To REST or not To REST

I wish I could write like Shakespeare but since I don't you are left with this blog about Representational State Transfer (REST) and specifically a discussion on whether the CloudStack API is a REST API or not. The short answer is that the CloudStack API is RESTlike but not RESTfull since it is only based on the GET method. Being an http based API that can return JSON does not make it a RESTfull API. This should not be seen as negative criticism but just a clarification.

A few words first about the CloudStack API and a few pointers. It is very extensive, from user creation, vm management to firewall configuration and more advanced networking features. Github has lots of clients written by the community and I am sure you can find your favorite language in there. The Developer Guide explains in details how to make requests. With CloudMonkey interacting with the API has become even easier. It is a terrific way to learn the API and check the required parameters of each call. Coupled with devcloud you can have a fully functional local CloudStack testbed.

To explore the API, I often also look at the GUI and see how the API calls are made. To do this, I access the UI in Firefox and launch Firebug console. I can then see the calls that I am interested in and specifically check the parameters/headers etc or the http call being made.

REST is known as an architectural style and to put it in french-english it is basically a way to design your API to create a web service that only uses the HTTP methods to manipulate the state of web resources. In my opinion it was really a response to the complexity of the SOAP based web services and the many standards that came out. REST really took off with Web 2.0 and was seen as a way to create easy to use web services. I recently found a few articles by Luis Rei and they clearly explain how to design a REST web service as well as implement on with Flask a lightweight python based web framework.

Looking closely at the CloudStack API we see that it uses http, can return JSON objects but only uses the GET method of HTTP. As such it is really a Query API that we can call RESTLike but not RESTfull similarly to the Amazon AWS EC2 Query API.

To illustrate this point I decided to write a REST wrapper on top of the CloudStack API. This was really the reason behind my CloudStack Silly Tuesday Hack. The actual usefulness of it is still questionable :) it would be better to re-write the CloudStack API into a REST native API. That being said "Cloud wrappers" like jclouds/deltacloud have adopted a stance where they expose an API on top of existing IaaS APIs. So who knows it may be useful.

First I wanted to be able to make http calls via curl and use the default GET based CloudStack API, then I wanted to make it more RESTFull. I decided to use Flask because it has a clean way to define web routes and specify the HTTP method being used. To do all of this I used DevCloud as mentioned above and I was checking all my calls with CloudMonkey. To make simple GET calls via curl and avoid constructing the signature of the calls, I took the requester.py file from CloudMonkey and imported it in my Flask application, checkout the gist for the entire code. Below is a sample "route". In this example we only consider user management.

@app.route('/list')
def list():
    print request.query_string
    res={}
    for key in request.args.iterkeys():
        res[key]=request.args.get(key)
    print res
    response, error = requester.make_request('listUsers',res,None,host,port,apikey,secretkey,protocol,path)
    return response

An http GET to http://localhost:5000/list will be routed to the list() function. The query parameters will be parsed and stored in a dictionary which in turn will be given to listUsers from CloudStack. This is where the silliness lies. A GET that does a GET. Using curl we can easily call this method:

curl -X GET -G 'http://localhost:5000/list'
{ "listusersresponse" : { "count":4 ,"user" : [  {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"sebgoa","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"62d866e4-da97-46c2-a3e1-10faf6197c73","username":"titi","firstname":"www","lastname":"rr","email":"joe@smith.com","created":"2013-04-15T06:21:59-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

In the response we see the list of all the users in the system. The same list route would also work if we were to pass the id of a single user:

curl -X GET -G 'http://localhost:5000/list' -d id=7ed6d5da-93b2-4545-a502-23d20b48ef2a
{ "listusersresponse" : { "count":1 ,"user" : [  {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

Where it becomes interesting is that deleting a user will be the same code but a different API call:

@app.route('/delete')
def delete():
    print request.query_string
    res={}
    for key in request.args.iterkeys():
        res[key]=request.args.get(key)
    print res
    response, error = requester.make_request('deleteUser',res,None,host,port,apikey,secretkey,protocol,path)
    return response

You would call it to delete a user by passing the id of the user like this:

curl -X GET -G 'http://localhost:5000/delete' -d id=62d866e4-da97-46c2-a3e1-10faf6197c73
{ "deleteuserresponse" : { "success" : "true"}  }

You are deleting a resource and yet, it is still a GET call. Why not using a DELETE call to delete a resource ? This would be more intuitive and a nice use of the HTTP grammar. The same situation will occur for updating a user or creating a user, still a GET while we could use a POST and a PATCH. In CloudStack-Flask I added some routes to create a proper REST service. For example a route becomes:

@app.route('/user/', methods=['GET','DELETE','PATCH'])
def user(uuid):
    if request.method =='GET':
        response, error = requester.make_request('listUsers',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
        return response
    elif request.method =='PATCH':
        data = request.json
        data['id']=uuid
        response, error = requester.make_request('updateUser',data,None,host,port,apikey,secretkey,protocol,path)
        return response
    else:
        response, error = requester.make_request('deleteUser',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
        return response

The id of the user is specified in the URL. The user is the actual web resource whose state we are changing. And the HTTP method is used to determine the type of action. In our case, the GET lists the user information, the PATCH updates the parameters of the user and the DELETE deletes the user. The id is now part of the URI and not a query parameters

curl -X GET http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "listusersresponse" : { "count":1 ,"user" : [  {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

curl -X  DELETE http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "deleteuserresponse" : { "success" : "true"}  }

curl -X  PATCH -H "Content-Type: application/json" http://localhost:5000/user/1fea6418-5576-4989-a21e-4790787bbee3 -d '{"firstname":"foobar"}'
{ "updateuserresponse" :  { "user" : {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"foobar","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} }  }

Identify your resources/entities and use the HTTP grammer to modify their state. Don't forget to check my cloudstack-flask app, it's very basic right now but could be a nice REST wrapper doubled-up with a bootstrap based UI, we will talk about the web views in the next post.

The Tuesday CloudStack Silly Hack

The problem with middleware/backend work is that nobody sees what you do and since I am terrible at graphics/design and know nothing of User Interface principles I am pretty much stuck in the dark. So today I invested couple hours on the Tuesday Silly CloudStack Hack. It is made of Flask, a good read, Twitter Bootstrap and some stolen code from CloudMonkey

Flask is a terrific web microframework for Python. Of course I like Python so I think Flask is terrific. It is also great because you can use it to design clean REST services. Bootstrap is en vogue these days, and CloudMonkey, also written in Python is the new CloudStack command line interface boasting some cool features, like auto-completion, interactive shell and so on.

I will keep it short, grab requester.py from the CloudMonkey source tree, it will help you make API calls to a CloudStack instance. Create a simple script with flask, and set your CloudStack endpoint variables, plus the keys (I am using DevCloud):

import requester

from flask import Flask, url_for, render_template, request
app = Flask(__name__)

apikey='plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg'
secretkey='VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ'
path='/client/api'
host='localhost'
port='8080'
protocol='http'

Setup a route that you will use to trigger a call to CloudStack. Something like this:

@app.route('/users')
def listusers():
    response, error = requester.make_request('listUsers',{},None,host,port,apikey,secretkey,protocol,path)
    resp=json.loads(str(response))
    return render_template('users.html',users=resp['listusersresponse'])

Now Download Bootstrap and stick it in a static directory in your Flask application. Then create html template files using the jinja2 syntax, something like this:

{% extends "base.html" %}

{% block content %}

{% if users %}
{{ users }}
{% else %}
Hello World!
{% endif %}

{% endblock content %}
sebmini:templates sebastiengoasguen$ 

Now run the app with Python and hit http://localhost:5000 and bang, you just got yourself your quarter end Bonus....! Well not quite, but that's a start :)

Joke aside, this should be the start of a fun Google Summer of Code project, I will put it on github if there is interest. Also if you want another silly hack, push requester.py to your android phone and using SL4a you can make calls to CloudStack from your phone...Silly CloudStack Wednesday hack anyone ?