How I look at Citrix...

I am closing in on my first year with Citrix working in the community team of Apache CloudStack, it's about time to describe how I look at Citrix and why I decided to leave a +10 year career in academia behind to work on Open Source...at Citrix. Disclaimer: if you don't want to read about Citrix products, stop right there.

Prior to joining Citrix I only knew it as the company that "brings windows to unix" delivering applications through an equivalent of remote desktop. It's a very unsavory and unfair summary at best and the XenDesktop folks will hate me for it :). What I quickly discovered is that Citrix has embraced the Cloud trying to deliver applications "anywhere, to anyone on any device", it is more than talk as I realized by going through the product list. This was especially intriguing to me because I did not understand why Citrix was getting into the Cloud Infrastructure as a Service (IaaS) layer area by acquiring cloud.com. IaaS software is more the purview of data centers vendors and Citrix struck me as a "top of the stack" company.

To understand why I think Citrix is a true Cloud company you need to get back to the Cloud definition that most of us refer to these days. The NIST definition, it defines Cloud as "...a model for enabling ubiquitous on-demand network access to a shared pool of network resources...that can rapidly be provisioned and released with minimal management effort or service provider interaction " it is the coming of age of utility computing where anyone can access any computing service at anytime from anywhere. Virtualization has been a key enabler of Cloud in the data center and the IaaS layer (or Fabric layer) has become the most mature of the Cloud layers. With Citrix acquiring Cloud.com and donating CloudStack to the Apache Software Foundation (ASF) it told me that Citrix was catching up and complementing its portfolio of products. What I did not expect is to realize that Citrix already had the other pieces of the Cloud puzzle building a suite of products that would enable on-demand/scalable network access to applications autonomically, in short: an app-focused cloud middleware.

At the fabric layer, resource sharing deals with computer, network and storage services. Server virtualization is a key component of IaaS and with XenServer Citrix has the leading hypervisor of the Cloud. To share compute resources, and manage farms of XenServer an orchestrator is needed. That's where CloudPlatform (commercial offering of Apache CloudStack) fits in. CloudPlatform orchestrates compute services and leverages existing storage and networking solutions. But here comes NetScaler, the networking piece, load-balancer, application firewall, branch repeater and cloud connector (via GRE tunnels, IPsec and so on). In storage, Sharefile is there. Unfortunately it is not an enterprise data center storage solution like today's large scale object store or traditional NAS solution but more of a DropBox equivalent (again not doing it justice) which includes corporate data governance.

At the Platform layer or PaaS, Citrix deals more with brokering and management of applications than a PaaS like Microsoft Azure or Google App Engine. CloudPortal is set to become a true Cloud broker offering a marketplace of applications, usage metering and billing making use of enterprise IaaS solutions to provision the applications in the datacenter. XenDesktop, XenApp and VDI-in-a-box are some of the solutions that CloudPlatform can broker. This will bring the traditional Citrix business to the Cloud (i.e Avalon project), providing on-demand, scalable, metered IT services.

At the SaaS layer, Citrix provides true collaboration services with it's GoTo* suite and Podio a social/collaborative platform. Surely these will soon be bundled in CloudPlatform for on-demand access. The nature of these apps speak to Citrix commitment to Clouds and a new way of working "anywhere, with anyone on any device" as they empower remote workers. And despite Yahoo's recent announcement about remote working I am very thankful for these apps and for Citrix walking the talk. To keep enabling mobility Citrix also offers additional middleware which I place in this SaaS layer even though they are not end-user application. XenMobileMDM and CloudGateway solve the problem of access management to application and data from the many devices that are now part of an enterprise IT.

Missing Pieces: This short mapping of Citrix products to the three basic layers of the Cloud highlights couple missing pieces. I am listing them here but I have no visibility and Citrix acquisition strategy so don't read anything else into it :). While Citrix has now a strong offering at the IaaS layer with XenServer, CloudPlatform (based on Apache CloudStack) and NetScaler. It does lack a true Cloud data center storage solution that would open the door to the Big Data market. Sharefile is more tailored for the individual end-user within the enterprise. Maybe packaging Apache Hadoop could fill this gap. In any case Citrix has strategic partnerships with NetApp and EMC as well as an innovative partner InkTank which is behind Ceph. At the PaaS layer, CloudPlatform is really a broker between app consumers and the cloud backends. A PaaS for application developers would be an interesting addition, even though these markets are still in the "trigger". A partnership with EMC on CloudFoundry and RedHat on OpenShift focused on integration with CloudStack/CloudPlatform would be interesting. I might also complement the middleware systems with a Data Analytics solution to mine one's data center and optimize application delivery, but that's my research hat thinking. The SaaS is a huge market and the goal is not to cover it all. I don't see any glaring gaps there, what strikes me the most at this layer is that the type of device we use will evolve quickly (e.g Google Glass), if one day people show up to work with "Glasses" what will Citrix Receiver be on Glass ?

Open Source: With this strong vision of Clouds, Citrix seemed perfect for me. What really made me accept the challenge of a new job was that Citrix open sourced CloudPlatform, donating it to the Apache Software Foundation. I had been an Open Source consumer (mostly) for a long time and it seemed right to get on the producer side. This was a true departure from its Microsoft ties, first by acquiring a data center software but second by reaching out to the Linux community. After the Xen Project this move spoke volume to me and was at the same time intriguing. Citrix has now stepped its Open Source efforts even more, putting the Xen Project under the stewardship of the Linux Foundation and being part of the OpenDayLight Software Defined Networking (SDN) consortium. There again IaaS is leading the way. I don't know yet what's in store for the other layers of the Cloud at Citrix, but if the recent Open Source activities are any signs, we can expect to see some Open Sourcing (in various forms: full software, API, community edition etc) at the other layers. This will further improve Citrix's reputation in the OSS community and help create an ecosystem around the terrific portfolio that we already have. Open Sourcing is key to innovation and we want Citrix to be a leading innovator of the Cloud.

In short, there is more to Clouds than virtual machine management and Citrix tackles the end-to-end cloud spectrum reaching out to the dream of computing as a utility in a mobile world.

To REST or not To REST

I wish I could write like Shakespeare but since I don't you are left with this blog about Representational State Transfer (REST) and specifically a discussion on whether the CloudStack API is a REST API or not. The short answer is that the CloudStack API is RESTlike but not RESTfull since it is only based on the GET method. Being an http based API that can return JSON does not make it a RESTfull API. This should not be seen as negative criticism but just a clarification.

A few words first about the CloudStack API and a few pointers. It is very extensive, from user creation, vm management to firewall configuration and more advanced networking features. Github has lots of clients written by the community and I am sure you can find your favorite language in there. The Developer Guide explains in details how to make requests. With CloudMonkey interacting with the API has become even easier. It is a terrific way to learn the API and check the required parameters of each call. Coupled with devcloud you can have a fully functional local CloudStack testbed.

To explore the API, I often also look at the GUI and see how the API calls are made. To do this, I access the UI in Firefox and launch Firebug console. I can then see the calls that I am interested in and specifically check the parameters/headers etc or the http call being made.

REST is known as an architectural style and to put it in french-english it is basically a way to design your API to create a web service that only uses the HTTP methods to manipulate the state of web resources. In my opinion it was really a response to the complexity of the SOAP based web services and the many standards that came out. REST really took off with Web 2.0 and was seen as a way to create easy to use web services. I recently found a few articles by Luis Rei and they clearly explain how to design a REST web service as well as implement on with Flask a lightweight python based web framework.

Looking closely at the CloudStack API we see that it uses http, can return JSON objects but only uses the GET method of HTTP. As such it is really a Query API that we can call RESTLike but not RESTfull similarly to the Amazon AWS EC2 Query API.

To illustrate this point I decided to write a REST wrapper on top of the CloudStack API. This was really the reason behind my CloudStack Silly Tuesday Hack. The actual usefulness of it is still questionable :) it would be better to re-write the CloudStack API into a REST native API. That being said "Cloud wrappers" like jclouds/deltacloud have adopted a stance where they expose an API on top of existing IaaS APIs. So who knows it may be useful.

First I wanted to be able to make http calls via curl and use the default GET based CloudStack API, then I wanted to make it more RESTFull. I decided to use Flask because it has a clean way to define web routes and specify the HTTP method being used. To do all of this I used DevCloud as mentioned above and I was checking all my calls with CloudMonkey. To make simple GET calls via curl and avoid constructing the signature of the calls, I took the requester.py file from CloudMonkey and imported it in my Flask application, checkout the gist for the entire code. Below is a sample "route". In this example we only consider user management.

@app.route('/list')
def list():
    print request.query_string
    res={}
    for key in request.args.iterkeys():
        res[key]=request.args.get(key)
    print res
    response, error = requester.make_request('listUsers',res,None,host,port,apikey,secretkey,protocol,path)
    return response

An http GET to http://localhost:5000/list will be routed to the list() function. The query parameters will be parsed and stored in a dictionary which in turn will be given to listUsers from CloudStack. This is where the silliness lies. A GET that does a GET. Using curl we can easily call this method:

curl -X GET -G 'http://localhost:5000/list'
{ "listusersresponse" : { "count":4 ,"user" : [  {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"sebgoa","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"62d866e4-da97-46c2-a3e1-10faf6197c73","username":"titi","firstname":"www","lastname":"rr","email":"joe@smith.com","created":"2013-04-15T06:21:59-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

In the response we see the list of all the users in the system. The same list route would also work if we were to pass the id of a single user:

curl -X GET -G 'http://localhost:5000/list' -d id=7ed6d5da-93b2-4545-a502-23d20b48ef2a
{ "listusersresponse" : { "count":1 ,"user" : [  {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

Where it becomes interesting is that deleting a user will be the same code but a different API call:

@app.route('/delete')
def delete():
    print request.query_string
    res={}
    for key in request.args.iterkeys():
        res[key]=request.args.get(key)
    print res
    response, error = requester.make_request('deleteUser',res,None,host,port,apikey,secretkey,protocol,path)
    return response

You would call it to delete a user by passing the id of the user like this:

curl -X GET -G 'http://localhost:5000/delete' -d id=62d866e4-da97-46c2-a3e1-10faf6197c73
{ "deleteuserresponse" : { "success" : "true"}  }

You are deleting a resource and yet, it is still a GET call. Why not using a DELETE call to delete a resource ? This would be more intuitive and a nice use of the HTTP grammar. The same situation will occur for updating a user or creating a user, still a GET while we could use a POST and a PATCH. In CloudStack-Flask I added some routes to create a proper REST service. For example a route becomes:

@app.route('/user/', methods=['GET','DELETE','PATCH'])
def user(uuid):
    if request.method =='GET':
        response, error = requester.make_request('listUsers',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
        return response
    elif request.method =='PATCH':
        data = request.json
        data['id']=uuid
        response, error = requester.make_request('updateUser',data,None,host,port,apikey,secretkey,protocol,path)
        return response
    else:
        response, error = requester.make_request('deleteUser',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
        return response

The id of the user is specified in the URL. The user is the actual web resource whose state we are changing. And the HTTP method is used to determine the type of action. In our case, the GET lists the user information, the PATCH updates the parameters of the user and the DELETE deletes the user. The id is now part of the URI and not a query parameters

curl -X GET http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "listusersresponse" : { "count":1 ,"user" : [  {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

curl -X  DELETE http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "deleteuserresponse" : { "success" : "true"}  }

curl -X  PATCH -H "Content-Type: application/json" http://localhost:5000/user/1fea6418-5576-4989-a21e-4790787bbee3 -d '{"firstname":"foobar"}'
{ "updateuserresponse" :  { "user" : {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"foobar","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} }  }

Identify your resources/entities and use the HTTP grammer to modify their state. Don't forget to check my cloudstack-flask app, it's very basic right now but could be a nice REST wrapper doubled-up with a bootstrap based UI, we will talk about the web views in the next post.

The Tuesday CloudStack Silly Hack

The problem with middleware/backend work is that nobody sees what you do and since I am terrible at graphics/design and know nothing of User Interface principles I am pretty much stuck in the dark. So today I invested couple hours on the Tuesday Silly CloudStack Hack. It is made of Flask, a good read, Twitter Bootstrap and some stolen code from CloudMonkey

Flask is a terrific web microframework for Python. Of course I like Python so I think Flask is terrific. It is also great because you can use it to design clean REST services. Bootstrap is en vogue these days, and CloudMonkey, also written in Python is the new CloudStack command line interface boasting some cool features, like auto-completion, interactive shell and so on.

I will keep it short, grab requester.py from the CloudMonkey source tree, it will help you make API calls to a CloudStack instance. Create a simple script with flask, and set your CloudStack endpoint variables, plus the keys (I am using DevCloud):

import requester

from flask import Flask, url_for, render_template, request
app = Flask(__name__)

apikey='plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg'
secretkey='VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ'
path='/client/api'
host='localhost'
port='8080'
protocol='http'

Setup a route that you will use to trigger a call to CloudStack. Something like this:

@app.route('/users')
def listusers():
    response, error = requester.make_request('listUsers',{},None,host,port,apikey,secretkey,protocol,path)
    resp=json.loads(str(response))
    return render_template('users.html',users=resp['listusersresponse'])

Now Download Bootstrap and stick it in a static directory in your Flask application. Then create html template files using the jinja2 syntax, something like this:

{% extends "base.html" %}

{% block content %}

{% if users %}
{{ users }}
{% else %}
Hello World!
{% endif %}

{% endblock content %}
sebmini:templates sebastiengoasguen$ 

Now run the app with Python and hit http://localhost:5000 and bang, you just got yourself your quarter end Bonus....! Well not quite, but that's a start :)

Joke aside, this should be the start of a fun Google Summer of Code project, I will put it on github if there is interest. Also if you want another silly hack, push requester.py to your android phone and using SL4a you can make calls to CloudStack from your phone...Silly CloudStack Wednesday hack anyone ?