How I look at Citrix...

I am closing in on my first year with Citrix working in the community team of Apache CloudStack, it's about time to describe how I look at Citrix and why I decided to leave a +10 year career in academia behind to work on Open Source...at Citrix. Disclaimer: if you don't want to read about Citrix products, stop right there.

Prior to joining Citrix I only knew it as the company that "brings windows to unix" delivering applications through an equivalent of remote desktop. It's a very unsavory and unfair summary at best and the XenDesktop folks will hate me for it :). What I quickly discovered is that Citrix has embraced the Cloud trying to deliver applications "anywhere, to anyone on any device", it is more than talk as I realized by going through the product list. This was especially intriguing to me because I did not understand why Citrix was getting into the Cloud Infrastructure as a Service (IaaS) layer area by acquiring cloud.com. IaaS software is more the purview of data centers vendors and Citrix struck me as a "top of the stack" company.

To understand why I think Citrix is a true Cloud company you need to get back to the Cloud definition that most of us refer to these days. The NIST definition, it defines Cloud as "...a model for enabling ubiquitous on-demand network access to a shared pool of network resources...that can rapidly be provisioned and released with minimal management effort or service provider interaction " it is the coming of age of utility computing where anyone can access any computing service at anytime from anywhere. Virtualization has been a key enabler of Cloud in the data center and the IaaS layer (or Fabric layer) has become the most mature of the Cloud layers. With Citrix acquiring Cloud.com and donating CloudStack to the Apache Software Foundation (ASF) it told me that Citrix was catching up and complementing its portfolio of products. What I did not expect is to realize that Citrix already had the other pieces of the Cloud puzzle building a suite of products that would enable on-demand/scalable network access to applications autonomically, in short: an app-focused cloud middleware.

At the fabric layer, resource sharing deals with computer, network and storage services. Server virtualization is a key component of IaaS and with XenServer Citrix has the leading hypervisor of the Cloud. To share compute resources, and manage farms of XenServer an orchestrator is needed. That's where CloudPlatform (commercial offering of Apache CloudStack) fits in. CloudPlatform orchestrates compute services and leverages existing storage and networking solutions. But here comes NetScaler, the networking piece, load-balancer, application firewall, branch repeater and cloud connector (via GRE tunnels, IPsec and so on). In storage, Sharefile is there. Unfortunately it is not an enterprise data center storage solution like today's large scale object store or traditional NAS solution but more of a DropBox equivalent (again not doing it justice) which includes corporate data governance.

At the Platform layer or PaaS, Citrix deals more with brokering and management of applications than a PaaS like Microsoft Azure or Google App Engine. CloudPortal is set to become a true Cloud broker offering a marketplace of applications, usage metering and billing making use of enterprise IaaS solutions to provision the applications in the datacenter. XenDesktop, XenApp and VDI-in-a-box are some of the solutions that CloudPlatform can broker. This will bring the traditional Citrix business to the Cloud (i.e Avalon project), providing on-demand, scalable, metered IT services.

At the SaaS layer, Citrix provides true collaboration services with it's GoTo* suite and Podio a social/collaborative platform. Surely these will soon be bundled in CloudPlatform for on-demand access. The nature of these apps speak to Citrix commitment to Clouds and a new way of working "anywhere, with anyone on any device" as they empower remote workers. And despite Yahoo's recent announcement about remote working I am very thankful for these apps and for Citrix walking the talk. To keep enabling mobility Citrix also offers additional middleware which I place in this SaaS layer even though they are not end-user application. XenMobileMDM and CloudGateway solve the problem of access management to application and data from the many devices that are now part of an enterprise IT.

Missing Pieces: This short mapping of Citrix products to the three basic layers of the Cloud highlights couple missing pieces. I am listing them here but I have no visibility and Citrix acquisition strategy so don't read anything else into it :). While Citrix has now a strong offering at the IaaS layer with XenServer, CloudPlatform (based on Apache CloudStack) and NetScaler. It does lack a true Cloud data center storage solution that would open the door to the Big Data market. Sharefile is more tailored for the individual end-user within the enterprise. Maybe packaging Apache Hadoop could fill this gap. In any case Citrix has strategic partnerships with NetApp and EMC as well as an innovative partner InkTank which is behind Ceph. At the PaaS layer, CloudPlatform is really a broker between app consumers and the cloud backends. A PaaS for application developers would be an interesting addition, even though these markets are still in the "trigger". A partnership with EMC on CloudFoundry and RedHat on OpenShift focused on integration with CloudStack/CloudPlatform would be interesting. I might also complement the middleware systems with a Data Analytics solution to mine one's data center and optimize application delivery, but that's my research hat thinking. The SaaS is a huge market and the goal is not to cover it all. I don't see any glaring gaps there, what strikes me the most at this layer is that the type of device we use will evolve quickly (e.g Google Glass), if one day people show up to work with "Glasses" what will Citrix Receiver be on Glass ?

Open Source: With this strong vision of Clouds, Citrix seemed perfect for me. What really made me accept the challenge of a new job was that Citrix open sourced CloudPlatform, donating it to the Apache Software Foundation. I had been an Open Source consumer (mostly) for a long time and it seemed right to get on the producer side. This was a true departure from its Microsoft ties, first by acquiring a data center software but second by reaching out to the Linux community. After the Xen Project this move spoke volume to me and was at the same time intriguing. Citrix has now stepped its Open Source efforts even more, putting the Xen Project under the stewardship of the Linux Foundation and being part of the OpenDayLight Software Defined Networking (SDN) consortium. There again IaaS is leading the way. I don't know yet what's in store for the other layers of the Cloud at Citrix, but if the recent Open Source activities are any signs, we can expect to see some Open Sourcing (in various forms: full software, API, community edition etc) at the other layers. This will further improve Citrix's reputation in the OSS community and help create an ecosystem around the terrific portfolio that we already have. Open Sourcing is key to innovation and we want Citrix to be a leading innovator of the Cloud.

In short, there is more to Clouds than virtual machine management and Citrix tackles the end-to-end cloud spectrum reaching out to the dream of computing as a utility in a mobile world.

To REST or not To REST

I wish I could write like Shakespeare but since I don't you are left with this blog about Representational State Transfer (REST) and specifically a discussion on whether the CloudStack API is a REST API or not. The short answer is that the CloudStack API is RESTlike but not RESTfull since it is only based on the GET method. Being an http based API that can return JSON does not make it a RESTfull API. This should not be seen as negative criticism but just a clarification.

A few words first about the CloudStack API and a few pointers. It is very extensive, from user creation, vm management to firewall configuration and more advanced networking features. Github has lots of clients written by the community and I am sure you can find your favorite language in there. The Developer Guide explains in details how to make requests. With CloudMonkey interacting with the API has become even easier. It is a terrific way to learn the API and check the required parameters of each call. Coupled with devcloud you can have a fully functional local CloudStack testbed.

To explore the API, I often also look at the GUI and see how the API calls are made. To do this, I access the UI in Firefox and launch Firebug console. I can then see the calls that I am interested in and specifically check the parameters/headers etc or the http call being made.

REST is known as an architectural style and to put it in french-english it is basically a way to design your API to create a web service that only uses the HTTP methods to manipulate the state of web resources. In my opinion it was really a response to the complexity of the SOAP based web services and the many standards that came out. REST really took off with Web 2.0 and was seen as a way to create easy to use web services. I recently found a few articles by Luis Rei and they clearly explain how to design a REST web service as well as implement on with Flask a lightweight python based web framework.

Looking closely at the CloudStack API we see that it uses http, can return JSON objects but only uses the GET method of HTTP. As such it is really a Query API that we can call RESTLike but not RESTfull similarly to the Amazon AWS EC2 Query API.

To illustrate this point I decided to write a REST wrapper on top of the CloudStack API. This was really the reason behind my CloudStack Silly Tuesday Hack. The actual usefulness of it is still questionable :) it would be better to re-write the CloudStack API into a REST native API. That being said "Cloud wrappers" like jclouds/deltacloud have adopted a stance where they expose an API on top of existing IaaS APIs. So who knows it may be useful.

First I wanted to be able to make http calls via curl and use the default GET based CloudStack API, then I wanted to make it more RESTFull. I decided to use Flask because it has a clean way to define web routes and specify the HTTP method being used. To do all of this I used DevCloud as mentioned above and I was checking all my calls with CloudMonkey. To make simple GET calls via curl and avoid constructing the signature of the calls, I took the requester.py file from CloudMonkey and imported it in my Flask application, checkout the gist for the entire code. Below is a sample "route". In this example we only consider user management.

@app.route('/list')
def list():
    print request.query_string
    res={}
    for key in request.args.iterkeys():
        res[key]=request.args.get(key)
    print res
    response, error = requester.make_request('listUsers',res,None,host,port,apikey,secretkey,protocol,path)
    return response

An http GET to http://localhost:5000/list will be routed to the list() function. The query parameters will be parsed and stored in a dictionary which in turn will be given to listUsers from CloudStack. This is where the silliness lies. A GET that does a GET. Using curl we can easily call this method:

curl -X GET -G 'http://localhost:5000/list'
{ "listusersresponse" : { "count":4 ,"user" : [  {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"sebgoa","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"}, {"id":"62d866e4-da97-46c2-a3e1-10faf6197c73","username":"titi","firstname":"www","lastname":"rr","email":"joe@smith.com","created":"2013-04-15T06:21:59-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

In the response we see the list of all the users in the system. The same list route would also work if we were to pass the id of a single user:

curl -X GET -G 'http://localhost:5000/list' -d id=7ed6d5da-93b2-4545-a502-23d20b48ef2a
{ "listusersresponse" : { "count":1 ,"user" : [  {"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a","username":"admin","firstname":"admin","lastname":"cloud","created":"2012-07-05T12:18:27-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg","secretkey":"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

Where it becomes interesting is that deleting a user will be the same code but a different API call:

@app.route('/delete')
def delete():
    print request.query_string
    res={}
    for key in request.args.iterkeys():
        res[key]=request.args.get(key)
    print res
    response, error = requester.make_request('deleteUser',res,None,host,port,apikey,secretkey,protocol,path)
    return response

You would call it to delete a user by passing the id of the user like this:

curl -X GET -G 'http://localhost:5000/delete' -d id=62d866e4-da97-46c2-a3e1-10faf6197c73
{ "deleteuserresponse" : { "success" : "true"}  }

You are deleting a resource and yet, it is still a GET call. Why not using a DELETE call to delete a resource ? This would be more intuitive and a nice use of the HTTP grammar. The same situation will occur for updating a user or creating a user, still a GET while we could use a POST and a PATCH. In CloudStack-Flask I added some routes to create a proper REST service. For example a route becomes:

@app.route('/user/', methods=['GET','DELETE','PATCH'])
def user(uuid):
    if request.method =='GET':
        response, error = requester.make_request('listUsers',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
        return response
    elif request.method =='PATCH':
        data = request.json
        data['id']=uuid
        response, error = requester.make_request('updateUser',data,None,host,port,apikey,secretkey,protocol,path)
        return response
    else:
        response, error = requester.make_request('deleteUser',{'id':uuid},None,host,port,apikey,secretkey,protocol,path)
        return response

The id of the user is specified in the URL. The user is the actual web resource whose state we are changing. And the HTTP method is used to determine the type of action. In our case, the GET lists the user information, the PATCH updates the parameters of the user and the DELETE deletes the user. The id is now part of the URI and not a query parameters

curl -X GET http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "listusersresponse" : { "count":1 ,"user" : [  {"id":"b3b60a8d-df6f-4ce6-a6f9-6194907457a5","username":"john","firstname":"sebgoa","lastname":"goa","email":"runseb@gmail.com","created":"2013-04-12T05:09:10-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }

curl -X  DELETE http://localhost:5000/user/b3b60a8d-df6f-4ce6-a6f9-6194907457a5
{ "deleteuserresponse" : { "success" : "true"}  }

curl -X  PATCH -H "Content-Type: application/json" http://localhost:5000/user/1fea6418-5576-4989-a21e-4790787bbee3 -d '{"firstname":"foobar"}'
{ "updateuserresponse" :  { "user" : {"id":"1fea6418-5576-4989-a21e-4790787bbee3","username":"runseb","firstname":"foobar","lastname":"goa","email":"joe@smith.com","created":"2013-04-10T16:52:06-0700","state":"enabled","account":"admin","accounttype":1,"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77","domain":"ROOT","apikey":"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA","secretkey":"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w","accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} }  }

Identify your resources/entities and use the HTTP grammer to modify their state. Don't forget to check my cloudstack-flask app, it's very basic right now but could be a nice REST wrapper doubled-up with a bootstrap based UI, we will talk about the web views in the next post.

The Tuesday CloudStack Silly Hack

The problem with middleware/backend work is that nobody sees what you do and since I am terrible at graphics/design and know nothing of User Interface principles I am pretty much stuck in the dark. So today I invested couple hours on the Tuesday Silly CloudStack Hack. It is made of Flask, a good read, Twitter Bootstrap and some stolen code from CloudMonkey

Flask is a terrific web microframework for Python. Of course I like Python so I think Flask is terrific. It is also great because you can use it to design clean REST services. Bootstrap is en vogue these days, and CloudMonkey, also written in Python is the new CloudStack command line interface boasting some cool features, like auto-completion, interactive shell and so on.

I will keep it short, grab requester.py from the CloudMonkey source tree, it will help you make API calls to a CloudStack instance. Create a simple script with flask, and set your CloudStack endpoint variables, plus the keys (I am using DevCloud):

import requester

from flask import Flask, url_for, render_template, request
app = Flask(__name__)

apikey='plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg'
secretkey='VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ'
path='/client/api'
host='localhost'
port='8080'
protocol='http'

Setup a route that you will use to trigger a call to CloudStack. Something like this:

@app.route('/users')
def listusers():
    response, error = requester.make_request('listUsers',{},None,host,port,apikey,secretkey,protocol,path)
    resp=json.loads(str(response))
    return render_template('users.html',users=resp['listusersresponse'])

Now Download Bootstrap and stick it in a static directory in your Flask application. Then create html template files using the jinja2 syntax, something like this:

{% extends "base.html" %}

{% block content %}

{% if users %}
{{ users }}
{% else %}
Hello World!
{% endif %}

{% endblock content %}
sebmini:templates sebastiengoasguen$ 

Now run the app with Python and hit http://localhost:5000 and bang, you just got yourself your quarter end Bonus....! Well not quite, but that's a start :)

Joke aside, this should be the start of a fun Google Summer of Code project, I will put it on github if there is interest. Also if you want another silly hack, push requester.py to your android phone and using SL4a you can make calls to CloudStack from your phone...Silly CloudStack Wednesday hack anyone ?

Quantifying the Apache CloudStack Community

CloudStack is now an Apache Top Level Project (TLP) at the Apache Software Foundation (ASF), the announcement just came out. What the incubation period has meant for CloudStack has nothing to do with code maturity. CloudStack was mature and used in enterprise settings before it entered incubation at the ASF. What incubation meant was that CloudStack evolved into an open source community, self-governed by the Apache Way: transparency, meritocracy, respect, non-affiliation and consensus in no particular order. The community has learned and demonstrated that it understands the principles and processes laid by the Apache Software Foundation and that it can now operate more autonomously.

Growing an open source community is challenging, folks who participate come from various backgrounds, may seldom meet and interact mostly via emails, social media, instant messaging. Participants come from all over the world, work in different time-zones and donate their time after (and sometime while) dealing with day jobs and family. Participants rally around a project that they deem interesting, sometime just to land a hand for a few months, or sometimes because their day job requires them to do it. In that very heterogeneous and fluctuating mix, an open source software community emerges, self-governed, sustainable and non-affiliated. In the last 12 months CloudStack has done just that, building a community from the ground up, developing and understanding the principles laid out in our bylaws, adapting -if need be- people's way of developing software, getting to know each other, welcoming new members every day and setting the foundation for a sustainable software.

When growing a community it is fairly natural to want to measure how well we are doing and how healthy the community is. Over the last several months I have started collecting some data to analyze our community, trying to see how we were doing and interacting. I mostly looked at our public mailing lists doing a study similar to the one done about comparing CloudStack, OpenNebula, OpenStack and Ecualyptus. Secretly, this was also a good way for me to sharpen a few skills on BigData, not that big actually but I used MongoDB instead of MySQL so that qualifies as BigData :). Defining membership in an open source community is a challenge since there is no concept of membership, even the concept of contribution is ill-defined. What constitutes a contribution ? Which channels need to be considered ? In the case of ASF for instance, contribution to code may only mean being a committer, but a committer is someone with write access to the code. Just counting committers will leave out all the folks sending patches, doing testing, doing user support, translating documentation, giving talks and so on. Also while at the ASF everything happens on the mailing list, what about IRC channels, social media like Facebook, Twitter and Linkedin, and what about other communities that may arise around a particular software: sub-projects, user groups etc. In this analysis I decided to only look at our public mailing lists but there is more to it than just this data source.

The two figures below show the number of individual contributors measured by unique email addresses used to send messages to the users and developers mailing lists. The red lines represent data from the users and developers mailing list prior to entering incubation at the ASF. The blue lines represent the ASF specific lists. Significant is the impact that the move to the ASF has had on the number of contributors. The developers list has peaked over 200 per month and the users list has peaked over 150 per month so far (figure on the right). The last data point is March (as of March 21st) and numbers will go up by the end of the month. The graph on the right shows the accumulation of contributors, adding all unique email addresses every month into a set. This shows us again that the move to ASF has had a huge impact on the growth rate of the community and that both list grow at relatively the same pace. Adding the accumulated number of contributors to both list and removing duplicates present in both sets, this gives us a magic number of 722 CloudStack contributors to date.

As mentioned earlier these numbers of contributors are different from the number of Apache CloudStack committers. According to our bylaws contributors can become committers and gain write access to the code when invited by the Project Management Committee (PMC). Just today we welcomed three new committers for a current total of 54. This number is again relative in terms of contribution to code, since committers apply patches from contributor who are not yet committers (I know confusing :) ). A quick look at our git repo, shows a current total of 159 code contributors. To that we could also add the 32 contributors who helped with the translation via transifex which may or may not have participated in the mailing lists.

These numbers show a growing community sparked by the move to the ASF. While at the start of incubation the initial set of committers and contributors where only from Citrix who donated the code, we are also seeing a diversification in the number of companies involved. Talking about affiliation is actually a cardinal sin of the Apache Way. At the ASF only individuals matter, but it is fair to say that for CloudStack to be successful we need to see adoption/participation by a diverse set of companies. The two figures below show just that with the number of companies identified by the email domain used by the contributors. This is not perfect since contributors often do not use their work email but gives a good idea of the trend. Similarly to the plots about contributors we plot the number of companies (really email domains). Clearly the growth in the number of companies involved has increased since joining the ASF, we currently see around 50 companies involved in CloudStack every month. Accumulating all these companies we reach almost 200 on both lists. Removing duplicates from both sets we estimate the total number of companies to 272. Even if some of those companies are only represented by a single individual this is still a very strong number that shows great diversification. Interestingly over the last three months we see that the participation in the user list has shown more diversity than on the developer list.

Finally, looking at the cloudstack-commits mailing list we try to identify the number of committers to the code. The figure below shows the number of committers and the number of commits measured by number of emails and unique email addresses used on the commits mailing list. We already mentioned that we had 54 committers but that some of those committers gained their rights based on contribution which could be as diverse as development, user support, translation etc. Therefore it is no surprise to see the number of committers peaking at 35 and the accumulated number reaching 45. The number of commits is growing, reaching 1500 a month during the 4.0 release, easing up during christmas time and peaking again over 2500 per month prior to code freeze of the 4.1 release.

All in all, analyzing lots of emails showed me that moving to ASF has had a tremendous impact on CloudStack, with over 700 contributors, close to 60 committers, 30 translators, over 1 million lines of code, CloudStack is self-governed and here to stay.

Security in the Cloud and the CCSK

Search for cloud computing and you will get approximately 190 million results, search for cloud computing security and you will get 120 million results. This is very rough data of course but it gives us an idea that when talking about Cloud, security is a big concern. Go to a conference and talk about Cloud, and you can be certain that one of the big questions you will get asked is "But what about Security ?"

Disclaimer and bias: This question always leaves me pondering, mostly because my personal background and bias always makes me wonder what people are afraid off in the Cloud and what do they see that Cloud brings to bear that is different from any existing distributed systems running over the internet. I am not an enterprise security expert, I used to teach an introductory course on network security, but I have spent my fair share thinking about Clouds especially at the IaaS layer. There, the new technology that could represent a new attack vector is virtualization and I only read about two non-traditional efforts that really challenged the security of virtualization: the controversial bluepill project in 2006 and the cross-VM side channel attack reported by a research group at MIT in 2009 (there are of course more...). Most problems publicly described with IaaS have been with spam and DDOS. Where on one hand cloud providers are being used to send spam and on the other hand cloud providers are victim of DDOS threatening the availability of services.

However, in the fall I had the chance to participate in the DELL in the Clouds Think Tank in London. It is there that I started to understand that what most people where worried about with the Cloud had more to do with legal issues, governance, compliance and contracts than hardcore attacks. Indeed when dealing with a cloud provider you are exposing your data to new risks for the simple fact that it is not under your total control and you need to manage those risks. Moving your data out of your secured premises and putting them in the hands of another party exposes you to new threats. This is the core of information assurance and risk management. Cloud security is therefore more about updating your security guidelines, making sure that you are compliant with the law and being confident that you can respond appropriately to any attack or business continuity issues. Cloud security is less about the fear of a new technology that exposes new attack vectors. The risks may be new to your enterprise but the attacks and vulnerabilities used are not new to the internet.

To learn more and come up with a plan I now point people to the Cloud Security Alliance (CSA) and their guidelines. It is a 176 pages document which coupled with the ENISA cloud security assessment (125 pages :)) forms the basis of the CSA Certificate of Cloud Security Knowledge (CCSK). I have finished reading the CSA guidelines and once I read the ENISA report I will take the CCSK exam.

The CSA guidelines are a set of reports covering fourteen domains of interest to Cloud security. From Governance and Legal Issues to Incident Response and Virtualization (to name a few). One sentence truly resonated with me due to my personal bias explained earlier. It is in the Application Security domain chapter which states: "Cloud-based software applications require a design rigor similar to an application connecting to the raw internet - the security must be provided by the application without any assumptions being made about the external environment" indeed doing the opposite would be one of the fallacies of distributed systems design enunciated by Peter Deutsch from SUN. There lies in my view the biggest risk, thinking that you can take an application that has been designed in-house assuming a secure local network and wanting to move it to the cloud as-is not managing the risks due to the fact that a) the network is not secure b) bandwidth is not infinite c) latency is not zero d) transport has a cost.

Any service, application, provider, data that is accessible over the public internet is being attacked and is subject to risk of being stolen, tempered with, disrupted and even shutdown. This is not fear mongering, it is just a fact and if you design an application or use a service thinking otherwise you will be exposing yourself and not managing risks properly. Similarly to the new cloud software being developed (e.g Hadoop, Cassandra, CloudStack), that are designed assuming failures of components, when moving to the Cloud one needs to assume attacks and unsecured networks. This is not saying that the Cloud is unsecure, this is saying that you need to adopt the proper security posture, a different one than if you have been operating under the -at least perceived- warmth and coziness of a secured local network.

Getting back to the CSA guidelines, the first domain Cloud Architecture is the perfect introduction to Cloud with reference to the NIST definition of cloud computing. It then follows with what I think is the most important section: Governing in the Cloud, it presents risk management as key to an enterprise governance and introduces legal issues and compliance management as it pertains to Cloud. The chapters help to define the proper security posture, defining or updating security policies that will make sense for Cloud use, understanding the assets that will be at risk and understanding if and how compliance will be enforced. As such it is not specific to Cloud computing, it is really best practices of risk management and understanding the contracts being signed with the cloud providers. Will those contracts expose you ? Do providers follow data protection standards ? Are the providers subject to any laws that may expose you (e.g Patriot Act) ? How can you remedy those risks ? Which providers can give you the compliance you need ? To help with these decisions, CSA created the Security Trust and Assurance Registry (STAR). Cloud providers who participate in the registry submit answers to a questionnaire that lists the standard they follow in 99 categories from audit and compliance to operations, business continuity, human resources, forensics. This registry is key to choose cloud providers that will match your security and governance needs.

The last section of the CSA guidelines is about Operating in the Cloud. From Disaster recovery, data center operations, incident response to encryption, authentication and virtualization. This section is not specific to cloud but comes into play in selecting providers to ensure for example, that the provider data center operations matches your requirements. Or to ensure that in case of incidents you will have access to the logs (defining which ones in a contract). I was pleased to see John Kinsella (@johnlkinsella) from Stratosec as one of the authors of the chapter on Application Security. John is a member of the CloudStack Project Management Committer (PMC). I was also happily surprised to see a chapter on Security as a Service something that Mice Xia from tcloudcomputing and a committer on CloudStack has been working on.

To summarize, knowledge is power (a bit cheezy I know). When moving to the Cloud, an enterprise should engage their security experts from the on-set making sure that risk is managed and that everything is in compliance, this is standard information assurance and part of good enterprise governance. When negotiating (or not negotiating) contracts with Cloud Providers the STAR registry can help choose the providers that will best match the requirements of the enterprise. I am not being paid by CSA or SANS, but I would recommend people to get the CCSK certification and probably also a SANS course on Cloud Security. In my opinion the cloud is not less secure than anything else. The Cloud (at least in its public form) is about accessing resources over the network and locating assets off-premises, this intrinsically presents risks but it is manageable risk that needs to be part of a design. You need to design for attacks, test your designs, monitor, counter-strike. Basic warfare. And remember, the network is secure...Oupss...sorry !

Activeeon ProActive integrates with Apache CloudStack

Since CloudStack entered incubation at the Apache Software Foundation, there has been lots of work in integrating existing software solutions with Apache CloudStack (ACS). On the networking side we have seen integration with Nicira, BigSwitch BVS, VMWare dvSwitch, Midokura Midonet. On the storage side we have seen integration with Ceph, Riak CS, Caringo and more recently Solidfire. All of these integrations are either already present in the 4.0 release, set for 4.1 at the end of March or in the works for the 4.2 release this summer. Most of these integration efforts need some tight integration with the CloudStack code, developing plugins, writing new classes, potentially defining new orchestration steps, and adding UI interaction. In this post, I want to introduce an integration with Proactive from a french company called Activeeon (@activeeon). They treated ACS as a black box and integrated with it using the default exposed API. A very powerful mechanism to integrate existing solutions and enterprise workflows with a private or public cloud.

Activeeon is a company that originated from an INRIA research lab (INRIA is the leading computer research organization in French , and a CloudStack user in their continuous integration department). One of their solutions, Proactive is an open source software available at the OW2 consortium. Proactive is an advanced workflow manager that combines a powerful IDE, a workflow engine and a resource manager. It aims to take complex computational workflows and ease their execution on distributed resources, such as HPC cluster, desktop grids and clouds. Existing applications using Proactive are from a diverse set of industries such as the financial, biological and automotive industries. Integrating with Clouds allows ProActive to dynamically provision resources to execute a workflow based on a set of pre-defined policies and constraints that are up to the user. Activeeon had developed a Amazon EC2 resource plugin and it made complete sense to integrate with Apache CloudStack either through the EC2 mapping and even directly via the ACS native API. Below I embed slides from Brian Amedro (@brianamedro) describing the integration, as well as a video demoing it live.

Practically speaking, how did the integration happen ? According to @brianamedro it took 1 person five days. They used the ACS sandbox DevCloud to stand up a live ACS cloud. They modified the template used in DevCloud to make it a Proactive nodesource template. To call the API they used an existing library which even though it was created for older CloudStack release still worked with ACS 4.0. The resulting java code was open sourced on the OW2 repository. Looking ahead, Activeeon wants to test the scalability of the CloudStack connector on thousand nodes cloud and add a stronger integration especially in terms of identity management and auto-scaling.

What their work showed me is that out of the box integration of existing middleware systems with CloudStack is quite straightforward. The rich ACS API opens the door for very powerful couplings and a very extensive CloudStack ecosystem. To top it off, I really appreciated that they developed straight up in Open source mode and that their work is available on the OW2 consortium forge.

Translating Apache CloudStack

We are coming down the stretch to submit translations for the Apache CloudStack Documentation. The 4.1 release should be cut on March 22nd, we need the translations in before that. It is a huge task, we need your help, here is how:

How to Translate Apache CloudStack Docs from Sebastien Goasguen

SDN in CloudStack

Software Defined Networking (SDN) has seen a lot of uptick in momentum since VMware acquired Nicira last summer, three months after Google announced that they were using OpenFlow to optimize their internal backbone. In this post we look at "SDN" support in CloudStack.

First, let's try to define SDN in a short paragraph. It is, was it spells out to be (like the french adage: "c'est comme le Port-Salut c'est écrit dessus" :)) a way to configure the network using software. This means that the network definition (routing, switching), optimization (load-balancing, firewall, etc) becomes a software problem with SDN. If you have seen the wikipedia page and read other articles, you most likely have read that SDN decouples the control plan and the data plane. This is short for saying that the forwarding tables used in switches/routers will be controlled by a software/applications that can be remote. Part of the SDN landscape is OpenFlow. OpenFlow is a standard defined by the ONF that allows you to implement a SDN solution. It defines the protocol used by the control plane to send forwarding information (a.k.a flow rules) to network devices. However while some early SDN companies embraced and led the development of openflow (e.g BigSwitch), an SDN solution may not use the OpenFlow protocol. This leads to a key point of todays SDN solutions: SDN != OpenFlow.

The academic in me can't help but point out the GENI initiative, which aims to re-design the internet and start from a clean slate. SDN research happens on GENI but it is also seen as a way to instrument the network, isolate experiments and dynamically reconfigure the network. Something impossible (especially over wide are networks) before SDN. With FutureGrid and Grid5000 being testbeds for IaaS solutions, GENI is a real-life testbed for future networking solutions. I had a chance to work a little bit on GENI while at Clemson. We modified the NOX OpenFlow controller to integrate it with OpenNebula and provide Security Groups as well as Elastic IP functionality.

While virtualization was a key enabler of IaaS, virtual switches are key enablers of SDN based networks. Open Virtual Switch (OVS) is the leading virtual switch. OVS is now used in most private and public clouds, it replaces the standard linux bridge and is used to connect virtual machine network interfaces to the physical network. An OVS can be connected with an OpenFlow controller and receive flow rules from the controller. However it does not need to, an SDN solution could talk directly to OVS. The main issue being that a single openflow controller may not be fast enough to process all "control" decisions on a large networks. We would have to see a distributed OpenFlow controller to be able to reach extremely large scale (It might exist, I just have not found references for it). OVS can do many things among them: VLAN tagging, QoS, Generic Routing Encapsulation(GRE) and Stateless Transport Tunneling (STT) tunnels. To know more about the difference between GRE and STT see this blog by Bruce Davie and Andrew Lambeth.

So where does SDN help IaaS ? Anyone who has worked on networking of virtual machines (VM) knows how complex this can get. VMs from multiple tenants need to be isolated from each other, they have private IP addresses but may need to be accessed form the public internet, VMs can be migrated within a single broadcast domain but one may want to migrate across domains. VMs from multiple data centers may need to be in the same subnet and broadcast domain (Layer2) etc. Networking is really complex issue in IaaS. Even more so that it is hard to understand conceptually. One really needs to think in terms of logical networks and forget about the physical network (at least at a high level). To enable all these things and reach large scale we need to be able to control the network devices from the application layer. This is where a significant shift is happening. The application developers are now going to describe the network they need and provision it on-demand. Yet again SDN is Cloud. On-demand and Elasticity in the network thanks to SDN.

So where does Apache CloudStack stands with SDN ? One of the main design decisions in CloudStack was to provide multi-tenancy and isolate guest networks. Pre-SDN, the way to achieve this was to use a VLAN per guest network. Creating an isolated Layer-2 broadcast domain for each tenant (and even multiple VLAN per-tenants if need be). Advanced networking in CloudStack was all about VLANs. VLAN ids however are 12 bit, that means that the grand maximum of VLANs is 4096. While it can seem big, you could very well run out of VLANs quickly. Here comes SDN. SDN allows you to build a new type of isolation for your tenants. The main tenet (pun intended) is to build a mesh of tunnels between all the virtual switches residing on all the hosts/hypervisors in your data center. OVS can do that. Creating those meshes you can create network overlays to build Layer 2 broadcast domains within zones, across zones and over WAN while ensuring isolation of tenants.

The previous snapshot on the left shows the CloudStack GUI when you are creating an advanced zone. You need to specify the type of isolation. Traditional isolation would be using VLANs, but you see two other types of isolations: GRE and STT. These are protocols used to create tunnels between OVS bridges (logical switches). The GRE isolation type will be used with what I call the "native SDN solution" in CloudStack. It is a SDN controller built-in the CloudStack code that creates GRE tunnels using OVS (Currently only supported with Xen, but KVM support should be in 4.1 if not 4.2 this summer). The wiki has an extensive functional specification titled OVS tunnel manager. The slides below are also a great presentation of this solution:

CloudStack + SDN from Chiradeep Vittal

Choosing the STT isolation type will you guessed it use STT tunnels between all the virtual switches. This is currently only being used by the Nicira NVP plugin described in our documentation. Hugo Trippaers (@Spark404) from Schuberg Phillis is the author of the plugin, he recently presented about the integration at a Build a Cloud Day workshop and talked about the upcoming features in the CloudStack 4.1 release (KVM support and Layer 3 routing). See his slides below:

CloudStack NVP Integration - BACD from Hugo Trippaers

We are seeing two more SDN "solutions" being integrated in CloudStack. First is Big Virtual Switch from BigSwitch. Development is happening right now, and the commits made the 4.1 code freeze. So expect to see it in the 4.1 release at the end of March. Expect to see open source OpenFlow controllers being used with CloudStack this summer. The last one is Midonet from Midokura. While documentation has been posted on slideshare (see below and skip the first page if you don't read japanese), the commits have not yet been made. So look at 4.2 release for Midonet support in CloudStack.

CloudStack Avent Calendar Midokura_20121221 from Takeshi Nakajima

This is only the beginning. While these solutions are used to provide multi-tenant isolation, we can bet that SDN will be used to provide load-balancing, elastic IPs, security groups, migration support, dynamic leasing and optimization of network. SDN brings network intelligence to your IaaS.

Build A Cloud Day, Ghent Feb1st Summary

Last friday we had a full day workshop (Build A Cloud Day) in Ghent, Belgium. It was co-located with Puppet Camp. All the logistics was planned by our friends from Inuits led by Kris Buytaert (@KrisBuytaert). The BACD had approximately 50 people in attendance throughout the day, with things winding down by 4pm, when people started heading to Brussels for Europe's biggest Open Source event: FOSDEM. We had an exciting day with terrific speakers that showed the complete range of the CloudStack ecosystem. Here are all the slides, enjoy.

I started the day with an introduction talk about CloudStack, bringing some high level vision about Clouds and how CloudStack fits in. I also presented the "Apache Way", what you can expect in terms of releases, and I also highlighted the main components of CloudStack. I introduced the rest of the agenda with a theme of covering the entire "stack" of Cloud computing and seeing how CloudStack is the core backend of it.

Intro to CloudStack Build a Cloud Day from Sebastien Goasguen

Hugo Trippaers (@Spark404) from Schuberg Phillis then presented the integration of the Nicira Private Gateway with CloudStack. Which brings an SDN solution (STT tunnels meshes for isolation of tenants) to ACS and complements the native SDN controller that can build meshes of GRE tunnels. Hugo is a committer and PMC member of CloudStack and the lead engineer for operation of the CloudStack private cloud at Schuberg Phillis.

CloudStack NVP Integration - BACD from Hugo Trippaers

Once we learned about SDN and advanced networking for multi-tenant isolation, we heard from Wido Den Hollander from PCExtreme. Wido is a committer and PMC member of CloudStack as well, and he is most notably the man behind the CloudStack/Ceph integration. Ceph has received a lot of attention for the last couple years as it offers an alternative to traditional parallel distributed file systems and builds a highly scalable object store and efficient storage for virtual machines with the so-called Rados Block Device (RBD). Wido had few slides but filled many questions on the board.

Ceph storage for CloudStack from Wido den Hollander

Three talks and the excitement rose learning about the latest features in networking and storage for CloudStack. A short lunch and we came back in to talk about the API exposed by CloudStack. When discussing APIs, everyone mentions standards. CloudStack API is not a standard and I doubt it will ever be (for good reasons). But what we have seen in the field is a de-facto standard in AWS APIs (which is supported in CloudStack to some extent) and some emerging standards from OGF and DMTF. While limited in scope they do provide some assurance against vendor lock-in. The talk by RedHat Oved Ourfali (@ovedou) was about CIMI, the DMTF standard. We were particularly interested to hear about CIMI to see how we could integrate it in CloudStack and potentially write a deltacloud driver.

Supporting and Using EC2/CIMI on top of Cloud Environments via Deltacloud from Oved Ourfali

Having learned about the backend networking and storage solutions, plus the API used to manage and access your cloud, it was now time to hear about an exciting use case: Spotify the on-line music service. Noa Resare (@blippie) has been active on the CloudStack mailing list and is now helping out with packaging. He presented us Spoticloud a private cloud built for their engineers/developers to allow them to "be developers". He gave us some great feedback with pointy details like adding a pod with the wrong id, and asked for features that we are working on, like removing the secondary storage VM. Also exciting was that Spotify is hiring a cloudstack engineer: http://spoti.fi/cloudstack

Cloudstack at Spotify from Noa Resare

Next up was Brian Amedro (@brianamedro) from Activeeon, I really wanted Brian to talk because he represents a SaaS application making use of CloudStack. His company Activeeon has a very interesting application: ProActive, that offers a powerful IDE, a workflow engine and a resource manager to automate the parallelization of compute intensive tasks. It is used in the automotive industry, pharmacy, finance and other fields that have a need for long running computing analysis. Where CloudStack comes in is at the resource management layer, Brian integrated ProActive with CloudStack using the API, it allows ProActive to dynamically provision machines on cloud providers and run the workflows. Check the video as well.

Elastic and Flexible Cloud with ProActive & CloudStack from Brian Amedro

To wrap up our day we had Charles Moulliard (@cmoulliard), (RedHat), an Apache committer on several projects and very active on the CloudStack mailing list. Charles is not shy to bring up issues with the API, DevCloud or thrid party clients like jclouds. Charles introduced us to Karaf and Fuse Fabric. By deploying Karaf on multiple cloud providers you can create a coordinated PaaS. All the providers coordinate with a Zookeeper instance, creating a "Fuse Fabric" that eases deployment of software on all cloud nodes. It gave me an idea to use FuseFabric to deploy a hadoop cluster in the cloud...

iPaas with Fuse Fabric Technology from Charles Moulliard

And that was it for BACD Ghent. If you were there you learned a ton :), if you missed it...well you can always come to the next one, but it may not come with Belgium beer :)

What a month !

What a month January 2013 was for Apache CloudStack mailing lists. I have barely recovered from what was our most intense month since CloudStack was donated to the Apache Software Foundation. As some of you know I have been keeping a close eye on our mailing lists, trying to extract valuable information for our community. This was triggered by Qingye Jiang study about CloudStack, OpenStack, Eucalyptus and Opennebula. Since all decisions happen on the mailing list at the ASF, it seemed like a very reasonable method to check the health of our community

I posted some early results in a prior post. With the whirlwind of emails in January I wanted to get a closer look at this past month alone.

We had a total of 5144 emails on the developers list and a total of 775 emails on the users list. 2036 of the 5144 where from JIRA.

First 5144 emails in a month is quite big to put it mildly. Second 2036 from JIRA is a lot. JIRA generates emails automatically, but what we have seen this month is an increase in conversation between developers happening within tickets. This is actually a mode of operation that is very familiar to the HDFS community for instance. I suspect that as we go through graduation, we may find that working tickets directly in JIRA will be very valuable and that JIRA emails will represent valid communication emails.

The developers list saw 175 unique contributors and the users list saw 145 unique contributors. I still need to clean up a few duplicates, but I don't expect it to represent more than 5% of the total number of contributors. That seems to me like a pretty robust community for a single month. If we compare both set of contributors we see that 76 are common to both list. This means that 48% of the contributors on the users list are not participating in the developers list, they are "true" users.

While affiliation of individual contributors is not recognized within the ASF, I can't help but mention that 57 domains were represented in the developers list versus 63 domains in the users lists. It would not be a stretch to associate an email domain with a company.

The graph below shows our daily number of contributors. First thing to notice is that thankfully we take a break on week-ends. The dips clearly show a lower number of contributors on saturday and sunday. The developers list approximately peaks at 60 contributors a day on Tuesday or Wednesday. The users list peaks at roughly 20 contributors a day mid-week.

In my previous post I also started a social network analysis. I did it again for this month alone. Below is the social graph of our developers list. To obtain this graph I group emails by thread and create a connection between two contributors if they exchange emails within a thread. Every time the same two contributors exchange an email within a thread, the "strength" of their connection increases. This is represented by the thickness of the line between all nodes. Also the size of the node/contributor matters. The more central a node is within our community, the bigger the node. Centrality is defined as being the shortest path between any two nodes in the network. The graph clearly shows that Chip Childers had the biggest centrality in our developers list in January. That means that for any two contributors to talk, the shortest path was through Chip (Sorry Chip, looks like your inbox is going to get even bigger). Other notables is Chiradeep Vittal, Alex Huang, David Nalley and Animesh Chaturvedi. Nodes that had a small number of connections were filtered out (but their contribution is very much appreciated. I only applied the filter to get a clearer picture). My boss would like to see a dynamic version of this graph, were we can see the evolution of the social network over time...:) Will see, I am just not sure how to automate that...

And below is the social graph of our users list. We see that Geoff Higginbottom from Shapeblue is the most influential node on the users list. Closely followed by Ahmad Emneina and Pranav Saxena from Citrix. A new comer is also showing up this month, Geoff's protege, Paul Angus from Shapeblue. It is great to see the leading CloudStack integrator taking leadership in helping users of Apache CloudStack. We also see a new user coming on board strongly: Nux also know as Lucian that I had the pleasure to meet in London. Interestingly we see that the thickness of the connections between the nodes is less than on the developers list and the network is larger. This just indicates that the most influential nodes answer a lot of questions from many different people but that they don't tend to exchange within a large number of threads. Also note that the traffic on the users list is much less than on the dev list.

And that's it for January, our highest traffic so far on the mailing lists. We see some strong and increasing participation every day, a clear technical leadership has emerged and some dedicated folks are helping out users.

Social Network Analysis of Apache CloudStack

Apache is about building communities of developers and users around an open source software. As such they can be analyzed with social networking tools to identify patterns of communications (communication networks), sub-communities within groups and bring up the most influential nodes. This type of analysis can be done over time using data from the Apache mailing lists.

Since I am a new Apache committer on CloudStack I wanted to have a look at the health of our community and thought a social network analysis (SNA) would do it. A little googling led me to this very nice research paper on SNA of the R mailing lists. I have not done all the analysis mentioned in the paper, especially the content based analysis but I wanted to post my early results.

Methodology: To get the graphs I grabbed the emails archive from Apache. I used Python to load the mbox files into single Mongo collections. I cleaned the data to avoid replications of senders as well as remove JIRA and Review Board entries. Then with a little bit of PyMongo I made the queries and build the graph with NetworkX. Finished up with the graph visualization and calculations using Gephi. Since there are thousands of emails and threads, there is still some work to pre-process the data, avoid duplicates and match individuals to multiple email addresses.

Using Gephi, I manipulated the graphs. Computing the degree of each node (i.e the number of direct connection to other nodes), the betweeness centrality (i.e a measure of how often a node serves as a bridge between the shortest path between two nodes. In other terms: is a node the best "proxy" between two other nodes ?), I then partitioned the graphs with a color code, trying to identify sub-communities. Finally for clarity I filtered nodes by degree. In CloudStack filtering is especially important since the list has grown quite large of late (This may actually be an indirect sign that it is time to split the dev list).

The graph of the cloudstack-dev mailing list can be seen below:

What stands out right away are the largest nodes, or the most influential nodes according to betweeness centrality. Chip, David, Edison, Chiradeep, Hugo, Wido, Alex are all members of the PMC and exhibit a high centrality. Prasanna and Rohit also exhibit a high centrality but are not currently in the PMC. Also of interest is that this graph is valid since CloudStack joined Apache in April 2012, we can identify contributors who are not active currently but once where and thus are still part of the overall communication network. The color code highlights communities within the community. There seems to be 4 to 5 sub-communities (green, blue, red, cyan, yellow, more investigation is necessary to give interesting meanings to these sub-communities. You will also notice that the edges have all the same thickness. This means that they have the same weight. Once two people exchange an email, an edge is drawn between the two nodes. If they communicate again, the edge is not modified. I will add edge weighting in a future study, this will show us "pathways" between community members and will also affect the influence of the nodes.

Update January 22nd: I added weight to the edges. In english this means that everytime two people communicated in a thread I increases their connectidness by 1. The graph below shows edges with a different thickness. Nodes and Edges were filtered to highlight the strongest connection. This clearly shows the "PMC" of ACS.

The graph of the cloudstack users mailing list can be seen below:

What stands out the most in this graph is that some of the PMC members are still influential (Chiradeep, David, Alex and Edison for instance). But new influential nodes have appeared. Most notably: mcirauqui, geoff.higginbottom and ahmad.emneina. Chip Childers is still present but his influence in this users community is much less. Based on this I am ready to campaign for mcirauqui and geoff to become committers, as they are clear contributors of the CloudStack users community :)

For comparison I checked the HDFS dev mailing list (note that this is fairly restrictive since Hadoop is a very large ecosystem with many mailing lists), followed the same process and obtained the following graph. Maybe the HDFS community can help me analyze it and see if this gives the right picture of their dev community :)

I plan to do more work on this. Cleaning the dataset a bit further, studying the community partitioning, and especially building content based graphs. These will allow us to identify communication network on a particular topic. Say you want to learn about SDN support in CloudStack, we could generate the graph and see who are the most "influential" nodes about SDN in CloudStack.

A Mahout Cluster across France and Luxembourg Using CloudStack

Early December I attended the Grid5000 Winter school held at the Ecole des Mines de Nantes (EMN) and organized by Adrien Lebre. Grid 5000 is "a scientific instrument designed to support experiment-driven research in all areas of computer science related to parallel, large-scale or distributed computing and networking ". Basically a large scale testbed to design, build and test distributed systems. The US also have such an infrastructure in academia called FutureGrid. These research infrastructures have become key to enable research in distributed systems approaching scales now seen in the industry rather than test systems on couple machines in a single lab. Currently Grid5000 operates 1195 physical hosts, for a total of 8184 cores across 10 sites.

While in Nantes I met with Alexandra Carpen Amarie an INRIA research engineer who developed an amazing tool G5k campaign. G5k campaign allows any user of Grid5000 (G5k) to book nodes on the infrastructure, deploy machines with bare-metal provisioning and then deploy their favorite Cloud IaaS framework (currently CloudStack, Opennebula and Nimbus). G5k scripts are available via git, of interest are some Chef recipes. Heavily tailored for Alexandra's scripts they could be useful for the CloudStack community. Alexandra held a tutorial on deploying a IaaS and PaaS on G5k. For the tutorial the PaaS was Apache Mahout. Lets' not get into a discussion about whether Mahout is a PaaS or not, the point is that a IaaS can be used to deploy and managed a set of nodes that run Hadoop and Mahout on top, to provide a high level functionality. In this case machine learning algorithms to analyze large data-sets. It was attended by approximately 30 people. How did it work exactly?

One thing about G5k and I believe the French research computing community is that they are very prolific in creating great tools. Unfortunately few people know about them. The clusters of G5k are operated like regular batch processing clusters. A batch scheduler is used to access the nodes. Tool #1: OAR a PBS/MOAB like equivalent. Once the nodes are allocated they are provisioned using Tool #2: Kadeploy a crowbar like equivalent. Of great interest is Tool #3 KaVLAN, a tool to lease the VLANs configured on G5k. While not currently used in Alexandra's G5k campaign, I hope the Apache CloudStack community can start making use of it to test Advanced Zones.

The beauty of G5k campaign is that Alexandra's has hidden most of the complexity of the provisioning and configuring. You only need to write a YAML configuration file for your deployment. Specifying the sites and the number of nodes that you want to run on/at, for example:

deployment:
  engine:
    name: CloudStack
    customization_type: multisiteChef
  walltime: 2:00:00
  sites:
    rennes:
      nodes: 10
      subnet: slash_22=1
    nancy:
      nodes: 10
      subnet: slash_22=1     
    sophia:
      nodes: 10
      subnet: slash_22=1
ssh:
  user: username

Launch your campaign and wait for the nodes to be allocated, provisioned and then configured with your IaaS. Depending on the number of nodes requested, you could have a Cloud working within 20 minutes. You can then interact with it. In the case of CloudStack, using the API, Alexandra developed some wrappers to manage VMs. She did it before CloudMonkey came out and is not needed now even though still a great exercise. Couple days after the tutorial I asked Alexandra to deploy CloudStack across several sites. Within 24 hours I had those snapshots in my inbox. A 5 sites cloud, one basic zone per physical site and 97 physical hosts setup, 800 cores and 100 VMs deployed running Mahout. It took 30 minutes to deploy the nodes, one hour to configure the hosts in CloudStack (serially, Alexandra is working on adding parallel configuration in her tool). The 100 VMs were deployed in roughly 10 minutes. The 3 physical nodes missing were due to bare metal provisioning problems.

The snapshot below shows the infrastructure/zone view of the CloudStack deployment. Five basic zones were configured at Rennes, Toulouse, Nancy, Sophia and Luxembourg. All cities connected via the RENATER fiber network.

Below the infrastructure view, shows five zones, 97 hosts, 10 system VMs (console proxy and secondary storage) and 4 virtual routers (One router was not started at the time of the snapshot).

A small detail that you may have seen from the YAML configuration file is that this is all based on ssh. Access to G5k is via ssh keys and not via a PKI infrastructure. Having worked on TeraGrid. This was a nice surprised. Using PKI across different organizations and managing authorization can be extremely complex. This was a sore point in the TeraGrid. It is also used in the LHC grid with more success but still requires a lot of work. In G5k the user base is smaller and more trusted. SSH keys are distributed among sites using a basic NFS setup on the private RENATER network. This makes it easy for users to access all sites.

Looking ahead, the basic question one might ask is whether it makes sense to run mahout within virtual machines. In cases where the dataset is not very large the use of HDFS as a large scale distributed storage systems is not the issue. Rather the time spent running the machine learning algorithms is. There the cpu overhead of virtualization is the main performance factor. Alexandra pointed me to a paper she wrote on performance of map-reduce in the Cloud. I asked her to do some more analysis specific to a CloudStack based Cloud, stay tuned for the results :).

Thoughts from the Frontline

I am stealing the title of this blog from John Mauldin an analyst who has a great newsletter. For almost five years he has been very bearish on the economy, calling it a "muddle through" economy. He gets on my nerves lately because he calls for the end of the euro...

Anyway :) I dont' think he knows anything about Cloud Computing except maybe the value of certain tech companies and trends in the sector. What's the link with Cloud you are asking ? Well, the link is that while we were in full blown sub-prime mortgage crisis back in 2007/2008 (with Bear Stearns crumbling in March 2008) the world of Cloud Computing was in full innovation mode.

What really triggered this post is a talk by Marten Mickos from Eucalyptus at Linux Con Europe in Barcelona. In it, he called CloudStack, Eucalyptus, OpenNebula and OpenStack the four sisters. He called them the four sisters apparently in response to VMware prince charming blog. The point is not to start a polemic on this, but rather that I thought Marten missed at least two other ones: Nimbus and VCL.

Looking back at those six sisters, I wanted to see when they started. CloudStack started as VMops in early 2008, Nimbus was announced in March 2008, OpenNebula first release came in July 2008. Eucalyptus started also around April 2008 with a tech report filed at UCSB and a paper later published at the CCGRID conference in 2009. VCL started much earlier via the IBM Virtual Computing Initiative ( VCI ) out of Research Triangle Park. OpenStack first release -Austin- came in October 2010, but NASA nova came out of the Nebula project which was started in 2008 as well ( I believe through stimulus money, but I could not find confirmation).

So Bear Stearns was crumbling, portfolios were taking a huge hit but Cloud was being created.

What is also striking with those "six sisters" is that three came out of educational institutions, and two (Nimbus and Openstack nova) came out of a federal lab (even though NASA IT isn't really a research lab). Eucalyptus came out of Rich Wolski's group at UCSB, OpenNebula came out of Ignacio de Lorente and Ruben Montero's group at University of Madrid, Nimbus came out of Kate Keahey's group out of Argonne National Lab and VCL came out of North Carolina State University under Mladen Vouk's leadership. Only CloudStack stands out as being 100% enterprise driven.

The research on Cloud started much earlier but did not have the Cloud tag on it. It started as virtualization research and then application of virtualization to build distributed systems. I could list hundred of references on the subject but I will just point to my friends from University of Florida and tell you to read "A case for Grid Computing on Virtual Machines.". You may have to order it or find a free version. This paper is dated 2003, it lays the foundation for how virtualization is going to allow us to build dynamic homogenous systems over a distributed set of physical resources: in one word, the Cloud. Yes it is compute centric, but remember that the Amazon Dynamo paper only came out late in 2007, while it's true that the Google File System was presented in 2003 as well.

Eucalyptus, OpenNebula, Nimbus and VCL all published description of their systems around 2009. Take into account at least two/three years of work leading to the actual presentation of the paper.

VCL is now a top level Apache project, having graduated from incubation last June. VCL is almost unknown in the Cloud community despite running software, great features and vision. Nimbus is still going strong in academic research, specifically in support of science through the science clouds but has yet to see wide adoption in national research infrastructure projects like OSG. These two seem out of the discussion when we talk about enterprise solution and large open source communities. One can wonder though, what is the differentiator that makes similar ideas successful or not.

We are left with the four sisters then - CloudStack, Eucalyptus, OpenNebula, OpenStack - by alphabetical order :)

Rather oddly OpenNebula has not received any venture capital or large support from a hardware vendor (at least none that I know of). It is a shame for a great project and software that works very well. Perhaps the fact that it is located in Europe makes it harder. Europe is more conservative when it comes to innovation and taking risks. The remaining three for that matter are all located or originated in California. Surf and computers seem to do well together :)

From a momentum stand point, it is difficult to argue that OpenStack is not leading. It has gathered a lot of interest and support from key hardware vendors. In retrospect though it was the last project to start -~summer 2010-. In my opinion the reason why it has gained a lot of user/developer support is that there was a vacuum. CloudStack was not yet in Apache and cloud.com operated silently. OpenNebula was an academic European project and Eucalyptus was going through growing pains of migrating from an academic project to an enterprise solution. OpenStack seemed the only choice.

Now that CloudStack is incubating in Apache with tons of enterprise experience and focus, Eucalyptus is re-focusing and OpenNebula still going strong hoping for the euro to survive, users have plenty of choices. This is good news.

This is good news because instead of being forced into the only potential solution on the market. Users will be able to test, deploy, benchmark, stress test, integrate, develop, take part in the community, and then make an educated choice. In my view this is what was in the core of the Gartner note. When making such an important choice for your IT, you need to look at available choices and make an educated technical decision based on your local constraints.

deploy & test != twitter & prod