Why CloudStack is not a Citrix project

I was at CloudExpo Europe in London last week for the Open Cloud Forum to give a tutorial on CloudStack tools. A decent crowd showed up, all carrying phones. Kind of problematic for a tutorial where I wanted the audience to install python packages and actually work :) Luckily I made it self-paced so you can follow at home. Giles from Shapeblue was there too and he was part of a panel on Open Cloud. He was told once again "But Apache CloudStack is a Citrix project !" This in itself is a paradox and as @jzb told me on twitter yesterday "Citrix donated CloudStack to Apache, the end". Apache projects do not have any company affiliation.

I don't blame folks, with all the vendors seemingly supporting OpenStack, it does seem that CloudStack is a one supporter project. The commit stats are also pretty clear with 39% of commits coming from Citrix. This number is also probably higher since those stats are reporting gmail and apache as domain contributing 20 and 15% respectively, let's say 60% is from Citrix. But nonetheless, this is ignoring and mis-understanding what Apache is and looking at the glass half empty.

When Citrix donated CloudStack to the Apache Software Foundation (ASF) it relinquished control of the software and the brand. This actually put Citrix in a bind, not being able to easily promote the CloudStack project. Indeed, CloudStack is now a trademark of the ASF and Citrix had to rename their own product CloudPlatform (powered by Apache CloudStack). Citrix cannot promote CloudStack directly, it needs to get approval to donate sponsoring and follow the ASF trademark guidelines. Every committer and especially PMC members of Apache CloudStack are now supposed to work and protect the CloudStack brand as part of the ASF and make sure that any confusion is cleared. This is what I am doing here.

Of course when the software was donated, an initial set of committers was defined, all from Citrix and mostly from the former cloud.com startup. Part of the incubating process at the ASF is to make sure that we can add committers from other organization and attract a community. "Community over Code" is the bread and butter of ASF and so this is what we have all been working on, expanding the community outside Citrix, welcoming anyone who thinks CloudStack is interesting enough to contribute a little bit of time and effort. Looking at the glass half empty is saying that CloudStack is a Citrix project "Hey look 60% of their commits is from Citrix", looking at it half full like I do is saying "Oh wow, in a year since graduation, they have diversified the committer based, 40% are not from Citrix". Is 40% enough ? of course not, I wish it were the other way around, I wish Citrix were only a minority in the development of CloudStack.

Couple other numbers: Out of the 26 members of the project management committee (PMC) only seven are from Citrix and looking at mailing lists participation since the beginning of the year, 20% of the folks on the users mailing list and 25% on the developer list are from Citrix. We have diversified the community a great deal but the "hand-over", that moment when new community members are actually writing more code than the folks who started it, has not happened yet. A community is not just about writing code, but I will give it to you that it is not good for a single company to "control" 60% of the development, this is not where we/I want to be.

This whole discussion is actually against Apache's modus operandi. Since one of the biggest tenant of the foundation is non-affiliation. When I participate on the list I am Sebastien, I am not a Citrix employee. Certainly this can put some folks in conflicting situations at times, but the bottom line is that we do not and should not take into account company affiliation when working and making decisions for the project. But if you really want some company name dropping, let's do an ASF faux-pas and let's look at a few features:

The Nicira/NSX and OpenDaylight SDN integration was done by Schuberg Phillis, the OpenContrail plugin was done by Juniper, Midokura created it's own plugin for Midonet and Stratosphere as well, giving us a great SDN coverage. The LXC integration was done by Gilt, Klarna is contributing in the ecosystem with the vagrant and packer plugins, CloudOps has been doing terrific job with Chef recipes, Palo-Alto networks integration and Netscaler support, a google summer of code intern did a brand new LDAP plugin and another GSoC did the GRE support for KVM. RedHat contributed the Gluster plugin and PCExtreme contributed the Ceph interface while Basho of course contributed the S3 plugin for secondary storage as well as major design decisions on the storage refactor. The Solidfire plugin was done by, well Solidfire and Netapp has developed a plugin as well for their virtual storage console. NTT contributed the CloudFoundry interface via BOSH. On the user side, Shapeblue is leading the user support company. So no it's not just Citrix.

Are all these companies members of the CloudStack project ? No. There is no such thing as a company being a member of an ASF project. There is no company affiliation, there is no lock in, just a bunch of guys trying to make good software and build a community. And yes, I work for Citrix and my job here will be done when Citrix only contributes 49% of the commits. Citrix is paying me to make sure they loose control of the software, that a healthy ecosystem develops and that CloudStack keeps on becoming a strong and vibrant Apache project. I hope one day folks will understand what CloudStack has become, an ASF project, like HTTP, Hadoop, Mesos, Ant, Maven, Lucene, Solr and 150 other projects. Come to Denver for #apachecon you will see ! The end.

PaaS with CloudStack

A few talks from CCC in Amsterdam

In November at the CloudStack Collaboration Conference I was pleased to see several talks on PaaS. We had Uri Cohen (@uri1803) from Gigaspaces, Marc-Elian Begin (@lemeb) from Sixsq and Alex Heneveld (@ahtweetin) from CloudSoft. We also had some related talks -depending on your definition of PaaS- with talks about Docker and Vagrant.

PaaS variations

The differences between PaaS solutions is best explained by this picture from AWS FAQ about application management.

There is clearly a spectrum that goes from operational control to pure application deployment. We could argue that true PaaS abstracts the operational details and that management of the underlying infrastructure should be totally hidden, that said, automation of virtual infrastructure deployment has reached such a sophisticated state that it blurs the definition between IaaS and PaaS. Not suprisingly AWS offers services that covers the entire spectrum.
Since I am more on the operation side, I tend to see a PaaS as an infrastructure automation framework. For instance I look for tools to deploy a MongoDB cluster or a RiakCS cluster. I am not looking for an abstract plaform that has Monogdb pre-installed and where I can turn a knob to increase the size of the cluster or manage my shards. An application person will prefer to look at something like Google App Engine and it's open source version Appscale. I will get back to all these differences in a next post on PaaS but this article by @DavidLinthicum that just came out is a good read.

Support for CloudStack

What is interesting for the CloudStack community is to look at the support for CloudStack in all these different solutions wherever they are in the application management spectrum.

• Cloudify from Gigaspaces was all over twitter about their support for OpenStack, and I was getting slightly bothered with the lack of CloudStack support. That's why it was great to see Uri Cohen in Amstredam. He delivered a great talk and he gave me a demo of Cloudify. I was very impressed of course by the slick UI but overall by the ability to provision complete application/infrastructure definitions on clouds. Underlying it uses Apache jclouds, so there was no reason that it could not talk to CloudStack. Over christmas Uri did a terrific job and the CloudStack support is now tested and documented. It works not only on the commercial version from Citrix CloudPlatform but also with Apache CloudStack. And of course it works with my neighbors Cloud exoscale
• Slipstream is not widely known but worth a look. At CCC @lemeb demoed a CloudStack driver. Since then, they now offer an hosted version of their slipstream cloud orchestration framework which turns out to be hosted on exoscale CloudStack cloud. Slipstream is more of a Cloud broker than a PaaS but it automates application deployment on multiple clouds abstracting the various cloud APIs and offering application templates for deployments of virtual infrastructure. Check it out.
• Cloudsoft main application deployment engine is brooklyn, it originated from Alex Heneveld contribution to Apache Whirr that I wrote about couple times. But it can use OpenShift for additional level of PaaS. I will need to check with Alex how they are doing this, as I believe Openshift uses LXC. Since CloudStack has LXC support, one ought to be able to use Brooklyn to deploy a LXC cluster on CloudStack and then use OpenShift to manage deployed applications.
• A quick note on OpenShift. As far as I understand, it actually uses a static cluster. The scalability comes from the use of containes in the nodes. So technically you could create an OpenShift cluster in CloudStack, but I don't think we will see OpenShift talking directly to the CloudStack API to add nodes. Openshift bypasses the IaaS APIs. Of course I have not looked at it in a while and I may be wrong :)
• Talking about PaaS for Vagrant is probably a bit far fetched, but it fits the infrastructure deployment criteria and could be compared with AWS OpsWorks. Vagrant helps to define reproducible machines so that devs and ops can actually work on the same base servers. But Vagrant with its plugins can also help deployment on public clouds, and can handle multiple server definitions. So one can look at a Vagrantfile as a template defintion for a virtual infrastructure deployment. As a matter of fact, there are many Vagrant boxes out there to deploy things like Apache Mesos clusters, MongoDB, RiakCS clusters etc. It's not meant to manage that stack in production but at a minimum can help develop it. Vagrant has a CloudStack plugin demoed by Hugo Correia from Klarna at CCC. Exoscale took the bait and created a set of -exoboxes- that's real gold for developers deploying in exoscale and any CloudStack provider should follow suit.
• Which brings me on to Docker, there is currently no support for Docker in CloudStack. We do have LXC support therefore it would not be to hard to have a 'docker' cluster in CloudStack. You could even install docker within an image and deploy that on KVM or Xen. Of course some would argue that using containers within VMs defeats the purpose. In any case, with the Docker remote API you could then manage your containers. OpenStack already has a Docker integration, we will dig deeper into Docker functionality to see how best to integrate it in CloudStack.
• AWS as I mentioned has several PaaS like layers with OpsWorks, CloudFormation, Beanstalk. CloudStack has an EC2 interface but also has a third party solution to enabled CloudFormation. This is still under development but pretty close to full functionality, check out stackmate and its web interface stacktician. With a CF interface to CloudStack we could see a OpSWork and a Beanstalk interface coming in the future.
• Finally, not present at CCC but the leader of PaaS for enterprise is CloudFoundry. I am going to see Andy Piper (@andypiper) in London next week and will make sure to talk to him about the recent CloudStack support that was merged in the cloudfoundry community repo. It came from folks in Japan and I have not had time to test it. Certainly we as a community should look at this very closely to make sure there is outstanding support for CloudFoundry in ACS.

It is not clear what the frontier between PaaS and IaaS is, it is highly dependent on the context, who you are and what you are trying to achieve. But CloudStack offers several interfaces to PaaS or shall I say PaaS offer several connectors to CloudStack :)

Veewee, Vagrant and CloudStack

Coming back from CloudStack conference the feeling that this is not about building clouds got stronger. This is really about what to do with them and how they bring you agility, faster-time to market and allow you to focus on innovation in your core business. A large component of this is Culture and a change of how we do IT. The DevOps movement is the embodiment of this change. Over in Amsterdam I was stoked to meet with folks that I had seen at other locations throughout Europe in the last 18 months. Folks from PaddyPower, SchubergPhilis, Inuits who all embrace DevOps. I also met new folks, including Hugo Correia from Klarna (CloudStack users) who came by to talk about Vagrant-cloudstack plugin. His talk and a demo by Roland Kuipers from Schuberg was enough to kick my butt and get me to finally check out Vagrant. I sprinkled a bit of Veewee and of course some CloudStack on top of it all. Have fun reading.

Automation is key to a reproducible, failure-tolerant infrastructure. Cloud administrators should aim to automate all steps of building their infrastructure and be able to re-provision everything with a single click. This is possible through a combination of configuration management, monitoring and provisioning tools. To get started in created appliances that will be automatically configured and provisioned, two tools stand out in the arsenal: Veewee and Vagrant.

Veewee: Veewee is a tool to easily create appliances for different hypervisors. It fetches the .iso of the distribution you want and build the machine with a kickstart file. It integrates with providers like VirtualBox so that you can build these appliances on your local machine. It supports most commonly used OS templates. Coupled with virtual box it allows admins and devs to create reproducible base appliances. Getting started with veewee is a 10 minutes exericse. The README is great and there is also a very nice post that guides you through your first box building.

Most folks will have no issues cloning Veewee from github and building it, you will need ruby 1.9.2 or above. You can get it via `rvm` or your favorite ruby version manager.

git clone https://github.com/jedi4ever/veewee
gem install bundler
bundle install

Setting up an alias is handy at this point `alias veewee="bundle exec veewee"`. You will need a virtual machine provider (e.g VirtualBox, VMware Fusion, Parallels, KVM). I personnaly use VirtualBox but pick one and install it if you don't have it already. You will then be able to start using `veewee` on your local machine. Check the sub-commands available (for virtualbox):

$ veewee vbox
Commands:
  veewee vbox build [BOX_NAME]                     # Build box
  veewee vbox copy [BOX_NAME] [SRC] [DST]          # Copy a file to the VM
  veewee vbox define [BOX_NAME] [TEMPLATE]         # Define a new basebox starting from a template
  veewee vbox destroy [BOX_NAME]                   # Destroys the virtualmachine that was built
  veewee vbox export [BOX_NAME]                    # Exports the basebox to the vagrant format
  veewee vbox halt [BOX_NAME]                      # Activates a shutdown the virtualmachine
  veewee vbox help [COMMAND]                       # Describe subcommands or one specific subcommand
  veewee vbox list                                 # Lists all defined boxes
  veewee vbox ostypes                              # List the available Operating System types
  veewee vbox screenshot [BOX_NAME] [PNGFILENAME]  # Takes a screenshot of the box
  veewee vbox sendkeys [BOX_NAME] [SEQUENCE]       # Sends the key sequence (comma separated) to the box. E.g for testing the :boot_cmd_sequence
  veewee vbox ssh [BOX_NAME] [COMMAND]             # SSH to box
  veewee vbox templates                            # List the currently available templates
  veewee vbox undefine [BOX_NAME]                  # Removes the definition of a basebox 
  veewee vbox up [BOX_NAME]                        # Starts a Box
  veewee vbox validate [BOX_NAME]                  # Validates a box against vagrant compliancy rules
  veewee vbox winrm [BOX_NAME] [COMMAND]           # Execute command via winrm

Options:
          [--debug]           # enable debugging
  -w, --workdir, [--cwd=CWD]  # Change the working directory. (The folder containing the definitions folder).
                              # Default: /Users/sebgoa/Documents/gitforks/veewee

Pick a template from the `templates` directory and `define` your first box:

veewee vbox define myfirstbox CentOS-6.5-x86_64-minimal

You should see that a `defintions/` directory has been created, browse to it and inspect the `definition.rb` file. You might want to comment out some lines, like removing `chef` or `puppet`. If you don't change anything and build the box, you will then be able to `validate` the box with `veewee vbox validate myfirstbox`. To build the box simply do:

veewee vbox build myfristbox

Everything should be successfull, and you should see a running VM in your virtual box UI. To export it for use with `Vagrant`, `veewee` provides an export mechanism (really a VBoxManage command): `veewee vbox export myfirstbox`. At the end of the export, a .box file should be present in your directory.

Vagrant: Picking up from where we left with `veewee`, we can now add the box to Vagrant and customize it with shell scripts or much better, with Puppet recipes or Chef cookbooks. First let's add the box file to Vagrant:

vagrant box add 'myfirstbox' '/path/to/box/myfirstbox.box'

Then in a directory of your choice, create the Vagrant "project":

 
vagrant init 'myfirstbox'

This will create a `Vagrantfile` that we will later edit to customize the box. You can boot the machine with `vagrant up` and once it's up , you can ssh to it with `vagrant ssh`.

While `veewee` is used to create a base box with almost no customization (except potentially a chef and/or puppet client), `vagrant` is used to customize the box using the Vagrantfile. For example, to customize the `myfirstbox` that we just built, set the memory to 2 GB, add a host-only interface with IP 192.168.56.10, use the apache2 Chef cookbook and finally run a `boostrap.sh` script, we will have the following `Vagrantfile`:

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "myfirstbox"
  config.vm.provider "virtualbox" do |vb|
    vb.customize ["modifyvm", :id, "--memory", 2048]
  end

  #host-only network setup
  config.vm.network "private_network", ip: "192.168.56.10"

  # Chef solo provisioning
  config.vm.provision "chef_solo" do |chef|
     chef.add_recipe "apache2"
  end

  #Test script to install CloudStack
  #config.vm.provision :shell, :path => "bootstrap.sh"
  
end

The cookbook will be in a `cookbooks` directory and the boostrap script will be in the root directory of this vagrant definition. For more information, check the Vagrant website and experiment.

Vagrant and CloudStack: What is very interesting with Vagrant is that you can use various plugins to deploy machines on public clouds. There is a `vagrant-aws` plugin and of course a `vagrant-cloudstack` plugin. You can get the latest CloudStack plugin from github. You can install it directly with the `vagrant` command line:

vagrant plugin install vagrant-cloudstack

Or if you are building it from source, clone the git repository, build the gem and install it in `vagrant`

git clone https://github.com/klarna/vagrant-cloudstack.git
gem build vagrant-cloudstack.gemspec
gem install vagrant-cloudstack-0.1.0.gem
vagrant plugin install /Users/sebgoa/Documents/gitforks/vagrant-cloudstack/vagrant-cloudstack-0.1.0.gem

The only drawback that I see is that one would want to upload his local box (created from the previous section) and use it. Instead one has to create `dummy boxes` that use existing templates available on the public cloud. This is easy to do, but creates a gap between local testing and production deployments. To build a dummy box simply create a `Vagrantfile` file and a `metadata.json` file like so:

$ cat metadata.json 
{
    "provider": "cloudstack"
}
$ cat Vagrantfile 
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.provider :cloudstack do |cs|
    cs.template_id = "a17b40d6-83e4-4f2a-9ef0-dce6af575789"
  end
end

Where the `cs.template_id` is a uuid of a CloudStack template in your cloud. CloudStack users will know how to easily get those uuids with `CloudMonkey`. Then create a `box` file with `tar cvzf cloudstack.box ./metadata.json ./Vagrantfile`. Note that you can add additional CloudStack parameters in this box definition like the host,path etc (something to think about :) ). Then simply add the box in `Vagrant` with:

vagrant box add ./cloudstack.box

You can now create a new `Vagrant` project:

mkdir cloudtest
cd cloudtest
vagrant init

And edit the newly created `Vagrantfile` to use the `cloudstack` box. Add additional parameters like `ssh` configuration, if the box does not use the default from `Vagrant`, plus `service_offering_id` etc. Remember to use your own api and secret keys and change the name of the box to what you created. For example on exoscale:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "cloudstack"

  config.vm.provider :cloudstack do |cs, override|
    cs.host = "api.exoscale.ch"
    cs.path = "/compute"
    cs.scheme = "https"
    cs.api_key = "PQogHs2sk_3..."
    cs.secret_key = "...NNRC5NR5cUjEg"
    cs.network_type = "Basic"

    cs.keypair = "exoscale"
    cs.service_offering_id = "71004023-bb72-4a97-b1e9-bc66dfce9470"
    cs.zone_id = "1128bd56-b4d9-4ac6-a7b9-c715b187ce11"

    override.ssh.username = "root" 
    override.ssh.private_key_path = "/path/to/private/key/id_rsa_example"
  end

  # Test bootstrap script
  config.vm.provision :shell, :path => "bootstrap.sh"

end

The machine is brought up with:

vagrant up --provider=cloudstack

The following example output will follow:

$ vagrant up --provider=cloudstack
Bringing machine 'default' up with 'cloudstack' provider...
[default] Warning! The Cloudstack provider doesn't support any of the Vagrant
high-level network configurations (`config.vm.network`). They
will be silently ignored.
[default] Launching an instance with the following settings...
[default]  -- Service offering UUID: 71004023-bb72-4a97-b1e9-bc66dfce9470
[default]  -- Template UUID: a17b40d6-83e4-4f2a-9ef0-dce6af575789
[default]  -- Zone UUID: 1128bd56-b4d9-4ac6-a7b9-c715b187ce11
[default]  -- Keypair: exoscale
[default] Waiting for instance to become "ready"...
[default] Waiting for SSH to become available...
[default] Machine is booted and ready for use!
[default] Rsyncing folder: /Users/sebgoa/Documents/exovagrant/ => /vagrant
[default] Running provisioner: shell...
[default] Running: /var/folders/76/sx82k6cd6cxbp7_djngd17f80000gn/T/vagrant-shell20131203-21441-1ipxq9e
Tue Dec  3 14:25:49 CET 2013
This works

Which is a perfect execution of my amazing bootstrap script:

#!/usr/bin/env bash

/bin/date
echo "This works"

You can now start playing with Chef cookbooks, Puppet recipes or SaltStack formulas and automate the configuration of your cloud instances, thanks to Veewee Vagrant and CloudStack.

Why I will go to CCC13 in Amsterdam ?

Aside from the fact that I work full-time on Apache CloudStack, that I am on the organizing committee and that my boss would kill me if I did not go to the CloudStack Collaboration conference, there are many great reasons why I want to go as an open source enthusiast, here is why:

It's Amsterdam and we are going to have a blast (the city of Amsterdam is even sponsoring the event). The venue -Beurs Van Berlage- is terrific, this is the same venue where the Hadoop summit is held and where the AWS Benelux Summit was couple weeks ago. We are going to have a 24/7 Developer room (thanks to CloudSoft) where we can meet to hack on CloudStack and its ecosystem, three parallel tracks in other rooms and great evening events. The event is made possible by the amazing local support from the team at Schuberg Philis, a company that has devops in its vein and organized DevOps days Amsterdam. I am not being very subtle in acknowledging our sponsors here, but hey, without them this would not be possible.

On the first day (November 20th) is the Hackathon sponsored by exoscale. In parallel to the hackathon, new users of CloudStack will be able to attend a full day bootcamp run by the super competent guys from Shapeblue, they also play guitar and drink beers so make sure to hang out with them :). Even as cool is that the CloudStack community recognizes that building a Cloud takes many components, so we will have a jenkins workshop and an elasticsearch workshop. I am big fan of elasticsearch, not only for keeping your infrastructure logs but also for other types of data. I actually store all CloudStack emails in an elasticsearch cluster. Jenkins of course is at the heart of everyone's continuous integration systems these days. Seeing those two workshops, it will be no surprise to see a DevOps track the next two days.

Kicking off the second day -first day of talks- we will have a keynote by Patrick Debois the jedi master of DevOps. We will then break up into a user track, a developer track, a commercial track and for this day only a devops track with a 'culture' flavor. The hard work will begin: choosing which talk to attend. I am not going to go through every talk, we received a lot of great submissions and choosing was hard. New CloudStack users or people looking into using CloudStack will gain a lot from the case studies being presented in the user track while the developers will get a deep dive into the advanced networking features of CloudStack including SDN support -right off the bat-. In the afternoon, the case studies will continue in the user track including a talk from NTT about how they built an AWS compatible cloud. I will have to head to the developer track for a session on 'interfaces' with a talk on jclouds, a new GCE interface that I worked on and my own talk on Apache libcloud for which I worked a lot on the CloudStack driver. The DevOps track will have an entertaining talk by Michael Ducy from Opscode, some real world experiences by John Turner and Noel King from Paddy Power and the VP of engineering for Citrix CloudPlatform will lead an interactive session on how to best work with the open source community of Apache CloudStack.

After recovering from the nights events, we will head into the second day with another entertaining keynote by John Willis. Here the choice will be hard between the storage session in the commercial track and the 'Future of CloudStack' session in the developer track. With talks from NetApp and SolidFire who have each developed a plugin in CloudStack plus our own Wido Den Hollander (PMC member) who wrote the Ceph integration the storage session will rock, but the 'Future of CloudStack' session will be key for developers, talking about frameworks, integration testing, system VMs...After lunch the user track will feature several intro to networking talks. Networking is the most difficult concept to grasp in clouds (IMHO). The storage session will continue with a talk by Basho on RiakCS (also integrated in CloudStack) and a panel. The dev track will be dedicated to discussions on PaaS, not to be missed if you ask me, as PaaS is the next step in Clouds. To wrap things up, I will have to decide between a session on metering/billing, a discussion on hypervisor choice and support, and a presentation on the CloudStack community in Japan after Ruv Cohen talking about trading cloud commodities.

The agenda is loaded and ready to fire, it will be tough to decide which sessions to attend but you will come out refreshed, energized with lots of new ideas to evolve your IT infrastructure, so one word: Register

And of course many thanks to our sponsors: Citrix, Schuberg Philis, Juniper, Sungard, Shapeblue, NetApp, cloudSoft, Nexenta, iKoula, leaseweb, solidfire, greenqloud, atom86, apalia, elasticsearch, 2source4, iamsterdam, cloudbees and 42on

CloudStack Google Summer of Code projects

Google Summer of Code is entering the final stretch with pencil down on Sept 16th and final evaluation on Sept 27th. Of the five projects CloudStack had this summer, one failed at mid-term and one led to committer status couple weeks ago. That's 20% failure and 20% outstanding results, on par with GSoC wide statistics I believe.

The LDAP integration has been the most productive project. Ian Duffy a 20 year old from Dublin did an outstanding job, developing his new feature in a feature branch, building a jenkins pipeline to test everything and submitting a merge request to master couple weeks ago. With 90% unittest coverage, static code analysis with Sonar in his jenkins pipeline and automatic publishing of rpms in a local yum repo, Ian exceed expectation. His code has even been already backported to the 4.1.1 release with the CloudSand distro of CloudStack.

The SDN extension project was about taking the native GRE controller in CloudStack and extend it to support XCP and KVM. Nguyen from Vietnam has done an excellent job quickly adding support for XCP thanks to his expertise with Xen. He is now putting the final touches on KVM support and building L3 services with OpenDaylight. The entire GRE controller was re-factored to be a plugin similar to the Nicira NVP, Midonet and BigSwitch BVS plugin. While native to CloudStack this controller brings another SDN solution to CloudStack. I expect to see his merge request before pencil down for what will be an extremely valuable project.

While the CloudStack UI is great, it was actually written has a demonstration of how the CloudStack API could be used to build a user facing portal. With the "new UI" project, Shiva Teja from India used boostrap and Angular to create a new UI. Originally the project suggested to use backbone but after feedback from the community Shiva switch to using Angular. Shiva's effort are to be commended as he truly worked on his own with in-consistent network connectivity and no local mentoring. Shiva is a bachelor student and had to learn bootstrap, angular and also Flask on his own. It must have been paying off since he is interviewing with Amazon and Goole for internships next summer. His code being independent of the CloudStack code has been committed to master in our tools directory. This creates a solid framework for other to build on and create their own CloudStack UI.

Perhaps the most research oriented project has been the one from Meng Han from Florida. This was no standard coding projects as it required not only to learn new technologies (aside from CloudStack) but also required investigation of the Amazon EMR API. Meng had to implement EMR in CloudStack using Apache Whirr. Whirr is a java library for provisioning of virtual machines on cloud providers. Whirr uses Apache jclouds and can interact with most cloud providers out there. Meng developed a new set of CloudStack APIs to launch hadoop clusters on-demand. At the start she had to learn CloudStack and install it, then learn the Whirr library and subsequently create a new API in CloudStack which would use Whirr to coordinate multiple node deployments. Meng's code is working but still a bit short from our goal of having a AWS EMR interface. This is partly my fault has this project could have required more mentoring. In any case, the work will go on and I expect to see an EMR implementation in CloudStack in the coming months.

All students faced the same challenge, not a code writing challenge but the OSS challenge and specifically learning the Apache Way. Apache is about consensus and public discussions on the mailing list. With several hundreds participants every month and very active discussion, the shear amount of email traffic can be intimidating. Sharing issues and asking for help on public mailing list is still a bit frightening. IRC, intense emailing, JIRA and git are basic tools used in all Apache project, but seldom used in academic settings. Learning these development tools and participating in a project with over a million line of code was the toughest challenge for students and the goal of GSoC. I am glad we got five students to join CloudStack this summer and tackle these challenges, if anything it is a terrific experience that will benefit their own academic endeavor and later their entire career. Great job Ian, Meng, Nguyen, Shiva and Dharmesh, we are not done yet but I wish you all the best.

And yes Elasticsearch cluster with Whirr and CloudStack as well

In my previous post I showed how to deploy hadoop clusters on-demand using Apache Whirr and CloudStack. Whirr can do much more than hadoop: Cassandra, Ganglia, Solr, Zookeeper etc...and of course Elasticsearch.

This post is really a little wink at a CloudStack PMC member and also a start of investigating if ES would be a better choice than Mongodb for all my email and logs analysis.

Let's get to it. In the Whirr source find the elasticsearch.properties file under /recipes modify it to your liking and to your cloud:

For this test I am using exoscale again, but any CloudStack cloud like ikoula, pcextreme or leaseweb will do (some European chauvinism right there :) ). In a basic zone, specify your ssh keypairs that whirr will use to boostrap ES.

whirr.cloudstack-keypair=exoscale

Set the number of machines in the cluster

# Change the number of machines in the cluster here
whirr.instance-templates=2 elasticsearch

Set the instance type of each cluster instanc (e.g large, extra-large etc..). I have not tried to pass them by name I just use the uuid from CloudStack.

whirr.hardware-id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8

And of course define the endpoint

whirr.provider=cloudstack
whirr.endpoint=https://api.exoscale.ch/compute

And define the template that you want to use (e.g Ubuntu, CentOS etc)

whirr.image-id=1d16c78d-268f-47d0-be0c-b80d31e765d2

Finally, define the ES tarball you want to use. Whirr has not updated this in a long time, so the default is still set at 0.15. Remember to change it.

# You can specify the version by setting the tarball url
whirr.elasticsearch.tarball.url=http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.2.tar.gz

Then launch the cluster

whirr launch-cluster --config elasticsearch.properties

When whirr is done launching and bootstrapping the instance you will get something like:

You can log into instances using the following ssh commands:
[elasticsearch]: ssh -i /Users/toto/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o 
StrictHostKeyChecking=no toto@185.19.28.90
[elasticsearch]: ssh -i /Users/toto/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o 
StrictHostKeyChecking=no toto@185.19.28.92
To destroy cluster, run 'whirr destroy-cluster' with the same options used to launch it.

Don't bother with those IPs, the cluster is already dead when you read this and the security group are set so that I am the only one who can access them. That said you now have a working elasticsearch cluster in the cloud and you can hit it with the API:

$ curl -XGET 'http://185.19.28.90:9200/_cluster/nodes?pretty=true'
{
  "ok" : true,
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "njCzrXYaTnKxtqKsMrV-lA" : {
      "name" : "Powderkeg",
      "transport_address" : "inet[/185.19.28.90:9300]",
      "hostname" : "elasticsearch-f76",
      "version" : "0.90.2",
      "http_address" : "inet[/185.19.28.90:9200]"
    },
    "bdqnkSNfTb63eGMM7CUjNA" : {
      "name" : "USAgent",
      "transport_address" : "inet[/185.19.28.92:9300]",
      "hostname" : "elasticsearch-a28",
      "version" : "0.90.2",
      "http_address" : "inet[/185.19.28.92:9200]"
    }
  }

Really cool, have fun !!!

Apache Whirr and CloudStack for Big Data in the Clouds

This post is a little more formal than usual as I wrote this for a tutorial on how to run hadoop in the clouds, but I thought this was very useful so I am posting it here for everyone's benefit (hopefully).

When CloudStack graduated from the Apache Incubator in March 2013 it joined Hadoop as a Top-Level Project (TLP) within the Apache Software Foundation (ASF). This made the ASF the only Open Source Foundation which contains a cloud platform and a big data solution. Moreover a closer look at the projects making the entire ASF shows that approximately 30% of the Apache Incubator and 10% of the TLPs is "Big Data" related. Projects such as Hbase, Hive, Pig and Mahout are sub-projects of the Hadoop TLP. Ambari, Kafka, Falcon and Mesos are part of the incubator and all based on Hadoop.

To Complement CloudStack, API wrappers such as Libcloud, deltacloud and jclouds are also part of the ASF. To connect CloudStack and Hadoop two interesting projects are also in the ASF: Apache Whirr a TLP, and Provisionr currently in incubation. Both Whirr and Provisionr aimed at providing an abstraction layer to define big data infrastructure based on Hadoop and instantiate those infrastructure on Clouds, including Apache CloudStack based clouds. This co-existence of CloudStack and the entire Hadoop ecosystem under the same Open Source Foundation means that the same governance, processes and development principles apply to both project bringing great synergy that promises an even better complementarity.

In this tutorial we introduce Apache Whirr, an application that can be used to define, provision and configure big data solutions on CloudStack based clouds. Whirr automatically starts instances in the cloud and boostrapps hadoop on them. It can also add packages such as Hive, Hbase and Yarn for map-reduce jobs.

Whirr [1] is a "set of libraries for running cloud services" and specifically big data services. Whirr is based on jclouds [2]. Jclouds is a java based abstraction layer that provides a common interface to a large set of Cloud Services and providers such as Amazon EC2, Rackspace servers and CloudStack. As such all Cloud providers supported in Jclouds are supported in Whirr. The core contributors of Whirr include four developers from Cloudera the well-known Hadoop distribution. Whirr can also be used as a command line tool, making it straightforward for users to define and provision Hadoop clusters in the Cloud.

As an Apache project, Whirr comes as a source tarball and can be downloaded from one of the Apache mirrors [3]. Similarly to CloudStack, Whirr community members can host packages. Cloudera is hosting whirr packages to ease the installation. For instance on Ubuntu and Debian based systems you can add the Cloudera repository by creating /etc/apt/sources.list.d/cloudera.list and putting the following contents in it:

deb [arch=amd64] http://archive.cloudera.com/cdh4/-cdh4 contrib 
deb-src http://archive.cloudera.com/cdh4/-cdh4 contrib

With this repository in place, one can install whirr with:

$sudo apt-get install whirr

The whirr command will now be available. Developers can use the latest version of Whirr by cloning the software repository, writing new code and submitting patches the same way that they would submit patches to CloudStack. To clone the git repository of Whirr do:

$git clone git://git.apache.org/whirr.git

They can then build their own version of whirr using maven:

$mvn install

The whirr binary will be located under the /bin directory. Adding it to one's path with:

$export PATH=$PATH:/path/to/whirr/bin

Will make the whirr command available in the user's environment. Successfull installation can be checked by simply entering:

$whirr --help

With whirr installed, one now needs to specify the credentials of the Cloud that will be used to create the Hadoop infrastructure. A ~/.whirr/credentials has been created during the installation phase. The type of provider (e.g cloudstack), the endpoint of the cloud and the access and secret keys need to be entered in this credentials file like so:

PROVIDER=cloudstack
IDENTITY=
CREDENTIAL=
ENDPOINT=

For instance on Exoscale [4] a CloudStack based cloud in Switzerland, the endpoint would be https://api.exoscale.ch/compute

Now that the CloudStack cloud endpoint and keys have been configured, the hadoop cluster that we want to instantiate needs to be defined. This is done in a properties file using a set of Whirr specific configuration variables [5]. Below is the content of the file with explanations in-line:

---------------------------------------
# Set the name of your hadoop cluster
whirr.cluster-name=hadoop

# Change the name of cluster admin user
whirr.cluster-user=${sys:user.name}

# Change the number of machines in the cluster here
# Below we define one hadoop namenode and 3 hadoop datanode
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,3 hadoop-datanode+hadoop-tasktracker

# Specify which distribution of hadoop you want to use
# Here we choose to use the Cloudera distribution
whirr.env.repo=cdh4
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop

# Use a specific instance type.
# Specify the uuid of the CloudStack service offering to use for the instances of your hadoop cluster
whirr.hardware-id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8

# If you use ssh key pairs to access instances in the cloud
# Specify them like so
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_exoscale
whirr.public-key-file=${whirr.private-key-file}.pub

# Specify the template to use for the instances
# This is the uuid of the CloudStack template
whirr.image-id=1d16c78d-268f-47d0-be0c-b80d31e765d2
------------------------------------------------------

To launch this Hadoop cluster use the whirr command line:

$whirr launch-cluster --config hadoop.properties

The following example output shows the instances being started and boostrapped. At the end of the provisioning, whirr returns the ssh command that shall be used to access the hadoop instances.

-------------------
Running on provider cloudstack using identity mnH5EbKcKeJd456456345634563456345654634563456345
Bootstrapping cluster
Configuring template for bootstrap-hadoop-datanode_hadoop-tasktracker
Configuring template for bootstrap-hadoop-namenode_hadoop-jobtracker
Starting 3 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(b9457a87-5890-4b6f-9cf3-1ebd1581f725)
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(9d5c46f8-003d-4368-aabf-9402af7f8321)
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(6727950e-ea43-488d-8d5a-6f3ef3018b0f)
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-namenode_hadoop-jobtracker} on node(6a643851-2034-4e82-b735-2de3f125c437)
<< success executing InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(b9457a87-5890-4b6f-9cf3-1ebd1581f725): {output=This function does nothing. It just needs to exist so Statements.call("retry_helpers") doesn't call something which doesn't exist
Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
Get:2 http://security.ubuntu.com precise-security Release [49.6 kB]
Hit http://ch.archive.ubuntu.com precise Release.gpg
Get:3 http://ch.archive.ubuntu.com precise-updates Release.gpg [198 B]
Get:4 http://ch.archive.ubuntu.com precise-backports Release.gpg [198 B]
Hit http://ch.archive.ubuntu.com precise Release
..../snip/.....
You can log into instances using the following ssh commands:
[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.xx.yy.zz
[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.zz.zz.rr
[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.tt.yy.uu
[hadoop-namenode+hadoop-jobtracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.ii.oo.pp
-----------

To destroy the cluster from your client do:

$whirr destroy-cluster --config hadoop.properties.

Whirr gives you the ssh command to connect to the instances of your hadoop cluster, login to the namenode and browse the hadoop file system that was created:

$ hadoop fs -ls /
Found 5 items
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:11 /hadoop
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:10 /hbase
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:10 /mnt
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:11 /tmp
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:11 /user

Create a directory to put your input data.

$ hadoop fs -mkdir input
$ hadoop fs -ls /user/sebastiengoasguen
Found 1 items
drwxr-xr-x   - sebastiengoasguen supergroup          0 2013-06-21 20:15 /user/sebastiengoasguen/input

Create a test input file and put in the hadoop file system:

$ cat foobar 
this is a test to count the words
$ hadoop fs -put ./foobar input
$ hadoop fs -ls /user/sebastiengoasguen/input
Found 1 items
-rw-r--r--   3 sebastiengoasguen supergroup         34 2013-06-21 20:17 /user/sebastiengoasguen/input/foobar

Define the map-reduce environment. Note that this default Cloudera distribution installation uses MRv1. To use Yarn one would have to edit the hadoop.properties file.

$ export HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce

Start the map-reduce job:

$ hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar wordcount input output
13/06/21 20:19:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/21 20:20:00 INFO input.FileInputFormat: Total input paths to process : 1
13/06/21 20:20:00 INFO mapred.JobClient: Running job: job_201306212011_0001
13/06/21 20:20:01 INFO mapred.JobClient:  map 0% reduce 0%
13/06/21 20:20:11 INFO mapred.JobClient:  map 100% reduce 0%
13/06/21 20:20:17 INFO mapred.JobClient:  map 100% reduce 33%
13/06/21 20:20:18 INFO mapred.JobClient:  map 100% reduce 100%
13/06/21 20:20:21 INFO mapred.JobClient: Job complete: job_201306212011_0001
13/06/21 20:20:22 INFO mapred.JobClient: Counters: 32
13/06/21 20:20:22 INFO mapred.JobClient:   File System Counters
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of bytes read=133
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of bytes written=766347
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of read operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of write operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of bytes read=157
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of bytes written=50
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of read operations=2
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of write operations=3
13/06/21 20:20:22 INFO mapred.JobClient:   Job Counters 
13/06/21 20:20:22 INFO mapred.JobClient:     Launched map tasks=1
13/06/21 20:20:22 INFO mapred.JobClient:     Launched reduce tasks=3
13/06/21 20:20:22 INFO mapred.JobClient:     Data-local map tasks=1
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=10956
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=15446
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/21 20:20:22 INFO mapred.JobClient:   Map-Reduce Framework
13/06/21 20:20:22 INFO mapred.JobClient:     Map input records=1
13/06/21 20:20:22 INFO mapred.JobClient:     Map output records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Map output bytes=66
13/06/21 20:20:22 INFO mapred.JobClient:     Input split bytes=123
13/06/21 20:20:22 INFO mapred.JobClient:     Combine input records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Combine output records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce input groups=8
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce shuffle bytes=109
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce input records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce output records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Spilled Records=16
13/06/21 20:20:22 INFO mapred.JobClient:     CPU time spent (ms)=1880
13/06/21 20:20:22 INFO mapred.JobClient:     Physical memory (bytes) snapshot=469413888
13/06/21 20:20:22 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=5744541696
13/06/21 20:20:22 INFO mapred.JobClient:     Total committed heap usage (bytes)=207687680

And you can finally check the output:

$ hadoop fs -cat output/part-* | head
this 1
to  1
the  1
a  1
count 1
is  1
test 1
words 1

Of course this is a silly example of map-reduce job and you will want to do much more critical tasks. In order to benchmark your cluster Hadoop comes with examples jar.

To benchmark your hadoop cluster you can use the TeraSort tools available in the hadoop distribution. Generate some 100 MB of input data with TeraGen (100 byte rows):

$hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar teragen 1000000 output3

Sort it with TeraSort:

$ hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar terasort output3 output4

And then validate the results with TeraValidate:

$hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar teravalidate output4 outvalidate

Performance of map-reduce jobs run in Cloud based hadoop clusters will be highly dependent on the hadoop configuration, the template and the service offering being used and of course on the underlying hardware of the Cloud. Hadoop was not designed to run in the Cloud and therefore some assumptions were made that do not fit the Cloud model, see [6] for more information. Deploying Hadoop in the Cloud however is a viable solution for on-demand map-reduce applications. Development work is currently under way within the Google Summer of Code program to provide CloudStack with a compatible Amazon Elastic Map-Reduce (EMR) service. This service will be based on Whirr or a new Amazon CloudFormation compatible interface called StackMate [7].

[1] http://whirr.apache.org
[2] http://jclouds.incubator.apache.org
[3] http://www.apache.org/dyn/closer.cgi/whirr/
[4] http://exoscale.ch
[5] http://whirr.apache.org/docs/0.8.2/configuration-guide.html
[6] http://wiki.apache.org/hadoop/Virtual%20Hadoop
[7] https://github.com/chiradeep/stackmate

CloudStack support in Apache Libcloud

A vote has started on the libcloud dev list for the 0.13 release. The release notes detail all the new features and fixes. I am stoked about this because a few patches that I submitted are included in this release candidate. I patched the CloudStack driver quite a bit to improve the support for Clouds with a basic zone like Exoscale. There is more work to do on this driver including better support for Advanced zone especially for port forwarding, firewall rules and more unit tests. A the CloudStack hackathon last sunday @pst418 submitted a few patches for unit tests and they made it as well into 0.13 RC, terrific.

If you don't know libcloud, it's a python based API wrapper to abstract the various cloud APIs. With libcloud you can create connections to multiple clouds, potentially using different API. At a high level, it is similar to jclouds for JAVA or deltacloud written in ruby. There was already a CloudStack driver but its functionality was limited. If you grab my quickie libcloud shell, you can try to follow this walkthrough of what you can do with libcloud and a CloudStack basic zone. Of course you will need a CloudStack endpoint.

Start the libshell and check what zone you are on:

$ python ./libshell.py 
Hello LibCloud Shell !!
You are running at: https://api.exoscale.ch/compute
>>> conn.list_locations()
[<NodeLocation: id=1128bd56-b4d9-4ac6-a7b9-c715b187ce11, name=CH-GV2, country=AU, driver=CloudStack>]

You might notice a wrong country code, it is hard-coded in libcloud, I need to file a bug for this. Get the list of templates (or images in libcloud speak):

>>> conn.list_images()
[<NodeImage: id=01df77c3-0150-412a-a580-413a50924a18, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=89ee852c-a5f5-4ab9-a311-89f39d133e88, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=ccd142ec-83e3-4108-b1c5-8e1fdb353ff9, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=f2101a0c-eaf7-4760-a143-0a5c940fd864, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=77d32782-6866-43d4-9524-6fe346594d09, name=CentOS 5.5(64-bit) no GUI (KVM), driver=CloudStack  ...>,
 <NodeImage: id=29cba09f-4569-4bb3-95e7-71f833876e3e, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=ee241b47-4303-40c8-af58-42ed6bf09f8c, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=754ea486-d649-49e5-a70e-5e5d458f0df0, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=0f9f4f49-afc2-4139-b26b-b05a9f51ea74, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=954752a8-0486-46bb-8e3f-0adb3e01c619, name=Linux CentOS 6.4 64-bit, driver=CloudStack  ...<]

I cut the previous output, but there are also Ubuntu and CentOS images on that cloud...You can then list the various instance types or sizes in libcloud speak.

>>> conn.list_sizes()
[<NodeSize: id=71004023-bb72-4a97-b1e9-bc66dfce9470, name=Micro, ram=512 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8, name=Tiny, ram=1024 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=21624abb-764e-4def-81d7-9fc54b5957fb, name=Small, ram=2048 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=b6e9d1e8-89fc-4db3-aaa4-9b4c5b1d0844, name=Medium, ram=4096 disk=0 bandwidth=0 price=0 driver=CloudStack ...>, 
<NodeSize: id=c6f99499-7f59-4138-9427-a09db13af2bc, name=Large, ram=8182 disk=0 bandwidth=0 price=0 driver=CloudStack ...>, 
<NodeSize: id=350dc5ea-fe6d-42ba-b6c0-efb8b75617ad, name=Extra-large, ram=16384 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=a216b0d1-370f-4e21-a0eb-3dfc6302b564, name=Huge, ram=32184 disk=0 bandwidth=0 price=0 driver=CloudStack ...>]

What I added is the management of ssh key pairs and security groups, you can now list, create and delete both keypairs and security groups, and use those when deploying nodes. (Don't bother trying to do anything with the keys below, I deleted everything.)

>>> conn.ex_list_keypairs() 
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'Default 
Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': u'0.0.0.0/0', u'protocol':
 u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'0.0.0.0/0', u'protocol':
 u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': 
u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test',
 u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

>>> conn.ex_create_keypair('test')
{u'privateKey': u'-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQCAuPufJnbzFkyIUaMymkXwfilA5qS2J9fx2Q3EWNDs1m89FLFq\n19ERjG43ZvRuI/KGwWZwHbhqoqJth7WQYNe
lYCmOUYRaepxTrpU4TGDGuhqMh9D9\
noNMIFx3ktkcTitxkSY/5h/pXqSB2XXURLpZWwZxjEYwWCpWE8i7uVtIR7wIDAQAB\nAoGAOv0Kik9lOVbhsaK/yAO8w039d7l6h+NQaYtPbMhKzg4iofomp9DJBWK2a3sp\
nzoN4s9pTKFPmXC+1gb4sLULD72ENSgyozrPMCJ0tytNa3ebVCCYDvlagEZ83KwGn\nnr2BIRLul1xyHVa+amvTemuxi7OOx0u/0aZ6jPVW9ocPahkCQQDnUACG3Q2p60Mx\
nCttW7Go2wz0BhaI8ibG8rHorhdrFJUrsTI6QrLh340uHCIEAuWUXjYDX2u5MTPBG\njWbL/A/9AkEAjnX8EzO/B/RooFTxhYdxv7D1Dzcx4H59mFzjez6N/mjAHjKL4p66\
nqGFgLa9HMhDPFOs83VetBXcihu1vueffWwJBALQUD2TvAS0wz82FYz8nrITXuE3Q\nCH7Sv8FgEXiCq89hehO+ghrVrIMBPBJzJ2M18iLE8fKaKXzTRRfYC5hwss0CQHY8\
nBdIKCGoZtxwaY7lnCEkIHNtb+9FOKf7iWQpYiJC1b32ghei3xEMrTh+ccYJj4PqD\noigyNC9tCQLi3O92OjECQGYR2z8IDlfnl7G8p7eiyBciuoDyyq5/oJHhkHbkbwzr\
nW0Uun+rEcjRnXbUN8wUQ6FSFrxk2VSajbCBteTOrF24=\n-----END RSA PRIVATE KEY-----\n', u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}

>>> conn.ex_list_keypairs()
[{u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}, {u'name': u'foobar',
 u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_delete_keypair('test')
u'true'

>>> conn.ex_list_keypairs()
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_create_security_group('heyhey')
{u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id':
 u'77ad73b5-a383-4e13-94be-a38ef9877996', u'domain': u'runseb@gmail.com', u'name': u'heyhey'}

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': 
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'
0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
 u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description':
 u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}, 
{u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id': u'77ad73b5-a383-4e13-94be-a38ef9877996',
 u'domain': u'foobar@gmail.com', u'name': u'heyhey'}]

>>> conn.ex_delete_security_group('heyhey')
u'true'

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': 
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
 u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

With a security group created you can now add an ingress rule:

>>> conn.ex_authorize_security_group_ingress(securitygroupname='toto',protocol='TCP',cidrlist='0.0.0.0./0',
startport=99)
{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', 
u'description': u'this is a test', u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 99, u'cidr': u'0.0.0.0./0', 
u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}], 
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', 
u'description': u'Default Security Group', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id':
 u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': u'default'}, {u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': 
u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': 
[{u'startport': 99, u'cidr': u'0.0.0.0./0', u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}], 
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

Great, keypair and security group work fine. Now let's do the basics of starting an instance and let's test the suspend and resume which was not in the Driver until now. Nothing earth shattering but it's an improvement.

>>> size=conn.list_sizes()
>>> image=conn.list_images()

>>> n=conn.create_node(name='yoyo',size=size[0],image=image[0])

>>> n.ex_stop()
u'Stopped'
>>> n.ex_start()
u'Running'

And that's it for now, CloudStack support is getting better, there is still tons of work to do: improve the advanced zone support, check the load-balancer and storage support, add unit-tests and new CloudStack features like auto-scaling. Carry on !!!!

The Tuesday CloudStack Silly Hack

The problem with middleware/backend work is that nobody sees what you do and since I am terrible at graphics/design and know nothing of User Interface principles I am pretty much stuck in the dark. So today I invested couple hours on the Tuesday Silly CloudStack Hack. It is made of Flask, a good read, Twitter Bootstrap and some stolen code from CloudMonkey

Flask is a terrific web microframework for Python. Of course I like Python so I think Flask is terrific. It is also great because you can use it to design clean REST services. Bootstrap is en vogue these days, and CloudMonkey, also written in Python is the new CloudStack command line interface boasting some cool features, like auto-completion, interactive shell and so on.

I will keep it short, grab requester.py from the CloudMonkey source tree, it will help you make API calls to a CloudStack instance. Create a simple script with flask, and set your CloudStack endpoint variables, plus the keys (I am using DevCloud):

import requester

from flask import Flask, url_for, render_template, request
app = Flask(__name__)

apikey='plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg'
secretkey='VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ'
path='/client/api'
host='localhost'
port='8080'
protocol='http'

Setup a route that you will use to trigger a call to CloudStack. Something like this:

@app.route('/users')
def listusers():
    response, error = requester.make_request('listUsers',{},None,host,port,apikey,secretkey,protocol,path)
    resp=json.loads(str(response))
    return render_template('users.html',users=resp['listusersresponse'])

Now Download Bootstrap and stick it in a static directory in your Flask application. Then create html template files using the jinja2 syntax, something like this:

{% extends "base.html" %}

{% block content %}

{% if users %}
{{ users }}
{% else %}
Hello World!
{% endif %}

{% endblock content %}
sebmini:templates sebastiengoasguen$ 

Now run the app with Python and hit http://localhost:5000 and bang, you just got yourself your quarter end Bonus....! Well not quite, but that's a start :)

Joke aside, this should be the start of a fun Google Summer of Code project, I will put it on github if there is interest. Also if you want another silly hack, push requester.py to your android phone and using SL4a you can make calls to CloudStack from your phone...Silly CloudStack Wednesday hack anyone ?

Quantifying the Apache CloudStack Community

CloudStack is now an Apache Top Level Project (TLP) at the Apache Software Foundation (ASF), the announcement just came out. What the incubation period has meant for CloudStack has nothing to do with code maturity. CloudStack was mature and used in enterprise settings before it entered incubation at the ASF. What incubation meant was that CloudStack evolved into an open source community, self-governed by the Apache Way: transparency, meritocracy, respect, non-affiliation and consensus in no particular order. The community has learned and demonstrated that it understands the principles and processes laid by the Apache Software Foundation and that it can now operate more autonomously.

Growing an open source community is challenging, folks who participate come from various backgrounds, may seldom meet and interact mostly via emails, social media, instant messaging. Participants come from all over the world, work in different time-zones and donate their time after (and sometime while) dealing with day jobs and family. Participants rally around a project that they deem interesting, sometime just to land a hand for a few months, or sometimes because their day job requires them to do it. In that very heterogeneous and fluctuating mix, an open source software community emerges, self-governed, sustainable and non-affiliated. In the last 12 months CloudStack has done just that, building a community from the ground up, developing and understanding the principles laid out in our bylaws, adapting -if need be- people's way of developing software, getting to know each other, welcoming new members every day and setting the foundation for a sustainable software.

When growing a community it is fairly natural to want to measure how well we are doing and how healthy the community is. Over the last several months I have started collecting some data to analyze our community, trying to see how we were doing and interacting. I mostly looked at our public mailing lists doing a study similar to the one done about comparing CloudStack, OpenNebula, OpenStack and Ecualyptus. Secretly, this was also a good way for me to sharpen a few skills on BigData, not that big actually but I used MongoDB instead of MySQL so that qualifies as BigData :). Defining membership in an open source community is a challenge since there is no concept of membership, even the concept of contribution is ill-defined. What constitutes a contribution ? Which channels need to be considered ? In the case of ASF for instance, contribution to code may only mean being a committer, but a committer is someone with write access to the code. Just counting committers will leave out all the folks sending patches, doing testing, doing user support, translating documentation, giving talks and so on. Also while at the ASF everything happens on the mailing list, what about IRC channels, social media like Facebook, Twitter and Linkedin, and what about other communities that may arise around a particular software: sub-projects, user groups etc. In this analysis I decided to only look at our public mailing lists but there is more to it than just this data source.

The two figures below show the number of individual contributors measured by unique email addresses used to send messages to the users and developers mailing lists. The red lines represent data from the users and developers mailing list prior to entering incubation at the ASF. The blue lines represent the ASF specific lists. Significant is the impact that the move to the ASF has had on the number of contributors. The developers list has peaked over 200 per month and the users list has peaked over 150 per month so far (figure on the right). The last data point is March (as of March 21st) and numbers will go up by the end of the month. The graph on the right shows the accumulation of contributors, adding all unique email addresses every month into a set. This shows us again that the move to ASF has had a huge impact on the growth rate of the community and that both list grow at relatively the same pace. Adding the accumulated number of contributors to both list and removing duplicates present in both sets, this gives us a magic number of 722 CloudStack contributors to date.

As mentioned earlier these numbers of contributors are different from the number of Apache CloudStack committers. According to our bylaws contributors can become committers and gain write access to the code when invited by the Project Management Committee (PMC). Just today we welcomed three new committers for a current total of 54. This number is again relative in terms of contribution to code, since committers apply patches from contributor who are not yet committers (I know confusing :) ). A quick look at our git repo, shows a current total of 159 code contributors. To that we could also add the 32 contributors who helped with the translation via transifex which may or may not have participated in the mailing lists.

These numbers show a growing community sparked by the move to the ASF. While at the start of incubation the initial set of committers and contributors where only from Citrix who donated the code, we are also seeing a diversification in the number of companies involved. Talking about affiliation is actually a cardinal sin of the Apache Way. At the ASF only individuals matter, but it is fair to say that for CloudStack to be successful we need to see adoption/participation by a diverse set of companies. The two figures below show just that with the number of companies identified by the email domain used by the contributors. This is not perfect since contributors often do not use their work email but gives a good idea of the trend. Similarly to the plots about contributors we plot the number of companies (really email domains). Clearly the growth in the number of companies involved has increased since joining the ASF, we currently see around 50 companies involved in CloudStack every month. Accumulating all these companies we reach almost 200 on both lists. Removing duplicates from both sets we estimate the total number of companies to 272. Even if some of those companies are only represented by a single individual this is still a very strong number that shows great diversification. Interestingly over the last three months we see that the participation in the user list has shown more diversity than on the developer list.

Finally, looking at the cloudstack-commits mailing list we try to identify the number of committers to the code. The figure below shows the number of committers and the number of commits measured by number of emails and unique email addresses used on the commits mailing list. We already mentioned that we had 54 committers but that some of those committers gained their rights based on contribution which could be as diverse as development, user support, translation etc. Therefore it is no surprise to see the number of committers peaking at 35 and the accumulated number reaching 45. The number of commits is growing, reaching 1500 a month during the 4.0 release, easing up during christmas time and peaking again over 2500 per month prior to code freeze of the 4.1 release.

All in all, analyzing lots of emails showed me that moving to ASF has had a tremendous impact on CloudStack, with over 700 contributors, close to 60 committers, 30 translators, over 1 million lines of code, CloudStack is self-governed and here to stay.