Eutester with CloudStack

Eutester

An interesting tool based on Boto is Eutester it was created by the folks at Eucalyptus to provide a framework to create functional tests for AWS zones and Eucalyptus based clouds. What is interesting with eutester is that it could be used to compare the AWS compatibility of multiple clouds. Therefore the interesting question that you are going to ask is: Can we use Eutester with CloudStack ? And the answer is Yes. Certainly it could use more work but the basic functionality is there.

Install eutester with:

pip install eutester

Then start Python/iPython interactive shell or write a script that will import ec2ops and create a connection object to your AWS EC2 compatible endpoint. For example, using ec2stack:

    #!/usr/bin/env python

    from eucaops import ec2ops
    from IPython.terminal.embed import InteractiveShellEmbed

    accesskey="my api key"
    secretkey="my secret key"

    conn.ec2ops.EC2ops(endpoint="localhost",
                   aws_access_key_id=apikey,
                   aws_secret_access_key=secretkey,
                   is_secure=False,
                   port=5000,
                   path="/",
                   APIVersion="2014-02-01")

    ipshell = InteractiveShellEmbed(banner1="Hello Cloud Shell !!")
    ipshell()

Eutester at the time of this writing has 145 methods. Only the methods available through the CloudStack AWS EC2 interface will be availble. For example, get_zones and get_instances would return:

In [3]: conn.get_zones()
Out[3]: [u'ch-gva-2']

In [4]: conn.get_instances()
[2014-05-21 05:39:45,094] [EUTESTER] [DEBUG]: 
--->(ec2ops.py:3164)Starting method: get_instances(self, state=None, 
     idstring=None, reservation=None, rootdevtype=None, zone=None,
     key=None, pubip=None, privip=None, ramdisk=None, kernel=None,
     image_id=None, filters=None)
Out[4]: 
[Instance:5a426582-3aa3-49e0-be3f-d2f9f1591f1f,
 Instance:95ee8534-b171-4f79-9e23-be48bf1a5af6,
 Instance:f18275f1-222b-455d-b352-3e7b2d3ffe9d,
 Instance:0ea66049-9399-4763-8d2f-b96e9228e413,
 Instance:7b2f63d6-66ce-4e1b-a481-e5f347f7e559,
 Instance:46d01dfd-dc81-4459-a4a8-885f05a87d07,
 Instance:7158726e-e76c-4cd4-8207-1ed50cc4d77a,
 Instance:14a0ce40-0ec7-4cf0-b908-0434271369f6]

This example shows that I am running eight instances at the moment in a zone called ch-gva-2, our familiar exoscale. Selecting one of these instance objects will give you access to all the methods available for instances. You could also list, delete and create keypairs. List, delete and create security groups etc.

Eutester is meant for building integration tests and easily creating test scenarios. If you are looking for a client to build an application with, use Boto.

The master branch of eutester may still cause problems to list images from a CloudStack cloud. I recently patched a fork of the testing branch and opened an issue on their github page. You might want to check its status if you want to use eutester heavily.

Migrating from Publican to Sphinx and Read The Docs

Migration from Publican to Sphinx and Read The Docs

When we started with Cloudstack we chose to use publican for our documentation. I don't actually know why, except that Red Hat documentation is entirely based on publican. Perhaps David Nalley's background with Fedora influenced us :) In any case publican is a very nice documentation building system, it is based on the docbook format and has great support for localization. However it can become difficult to read and organize lots of content, and builds may break for strange reasons. We also noticed that we were not getting many contributors to the documentation, in contrast, the translation efforts via transifex has had over 80 contributors. As more features got added to CloudStack the quality of the content also started to suffer and we also faced issues with publishing the translated documents. We needed to do something, mainly making it easier to contribute to our documentation. Enters ReStructured Text (RST) and Read The Docs (RTD).

Choosing a new format

We started thinking about how to make our documentation easier to contribute to. Looking at Docbook, purely xml based, it is a powerful format but not very developer friendly. A lot of us are happy with basic text editor, with some old farts like me mainly stuck with vi. Markdown has certainly helped a lot of folks in writing documentation and READMEs, just look at Github projects. I started writing in Markdown and my production in terms of documentation and tutorials skyrocketed, it is just a great way to write docs. Restructured Text is another alternative, not really markdown, but pretty close. I got familiar with RST in the Apache libcloud project and fell in love with it, or at least liked it way more than docbook. RST is basically text, only a few markups to learn and your off.

Publishing Platform

A new format is one thing but you then need to build documentation in multiple formats: html, pdf, epub potentially more. How do you move from .rst to these formats for your projects ? Comes in Sphinx, pretty much an equivalent to publican originally aimed at Python documentation but now aimed at much more. Installing sphinx is easy, for instance on Debian/Ubuntu:

apt-get install python-sphinx

You will then have the sphinx-quickstart command in your path, use it to create your sphinx project, add content in index.rst and build the docs with make html. Below is a basic example for a ff sample project.

What really got me sold on reStructuredText and Sphinx was ReadTheDocs (RTD). It hosts documentation for open source projects. It automatically pulls your documentation from your revision control system and builds the docs. The killer feature for me was the integration with github (not just git). Using hooks, RTD can trigger builds on every commit and it also displays an edit on github icon on each documentation page. Click on this icon, and the docs repository will get forked automatically on your github account. This means that people can edit the docs straight up in the github UI and submit pull requests as they read the docs and find issues.

Conversion

After [PROPOSAL] and [DISCUSS] threads on the cloudstack mailing list, we reached consensus and decided to make the move. This is still on-going but we are getting close to going live with our new docs in RST and hosted by RTD. There were couple challenges:

1. Converting the existing docbook based documentation to RST
2. Setting up new repos, CNAMEs and Read The Docs projects
3. Setting up the localization with transifex

The conversion was much easier than expected thanks to pandoc, one of those great command line utility that saves your life.

pandoc -f docbook -t rst -o test.rst test.docbook

You get the just of it, iterate through your docbook files and generate the RST files, combine everything to reconstruct your chapters and books and re-organize as you wish. They are off course couple gotchas, namely the table formatting may not be perfect, the note and warnings may be a bit out of whack and the heading levels should probably be checked. All of these are actually good to check as a first pass through the docs to revamp the content and the way it is organized.
One thing that we decided to do before talking about changing the format was to move our docs to a separate repository. What we wanted to do was to be able to release docs on a different time frame than the code release, as well as make any doc bug fixes go live as fast as possible and not wait for a code release (that's a long discussion...). With a documentation specific repo in place, we used Sphinx to create the proper directory structure and add the converted RST files. Then we created a project on Read The Docs and pointed to the github mirror of our Apache git repo. Pointing to the github mirror allowed us to enable the nice github interaction that RTD provides. The new doc site looks like this.

There is a bit more to it, as we actually created several repositories and used a RTD feature called subprojects to make all the docs live under the same CNAME docs.cloudstack.apache.org. This is still work in progress but in track for the 4.3 code release. I hope to be able to announce the new documentation sites shortly after 4.3 is announced.
The final hurdle is the localization support. Sphinx provides utilities to generate POT files. They can then be uploaded to transifex and translation strings can be pulled to construct the translated docs. The big challenge that we are facing is to not loose the existing translation that were done from the docbook files. Strings may have changed. We are still investigating how to not loose all that work and get back on our feet to serve the translated docs. The Japanese translators have started to look at this.
Overall the migration was easy, ReStructuredText is easy, Sphinx is also straigthfoward and Read The Docs provides a great hosting platform well integrated with Github. Once we go live, we will see if our doc contributors increase significantly, we have already seen a few pull requests come in, which is very encouraging.
I will be talking about all of this at the Write The Docs conference in Budapest on March 31st, april 1st. If you are in the area stop by :)

OCCI interface to CloudStack

CloudStack has its own API. Cloud wrappers like libcloud and jclouds work well with this native API, but CloudStack does not expose any standard API like OCCI and CIMI. We (Isaac Chiang really, I just tested and pointed him in the right direction) started working on a CloudStack backend for rOCCI using our CloudStack ruby gem. The choice of rOCCI was made due to the existence of an existing Opennebula backend and the adoption of OCCI in the European Grid Initiative Federated cloud testbed.

Let's get started with installing the rOCCI server, this work has not yet been merged upstream so you will need to work from Isaac Chiang's fork.

    git clone https://github.com/isaacchiang/rOCCI-server.git
    bundle install
    cd etc/backend
    cp cloudstack/cloudstack.json default.json

Edit the defautl.json file to contain the information about your CloudStack cloud (e.g apikey, secretkey, endpoint). Start the rOCCI server:

    bundle exec passenger start

The server should be running on http://0.0.0.0:3000 and run the tests:

    bundle exec rspec

This was tested with the CloudStack simulator and a basic zone configuration, help us test it in production clouds.

You can also try an OCCI client. Install the rOCCI client from Github:

    git clone https://github.com/gwdg/rOCCI-cli.git

    cd rOCCI-cli
    gem install bundler
    bundle install
    bundle exec rake test
    rake install

You will then be able to use the OCCI client:

    occi --help

Test it against the server that you are started previously. You will need a running CloudStack cloud. Either a production one or a dev instance using DevCloud. The credentials and the endpoint to this cloud will have been entered in `default.json` file that you created in the previous section. Try a couple OCCI client command:

    $ occi --endpoint http://0.0.0.0:3000/ --action list --resource os_tpl

    Os_tpl locations:
     os_tpl#6673855d-ce9b-4997-8613-6830de037a8f

    $ occi --endpoint http://0.0.0.0:3000/ --action list --resource resource_tpl

    Resource_tpl locations:
     resource_tpl##08ba0343-bd39-4bf0-9aab-4953694ae2b4
     resource_tpl##f78769bd-95ea-4139-ad9b-9dfc1c5cb673
     resource_tpl##0fd364a9-7e33-4375-9e10-bb861f7c6ee7

You will recognize the `uuid` from the templates and service offerings that you have created in CloudStack. To start an instance:

    $ occi --endpoint http://0.0.0.0:3000/ --action create --resource compute 
           --mixin os_tpl#6673855d-ce9b-4997-8613-6830de037a8f 
           --mixin resource_tpl#08ba0343-bd39-4bf0-9aab-4953694ae2b4
           --resource-title foobar

A handle on the resource created will be returned. That's it !

We will keep on improving this driver to provide a production quality OCCI interface to users who want to use a standard. In all fairness we will also work on a CIMI implementation. Hopefully some of the clouds in the EGI federated cloud will pick CloudStack and help us improve this OCCI interface. In CloudStack we aim to provide the interfaces that the users want and keep them up to date and of production quality so that users can depend on it.

About those Cloud APIs....

There has been a lot of discussions lately within the OpenStack community on the need for an AWS API interface to OpenStack nova. I followed the discussion from far via a few tweets, but I am of the opinion that any IaaS solution does need to expose an AWS interface. AWS is the leader in Cloud and has been since 2006 -yes that's seven years- Users are accustomed to it and the AWS API is the de-factor standard.

When Eucalyptus started, it's main goal was to become an AWS clone and in 2012 signed an agreement with Amazon to offer seamless AWS support in Eucalyptus. Opennebula has almost always offered an AWS bridge and CloudStack has too, even though in total disclosure the interface was broken in the Apache CloudStack 4.1 release. Thankfully the AWS interface is now fixed in 4.1.2 release and will also be in the upcoming 4.2 release. To avoid breaking this interface we are developing a jenkins pipeline which will test it using the Eucalyptus testing suite.

Opennebula recently ran a survey to determine where to best put its efforts in API development. The results where clear with 47% of respondents asking for better AWS compatibility. There are of course developing official standards from standard organizations, most notably OCCI from OGF and CIMI from DMTF. The opennebula survey seems to indicate a stronger demand for OCCI than CIMI, but IMHO this is due to historical reasons: Opennebula early efforts in being the first OCCI implementation and Opennebula user base especially within projects like HelixNebula.

CIMI was promising and probably still is but it will most likely face an up-hill battle since RedHat announced it's scaling back on supporting Apache DeltaCloud. I recently heard about a new CIMI implementation project for Stratuslab from some of my friends at Sixsq, it is interesting and fun because written in Clojure and I hope to see it used with Clostack to provide a CIMI interface to CloudStack. We may be couple weeks out :)

While AWS is the de-facto standard, I want to make sure that CloudStack offers choices for its users. If someone wants to use OCCI and CIMI or AWS or the native CloudStack API they should be able to. I will be at the CloudPlugfest Interoperability week in Madrid Sept 18-20 and I hope to demonstrate a brand new OCCI interface to CloudStack using rOCCI and CloudStack ruby gem. A CloudStack contributor from Taiwan has been working on it.

The main issue with all these "standard" interfaces is that they will never give you the complete API of a given IaaS implementation. They by nature provide the lowest common denominator. That roughly means that the user facing APIs could be standardized but the administrator API will always remain hidden and non-standard. In CloudStack for instance, there are over 300 API calls. While we can expose a compatible interface, this will always only cover a subset of the overall API. It also brings the question of all the other AWS services: EMR, Beanstalk, CloudFormation... Standardizing on those will be extremely difficult if not impossible.

So yes, we should expose an AWS compatible interface, but we should also have OCCI, CIMI and of course our native API. Making those bridges is not hard, what's hard is the implementation behind it.

All of this would leave us with Google Compute Engine (GCE), and I should be able to bring back some good news by the end of september, stay tuned !!!

And yes Elasticsearch cluster with Whirr and CloudStack as well

In my previous post I showed how to deploy hadoop clusters on-demand using Apache Whirr and CloudStack. Whirr can do much more than hadoop: Cassandra, Ganglia, Solr, Zookeeper etc...and of course Elasticsearch.

This post is really a little wink at a CloudStack PMC member and also a start of investigating if ES would be a better choice than Mongodb for all my email and logs analysis.

Let's get to it. In the Whirr source find the elasticsearch.properties file under /recipes modify it to your liking and to your cloud:

For this test I am using exoscale again, but any CloudStack cloud like ikoula, pcextreme or leaseweb will do (some European chauvinism right there :) ). In a basic zone, specify your ssh keypairs that whirr will use to boostrap ES.

whirr.cloudstack-keypair=exoscale

Set the number of machines in the cluster

# Change the number of machines in the cluster here
whirr.instance-templates=2 elasticsearch

Set the instance type of each cluster instanc (e.g large, extra-large etc..). I have not tried to pass them by name I just use the uuid from CloudStack.

whirr.hardware-id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8

And of course define the endpoint

whirr.provider=cloudstack
whirr.endpoint=https://api.exoscale.ch/compute

And define the template that you want to use (e.g Ubuntu, CentOS etc)

whirr.image-id=1d16c78d-268f-47d0-be0c-b80d31e765d2

Finally, define the ES tarball you want to use. Whirr has not updated this in a long time, so the default is still set at 0.15. Remember to change it.

# You can specify the version by setting the tarball url
whirr.elasticsearch.tarball.url=http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.2.tar.gz

Then launch the cluster

whirr launch-cluster --config elasticsearch.properties

When whirr is done launching and bootstrapping the instance you will get something like:

You can log into instances using the following ssh commands:
[elasticsearch]: ssh -i /Users/toto/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o 
StrictHostKeyChecking=no toto@185.19.28.90
[elasticsearch]: ssh -i /Users/toto/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o 
StrictHostKeyChecking=no toto@185.19.28.92
To destroy cluster, run 'whirr destroy-cluster' with the same options used to launch it.

Don't bother with those IPs, the cluster is already dead when you read this and the security group are set so that I am the only one who can access them. That said you now have a working elasticsearch cluster in the cloud and you can hit it with the API:

$ curl -XGET 'http://185.19.28.90:9200/_cluster/nodes?pretty=true'
{
  "ok" : true,
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "njCzrXYaTnKxtqKsMrV-lA" : {
      "name" : "Powderkeg",
      "transport_address" : "inet[/185.19.28.90:9300]",
      "hostname" : "elasticsearch-f76",
      "version" : "0.90.2",
      "http_address" : "inet[/185.19.28.90:9200]"
    },
    "bdqnkSNfTb63eGMM7CUjNA" : {
      "name" : "USAgent",
      "transport_address" : "inet[/185.19.28.92:9300]",
      "hostname" : "elasticsearch-a28",
      "version" : "0.90.2",
      "http_address" : "inet[/185.19.28.92:9200]"
    }
  }

Really cool, have fun !!!

Apache Whirr and CloudStack for Big Data in the Clouds

This post is a little more formal than usual as I wrote this for a tutorial on how to run hadoop in the clouds, but I thought this was very useful so I am posting it here for everyone's benefit (hopefully).

When CloudStack graduated from the Apache Incubator in March 2013 it joined Hadoop as a Top-Level Project (TLP) within the Apache Software Foundation (ASF). This made the ASF the only Open Source Foundation which contains a cloud platform and a big data solution. Moreover a closer look at the projects making the entire ASF shows that approximately 30% of the Apache Incubator and 10% of the TLPs is "Big Data" related. Projects such as Hbase, Hive, Pig and Mahout are sub-projects of the Hadoop TLP. Ambari, Kafka, Falcon and Mesos are part of the incubator and all based on Hadoop.

To Complement CloudStack, API wrappers such as Libcloud, deltacloud and jclouds are also part of the ASF. To connect CloudStack and Hadoop two interesting projects are also in the ASF: Apache Whirr a TLP, and Provisionr currently in incubation. Both Whirr and Provisionr aimed at providing an abstraction layer to define big data infrastructure based on Hadoop and instantiate those infrastructure on Clouds, including Apache CloudStack based clouds. This co-existence of CloudStack and the entire Hadoop ecosystem under the same Open Source Foundation means that the same governance, processes and development principles apply to both project bringing great synergy that promises an even better complementarity.

In this tutorial we introduce Apache Whirr, an application that can be used to define, provision and configure big data solutions on CloudStack based clouds. Whirr automatically starts instances in the cloud and boostrapps hadoop on them. It can also add packages such as Hive, Hbase and Yarn for map-reduce jobs.

Whirr [1] is a "set of libraries for running cloud services" and specifically big data services. Whirr is based on jclouds [2]. Jclouds is a java based abstraction layer that provides a common interface to a large set of Cloud Services and providers such as Amazon EC2, Rackspace servers and CloudStack. As such all Cloud providers supported in Jclouds are supported in Whirr. The core contributors of Whirr include four developers from Cloudera the well-known Hadoop distribution. Whirr can also be used as a command line tool, making it straightforward for users to define and provision Hadoop clusters in the Cloud.

As an Apache project, Whirr comes as a source tarball and can be downloaded from one of the Apache mirrors [3]. Similarly to CloudStack, Whirr community members can host packages. Cloudera is hosting whirr packages to ease the installation. For instance on Ubuntu and Debian based systems you can add the Cloudera repository by creating /etc/apt/sources.list.d/cloudera.list and putting the following contents in it:

deb [arch=amd64] http://archive.cloudera.com/cdh4/-cdh4 contrib 
deb-src http://archive.cloudera.com/cdh4/-cdh4 contrib

With this repository in place, one can install whirr with:

$sudo apt-get install whirr

The whirr command will now be available. Developers can use the latest version of Whirr by cloning the software repository, writing new code and submitting patches the same way that they would submit patches to CloudStack. To clone the git repository of Whirr do:

$git clone git://git.apache.org/whirr.git

They can then build their own version of whirr using maven:

$mvn install

The whirr binary will be located under the /bin directory. Adding it to one's path with:

$export PATH=$PATH:/path/to/whirr/bin

Will make the whirr command available in the user's environment. Successfull installation can be checked by simply entering:

$whirr --help

With whirr installed, one now needs to specify the credentials of the Cloud that will be used to create the Hadoop infrastructure. A ~/.whirr/credentials has been created during the installation phase. The type of provider (e.g cloudstack), the endpoint of the cloud and the access and secret keys need to be entered in this credentials file like so:

PROVIDER=cloudstack
IDENTITY=
CREDENTIAL=
ENDPOINT=

For instance on Exoscale [4] a CloudStack based cloud in Switzerland, the endpoint would be https://api.exoscale.ch/compute

Now that the CloudStack cloud endpoint and keys have been configured, the hadoop cluster that we want to instantiate needs to be defined. This is done in a properties file using a set of Whirr specific configuration variables [5]. Below is the content of the file with explanations in-line:

---------------------------------------
# Set the name of your hadoop cluster
whirr.cluster-name=hadoop

# Change the name of cluster admin user
whirr.cluster-user=${sys:user.name}

# Change the number of machines in the cluster here
# Below we define one hadoop namenode and 3 hadoop datanode
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,3 hadoop-datanode+hadoop-tasktracker

# Specify which distribution of hadoop you want to use
# Here we choose to use the Cloudera distribution
whirr.env.repo=cdh4
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop

# Use a specific instance type.
# Specify the uuid of the CloudStack service offering to use for the instances of your hadoop cluster
whirr.hardware-id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8

# If you use ssh key pairs to access instances in the cloud
# Specify them like so
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_exoscale
whirr.public-key-file=${whirr.private-key-file}.pub

# Specify the template to use for the instances
# This is the uuid of the CloudStack template
whirr.image-id=1d16c78d-268f-47d0-be0c-b80d31e765d2
------------------------------------------------------

To launch this Hadoop cluster use the whirr command line:

$whirr launch-cluster --config hadoop.properties

The following example output shows the instances being started and boostrapped. At the end of the provisioning, whirr returns the ssh command that shall be used to access the hadoop instances.

-------------------
Running on provider cloudstack using identity mnH5EbKcKeJd456456345634563456345654634563456345
Bootstrapping cluster
Configuring template for bootstrap-hadoop-datanode_hadoop-tasktracker
Configuring template for bootstrap-hadoop-namenode_hadoop-jobtracker
Starting 3 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(b9457a87-5890-4b6f-9cf3-1ebd1581f725)
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(9d5c46f8-003d-4368-aabf-9402af7f8321)
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(6727950e-ea43-488d-8d5a-6f3ef3018b0f)
>> running InitScript{INSTANCE_NAME=bootstrap-hadoop-namenode_hadoop-jobtracker} on node(6a643851-2034-4e82-b735-2de3f125c437)
<< success executing InitScript{INSTANCE_NAME=bootstrap-hadoop-datanode_hadoop-tasktracker} on node(b9457a87-5890-4b6f-9cf3-1ebd1581f725): {output=This function does nothing. It just needs to exist so Statements.call("retry_helpers") doesn't call something which doesn't exist
Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
Get:2 http://security.ubuntu.com precise-security Release [49.6 kB]
Hit http://ch.archive.ubuntu.com precise Release.gpg
Get:3 http://ch.archive.ubuntu.com precise-updates Release.gpg [198 B]
Get:4 http://ch.archive.ubuntu.com precise-backports Release.gpg [198 B]
Hit http://ch.archive.ubuntu.com precise Release
..../snip/.....
You can log into instances using the following ssh commands:
[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.xx.yy.zz
[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.zz.zz.rr
[hadoop-datanode+hadoop-tasktracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.tt.yy.uu
[hadoop-namenode+hadoop-jobtracker]: ssh -i /Users/sebastiengoasguen/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no sebastiengoasguen@185.ii.oo.pp
-----------

To destroy the cluster from your client do:

$whirr destroy-cluster --config hadoop.properties.

Whirr gives you the ssh command to connect to the instances of your hadoop cluster, login to the namenode and browse the hadoop file system that was created:

$ hadoop fs -ls /
Found 5 items
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:11 /hadoop
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:10 /hbase
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:10 /mnt
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:11 /tmp
drwxrwxrwx   - hdfs supergroup          0 2013-06-21 20:11 /user

Create a directory to put your input data.

$ hadoop fs -mkdir input
$ hadoop fs -ls /user/sebastiengoasguen
Found 1 items
drwxr-xr-x   - sebastiengoasguen supergroup          0 2013-06-21 20:15 /user/sebastiengoasguen/input

Create a test input file and put in the hadoop file system:

$ cat foobar 
this is a test to count the words
$ hadoop fs -put ./foobar input
$ hadoop fs -ls /user/sebastiengoasguen/input
Found 1 items
-rw-r--r--   3 sebastiengoasguen supergroup         34 2013-06-21 20:17 /user/sebastiengoasguen/input/foobar

Define the map-reduce environment. Note that this default Cloudera distribution installation uses MRv1. To use Yarn one would have to edit the hadoop.properties file.

$ export HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce

Start the map-reduce job:

$ hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar wordcount input output
13/06/21 20:19:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/21 20:20:00 INFO input.FileInputFormat: Total input paths to process : 1
13/06/21 20:20:00 INFO mapred.JobClient: Running job: job_201306212011_0001
13/06/21 20:20:01 INFO mapred.JobClient:  map 0% reduce 0%
13/06/21 20:20:11 INFO mapred.JobClient:  map 100% reduce 0%
13/06/21 20:20:17 INFO mapred.JobClient:  map 100% reduce 33%
13/06/21 20:20:18 INFO mapred.JobClient:  map 100% reduce 100%
13/06/21 20:20:21 INFO mapred.JobClient: Job complete: job_201306212011_0001
13/06/21 20:20:22 INFO mapred.JobClient: Counters: 32
13/06/21 20:20:22 INFO mapred.JobClient:   File System Counters
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of bytes read=133
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of bytes written=766347
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of read operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     FILE: Number of write operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of bytes read=157
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of bytes written=50
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of read operations=2
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/06/21 20:20:22 INFO mapred.JobClient:     HDFS: Number of write operations=3
13/06/21 20:20:22 INFO mapred.JobClient:   Job Counters 
13/06/21 20:20:22 INFO mapred.JobClient:     Launched map tasks=1
13/06/21 20:20:22 INFO mapred.JobClient:     Launched reduce tasks=3
13/06/21 20:20:22 INFO mapred.JobClient:     Data-local map tasks=1
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=10956
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=15446
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/21 20:20:22 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/21 20:20:22 INFO mapred.JobClient:   Map-Reduce Framework
13/06/21 20:20:22 INFO mapred.JobClient:     Map input records=1
13/06/21 20:20:22 INFO mapred.JobClient:     Map output records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Map output bytes=66
13/06/21 20:20:22 INFO mapred.JobClient:     Input split bytes=123
13/06/21 20:20:22 INFO mapred.JobClient:     Combine input records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Combine output records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce input groups=8
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce shuffle bytes=109
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce input records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Reduce output records=8
13/06/21 20:20:22 INFO mapred.JobClient:     Spilled Records=16
13/06/21 20:20:22 INFO mapred.JobClient:     CPU time spent (ms)=1880
13/06/21 20:20:22 INFO mapred.JobClient:     Physical memory (bytes) snapshot=469413888
13/06/21 20:20:22 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=5744541696
13/06/21 20:20:22 INFO mapred.JobClient:     Total committed heap usage (bytes)=207687680

And you can finally check the output:

$ hadoop fs -cat output/part-* | head
this 1
to  1
the  1
a  1
count 1
is  1
test 1
words 1

Of course this is a silly example of map-reduce job and you will want to do much more critical tasks. In order to benchmark your cluster Hadoop comes with examples jar.

To benchmark your hadoop cluster you can use the TeraSort tools available in the hadoop distribution. Generate some 100 MB of input data with TeraGen (100 byte rows):

$hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar teragen 1000000 output3

Sort it with TeraSort:

$ hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar terasort output3 output4

And then validate the results with TeraValidate:

$hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples.jar teravalidate output4 outvalidate

Performance of map-reduce jobs run in Cloud based hadoop clusters will be highly dependent on the hadoop configuration, the template and the service offering being used and of course on the underlying hardware of the Cloud. Hadoop was not designed to run in the Cloud and therefore some assumptions were made that do not fit the Cloud model, see [6] for more information. Deploying Hadoop in the Cloud however is a viable solution for on-demand map-reduce applications. Development work is currently under way within the Google Summer of Code program to provide CloudStack with a compatible Amazon Elastic Map-Reduce (EMR) service. This service will be based on Whirr or a new Amazon CloudFormation compatible interface called StackMate [7].

[1] http://whirr.apache.org
[2] http://jclouds.incubator.apache.org
[3] http://www.apache.org/dyn/closer.cgi/whirr/
[4] http://exoscale.ch
[5] http://whirr.apache.org/docs/0.8.2/configuration-guide.html
[6] http://wiki.apache.org/hadoop/Virtual%20Hadoop
[7] https://github.com/chiradeep/stackmate

CloudStack support in Apache Libcloud

A vote has started on the libcloud dev list for the 0.13 release. The release notes detail all the new features and fixes. I am stoked about this because a few patches that I submitted are included in this release candidate. I patched the CloudStack driver quite a bit to improve the support for Clouds with a basic zone like Exoscale. There is more work to do on this driver including better support for Advanced zone especially for port forwarding, firewall rules and more unit tests. A the CloudStack hackathon last sunday @pst418 submitted a few patches for unit tests and they made it as well into 0.13 RC, terrific.

If you don't know libcloud, it's a python based API wrapper to abstract the various cloud APIs. With libcloud you can create connections to multiple clouds, potentially using different API. At a high level, it is similar to jclouds for JAVA or deltacloud written in ruby. There was already a CloudStack driver but its functionality was limited. If you grab my quickie libcloud shell, you can try to follow this walkthrough of what you can do with libcloud and a CloudStack basic zone. Of course you will need a CloudStack endpoint.

Start the libshell and check what zone you are on:

$ python ./libshell.py 
Hello LibCloud Shell !!
You are running at: https://api.exoscale.ch/compute
>>> conn.list_locations()
[<NodeLocation: id=1128bd56-b4d9-4ac6-a7b9-c715b187ce11, name=CH-GV2, country=AU, driver=CloudStack>]

You might notice a wrong country code, it is hard-coded in libcloud, I need to file a bug for this. Get the list of templates (or images in libcloud speak):

>>> conn.list_images()
[<NodeImage: id=01df77c3-0150-412a-a580-413a50924a18, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=89ee852c-a5f5-4ab9-a311-89f39d133e88, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=ccd142ec-83e3-4108-b1c5-8e1fdb353ff9, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=f2101a0c-eaf7-4760-a143-0a5c940fd864, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=77d32782-6866-43d4-9524-6fe346594d09, name=CentOS 5.5(64-bit) no GUI (KVM), driver=CloudStack  ...>,
 <NodeImage: id=29cba09f-4569-4bb3-95e7-71f833876e3e, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=ee241b47-4303-40c8-af58-42ed6bf09f8c, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=754ea486-d649-49e5-a70e-5e5d458f0df0, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=0f9f4f49-afc2-4139-b26b-b05a9f51ea74, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=954752a8-0486-46bb-8e3f-0adb3e01c619, name=Linux CentOS 6.4 64-bit, driver=CloudStack  ...<]

I cut the previous output, but there are also Ubuntu and CentOS images on that cloud...You can then list the various instance types or sizes in libcloud speak.

>>> conn.list_sizes()
[<NodeSize: id=71004023-bb72-4a97-b1e9-bc66dfce9470, name=Micro, ram=512 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8, name=Tiny, ram=1024 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=21624abb-764e-4def-81d7-9fc54b5957fb, name=Small, ram=2048 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=b6e9d1e8-89fc-4db3-aaa4-9b4c5b1d0844, name=Medium, ram=4096 disk=0 bandwidth=0 price=0 driver=CloudStack ...>, 
<NodeSize: id=c6f99499-7f59-4138-9427-a09db13af2bc, name=Large, ram=8182 disk=0 bandwidth=0 price=0 driver=CloudStack ...>, 
<NodeSize: id=350dc5ea-fe6d-42ba-b6c0-efb8b75617ad, name=Extra-large, ram=16384 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=a216b0d1-370f-4e21-a0eb-3dfc6302b564, name=Huge, ram=32184 disk=0 bandwidth=0 price=0 driver=CloudStack ...>]

What I added is the management of ssh key pairs and security groups, you can now list, create and delete both keypairs and security groups, and use those when deploying nodes. (Don't bother trying to do anything with the keys below, I deleted everything.)

>>> conn.ex_list_keypairs() 
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'Default 
Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': u'0.0.0.0/0', u'protocol':
 u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'0.0.0.0/0', u'protocol':
 u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': 
u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test',
 u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

>>> conn.ex_create_keypair('test')
{u'privateKey': u'-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQCAuPufJnbzFkyIUaMymkXwfilA5qS2J9fx2Q3EWNDs1m89FLFq\n19ERjG43ZvRuI/KGwWZwHbhqoqJth7WQYNe
lYCmOUYRaepxTrpU4TGDGuhqMh9D9\
noNMIFx3ktkcTitxkSY/5h/pXqSB2XXURLpZWwZxjEYwWCpWE8i7uVtIR7wIDAQAB\nAoGAOv0Kik9lOVbhsaK/yAO8w039d7l6h+NQaYtPbMhKzg4iofomp9DJBWK2a3sp\
nzoN4s9pTKFPmXC+1gb4sLULD72ENSgyozrPMCJ0tytNa3ebVCCYDvlagEZ83KwGn\nnr2BIRLul1xyHVa+amvTemuxi7OOx0u/0aZ6jPVW9ocPahkCQQDnUACG3Q2p60Mx\
nCttW7Go2wz0BhaI8ibG8rHorhdrFJUrsTI6QrLh340uHCIEAuWUXjYDX2u5MTPBG\njWbL/A/9AkEAjnX8EzO/B/RooFTxhYdxv7D1Dzcx4H59mFzjez6N/mjAHjKL4p66\
nqGFgLa9HMhDPFOs83VetBXcihu1vueffWwJBALQUD2TvAS0wz82FYz8nrITXuE3Q\nCH7Sv8FgEXiCq89hehO+ghrVrIMBPBJzJ2M18iLE8fKaKXzTRRfYC5hwss0CQHY8\
nBdIKCGoZtxwaY7lnCEkIHNtb+9FOKf7iWQpYiJC1b32ghei3xEMrTh+ccYJj4PqD\noigyNC9tCQLi3O92OjECQGYR2z8IDlfnl7G8p7eiyBciuoDyyq5/oJHhkHbkbwzr\
nW0Uun+rEcjRnXbUN8wUQ6FSFrxk2VSajbCBteTOrF24=\n-----END RSA PRIVATE KEY-----\n', u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}

>>> conn.ex_list_keypairs()
[{u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}, {u'name': u'foobar',
 u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_delete_keypair('test')
u'true'

>>> conn.ex_list_keypairs()
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_create_security_group('heyhey')
{u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id':
 u'77ad73b5-a383-4e13-94be-a38ef9877996', u'domain': u'runseb@gmail.com', u'name': u'heyhey'}

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': 
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'
0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
 u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description':
 u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}, 
{u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id': u'77ad73b5-a383-4e13-94be-a38ef9877996',
 u'domain': u'foobar@gmail.com', u'name': u'heyhey'}]

>>> conn.ex_delete_security_group('heyhey')
u'true'

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': 
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
 u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

With a security group created you can now add an ingress rule:

>>> conn.ex_authorize_security_group_ingress(securitygroupname='toto',protocol='TCP',cidrlist='0.0.0.0./0',
startport=99)
{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', 
u'description': u'this is a test', u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 99, u'cidr': u'0.0.0.0./0', 
u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}], 
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', 
u'description': u'Default Security Group', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id':
 u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': u'default'}, {u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': 
u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': 
[{u'startport': 99, u'cidr': u'0.0.0.0./0', u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}], 
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

Great, keypair and security group work fine. Now let's do the basics of starting an instance and let's test the suspend and resume which was not in the Driver until now. Nothing earth shattering but it's an improvement.

>>> size=conn.list_sizes()
>>> image=conn.list_images()

>>> n=conn.create_node(name='yoyo',size=size[0],image=image[0])

>>> n.ex_stop()
u'Stopped'
>>> n.ex_start()
u'Running'

And that's it for now, CloudStack support is getting better, there is still tons of work to do: improve the advanced zone support, check the load-balancer and storage support, add unit-tests and new CloudStack features like auto-scaling. Carry on !!!!

Build a Cloud Day Amsterdam recap

On June 13th we had a Build a Cloud Day workshop in Schuberg Phillis offices in Amsterdam. We had a good crowd with a little over 30 people gathering to learn about CloudStack, its use cases, monitoring, PaaS, SDN and client tools. We had good fun and it was a very productive meeting.

We started off with a talk from Arjan Eriks from Schuberg Phillis, Arjan gave us a terrific overview of the Schuberg's culture and how the Cloud and DevOps processes naturally appeared within Schuberg. Their team oriented approach to customer support evolved to using CloudStack as their core IT solution to provide an agile infrastructure that meets the need of their customers. Adding to the Cloud infrastructure, a DevOps culture permeated Schuberg even before DevOps really existed. His slides below will give you a good idea of how a company moves to the Cloud and uses agile methods to make customers and employees "happy".

Cloud and Devops at Schuberg Philis from Arjan Eriks

Following Arjan we had the chance to hear from Julien Pivoto (@roidelapluie) of Inuits an IT consulting company that helps its customers adopt open source solutions and embrace a DevOps culture. Julien presented the suite of open source monitoring tools that can be used to monitor a Cloud infrastructure at scale. From logstash, elasticsearch and Kibana, to collectd, graphite and riemann. Julien did a great job introducing all these tools and explaining how they stitch together to help automation of a Cloud infrastructure.

Monitoring at Cloud Scale from Julien Pivotto

Hugo Trippaers (@Spark404) then gave us a very practical talk about SDN (Software Defined Networking). Hugo has developed the Nicira NVP integraton in Apache CloudStack and manages the Cloud at Schuberg Phillis. He reviewed all the SDN support in CloudStack (e.g Nicira, native OVS manager, Big Switch, Midonet) and then dove into a Nicira use case. This really brought home the fact that SDN is not a fiction but a reality.

SDN in CloudStack from buildacloud

After these first talks about a company's view of Cloud and DevOps, a review of key monitoring tools and SDN solutions in CloudStack, Giles Sirett from Shapeblue presented us with several CloudStack use cases. Shapeblue is a CloudStack integrator in the UK with offices in India, US and Brazil. Giles insider's look on how company choose a Cloud solution and how CloudStack empowers them is unique. This was a very energizing talk showing use that CloudStack is gaining momentum and being adopted all over the world.

Shapeblue BACD Amsterdam from buildacloud

CloudStack is a IaaS solution, but once you have built your IaaS you need to focus on application delivery. Arash Kaffamanesh from Cloudssky talked to us about PaaS and showed a few demos on Appscale and Stackato integration with CloudStack. PaaS integration is set to become a hot topic within the next year and Arash lightning talk certainly set the stage. I know of folks working on better CloudStack support in Cloudfoundry/BOSH as well, so expect to see PaaS/CloudStack integration soon.

PaaS on top of CloudStack from buildacloud

After these relatively strategic talk we went back to developer tools with Sander Botman. Sander is currently taking the lead in maintaing the Knife CloudStack plugin. Knife is the command line utility from Opscode, it is a great tool to manage your Chef repository. Written in ruby it is highly customizable and has numerous plugins. Sander demoed the knife-cloudstack plugin which let's you provision instances on your CloudStack based cloud and automatically boostrap these instances using chef recipes. I have used knife-cs recently and even submitted a few patches. It is a terrific tool, easy to use with lots of options. If you are looking for a tool to easily provision, configure and manage instances in your cloud, this is it.

Knife CloudStack from buildacloud

Finally, I wrapped up the talks with demos about several CloudStack clients and tools. I have been testing lots of tools lately to prepare content for CloudStack University. I focused on Apache tools form the ASF such as libcloud and jclouds which just entered the incubator. I also looked at higher level wrappers like Apache whirr to create hadoop-clusters on demand. The slides below are still a work in progress and I will put up screencasts up really soon, Stay tuned.

CloudStack Clients and Tools from Sebastien Goasguen

To end the day, we had a few beers, before heading out to the DevOps days pre-conference party. A few folks sneaked out before we took the picture but we all left energized, having learned a ton. We all look forward to the CloudStack conference November 20-22nd in Amsterdam. Should be a blast.

Exoscale, a Swiss Cloud Provider with CloudStack

Over the last couple weeks I have been using exoscale, a swiss public cloud provider based on CloudStack. They just launched after a beta testing phase that I had the chance to be part of. Their offering is primarily aimed at developers. The folks at Exoscale were kind enough to give me couple "gift cards" during the CloudStack Geneva meetup and I was able to get going. Together with PCExtreme, Leaseweb, and iKoula they are one more European public cloud provider in production with Apache CloudStack that I know of.

Their cloud is almost straightforward: two data centers in Geneva and Vernier, with hardware hosted by Equinix. They run Apache CloudStack 4.0.2, the latest release and use KVM hypervisors on Ubuntu based servers. One customization that they made and that I am aware of is that they patched CloudStack to output logs using logstash and use Kibana for visuzalization. They offer CentOS 6.4 and Ubuntu 12.04/13.04 64 bit templates with instance types from 512MB with 1 core to 32 GB with 8 cores. Their development and operations team is relatively small for such an offering but they are backed by the Veltigroup a leading IT provider in Switzerland, which gives them a 20 person team for support. Their developers are seasoned IT infrastructure enthusiasts who participate in the DevOps, openBSD, Clojure and Pallet community. The lead developer, Pierre-Yves Ritschard, formerly with paper.li, recently participated in DevOps Days Paris and has contributed a Clojure client to CloudStack: clostack. They are embracing open source, not only by using it, but also by contributing to the various communities that make up the foundation of Cloud services.

While CloudStack comes with a powerful and efficient Web UI, exoscale decided to create their own UI and integrate it with a ticketing, monitoring and billing system that they developed. It reinforces the fact the CloudStack API is extremely rich and that the default UI was actually designed as a proof of concept rather than something that all users should use. The UI will please developers by its simplicity and straigthforward ease of use. From talking to them, I know they they will soon open source the python client they developed to build the UI backend. Pierre Yves told me it resembled a little bit my toy UI that uses Flask and builds a REST wrapper on top of the CloudStack API. See a snapshot of an instance view below:

They envision an entirely public cloud, meaning that all their instances are on the public network (nb: They also have a premium service for the enterprise with secure VPN access). They use a CloudStack basic zone with security groups ala EC2. Users can get VNC access and of course ssh access to the instance using ssh keys created via the console. Since it is still in beta-testing they have currently turned off some of the features like snapshotting. These feature limitations are well documented and in their roadmap for future releases of their Cloud services. In the meantime they are also working on high level services like Jenkins continuous infrastructure services and most probably big data services. The Big Data service is just a guess on my part but would make sense considering Pierre-Yes involvement with Pallet and a recent demo he did at Eurocloud.

I have used exoscale a lot in those last two weeks and I am extremely pleased with it. It is actually becoming my favorite testbed. The UI is great as I mentioned, but all the CloudStack clients also work (of course :) ), CloudMonkey is easily setup to talk to their endpoint, I submitted patches to libcloud last week and you can now use with exoscale. I even contributed a CloudStack driver for SaltStack (an alternative to Chef and Puppet) based on testing with exoscale. Right away I felt comfortable with exoscale and even gave a lecture at HEPIA where I put 7 students on it in 20 minutes. If you want to learn more, Exoscale will be at Build a Cloud Day Paris on June 19th.

So here is the catchphrase of the blog: Exoscale, it's simple and it works