Monday, May 27, 2013

Exoscale, a Swiss Cloud Provider with CloudStack

Over the last couple weeks I have been using exoscale, a swiss public cloud provider based on CloudStack. They just launched after a beta testing phase that I had the chance to be part of. Their offering is primarily aimed at developers. The folks at Exoscale were kind enough to give me couple "gift cards" during the CloudStack Geneva meetup and I was able to get going. Together with PCExtreme, Leaseweb, and iKoula they are one more European public cloud provider in production with Apache CloudStack that I know of.


Their cloud is almost straightforward: two data centers in Geneva and Vernier, with hardware hosted by Equinix. They run Apache CloudStack 4.0.2, the latest release and use KVM hypervisors on Ubuntu based servers. One customization that they made and that I am aware of is that they patched CloudStack to output logs using logstash and use Kibana for visuzalization. They offer CentOS 6.4 and Ubuntu 12.04/13.04 64 bit templates with instance types from 512MB with 1 core to 32 GB with 8 cores. Their development and operations team is relatively small for such an offering but they are backed by the Veltigroup a leading IT provider in Switzerland, which gives them a 20 person team for support. Their developers are seasoned IT infrastructure enthusiasts who participate in the DevOps, openBSD, Clojure and Pallet community. The lead developer, Pierre-Yves Ritschard, formerly with paper.li, recently participated in DevOps Days Paris and has contributed a Clojure client to CloudStack: clostack. They are embracing open source, not only by using it, but also by contributing to the various communities that make up the foundation of Cloud services.

While CloudStack comes with a powerful and efficient Web UI, exoscale decided to create their own UI and integrate it with a ticketing, monitoring and billing system that they developed. It reinforces the fact the CloudStack API is extremely rich and that the default UI was actually designed as a proof of concept rather than something that all users should use. The UI will please developers by its simplicity and straigthforward ease of use. From talking to them, I know they they will soon open source the python client they developed to build the UI backend. Pierre Yves told me it resembled a little bit my toy UI that uses Flask and builds a REST wrapper on top of the CloudStack API. See a snapshot of an instance view below:

They envision an entirely public cloud, meaning that all their instances are on the public network (nb: They also have a premium service for the enterprise with secure VPN access). They use a CloudStack basic zone with security groups ala EC2. Users can get VNC access and of course ssh access to the instance using ssh keys created via the console. Since it is still in beta-testing they have currently turned off some of the features like snapshotting. These feature limitations are well documented and in their roadmap for future releases of their Cloud services. In the meantime they are also working on high level services like Jenkins continuous infrastructure services and most probably big data services. The Big Data service is just a guess on my part but would make sense considering Pierre-Yes involvement with Pallet and a recent demo he did at Eurocloud.

I have used exoscale a lot in those last two weeks and I am extremely pleased with it. It is actually becoming my favorite testbed. The UI is great as I mentioned, but all the CloudStack clients also work (of course :) ), CloudMonkey is easily setup to talk to their endpoint, I submitted patches to libcloud last week and you can now use with exoscale. I even contributed a CloudStack driver for SaltStack (an alternative to Chef and Puppet) based on testing with exoscale. Right away I felt comfortable with exoscale and even gave a lecture at HEPIA where I put 7 students on it in 20 minutes. If you want to learn more, Exoscale will be at Build a Cloud Day Paris on June 19th.

So here is the catchphrase of the blog: Exoscale, it's simple and it works

Thursday, May 23, 2013

The LinuxTag Hack

What do you do when you go to LinuxTag the premier Open Source conference in Berlin Germany ? You give a talk, you hand out tee-shirts at the CloudStack booth, you explain Cloud computing and you hack a CloudStack driver for SaltStack while patching libcloud.

The talk: Talking about Clouds is nice and all, but after many years and many talks, I shamelessly admit that it gets a little old. So lately I have been working on BigData, both as a backend to CloudStack (think Ceph, Riack CS, Gluster) and as a workload to a cloud. I am talking about using Apache Whirr or Apache Provisionr incubating to deploy "one-click" hadoop clusters on public clouds. It is a long story that I will keep for another post as I am trying to write this before going to bed. but check out the slides and keep an eye on pallet and exoscale.

The booth: An open source booth is ..well a booth. I came with my pop-up banner, table cloth, tee-shirts, post-cards, USB stick/bottle openers, it feels a little bit like a traveling sales man, not that I would know but I imagine it like this. I have to explain that the 2 and 3 XL shirts will shrink quite a bit and will fit perfectly people's M or L frames. Then I point at the banner to showcase the magnificient CloudStack UI, I explain that there is an API server behind it. Sometimes I launch devcloud and do a live demo to bring them to their knees, sometimes I have to ask for help on IRC to answer a question, and sometimes a german developer wants to trade a tee-shirt against illegal substances not to be named. Life in the fastlane let me tell you. But that's what it's like to build a community, it is very much an evangelization process.

The hack: Then there is the hack, going to an open source conference without writing code would be a sin. The OSS gods decided to put me next to Tom Hatch the CTO of Saltstack. Tom is a funny guy with a deep voice and madenning Python skills. Saltstack is an alternative to Chef and Puppet which are written in this foreign language Ruby. It does configuration management, remote execution, cloud deployments and tons of other things. I was also happily surprised to find 300 folks on IRC, very chatty folks I might add. Anyway, Saltstack was not going to go anywhere because they did not have a CloudStack driver. EC2 yes, Rackspace yes, Joyent yes, OpenStack yes...but no CloudStack. I had to do something about it. A quick git clone and a RTFM later I was on my way, thanks to my new best friend exoscale, a Swiss public cloud provider (based on CloudStack off course) who gave me 50 Francs worth of Cloud Brownies I could test everything live.

I did not want to run any silly master on my laptop, behind crazy NAT non sense, so I used Salt-cloud, I copied the linode driver and started hacking it. It uses libcloud which thankfully I had looked at just couple weeks ago while writing a bit of CloudStack doc. There was a few issues with the libcloud driver so I opened couple bugs there and committed to patches to fix my own bugs, isn't it nice ? Grant it there are not big bugs, but bugs they are, 329 and 330 to be exact.

I got it working and finished it tonight which explains my excitement. I forked their repo on git and made two pull requests that got merged right away. Let's get down to it, shall we. You need two configuration file: cloud and profile

Your cloud conf defines your Cloud Provider and which driver it uses, that's where I define exoscale, my keys etc...

providers:
  exoscale:
    apikey:  
    secretkey: 
    host: api.exoscale.ch
    path: /compute
    securitygroup: default
    user: root
    private_key: 
    provider: cloudstack

Your profile conf defines the instance type that you are going to use, it's a combination of the image, the service offerings and the keys used to access the instance that it will create...

ubuntu-exoscale:
    provider: exoscale
    image: 1d16c78d-268f-47d0-be0c-b80d31e765d2 
    size: b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8 
    ssh_interface: public
    ssh_username: root
    keypair: exoscale

With this you can know list-location, list-sizes, list-images:

salt-cloud --list-locations exoscale
Password:
[INFO    ] Configuration file path: /Users/sebastiengoasguen/.saltcloud/cloud
[INFO    ] salt-cloud starting
cloudstack
  CH-GV2
    country: AU
    id: 1128bd56-b4d9-4ac6-a7b9-c715b187ce11

salt-cloud --list-sizes exoscale
Password:
[INFO    ] Configuration file path: /Users/sebastiengoasguen/.saltcloud/cloud
[INFO    ] salt-cloud starting
cloudstack
  Extra-large
    bandwidth: 0
    disk: 0
    id: 350dc5ea-fe6d-42ba-b6c0-efb8b75617ad
    price: 0
    ram: 16384
    uuid: edb4cd4ae14bbf152d451b30c4b417ab095a5bfe
...snip...

salt-cloud --list-images exoscale
[INFO    ] Configuration file path: /Users/sebastiengoasguen/.saltcloud/cloud
[INFO    ] salt-cloud starting
cloudstack
  CentOS 5.5(64-bit) no GUI (KVM)
    extra:
      format: QCOW2
      hypervisor: KVM
      os: CentOS 5.5 (64-bit)
    id: 77d32782-6866-43d4-9524-6fe346594d09
    uuid: 2fb9ae4b32b4ea5eafd3341166cf948cfe24aa7f
...snip...
And finally of course:
salt-cloud -p ubuntu-exoscale lll
This will start the instance on the cloud, do key based ssh access to it and start bootstrapping saltstack. Tomorrow I will check out their actual configuration management scheme, launch couple minions, hoping to create a monogd cluster or even a hadoop cluster. Before you ask, lll is the name you give to an instance when you get tired.

The caveat to all of this is that my patches to libcloud need to be accepted before the CloudStack SaltStack driver is usable. Enjoy ! Signing off. Busy day: Talk, booth, hack

CloudStack European Events this Summer

The summer is going to be full with CloudStack events and talks in Europe.

Choose your event, let's meet, brainstorm and build a great community

Friday, May 17, 2013

CloudStack University

At Apache CloudStack we recently started an initiative to organize our content into learning modules. We call this initiative CloudStack University. Everyone is invited to participate by contributing content (slides and screencasts), suggesting new learning modules that are needed and even creating exercises and assignments. School fun ! As we were discussing the initiative on the mailing list we started by looking at our existing content: slideshares, youtube videos and thought about organizing them into a CloudStack 101 course. This is still a work in progress that requires everyones participation to make it a great resource.

In the meantime I have been putting all my CloudStack content on slideshare and I wanted to provide a narrated version of these slides together with hands-on demo to show folks how to do a few things with CloudStack specifically but also related Cloud and OSS tips and tricks. Here comes the CloudStack university screencasts. I will add more of them as I go along and receive requests from the community (reach out on twitter @sebgoa and tell me what you want to see). I wanted to give you a preview of what this looks like. To create a self-paced learning module, I decided to create slide decks that people can download from slideshare and cross-post the corresponding screencasts (for most of them at least) on youtube. People can choose a particular topic, or take the entire series. The idea is that at the end of watching all the screencasts and reading the material people graduate from CloudStack University.

Certainly one can imagine how this could evolve into a full fledge training and certification program. I do plan to create a final exam once I am done with a consistent set of modules :) In this post I wanted to introduce you to some of the first modules I created. I welcome all feedback and suggestions to improve them. Reach out to me on twitter (@sebgoa) or contribute your own modules via the wiki and the mailing lists.

To get started, I show you below the screencast of testing the Apache CloudStack (ACS) 4.0.2 testing procedure. We used this basic procedure as a smoke test for the release and as way to vote on a release. There is far more QA going on for a release, this is just a basic testing to vote on the release. This is definitely geared towards developers, I plan to create a more end-user version of an introduction to CloudStack.

Once you have been introduced to ACS with this testing procedure you can learn the API. CloudStack has a native API as well as an EC2 compatible interface. The following screencasts and slides dives into the ACS API, showing how to do unauthenticated and authenticated calls, create a signature, it finishes with a discussion on REST and nice exercise.

Learning the API and the details of how to create a call by hand is a very nice skill to have. CloudStack has 16 clients on github in various languages, these clients make it extremely easy to use the CloudStack API. However ACS comes with a terrific interactive shell: CloudMonkey. The following module shows you how to install CloudMonkey and how to use it to manage your ACS backed cloud. If you followed the testing procedure, you can use CloudMonkey with your local CloudStack and explore the API.

If you want to participate in the development of CloudStack you can contribute in many different ways, but definitely modifying the source to include new features, fix bugs and add documentation are some of the basic contributions. The following module gives you an introduction to Git the version control system used by ACS. This is aimed at beginners and starts with a demo of gist on github, it then shows a walkthrough of the CloudStack git repo, looks at remote feature branches and finishes with the workflow to create a patch and submit for review.

At ACS we put great care in making sure that everyone in the world can use our software and that means use the UI in their own language and read the documentation in their own language. Translation is a tedious work but very much appreciated by non english speaking users and developers. In this module we show you how to contribute to the translation of CloudStack UI and docs using the Transifex service. We are proud to have over 30 translators who allow us to support the CloudStack UI in 10 languages and have documentation almost complete in 5 languages.

As a final note, Apache Software Foundation is a mentoring organization in the 2013 Google Summer of code. As such CloudStack is participating in GSoC, we are currently reviewing proposals from students and are eager to see the program started. I embed a deck that introduced the various projects we proposed. Stay tuned to learn more about which ones got awarded, notification on May 27th. And remember to keep an eye out for CloudStack University, a great resource for Cloud training.

Monday, May 06, 2013

Update on Apache CloudStack Community Analysis

Since my last post on the analysis of the CloudStack community, we have graduated and became a top-level project. It's about time to give an update on what can be seen as a metric of the health of our community.

All the data presented is based on the analysis of the mailing lists, the data is publicly accessible, I have used it previously, just when we graduated March 22nd, in January and back in November when I did some social network analysis. This study was inspired by John Jiang, now working at Eucalyptus, you can read his analysis, note that he moved it to the Eucalyptus website.

Methodology: As explained in previous posts, a Contributor is considered as someone who sent an email to one of the CloudStack mailing lists. This is not to be confused with a Committer which at the ASF is meant to represent someone with write access to the code. Not all code contributors have write access. I identify Companies as the email domain used by the Contributors. This is because Contributors are none-affiliated in the ASF. Obviously it has some limitations as email domains such as gmail.com can represent different companies. All emails are loaded in a mongodb database and queries are performed to extract the plots that you will see below. We currently have seven mailing lists of varying traffic: announce, users, users-cn, dev, marketing, commits, issues. Note that all JIRA emails are now sent to the issues list. Subscription to these lists and number of messages last month is as follows:

* dev@ 609 subs / ~2600 msgs in Apr
* users@ 782 subs / ~800 msgs in Apr
* issues@ 109 subs / ~2400 msgs in Apr
* commits@ 166 subs / ~3300 msgs in Apr
* marketing@ 85 subs / ~260 msg in Apr
* users-cn@ ~300 subs / ~260 msgs in Apr

Contributors: The plots below show the number of contributors per month since we became an ASF project as well as an accumulation to date. Comparison with traffic prior to joining ASF can be seen in the previous posts. The number of monthly contributors in dev is reaching 225 , while the number of monthly contributors in users is reaching 175. Most notable is that the number of contributors in the users list seem to be closing on the number of contributors in dev. It may indicate a stabilization of the number of developers and an increase in the user base. The accumulation on both lists is now over 500. A comparison of both contributor sets gives us an estimate of 806 for the entire CloudStack community. Of course this does not include people who may only participate in the marketing or announce list, but they are much lower traffic lists. It also does not include participants in the Chinese user lists. This will be in the next post hopefully. From the subscription data listed above you can also see that we have roughly a 30% activity ratio, meaning that 1/3 of the subscribers actually send emails to the lists. Difficult to know if this is a good or bad number, one would need to compare with other ASF projects.

Companies: The plots below show the number of companies contributing on the dev and users lists as well as an accumulation to date. Similarly to the monthly contributor count we are seeing a faster diversification on the users list. This shows that the users list is adding more companies faster than the dev list (~80 and ~60 per month respectively). The accumulation has reached ~230 on the users list and ~190 on the dev list, for a combined 319 total.

Commits and Marketing: The commits list represents the number of committers than modify the code. These committers often apply patches submitted by other contributors without write access to the code. Therefore the plots should not be seen as the total number of code contributors. The plot below shows an increase in the number of Committers just shy of 40. The marketing list is a new list that deals with event planning, its trend is not yet established but the data shows between 50 and 30 contributors per month.

Social Networks: The following graphs shows social networks of the dev and users list. They are aimed at identifying who is most central to the community. It can be used to identify great contributors that should be recognized and be invited to become an Apache committer. Ultimately I want to use it to build a topic based network, so that people searching for a particular subject know who to talk to. I plan to build an interface that would use keywords to dynamically build these graphs and identify the people who contribute to that topic the most. The graphs show the networks for the last four months. Our new Apache CloudStack vice-president Chip Childers is clearly the most central node in the dev list and Ahmad Emneina is the most central contributor on the users list. The size of the nodes is proportional to their centrality and the thickness of the edges shows the strength of the connection between two nodes. Several nodes (contributors) have been filtered to render a readable picture.

Finally, it is important to note that some contributors are not active on the mailing lists. Even though this is an ASF mantra. Specifically we have engaged in a very active translation effort to bring CloudStack to all countries worldwide. Our translation team has 32 members as off the last count. All translation is handled via transifex. I am also working on git analysis to show better information on commits and I pointed out to John Jiang that he used the wrong repository in his latest study. Stay tuned.