Tuesday, September 03, 2013

CloudStack Google Summer of Code projects

Google Summer of Code is entering the final stretch with pencil down on Sept 16th and final evaluation on Sept 27th. Of the five projects CloudStack had this summer, one failed at mid-term and one led to committer status couple weeks ago. That's 20% failure and 20% outstanding results, on par with GSoC wide statistics I believe.

The LDAP integration has been the most productive project. Ian Duffy a 20 year old from Dublin did an outstanding job, developing his new feature in a feature branch, building a jenkins pipeline to test everything and submitting a merge request to master couple weeks ago. With 90% unittest coverage, static code analysis with Sonar in his jenkins pipeline and automatic publishing of rpms in a local yum repo, Ian exceed expectation. His code has even been already backported to the 4.1.1 release with the CloudSand distro of CloudStack.

The SDN extension project was about taking the native GRE controller in CloudStack and extend it to support XCP and KVM. Nguyen from Vietnam has done an excellent job quickly adding support for XCP thanks to his expertise with Xen. He is now putting the final touches on KVM support and building L3 services with OpenDaylight. The entire GRE controller was re-factored to be a plugin similar to the Nicira NVP, Midonet and BigSwitch BVS plugin. While native to CloudStack this controller brings another SDN solution to CloudStack. I expect to see his merge request before pencil down for what will be an extremely valuable project.

While the CloudStack UI is great, it was actually written has a demonstration of how the CloudStack API could be used to build a user facing portal. With the "new UI" project, Shiva Teja from India used boostrap and Angular to create a new UI. Originally the project suggested to use backbone but after feedback from the community Shiva switch to using Angular. Shiva's effort are to be commended as he truly worked on his own with in-consistent network connectivity and no local mentoring. Shiva is a bachelor student and had to learn bootstrap, angular and also Flask on his own. It must have been paying off since he is interviewing with Amazon and Goole for internships next summer. His code being independent of the CloudStack code has been committed to master in our tools directory. This creates a solid framework for other to build on and create their own CloudStack UI.

Perhaps the most research oriented project has been the one from Meng Han from Florida. This was no standard coding projects as it required not only to learn new technologies (aside from CloudStack) but also required investigation of the Amazon EMR API. Meng had to implement EMR in CloudStack using Apache Whirr. Whirr is a java library for provisioning of virtual machines on cloud providers. Whirr uses Apache jclouds and can interact with most cloud providers out there. Meng developed a new set of CloudStack APIs to launch hadoop clusters on-demand. At the start she had to learn CloudStack and install it, then learn the Whirr library and subsequently create a new API in CloudStack which would use Whirr to coordinate multiple node deployments. Meng's code is working but still a bit short from our goal of having a AWS EMR interface. This is partly my fault has this project could have required more mentoring. In any case, the work will go on and I expect to see an EMR implementation in CloudStack in the coming months.

All students faced the same challenge, not a code writing challenge but the OSS challenge and specifically learning the Apache Way. Apache is about consensus and public discussions on the mailing list. With several hundreds participants every month and very active discussion, the shear amount of email traffic can be intimidating. Sharing issues and asking for help on public mailing list is still a bit frightening. IRC, intense emailing, JIRA and git are basic tools used in all Apache project, but seldom used in academic settings. Learning these development tools and participating in a project with over a million line of code was the toughest challenge for students and the goal of GSoC. I am glad we got five students to join CloudStack this summer and tackle these challenges, if anything it is a terrific experience that will benefit their own academic endeavor and later their entire career. Great job Ian, Meng, Nguyen, Shiva and Dharmesh, we are not done yet but I wish you all the best.