Thursday, December 19, 2013

Clojure with CloudStack

CloStack

CloStack is a Clojure client for Apache CloudStack. Clojure is a dynamic programming language for the Java Virtual Machine (JVM). It is compiled directly in JVM bytecode but offers a dynamic and interactive nature of an interpreted language like Python. Clojure is a dialect of LISP and as such is mostly a functional programming language.

You can try Clojure in your browser and get familiar with its read eval loop (REPL). To get started, you can follow the tutorial for non-LISP programmers through this web based REPL.

To give you a taste for it, here is how you would 2 and 2:

user=> (+ 2 2)
4

And how you would define a function:

user=> (defn f [x y]
  #_=> (+ x y))
#'user/f
user=> (f 2 3)
5

This should give you a taste of functional programming :)

Install Leinigen

leiningen is a tool for managing Clojure projects easily. With lein you can create the skeleton of clojure project as well as start a read eval loop (REPL) to test your code.

Installing the latest version of leiningen is easy, get the script and set it in your path. Make it executable and your are done.

The first time your run lein repl it will boostrap itself:

$ lein repl
Downloading Leiningen to /Users/sebgoa/.lein/self-installs/leiningen-2.3.4-standalone.jar now...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.0M  100 13.0M    0     0  1574k      0  0:00:08  0:00:08 --:--:-- 2266k
nREPL server started on port 58633 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> exit
Bye for now!

Download CloStack

To install CloStack, clone the github repository and start lein repl:

git clone https://github.com/pyr/clostack.git
$ lein repl
Retrieving codox/codox/0.6.4/codox-0.6.4.pom from clojars
Retrieving codox/codox.leiningen/0.6.4/codox.leiningen-0.6.4.pom from clojars
Retrieving leinjacker/leinjacker/0.4.1/leinjacker-0.4.1.pom from clojars
Retrieving org/clojure/core.contracts/0.0.1/core.contracts-0.0.1.pom from central
Retrieving org/clojure/core.unify/0.5.3/core.unify-0.5.3.pom from central
Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom from central
Retrieving org/clojure/core.contracts/0.0.1/core.contracts-0.0.1.jar from central
Retrieving org/clojure/core.unify/0.5.3/core.unify-0.5.3.jar from central
Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.jar from central
Retrieving codox/codox/0.6.4/codox-0.6.4.jar from clojars
Retrieving codox/codox.leiningen/0.6.4/codox.leiningen-0.6.4.jar from clojars
Retrieving leinjacker/leinjacker/0.4.1/leinjacker-0.4.1.jar from clojars
Retrieving org/clojure/clojure/1.3.0/clojure-1.3.0.pom from central
Retrieving org/clojure/data.json/0.2.2/data.json-0.2.2.pom from central
Retrieving http/async/client/http.async.client/0.5.2/http.async.client-0.5.2.pom from clojars
Retrieving com/ning/async-http-client/1.7.10/async-http-client-1.7.10.pom from central
Retrieving io/netty/netty/3.4.4.Final/netty-3.4.4.Final.pom from central
Retrieving org/clojure/data.json/0.2.2/data.json-0.2.2.jar from central
Retrieving com/ning/async-http-client/1.7.10/async-http-client-1.7.10.jar from central
Retrieving io/netty/netty/3.4.4.Final/netty-3.4.4.Final.jar from central
Retrieving http/async/client/http.async.client/0.5.2/http.async.client-0.5.2.jar from clojars
nREPL server started on port 58655 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> exit

The first time that you start the REPL lein will download all the dependencies of clostack.

Prepare environment variables and make your first clostack call

Export a few environmen variables to define the cloud you will be using, namely:

export CLOUDSTACK_ENDPOINT=http://localhost:8080/client/api
export CLOUDSTACK_API_KEY=HGWEFHWERH8978yg98ysdfghsdfgsagf
export CLOUDSTACK_API_SECRET=fhdsfhdf869guh3guwghseruig

Then relaunch the REPL

$lein repl
nREPL server started on port 59890 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (use 'clostack.client)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
nil

You can safely discard the warning message which only indicates that 'clostack' is meant to be used as a library in a clojure project.
Define a client to your CloudStack endpoint

user=> (def cs (http-client))
#'user/cs

And call an API like so:

user=> (list-zones cs)
{:listzonesresponse {:count 1, :zone [{:id "1128bd56-b4d9-4ac6-a7b9-c715b187ce11", :name "CH-GV2", :networktype "Basic", :securitygroupsenabled true, :allocationstate "Enabled", :zonetoken "ccb0a60c-79c8-3230-ab8b-8bdbe8c45bb7", :dhcpprovider "VirtualRouter", :localstorageenabled true}]}}

To explore the API calls that you can make, the REPL features tab completion. Enter list or de and press the tab key you should see:

user=> (list
list                                list*                               list-accounts                       list-async-jobs                     
list-capabilities                   list-disk-offerings                 list-event-types                    list-events                         
list-firewall-rules                 list-hypervisors                    list-instance-groups                list-ip-forwarding-rules            
list-iso-permissions                list-isos                           list-lb-stickiness-policies         list-load-balancer-rule-instances   
list-load-balancer-rules            list-network-ac-ls                  list-network-offerings              list-networks                       
list-os-categories                  list-os-types                       list-port-forwarding-rules          list-private-gateways               
list-project-accounts               list-project-invitations            list-projects                       list-public-ip-addresses            
list-remote-access-vpns             list-resource-limits                list-security-groups                list-service-offerings              
list-snapshot-policies              list-snapshots                      list-ssh-key-pairs                  list-static-routes                  
list-tags                           list-template-permissions           list-templates                      list-virtual-machines               
list-volumes                        list-vp-cs                          list-vpc-offerings                  list-vpn-connections                
list-vpn-customer-gateways          list-vpn-gateways                   list-vpn-users                      list-zones                          
list?

user=> (de
dec                           dec'                          decimal?                      declare                       def                           
default-data-readers          definline                     definterface                  defmacro                      defmethod                     
defmulti                      defn                          defn-                         defonce                       defprotocol                   
defrecord                     defreq                        defstruct                     deftype                       delay                         
delay?                        delete-account-from-project   delete-firewall-rule          delete-instance-group         delete-ip-forwarding-rule     
delete-iso                    delete-lb-stickiness-policy   delete-load-balancer-rule     delete-network                delete-network-acl            
delete-port-forwarding-rule   delete-project                delete-project-invitation     delete-remote-access-vpn      delete-security-group         
delete-snapshot               delete-snapshot-policies      delete-ssh-key-pair           delete-static-route           delete-tags                   
delete-template               delete-volume                 delete-vpc                    delete-vpn-connection         delete-vpn-customer-gateway   
delete-vpn-gateway            deliver                       denominator                   deploy-virtual-machine        deref                         
derive                        descendants                   destroy-virtual-machine       destructure                   detach-iso                    
detach-volume

To pass arguments to a call follow the syntax:

user=> (list-templates cs :templatefilter "executable")

Start a virtual machine

To deploy a virtual machine you need to get the serviceofferingid or instance type, the templateid also known as the image id and the zoneid, the call is then very similar to CloudMonkey and returns a jobid

user=> (deploy-virtual-machine cs :serviceofferingid "71004023-bb72-4a97-b1e9-bc66dfce9470" :templateid "1d961c82-7c8c-4b84-b61b-601876dab8d0" :zoneid "1128bd56-b4d9-4ac6-a7b9-c715b187ce11")
{:deployvirtualmachineresponse {:id "d0a887d2-e20b-4b25-98b3-c2995e4e428a", :jobid "21d20b5c-ea6e-4881-b0b2-0c2f9f1fb6be"}}

You can pass additional parameters to the deploy-virtual-machine call, such as the keypair and the securitygroupname:

user=> (deploy-virtual-machine cs :serviceofferingid "71004023-bb72-4a97-b1e9-bc66dfce9470" :templateid "1d961c82-7c8c-4b84-b61b-601876dab8d0" :zoneid "1128bd56-b4d9-4ac6-a7b9-c715b187ce11" :keypair "exoscale")
{:deployvirtualmachineresponse {:id "b5fdc41f-e151-43e7-a036-4d87b8536408", :jobid "418026fc-1009-4e7a-9721-7c9ad47b49e4"}}

To query the asynchronous job, you can use the query-async-job api call:

user=> (query-async-job-result cs :jobid "418026fc-1009-4e7a-9721-7c9ad47b49e4")
{:queryasyncjobresultresponse {:jobid "418026fc-1009-4e7a-9721-7c9ad47b49e4", :jobprocstatus 0, :jobinstancetype "VirtualMachine", :accountid "b8c0baab-18a1-44c0-ab67-e24049212925", :jobinstanceid "b5fdc41f-e151-43e7-a036-4d87b8536408", :created "2013-12-16T12:25:21+0100", :jobstatus 0, :jobresultcode 0, :cmd "com.cloud.api.commands.DeployVMCmd", :userid "968f6b4e-b382-4802-afea-dd731d4cf9b9"}}

And finally to destroy the virtual machine you would pass the id of the VM to the destroy-virtual-machine call like so:

user=> (destroy-virtual-machine cs :id "d0a887d2-e20b-4b25-98b3-c2995e4e428a")
{:destroyvirtualmachineresponse {:jobid "8fc8a8cf-9b54-435c-945d-e3ea2f183935"}}

With these simple basics you can keep on exploring clostack and the CloudStack API.

Use CloStack within your own clojure project

Hello World in clojure

To write your own clojure project that makes user of clostack, use leiningen to create a project skeleton

lein new toto

Lein will automatically create a src/toto/core.clj file, edit it to replace the function foo with -main. This dummy function returns Hellow World !. Let's try to execute it. First we will need to define the main namespace in the project.clj file. Edit it like so:

defproject toto "0.1.0-SNAPSHOT" :description "FIXME: write description" :url "http://example.com/FIXME" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :main toto.core :dependencies [[org.clojure/clojure "1.5.1"]])

Note the :main toto.core

You can now execute the code with lein run john . Indeed if you check the -main function in src/toto/core.clj you will see that it takes an argument. Surprisingly you should see the following output:

$ lein run john
john Hello, World!

Let's now add the CloStack dependency and modify the main function to return the zone of the CloudStack cloud.

Adding the Clostack dependency

Edit the project.clj to add a dependency on clostack and a few logging packages:

:dependencies [[org.clojure/clojure "1.5.1"]
               [clostack "0.1.3"]
               [org.clojure/tools.logging "0.2.6"]
               [org.slf4j/slf4j-log4j12   "1.6.4"]
               [log4j/apache-log4j-extras "1.0"]
               [log4j/log4j               "1.2.16"
                :exclusions [javax.mail/mail
                             javax.jms/jms
                             com.sun.jdkmk/jmxtools
                             com.sun.jmx/jmxri]]])
                             

lein should have created a resources directory. In it, create a log4j.properties file like so:

$ more log4j.properties 
# Root logger option
log4j.rootLogger=INFO, stdout

# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

A discussion on logging is beyond the scope of this recipes, we merely add it in the configuration for a complete example.

Now you can edit the code in src/toto/core.clj with some basic calls.

(ns testclostack.core
  (:require [clostack.client :refer [http-client list-zones]]))

(defn foo
  "I don't do a whole lot."
  [x]
  (println x "Hello, World!"))

(def cs (http-client))

(defn -main [args]
  (println (list-zones cs))
  (println args "Hey Wassup")
  (foo args)
)

Simply run this clojure code with lein run joe in the source of your project. And that's it, you have sucessfully discovered the very basics of Clojure and used the CloudStack client clostack to write your first Clojure code. Now for something more significant, look at Pallet

No comments:

Post a Comment