Control the Cloud Programmatically with Java at CESWP Tech Blog

CESWP Tech Blog

Technical notes from the CESWP Project

Control the Cloud Programmatically with Java

without comments

Summary
The cloud is rapidly becoming an essential tool that allows organizations, entrepreneurs
and scientists to innovate without a large, upfront cost for computing and storage
infrastructure. Anyone can start virtual machines (VMs) in the cloud and use them to
create the next critical business application, killer web application or simulation to
predict hurricanes. To support that innovation this tutorial is for Java developers looking
to control both a Eucalyptus cloud and the Amazon Elastic Compute Cloud and access
the virtual machines within them programmatically using Java APIs.

Objective
At the end of this tutorial you should be able to:

1. start a Linux virtual machine

2. create storage and attach it to the VM
3. make a filesystem storage and mount it in the VM
4. do something interesting within the VM
5. detach the storage and delete it
6. terminate the VM

using Java code.

Overview
When developers want to harness the full capability of the cloud (in the IaaS sense of the
word) they need to be able to control it programmatically from within their applications.
When it comes to Java there are a number of options available to developers for choosing
an API that will allow them to control multiple cloud providers. There are:

typica
jclouds
Deltacloud
Dasein
libcloud

Our requirements for the CESWP project are to support a Eucalyptus cloud and the

1 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

Amazon Elastic Compute Cloud (EC2). typica has initially been selected because it
supports both clouds out-of-the-box.

jclouds support for Eucalyptus is only in the nightly snapshot builds. A little too bleeding
edge for me and there is no documentation on using jclouds with Eucalyptus. Neither
Deltacloud nor Dasein explicitly support Eucalyptus. Although Eucalyptus claims to be
compatible with the EC2 API I’ve found that unless it’s documented, tested and officially
supported by the API then it probably won’t work at all. libcloud (a Java port of the
python libcloud) is still in the Apache incubator but it looks promising.

To this end I’ve begun a Java Cloud (Agnostic) API Survey that we may add to, if we
evaluate and experiment with other APIs.

What follows is a tutorial on using typica to control virtual machines and their associated
storage in the cloud. One final piece of the puzzle for controlling VMs in the cloud is an
SSH library to run commands directly on the VMs. For this I’ve used the excellent sshj.
The relevant versions used in this tutorial are:

typica 1.7.2
sshj 0.3.0
Eucalyptus 2.0.0

The code in this tutorial is not production ready. The code is only here to illustrate
control of the cloud. Most error handling has been omitted to make the code clearer and
is left as an exercise for the reader.

Before You Start

Account(s) needed for the tutorial:

One or both of the accounts below will be need for the tutorial.

Eucalyptus

Cost: Free

Most people won’t have the time, resources or inclination to build their own
Eucalyptus cloud. Instead I suggest you apply for an account on the Eucalyptus
Community Cloud (ECC). Follow the instructions to get an account. It may take a
day or longer to get your account approved so start as early as possible. Once you
have your account follow the guide Getting Started Using Eucalyptus 2.0 guide.
Important! Make a note of the key name and where you saved the private key
when creating your key pair. Do not skip the euca-authorize step!

Note that you cannot create storage and attach it to the VM in ECC. It’s a limitation
Eucalyptus put into this particular cloud to prevent abuse because the accounts are
free.

Amazon Elastic Compute Cloud

2 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

Cost: Get your credit card out (but prices are very low for short lived VMs like we’ll
be using)

Simply follow the directions in the Get Started with EC2 guide. Important! Make a
note of the key name and where you saved the private key when creating your
key pair. As part of this tutorial when you’re in the AWS Management Console click
on Security Groups. Choose default and in the window below add the Connection
Method SSH and click Save.

Software needed to run through the tutorial:

Java Development Kit 6 – openjdk-6-jdk is fine if you’re on Linux, e.g. yum/apt-get

install openjdk-6-jdk
Eclipse IDE for Java Developers (optional) – Optional but highly recommended. This
is the easiest way to get started with this tutorial but if you can’t (or won’t) install
Eclipse we’ve got you covered.
Subclipse – Having the Subclipse plugin will make things really, really easy.
Follow the Installation Instructions on the Download and Install page.
Subversion (if not using Eclipse w/Subclipse) – Download and install for your
platform.
EC2 API Tools – The Windows/Linux command line tools for controlling VMs in EC2.
You’ll want to read the Getting Started with the Command Line Tools guide.
Euca2ools – The Linux command line tools for controlling VMs in Eucalyptus. You’ll
want to read the Using Euca2ools Overview.
Hybridfox (optional) – A very useful FireFox add-on for controlling VMs in the
cloud.

Tutorial
Get and Compile the Code

The code for this tutorial is hosted at http://code.google.com/p/typica-tutorial. You can

browse the code online starting at http://code.google.com/p/typica-tutorial/source
/browse/#svn/trunk/src/org/cybera.

Eclipse

It’s done as an Eclipse project so for those of you using Eclipse the quickest way to get
started is:

1. Fire up Eclipse
2. File > New > Other > type “svn” > Checkout Projects from SVN > Next
3. Create a new repository location > Next
4. Select the URL > Finish

This will checkout and compile the code for you. Done.

Non-Eclipse

3 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

To get and compile the source code run the following commands:

1. svn co http://typica-tutorial.googlecode.com/svn/trunk/ typica-tutorial

2. cd typica-tutorial
3. mkdir bin
4. javac -d bin -cp “lib/*” src/org/cybera/ssh/* (on Windows change the / to \)
5. javac -d bin -cp bin:”lib/*” src/org/cybera/TypicaTutorial.java (on Windows change
the / to \ and : to ; )

Configuration

Now we need to setup the configuration files that contain the cloud specific parameters
for Eucalyptus and EC2. Open both typica-tutorial/conf/aws.properties and typica-
tutorial/conf/euca.properties.

aws.properties

cloud.accessId – Look at your AWS Security Credentials and it’s the one labelled
Access Key ID
cloud.secretKey – Look at your AWS Security Credentials and it’s the one labelled
Secret Access Key
cloud.username – Look at your AWS Security Credentials and it’s the one labelled
AWS Account ID (near the bottom of the page). Important! Remove the hyphens!
launchConfig.imageID – The ID of the Amazon Machine Image (AMI) you want to
run. You can use ami-ff6c8596, it’s a CentOS 5.5 image. A good place to find more
images is the cloud market.
launchConfig.availabilityZone – The zone where your AMI will run. You can set this
to “us-east-1d” (no quotes).
launchConfig.keyName – The key name from when you got started with your
Amazon account in the Before You Start section.
launchConfig.minCount=1 – the minimum number of VMs to run. Leave it as 1.
launchConfig.maxCount=1 – the maximum number of VMs to run. Leave it as 1.
attachVolume.deviceName=/dev/sdf – This is the Linux device you’ll attached your
storage to. Leave it as /dev/sdf.
vmAuth.username – The username you’ll use to login to your VM. If you use the
suggest AMI above set this to “root” (no quotes). If it was an Ubuntu image you’d
most likely use “ubuntu”.
vmAuth.privateKeyLocation – The absolute path to the private key from when you
got started with your Amazon account in the Before You Start section.
vmAuth.sudo – Set this property to “sudo” (no quotes) if your username is anything
other than root.

euca.properties

cloud.accessId – Look at your ECC Credentials and it’s the one labelled Query ID
cloud.secretKey – Look at your ECC Credentials and it’s the one labelled Secret Key
cloud.URL – Set this property to “ecc.eucalyptus.com” (no quotes)
cloud.username – The username you used to sign up with ECC

4 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

launchConfig.imageID – The ID of the Eucalyptus Machine Image (EMI) you want to

run. You can use emi-9ACB1363, it’s a CentOS 5.3 image. The place to find more
images is the ECC Images.
launchConfig.availabilityZone – The zone where your EMI will run. Set this to
“open” (no quotes)
launchConfig.keyName – The key name from when you got started with your
Eucalyptus account in the Before You Start section.
launchConfig.minCount=1 – the minimum number of VMs to run. Leave it as 1.
launchConfig.maxCount=1 – the maximum number of VMs to run. Leave it as 1.
attachVolume.deviceName=/dev/vdb – This is the Linux device you’ll attached your
storage to. Leave it as /dev/vdb.
vmAuth.username – The username you’ll use to login to your VM. If you use the
suggest AMI above set this to “root” (no quotes). If it was an Ubuntu image you’d
most likely use “ubuntu”.
vmAuth.privateKeyLocation – The absolute path to the private key from when you
got started with your Eucalyptus account in the Before You Start section.
vmAuth.sudo – Set this property to “sudo” (no quotes) if your username is anything
other than root.

Explore the Code

Note that the source and Javadoc of the primary dependencies typica and sshj are
included in typica-tutorial/lib/sources so you can see what’s going on under the hood.

TypicaTutorial.TypicaTutorial()

Setup all of the properties.

TypicaTutorial.start()

Top level method to start all of the other methods.

TypicaTutorial.describeImages()

Calls Jec2.describeImagesByOwner() which returns all of the AMIs/EMIs that the

owner has created. This call won’t actually return anything for you because you
haven’t created any images but it’s a simple call to make that verifies connectivity to
the cloud.

TypicaTutorial.runInstance()

Calls Jec2.runInstances() which runs an instance of a virtual machine based on the

launch configuration specified by the properties you changed in the Configuration
section above. It also tests the state of the instance to determine when it’s running
and waits and extra 30 seconds to give the SSH server on the VM a chance to startup
too. This way the method won’t return until you have a fully functioning VM.

TypicaTutorial.createAttachAndMountVolume()

Calls Jec2.createVolume() to create some storage for the VM (unfortunately this

5 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

doesn’t work on the Eucalyptus Community Cloud). Waits for the storage to be ready
and the calls Jec2.attachVolume() to attach that storage to the VM. Waits for the
storage to be attached and then runs the commands via SSH to make a filesystem
and mount the storage in the VM.

TypicaTutorial.doInterestingStuff()

This is where you come in. Right now the tutorial just runs a simple ping
command over SSH. Change the command and recompile! Experiment what you can
do with a complete virtual machine at your disposal. Run multiple commands over
during one connection like is done when the storage is mounted. For example, install
subversion (yum install -y subversion), download some code from the web, compile
and execute it.

TypicaTutorial.unmountDetachAndDeleteVolume()

First unmounts the storage so it can be detached without corrupting the storage
(even though were just going to delete it anyway). Calls Jec2.detachVolume() to
detach the storage from the VM and waits for it to detach. Calls Jec2.deleteVolume()
to delete the volume but doesn’t wait for it to be deleted because this can take a
while and we’re not dependant on the storage being deleted.

TypicaTutorial.terminateInstance()

Calls Jec2.terminateInstances to kill the running VM but doesn’t wait for it to die
because this can take a while and we’re not dependant on the VM being terminated.
One thing to note is that the call to terminate the instance goes through but it looks
like Eucalyptus returns a null value and typica chokes on it so we ignore the
Exception (see http://code.google.com/p/typica/issues/detail?id=105).

Run the Code

Note that it can take a long time for VMs to start running or for storage to be created so
be patient with the process.

Eclipse

1. Open the file typica-tutorial/src/org/cybera/TypicaTutorial.java

2. Run > Run Configurations…
3. Select Java Application and press the New launch configuration button
4. The Name (TypicaTutorial), Project (typica-tutorial) and Main class
(org.cybera.TypicaTutorial) should all be filled in since your editor was on the
TypicaTutorial.java file
5. Switch to the Arguments tab and type “euca” (no quotes) in the Program arguments
textfield
6. If you’re ready to go click Run, otherwise click Apply and Close and you can run it
later.
7. For EC2 do steps 1-6 above but in the Arguments tab type “aws” (no quotes)

6 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

Non-Eclipse

1. cd typica-tutorial
2. For Eucalyptus: java -cp conf:bin:”lib/*” org.cybera.TypicaTutorial euca (on
Windows change the / to \ and : to ; )
3. For EC2: java -cp conf:bin:”lib/*” org.cybera.TypicaTutorial aws (on Windows
change the / to \ and : to ; )

References
typica Google Group
sshj Google Group
Eucalyptus support forum

Future Work
Building a library on top of typica for coarser grained functionality (e.g. a single method
to start a VM and attach and mount X GBs of storage).

Try different Java Cloud APIs and add our findings to Java Cloud (Agnostic) API Survey.

Try different cloud management software (e.g. OpenStack).

Experiment with Java APIs that control storage in the cloud (e.g. S3 on AWS and Walrus
on Eucalyptus) and write a similar tutorial for that.

Written by Everett Toews

September 17th, 2010 at 5:28 pm

Posted in Uncategorized

« Troubleshooting Eucalyptus
Eucalyptus User-data »

Leave a Reply

Name (required)

Mail (will not be published) (required)

Website

7 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

Submit Comment

Pages

About
Project Tools

Search

search site archives

search

Blogroll

CESWP Blog

Archives

April 2011
January 2011
December 2010
October 2010
September 2010
June 2010
May 2010
April 2010
January 2010
December 2009
November 2009

Categories

Hypervisors
Sys Admin
Uncategorized

Meta

8 of 9
Control the Cloud Programmatically with Java at CESWP Tech Blog

Log in
Site Feed
Comments Feed
Back to top

The Journalist template by Lucian E. Marin — Built for WordPress

9 of 9