Hybrid Cloud Tutorial: From Planning to Classroom

David Rilett and Charlie Wiseman

Department of Computer Science and Networking Wentworth Institute of Technology

Heading into the Cloud

Cloud computing means many different things today This talk will do two things:
Provide an overview of the cloud world Go through the procedure to build a private or hybrid cloud platform using Eucalyptus, an open source cloud system that is compatible with Amazon Web Services (EC2/S3)

What was old

Cloud computing is the evolution of many other computing paradigms
Grid, Utility, Mainframes,

Virtualization is the key enabler

Better utilization of physical servers and storage with fast backup recovery Simpler migration of virtual instances Easier provisioning on demand

Big Picture
End Users

Application

Cloud Consumer

Cloud Provider

Example
Many R&D businesses rely on cloud environments for both long -term and shortterm services For example:
Deployed 50,000 cores of computing power for 3 hours Cost $15,000

Types of Cloud Models

Cloud computing systems are characterized in two ways: deployment model and service model The deployment model describes who the cloud consumers are The service model describes what types of resources are provided to the cloud consumers

Deployment Models
Public cloud
Resources are shared by many consumers and are provided to anyone (for a fee)

Private cloud
Resources are utilized by a single organization Can still be operated by a third party

Hybrid cloud
Mix of public and private, allowing the consumer to deploy applications using either public or private resources

Service Models
Infrastructure as a Service (IaaS)
Virtual machines and storage are provided Consumers install their own OS from the ground up

Platform as a Service (PaaS)

Preconfigured computing platforms are provided For example, pre-built web systems

Consumers only write the application

Software as a Service (SaaS)

Everything through the application is provided

Consumers can only customize and tweak

IaaS or PaaS?
Depending on the requirements, you should be using an IaaS or PaaS model
Give students a preconfigured image with everything they need on it (PaaS) Allow them to install their own OS and do all the configuration themselves (IaaS)

The former is more likely in a software development course, the later more likely in a networking or administration course

Public, Private, or Hybrid?

Benefits of public clouds like Amazon Web Services (AWS):
Requires no physical resources at your institution No limit on how many instances you can run

Drawback of using a public cloud:

Pay per usage (typically per server hour and per GB of data transfer) For example: http://aws.amazon.com/ec2/pricing/

Public, Private, or Hybrid?

Benefits of building your own private cloud:
Full control over the resources No per use fees

Drawbacks of a private cloud:

Have to buy all the equipment up front Can't exceed the resources you buy (locally)

Public, Private, or Hybrid? (2)

A common solution, therefore, is to use a hybrid environment Build your own private cloud with a reasonable number of resources Do all development and initial testing in the private cloud For scale-out testing or demos, move the applications to the public cloud

Enter Eucalyptus
Eucalyptus is an open source IaaS cloud platform Compatible with AWS (EC2/S3) Deploy your own private Eucalyptus cloud Move the Eucalyptus images to AWS if you need more resources RightScale, euca2ools and HybridFox help with moving and management

Building a Eucalyptus Cloud

Requirements:
At least two servers (more depending on how many instances you need to support) Cooperation of your friendly local network administrator for public IP addresses and firewall configuration

Recommendations:
External storage array for storing images Switch for building a separate private network for the cloud

Building a Eucalyptus Cloud

Eucalyptus website provides a lot of information but may lack adequate explanation
http://open.eucalyptus.com/wiki/EucalyptusInstallation_v2.0

There is some terminology to get out of the way You need one server to act as the front-end, which will run all of the cloud-level services and store S3 snapshots in Walrus Other servers are nodes for deploying instances of virtual machine images and EBS volumes

Building a Eucalyptus Cloud

The most important decision you have to make is which networking mode you are going to use with Eucalyptus There are four networking modes to choose from We recommend a setup with the MANAGEDNOVLAN mode as it provides good flexibility without too much extra hassle

Networking Modes
MANAGED-NOVLAN mode
Require a switch to interconnect the private cloud network ( or crossover cable to get one node up) The nodes will not be connected to your regular network directly but use private network

MANAGED mode is similar

Requires VLAN-configured switches (not too difficult) The only benefit is layer 2 isolation of instances which is less important for a course

Networking Modes
The other two modes are STATIC and SYSTEM STATIC allows you to configure IP addressing by hand in the cloud
Convenient but very hard to manage in even a small environment

SYSTEM mode should only be used if can't get a switch for MANAGED-NOVLAN mode
Requires more cooperation with your local network admin since not managed by Eucalyptus

Network Configuration
From now on, we will be using MANAGEDNOVLAN mode One very tricky part that isn't explained well in the documentation is that there are really three different sets of IP addresses you must work with
Public, Private, and Virtual Instance IPs

Network Configuration
LAN/ Internet
public address: 5.5.5.5 (eth0)

public network
front-end (cloud controller)

private address: 192.168.0.1 (eth1) private addresses: 192.168.0.X (eth0) internal network addresses: 192.168.0.X, optional external addresses 5.5.5.X

private network nodes (node controllers) VMs

Virtual Instances IPs

When you configure Eucalyptus on the front-end, you have to tell it the public and private IP addresses of the frontend You also have to choose a third set of IP addresses used for the virtual network of instances running on the nodes This must be separate from the other IP ranges Assuming you use 192.168.0.0/24 for the private ones, use something like 10.0.0.0/16 for the virtual IPs

Additional Public IPs

Lastly, you need an additional set of public IP addresses that can be assigned to instances that want to be accessible from outside of the private network Ask your network admin for these
For example, might get 5.5.5.10 5.5.5.50 Must ensure that regular network doesn't use these addresses already and that firewalls/routers (DNS?) know about them

Example Configuration File

Example virtual network configuration on the cloud controller (eucalyptus.conf):
VNET_PUBINTERFACE="eth0" VNET_PRIVINTERFACE="eth1" VNET_DHCPDAEMON="/usr/sbin/dhcpd" VNET_MODE="MANAGED-NOVLAN" VNET_SUBNET="10.10.0.0" VNET_NETMASK="255.255.0.0" VNET_DNS="5.5.1.1" VNET_ADDRSPERNET="32" VNET_PUBLICIPS="5.5.5.10-5.5.5.50"

Other Gotchas *
Installing the front-end (we used CentOS 5.8)

Be sure to disable SELinux Enable NTP Install a DHCP server, but DON'T enable it or configure it Configure both public and private NICs statically * Faststart manages this for you

Other Gotchas*
Installing the nodes (we used CentOS 5.8)
They won't have Internet access in the given configuration, so you could temporarily connect them to your regular network until the install is done Disable SELinux Disable the local firewall (iptables) Enable NTP Set the IP address statically After you install xen, you might have to edit the GRUB configuration to tell it to boot the xen kernel Start with a single node until things are working to simplify debugging * Faststart manages this

Faststart Install CENTOS 5.7 +

EucalyptusFastInstallGuide-CentOS.pdf
Need: CentOS 5.7 install CD #1. You can find the image on one of the mirrors here: http://isoredirect.centos.org/centos/5/isos/x86_64/. The file is labeled CentOS-5.7-x86_64-bin-1of8.iso. Need: Eucalyptus Fast Start media (1GB USB) Video: http://vimeo.com/25163094 On install, do network interface configuration, select Edit and manually configure IP, netmask, hostname, gateway and DNS. You will need to check the DNS in resolv.conf, the default route in /etc/sysconfig/network and you interface setup in /etc/sysconfig/network-scripts/ifcfg-eth0 etc. DEVICE=eth0 BOOTPROTO=none HWADDR=78:2B:CB:3A:A9:3A ONBOOT=yes IPADDR=69.43.72.190 NETMASK=255.255.255.240 GATEWAY=69.43.72.177

http://www.eucalyptus.com/sites/all/files/EucalyptusFastInstallGuide-CentOS.pdf

 There are two of these, one on each machine: Frontend (cc walrus) and Node Controller (nc) Restart /etc/init.d/eucalyptus-cc cleanrestart for cc when edit on frontend Restart /etc/init.d/eucalyptus-nc restart for nc when edit node(s) Be careful with /etc/init.d/eucalyptus-cloud restart on Frontend -May lose external addresses on VM instances or may forget to take snapshots of data

EucalyptusConfiguration (/etc/eucalyptus/eucalyptus.conf)

eucarc run-time environment

. /root/.euac/eucarc (source it for euca2ools)
EUCA_KEY_DIR=$(dirname $(readlink -f ${BASH_SOURCE})) export S3_URL=http://69.43.72.190:8773/services/Walrus export EC2_URL=http://69.43.72.190:8773/services/Eucalyptus export EC2_PRIVATE_KEY=${EUCA_KEY_DIR}/euca2-admin-f5052272-pk.pem export EC2_CERT=${EUCA_KEY_DIR}/euca2-admin-f5052272-cert.pem export EC2_JVM_ARGS=-Djavax.net.ssl.trustStore=${EUCA_KEY_DIR}/jssecacerts export EUCALYPTUS_CERT=${EUCA_KEY_DIR}/cloud-cert.pem export EC2_ACCESS_KEY='WKy3rMzOWPouVOxK1p3Ar1C2uRBwa2FBXnCw' export EC2_SECRET_KEY='GbxmI31soVHbDr8o6dQpOwTAmlrrrisP1HSQ' # This is a bogus value; Eucalyptus does not need this but client tools do. export EC2_USER_ID='339620359308903565207410324383178424330' alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 339620359308903565207410324383178424330 --ec2cert ${EUCALYPTUS_CERT}" alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${EUCALYPTUS_CERT} * Need lines in red to register with HyperFox etc.

euca2ools
Initialization variables: $ . ~/.euca/eucarc # This sources the environment, check with env Informational $ euca-describe-availability-zones verbose $ euca-describe-images $ euca-describe-keypairs $ euca-describe-instances $ euca-describe-volumes $ euca-describe-snapshots $ euca-describe-addresses $ euca-describe-groups

euca2ools 2
Keypairs: $ euca-add-keypair mykey > mykey.priv $ euca-delete-keypair mykey Registering kernel image: $ euca-bundle-image -i vmlinuz-2.6.28-11-server --kernel true $ euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.28-11-server.manifest.xml $ euca-register mybucket/vmlinuz-2.6.28-11-server.manifest.xml Registering ramdisk image: $ euca-bundle-image -i initrd.img-2.6.28-11-server $ euca-upload-bundle -b mybucket -m /tmp/initrd.img-2.6.28-11-server.manifest.xml $ euca-register mybucket/initrd.img-2.6.28-11-server.manifest.xml

euca2ools 3
Registering disk image: $ euca-bundle-image -i image.img --kernel eki-XXXXXXXX --ramdisk eri-XXXXXXXX $ euca-upload-bundle -b mybucket -m /tmp/image.img.manifest.xml $ euca-register mybucket/image.img.manifest.xml Launching and manage instances: $ euca-run-instances emi-XXXXXXXX -k mykey -t c1.medium $ euca-get-console-output i-XXXXXXXX Launching an instance associated with the security group webservers : $ euca-run-instances emi-XXXXXXXX -k mykey -g webservers

euca2ools 4
Connecting to the instance: $ ssh -i mykey.priv user@ip $ euca-reboot-instances i-XXXXXXXX $ euca-terminate-instances i-XXXXXXXX Create new 10GB volume: $ euca-create-volume -s 10 -z mycloud $ euca-create-volume -s 10 --snapshot snap-ZZZZZZZZ --zone mycloud Attach a volume to a runing instance: $ euca-attach-volume -i i-XXXXXXX -d /dev/sdb vol-YYYYYYYY

euca2ools 5
Detach a volume: $ euca-detach-volume vol-YYYYYYYY $ euca-delete-volume vol-YYYYYYYY ### for windows only Snapshots: $ euca-create-snapshot vol-YYYYYYYY $ euca-delete-snapshot snap-ZZZZZZZZ Allocating IP address: $ euca-allocate-address ADDRESS IP Releasing a public ip address from a specify user: $ euca-release-address IP

euca2ools 6
MORE ADDRESS IP Associating a public ip address to a running instance $ euca-associate-address -i i-XXXXXXXX ip Disassociating a public ip address from a running instance $ euca-disassociate-address ip Creating a security group named webservers: $ euca-add-group -d "Web Servers" webservers Adding a rule to the security group webservers allowing icmp and tcp traffic from a.b.c.d: $ euca-authorize -P tcp -s a.b.c.d webservers $ euca-authorize -P icmp -s a.b.c.d webservers

euca2ools 7
Removing the rule for icmp traffic from the source ip a.b.c.d from the security group webservers: $ euca-revoke -P icmp -s a.b.c.d webservers Deleting the security group webservers: $ euca-delete-group webservers SOURCE: http://cssoss.wordpress.com/2010/05/10/eucalyptus-beginner%E2%80%99s-guide-%E2%8093-uec-edition-chapter-10%E2%80%93-euca%C2%A0commands

Set Up Images: Create Kernel

initrd-2.6.27.21-0.1-xen vmlinuz-2.6.27.21-0.1-xen [root@css1 xen-kernel]# euca-bundle-image -i vmlinuz-2.6.27.21-0.1-xen -kernel = true Invalid private key [root@css1 xen-kernel]# euca-bundle-image -i vmlinuz-2.6.27.21-0.1-xen --kernel=true Checking image Tarring image Encrypting image Splitting image... Part: vmlinuz-2.6.27.21-0.1-xen.part.0 Generating manifest /tmp/vmlinuz-2.6.27.21-0.1-xen.manifest.xml

Upload and Register

[root@css1 xen-kernel]# euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.27.21-0.1-xen.manifest.xml Checking bucket: mybucket Uploading manifest file Uploading part: vmlinuz-2.6.27.21-0.1-xen.part.0 Uploaded image as mybucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml [root@css1 xen-kernel]# euca-register mybucket/vmlinuz-2.6.27.21-0.1-xen.manifest.xml IMAGE eki-5FB9126E

Set Up Images: Create Ramdisk

euca-bundle-image -i initrd-2.6.27.21-0.1-xen Checking image Tarring image Encrypting image Splitting image... Part: initrd-2.6.27.21-0.1-xen.part.0 Generating manifest /tmp/initrd-2.6.27.21-0.1-xen.manifest.xml

Upload and Register

euca-upload-bundle -b mybucket -m /tmp/initrd-2.6.27.21-0.1-xen.manifest.xml Checking bucket: mybucket Uploading manifest file Uploading part: initrd-2.6.27.21-0.1-xen.part.0 Uploaded image as mybucket/initrd-2.6.27.21-0.1-xen.manifest.xml [root@css1 xen-kernel]# euca-register mybucket/initrd-2.6.27.21-0.1-xen.manifest.xml IMAGE eri-410611E1

Set Up Images: Create Image

[root@css1 euca-ubuntu-9.04-x86_64]# euca-bundle-image -i ubuntu.9-04.x86-64.img --kernel eki-5FB9126E --ramdisk eri-410611E1 Checking image Tarring image Encrypting image Splitting image... Part: ubuntu.9-04.x86-64.img.part.0 Part: ubuntu.9-04.x86-64.img.part.1 Part: ubuntu.9-04.x86-64.img.part.2 Part: ubuntu.9-04.x86-64.img.part.3 Part: ubuntu.9-04.x86-64.img.part.4 Part: ubuntu.9-04.x86-64.img.part.5 Part: ubuntu.9-04.x86-64.img.part.6 Part: ubuntu.9-04.x86-64.img.part.7 Part: ubuntu.9-04.x86-64.img.part.8 Part: ubuntu.9-04.x86-64.img.part.9 Part: ubuntu.9-04.x86-64.img.part.10 Part: ubuntu.9-04.x86-64.img.part.11 Part: ubuntu.9-04.x86-64.img.part.12 Part: ubuntu.9-04.x86-64.img.part.13 Generating manifest /tmp/ubuntu.9-04.x86-64.img.manifest.xml

Upload
[root@css1 euca-ubuntu-9.04-x86_64]# euca-upload-bundle -b mybucket -m /tmp/ubuntu.9-04.x8664.img.manifest.xml Checking bucket: mybucket Uploading manifest file Uploading part: ubuntu.9-04.x86-64.img.part.0 Uploading part: ubuntu.9-04.x86-64.img.part.1 Uploading part: ubuntu.9-04.x86-64.img.part.2 . Uploading part: ubuntu.9-04.x86-64.img.part.10 Uploading part: ubuntu.9-04.x86-64.img.part.11 Uploading part: ubuntu.9-04.x86-64.img.part.12 Uploading part: ubuntu.9-04.x86-64.img.part.13 Uploaded image as mybucket/ubuntu.9-04.x86-64.img.manifest.xml

Register
[root@css1 euca-ubuntu-9.04-x86_64]# euca-register mybucket/ubuntu.9-04.x86-64.img.manifest.xml IMAGE emi-345611F6

Wentworth Eucalyptus Front End

Use to view images/instances/credentials/ and manage users

hybridfox

Use to manage instances/AWS/Security Groups/Volumes/Snapshots/etc.

http://code.google.com/p/hybridfox/

SUSE Studio
http://susestudio.com/home

Troubleshooting
There are many moving parts in Eucalyptus and it can frustrating to fix The most common failure case is that you try to run a new instance and it stays "pending" for a long time before terminating When this happens, there are a few places to check

Troubleshooting: DHCP
If the instance does not get an IP address (public or private) then mostly likely the DHCP assignments are not working
When you actually run an instance, the cloud controller on the frontend will start a dhcpd process to assign IP addresses to the new instance If you do a 'ps ef | grep dhcpd' and don't see anything, then DHCP is not running during setup

This almost certainly means that the VNET configuration is wrong (VNET overlap with private network, etc)
You can also look in /var/run/eucalyptus/net/ for the DHCP configuration and lease files

Troubleshooting: Log Files

The best place to look for clues is in the log files As with most log files, it can be hard to pull the relevant information out of the noise On the front-end, look in /var/log/eucalyptus/cc.log On the nodes, look in /var/log/eucalyptus/nc.log

Troubleshooting Example: cc.log

[DAY MON DD HH:MM:SS YYYY][004513][EUCAINFO ] DescribeResources(): resource response summary (name{avail/max}): m1.small {0/0} c1.medium{0/0} m1.large{0/0} m1.xlarge{0/0} c1.xlarge{0/0}

Look for the above "resource response summary" messages They will tell you how many resources are available for each instance size In the above, you see that it says there are 0 available and 0 total for each size which means that the front-end can't talk to the nodes, probably because of a firewall problem

Troubleshooting Example: cc.log

When the nodes and front-end can talk, you should messages like this:
[DAY MON DD HH:MM:SS YYYY][004513][EUCAINFO ] DescribeResources(): resource response summary (name{avail/max}): m1.small{16/16} c1.medium{16/16} m1.large{8/8} m1.xlarge{8/8} c1.xlarge{4/4}

And like this from the individual nodes:

[DAY MON DD HH:MM:SS YYYY][011297][EUCADEBUG ] refresh_resources(): received data from node=192.168.0.2 mem=16083/16083 disk=856137/856137 cores=16/16

Troubleshooting Example: nc.log

[DAY MON couldn't [DAY MON download DD HH:MM:SS YYYY][006411][EUCAERROR ] walrus_request(): connect to host (7) DD HH:MM:SS YYYY][006411][EUCAERROR ] retry 10 of 10 will commence in 4 seconds

In general, look for the [EUCAERROR lines to find errors that you need to address In the above example, it tells you that the node can't contact the walrus component of front end (for S3 storage) (i.e. snapshots and images) Again, this is probably a firewall issue, so be sure the firewall is off on the node

Troubleshooting: Images
Another issue is broken or incompatible images If Eucalyptus tells you that an instance is running but you can't log in, try pinging it If you can't ping it, the instance didn't boot correctly You can test it with xen
Log in to the node and run "xm list" You can then log in directly to the instance with "xm console INSTANCE" If it's waiting at a maintenance prompt then something is wrong with the image try another image Note, do a CTRL-] to exit the xen console

How Many Servers Do I Need?

For most purposes, you should think about 1 core, 1GB of memory, and 10GB of storage for each student
So, if you have a course of 30 students, you need roughly 32 cores and 32 GB of memory (HDDs are not costly) You can get a single server with dual hyper-threaded quad core processors and 16GB of memory for ~$4k

Add another $2k for the front-end and it's about $10k total (but dont forget switches etc.)

Cloud in the Classroom

There are many ways to bring the cloud into a course In software courses, every student could be assigned their own virtual machine for development and testing or run high performance or parallel applications In networking courses, students could create entire virtual networks to study protocols and network architectures

Conclusion

If you are interested in setting up a private or hybrid cloud for a course, Xen is reasonable option Eucalyptus/AWS are an infrastructure and web services platform in the cloud Cloud networking is an important new tool for education, scientific discovery and business logic services The benefit is that you can validate and verify locally before for-fee AWS cloud with a larger implementation EBS coupled with S3 gives a fast and safe way to protect data. Freddy Fender Wasted Days is good Koala theme music

AWS Info
http://aws.amazon.com/govcloud-us/ http://aws.amazon.com/resources/webinars/ http://aws.typepad.com/ http://www.youtube.com/user/AmazonWebServices/videos?vie w=pl http://calculator.s3.amazonaws.com/calc5.html http://aws.amazon.com/ec2/instance-types/

CONTACTS
David Rilett Charlie Wiseman