CS6712 Grid and Cloud Computing Laboratory

Website: https://larshansolutions.blogspot.
in
http://liveinternetjobs.blogspot.com
http://auquestion.blogspot.in
Contact: contactlskgr@gmail.com
CS6712 GRID AND CLOUD COMPUTING LABORATORY LTPC0032
OBJECTIVES:
The student should be made to:

Be exposed to tool kits for grid and cloud environment.
Be familiar with developing web services/Applications in grid framework
Learn to run virtual machines of different configuration.
Learn to use Hadoop
LIST OF EXPERIMENTS:
GRID COMPUTING LAB
Use Globus Toolkit or equivalent and do the following:
1. Develop a new Web Service for Calculator.
2. Develop new OGSA-compliant Web Service.
3. Using Apache Axis develop a Grid Service.
4. Develop applications using Java or C/C++ Grid APIs
5. Develop secured applications using basic security mechanisms available in Globus Toolkit.
6. Develop a Grid portal, where user can submit a job and get the result. Implement it with
and without GRAM concept.
CLOUD COMPUTING LAB
Use Eucalyptus or Open Nebula or equivalent to set up the cloud and demonstrate.
1. Find procedure to run the virtual machine of different configuration. Check how many
virtual
machines can be utilized at particular time.
2. Find procedure to attach virtual block to the virtual machine and check whether it holds the
data even after the release of the virtual machine.
3. Install a C compiler in the virtual machine and execute a sample program.
4. Show the virtual machine migration based on the certain condition from one node
to the other.
6. Find procedure to set up the one node Hadoop cluster.
7. Mount the one node Hadoop cluster using FUSE.
8. Write a program to use the API's of Hadoop to interact with it.
9. Write a wordcount program to demonstrate the use of Map and Reduce tasks
TOTAL: 45 PERIODS
OUTCOMES:
At the end of the course, the student should be able to
Use the grid and cloud tool kits.
Design and implement applications on the Grid.
Design and Implement applications on the Cloud.
LIST OF EQUIPMENT FOR A BATCH OF 30 STUDENTS:
SOFTWARE:
Globus Toolkit or equivalent
Eucalyptus or Open Nebula or equivalent
HARDWARE
Standalone desktops 30 Nos
EX No:1 Basic Linux Comments
Aim:To Develop basic commands in Linux Environment.
General Commands
Using the utility Terminal / Console
The Linux console provides a way for the kernel and other processes to output text-
based messages to the user, and to receive text-based input from the user. In order to
invoke a terminal,
Press ALT + F2
Type gnome-terminal
Creating a new user
In order to create a new user from the terminal adduser command is to be used.
boss@localhost:/$ adduser hadoop
Adding user `hadoop' ...
Adding new group `hadoop' (1002) ...
Adding new user `hadoop' (1002) with group `hadoop' ...
Creating home directory `/home/hadoop' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:

Retype new UNIX password:
Enter the new value, or press ENTER for the default
Full Name []:

Room Number
[]: Work Phone
[]: Home Phone
[]: Other []:
Is the information correct? [Y/n]

Adding the new user to the “sudoers” List
SUDOER - sudo is a program for Unix-like computer operating systems that allows
users to run programs with the security privileges of another user (normally the superuser, or
root). Its name is a concatenation of “su” (substitute user) and “do”, or takes action. Since,
ours newly created users often need to perform certain privileged operations; these new users
should make be part of sudoers of the OS. In order to do this,
boss@localhost:/$ adduser hadoop sudo

Adding user `hadoop' to group `sudo' ...
Adding user hadoop to group sudo Done.
Beginning as new User
In order to do further operations as the newly created user, enter the following
at the terminal and you can check your identity,
boss@localhost:/$su - hadoop
hadoop@localhost:/$ whoami
hadoop
Check System's architecture and Kernel
BigData environment generally, is to be established on 64 bit architecture. In

order to support system's hardware architecture, kernel's support use the following at
the terminal,
hadoop@localhost:/$ cat /proc/cpuinfo | grep --color lm
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc
arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu
pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid
sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb
xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
Here lm indicate Long Mode i.e. 64 bit CPU architecture
hadoop@localhost:/$ uname -a
Linux localhost 3.8.0-38-generic #[YOUR_DISTRO] SMP Thu Mar 13 16:22:48

UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
By executing the above two commands, and your result is similar to the respective
screenshots, it indicates,
• Your System's hardware is 64 bit supported and
• It is installed with 64 bit OS.
Create a new folder for all your operations with Access Control
Since Linux is multi user OS, and each user is assigned with certain access control
mechanism, it is needed to ensure that the newly created user is able assigned with necessary
access control mechanism. In order to do this, here
1. Create a new directory for the newly created user – All the software will be installed
here
2. Move all the necessary software to this directory
3. Assign permissions to this directory and all it's child directories
1. Create a new directory for the newly created user – All the software will be installed
here
hadoop@localhost:/$ sudo mkdir /opt/bigdata

[sudo] password for hadoop: <-- Enter Password here
hadoop@localhost:/$ ls /opt/
jdk1.7.0_65 lost+found bigdata google hadoop@localhost:/$
2. Move all the necessary software to this directory
hadoop@localhost:/$cp <SOURCE_PATH>/<SRC_FILE_NAME><TARGET_PATH>
To unzip (Install) any archive file
hadoop@localhost:/$ sudo tar zxvf <ARVHIVE_FILE_NAME>
3. Assign permissions to this directory and all it's child directories First Check the default
permission
hadoop@localhost:/$ ls -l /opt/ total 44
drwxr-xr-x 2 root root 4096 Aug 7 12:20 bigdata drwxr-xr-x 3 root root 4096
Apr 7 13:25 google drwxr-xr-x 8 root root 4096 Aug 4 19:13 jdk1.7.0_65
drwxr-xr-x d – Directory
rwxr-xr-x – {User Group Others}{READ WRITE EXECUTE} root root – User and
Group is root
4. Assigning Permissions mean – Change User and Group to hadoop

hadoop@localhost:/$sudo chown -hR hadoop:hadoop /opt/bigdata
hadoop@localhost:/$
hadoop@localhost:/$ ls -l /opt total 44
drwxr-xr-x 2 test test 4096 Aug 7 12:20 bigdata
Software Installation
On Linux, mostly software installation is simply untar the archive if the software is
in archived format. If it is in binary / script format execute it. In order to execute such files
current user needs to have EXECUTE permission.
1. Installation from .bin File
Check if “X – (Execute)” flag is enabled for <BINARY / SCRIPT FILE>
hadoop@localhost:/$ls <BINARY / SCRIPT FILE>

-rw-r-xr-x 2 test test 4096 Aug 7 12:20 <BINARY /SCRIPT FILE>
If “X” flag is not set for the current user then
hadoop@localhost:/$chmod u+x <BINARY / SCRIPT FILE>
And check again
hadoop@localhost:/$ls <BINARY / SCRIPT FILE> -rwxr-xr-x 2 test test

4096 Aug 7 12:20 <BINARY /SCRIPT FILE>
To run / Execute the BINARY / SCRIPT File
hadoop@localhost:/$bash <BINARY / SCRIPT FILE>
Invoke 6.3 if any folder / file is not belonging the current user whenever necessary.
Now, in order to complete the installation process system wide / session wide variables are to
be update.
1. System Wide – Changes will be available to the entire system and to all the users (meant
for production environment)
▪ In order to set any variable to be available in System Wide, a file called profile is to
updated with appropriate values.
▪ This file is available in /etc/profile.
▪ Since this file belongs to the privileged user, it is to be edited with sudo command.
▪ Example : Installation of Java on Linux

▪ Introduction to Linux Environment
▪ Download Java (Either 32 / 64 bit)
▪ If it is in .tar.gz format untar as described in 6.2 to a directory where you wish all your software is to
be installed.
▪ If it is in .bin format untar as described in 7.1 to a directory where you wish all your software is to be
installed.
2. Session Wide – Changes will be available ONLY TO THE CURRENT SESSION (USUALLY A
TERMINAL / CONSOLE) AND TO THE CURRENT USER (meant for development and testing
environment)
1. This is done at the current working terminal using the command export.
1. To set HOME Variable
hadoop@localhost:/$ export JAVA_HOME=/opt/jdk1.7.0_65/
2. To Verify
hadoop@localhost:/$ echo $JAVA_HOME /opt/jdk1.7.0_65/
3. To Update PATH variable with JAVA_HOME
hadoop@localhost:/$ export PATH=$PATH: $JAVA_HOME/bin
4. To Verify
hadoop@localhost:/$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin :/usr/games:/opt/jdk1.7.0_65/bin
1. Open /etc/profile by
hadoop@localhost:/$ sudo vi /etc/profile
2. Create lines that are underlined at /etc/profile file

Enter into INSERT mode by pressing “i” from the keyboard.
Enter lines without making any space around the “=”
symbol. You may also from another Terminal ,
- cd /opt/<APPROPRIATE_SOFTWARE_DIRECTORY>
- pwd
- Copy the value from the output at this step from terminal and paste it in /etc/profile
#dafj/etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).
JAVA_HOME=/opt/<OUTPUT OF PWD COMMAND TO BE PASTED HERE>
if [ "$PS1" ]; then
if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ]; then
# The file bash.bashrc already sets the default PS1.
# PS1='\h:\w\$ '
if [ -f /etc/bash.bashrc ]; then
. /etc/bash.bashrc fi
else
if [ "ìd -u`" -eq 0 ]; then PS1='# '
else PS1='$ '
fi
fi
fi
# The default umask is now handled by pam_umask.
# See pam_umask(8) and /etc/login.defs.
if [ -d /etc/profile.d ]; then
for i in /etc/profile.d/*.sh; do if [ -r $i ]; then
. $i fi
done
unset i
fi
export JAVA_HOME PATH=$PATH:$JAVA_HOME/bin

File Management Commands
1 vi Editor / vim (vi improved) Editor – Text Editor
General Startup
To use vi: vi filename
To exit vi and save changes: ZZ or :wq
To exit vi without saving changes: :q!
To enter vi Insert mode: i
To enter vi command mode: [esc]
vi commands are applicable only in “vi command mode”.
2 To create directory
hadoop@localhost:~$ mkdir test
3 To display File Contents

hadoop@localhost:~$ cat <FILE_NAME>
4 To Copy / Rename File
To Copy a File
hadoop@localhost:~$ cp <SOURCE_FILE_NAME> <TARGET_PATH>
To Rename a File
hadoop@localhost:~$ mv <OLD_FILE_NAME> <NEW_FILE_NAME>
5 To delete a File
To delete a File
hadoop@localhost:~$ rm -rf <FILE_NAME>
Result:
Thus the above Linux Commands are executed successfully.
GRID COMPUTING LAB
EX No:2 DEVELOP A NEW WEB SERVICE FOR CALCULATOR
Aim:
To develop a new Web Service for Calculator using Globus toolkit.
Algorithm
1. Create new project

2. Select java Empty Web Application
3. Give a name to your project and click ok button
4. Go to Solution Explorer and right click at your project
5. Select Add New Item and select Web Service application
6. Give it name and click ok button
Program:
package gt3tutorial.core.first.impl;
import org.globus.ogsa.impl.ogsi.GridServiceImpl;
import gt3tutorial.core.first.Math.MathPortType;
import java.rmi.RemoteException;
public class MathImpl extends GridServiceImpl implements MathPortType
{
public MathImpl()
{
super("Simple Math Service");
}
public int add(int a, int b) throws
RemoteException
{
return a + b;
}
public int subtract(int a, int b) throws
RemoteException
{
return a - b;
}
public int multiply(int a, int b) throws
RemoteException
{
return a * b;
}
public float divide(int a, int b) throws
RemoteException
{
return a / b;
}}
OUTPUT:
Result:
Thus the Web Service for Calculator is developed using Globus Toolkit successfully.
EX No:3 OGSA-compliant Web Service
Aim:
To develop a OGSA-compliant Web Service using Globus Toolkit.
Algorithm:
The Global Grid Forum (GGF)’s Open Grid Services Architecture working group (OGSA-WG) has
recently completed two major work products. The OGSA Use Cases document describes a set of use
cases from a range of enterprise and scientific settings, intended as a source of requirements for
OGSA services. The OGSA Version 1 document collates requirements for an Open Grid Services
Architecture and identifies a large number of service interfaces that may be required to meet those
requirements.
The completion of these two documents leads to the question: what is the path by which OGSA
should now be further developed and defined? An answer to this question is important to a variety of
people.
 Many GGF participants have bought into the notion that OGSA can serve as an overarching
architectural framework for different GGF activities. They now want to understand what this
framework implies for their work.
 Developers and users want to know “what they can expect when” in terms of standards, so
that they can make plans for product developments and technology acquisitions.
 Arguably the credibility of OGSA, GGF, and Grid as a whole depends in part on a coherent
answer to this question.
These and other pressures encourage the view that we must move quickly to fill out the OGSA
definition and produce a set of normative specifications that define in great detail what it means to be
OGSA compliant.
However, before rushing into this task, we must also be aware of a number of other factors:
 The broad importance of Grid and the tight alignment of OGSA with Web services means
that further work on OGSA cannot proceed as a purely GGF activity, but must rather be
viewed as one (hopefully important) input to a larger process aimed at defining service-
oriented solutions to distributed computing problems.
 As in any standardization process, we need to be acutely sensitive to the dangers of premature
standardization, i.e., standardization without adequate experience and/or buy in from its
eventual users. These issues are particularly important in the case of OGSA, due to the
particularly large gap between our ambition and experience.
 While the OGSA design team has worked hard and considered a variety of use cases, the
team remains relatively small. It would seem likely that there are important perspectives that
have not yet been considered.
 The human resources available to work on OGSA activities are small, certainly far fewer than
are needed to do justice to the full spectrum of issues described in OGSA Version 1.
These considerations motivate this document, which seeks to clarify the role of OGSA and the steps
required to refine its definition by addressing the following three issues.
 With a view to identifying external constraints on OGSA, we review major relevant
standardization activities external to GGF. We discuss both past activities that have produced
specifications on which OGSA can build, and current and planned future activities that may
contribute to, and/or constrain, OGSA’s evolution.
 With a view to identifying factors that might help prioritize work on OGSA, we identify
dependencies among different OGSA interfaces and the interfaces that appear needed within
different deployment and application profiles.
 With a view to clarifying the process by which OGSA definition may proceed, we
recommend a process by which technical specifications developed within or outside GGF can
be identified as meeting OGSA requirements.
1 Open Grid Services Architecture: Goals and Status

A few brief words on goals and status, referring to other documents for details of course.
 Overarching goals.
 Documents produced.
2 The Standards Landscape

The roles of GGF, OASIS, W3C, DMTF, WS-I, IETF. Products that may be expected from each of
these groups in the coming 1-2 years, and their relevance to OGSA. Approaches that may be taken to
facilitate coordination.
The following are a few rough notes that may be relevant.
There are many places where bilateral coordination has worked, e.g. WSDM/CMM, but there is still
a lot of potential for conflict, e.g. Notification and Eventing.
Several low hanging pieces of fruit were identified: Notification, WMX, CML,
Naming, Reliable Messaging, Workflow (CAF/BEPL), Code footprint vs. minimal functionality,
common data model, policy models/languages (constraints and capabilities), provisioning and
deployment (Solution Installation Schema, CDDLM).
3 OGSA Definition Process

The process by which we see OGSA being further refined. Figure 1 may be relevant here.
A key point that Foster wants to see expressed here is that the “top down” and “bottom up” worlds
are to be coordinated as follows:
 OGSA-WG is concerned with defining requirements and overall architecture: the lighthouse
towards which others may steer.
 WGs within GGF or other bodies may (hopefully will!) be formed to develop specifications
that speak to requirements identified by OGSA-WG. However, OGSA-WG is not in the
business of endorsing such activities ahead of time.
 The steps by which a technical specification may become identified as “OGSA compliant”
remains to be clearly defined. I’d suggest that a key requirement should be identification as a
“recommendation” in the sense that there are two or more interoperable implementations.
Materials from OGSA F2F

2004.08.20
v2…vN Coarser granularity

Architecture Roadmap
OGSA-WG
(Ian, Hiro, Jay)
v1…vN-1
EMS Data Security …

Design Team (Ravi) (DaveB) (Frank)
Normative specs
Capability We are
here
EPS CSG …
External standards (e.g., WS specs) that

satisfy our requirements
Or identify what we’d like to see happen Finer granularity
And a detailed chart with stages of individual specs and when
we expect they would be available, e.g.,
* Roadmap discussion
- Identify dependencies and priorities we have received by the community
- Also to help people looking from the outside on what is OGSA and what is happening
- List of what is required and in what order
- Intended audience: need to identify in more detail.
Resolved
 Roadmap document per design team and a document that links them for OGSA. OGSA
document to also describe dependencies between the design teams' roadmaps.
 And a one page powerpoint slide that shows the overall map (and dates)
 Documents are internal to ogsa-wg; the one page powerpoint slide is public
- Agreement that it is ok to continue working on higher levels based on the expection of specs at
lower level that will define some needed functionality; as long as there is a description of what can
be expected of the lower spec (in terms of requirements).
- Create an ordering of the specs, and how easy it is to do things; which are the low hanging fruits)
Proposal: That roadmap should be independent of the dependencies but should include the
dependencies)
- (Essentially milestones; but need to do more work before that can be done.)
- Agreed that relations and dependencies with other groups need to be explicitly spelled out:
- What we will work with them or defer to them, priorities, etc. Document decisions and what they
imply and who is doing what.
- Need to communicate to them our requirements. This is not just liaison work; someone should
'own' that piece of work.
- E.g., a joint work/document or some more explicit understanding
- In the case of WSDM we have identitified:
- Naming/identity
- Events
- EMS overlap (or EMS usage of WSDM) --- as an initial 'application' area
- (Metadata --- query to data design team)
- (IP policy issues; when work is done in different standards bodies)
Action: Jay to talk with Ian and Hiro to form a core design team to work on dependencies at
architecture level (and roadmap)
Call 2.
Figure 2 is the result of a brainstorming session (& was put to the side on the next day).
Dave talked about the importance of the meeting that Hiro attended last week
- Inter-standards bodies meeting focusing on WS-management (as a common area of interest) to
discuss collaboration between standards bodies
- At the moment all collaboration is at the grassroots level only
- Trying to come up with ways to collaborate at a higher level
- The OGSA roadmap could be used as one important input to this discussion
- In particular we should express OGSA priorities and feedback
- OGSA v1 (currently in public comment) will hopefully act as milestone/catalyst to push this work
forward
- Currently only WSDM MUWS/MOWS is the other similar document
- (Jem trying to get OGSA v1 comments from HP; other people are encouraged to do the same
within their companies.)
- (Draft roadmap shouldn't have to go into public comment)
- Comments to Ian:
- Top level roadmap should (also) be in 2-dimensions: time and level of detail
- At the moment we have 2 levels of detail; need a coarser level of detail for the top level roadmap.
- At the OGSA level talk in terms of EMS/Data/Security (and maybe Naming)
- (Lowest level: specification and implementations)
- Dave also volunteered to get involved to this effort.
- Postpone further discussion to next call (next week)
Result
Thus the OGSA-compliant Web Service using Globus Toolkit is developed successfully.
Ex No:4 Using Apache Axis develop a Grid Service
Aim:
To develop a web Service using Apache Axis Webserver
Algorithum:
Create a new project
start by creating a new Java Project called ProvisionDirService.SelectFile > New > Project...and
select Java > Java Project from the selection wizard. Click Next and enter ProvisionDirService in the
Project Name textbox. Accept the remaining project creation defaults by clicking Finish.
Make the project a Tomcat project

The first thing we need to do is to make this project a "Tomcat Project." Doing so enables Tomcat to
run from .class files inside our project as soon as they are compiled by Eclipse (which appens every
time they are saved). Hence, minor changes to the service logic will be reflected in the running
service without having to regenerate or redeploy any GARs. Open the project properties
page shown in Figure 8 (select Properties from the project's pop-up menu), select the Tomcat page,
and check the "Is a Tomcat Project" checkbox.
Add project source to Tomcat source path

Now we need to add the project's source to the Tomcat source path. The last step allows Tomcat to
reload any updates to the implementation. This step allows the live debugger to pull up the fresh
source when debugging. Go to Window > Preferences and select the Tomcat > Source Path page
(shown in Figure 9). Select the checkbox next to our ProvisionDirService project.
Add project source to Tomcat source path

Now we need to add the project's source to the Tomcat source path. The last step allows Tomcat to
reload any updates to the implementation. This step allows the live debugger to pull up the fresh
source when debugging.
Go to
Window > Preferences
and select the
Tomcat > Source Path
Select the checkbox next to our
ProvisionDirService
project
Create the GT4 library

To create a user library from the GT4 library directory, use the
User Libraries...
button. Click
New...
in the User Libraries dialog (see Figure 11) and create a
Library called GT4 Library.
Finish the configuration
Result
Thus the Apache Axis webserver to develop a Grid Service using Globus Toolkit is developed
successfully.
EX No: 5 Develop secured applications using basic security in Globus
Aim:
To develop a Develop secured applications using basic security in Globus
Algorithm:
Authenticating Users Programmatically
Servlet 3.0 specifies the following methods of the HttpServletRequest interface that enable you to
authenticate users for a web application programmatically:
 authenticate
The authenticate method allows an application to instigate authentication of the request caller
by the container from within an unconstrained request context. A login dialog box displays
and collects the user's name and password for authentication purposes.
 login
The login method allows an application to collect username and password information as an
alternative to specifying form-based authentication in an application deployment descriptor.
 logout
The logout method is provided to allow an application to reset the caller identity of a request.
The following example code shows how to use the login and logout methods:
package test;
import java.io.IOException;
import java.io.PrintWriter;
import java.math.BigDecimal;
import javax.ejb.EJB;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@WebServlet(name="TutorialServlet", urlPatterns={"/TutorialServlet"})
public class TutorialServlet extends HttpServlet {
@EJB
private ConverterBean converterBean;
/**
* Processes requests for both HTTP <code>GET</code>
* and <code>POST</code> methods.
* @param request servlet request
* @param response servlet response
* @throws ServletException if a servlet-specific error occurs
* @throws IOException if an I/O error occurs
*/
protected void processRequest(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
out.println("<html>");
out.println("<head>");
out.println("<title>Servlet TutorialServlet</title>");
out.println("</head>");
out.println("<body>");
request.login("TutorialUser", "TutorialUser");
BigDecimal result = converterBean.dollarToYen(new BigDecimal("1.0"));
out.println("<h1>Servlet TutorialServlet result of dollarToYen= " + result + "</h1>");
out.println("</body>");
out.println("</html>");
} catch (Exception e) {
throw new ServletException(e);
} finally {
request.logout();
out.close();
}
}
}
This code sample shows how to use the authenticate method:
package com.sam.test;
import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
public class TestServlet extends HttpServlet {
protected void processRequest(HttpServletRequest request,

HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
request.authenticate(response);
out.println("Authenticate Successful");
} finally {
out.close();
}
}
Writing GSI enabled Web Services
Install Tomcat according to the instructions and check that it works.
Installing the software
Install Tomcat
 Install Tomcat according to the instructions and check that it works.
Deploy Axis on Tomcat
 Install Axis according to the instructions and check that it works.
Note that a bug in Tomcat means that any jars containing java.* or javax.* classes will not be
executed if there are in the webapps/ tree. Instead, copy the jars to Tomcat's common/lib directory. In
Axis alpha 3 this applies to axis.jar and wsdl4j.jar; in Axis beta 1 this applies to jaxrpc.jar and
wsdl4j.jar.
Install libraries to provide GSI support for Tomcat
 Copy cog.jar, cryptix.jar, iaik_ssl.jar, iaik_jce_full.jar, iaik_javax_crypto.jar to Tomcat's

common/lib directory.
 Check that log4j-core.jar and xerces.jar (or other XML parser) are in Tomcat's common/lib
directory.
 Copy gsicatalina.jar to Tomcat's server/lib directory.
Deploy GSI support in Tomcat
 Edit Tomcat's conf/server.xml
Add GSI Connector in <service> section:

<Connector className="org.apache.catalina.connector.http.HttpConnector"
port="8443" minProcessors="5" maxProcessors="75"
enableLookups="true" authenticate="true"
acceptCount="10" debug="1" scheme="httpg" secure="true">
<Factory className="org.globus.tomcat.catalina.net.GSIServerSocketFactory"
cert="/etc/grid-security/hostcert.pem"
key="/etc/grid-security/hostkey.pem"
cacertdir="/etc/grid-security/certificates"
gridmap="/etc/grid-security/gridmap-file"
debug="1"/>
</Connector>
If you are testing under a user account, make sure that the proxy or certificates and
keys are readable by Tomcat. For testing purposes you can use user proxies or
certificates instead of host certificates e.g.:
<Connector className="org.apache.catalina.connector.http.HttpConnector"
port="8443" minProcessors="5" maxProcessors="75"
enableLookups="true" authenticate="true"
acceptCount="10" debug="1" scheme="httpg" secure="true">
<Factory className="org.globus.tomcat.catalina.net.GSIServerSocketFactory"
proxy="/tmp/x509u_up_neilc"
debug="1"/>
</Connector>
If you do test using user proxies, make sure the proxy has not expired!
Add a GSI Valve in the <engine> section:
<Valve className="org.globus.tomcat.catalina.valves.CertificatesValve"
debug="1" />
Install libraries to provide GSI support for Axis
 Copy gsiaxis.jar to the WEB-INF/lib directory of your Axis installation under Tomcat.
Set your CLASSPATH correctly
 You should ensure that the following jars from the axis/lib directory are in your classpath:
o axis.jar
o clutil.jar
o commons-logging.jar
o jaxrpc.jar
o log4j-core.jar
o tt-bytecode.jar
o wsdl4j.jar
 You should also have these jars in your classpath:
o gsiaxis.jar
o cog.jar
o xerces.jar (or other XML parser)
Start the GSI enabled Tomcat/Axis server
 Start up Tomcat as normal
Check the logs in Tomcat's logs/ directory to ensure the server started correctly. In particular check
that:
 apache_log.YYYY-MM-DD.txt does not contain any GSI related error messages

 catalina.out contains messages saying "Welcome to the IAIK ... Library"
 catalina_log.YYYY-MM-DD.txt contains messages saying "HttpConnector[8443] Starting
background thread" and "HttpProcessor[8443][N] Starting background thread"
 localhost_log.YYYY-MM-DD.txt contains a message saying "WebappLoader[/axis]: Deploy
JAR /WEB-INF/lib/gsiaxis.jar"
Writing a GSI enabled Web Service
Implementing the service
The extensions made to Tomcat allow us to receive credentials through a transport-level security
mechanism. Tomcat exposes these credentials, and Axis makes them available as part of the
MessageContext.
Alpha 3 version
Let's assume we already have a web service called MyService with a single method, myMethod.
When a SOAP message request comes in over the GSI httpg transport, the Axis RPC despatcher will
look for the same method, but with an additional parameter: the MessageContext. So we can write a
new myMethod which takes an additional argument, the MessageContext.
This can be illustrated in the following example:
package org.globus.example;
import org.apache.axis.MessageContext;
import org.globus.axis.util.Util;
public class MyService {
// The "normal" method

public String myMethod(String arg) {
System.out.println("MyService: http request\n");
System.out.println("MyService: you sent " + arg);
return "Hello Web Services World!";
}
// Add a MessageContext argument to the normal method

public String myMethod(MessageContext ctx, String arg) {
System.out.println("MyService: httpg request\n");
System.out.println("GOT PROXY: " + Util.getCredentials(ctx));
}
Beta 1 version
In the Beta 1 version, you don't even need to write a different method. Instead the Message Context
is put on thread local store. This can be retrieved by calling MessageCOntext.getCurrentContext():
import org.apache.axis.MessageContext;
public class MyService {
// Beta 1 version
public String myMethod(String arg) {
System.out.println("MyService: httpg request\n");
// Retrieve the context from thread local

MessageContext ctx = MessageContext.getCurrentContext();
System.out.println("GOT PROXY: " + Util.getCredentials(ctx));
}
Part of the code provided by ANL in gsiaxis.jar is a utility package which includes the
getCredentials() method. This allows the service to extract the proxy credentials from the
MessageContext.
Deploying the service
Before the service can be used it must be made available. This is done by deploying the service. This
can be done in a number of ways:
1. Use the Axis AdminClient to deploy the MyService classes.

2. Add the following entry to the server-config.wsdd file in WEB-INF directory of axis on
Tomcat:
3. <service name="MyService" provider="java:RPC">
4. <parameter name="methodName" value="*"/>
5. <parameter name="className" value="org.globus.example.MyService"/>
6. </service>
Writing a GSI enabled Web Service client
As in the previous example, this is very similar to writing a normal web services client. There are
some additions required to use the new GSI over SSL transport:
 Deploy a httpg transport chain

 Use the Java CoG kit to load a Globus proxy
 Use setProperty() to set GSI specifics in the Axis "Property Bag":
o globus credentials (the proxy certificate)
o authorisation type
o GSI mode (SSL, no delegation, full delegation, limited delegation)
 Continue with the normal Axis SOAP service invokation:
o Set the target address for the service
o Provide the name of the method to be invoked
o Pass on any parameters required
o Set the type of the returned value
o Invoke the service
Here's an example which can be used to call the service you wrote in the last section:
import org.apache.axis.client.Call;
import org.apache.axis.client.Service;
import org.apache.axis.encoding.XMLType;
import org.apache.axis.configuration.SimpleProvider;
import org.apache.axis.utils.Options;
import org.apache.axis.AxisFault;
import org.apache.axis.SimpleTargetedChain;
import org.apache.axis.transport.http.HTTPSender;
import org.globus.axis.transport.GSIHTTPSender;
import org.globus.axis.transport.GSIHTTPTransport;
import org.globus.security.auth.SelfAuthorization;
import org.globus.security.GlobusProxy;
import javax.xml.rpc.namespace.QName;
import javax.xml.rpc.ParameterMode;
public class Client

{
public static void main(String [] args)
{
Util.registerTransport();
try {
Options options = new Options(args);
String endpointURL = options.getURL();

String textToSend;
// Parse the arguments for text to send

args = options.getRemainingArgs();
if ((args == null) || (args.length < 1)) {
textToSend = "";
} else {
textToSend = args[0];
}
// Set up transport handler chains and deploy

SimpleProvider provider = new SimpleProvider();
SimpleTargetedChain c = null;
c = new SimpleTargetedChain(new GSIHTTPSender());

provider.deployTransport("httpg", c);
c = new SimpleTargetedChain(new HTTPSender());

provider.deployTransport("http", c);
// Load globus proxy

GlobusProxy proxy = GlobusProxy.getDefaultUserProxy();
// Create a new service call

Service service = new Service(provider);
Call call = (Call) service.createCall();
// Set globus credentials

call.setProperty(GSIHTTPTransport.GSI_CREDENTIALS,
proxy);
// Set authorization type

call.setProperty(GSIHTTPTransport.GSI_AUTHORIZATION,
new SelfAuthorization(proxy));
// Set gsi mode

call.setProperty(GSIHTTPTransport.GSI_MODE,
GSIHTTPTransport.GSI_MODE_LIMITED_DELEG);
// Set the address of the service (from cmd line arguments)

call.setTargetEndpointAddress( new java.net.URL(endpointURL) );
// Set the name of the method we're invoking

call.setOperationName(new QName("MyService",
"myMethod"));
// Setup a target parameter

call.addParameter( "arg1",
XMLType.XSD_STRING,
ParameterMode.PARAM_MODE_IN);
// Set the return type

call.setReturnType( XMLType.XSD_STRING );
// Invoke the method, passing in the value of "arg1"

String ret = (String) call.invoke( new Object[] { textToSend } );
// Print out the returned value

System.out.println("MyService returned: " + ret);
} catch (Exception e) {
if ( e instanceof AxisFault ) {
((AxisFault)e).dump();
} else
e.printStackTrace();
}
}
}
You can invoke this client by running:

java org.globus.example.Client -l httpg://127.0.0.1:8443/axis/servlet/AxisServlet "Hello!"
assuming that you are running the client on the same machine (localhost) as the Axis/Tomcat server,
and that you've installed Axis in the webapps/axis directory of Tomcat.
If you examine logs/catalina.out you should see the messages from the Client received by the
service, as well as the proxy credentials.
Descriptions of the GSI extensions to Tomcat and Axis
1. Build a server-side SOAP service using Tomcat and Axis

2. Create connection stubs to support client-side use of the SOAP service
3. Build a custom client-side ClassLoader
4. Build the main client application
5. Build a trivial compute task designed to exercise the client ClassLoader
Build the SOAP service
The SOAP service I build in this article is the closest thing to a management layer that this
framework will have. The SOAP service provides a way for our grid computing application to pull
the classes it needs from the SOAP server. While my example service simply delivers a single
specific jar file, this service's actual production version would likely have access to multiple jar files
(each containing a different computing task), and it would contain additional logic to control which
JAR was delivered to whom.
The first step in providing the SOAP service is to set up the SOAP infrastructure. I chose Tomcat as
the servlet container/HTTP server because it is an open source project and proves to be extremely
reliable and easy to use. I chose Axis as the SOAP services provider because it too is open source,
supports an easy-to-use drag-and-drop service installer, and comes with a tool that creates SOAP
client-side stubs from WSDL (Web Services Description Language) files (a feature I exploit later).
After downloading and installing Tomcat 4.0.6 and Axis 1.0, I wrote the SOAP service class
GridConnection. This service fetches a known jar file, loads the file into a byte array, and returns the
byte array to the caller. The following code is the entire file GridConnection.java:
//// GridConnection.java
//
import java.util.*;
import java.io.* ;
public class GridConnection {
public byte[] getJarBytes () {
byte[] jarBytes = null ;
try {
FileInputStream fi = new FileInputStream("/Users/tkarre/MySquare/build/MySquare.jar");
jarBytes = new byte[fi.available()];
fi.read(jarBytes);
fi.close() ;
}
catch(Exception e) {}
return jarBytes ;
}
}
Result
Thus the above application basic security in Globus executed successfully.

EX.NO:6 Develop Grid API’s using C++
AIM:
To write a program for developing Grid API’s using C++.
Algorithm:
The Simple API for Grid Applications (SAGA) is a family of related standards specified by the
Open Grid Forum to define an application programming interface (API) for common distributed
computing functionality.
The SAGA specification for distributed computing originally consisted of a single document,
GFD.90, which was released in 2009.
The SAGA API does not strive to replace Globus or similar grid computing middleware systems, and
does not target middleware developers, but application developers with no background on grid
computing. Such developers typically wish to devote their time to their own goals and minimize the
time spent coding infrastructure functionality. The API insulates application developers from
middleware.
The specification of services, and the protocols to interact with them, is out of the scope of SAGA.
Rather, the API seeks to hide the detail of any service infrastructures that may or may not be used to
implement the functionality that the application developer needs. The API aligns, however, with all
middleware standards within Open Grid Forum
Implementations
Since the SAGA interface definitions are not bound to any specific programming language, several
implementations of the SAGA standards exist in different programming languages. Apart from the
implementation language, they differ from each other in their completeness in terms of standard
coverage, as well as in their support for distributed middleware.
SAGA C++
SAGA C++ was the first complete implementation of the SAGA Core specification, written in C++.
Currently the C++ implementation is not under active development.
Job submission
A typical task in a distributed application is to submit a job to a local or remote distributed resource
manager. SAGA provides a high-level API called the job package for this. The following two simple
examples show how the SAGA job package API can be used to submit a Message Passing Interface
(MPI) job to a remote Globus GRAM resource manager.
C++ Program:
#include <saga/saga.hpp>
int main (int argc, char** argv)
{
namespace sa = saga::attributes;
namespace sja = saga::job::attributes;
try
{
saga::job::description jd;
jd.set_attribute (sja::description_executable, "/home/user/hello-mpi");

jd.set_attribute (sja::description_output, "/home/user/hello.out");
jd.set_attribute (sja::description_error, "/home/user/hello.err");
// Declare this as an MPI-style job

jd.set_attribute (sja::description_spmd_variation, "mpi");
// Name of the queue we want to use

jd.set_attribute (sja::description_queue, "checkpt");
jd.set_attribute (sja::description_spmd_variation, "mpi");
// Number of processors to request
jd.set_attribute (sja::description_number_of_processes, "32");
saga::job::service js("gram://my.globus.host/jobmanager-pbs");
saga::job::job j = js.create_job(jd);
j.run()
}
catch(saga::exception const & e)
{
std::cerr << "SAGA exception caught: " << e.what() << std::endl;
}
}
Result
Thus the program for developing Grid API using C++ was executed successfully.
EX No:7 Develop a Grid portal, where user can submit a job and get the result.
Implement it with and without GRAM concept
AIM:
To Develop a Grid portal it with and without GRAM concept.
Algorithm:
The Grid Portal Development Kit leverages off existing Globus/Grid middleware infrastructure as
well as commodity web technology including Java Server Pages and servlets. Present the design and
architecture of GPDK as well as a discussion on the portal building capabilities of GPDK allowing
application developers to build customized portals more effectively by reusing common core services
provided by GPDK.
The Grid Portal Development Kit
The Grid Portal Development Kit is based on the standard n-tier architecture adopted by most web
application servers as shown in Figure 1. Tiers represent physical and administrative boundaries
between the end user and the web application server. The client tier is represented as tier 1 and
consists of the end-user’s workstation running a web browser. The only requirements placed upon
the client tier is a secure (SSL-capable) web browser that supports DHTML/Javascript for improved
interactivity, and cookies to allow session data to be transferred between the client and the web
application server.
Job Submission Both interactive and batch queue job submissions are enabled using either the GSI
enhance SSH client [] or using the Globus GRAM protocol to submit jobs to Globus gatekeepers
deployed on Grid resources. The major GPDK components used to submit jobs are the JobBean, the
JobSubmissionBean and the JobInfoBean. The JobBean provides a description of the job to be
submitted.
It includes methods for setting and returning values for the executable, additional arguments passed
to the executable, number of processors for parallel jobs, batch queue if submitting a batch mode and
more. The JobSubmissionBean is actually an abstract class that is subclassed by the
GramSubmissionBean in the case of submitting a job to a Globus gatekeeper or a
GSISSHSubmissionBean idf using the GSI enhanced SSH client. The GramSubmissionBean
capabilities are provided once again by the Java CoG library. Once a job has been succesfully
submitted, a JobInfoBean is created which contains a time stamp of when the job was submitted and
other useful information about the job, including a GRAM URL that can be used to query on the
status of the job.
File Transfer
Data access capabilities are provided by the GridFTP [] API implemented as part of the CoG toolkit
and encapsulated into core GPDK service beans. Capabilities include file transfer, including third-
party file transfer between GSI enabled FTP servers, as well as file browsing capabilities. The
FileTransferBean provides a generic file transfer API that is extended by the GSIFTPTransferBean
and the GSISCPTransferBean, an encapsulation of file transfer via the GSI enhanced scp coommand
tool. The GSIFTPServiceBean provides a session scoped bean that manages multiple FTP
connections to GSI enabled FTP servers. The GSIFTPServiceBean allows users to browse multiple
GSI FTP servers simultaneously and a separate thread monitors server timeouts. The
GSIFTPViewBean is an example view bean used by a JSP to display the results of browsing a
remote GSI FTP server.
Information Services
The Grid Forum Information Services working group has proposed the Grid Information Services
(GIS) architecture for deploying information services on the Grid and supported the Lightweight
Directory Access Protocol (LDAP) as the communication protocol used to query information
services. Information services on the Grid are useful for obtaining both static and dynamic
information on software and hardware resources. The Globus toolkit provides a Metacomputing
Directory Service (MDS), which is an implementation of a Grid Information Service using
OpenLDAP, an open source LDAP server. Although, the Java CoG toolkit provides support for
LDAP using the Java Naming and Directory Interface (JNDI), GPDK uses the open source
Netscape/Mozilla Directory SDK [] as it proved easier to use in practice and also provides support
for developing a connection pool for maintaining multiple connections to several Grid Information
service providers, thus eliminating the need for clients to reconnect during each query. However, this
model will need to be re-evaluated with the widespread deployment of the MDS-2 architecture which
includes GSI enhancements making it necessary for clients to reauthenticate to the MDS for each
query. GPDK provides an MDSQueryBean and MDSResultsBean for querying and formatting
results obtained from the MDS. Currently GDK supports querying the MDS for hardware
information such as CPU type, number of processors and other details as well as cpu load and queue
information that can be used by the user to make more effective job scheduling decisions.
Result:
Thus the above Grid portal application is executed successfully.

CLOUD COMPUTING LAB
EX.No 1 CREATION OF VIRTUAL MACHINES
AIM:
To Find procedure to run the virtual machine of different configuration. And also Check how many
virtual machines can be utilized at particular time.
ALGORITHM:
This experiment is to be performed through portal. Login into Open stack portal, in instances, create
virtual machines.
TO RUN VM
Step 1 : Under the Project Tab, Click Instances. In the right side screen Click Launch
Instance.
Step 2 : In the details, Give the instance name(eg. Instance1).
Step 3: Click Instance Boot Source list and choose 'Boot from image'
Step 4: Click Image name list and choose the image currently uploaded.
Step 5: Click launch.
Your VM will get created.
For further details refer : http://docs.openstack.org/user-guide/dashboard_launch_instances.html
Follow these steps to create a virtual machine using a virtual disk.
1. Start VMware Workstation.
Windows hosts: Double-click the VMware Workstation icon on your desktop or use the Start
menu (Start > Programs > VMware > VMware Workstation).
Linux hosts: In a terminal window, enter the command
vmware &
Note: On Linux hosts, the Workstation installer adds an entry to the Start menu for VMware
Workstation. However, this menu entry is located in different submenus, depending on your
Linux distribution. For example:
SUSE Linux 9.1 — Start > System > More Programs > VMware Workstation
Red Hat Enterprise Linux AS/WS Release 3 — Start > System Tools > More System Tools >
VMware Workstation
2. If this is the first time you have launched VMware Workstation and you did not enter the serial
number when you installed the product (an option available on a Windows host), you are
prompted to enter it. The serial number is on the registration card in your package or in the email
message confirming your electronic distribution order. Enter your serial number and click OK.
The serial number you enter is saved and VMware Workstation does not ask you for it again. For
your convenience, VMware Workstation automatically sends the serial number to the VMware
Web site when you use certain Web links built into the product (for example, Help > VMware
on the Web > Register Now! and Help > VMware on the Web > Request Support). This
allows us to direct you to the correct Web page to register and get support for your product.
3. Start the New Virtual Machine Wizard.
When you start VMware Workstation, you can open an existing virtual machine or create a new
one. Choose File > New > Virtual Machine to begin creating your virtual machine.
4. The New Virtual Machine Wizard presents you with a series of screens that you navigate using
the Next and Prev buttons at the bottom of each screen. At each screen, follow the instructions,
then click Next to proceed to the next screen.
5. Select the method you want to use for configuring your virtual machine.
If you select Typical, the wizard prompts you to specify or accept defaults for the following
choices:
The guest operating system

The virtual machine name and the location of the virtual machine's files
The network connection type
Whether to allocate all the space for a virtual disk at the time you create it
Whether to split a virtual disk into 2GB files
If you select Custom, you also can specify how to set up your disk — create a new virtual disk,
use an existing virtual disk or use a physical disk — and specify the settings needed for the type
of disk you select. There is also an option to create a legacy virtual disk for use in environments
with other VMware products.
Select Custom if you want to
Make a legacy virtual machine that is compatible with Workstation 4.x, GSX Server 3.x, ESX
Server 2.x and VMware ACE 1.x.
Store your virtual disk's files in a particular location
Use an IDE virtual disk for a guest operating system that would otherwise have a SCSI virtual disk
created by default
Use a physical disk rather than a virtual disk (for expert users)
Set memory options that are different from the defaults
Assign more than one virtual processor to the virtual machine
6. If you selected Typical as your configuration path, skip to step 7.
If you selected Custom as your configuration path, you may create a virtual machine that fully
supports all Workstation 5 features or a legacy virtual machine compatible with specific VMware
products.
This screen asks whether you want to create a Workstation 5 virtual machine or a legacy virtual
machine. See Legacy Virtual Disks for more information.
7. Select a guest operating system.

This screen asks which operating system you plan to install in the virtual machine. Select both an
operating system and a version. The New Virtual Machine Wizard uses this information to
Select appropriate default values, such as the amount of memory needed
Name files associated with the virtual machine
Adjust settings for optimal performance
Work around special behaviors and bugs within a guest operating system
If the operating system you plan to use is not listed, select Other for both guest operating system and
version.
Note: Workstation supports 64-bit guest operating systems only in Workstation versions 5.5 and
later, and only on host machines with supported processors. For the list of processors Workstation
supports for 64-bit guest operating systems, see Support for 64-Bit Guest Operating Systems.
Caution: Do not attempt to install a 64-bit operating system after selecting a 32-bit guest
operating system type here.
The remaining steps assume you plan to install a Windows XP Professional guest operating
system. You can find detailed installation notes for this and other guest operating systems in the
VMware Guest Operating System Installation Guide, available from the VMware Web site or
from the Help menu.
8. Select a name and folder for the virtual machine.
The name specified here is used if you add this virtual machine to the VMware Workstation
Favorites list. This name is also used as the name of the folder where the files associated with
this virtual machine are stored.
Each virtual machine should have its own folder. All associated files, such as the configuration
file and the disk file, are placed in this folder.
Windows hosts: On Windows 2000, Windows XP and Windows Server 2003, the default folder
for this Windows XP Professional virtual machine is C:\Documents and
Settings\<username>\My Documents\My Virtual Machines\Windows XP Professional. On
Windows NT, the default folder is C:\WINNT\Profiles\<username>\Personal\My Virtual
Machines\Windows XP Professional.
Linux hosts: The default location for this Windows XP Professional virtual machine is
<homedir>/vmware/winXPPro, where <homedir> is the home directory of the user who is
currently logged on.
Virtual machine performance may be slower if your virtual hard disk is on a network drive. For
best performance, be sure the virtual machine's folder is on a local drive. However, if other users
need to access this virtual machine, you should consider placing the virtual machine files in a
location that is accessible to them. For more information, see Sharing Virtual Machines with
Other Users.
9. Specify the number of processors for the virtual machine. The setting Two is supported only for
host machines with at least two logical processors.
Note: The following are all considered to have two logical processors:
A single-processor host with hyperthreading enabled
A single-processor host with a dual-core CPU
A multiprocessor host with two CPUs, neither of which are dual-core or have hyperthreading
enabled
If the host does not have at least two logical processors, assigning two virtual processors is
neither supported nor recommended: a warning message will appear. You can disregard this
message and assign two virtual processors to the virtual machine, but, once you have finished
creating the virtual machine, you will not be able to power it on unless you move it to a host
machine with at least two logical processors. For more about Workstation support for virtual
Symmetric Multiprocessing (SMP), see Using Two-Way Virtual Symmetric Multiprocessing
(Experimental).
10. If you selected Typical as your configuration path, skip to step 11.
If you selected Custom as your configuration path, you may adjust the memory settings or
accept the defaults, then click Next to continue.
In most cases, it is best to keep the default memory setting. If you plan to use the virtual machine
to run many applications or applications that need high amounts of memory, you may want to
use a higher memory setting. For more information, see Virtual Machine Memory Size.
Note: You cannot allocate more than 2GB of memory to a virtual machine if the virtual
machine's files are stored on a file system such as FAT32 that does not support files greater than
2GB.
11. Configure the networking capabilities of the virtual machine.
If your host computer is on a network and you have a separate IP address for your virtual
machine (or can get one automatically from a DHCP server), select Use bridged networking.
If you do not have a separate IP address for your virtual machine but you want to be able to
connect to the Internet, select Use network address translation (NAT). NAT allows you to
share files between the virtual machine and the host operating system.
For more details about VMware Workstation networking options, see Configuring a Virtual
Network.
12. If you selected Typical as your configuration path, click Finish and the wizard sets up the files
needed for your virtual machine.
If you selected Custom as your configuration path, continue with the steps below to configure a
disk for your virtual machine.
13. Select the type of SCSI adapter you want to use with the virtual machine.
An IDE and a SCSI adapter are installed in the virtual machine. The IDE adapter is always
ATAPI. You can choose a BusLogic or an LSI Logic SCSI adapter. The default for your guest
operating system is already selected. All guests except for Windows Server 2003, Red Hat
Enterprise Linux 3 and NetWare default to the BusLogic adapter.
The LSI Logic adapter has improved performance and works better with generic SCSI devices.
The LSI Logic adapter is also supported by ESX Server 2.0 and higher. Keep this in mind if you
plan to migrate the virtual machine to another VMware product.
Your choice of SCSI adapter does not affect your decision to make your virtual disk an IDE or
SCSI disk. However, some guest operating systems — such as Windows XP — do not include a
driver for the Buslogic or LSI Logic adapter. You must download the driver from the LSI Logic
Web site.
Note: Drivers for a Mylex (BusLogic) compatible host bus adapter are not obvious on the LSI
Logic Web site. Search the support area for the numeric string in the model number. For
example, search for "958" for BT/KT-958 drivers.
See the VMware Guest Operating System Installation Guide for details about the driver and the
guest operating system you plan to install in this virtual machine.
14. Select the disk you want to use with the virtual machine.
Select Create a new virtual disk.
Virtual disks are the best choice for most virtual machines. They are quick and easy to set up and
can be moved to new locations on the same host computer or to different host computers. By
default, virtual disks start as small files on the host computer's hard drive, then expand as needed
— up to the size you specify in the next step. The next step also allows you to allocate all the
disk space when the virtual disk is created, if you wish.
To use an existing operating system on a physical hard disk (a "raw" disk), read Configuring a
Dual-Boot Computer for Use with a Virtual Machine. To install your guest operating system
directly on an existing IDE disk partition, read the reference note Installing an Operating System
on a Physical Partition from a Virtual Machine.
Note: Physical disk configurations are recommended only for expert users.
Caution: If you are using a Windows Server 2003, Windows XP or Windows 2000 host, see Do
Not Use Windows 2000, Windows XP and Windows Server 2003 Dynamic Disks as Physical
Disks.
To install the guest operating system on a IDE physical disk, select Existing IDE Disk
Partition. To use a SCSI physical disk, add it to the virtual machine later with the virtual
machine settings editor. Booting from a SCSI physical disk is not supported. For a discussion of
some of the issues involved in using a SCSI physical disk, see Configuring Dual- or Multiple-
Boot SCSI Systems to Run with VMware Workstation on a Linux Host.
15. Select whether to create an IDE or SCSI disk.
The wizard recommends the best choice based on the guest operating system you selected. All
Linux distributions you can select in the wizard use SCSI virtual disks by default, as do
Windows NT, Windows 2000, and Windows Vista. All Windows operating systems except
Windows NT, Windows 2000, and Windows Vista use IDE virtual disks by default; NetWare,
FreeBSD, MS-DOS and other guests default to IDE virtual disks.
16. Specify the capacity of the virtual disk.
Enter the size of the virtual disk that you wish to create.
You can set a size between 0.1GB and 950 GB for a SCSI virtual disk.
The option Allocate all disk space now gives somewhat better performance for your virtual
machine. If you do not select Allocate all disk space now, the virtual disk's files start small and
grow as needed, but they can never grow larger than the size you set here.
Note: Allocate all disk space now is a time-consuming operation that cannot be cancelled, and
requires as much physical disk space as you specify for the virtual disk.
Select the option Split disk into 2GB files if your virtual disk is stored on a file system that does
not support files larger than 2GB.
17. Specify the location of the virtual disk's files.

18. Click Finish.
The wizard sets up the files needed for your virtual machine.
https://www.vmware.com/support/ws55/doc/ws_newguest_setup_simple_steps.html
OUTPUT:
Result:
Thus the above application is Login into Open stack portal, in instances, create virtual
machines executed successfully.
CREATION OF VIRTUAL BLOCK TO VIRTUAL MACHINE
EX No: 2
Aim:
To Find procedure to attach virtual block to the virtual machine and check whether it holds the data
even after the release of the virtual machine.
Algorithum
Objective: Installation and Configuration of Justcloud.
Requirement: Justcloud exe File
THEORY:
Professional Cloud Storage from JustCloud is Simple, Fast and Secure.

Just Cloud will automatically backup the documents, photos, music and videos
stored on your computer, to the cloud so you are never without files again.
Installation :
1. Download Software this

linkhttp://www.justcloud.com/download/
2. By following these steps you will download and install the JustCloud software application on
this computer. This software will automatically start backing up files from your computer and
saving them securely in an online cloud user account. Your free account gives you 15MB
storage space or 50 files for 14 days. Once installed a sync folder will be added to your
desktop for you to easily drag and drop files you wish to backup.
Result:
The above application allots virtual block and virtual machine is successfully developed.
EX No:3 Install a C compiler in the virtual machine and execute a sample program.
Aim :
To develop a C program using compiler in the virtual machine.
Algorithm:
Let's start with a simple "Hello Cloud9" example.
1. From your dashboard, click 'create new workspace' and then select 'create new workspace'.
2. Enter a catchy workspace name, visibility: open (proud to share your creations), hosting:
hosted and choose a 'custom' workspace type. Click 'create'.
3. The workspace is being prepared and when done, select the project in the dashboard and click
'start editing'.
4. The workspace is opened, right click the project tree and select 'new file'. Name it
'helloCloud9.cc'.
5. Open the file by double clicking it in the file tree. Copy / paste the following code in the file:
int main() {
cout << "Hello world\n";
6. Compile the code using:
g++ helloCloud9.cc -o helloCloud9
7. Run the file by typing:
./helloWorld
Result:
Thus the above C program was developed using in C compiler in the virtual machine application
executed successfully.
EX No: 4 VIRTUAL MACHINE MIGRATION
AIM:
To develop a virtual machine migration based on the certain condition from one node to the other.
Algorithm
1. On the first node, obtain a key pair (public key and private key). Use the root key that is in
the /root/.ssh/id_rsa and /root/.ssh/id_ras.pub directories or generate a new key pair.
2. Run setenforce 0 to put SELinux into permissive mode.
3. Enable login abilities for the nova user:
4. # usermod -s /bin/bash nova
Switch to the nova account.
# su nova
5. As root, create the folder that is needed by SSH and place the private key that you obtained in
step 1 into this folder:
6. mkdir -p /var/lib/nova/.ssh
7. cp <private key> /var/lib/nova/.ssh/id_rsa
8. echo 'StrictHostKeyChecking no' >> /var/lib/nova/.ssh/config
9. chmod 600 /var/lib/nova/.ssh/id_rsa /var/lib/nova/.ssh/authorized_keys
10. Repeat steps 2-4 on each node.
Note
The nodes must share the same key pair, so do not generate a new key pair for any
subsequent nodes.
11. From the first node, where you created the SSH key, run:
12. ssh-copy-id -i <pub key> nova@remote-host
This command installs your public key in a remote machine’s authorized_keys folder.
13. Ensure that the nova user can now log in to each node without using a password:
14. # su nova
15. $ ssh *computeNodeAddress*
16. $ exit
17. As root on each node, restart both libvirt and the Compute services:
18. # systemctl restart libvirtd.service
19. # systemctl restart openstack-nova-compute.service
1. To list the VMs you want to migrate, run:

$ nova list
2. After selecting a VM from the list, run this command where VM_ID is set to the ID in the list
returned in the previous step:
$ nova show VM_ID
3. Use the nova migrate command.
$ nova migrate VM_ID
4. To migrate an instance and watch the status, use this example script:
#!/bin/bash
# Provide usage
usage() {
echo "Usage: $0 VM_ID"
exit 1
}
[[ $# -eq 0 ]] && usage
# Migrate the VM to an alternate hypervisor

echo -n "Migrating instance to alternate host"
VM_ID=$1
nova migrate $VM_ID
VM_OUTPUT=`nova show $VM_ID`
VM_STATUS=ècho "$VM_OUTPUT" | grep status | awk '{print $4}'`
while [[ "$VM_STATUS" != "VERIFY_RESIZE" ]]; do
echo -n "."
sleep 2
VM_OUTPUT=`nova show $VM_ID`
VM_STATUS=ècho "$VM_OUTPUT" | grep status | awk '{print $4}'`
done
nova resize-confirm $VM_ID
echo " instance migrated and resized."
echo;
# Show the details for the VM

echo "Updated instance details:"
nova show $VM_ID
# Pause to allow users to examine VM details

read -p "Pausing, press <enter> to exit."
Result:
Thus the above program for virtual machine migration using Linux commands are executed
successfully.
EXNo:5 Find procedure to install storage controller and interact with it.
Aim:
To develop a storage controller and interact with it.
Algorithum:
Create a container
1. Log in to the dashboard.

2. Select the appropriate project from the drop down menu at the top left.
3. On the Project tab, open the Object Store tab and click Containers category.
4. Click Create Container.
5. In the Create Container dialog box, enter a name for the container, and then click Create
Container.
You have successfully created a container.
Upload an object

4. Select the container in which you want to store your object.
5. Click Upload Object.
The Upload Object To Container: <name> dialog box appears. <name> is the name of the
container to which you are uploading the object.
6. Enter a name for the object.

7. Browse to and select the file that you want to upload.
You have successfully uploaded an object to the container.
Manage an object
To edit an object

5. Click the menu button and choose Edit from the dropdown list.
The Edit Object dialog box is displayed.
6. Browse to and select the file that you want to upload.

7. Click Update Object.
To create a metadata-only object without a file
You can create a new object in container without a file available and can upload the file later when it
is ready. This temporary object acts a place-holder for a new object, and enables the user to share
object metadata and URL info in advance.

The Upload Object To Container: <name> dialog box is displayed.
<name> is the name of the container to which you are uploading the object.
6. Enter a name for the object.

7. Click Update Object.
To create a pseudo-folder
Pseudo-folders are similar to folders in your desktop operating system. They are virtual collections
defined by a common prefix on the object’s name.

5. Click Create Pseudo-folder.
The Create Pseudo-Folder in Container <name> dialog box is displayed. <name> is the name
of the container to which you are uploading the object.
6. Enter a name for the pseudo-folder.
A slash (/) character is used as the delimiter for pseudo-folders in Object Storage.
7. Click Create.
Result:
Thus the above programme is executed with storage controller.
EX NO:6 Find procedure to set up the one node Hadoop cluster.
AIM
To develop a procedure to set up the one node Hadoop cluster.
Algorithm:
Hadoop installation
Now Download Hadoop from the official Apache, preferably a stable release version
of Hadoop 2.7.x and extract the contents of the Hadoop package to a location of your choice.
We chose location as “/opt/”
Step 1: Download the tar.gz file of latest version Hadoop ( hadoop-2.7.x) from the official site .
Step 2: Extract(untar) the downloaded file from this commands to /opt/bigdata
root@solaiv[]# cd /opt
root@solaiv[/opt]# sudo tar xvpzf /home/itadmin/Downloads/hadoop-2.7.0.tar.gz
root@solaiv[/opt]# cd hadoop-2.7.0/
Like java, update Hadop environment variable in /etc/profile
boss@solaiv[]# sudo vi /etc/profile

#--insert HADOOP_PREFIX
HADOOP_PREFIX=/opt/hadoop-
2.7.0
#--in PATH variable just append at the end of the line

PATH=$PATH:$HADOOP_PREFIX/bin
#--Append HADOOP_PREFIX at end of the export statement

export PATH JAVA_HOME HADOOP_PREFIX
save the file using by pressing “Esc” key followed by :wq!
Step 3: Source the /etc/profile

boss@solaiv[]# source /etc/profile
Verify Hadoop installation
boss@solaiv[]# cd $HADOOP_PREFIX
boss@solaiv[]# bin/hadoop version
3.1) Modify the Hadoop Configuration Files

In this section, we will configure the directory where Hadoop will store its configuration
files, the network ports it listens to, etc. Our setup will use Hadoop Distributed File
System,(HDFS), even though we are using only a single local machine.
Add the following properties in the various hadoop configuration files which is available
under $HADOOP_PREFIX/etc/hadoop/
core-site.xml, hdfs-site.xml, mapred-site.xml & yarn-site.xml
Update Java, hadoop path to the Hadoop environment file
boss@solaiv[]# cd $HADOOP_PREFIX/etc/hadoop
boss@solaiv[]# vi hadoop-env.sh
Paste following line at beginning of the file

export JAVA_HOME=/usr/local/jdk1.8.0_05 export HADOOP_PREFIX=/opt/hadoop-2.7.0
Modify the core-site.xml
boss@solaiv[]# cd $HADOOP_PREFIX/etc/hadoop boss@solaiv[]# vi core-site.xml

Paste following between <configuration> tags
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Modify the hdfs-site.xml
boss@solaiv[]# vi hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
/configuration>
YARN configuration - Single Node modify the mapred-site.xml

boss@solaiv[]# cp mapred-site.xml.template mapred-site.xml
boss@solaiv[]# vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Modify yarn-site.xml
boss@solaiv[]# vi yarn-site.xml
<configuration>
<property> <name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
Formatting the HDFS file-system via the NameNode
The first step to starting up your Hadoop installation is formatting the Hadoop files system
which is implemented on top of the local file system of our “cluster” which includes only our
local machine. We need to do this the first time you set up a Hadoop cluster.
Do not format a running Hadoop file system as you will lose all the data currently in the
cluster (in HDFS)
root@solaiv[]# cd $HADOOP_PREFIX
root@solaiv[]# bin/hadoop namenode -format
Start NameNode daemon and DataNode daemon: (port 50070)
root@solaiv[]# sbin/start-dfs.sh
To know the running daemons jut type jps or /usr/local/jdk1.8.0_05/bin/jps Start

ResourceManager daemon and NodeManager daemon: (port 8088)
root@solaiv[]# sbin/start-yarn.sh
To stop the running process
root@solaiv[]# sbin/stop-dfs.sh
To know the running daemons jut type jps or /usr/local/jdk1.8.0_05/bin/jps Start

ResourceManager daemon and NodeManager daemon: (port 8088)
root@solaiv[]# sbin/stop-yarn.sh
Result:
Thus the above application for setting up the one Hadoop cluster is executed successfully.
EX.NO:7 Mount the one node Hadoop cluster using FUSE.
AIM:
To develop a Mount the one node Hadoop cluster using FUSE.
Algorithm:
1. FUSE (Filesystem in Userspace) enables you to write a normal user application as a
bridge for a traditional filesystem interface.
2. The hadoop-hdfs-fuse package enables you to use your HDFS cluster as if it were a
traditional filesystem on Linux. It is assumed that you have a working HDFS cluster
and know the hostname and port that your NameNode exposes.
To install fuse-dfs on Ubuntu systems:

sudo apt-get install hadoop-hdfs-fuse
To set up and test your mount point:
mkdir -p <mount_point>
hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>
You can now run operations as if they are on your mount point. Press Ctrl+C to end the fuse-
dfs program, and umount the partition if it is still mounted.
Note:
-fuse-dfs uses the HADOOP_CONF_DIR
configured at the time the mount command is invoked.
-fuse-dfs may exit
immediately because ld.so can't find libjvm.so. To work around this issue, add
/usr/java/latest/jre/lib/amd64/server to the LD_LIBRARY_PATH.
To clean up your test:

$ umount <mount_point>
You can now add a permanent HDFS mount which persists through reboots. To add a
system mount:
1. Open /etc/fstab and add lines to the bottom similar to these:
hadoop-fuse-dfs#dfs://<name_node_hostname>:<namenode_port> <mount_point> fuse

allow_other,usetrash,rw 2 0
For example:
hadoop-fuse-dfs#dfs://localhost:8020 /mnt/hdfs fuse allow_other,usetrash,rw 2 0
2. Test to make sure everything is working properly:
$ mount <mount_point>
Your system is now configured to allow you to use the ls command and use that mount point
as if it were a normal system disk.
Result:
Thus the above application is executed successfully.

EX No 8. Write a program to use the API's of Hadoop to interact with it.
AIM:
To develop a API of hadoop
Algorithm
Modify the Hadoop Configuration Files

In this section, we will configure the directory where Hadoop will store its
configuration files, the network ports it listens to, etc. Our setup will use Hadoop
Distributed File System,(HDFS), even though we are using only a single local
machine.
Add the following properties in the various hadoop configuration files which is
available under $HADOOP_PREFIX/etc/hadoop/
core-site.xml, hdfs-site.xml, mapred-site.xml & yarn-site.xml
Update Java, hadoop path to the Hadoop environment file
boss@solaiv[]# cd $HADOOP_PREFIX/etc/hadoop
boss@solaiv[]# vi hadoop-env.sh
Paste following line at beginning of the file

export JAVA_HOME=/usr/local/jdk1.8.0_05 export HADOOP_PREFIX=/opt/hadoop-2.7.0
Modify the core-site.xml
boss@solaiv[]# cd $HADOOP_PREFIX/etc/hadoop boss@solaiv[]# vi core-site.xml

<configuration>
<property>
</property>
</configuration>
Modify the hdfs-site.xml
boss@solaiv[]# vi hdfs-site.xml
<configuration>
<property>
<value>1</value>
</property>
</configuration>
YARN configuration - Single Node modify the mapred-site.xml
boss@solaiv[]# cp mapred-site.xml.template mapred-site.xml

boss@solaiv[]# vi mapred-site.xml
<configuration>
<property>
<value>yarn</value>
</property>
</configuration>
Modfiy yarn-site.xml
boss@solaiv[]# vi yarn-site.xml
<configuration>
<property> <name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</v
alue> </property>
</configuration>
Formatting the HDFS file-system via the NameNode
The first step to starting up your Hadoop installation is formatting the Hadoop files
system which is implemented on top of the local file system of our “cluster” which
includes only our local machine. We need to do this the first time you set up a Hadoop
cluster.
Do not format a running Hadoop file system as you will lose all the data currently in
the cluster (in HDFS)
Result:
Ex No: 9. Write a word count program to demonstrate the use of Map and Reduce tasks
Aim:
To develop a word count program to demonstrate the use of Map
Algorithm
Required software for Linux includes
1. Java™ must be installed. Recommended Java versions are described at

HadoopJavaVersions.
2. ssh must be installed and sshd must be running to use the Hadoop scripts that manage
remote Hadoop daemons.
3. f your cluster doesn’t have the requisite software you will need to install it.
4. For example on Ubuntu Linux:
5. $ sudo apt-get install ssh
6. $ sudo apt-get install rsync
7. Unpack the downloaded Hadoop distribution. In the distribution, edit the file
etc/hadoop/hadoop-env.sh to define some parameters as follows:
8. # set to the root of your Java installation
9. export JAVA_HOME=/usr/java/latest
10. Try the following command:
11. $ bin/hadoop
12. e following example copies the unpacked conf directory to use as input and then finds
and displays every match of the given regular expression. Output is written to the
given output directory.
13. $ mkdir input
14. $ cp etc/hadoop/*.xml input
15. $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar
grep input output 'dfs[a-z.]+'
16. $ cat output/*
Program:
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,

Context context
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
Input/output
$ bin/hadoop fs -ls /user/joe/wordcount/input/ /user/joe/wordcount/input/file01

/user/joe/wordcount/input/file02
$ bin/hadoop fs -cat /user/joe/wordcount/input/file01

Hello World Bye World
$ bin/hadoop fs -cat /user/joe/wordcount/input/file02
Hello Hadoop Goodbye Hadoop
Run the application:

$ bin/hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output
Result:
Website: https://larshansolutions.blogspot.in
You like my Website Please Share it and Following
Thanking you Welcome

EX. NO. : 1 INSTALLATION OF VIRTUAL MACHINE
Date:
Aim:
To find procedure to run the virtual machine of different configuration. Check how many
virtual machines can be utilized at particular time.
KVM:
In computing, virtualization refers to the act of creating a virtual (rather than actual) version
of something, including virtual computer hardware platforms, operating systems, storage devices,
and computer network resources. Kernel-based Virtual Machine (KVM) is a virtualization
infrastructure for the Linux kernel that turns it into a hypervisor.
Steps for KVM Installation:
1. To run KVM, you need a processor that supports hardware virtualization. So check that
your CPU supports hardware virtualization
egrep -c '(vmx|svm)' /proc/cpuinfo
If 0 - CPU doesn't support hardware virtualization

1 - CPU support hardware virtualization
2. To see if your processor is 64-bit,

egrep -c ' lm ' /proc/cpuinfo
If 0 - printed, it means that your CPU is not 64-bit.

1- is 64-bit.
3. $ ls /lib/modules/3.16.0-3 generic/kernel/arch/x86/kvm/kvm
kvm-amd.ko kvm-intel.ko kvm.ko
4. $ ls /dev/kvm
/dev/kvm
5. Install Necessary Packages using the following commands,
1
qemu-kvm
libvirt-bin
bridge-utils
virt-manager
qemu-system
6. Creating VM’s
virt-install --connect qemu:///system -n hardy -r 512 -f hardy1.qcow2 -s 12 -c ubuntu-
14.04.2-server-amd64.iso --vnc --noautoconsole --os-type linux --os-variant ubuntuHardy
Output:
1. New virtual machine is created using KVM:
2
Conclusion:
Thus the virtual machine of different configuration is created successfully.
3
EX. NO. : 2 INSTALLATION OF C COMPILER
Date:
Aim:
To find the procedure to install a C Compiler in the Virtual Machine and execute a C
program.
Steps:
1. To install the C Compiler in the guest os, install the following package.
gcc
2. Write a sample program using gedit/vim editor.
3. Compile the C program using the compiler installed.
gcc sample_c_program.c –o output
4. Run the object file and get the output.
Conclusion:
Thus the C Compiler is installed successfully and executed a sample C program.
4
EX. NO. : 3 INSTALLATION OF VIRTUAL MACHINE
Date:
Aim:
To find procedure to install storage controller and interact with it.
KVM:
In computing, virtualization refers to the act of creating a virtual (rather than actual) version
of something, including virtual computer hardware platforms, operating systems, storage devices,
and computer network resources. Kernel-based Virtual Machine (KVM) is a virtualization
infrastructure for the Linux kernel that turns it into a hypervisor.
Steps for KVM Installation:
1. To run KVM, you need a processor that supports hardware virtualization. So check that your
CPU supports hardware virtualization
egrep -c '(vmx|svm)' /proc/cpuinfo
If 0 - CPU doesn't support hardware virtualization

1 - CPU support hardware virtualization
2. To see if your processor is 64-bit,

egrep -c ' lm ' /proc/cpuinfo
If 0 - printed, it means that your CPU is not 64-bit.

2- is 64-bit.
3. $ ls /lib/modules/3.16.0-3 generic/kernel/arch/x86/kvm/kvm
kvm-amd.ko kvm-intel.ko kvm.ko
4. $ ls /dev/kvm
/dev/kvm
5. Install Necessary Packages using the following commands,
qemu-kvm
libvirt-bin
bridge-utils
virt-manager
qemu-system
5
6. Creating VM’s
virt-install --connect qemu:///system -n hardy -r 512 -f hardy1.qcow2 -s 12 -c ubuntu-
14.04.2-server-amd64.iso --vnc --noautoconsole --os-type linux --os-variant ubuntuHardy
Output:
2. New virtual machine is created using KVM:
6
Conclusion:
Thus the storage controller is inatlled successfully in virtual machine.
7
EX. NO. : 4 VIRTUAL MACHINE MIGRATION
Date:
Aim:
To show the virtual machine migration based on the certain condition from one node to the
other.
Steps to Migrate the Virtual Machine:
1. Open virt-manager
2. Connect to the target host physical machine
Connect to the target host physical machine by clicking on the File menu, then click Add
Connection.
8
3. Add connection
The Add Connection window appears.
Enter the following details:
Hypervisor: Select QEMU/KVM.
Method: Select the connection method.
Username: Enter the username for the remote host physical machine.
Hostname: Enter the hostname/IP address for the remote host physical machine.
Click the Connect button. An SSH connection is used in this example, so the specified user's
password must be entered in the next step.
9
4. Migrate guest virtual machines
Open the list of guests inside the source host physical machine (click the small triangle on the left
of the host name) and right click on the guest that is to be migrated (guest1-rhel6-64 in this
example) and click Migrate.
In the New Host field, use the drop-down list to select the host physical machine you wish to
migrate the guest virtual machine to and click Migrate.
10
A progress window will appear.
virt-manager now displays the newly migrated guest virtual machine running in the destination
host. The guest virtual machine that was running in the source host physical machine is now listed
in the Shutoff state.
Conclusion:
Thus the virtual machine is migrated from one node to another node successfully.
11
EX. NO.: 5 VIRTUAL BLOCK ATTACHMENT
Date:
Aim:
To find the procedure to attach virtual block to the virtual machine and check whether it
holds the data even after the release of the virtual machine.
Steps:
1. Make sure that you have shut down your virtual machine.
2. Select your VM and then click Edit settings.
3. Select the Hardware tab and then click Add.
4. Select Hard Disk from the list of device types and then click Next.
5. Choose Create a new virtual disk.
6. Specify the disk size.
7. Choose Thick Provision Lazy Zeroed.
8. Choose Specify a datastore or datastore cluster: and then click Browse
9. Select your datastore from the provided list and then click OK.
10. Click Next to accept the default advanced options. (By default, the new disk will be included
in full VM snapshots. To keep them consistent, we recommend that you leave the Independent
option unselected.)
11. Click Finish to proceed with adding the disk.
12. Click OK once the new hard disk has been added. This may take some time, depending on
how much storage you're adding.
Conclusion:
Thus the new virtual block is successfully added to existing virtual machine.
12
EX. NO.: 6 HADOOP SETUP AND INSTALLATION
Date:
Aim:
To find the procedure to set up the one node Hadoop cluster.
HADOOP:
Apache Hadoop is an open-source software framework for storage and large-scale
processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level
project being built and used by a global community of contributors and users. It is licensed under
the Apache License 2.0.
The Apache Hadoop framework is composed of the following modules:
 Hadoop Common – contains libraries and utilities needed by other Hadoop modules
 Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on
commodity machines, providing very high aggregate bandwidth across the cluster.
 Hadoop YARN – a resource-management platform responsible for managing compute
resources in clusters and using them for scheduling of users' applications.
 Hadoop MapReduce – a programming model for large scale data processing.
Installation Steps:
1. Install Java
Check the Java version in the system.
“java –version”
2. Open the “/etc/profile” file and Add the following line as per the version to set a
environment for Java
$ sudo vi /etc/profile
#--insert JAVA_HOME
JAVA_HOME=/opt/jdk1.8.0_05
PATH=$PATH:$JAVA_HOME/bin
#--Append JAVA_HOME at end of the export statement
export PATH JAVA_HOME
$ source /etc/profile
3. Install SSH using the command
13
$ sudo apt-get install openssh-server openssh-client
4. Generate an SSH key for the user. Then Enable password-less SSH access
$ ssh localhost
$ ssh-keygen
$ exit
5. Hadoop installation:
 Download the tar.gz file of latest version Hadoop ( hadoop-2.7.x) from the official
site .
 Extract(untar) the downloaded file from commands
$ sudo tar zxvf hadoop-2.7.0.tar.gz
$ cd hadoop-2.7.0/
6. Update Hadoop environment variable in /etc/profile
$ sudo vi /etc/profile
#--insert HADOOP_PREFIX
HADOOP_PREFIX=/opt/hadoop-2.7.0
PATH=$PATH:$HADOOP_PREFIX/bin
#--Append HADOOP_PREFIX at end of the export statement
export PATH JAVA_HOME HADOOP_PREFIX
Source the /etc/profile
$ source /etc/profile
Verify Hadoop installation
$ cd $HADOOP_PREFIX
$ bin/hadoop version
7. Update Java, hadoop path to the Hadoop environment file
$HADOOP_PREFIX/etc/Hadoop
$ vi core-site.xml
Paste following between <configuration> tags in core-site.xml
<property>
</property>
14
$ vi hdfs-site.xml
Paste following between <configuration> tags in hdfs-site.xml
<property>
<value>1</value>
</property>
$ cp mapred-site.xml.template mapred-site.xml
$ vi mapred-site.xml
Paste following between <configuration> tags in mapred-site.xml
<property>
<value>yarn</value>
</property>
$vi yarn-site.xml
Paste following between <configuration> tags in yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
8. Formatting the HDFS file-system via the NameNode
$bin/hadoop namenode –format
9. Start NameNode daemon and DataNode daemon: (port 50070)
$ sbin/start-dfs.sh
10. Start ResourceManager daemon and NodeManager daemon: (port 8088)

$ sbin/start-yarn.sh
11. To stop the running process

$ sbin/stop-dfs.sh
$ sbin/stop-yarn.sh
15
Output:
Hadoop installation:
Create the HDFS directories:
Conclusion:
Thus the one node Hadoop cluster is installed successfully.
16
EX. NO.: 7 HADOOP CLUSTER USING FUSE
Date:
Aim:
To mount the one node Hadoop cluster using FUSE.
Steps:
Download the cdh3 repository from the internet.
$ wget http://archive.cloudera.com/one-click-install/maverick/cdh3-repository_1.0_all.deb
Add the cdh3 repository to default system repository.
$ sudo dpkg -i cdh3-repository_1.0_all.deb
Update the package information using the following command.
$ sudo apt-get update
Install the hadoop-fuse.
$ sudo apt-get install hadoop-0.20-fuse
Once fuse-dfs is installed, go ahead and mount HDFS using FUSE as follows:
$ sudo hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>
Conclusion:
Thus the one node Hadoop cluster is mounted using FUSE successfully.
17
EX. NO.: 8 MAP AND REDUCE – WORD COUNT
Date:
Aim:
To write a word count program to demonstrate the use of Map and Reduce tasks.
Mapreduce:
MapReduce is a programming model and an associated implementation for processing and

generating large data sets with a parallel, distributed algorithm on a cluster. A MapReduce program
is composed of a Map() procedure that performs filtering and sorting and a Reduce() method that
performs a summary operation.
 "Map" step: Each worker node applies the "map()" function to the local data, and writes
the output to a temporary storage. A master node ensures that only one copy of redundant
input data is processed.
 "Shuffle" step: Worker nodes redistribute data based on the output keys (produced by the
"map()" function), such that all data belonging to one key is located on the same worker
node.
 "Reduce" step: Worker nodes now process each group of output data, per key, in parallel.
Steps:
Source Code:
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper

extends Mapper<Object, Text, Text, IntWritable>{
18
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context

StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer

extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,

Context context
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
19
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
1. Set Environmental Variables:
export JAVA_HOME=/usr/java/default
export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
2. Compile the source file to jar file,
$ bin/hadoop com.sun.tools.javac.Main WordCount.java

$ jar cf wc.jar WordCount*.class
3. Run the Application
$ bin/hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output
Output:
$ bin/hadoop fs -cat /user/joe/wordcount/output/part-r-00000`
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2`
Conclusion:
Thus the word count program to demonstrate the Map and Reduce task is done
successfully.
20
EX. NO.: 9 API’S OF HADOOP
Date:
Aim:
To write a program to use the API's of Hadoop to interact with it.
Steps:
1. Start NameNode daemon and DataNode daemon: (port 50070)

$ sbin/start-dfs.sh
2. Start ResourceManager daemon and NodeManager daemon: (port 8088)
$ sbin/start-yarn.sh
3. Make the HDFS directories required to execute MapReduce jobs:

$ bin/hdfs dfs -mkdir /user
4. Copy the input files into the distributed filesystem:
$ bin/hdfs dfs -put <input-path>/* /input

Run some of the examples provided:
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-
2.5.1.jar wordcount /input /output1
5. Examine the output files:
View the output files on the distributed filesystem:
$ bin/hdfs dfs -cat /output/*
21
Conclusion:
Thus the program to use the API of Hadoop is implemented successfully.
You like my Website Please Share it and Following
Thanking you Welcome

22

CS6712 Grid and Cloud Computing Laboratory

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

CS6712 Grid and Cloud Computing Laboratory

Hochgeladen von

Copyright:

Verfügbare Formate

Website: https://larshansolutions.blogspot.

CS6712 GRID AND CLOUD COMPUTING LABORATORY LTPC0032

The student should be made to:

LIST OF EQUIPMENT FOR A BATCH OF 30 STUDENTS:

Aim:To Develop basic commands in Linux Environment.

Using the utility Terminal / Console

Creating a new user

boss@localhost:/$ adduser hadoop

Adding user `hadoop' ...

Adding new group `hadoop' (1002) ...

Adding new user `hadoop' (1002) with group `hadoop' ...

Creating home directory `/home/hadoop' ...

Copying files from `/etc/skel' ...

Enter new UNIX password:

Enter the new value, or press ENTER for the default

Full Name []:

Is the information correct? [Y/n]

boss@localhost:/$ adduser hadoop sudo

Beginning as new User

Check System's architecture and Kernel

BigData environment generally, is to be established on 64 bit architecture. In

hadoop@localhost:/$ cat /proc/cpuinfo | grep --color lm

Here lm indicate Long Mode i.e. 64 bit CPU architecture

Linux localhost 3.8.0-38-generic #[YOUR_DISTRO] SMP Thu Mar 13 16:22:48

• It is installed with 64 bit OS.

hadoop@localhost:/$ sudo mkdir /opt/bigdata

2. Move all the necessary software to this directory

hadoop@localhost:/$ ls -l /opt/ total 44

4. Assigning Permissions mean – Change User and Group to hadoop

1. Installation from .bin File

Check if “X – (Execute)” flag is enabled for <BINARY / SCRIPT FILE>

hadoop@localhost:/$ls <BINARY / SCRIPT FILE>

If “X” flag is not set for the current user then

hadoop@localhost:/$chmod u+x <BINARY / SCRIPT FILE>

And check again

hadoop@localhost:/$ls <BINARY / SCRIPT FILE> -rwxr-xr-x 2 test test

To run / Execute the BINARY / SCRIPT File

hadoop@localhost:/$bash <BINARY / SCRIPT FILE>

▪ This file is available in /etc/profile.

▪ Example : Installation of Java on Linux

1. To set HOME Variable

hadoop@localhost:/$ export JAVA_HOME=/opt/jdk1.7.0_65/

hadoop@localhost:/$ echo $JAVA_HOME /opt/jdk1.7.0_65/

3. To Update PATH variable with JAVA_HOME

hadoop@localhost:/$ export PATH=$PATH: $JAVA_HOME/bin

hadoop@localhost:/$ echo $PATH

hadoop@localhost:/$ sudo vi /etc/profile

[sudo] password for hadoop: <-- Enter Password here

2. Create lines that are underlined at /etc/profile file

# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

JAVA_HOME=/opt/<OUTPUT OF PWD COMMAND TO BE PASTED HERE>

if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ]; then

# The file bash.bashrc already sets the default PS1.

if [ "`id -u`" -eq 0 ]; then PS1='# '

else PS1='$ '

# The default umask is now handled by pam_umask.

# See pam_umask(8) and /etc/login.defs.

for i in /etc/profile.d/*.sh; do if [ -r $i ]; then

export JAVA_HOME PATH=$PATH:$JAVA_HOME/bin

1 vi Editor / vim (vi improved) Editor – Text Editor

To use vi: vi filename

To exit vi and save changes: ZZ or :wq

To exit vi without saving changes: :q!

To enter vi Insert mode: i