CREAM Guide

EGEE
EGEE User’s Guide

CREAM S ERVICE
Document identifier: EGEE-JRA1-TEC-595770
Date: January 18, 2010
Activity: JRA1: Middleware Engineering and

Integration
Lead Partner:
Document status: DRAFT
Document link: https://edms.cern.ch/document/595770
Abstract: This user’s guide explains how to use the CREAM CE Service.
INFSO-RI-508833 PUBLIC 1/60

Doc. Identifier:
EGEE-JRA1-TEC-595770
EGEE USER’S GUIDE
Delivery Slip
Name Partner Date Signature

From
Reviewed by
Approved by
Document Change Log

Doc. Identifier:
EGEE USER’S GUIDE
Issue Date Comment Author

July 12, 2005 Section on glite-ce-job-status M. Sgaravatto
added.
July 14, 2005 Section on glite-ce-job-cancel M. Sgaravatto
added.
Appendix on job states added.
Doc on –status, –from and –
to option added for glite-ce-job-
status.
Other minor changes.
July 22, 2005 Section on glite-ce-job-purge M. Sgaravatto
added.
Security model on job mgmt op-
erations explained.
Introduction added.
August 04, 2005 Suspend and resume docu- M. Sgaravatto
mented.
October 14, 2005 UI configuration added. M. Sgaravatto
January 11, 2006 Specified that port is optional. M. Sgaravatto
February 06, 2006 Added documentation on M. Sgaravatto
glite-ce-get-cemon-url com-
mand.
February 26, 2006 Changes on job status docu- M. Sgaravatto
mented.
March 01, 2006 CREAM architecture added. M. Sgaravatto
March 09, 2006 New functionality to restrict job M. Sgaravatto
submissions documented.
New “Issued commands” field on
job status documented.
March 26, 2006 Really-Running job status docu- M. Sgaravatto
mented.
March 31, 2006 Proxy renewal documented. M. Sgaravatto
April 03, 2006 Minor changes. M. Sgaravatto
April 19, 2006 Minor changes. M. Sgaravatto
August 03, 2006 X509_VOMS_DIR and M. Sgaravatto
X509_CERT_DIR env. vari-
ables documented
September 27, 2006 Modifications because of use of M. Sgaravatto
delegation 2.0
February 28, 2008 General review M. Sgaravatto
November 12, 2009 Added documentation on M. Sgaravatto
glite-ce-service-info command.
January 18, 2010 Added documentation on M. Sgaravatto
-L option of glite-ce-service-info
command.
Doc. Identifier:
EGEE USER’S GUIDE
Document Change Record
Issue Item Reason for Change
Copyright Members
c of the EGEE Collaboration. 2004. See http://eu-egee.org/partners for de-
tails on the copyright holders.
EGEE (“Enabling Grids for E-science in Europe”) is a project funded by the European Union. For
more information on the project, its partners and contributors please see http://www.eu-egee.org.
You are permitted to copy and distribute verbatim copies of this document containing this copy-
right notice, but modifying this document is not allowed. You are permitted to copy this document
in whole or in part into other documents if you attach the following reference to the copied ele-
ments: “Copyright 2004.
c Members of the EGEE Collaboration. http://www.eu-egee.org”
The information contained in this document represents the views of EGEE as of the date they are
published. EGEE does not guarantee that any information contained herein is error-free, or up to
date.
EGEE MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING
THIS DOCUMENT.

Doc. Identifier:
EGEE USER’S GUIDE
C ONTENTS
1 INTRODUCTION 7
1.1 SERVICE ARCHITECTURE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 SECURITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 AUTHORIZATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.2 CREAM “SUPER-USERS” . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.3 AUTHORIZATION ON JOB MANAGEMENT OPERATIONS . . . . . . . . . 9
1.2.4 PROXY CREDENTIAL DELEGATION . . . . . . . . . . . . . . . . . . . . . 9
1.2.5 PROXY RENEWAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 QUICK-START GUIDE 10
2.1 BEFORE STARTING: GET YOUR USER PROXY . . . . . . . . . . . . . . . . . . . . 10
2.2 CREAM CLIENT COMMANDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 SUBMITTING JOBS TO CREAM BASED CES . . . . . . . . . . . . . . . . . . . . . 12
2.4 MONITORING JOBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5 OTHER JOB MANAGEMENT OPERATIONS . . . . . . . . . . . . . . . . . . . . . . 14
2.6 HANDLING JOB IDENTIFIERS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.7 RESTRICTING JOB SUBMISSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.8 OTHER OPERATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 COMMAND-LINE REFERENCE GUIDE 18

3.1 CREAM UI CONFIGURATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.1 CONFIGURATION ATTRIBUTES . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.2 EXAMPLE OF CONFIGURATION FILE . . . . . . . . . . . . . . . . . . . . . 20
3.2 GLITE-CE-JOB-SUBMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3 GLITE-CE-DELEGATE-PROXY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4 GLITE-CE-JOB-STATUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.5 GLITE-CE-JOB-LIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.6 GLITE-CE-JOB-SUSPEND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.7 GLITE-CE-JOB-RESUME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.8 GLITE-CE-JOB-CANCEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.9 GLITE-CE-JOB-PURGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.10 GLITE-CE-PROXY-RENEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.11 GLITE-CE-ALLOWED-SUBMISSION . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.12 GLITE-CE-ENABLE-SUBMISSION . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.13 GLITE-CE-DISABLE-SUBMISSION . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Doc. Identifier:
EGEE USER’S GUIDE
3.14 GLITE-CE-SERVICE-INFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.15 GLITE-CE-GET-CEMON-URL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A CREAM JOB STATES 58
B KNOWN PROBLEMS AND CAVEATS 60

Doc. Identifier:
EGEE USER’S GUIDE
1 I NTRODUCTION
The CREAM [1] (Computing Resource Execution And Management) Service is a simple, lightweight
service for job management operations at the Computing Element (CE) level. CREAM accepts job sub-
mission requests (which are described with the same JDL language used to describe the jobs submitted
to the Workload Management System) and other job management requests (e.g. job cancellation, job
monitoring, etc).
CREAM can be used by the Workload Management System (WMS), via the ICE (Interface to Cream
Environment) component, or by a generic client, e.g. an end-user willing to directly submit jobs to a
CREAM CE. For the latter user case a C++ command line interface (described in Sect. 3) is available.
For the submission to CREAM based CEs via the Workload Management System (WMS), please refer
to the WMS documentation ([8], [9]).
CREAM exposes a web service interface.
1.1 S ERVICE A RCHITECTURE
Information about the architecture of the CREAM service can be found in the CREAM web site [1].
1.2 S ECURITY
For the Grid to be an effective framework for largely distributed computation, users, user processes and
Grid services must work in a secure environment.
Due to this, the communication between the client and the CREAM service is mutually authenticated.
Depending on the specific interaction, an entity authenticates itself to the other peer using either its
own credential or a delegated user credential or both. For example when the CREAM user interface
passes a job to the CREAM service, the CREAM-UI authenticates using a proxy user credential (a proxy
certificate) whereas CREAM uses its own service credential.
The CREAM-UI uses a proxy user credential to limit the risk of compromising the original credential in
the hands of the user.
The user or service identity and their public key are included in a X.509 certificate signed by a trusted
Certification Authority (CA), whose purpose is to guarantee the association between that public key and
its owner.
According to what just premised, the user has to possess a valid X.509 certificate on the submit-
ting machine, consisting of two files: the certificate file and the private key file. The location of
the two mentioned files is assumed to be either pointed to respectively "$X509_USER_CERT" and
"$X509_USER_KEY" or by "$HOME/.globus/usercert.pem" and "$HOME/.globus/userkey.pem" if the
X509 environment variables are not set.

Doc. Identifier:
EGEE USER’S GUIDE
How to extract userkey.pem and usercert.pem from your certificate

Usually X.509 Certificates are downloaded using a browser and managed by the browser itself.
Anyway it is possible to export your certificate in a file PKCS12 (which will probably have the
extension .p12 or .pfx).
The procedure to export the certificate vary from browser to browser, for example Internet Explorer
starts with "Tools − > Internet Options − > Content"; Netscape Communicator has a "Security"
button on the top menu bar; Mozilla starts with "Edit − > Preferences − > Privacy and Security
− > Certificates" and Firefox has "Edit − > Preferences − > Advanced − > Certificates − >
manage certificates − > backup".
PKCS12 format is not accepted by the Grid Security Infrastructure, but you can easily convert
it into the supported standard (PEM). This operation will split your *.p12 file in two files: the
certificate (usercert.pm) and the private key (userkey.pm). The conversion can be performed with
openssl tool:
$ openssl pkcs12 -nocerts -in mycert.p12 -out userkey.pem
$ openssl pkcs12 -clcerts -nokeys -in mycert.p12 -out usercert.pem
$ chmod 0400 userkey.pem
$ chmod 0600 usercert.pem
Permission must be set as shown not only for security reasons: voms-proxy-init and grid-proxy-init
commands will fail if your private key is not protected as listed above.
Actually the user certificate and private key files are not mandatory on the CREAM-UI machine; indeed
they are only needed for the creation of the proxy user credentials through the grid-proxy-init or voms-
proxy-init commands. Please refer to the VOMS User’s Guide [4] for detailed information about the
VOMS client commands.
Alternatively you can download the proxy credentials from a trusted site and work with it without having
the cert and key available locally.
All CREAM-UI commands, when started, check for the existence and expiration date of a user proxy
credentials in the location pointed to by "$X509_USER_PROXY" or in "/tmp/x509up_u<UID>" (where
<UID> is the user identifier in the submitting machine OS) if the X509 environment variable is not set.
If the proxy certificate does not exist or has expired, the CREAM-UI returns an error message to the user
and exits.
It is important to note that besides authentication, proxy credential issued by VOMS, i.e. containing
FQANs (Fully Qualified Attribute Names [4]), are used by CREAM to get the VO the user is currently
working for. If a given proxy credential contains attributes for more then one VO, than the default one
(i.e. the one first position) is considered.
1.2.1 AUTHORIZATION
The client must be properly authorized when interacting with the CREAM service. In CREAM it is
possible to specify the Virtual Organizations (VOs) authorized to access the CREAM service and/or
specific Grid users.
This is supported by two AuthZ PDPs (Policy Decision Points):
• the VOMS PDP;
• the ban-allow (grid-mapfile) PDP.

Doc. Identifier:
EGEE USER’S GUIDE
The former allows specifying the Virtual Organizations whose users are allowed to use the CREAM
service. The latter allows authorizing specific Distinguished Names (DNs).
1.2.2 CREAM “ SUPER - USERS ”
The administrator of a CREAM based CE can specify a list of “super-users” (the user DNs of these
privileged users are specified in the /etc/grid-security/admin-list file of the CREAM CE host): these
special users are allowed to manage also jobs submitted by other persons (see Sect. 1.2.3), and are
allowed to enable/disable job submissions to the considered CREAM CE (see Sect. 2.7).
1.2.3 AUTHORIZATION ON JOB MANAGEMENT OPERATIONS
As a general rule, a user can manage only her own jobs (i.e. the ones she submitted). So, for example, a
user can’t cancel or check the status of jobs submitted by other users.
On the other, as mentioned in Sect. 1.2.2, CREAM “super-users” are allowed to manage also jobs sub-
mitted by other persons.
1.2.4 P ROXY CREDENTIAL DELEGATION
Each job submitted to the CREAM service is given the delegated credentials of the user who submitted
it. These credentials can then be used when operations requiring security support has to be performed by
the job (e.g. a GridFTP file transfer).
As described in Sects. 3.2 and 3.3, for what concerns the “direct” submission to CREAM it is possible
to ask the “automatic” delegation of the credentials during the submission operation, or it is possible to
explicitly delegate credentials, and then ask to rely on these previously delegated credentials. It is recom-
mended to rely on this latter mechanism, using the same delegated proxy for multiple job submissions,
instead of delegating each time a proxy. Delegating a proxy, in fact, is an operation that can require a
non negligible time.
1.2.5 P ROXY RENEWAL
It is possible that long jobs may outlive the validity of the initial delegated credentials; if so the job
will die prematurely. To avoid this it is possible to renew the proxy of jobs submitted to CREAM CEs.
For “direct” submissions (i.e. not via the WMS) to CREAM, this is discussed in Sect. 3.10, while for
submissions to CREAM based CEs using the WMS, please refer to the WMS guide [8].

Doc. Identifier:
EGEE USER’S GUIDE
2 Q UICK - START G UIDE
This section briefly explains the sequence of operations to be performed by a user to submit and then
manage jobs on CREAM based CEs, referring to the C++ Command Line Interface (CLI).
2.1 B EFORE STARTING : GET YOUR USER PROXY
Before using any of the CREAM client commands, it is necessary to have a valid proxy credential
available on the client machine. You can create it using the voms-proxy-init command or alternatively
the grid-proxy-init one. If you already have a valid proxy available on your machine just make the
X509_USER_PROXY environment variable point to it.
In order to get a proxy certificate issued by VOMS [4] you should have in the directory
/opt/glite/etc/vomses the proper VOMS file containing a line as follows:
"EGEE" "kuiken.nikhef.nl" "15001" "/O=dutchgrid/O=hosts/OU=nikhef.nl/CN=kuiken.nikhef.nl" "EGEE" "22"
or the corresponding line for your VO (ask your VO admin for that). You also need to install the VOMS
certificate in the /etc/grid-security/vomsdir directory (ask your VO admin for this as well).
Make moreover sure you have in the directory $HOME/.globus your certificate/key pair, i.e. the follow-
ing files:
usercert.pem
userkey.pem
Note that file permissions are important: the two files must have respectively 0600 and 0400 permissions.
Then you can issue the VOMS client command (you will be prompted for the pass-phrase):
> voms-proxy-init -voms EGEE

Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto/Email=massimo.sgaravatto@pd.infn.it
Enter GRID pass phrase:
Creating temporary proxy .................................... Done
Contacting kuiken.nikhef.nl:15001 [/O=dutchgrid/O=hosts/OU=nikhef.nl/CN=kuiken.nikhef.nl] ‘‘EGEE’’ Done
Creating proxy ............................................. Done
Your proxy is valid until Thu Jul 28 23:11:01 2005
> voms-proxy-info
subject : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto/Email=massimo.sgaravatto@pd.infn.it/CN=proxy
issuer : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto/Email=massimo.sgaravatto@pd.infn.it
identity : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto/Email=massimo.sgaravatto@pd.infn.it
type : proxy
strength : 512 bits
path : /tmp/x509up_u756
timeleft : 11:59:25
More information about security in CREAM is available in Sect. 1.2
2.2 CREAM CLIENT COMMANDS
The most relevant commands to interact with CREAM based CEs are:

Doc. Identifier:
EGEE USER’S GUIDE
• glite-ce-job-submit <jdl_file>
• glite-ce-delegate-proxy <delegation_Id>
• glite-ce-job-status <job_Ids>
• glite-ce-job-list <host[:port]>
• glite-ce-job-cancel <job_Ids>
• glite-ce-job-suspend <job_Ids>
• glite-ce-job-resume <job_Ids>
• glite-ce-job-purge <job_Ids>
• glite-ce-proxy-renew <delegation_Ids>
• glite-ce-service-info <host[:port]>
• glite-ce-get-cemon-url <host[:port]>
• glite-ce-enable-submission <host[:port]>
• glite-ce-disable-submission <host[:port]>
• glite-ce-allowed-submission <host[:port]>
Man pages are available for all the CREAM client commands. You can also access information about
the usage of each command by issuing:
> <command> --help
glite-ce-job-submit submits a job to a CREAM based CE. It requires a JDL file as input and returns a
CREAM job identifier.
glite-ce-delegate-proxy allows the user to delegate her proxy credential to the CREAM service. This
delegated credential can then be used for job submissions.
glite-ce-job-status displays information (in particular the states) of jobs previously submitted to
CREAM based CEs.
glite-ce-job-list lists the identifiers of jobs submitted to a CREAM based CE by the user issuing the
command.
glite-ce-job-cancel cancels one or more jobs previously submitted to CREAM based CEs.
glite-ce-job-purge clears one or more jobs from CREAM based CEs. After this operation the purged
jobs can’t be managed anymore.
glite-ce-proxy-renew renews delegations, and therefore refreshes the proxy of the jobs submitted to
CREAM based CEs using the considered delegations.
glite-ce-service-info returns information about the CREAM service (version, status, etc.).
glite-ce-get-cemon-url returns the end-point of the CEMon service coupled with the considered
CREAM CE.

Doc. Identifier:
EGEE USER’S GUIDE
glite-ce-enable-submission (re-)enables job submissions on the specified CREAM CE.
glite-ce-disable-submission disables job submissions on the specified CREAM CE.
glite-ce-allowed-submission checks if jobs submissions on the specified CREAM CE are allowed or

have been disabled.
All these commands, whose use and purpose is briefly described in the following subsections, are de-
scribed in more detail in Sect. 3.
2.3 S UBMITTING JOBS TO CREAM BASED CE S
To submit jobs to CREAM based CEs, the command glite-ce-job-submit (described in Sect 3.2) must be
used. The glite-ce-job-submit command requires as input a job description file, which describes the job
characteristics and requirements through the JDL (Job Description Language).
A typical example of a JDL job description file is:
[
Type = "Job";
JobType = "Normal";
Executable = "myexe";
StdInput = "myinput.txt";
StdOutput = "message.txt";
StdError = "error.txt";
InputSandbox = {"/users/seredova/example/myinput.txt",
"/users/seredova/example/myexe"};
OutputSandbox = {"message.txt", "error.txt"};
OutputSandboxBaseDestUri = "gsiftp://se.pd.infn.it/data/seredova";
]
Such a JDL would make the myexe executable be transferred on the remote CREAM CE and be run
taking the myinput.txt file (also copied from the client node) as input. The standard streams of the job are
redirected to files message.txt and error.txt, and when job completes its execution they are automatically
uploaded on gsiftp://se.pd.infn.it/data/seredova.
A detailed description of the available JDL attributes and of the rules for building correct JDL files is
provided by the "CREAM Job Description Language Attributes Specification" document [3].
Each job submitted to a CREAM based CE is given the delegated credentials of the user who submitted
it. These credentials can then be used when operations requiring security support has to be performed by
the job.
There are two possible options to deal with proxy delegation:
• asking the “automatic” delegation of the credentials during the submission operation;
• explicitly delegating credentials, and then asking to rely on these previously delegated credentials
on the actual submission operations.

Doc. Identifier:
EGEE USER’S GUIDE
It is highly suggested to rely on this latter mechanism, using the same delegated proxy for multiple job
submissions, instead of delegating each time a proxy. Delegating a proxy, in fact, is an operation that can
require a non negligible time.
The command glite-ce-delegate-proxy (described in Sect. 3.3) is the command to be used to explicitly
delegate the user credentials to a CREAM CE.
The following shows an example of job submission, performed explicitly delegating credentials.
So first of all the credentials are delegated to a CREAM based CE (whose endpoint is specified with the
option –endpoint (-e):
> glite-ce-delegate-proxy -e cream-ce-01.pd.infn.it mydelid

2006-02-26 15:03:37,286 NOTICE - Proxy with delegation id [mydelid] successfully
delegated to endpoint [https://cream-ce-01.pd.infn.it:8443//ce-cream/services/CREAMDelegation]
The identifier of the delegation is then specified with the –delegationId (-D) option in the job submit
operation:
> glite-ce-job-submit -D mydelid -r cream-ce-01.pd.infn.it:8443/cream-lsf-grid02 myjob1.jdl
The option -r (–resource) has been used to specify the identifier of the CREAM CE where the job has to
be submitted to. myjob1.jdl is the JDL file describing the job to be submitted.
The command returns the CREAM job identifier associated with this job (e.g. https://cream-ce-
01.pd.infn.it:8443/CREAM116j9vgnf ) which identifies it in clear and unique way all over the Grid system
scope.
2.4 M ONITORING JOBS
Passing the CREAM job id returned by the glite-ce-job-submit command to the glite-ce-job-status com-
mand (described in detail in Sect. 3.4) it is possible to monitor the submitted job. Several (static and
dynamic) information can be shown, depending on the chosen verbosity level. The verbosity level can
be 0 (less verbosity), 1 or 2 (most verbosity). Please note that specifying 0 as verbosity level means
calling on the CREAM service a faster operation than when using 1 or 2 as verbosity level.
The most relevant attribute is the job status: the list of the possible CREAM job states (and their meaning)
is reported in Sect. A.
The following is an example of job status operation, specifying 1 as verbosity level:
$ glite-ce-job-status -L 1 https://cream-02.pd.infn.it:8443/CREAM738582717
****** JobID=[https://cream-02.pd.infn.it:8443/CREAM738582717]
Current Status = [DONE-FAILED]
ExitCode = [N/A]
FailureReason = [lsf_reason=256; Cannot move ISB (${globus_transfer_cmd}
gsiftp://cream-02.pd.infn.it//CREAMTests/Exe1/ssh1.sh file:///home/infngrid001/home_cream_738582717/CREAM738582717/ssh1.sh):
error: globus_ftp_client: the server responded with an error 500 500-Command failed. : globus_l_gfs_file_open failed.
500-globus_xio: Unable to open file //CREAMTests/Exe1/ssh1.sh
500-globus_xio: System error in open: No such file or directory
500-globus_xio: A system call failed: No such file or directory 500 End.]
Grid JobID = [N/A]

Doc. Identifier:
EGEE USER’S GUIDE
Job status changes:

-------------------
Status = [REGISTERED] - [Tue 22 Jan 2008 15:55:08] (1201013708)
Status = [PENDING] - [Tue 22 Jan 2008 15:55:08] (1201013708)
Status = [IDLE] - [Tue 22 Jan 2008 15:55:11] (1201013711)
Status = [RUNNING] - [Tue 22 Jan 2008 15:55:18] (1201013718)
Status = [DONE-FAILED] - [Tue 22 Jan 2008 16:03:10] (1201014190)
Issued Commands:
-------------------
*** Command Name = [JOB_REGISTER]

Command Category = [JOB_MANAGEMENT]
Command Status = [SUCCESSFULL]
*** Command Name = [JOB_START]

The meaning of the various fields reported by the glite-ce-job-status is explained in Sect. 3.4. In this
example it is interesting to note that the job failed (as reported by the Current Status field) for the
problem reported in the FailureReason field: the file to be transferred was not found.
Instead of explicitly specifying the identifiers of the jobs to monitor, the user can also ask to monitor
all her jobs, in case specifying conditions (on the submission date and/or on the job status) that must
be met. For example to monitor all jobs, whose status is DONE-OK or DONE-FAILED, submitted to
the grid005.pd.infn.it CREAM CE between July 23, 2005 10:00 and July 28, 2005 11:00, the following
command must be issued:
> glite-ce-job-status --all -e grid005.pd.infn.it:8443 --from ’2005-07-23 10:00:00’ \

--to ’2005-07-28 11:00:00’ -s DONE-OK:DONE-FAILED
2.5 OTHER JOB MANAGEMENT OPERATIONS
If a user is interested to get the identifiers of all her jobs submitted to a specific CREAM CE, she can use
the command glite-ce-job-list command, described in Sect. 3.5. For example the following command
returns the identifiers of all the jobs submitted to the specified CREAM CE, owned by the user issuing
the command:
> glite-ce-job-list grid005.pd.infn.it:8443
In some cases it might be needed to cancel jobs which have been previously submitted to CREAM based
CEs. This can be achieved via the glite-ce-job-cancel command, described in Sect. 3.8.
E.g., the command:
> glite-ce-job-cancel https://grid005.pd.infn.it:8443/CREAM115j5vfnf
cancels the specified job.
A running or idle job can be suspended (i.e. its execution will be stopped), and be resumed (i.e. it will
run again) later. This can be achieved with the glite-ce-job-suspend (described in detail in Sect. 3.6)

Doc. Identifier:
EGEE USER’S GUIDE
and glite-ce-job-resume (documented in Sect. 3.7) commands. The following example shows that after
having issued the glite-ce-job-suspend command, after a while the job status will become HELD.
> glite-ce-job-suspend https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2
Are you sure you want to suspend specified job(s) [y/n]: y
> glite-ce-job-status -L 0 https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2
****** JobID=[https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2]
Status = [HELD]
Issuing the glite-ce-job-resume command, the job will run/will be idle again:
> glite-ce-job-resume https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2
Are you sure you want to resume specified job(s) [y/n]: y
> glite-ce-job-status -L 0 https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2
****** JobID=[https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2]
Status = [REALLY-RUNNING]
A CREAM job can be monitored (via the glite-ce-job-status) even after it has completed its execution. A
job gets “lost” (i.e. it is not possible to monitor or manage it anymore) only when the user who submitted
it decides to explicitly clear it, or when the CREAM system administrator decides to do this purging
operation. A user can purge her own jobs, using the glite-ce-job-purge command, described in detail in
Sect. 3.9.
E.g., after having issued the command:
> glite-ce-job-purge https://cream-ce-01.pd.infn.it:8443/CREAM116jbi4o0
the specified job can’t be managed anymore (e.g. it is not possible to check its status anymore).
As introduced in Sect. 1.2.5, a job dies if its proxy expires. To avoid this, it is possible to explicitly renew
the proxy of a job with the glite-ce-proxy-renew command, discussed in Sect. 3.10.
E.g. the following command:
> glite-ce-proxy-renew -e cream-ce-01.pd.infn.it:8443 mydelid
renews the proxy of all the jobs having mydelid as delegation id.
It must be stressed that for jobs submitted to CREAM based CEs via the Workload Management System
(WMS), proxy renewal is automatically dealt by the middleware, as explained in the WMS guide [8].
2.6 H ANDLING JOB IDENTIFIERS
Handling the job identifiers directly quickly becomes tedious. To avoid this, you can make the glite-ce-
job-submit and glite-ce-job-list commands append the job Id(s) to a named file using the –output (-o)

Doc. Identifier:
EGEE USER’S GUIDE
option. On the other side, the CREAM client commands which take job identifier(s) as argument accept
also the –input (-i) option which allows the job identifier(s) to be read from a file.
The following shows an example:
> glite-ce-job-submit -a -r cream-ce-01.pd.infn.it:8443/cream-lsf-grid02 -o idfile myjob.jdl
https://cream-ce-01.pd.infn.it:8443/CREAM116jbs5b9
The returned job id got also inserted in the specified file (idfile), which can be specified with the –input
(-i) option e.g. with the glite-ce-job-status command:
> glite-ce-job-status -i idfile
****** JobID=[https://cream-ce-01.pd.infn.it:8443/CREAM116jbs5b9]
Status=[REALLY-RUNNING]
2.7 R ESTRICTING JOB SUBMISSIONS
In order to prevent that a CREAM CE gets overloaded by too many jobs, the CREAM CE administrator
can set a specific policy on the number of CREAM pending and/or idle and/or running jobs. For example
it can be specified that if the number of “active” jobs (pending + idle + running jobs) is greater than a
certain threshold, new job submissions have to be disabled. In this case, if newer job submissions are
attempted, users will get an error message such as:
> glite-ce-job-submit -a -r grid006.pd.infn.it:8443/cream-lsf-grid02 lnl_test.jdl

MethodName=[jobRegister] ErrorCode=[0] Description=[The CREAM2 service cannot
accept jobs anymore] FaultCause=[The CREAM2 service cannot accept jobs anymore]
Timestamp=[Tue 22 Jan 2008 16:28:47]
In order to avoid degrading the performance of the system, the specified policy is not evaluated for each
job submission, but instead it is evaluated and imposed from time to time (so it might happen that for a
short time job submissions are allowed even if the specified threshold has been reached).
The procedure for setting such policies is described in the CREAM configuration page available in the
CREAM Web site [1].
CREAM “super-users” (see Sect. 1.2.2) can also disable newer job submissions via the command glite-
ce-disable-submission (described in Sect. 3.13). Submissions can then be re-enabled by a CREAM
“super-user” via the command glite-ce-enable-submission, described in Sect. 3.12.
To check if job submissions on a specific CREAM CE are allowed, the command glite-ce-allowed-
submission (described in Sect. 3.11) can be used.
E.g.:
> glite-ce-disable-submission grid006.pd.infn.it:8443

Operation for disabling new submissions succeeded
>
> glite-ce-allowed-submission grid006.pd.infn.it:8443
Job Submission to this CREAM CE is disabled

Doc. Identifier:
EGEE USER’S GUIDE
>
> glite-ce-enable-submission grid006.pd.infn.it:8443
Operation for enabling new submissions succeeded
>
Job Submission to this CREAM CE is enabled
It must be stressed that if job submissions to a specific CREAM CE are disabled, all other operations
(job status, job cancellations, etc.) can still be performed.
2.8 OTHER OPERATIONS
It is possible to get information about the CREAM service (interface and service version, status, etc)
using the glite-ce-service-info command1 , e.g.:
> glite-ce-service-info cream-13.pd.infn.it:8443
Interface Version = [2.1]

Service Version = [1.12]
Description = [CREAM 2]
Started at = [Tue Nov 10 14:42:12 2009]
Submission enabled = [YES]
Status = [RUNNING]
Service Property = [SUBMISSION_THRESHOLD_MESSAGE]->
[Threshold for Load Average
(1 min): 10 => Detected value for Load Average(1 min): 0.03
Threshold for Load Average(5 min): 10 => Detected value for Load Average(5 min):
0.03
Threshold for Load Average(15 min): 10 => Detected value for Load Average(15 min
): 0.00
Threshold for Memory Usage: 95 => Detected value for Memory Usage: 57.41%
Threshold for Swap Usage: 95 => Detected value for Swap Usage: 2.02%
Threshold for Free FD: 500 => Detected value for Free FD: 204500
Threshold for tomcat FD: 800 => Detected value for Tomcat FD: 107
Threshold for FTP Connection: 30 => Detected value for FTP Connection: 1
Threshold for Number of active jobs: -1 => Detected value for Number of active j
obs: 0
Threshold for Number of pending commands: -1 => Detected value for Number of pen
ding commands: 0
A CREAM CE is usually coupled with a CEMon [2] service, which can be queried to get information
about the CE and/or can notify clients with specific CE events. CEMon also notifies ICE (i.e. the WMS
component responsible to deal with CREAM based CEs) about CREAM jobs. The command glite-ce-
get-cemon-url can be used to get the end-point of this CEMon service, e.g.:
> glite-ce-get-cemon-url grid005.pd.infn.it:8443

https://grid005.pd.infn.it:8443/ce-monitor/services/CEMonitor
1 Command introduced with glite-ce-cream-cli v. 1.12

Doc. Identifier:
EGEE USER’S GUIDE
3 C OMMAND -L INE R EFERENCE G UIDE
In this section we describe syntax and behavior of the commands made available by the CREAM-UI to
allow “direct” job submission, monitoring and control to CREAM based CEs.
Before describing the single commands, it is described how the CREAM-UI can be configured.
3.1 CREAM UI C ONFIGURATION
The configuration of the CREAM UI is accomplished via three possible configuration files:
1. A general configuration file.

This file is looked for in the following order:
• $GLITE_LOCATION/etc/glite_cream.conf
• /opt/glite/etc/glite_cream.conf
2. A VO specific configuration file.

• $GLITE_LOCATION/etc/<VO>/glite_cream.conf
• /opt/glite/etc/<VO>/glite_cream.conf
3. A user specific configuration file.

• The file specified with the –conf option of the considered command
• The file referenced by the $GLITE_CREAM_CLIENT_CONFIG environment variable
• $HOME/.glite/<VO>/glite_cream.conf (if the VO is defined) or
$HOME/.glite/glite_cream.conf otherwise
Each of these files is a classad containing definitions (see Sect. 3.1.1 for the set of possible attributes that
can be defined).
If the same attribute is defined in more than a configuration file, the definition in the user specific con-
figuration file (if any) is considered. Likewise the definitions in the VO specific configuration file have
higher priority than the ones specified in the general configuration file.
It must be noted that one or more (even all) of these three configuration files can be missing.
3.1.1 C ONFIGURATION ATTRIBUTES
We list here the possible attributes that can be specified in the configuration files:
• CREAM_URL_PREFIX: the prefix to the <hostname>:<port> to build the CREAM service

endpoint.
The default is https://.

Doc. Identifier:
EGEE USER’S GUIDE
• CREAMDELEGATION_URL_PREFIX: the prefix to the <hostname>:<port> to build the

CREAM delegation service endpoint.
The default is https://.
• DEFAULT_CREAM_TCPPORT: the port to be appended to the hostname (if not specified by
the user) to build the CREAM and CREAM delegation service endpoint.
The default is 8443.
• CREAM_URL_POSTFIX: the postfix to be appended to the hostname:port to build the CREAM
service endpoint.
The default is /ce-cream/services/CREAM2.
• CREAMDELEGATION_URL_POSTFIX: the postfix to be appended to the hostname:port to
build the CREAM delegation service endpoint.
The default is /ce-cream/services/gridsite-delegation.
• JDL_DEFAULT_ATTRIBUTES: the classad that must be included by default in the user’s JDLs.
The default is an empty classad.
• STATUS_VERBOSITY_LEVEL: the default verbosity level to be used for the glite-ce-job-status
command: see Sect. 3.4.
The default value is 0.
• SUBMIT_LOG_DIR: the directory where by default the log file glite-ce-job-
submit_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-job-submit command) is created.
The default is /tmp/glite_cream_cli_logs.
• DELEGATE_LOG_DIR: the directory where by default the log file glite-ce-delegate-
proxy_CREAM_<username>_<date>_<time>.log (created when the –debug option is used with
the glite-ce-delegate-proxy command) is created.
• STATUS_LOG_DIR: the directory where by default the log file glite-ce-job-
status_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-job-status command) is created.
• LIST_LOG_DIR: the directory where by default the log file glite-ce-job-
list_CREAM_<username>_<date>_<time>.log (created when the –debug option is used
with the glite-ce-job-list command) is created.
• SUSPEND_LOG_DIR: the directory where by default the log file glite-ce-job-
suspend_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-job-suspend command) is created.
• RESUME_LOG_DIR: the directory where by default the log file glite-ce-job-
resume_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-job-resume command) is created.

Doc. Identifier:
EGEE USER’S GUIDE
• CANCEL_LOG_DIR: the directory where by default the log file glite-ce-job-

cancel_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-job-cancel command) is created.
• PURGE_LOG_DIR: the directory where by default the log file glite-ce-job-

purge_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-job-purge command) is created.
• ALLOWEDSUB_LOG_DIR: the directory where by default the log file glite-ce-allowed-

submission_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-allowed-submission command) is created.
• ENABLE_LOG_DIR: the directory where by default the log file glite-ce-enable-

submission_CREAM_<username>_<date>_<time>.log (created when the –debug option
is used with the glite-ce-enable-submission command) is created.
• DISABLE_LOG_DIR: the directory where by default the log file glite-ce-disable-

submission_CREAM_<username>_<date>_<time>.log (created when the –debug option is
used with the glite-ce-disable-submission command) is created.
• PROXYRENEW_LOG_DIR: the directory where by default the log file glite-ce-proxy-

renew_CREAM_<username>_<date>_<time>.log (created when the –debug option is used
with the glite-ce-proxy-renew command) is created.
• GETSERVICEINFO_LOG_DIR: the directory where by default the log file glite-ce-service-

info_CREAM_<username>_<date>_<time>.log (created when the –debug option is used with
the glite-ce-service-info command) is created.
• GETCEMONURL_LOG_DIR: the directory where by default the log file glite-ce-get-cemon-

url_CREAM_<username>_<date>_<time>.log (created when the –debug option is used with
the glite-ce-get-cemon-url command) is created.
As mentioned above, if the same attribute is defined in more than a configuration file, the definition
in the user specific configuration file (if any) has higher priority than the definition in the VO specific
configuration file (if any), which has higher priority than the definition in the generic configuration file.
If an attribute is not defined anywhere, the default value is considered.
3.1.2 E XAMPLE OF CONFIGURATION FILE
The following represents an example of a CREAM UI configuration file:

Doc. Identifier:
EGEE USER’S GUIDE
[
JDL_DEFAULT_ATTRIBUTES = [
JobType=’’Normal’’;
Type=’’job’’;
];
STATUS_VERBOSITY_LEVEL = 2;
CANCEL_LOG_DIR=’’/tmp/CREAMLogs’’;
PURGE_LOG_DIR=’’/tmp/CREAMLogs’’;
RESUME_LOG_DIR=’’/tmp/CREAMLogs’’;
STATUS_LOG_DIR=’’/tmp/CREAMLogs’’;
SUBMIT_LOG_DIR=’’/tmp/CREAMLogs’’;
SUSPEND_LOG_DIR=’’/tmp/CREAMLogs’’;
LIST_LOG_DIR=’’/tmp/CREAMLogs’’;
DELEGATE_LOG_DIR=’’/tmp/CREAMLogs’’;
]

Doc. Identifier:
EGEE USER’S GUIDE
3.2 GLITE - CE - JOB - SUBMIT
Allows the user to submit a job for execution on a CREAM based CE.
SYNOPSIS
glite-ce-job-submit –resource CEId [options] <jdl_file>
Options:
--help, -h
--conf, -c <file_name>
--autm-delegation, -a
--delegationId, -D <ID>
--vo, -V <VO_name>
--output, -o <file_name>
--noint, -N
--nomsg, -n
--debug, -d
--logfile, -l <file_name>
--version, -v
DESCRIPTION
glite-ce-job-submit is the command for submitting jobs to CREAM based Computing Elements. glite-
ce-job-submit requires as mandatory inputs:
• the identifier of the CREAM CE, specified using the option –resource (-r), where the job has to be
submitted;
• a job description (JDL) file in which job characteristics and requirements have been specified
via a CREAM JDL expression. The guide explaining how to write a CREAM JDL expression is
available at [3].
The job description file given in input to this command is syntactically checked and default values are
assigned to the not provided mandatory attributes in order to create a meaningful class-ad. The resulting
ad is then sent to the CREAM CE.
Upon successful submission, this command returns to the user the submitted CREAM job identifier
(jobId), a string that identifies unambiguously the job, that can be later used as a handle to perform
monitor and control operations on the job (e.g. see glite-ce-job-status described in Sect. 3.4).
The format of the CREAM jobId is as follows:
https://<CREAM_full_hostname>:<port>/CREAM<unique_string>
It is possible to redirect the returned jobId to an output file using the –output (-o) option. If the file
already exists and it is not a CREAM job id file, the user is asked if she wants to overwrite the file or if
she prefers to abort the operation.
The –resource (-r) directive must be used to target the job submission to a specific CREAM CE, identified
by its identifier CEId. The CREAM CE identifier is a string that unambiguously identifies a CE CREAM
service belonging to the Grid. The standard format for a CREAM CEId is:

Doc. Identifier:
EGEE USER’S GUIDE
<full-hostname>:<port-number>/cream-<service>-<queue-name>
where <service> identifies the underlying resource management system (e.g. lsf, pbs, etc.).
The submitted job is given the delegated credentials of the user who submitted the job: these creden-
tials can then be used when operations requiring security support (e.g. gridftp file transfers) have to be
performed by the job. It is possible to rely on credentials previously delegated with the glite-ce-delegate-
proxy command (see Sect. 3.3) or it is possible to ask the “automatic” delegation of the same credentials
used during the submission operation. In the former case the –delegationId (-D) option must be speci-
fied, while in the latter case the —-autm-delegation (-a) option must be used. Please note that one (and
only one) among these two options must be specified.
OPTIONS
–help, -h
Displays command usage.
–conf file_name, -c file_name

Specifies the user specific configuration file (see Sect. 3.1).
–resource ce_id, -r ce_id

Specifies that the job has to be submitted to the CREAM CE identified by the specified ce_id.
Standard format for the CREAM CEId is:
<full hostname>:<port number>/cream-<service>-<queue name>
where <service> specifies the underlying local resource management system (e.g. lsf, pbs, etc.)
–autm-delegation, -a
Specifies that the proxy credentials that have to be delegated are the ones used for this job submission.
–delegationId ID, -D ID,

Forces the submitted job to use previously delegated (via the command glite-ce-delegate-proxy: see
Sect. 3.3) user credentials, identified with the specified ID.
–vo VO_name, -V VO_name

Specifies the Virtual Organisation the user belongs to. This option is ignored if the user issued the
command using a VOMS proxy: in this case in fact the VO to be considered is the one specified in the
VOMS proxy2 .
–output file_name, -o file_name

Writes the generated jobId assigned to the submitted job in the file specified by file_name. file_name
can be either a simple name or an absolute path on the submitting machine. In the former case the file
file_name is considered in the current working directory.
–noint, -N If this option is specified every interactive question to the user is skipped, and default values
(yes; all: consider all jobs of the given list) will be considered.
2 When dealing with VOMS proxies, the VOMS server certificate is looked for in the directory pointed out by the
X509_VOMS_DIR environment variable (default: /etc/grid-security/vomsdir) while the CA certificates are looked for in the
directory pointed out by the X509_CERT_DIR environment variable (default: /etc/grid-security/certificates)

Doc. Identifier:
EGEE USER’S GUIDE
–nomsg, -n
This option makes the command print on the standard output only the jobId generated for the job if
submission was successful (or error messages in case of failure). Therefore any warning or info message
is suppressed if this option is used.
–debug, -d
When this option is specified, debugging information is displayed on the standard output and written also
into the file glite-ce-job-submit_CREAM_<username>_<date>_<time>.log file under (by default) the
/tmp/glite_cream_cli_logs directory.
–logfile file_name, -l file_name

When this option is specified, a command log file (whose pathname is file_name) is created.
–version, -v
Displays the version of the CREAM CLI software.
EXIT STATUS
glite-ce-job-submit exits with a status value of 0 (zero) upon success, and >0 (greater than zero) upon
failure.
EXAMPLES
> glite-ce-job-submit -a -r grid005.pd.infn.it:8443/cream-lsf-grid02 myjob1.jdl

where myjob1.jdl is e.g. as follows:
##############################################!
#
# -------- Job description file ----------
#
##############################################!
[
JobType = "Normal";
Executable = "sum.exe";
StdOutput="out.out";
StdError="err.err";
Environment={"CMSVER=321"};
InputSandbox={"file:///home/user/sum.exe", "otherfile"};
InputSandboxBaseUri="gsiftp://se1.pd.infn.it/cmssw";
OutputSandbox={"out.out", "err.err", ‘‘hist.out’’};
OutputSandboxBaseDestUri="gsiftp://se2.pd.infn.it/myfiles";
]
If the operation succeeds, the output will be the generated jobid, e.g:
https://grid005.pd.infn.it:8443/CREAM116jbs5b9
Examples of JDL files to be used with the glite-ce-job-submit command are available in [3].

Doc. Identifier:
EGEE USER’S GUIDE
3.3 GLITE - CE - DELEGATE - PROXY
Allows the user to delegate her proxy credentials, that can be used later for her job submissions.
SYNOPSIS
glite-ce-delegate-proxy –endpoint <host>[:<port>] <DelegationId>
Options:
--help, -h
--endpoint, -e <endpoint>
--debug, -d
--version, -v
DESCRIPTION
glite-ce-delegate-proxy is the command to delegate proxy credentials, that can be used later in job
submission operations (see option –delegationId of the glite-ce-job-submit command in Sect. 3.2).
In this way the user can delegate her proxy credentials, and then perform multiple submissions relying on
the same delegated credentials, therefore speeding up the submission process (as explained in Sect. 1.2.4,
delegating a proxy can take some time).
glite-ce-delegate-proxy requires as mandatory inputs:
• the endpoint (that is the full hostname and the port number) of the considered CREAM CE service,
specified using the –endpoint (-e) option;
• a string, chosen by the end user, specifying the id of the delegated proxy. This id will then have to
be specified in the following submission operations (with the option –delegationId of the glite-ce-
job-submit command).
The proxy file considered by this command (that is the proxy to delegate to the remote CREAM service)
is:
• the one pointed by the environment variable X509_USER_PROXY;
• the file /tmp/x509up_u<uid>, if the environment variable X509_USER_PROXY is undefined.
If a delegation with the same id and belonging to the same user already exists in that CREAM CE, an
error message is returned.
The -endpoint (-e) directive must be used to target the operation to a specific CREAM CE. The format
to be used is the following:
<fullhostname>[:<portnumber>]

Doc. Identifier:
EGEE USER’S GUIDE
OPTIONS
–help, -h

–endpoint <host>[:<port>], -e <host>[:<port>]

Specifies the hostname and port number of the service where the user credentials have to be delegated.
If the port is not specified, the port to be used is the one specified in the CREAM UI configuration file
(see Sect. 3.1). Specifying the –endpoint option is mandatory.
–debug, -d
into the file glite-ce-delegate-proxy_CREAM_<username>_<date>_<time>.log file under (by default)
the /tmp/glite_cream_cli_logs directory.

–version, -v
EXIT STATUS
glite-ce-delegate-proxy exits with a status value of 0 (zero) upon success, and >0 (greater than zero)
upon failure.
EXAMPLES
> glite-ce-delegate-proxy -e ce1.pd.infn.it:8443 myproxyid1

2006-02-26 15:31:09,924 NOTICE - Proxy with delegation id [myproxyid1] succesfully
delegated to endpoint [https://ce1.pd.infn.it:8443//ce-cream/services/CREAMDelegation]
The following example shows what happens if a delegation with the same id already exists for the same
user in the considered CREAM CE:
> glite-ce-delegate-proxy -e cream-01.pd.infn.it:8443 idalreadyused

2006-09-27 09:31:42,233 ERROR - Delegation ID ’idalreadyused’ already exists
for client (DN=’C=IT,O=INFN,OU=Personal Certificate,L=Padova,CN=Massimo Sgaravatto,
Email=massimo.sgaravatto@pd.infn.it; VOMS ATTRS=’ /EGEE/Role=NULL/Capability=NULL’).
Call renewProxyReq.

Doc. Identifier:
EGEE USER’S GUIDE
3.4 GLITE - CE - JOB - STATUS
Displays information concerning jobs submitted to CREAM based CEs.
SYNOPSIS
glite-ce-job-status [options] <jobId1> <jobId2> ... <jobIdn>
Options:
--help, -h
--level, -L <verb_num>
--status, -s <status1>:<status2>:..:<statusm>
--from, -f <time>
--to, -t <time>
--all, -a
--input, -i <file_name>
--nomsg, -n
--debug, -d
--logfile, -l <file_path>
--version, -v
DESCRIPTION
glite-ce-job-status is the command for getting information concerning jobs submitted to CREAM based
Computing Elements.
glite-ce-job-status can monitor one or more jobs: the jobs to be checked are identified by their job
identifiers (job Ids returned by glite-ce-job-submit) provided as arguments to the command and separated
by a blank space.
It has to be remarked that a job can be monitored only by his/her owner (i.e. the user who submitted it).
As reported in Sect. 1.2.3, only the “super-users” can check the status of jobs submitted by other persons.
If the –all option is specified, information about all the jobs submitted to the interested CREAM CE
(whose endpoint must be specified with the –endpoint option, which is mandatory in this case) owned by
the user issuing the command is returned. When the command is launched with the –all option, neither
can a jobId be specified nor can the –input option be used.
The –input option permits to specify a file (file_name) that contains the jobIds to monitor. This file must
be the one generated with the –output option of the glite-ce-job-submit or glite-ce-job-list commands
(see Sects. 3.2 and 3.5). If the file_name does not represent an absolute path, it will be searched in the
current working directory.
When a glite-ce-job-status request is issued, the host name and the port number to be used to contact the
CREAM service(s) are retrieved from the specified jobIds (which encompass this information) unless the
–endpoint option is used. As stated before, the –endpoint option is mandatory if the –all option is used.
The –level option allows setting the detail level of the returned information. This option can be specified
with three values: 0, 1 and 2. The default level of verbosity (the one considered if this option is not
used and if the attribute STATUS_VERBOSITY_LEVEL hasn’t been specified in the CREAM CLI conf
file) is 0. Please note that specifying 0 as verbosity level means calling on the CREAM service a faster
operation than when using 1 or 2 as level.

Doc. Identifier:
EGEE USER’S GUIDE
Hereafter are listed the information displayed according to the verbosity level:
Verbosity level equal 0:
• JobID: the CREAM job unique identifier
• Status: the job current status: see Sect. A
• ExitCode: the Unix exit code - if applicable
• FailureReason: the failure reason (for failed and aborted jobs)
• Description: a description, used for example to explain why a job has been killed by CREAM
Verbosity level equal 1:
• JobID: the CREAM job unique identifier
• GridJobID: the Grid (WMS) job unique identifier, for jobs submitted to CREAM via the WMS
• Current Status: the job current status
• ExitCode: the Unix exit code - if applicable
• FailureReason: the failure reason (for failed and aborted jobs)
• Description: a description, used for example to explain why a job has been killed by CREAM
• Job status changes: the status changes (along with their timestamps) occured so far for the consid-
ered job
• Issued Commands: the commands (all but the status/list operations) and the relevant results issued
on this job
With a verbosity level equal 2, some additional information fields are added:
• Worker Node: the worker node where the job is being/was executed
• Local User: the user name on the CE mapped to the considered Grid user
• Deleg Proxy ID: the id of the delegated user proxy
• DelegProxyInfo: information about the user proxy used for this job
• Lease ID: the id of the delegation (if the lease mechanism has been used)
• Lease Time: the lease expiration time (if the lease mechannism has been used)
• CREAM ISB URI: the URI of the CREAM directory for the InputSandbox
• CREAM OSB URI: the URI of the CREAM directory for the OutputSandbox
• JDL: the job JDL
• Issued Commands: the same information as for verbosity level 1 plus the timestamps when the
commands have been processed and completed

Doc. Identifier:
EGEE USER’S GUIDE
If the user issuing the glite-ce-job-status is a “superuser” of that CREAM CE (see Sect. 1.2.3), also the
following information is printed specifying ’2’ as verbosity level:
• LRMS Abs JobID: the id of the job wrt the LRMS Abstraction Layer (i.e. the BLAH job id, since
BLAH is used as middleware component to interact with the underlying resource management
system)
• LRMS JobID: the id of the job in the LRMS (i.e. the PBS id /the LSF id, ...)
• Working Dir: the directory used by CREAM to store information about this job. Please note that
this is not the job working directory on the worker node.
It is possible to specify filters on the jobs to be considered. A first type of filter is on the job status: with
the –status (-s) option it is possible to specify a list of possible job states: only the jobs being in one of
the listed status will be considered.
Another possible filter that can be considered is on the submission time: it is possible to select, with
the –from (-f ) and/or –to (-t) option a temporal interval: only jobs submitted in this time range will be
considered.
OPTIONS
–help, -h

–level verb_level, -L verb_level

Sets the detail level of information about the job displayed to the user. Possible values for verb_level are
0,1 and 2. Please note that specifying 0 as verbosity level means calling on the CREAM service a faster
operation than when using 1 or 2 as level.
–status <status1>:<status2>: ... :<statusm>, -s <status1>:<status2>: ... :<statusm>

Specifies a filter on job status: only jobs being in one of the listed states will be considered; all jobs that
are not in one of the specified states will be filtered out. In Sect. A the list of the possible CREAM job
states, that can therefore be specified, is reported.
–from <DATE>, -f <DATE>

Specifies a filter on job submission time: only jobs submitted after the specified date will be consid-
ered. <TIME>, which must be specified wrt the local timezone, must have the format ’YYYY-MM-DD
HH:mm:ss’ where:
• YYYY is the year as a decimal number including the century
• MM is the month (1-12)
• DD is the day of month (1-31)
• HH, mm, ss represent respectively the hour (0-23), the minutes (0-59) and the seconds (0-59)

Doc. Identifier:
EGEE USER’S GUIDE
–to <DATE>, -t <DATE>

Specifies a filter on job submission time: only jobs submitted before the specified date will be consid-
ered. <TIME>, which must be specified wrt the local timezone, must have the format ’YYYY-MM-DD
HH:mm:ss’ where:
• YYYY is the year as a decimal number including the century
• MM is the month (1-12)
• DD is the day of month (1-31)
• HH, mm, ss are respectively the hour (0-23), the minutes (0-59) and the seconds (0-59)
–all, -a
Displays information about all known jobs in the specified (with the –endpoint option, which is manda-
tory in this case) CREAM CE, owned by the user issuing the command. This option can’t be used either
if one or more jobIds have been specified or if the –input option has been used.
–endpoint host[:port], -e host[:port]

Selects the endpoint of the CREAM CE to send the status request to. If this option is not specified, the
endpoint will be extracted from the JobID strings. The endpoint must have the format <host>:<port>
or <host>.
–input file_name, -i file_name

Displays information about the jobs whose Ids are contained in the file file_name. This option can’t be
used if the –all option has been specified as well.
–nomsg, -n
When this option is specified, any warning or info message is suppressed.
–debug, -d
into the file glite-ce-job-status_CREAM_<username>_<date>_<time>.log file under (by default) the

–version, -v
EXIT STATUS
glite-ce-job-status exits with a value of 0 if the status of all the specified jobs is retrieved correctly, >0
if errors occurred.
EXAMPLES
1.> glite-ce-job-status -L 1 https://cream-02.pd.infn.it:8443/CREAM955790315
displays the following lines:

Doc. Identifier:
EGEE USER’S GUIDE
Current Status = [REALLY-RUNNING]
Grid JobID = [N/A]
Job status changes:

-------------------
Status = [REGISTERED] - [Fri 21 Dec 2007 17:49:38] (1198255778)
Status = [PENDING] - [Fri 21 Dec 2007 17:49:38] (1198255778)
Status = [IDLE] - [Fri 21 Dec 2007 17:49:41] (1198255781)
Status = [RUNNING] - [Fri 21 Dec 2007 17:49:51] (1198255791)
Status = [REALLY-RUNNING] - [Fri 21 Dec 2007 17:50:04] (1198255804)
Issued Commands:
-------------------
*** Command Name = [JOB_REGISTER]

*** Command Name = [JOB_START]

2.> glite-ce-job-status --endpoint grid005.pd.infn.it:8443 --all --status DONE-OK:DONE-FAILED --from ’2005-07-13 10:00:00’
Status information of all jobs belonging to the user who issued the command, submitted to the specified
(with the –endpoint option) CREAM CE after the specified date and time, and being in DONE-OK or in
DONE-FAILED status, will be printed.
3. The following example shows a job that has been killed by CREAM because the lease time expired:
> glite-ce-job-status https://cream-03.pd.infn.it:8443/CREAM530773588
Status = [CANCELLED]
ExitCode = []
Description = [Job cancelled by Lease Manager! (try 1/3)!]

Doc. Identifier:
EGEE USER’S GUIDE
3.5 GLITE - CE - JOB - LIST
Displays the Ids of all the jobs, belonging to the user issuing the command, which are “known” to a
specified CREAM based CE.
SYNOPSIS
glite-ce-job-list [options] <host>[:<port>]
Options:
--help, -h
--output, -o <file_name>
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
glite-ce-job-list is the command to get the identifiers of all the jobs owned by the user issuing the com-
mand, which have been submitted to the CREAM based CE whose host and port number are specified
as argument of the command. If the port is not specified, the port to be used is the one specified in the
CREAM UI configuration file (see Sect. 3.1).
These returned CREAM job identifiers can be later used as a handle to perform monitor and control
operations on the jobs (e.g. see glite-ce-job-status, described in Sect. 3.4).
It is possible to redirect the returned CREAM jobIds into an output file using the –output (-o) option. If
the file already exists and it is not a CREAM id file, the user is asked if she wants to overwrite the file,
or if she prefers to abort the operation.
OPTIONS
–help, -h

–output file_name, -o file_name

Writes the resulting jobIds in the file specified by file_name. file_name can be either a simple name or
an absolute path on the submitting machine. In the former case the file file_name is considered in the
current working directory.
–nomsg, -n
This option makes the command print on the standard output only the jobIds. Therefore any warning or
info message is suppressed if this option is used.
–debug, -d

Doc. Identifier:
EGEE USER’S GUIDE
When this option is specified, debugging information is displayed on the standard output and written
also into the file glite-ce-job-list_CREAM_<username>_<date>_<time>.log file under (by default)

–version, -v
EXIT STATUS
glite-ce-job-list exits with a status value of 0 (zero) upon success and >0 (greater than zero) upon failure.
EXAMPLES
> glite-ce-job-list cream-ce-01.pd.infn.it:8443
If the operation succeeded, the output will be a list of jobid, e.g:
https://cream-ce-01.pd.infn.it:8443/CREAM116j8te4j
https://cream-ce-01.pd.infn.it:8443/CREAM116j8uojl
https://cream-ce-01.pd.infn.it:8443/CREAM116j92f39
https://cream-ce-01.pd.infn.it:8443/CREAM116j93te1
https://cream-ce-01.pd.infn.it:8443/CREAM116j9403v
https://cream-ce-01.pd.infn.it:8443/CREAM116j942oc
https://cream-ce-01.pd.infn.it:8443/CREAM116j9457d
https://cream-ce-01.pd.infn.it:8443/CREAM116j947mc
https://cream-ce-01.pd.infn.it:8443/CREAM116j94a99
https://cream-ce-01.pd.infn.it:8443/CREAM116j9vgnf
https://cream-ce-01.pd.infn.it:8443/CREAM116jbhdk0
https://cream-ce-01.pd.infn.it:8443/CREAM116jbhg67
https://cream-ce-01.pd.infn.it:8443/CREAM116jbhip8
https://cream-ce-01.pd.infn.it:8443/CREAM116jbhla7
https://cream-ce-01.pd.infn.it:8443/CREAM116jbhvij
https://cream-ce-01.pd.infn.it:8443/CREAM116jbi25a
https://cream-ce-01.pd.infn.it:8443/CREAM116jbi4o0
https://cream-ce-01.pd.infn.it:8443/CREAM116jbs5b9

Doc. Identifier:
EGEE USER’S GUIDE
3.6 GLITE - CE - JOB - SUSPEND
Suspends one or more jobs running or idle on CREAM based CEs.
SYNOPSIS
glite-ce-job-suspend [options] <jobId1> <jobId2> ... <jobIdn>
Options:
--help, -h
--all, -a
--input, -i <file>
--noint, -N
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
The command glite-ce-job-suspend suspends jobs running or idle on CREAM based CEs.
glite-ce-job-suspend can suspend one or more jobs: the jobs to be suspended are identified by their job
identifiers (job Ids returned by the glite-ce-job-submit command) provided as arguments to the command
and separated by a blank space.
It has to be remarked that a job can be suspended only by his/her owner (i.e. the user who submitted it).
As reported in Sect. 1.2.3, only the “super-users” can suspend jobs submitted by other persons.
Before actually suspending the specified jobs, the command prompts the user for confirmation.
Only jobs in RUNNING, REALLY-RUNNING or IDLE status can be suspended, even if some other con-
straints can be imposed by the underlying resource manager system (e.g. PBS doesn’t allow suspending
running jobs). For all the other job states the suspend request will result in a failure. An error message is
of course printed also if a specified jobId doesn’t exist.
If the –all option is specified, all the jobs submitted to the interested CREAM CE (whose endpoint must
be specified with the –endpoint option, which is mandatory in this case), owned by the user issuing the
command, are requested to be suspended. When the command is launched with the –all option, neither
The –input option permits to specify a file (file_name) that contains the jobIds to be suspended. This
file must be the one generated with the –output option of the glite-ce-job-submit or glite-ce-job-list
commands (see Sects. 3.2 and 3.5). When using this option the user is interrogated for choosing among
all, one or a subset of the listed job identifiers. If the file file_name does not represent an absolute path
the file will be searched in the current working directory.
When a glite-ce-job-suspend request is issued, the host name and the port number to be used to contact
the CREAM service(s) are retrieved from the specified jobIds (which encompass this information) unless
the –endpoint option is used. As stated before, the –endpoint option is mandatory if the –all option is
used.
OPTIONS

Doc. Identifier:
EGEE USER’S GUIDE
–help, -h

–all, -a
Suspends all known jobs in the specified (with the –endpoint option, which is mandatory in this case)
CREAM CE, owned by the user issuing the command. This option can’t be used either if one or more
jobIds have been specified or if the –input option has been specified.

Selects the endpoint of the CREAM CE to send the suspend request to. If this option is not specified, the
or <host>.

Suspends jobs whose Ids are contained in the file file_name. This option can’t be used if the –all option
has been specified as well.
–noint, -N
If this option is specified every interactive question to the user is skipped, and default values (yes; all:
consider all jobs of the given list) will be considered.
–nomsg, -n
–debug, -d
into the file glite-ce-job-suspend_CREAM_<username>_<date>_<time>.log file under (by default)

–version, -v
EXIT STATUS
glite-ce-job-suspend exits with a value of 0 if the suspend operation succeeds, >0 if errors occurred.
EXAMPLES
1.> glite-ce-job-suspend https://grid005.pd.infn.it:8443/CREAM116j9457d

requests to suspend the specified job. After a while the job status will become HELD:
> glite-ce-job-status -L 0 https://grid005.pd.infn.it:8443/CREAM116j9457d
****** JobID=[https://grid005.pd.infn.it:8443/CREAM116j9457d]

Doc. Identifier:
EGEE USER’S GUIDE
Status=[HELD]
2.If it is requested to suspend a job which is not running or idle, an error message will be returned, as in
the following example:
> glite-ce-job-suspend https://cream-02.pd.infn.it:8443/CREAM302651331
Are you sure you want to suspend specified job(s) [y/n]: y

2008-01-22 14:46:26,402 ERROR - This job was not matching the status or had a status incompatible for operation: [CREAM302651331]
Server error Message is: [MethodName=[JOB_SUSPEND]
Timestamp=[Tue 22 Jan 2008 14:46:26] ErrorCode=[2]
Description=[the job has a status not compatible with the JOB_SUSPEND command!]
FaultCause=[N/A]]
3.> glite-ce-job-suspend --endpoint grid005.pd.infn.it:8443 --all --noint
requests to suspend all jobs, submitted to the specified CREAM CE, owned by the user issuing the
command.

Doc. Identifier:
EGEE USER’S GUIDE
3.7 GLITE - CE - JOB - RESUME
Resumes one or more jobs, which have been previously suspended.
SYNOPSIS
glite-ce-job-resume [options] <jobId1> <jobId2> ... <jobIdn>
Options:
--help, -h
--all, -a
--input, -i <file>
--noint, -N
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
The command glite-ce-job-resume resumes jobs, which therefore can run again, previously suspended
with the glite-ce-job-suspend command.
glite-ce-job-resume can resume one or more jobs: the jobs to be resumed are identified by their job
It has to be remarked that a job can be resumed only by his/her owner (i.e. the user who submitted it).
As reported in Sect. 1.2.3, only the “super-users” can resume jobs submitted by other persons.
Before actually resuming the specified jobs, the command prompts the user for confirmation.
Only jobs which have been previosly suspended, and therefore are in Held status, can be resumed. For
all the other job states the resume request will result in a failure. An error message is of course printed
also if a specified jobId doesn’t exist.
command, are requested to be resumed. When the command is launched with the –all option, neither
The –input option permits to specify a file (file_name) that contains the jobIds to be resumed. This
When a glite-ce-job-resume request is issued, the host name and the port number to be used to contact
the CREAM service(s) are retrieved from the specified jobIds (which encompass this information) unless
the –endpoint option is used. As stated before, the –endpoint option is mandatory if the –all option is
used.
OPTIONS

Doc. Identifier:
EGEE USER’S GUIDE
–help, -h

–all, -a
Resumes all known jobs in the specified (with the –endpoint option, which is mandatory in this case)

Selects the endpoint of the CREAM CE to send the resume request to. If this option is not specified, the
or <host>.

Resumes jobs whose Ids are contained in the file file_name. This option can’t be used if the –all option
–noint, -N
–nomsg, -n
–debug, -d
into the file glite-ce-job-resume_CREAM_<username>_<date>_<time>.log file under (by default) the

–version, -v
EXIT STATUS
glite-ce-job-resume exits with a value of 0 if the resume operation succeeds, >0 if errors occurred.
EXAMPLES
1.> glite-ce-job-resume https://grid005.pd.infn.it:8443/CREAM116j8qi1s
requests to resume the specified job, which must be previously suspended, and which therefore must be
in HELD status.
2. If it is requested to resume a job which hasn’t been previously suspended, an error message will be
returned, as in the following example:

Doc. Identifier:
EGEE USER’S GUIDE
$ glite-ce-job-resume https://cream-02.pd.infn.it:8443/CREAM302651331
Are you sure you want to resume specified job(s) [y/n]: y

2008-01-22 15:09:53,959 ERROR - This job was not matching the status or had a status incompatible for operation: [CREAM302651331]
Server error Message is: [MethodName=[JOB_RESUME]
Timestamp=[Tue 22 Jan 2008 15:09:53] ErrorCode=[2]
Description=[the job has a status not compatible with the JOB_RESUME command!]
FaultCause=[N/A]]

Doc. Identifier:
EGEE USER’S GUIDE
3.8 GLITE - CE - JOB - CANCEL
Cancels one or more jobs previously submitted to CREAM based CEs.
SYNOPSIS
glite-ce-job-cancel [options] <jobId1> <jobId2> ... <jobIdn>
Options:
--help, -h
--all, -a
--input, -i <file>
--noint, -N
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
The command glite-ce-job-cancel cancels jobs previously submitted, using the glite-ce-job-submit com-
mand, to CREAM based CEs.
glite-ce-job-cancel can remove one or more jobs: the jobs to be removed are identified by their job
It has to be remarked that a job can be cancelled only by his/her owner (i.e. the user who submitted it).
As reported in Sect. 1.2.3, only the “super-users” can cancel jobs submitted by other persons.
Before actually cancelling the specified jobs, the command prompts the user for confirmation.
Job states for which cancellation is allowed are:
• Pending
• Idle
• Running
• Really-Running
• Held
For all the other job states the cancellation request will result in a failure. An error message is of course
printed also if the specified jobId doesn’t exist.
command, are removed. When the command is launched with the –all option, neither can a jobId be
specified nor can the –input option be used.

Doc. Identifier:
EGEE USER’S GUIDE
The –input option permits to specify a file (file_name) that contains the jobIds to be removed. This
When a glite-ce-job-cancel request is issued, the host name and the port number to be used to contact the
OPTIONS
–help, -h

–all, -a
Cancels all known jobs in the specified (with the –endpoint option, which is mandatory in this case)

Selects the endpoint of the CREAM CE to send the cancel request to. If the port is not specified, the
port to be used is the one specified in the CREAM UI configuration file (see Sect. 3.1). If this option is
not specified, the endpoint will be extracted from the JobID strings. The endpoint must have the format
<host>:<port> or <host>.

Cancels jobs whose Ids are contained in the file file_name. This option can’t be used if the –all option
–noint, -N
–nomsg, -n
–debug, -d
into the file glite-ce-job-cancel_CREAM_<username>_<date>_<time>.log file under (by default) the

–version, -v

Doc. Identifier:
EGEE USER’S GUIDE
EXIT STATUS
glite-ce-job-cancel exits with a value of 0 if the cancel operation succeeds, >0 if errors occurred.
EXAMPLES
1.> glite-ce-job-cancel --input joblist.txt
where joblist.txt is a file containing 2 jobIds, displays the following confirmation message:
0. https://cream-ce-01.pd.infn.it:8443/CREAM116jbs5b9
1. https://cream-ce-01.pd.infn.it:8443/CREAM116jbi4o0
a. Cancel all jobs
q. Quit
Your choice ? (multiple choices separated by ’,’) a
Are you sure you want to cancel specified job(s) [y/n]: y
2.> glite-ce-job-cancel --endpoint grid005.pd.infn.it:8443 --all
cancels all jobs, on the specified CREAM CE, owned by the user issuing the command.

Doc. Identifier:
EGEE USER’S GUIDE
3.9 GLITE - CE - JOB - PURGE
Cleans one or more jobs from CREAM based CEs. After this operation the purged jobs can’t be managed
anymore.
SYNOPSIS
glite-ce-job-purge [options] <jobId1> <jobId2> ... <jobIdn>
Options:
--help, -h
--all, -a
--input, -i <file_name>
--noint, -N
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
The command glite-ce-job-purge purges jobs from CREAM based CEs, i.e. any reference to these
jobs is cancelled. Therefore after this operation it is not possible anymore to monitor (e.g. with the
glite-ce-job-status command) or in some other way “manage” these jobs.
glite-ce-job-purge can purge one or more jobs: the jobs to be removed are identified by their job identi-
fiers (job Ids returned by the glite-ce-job-submit command) provided as arguments to the command and
separated by a blank space.
Before actually purging the considered jobs, the user is asked for confirmation.
It has to be remarked that a job can be purged only by his/her owner (i.e. the user who submitted it). As
reported in Sect. 1.2.3, only the “super-users” can purge jobs submitted by other persons.
Job states for which the purge is allowed are:
• REGISTERED
• DONE-OK
• DONE-FAILED
• ABORTED
• CANCELLED
For all the other job states the purge request will result in a failure. An error message is of course printed
also if the specified jobId doesn’t exist.
be specified with the –endpoint option), owned by the user issuing the command, are purged. When the
command is launched with the –all option, neither can a jobId be specified nor can the –input option be
used.

Doc. Identifier:
EGEE USER’S GUIDE
The –input option permits to specify a file (file_name) that contains the jobIds to be removed. This
commands (see Sects. 3.2 and 3.5). If the file file_name does not represent an absolute path, the file will
be searched in the current working directory.
When a glite-ce-job-purge request is issued, the host name and the port number to be used to contact the
OPTIONS
–help, -h

–all, -a
Purges all known jobs in the specified (with the –endpoint option, which is mandatory in this case)

Selects the endpoint of the CREAM CE to send the purge request to. If the port is not specified, the
port to be used is the one specified in the CREAM UI configuration file (see Sect. 3.1). If this option is
not specified, the endpoint will be extracted from the JobID strings. The endpoint must have the format
<host>:<port> or <host>.

Purges jobs whose Ids are contained in the file file_name. This option can’t be used if the –all option has
been specified as well.
–noint, -N
–nomsg, -n
–debug, -d
into the file glite-ce-job-purge_CREAM_<username>_<date>_<time>.log file under (by default) the

–version, -v
EXIT STATUS

Doc. Identifier:
EGEE USER’S GUIDE
glite-ce-job-purge exits with a value of 0 if the cancel operation succeeds, >0 if errors occurred.
EXAMPLES
1.> glite-ce-job-purge --input joblist.txt
where joblist.txt is a file containing 2 jobIds, displays the following confirmation message:
0. https://cream-ce-01.pd.infn.it:8443/CREAM116jbs5b9
1. https://cream-ce-01.pd.infn.it:8443/CREAM116jbi4o0
a. Purge all jobs
q. Quit
Your choice ? (multiple choices separated by ’,’) a
Are you sure you want to purge specified job(s) [y/n]: y
2.> glite-ce-job-purge --endpoint grid005.pd.infn.it:8443 --all --noint
purges all jobs, on the specified CREAM CE, owned by the user issuing the command.

Doc. Identifier:
EGEE USER’S GUIDE
3.10 GLITE - CE - PROXY - RENEW
Renews delegations, and therefore renews the proxy of the jobs previously submitted to CREAM based
CEs using these delegations.
SYNOPSIS
glite-ce-proxy-renew –endpoint <endpoint> [options] <delId1> <delId2> ... <delIdn>
Options:
--help, -h
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
The command glite-ce-proxy-renew renews delegations (previously done explicitly with the command
glite-ce-delegate-proxy or using the automatic delegation in the job submission process). For all jobs
using these delegations, the proxy will be therefore renewed.
This is necessary if the proxy associated to a job is expiring (this can be checked using the glite-ce-job-
status command: the attribute to be checked is DelegProxyInfo).
glite-ce-proxy-renew can renew one or more delegations. The delegation identifier associated to a specific
job can be found using the glite-ce-job-status command: the attribute to be checked is Deleg Proxy
ID. So, when the proxy has been explicitly delegated, the delegation identifier is the one specified as
argument of the glite-ce-delegate-proxy command.
The option –endpoint (-e) is mandatory.
The proxy file considered by this command (that is the “fresh” proxy to be sent to the remote CREAM
service) is:
• the one pointed by the environment variable X509_USER_PROXY;
• the file /tmp/x509up_u<uid>, if the environment variable X509_USER_PROXY is undefined.
If the specified delegationId doesn’t exist, an error message is printed.
OPTIONS
–help, -h


Doc. Identifier:
EGEE USER’S GUIDE
Selects the endpoint of the CREAM CE to send the proxy renewal request to. If the port is not specified,
the port to be used is the one specified in the CREAM UI configuration file (see Sect. 3.1). The endpoint
must have the format <host>:<port> or <host>.
This option is mandatory.
–nomsg, -n
–debug, -d
into the file glite-ce-proxy-renew_CREAM_<username>_<date>_<time>.log file under (by default)

–version, -v
EXIT STATUS
glite-ce-proxy-renew exits with a value of 0 if the proxy renewal request operation succeeds, >0 if
errors occurred.
EXAMPLES
> glite-ce-job-status -L 2 https://cream-02.pd.infn.it:8443/CREAM051924507 | grep ‘‘Deleg Proxy ID’’

Deleg Proxy ID = [mydelid]
> glite-ce-proxy-renew -e cream-02.pd.infn.it:8443 mydelid
2008-01-28 13:35:22,699 NOTICE - Proxy with delegation id [mydelid] succesfully renewed to endpoint
[https://cream-02.pd.infn.it:8443//ce-cream/services/CREAMDelegation]

Doc. Identifier:
EGEE USER’S GUIDE
3.11 GLITE - CE - ALLOWED - SUBMISSION
Checks if new job submissions to the specified CREAM CE are allowed.
SYNOPSIS
glite-ce-allowed-submission [options] <host>[:<port>]
Options:
--help, -h
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
glite-ce-allowed-submission is the command used to check if job submissions to the considered

CREAM service are allowed or have been disabled by the system administrator via the glite-ce-disable-
submission (see Sect. 3.13) or because the condition specified in the CREAM configuration file has been
met (see Sect. 2.7).
OPTIONS
–help, -h

–nomsg, -n
This option makes the command print on the standard output only the result of the operation (or error
messages in case of failure). Therefore any warning or info message is suppressed if this option is used.
–debug, -d
also into the file glite-ce-allowed-submission_CREAM_<username>_<date>_<time>.log file under
(by default) the /tmp/glite_cream_cli_logs directory.

–version, -v
EXIT STATUS
glite-ce-allowed-submission exits with a status value of 0 (zero) upon success and >0 (greater than
zero) upon failure.

Doc. Identifier:
EGEE USER’S GUIDE
EXAMPLES

Job Submission to this CREAM CE is disabled

Doc. Identifier:
EGEE USER’S GUIDE
3.12 GLITE - CE - ENABLE - SUBMISSION
Enables job submissions to the specified CREAM CE.
SYNOPSIS
glite-ce-enable-submission [options] <host>[:<port>]
Options:
--help, -h
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
glite-ce-enable-submission is an operation supposed to be used only by a “super-user” (see Sect. 1.2.2)

of the considered CE. It is used to (re-)allow job submissions to the considered CREAM CE.
OPTIONS
–help, -h

–nomsg, -n
–debug, -d
also into the file glite-ce-enable-submission_CREAM_<username>_<date>_<time>.log file under (by
default) the /tmp/glite_cream_cli_logs directory.

–version, -v
EXIT STATUS
glite-ce-enable-submission exits with a status value of 0 (zero) upon success and >0 (greater than zero)
upon failure.
EXAMPLES

Doc. Identifier:
EGEE USER’S GUIDE

Operation for enabling new submissions succeded
The following example shows what happens if the considered user is not a “super-user” of the specified
CREAM CE:

2008-01-16 17:22:01,135 ERROR - MethodName=[acceptNewJobSubmissions] ErrorCode=[0]
Description=[Operation reserved only to administrator!]
FaultCause=[Operation reserved only to administrator!]
Timestamp=[Wed 16 Jan 2008 17:22:01]

Doc. Identifier:
EGEE USER’S GUIDE
3.13 GLITE - CE - DISABLE - SUBMISSION
Disables job submissions to the specified CREAM CE.
SYNOPSIS
glite-ce-disable-submission [options] <host>[:<port>]
Options:
--help, -h
--nomsg, -n
--debug, -d
--version, -v
DESCRIPTION
glite-ce-disable-submission is an operation supposed to be used only by a “super-user” (see Sect. 1.2.2)

of the considered CE. It is used to disable job submissions to the considered CREAM CE. So after having
issued this command it is not possible to submit anymore new jobs on the considered CREAM CE, while
the other operations (job status, job cancellation, etc.) are still allowed.
OPTIONS
–help, -h

–nomsg, -n
–debug, -d
into the file glite-ce-disable-submission_CREAM_<username>_<date>_<time>.log file under (by de-
fault) the /tmp/glite_cream_cli_logs directory.

–version, -v
EXIT STATUS
glite-ce-disable-submission exits with a status value of 0 (zero) upon success and >0 (greater than zero)
upon failure.

Doc. Identifier:
EGEE USER’S GUIDE
EXAMPLES

Operation for disabling new submissions succeded
The following example shows what happens if the considered user is not a “super-user” of the specified
CREAM CE:
2008-01-16 18:35:06,477 ERROR - MethodName=[acceptNewJobSubmissions] ErrorCode=[0]

Description=[Operation reserved only to administrator!]
FaultCause=[Operation reserved only to administrator!]
Timestamp=[Wed 16 Jan 2008 18:35:06]

Doc. Identifier:
EGEE USER’S GUIDE
3.14 GLITE - CE - SERVICE - INFO
Returns information about the CREAM service (version, status, etc.)
SYNOPSIS
glite-ce-service-info [options] <host>[:<port>]
Options:
--help, -h
--debug, -d
--version, -v
DESCRIPTION
glite-ce-service-info3 is the command used to get information about the CREAM service. Information
provided by the command includes the interface version, the service version, the status of the CREAM
service.
OPTIONS
–help, -h

–level verb_level, -L verb_level

Sets the detail level of information. Possible values for verb_level are 1 and 2.
–debug, -d
into the file glite-ce-service-info_CREAM_<username>_<date>_<time>.log file under (by default)

–version, -v
EXIT STATUS
glite-ce-service-info exits with a status value of 0 (zero) upon success and >0 (greater than zero) upon
failure.
EXAMPLES
3 Command introduced with glite-ce-cream-cli v. 1.12

Doc. Identifier:
EGEE USER’S GUIDE
> glite-ce-service-info cream-13.pd.infn.it:8443
Interface Version = [2.1]

Service Version = [1.12]
Description = [CREAM 2]
Started at = [Tue Nov 10 14:42:12 2009]
Submission enabled = [YES]
Status = [RUNNING]
Service Property = [SUBMISSION_THRESHOLD_MESSAGE]->
[Threshold for Load Average
(1 min): 10 => Detected value for Load Average(1 min): 0.03
Threshold for Load Average(5 min): 10 => Detected value for Load Average(5 min): 0.03
Threshold for Load Average(15 min): 10 => Detected value for Load Average(15 min): 0.00
Threshold for Memory Usage: 95 => Detected value for Memory Usage: 57.41%
Threshold for Swap Usage: 95 => Detected value for Swap Usage: 2.02%
Threshold for Free FD: 500 => Detected value for Free FD: 204500
Threshold for tomcat FD: 800 => Detected value for Tomcat FD: 107
Threshold for FTP Connection: 30 => Detected value for FTP Connection: 1
Threshold for Number of active jobs: -1 => Detected value for Number of active jobs: 0
Threshold for Number of pending commands: -1 => Detected value for Number of pending commands: 0

Doc. Identifier:
EGEE USER’S GUIDE
3.15 GLITE - CE - GET- CEMON - URL
Returns the end-point of the CEMon service coupled to the considered CREAM service.
SYNOPSIS
glite-ce-get-cemon-url [options] <host>[:<port>]
Options:
--help, -h
--debug, -d
--version, -v
DESCRIPTION
glite-ce-get-cemon-url is the command used to get the URL of the CEMon service coupled to the con-
sidered CREAM service. This CEMon service can then be contacted to get information about CREAM
jobs, about the CE characteristics and status, etc.4 .
OPTIONS
–help, -h

–debug, -d
into the file glite-ce-get-cemon-url_CREAM_<username>_<date>_<time>.log file under (by default)

–version, -v
EXIT STATUS
glite-ce-get-cemon-url exits with a status value of 0 (zero) upon success and >0 (greater than zero)
upon failure.
EXAMPLES
> glite-ce-get-cemon-url grid005.pd.infn.it:8443

https://grid005.pd.infn.it:8443//ce-monitor/services/CEMonitor
4 This depends on the installed CEMon sensors

Doc. Identifier:
EGEE USER’S GUIDE
R EFERENCES
[1] CREAM Home page, http://grid.pd.infn.it/cream
[2] CEMon Home page, http://grid.pd.infn.it/cemon
[3] CREAM Job Description Language Attributes Specification, EGEE-JRA1-TEC-592336,

https://edms.cern.ch/document/592336.
[4] VOMS User’s Guide, EGEE-JRA1-TEC-571991 ,

https://edms.cern.ch/document/571991.
[5] CREAM WSDL, http://grid.pd.infn.it/cream/wsdl
[6] CREAM Java Client, http://grid.pd.infn.it/cream/field.php?n=CREAM.CREAMJavaCLIDocumentation
[7] CREAM Job Purger, http://grid.pd.infn.it/cream/field.php?n=Main.CREAMJobPurger
[8] WMProxy User’s Guide, https://edms.cern.ch/document/674643
[9] WMProxy Client Quick Start, http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=WMProxyClient.

QuickStart

Doc. Identifier:
EGEE USER’S GUIDE
A CREAM JOB STATES
Here below is provided a brief description of the meaning of each possible state a CREAM job can enter:
• REGISTERED: the job has been registered but it has not been started yet.
• PENDING the job has been started, but it has still to be submitted to the LRMS abstraction layer
module (i.e. BLAH).
• IDLE: the job is idle in the Local Resource Management System (LRMS).
• RUNNING: the job wrapper, which “encompasses” the user job, is running in the LRMS.
• REALLY-RUNNING: the actual user job (the one specified as Executable in the job JDL) is running
in the LRMS.
• HELD: the job is held (suspended) in the LRMS.
• CANCELLED: the job has been cancelled.
• DONE-OK: the job has successfully been executed.
• DONE-FAILED: the job has been executed, but some errors occurred.
• ABORTED: errors occurred during the “management” of the job, e.g. the submission to the LRMS
abstraction layer software (BLAH) failed.
• UNKNOWN: the job is an unknown status.
Figure 1 shows the possible job states transitions.

Doc. Identifier:
EGEE USER’S GUIDE
Figure 1: CREAM job states

Doc. Identifier:
EGEE USER’S GUIDE
B K NOWN P ROBLEMS AND C AVEATS
• Since the gridftp protocol (the protocol used for the InputSanbox files staging) in general doesn’t
preserve the x flag, the script specified as Executable in the JDL, should perform a chmod +x for
all the files needing execution permission, that are transferred within the InputSandbox of the job.
• The Arguments attribute in the JDL allows the user to specify all the command line arguments
needed to start the job They have to be specified as a single string, e.g. the job sum that is started
with:
$ sum N1 N2 -out result.out
is described by:
Executable = ‘‘sum’’;
Arguments = ‘‘N1 N2 -out result.out’’;
If you want to specify a quoted string inside the Arguments then you have to escape quotes with
the \ character. e.g. when describing a job like:
$ grep -i ‘‘my name’’ *.txt you will have to specify:
Executable = ‘‘/bin/grep’’;
Arguments = ‘‘-i \‘‘my name\’’ *.txt’’;

CREAM Guide

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

CREAM Guide

Hochgeladen von

Copyright:

Verfügbare Formate

EGEE

EGEE User’s Guide

Document identifier: EGEE-JRA1-TEC-595770

Date: January 18, 2010

Activity: JRA1: Middleware Engineering and

Document status: DRAFT

Document link: https://edms.cern.ch/document/595770

INFSO-RI-508833 PUBLIC 1/60

Name Partner Date Signature

Document Change Log

INFSO-RI-508833 PUBLIC 2/60

Issue Date Comment Author

Document Change Record

Issue Item Reason for Change

INFSO-RI-508833 PUBLIC 4/60

3 COMMAND-LINE REFERENCE GUIDE 18

INFSO-RI-508833 PUBLIC 5/60

A CREAM JOB STATES 58

B KNOWN PROBLEMS AND CAVEATS 60

INFSO-RI-508833 PUBLIC 6/60

1.1 S ERVICE A RCHITECTURE

INFSO-RI-508833 PUBLIC 7/60

How to extract userkey.pem and usercert.pem from your certificate

• the VOMS PDP;

• the ban-allow (grid-mapfile) PDP.

INFSO-RI-508833 PUBLIC 8/60

1.2.2 CREAM “ SUPER - USERS ”

1.2.3 AUTHORIZATION ON JOB MANAGEMENT OPERATIONS

1.2.4 P ROXY CREDENTIAL DELEGATION

1.2.5 P ROXY RENEWAL

INFSO-RI-508833 PUBLIC 9/60

2 Q UICK - START G UIDE

2.1 B EFORE STARTING : GET YOUR USER PROXY

"EGEE" "kuiken.nikhef.nl" "15001" "/O=dutchgrid/O=hosts/OU=nikhef.nl/CN=kuiken.nikhef.nl" "EGEE" "22"

> voms-proxy-init -voms EGEE

More information about security in CREAM is available in Sect. 1.2

2.2 CREAM CLIENT COMMANDS

INFSO-RI-508833 PUBLIC 10/60

> <command> --help

INFSO-RI-508833 PUBLIC 11/60

glite-ce-enable-submission (re-)enables job submissions on the specified CREAM CE.

glite-ce-disable-submission disables job submissions on the specified CREAM CE.

glite-ce-allowed-submission checks if jobs submissions on the specified CREAM CE are allowed or

2.3 S UBMITTING JOBS TO CREAM BASED CE S

INFSO-RI-508833 PUBLIC 12/60

> glite-ce-delegate-proxy -e cream-ce-01.pd.infn.it mydelid

> glite-ce-job-submit -D mydelid -r cream-ce-01.pd.infn.it:8443/cream-lsf-grid02 myjob1.jdl

2.4 M ONITORING JOBS

The following is an example of job status operation, specifying 1 as verbosity level:

INFSO-RI-508833 PUBLIC 13/60

Job status changes:

*** Command Name = [JOB_REGISTER]

*** Command Name = [JOB_START]

> glite-ce-job-status --all -e grid005.pd.infn.it:8443 --from ’2005-07-23 10:00:00’ \

2.5 OTHER JOB MANAGEMENT OPERATIONS

> glite-ce-job-list grid005.pd.infn.it:8443

> glite-ce-job-cancel https://grid005.pd.infn.it:8443/CREAM115j5vfnf

cancels the specified job.

INFSO-RI-508833 PUBLIC 14/60

> glite-ce-job-suspend https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2

Are you sure you want to suspend specified job(s) [y/n]: y

> glite-ce-job-status -L 0 https://cream-ce-01.pd.infn.it:8443/CREAM11a79tnb2