Building Linux Clusters

Cluster Management & Administration Tools

Version 2.0BETA

David HM Spector
December 18, 2000
Acknowledgments.............................................................................................. 3

Introduction ........................................................................................................ 4

What’s in this Release?...................................................................................... 5

Known Issues and Bugs..................................................................................... 6

Installation Requirements .................................................................................. 7

Obtaining the Cluster Administration System ................................................. 9

Installing the Required Software..................................................................... 10

Installing the Cluster Administration System ................................................ 11

The Cluster Build Process ............................................................................... 12

Installing ClusterForge Step-by-Step............................................................. 13

Using the System.............................................................................................. 21

Connecting to Cluster Management System.....................................21

Logging in to the Cluster Management System................................22

Overview of the Cluster Management System .................................23

Adding a Node..................................................................................24

Viewing the Node Database .............................................................26

Showing a Node’s Characteristics ....................................................27

Adding a New Network Interface.....................................................27

Other Forms ......................................................................................28

Generating the DHCP File............................................................................... 29

SourceForge Menus and Commands.............................................................. 30

Creating Users .................................................................................................. 31

Appendix A : The ClusterForge Configuration File...................................... 33

Appendix B : The ClusterForge Crontab Entries .......................................... 36

Root tasks: ........................................................................................36

Tasks for clusteradmin......................................................................37

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Acknowledgments

SourceForge: The author would like to thank the SourceForge crew, and especially Tim Purdue,
for the creation of SourceForge – it’s a great system that will prove invaluable for developers of
all stripes. It was a fun challenge to write all of the portability code; I hope I can continue to
make valuable contributions to such an important project!

O’Reilly: An extra-special thanks to Laurie Petrycki, the executive editor at O’Reilly &
Associates for being so understanding about all the things that can conspire to get in the way of
getting stuff done.

Michelle Smith: (My wife) for putting up with all the stress and angst that accompanied parts of
this project, especially with a new baby in the house and the unbelievable amount of work that
entails.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Introduction

This document details the installation of version 2 of Building Linux Clusters “Cluster
Management System” software. This software is a replacement for the version shipped with the
1st Edition of Building Linux Clusters and adds significantly more functionality.

This new software release is based on the framework provided by SourceForge, an Open Source
collaborative development environment produced by VA Linux Systems and published under the
GNU General Public License (GPL). SourceForge provides a wide range of collaborative tools
for developing software, such as an easy to use interface to software version control (CVS), bug
tracking, discussion forums and a way to archive “code snippets” (useful code libraries that are
not necessarily part of a complete software package), and many other tools – all of which are
useful in the clustered environment where software development projects can quickly become
quite complicated due to the nature of parallel systems.

The Cluster Administration System components add a number of internal tables to the base
SourceForge1 system that aid in the management of Linux Clusters, including the ability to add,
edit and delete nodes to the cluster, the ability to manage batch queues and so on. The system is
deigned so that it is extensible and new functionality may be easily added without re-releasing (or
forcing you to re-install) the entire package. The additions have been made in such a way that the
cluster management database components are separate from the base SourceForge tables which
will make upgrading the SourceForge package itself easier as that package evolves.

Because of problems with the first version of the management software, the author has decided to
make this update available via the O’Reilly & Associates web site rather than wait for a new
edition of the book. This will enable faster updates, bug fixes and other new information to be
delivered to you more quickly. These updates will also be available via the author’s home web
server which can be found at http://www.zeitgeist.com/software/clusteradmin/ .

Finally, this document is a work in progress2 and is does not document all of the capabilities of
this system – it is meant to help you get your cluster up and running by getting the cluster
administration portion of the ClusterForge set up. There are many, many more features in this
package than are described/documented here.

A full description and complete manual along with an updated/replacement Cluster Management
chapter one in the 1st edition of Building Linux Clusters will be made available via O’Reilly and
from the author directly.

1
Occasionally I will refer to the Cluster Administration & Management system as “ClusterForge.” It’s a heck of a lot shorter and a
homage to its roots in the SourceForge code-base even if the whole “XXXXforge” thing is getting a little bit hackneyed.
2
It also has not been edited by O’Reilly & Associates, so any typos or other faux-pas are the author’s alone!

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 What’s in this Release?

The new release of the Building Linux Clusters Cluster Management tool kit is a completely new
system for managing your cluster. It is based on SourceForge, the Open Source collaborative
development system, and uses this package as the framework into which cluster management
tools are inserted.

The base Source Forge system provides a number of tools that can aid in the software
development efforts, including:

• A database centric, browser-based user environment that offers several levels of user
privilege
• A browser-based project description/creation system
• A browser-based interface to CVS, the Concurrent Versions System
• A bug reporting/tracking system that ties bug reports to projects or modules with projects
• A system for categorizing free-standing bits and pieces of code into a libraries that aid in
code-reuse

To this, the cluster administration tools add (at this first release):

• Node definition/management
• Cluster-node Interface Management
• Cluster user management

To these basic tools, any number of other facilities may be added; among those in the works are
modules for:

• Batch Queue Management (January 2001)

• Resource management: NIS groups, disk and CPU quotas (February 2001)
• DNS management for clusters (February 2001)
• Cluster-wide process accounting (March 2001)
• Cluster load monitoring (March/April 2001)

Any ideas and suggestions that you may have for other modules that would be useful in this
system would be greatly appreciated. You can send your ideas to:

blc-clusteradmin@zeitgeist.com

Code contributions and bug fixes are also welcome and can be sent to the same address.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Known Issues and Bugs

As with any (every?) software package, ClusterForge has some bugs and other issues. Here’s the
list of issues for the 2.0Beta version:

1) Display quirks: There are small gray artifacts on some of the ClusterForge screens. This
is due to bugs in the theme support in the SourceForge code base. I’ll be trying to nail it
down, but the theme support in SourceForge is not fully baked.

2) SourceForge administrative interface is really hard to use. I’ll be adding both

documentation and better UI elements over time.

3) There is no sanity check of the host IP addresses. At this release the add-node.php3
code does not check to see if the IP address you enter is on the same network as the
master node.In order for DHCP (and, in fact for the cluster as a whole) to work, the
primary interface of each node must be on the same network as the master node of the
cluster.

4) There is no sanity check for the hardware address. At this release the add-node.php3
code doesn’t check to see if the format of the hardware address matches the kind of
interface you’ve specified in the form (i.e., ETHn [Ethernet] type addresses should have a
48-bits of address). There is support in the cluster_node_interfaces table for
this feature and this will be fixed in a subsequent release.

5) The software manuals included as part of the cluster documentation (e.g., MPI, PVM,
PADE, etc.) are vanilla HTML and have not yet been modified to match the look and feel
of the rest of ClusterForge. This will be fixed in an upcoming release.

6) The configuration file has a flag that controls whether or not shell accounts are
automatically created – in this release this feature is not implemented. Shell accounts are
always created when new users are registered with the ClusterForge system.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 7)

Installation Requirements

In order to install and use this version of the Cluster Management & Administration Tools you
will need a number of other ancillary software packages. In the next release of the printed book
these packages will be included on the CD-ROM, but for this stand-alone release, you will need
to FTP these packages yourself3.

Package Description Version Download Location

MySQL SQL Database >= 3.22 http://www.mysql.com/downloads/
MySQL Supporting libraries >= 3.22 (same as above)
shared for the database
MySQL Client interfaces >= 3.22 (same as above)
client
Perl DBI Perl database >= 1.14 http://www.cpan.org/authors/id/TIMB/
Interface independent interface
Perl MySQL The MySQL >= 1.2 http://www.cpan.org/authors/id/JWIED/Msql
Interface database driver for -Mysql-modules-1.2215.tar.gz
DBI
PHP version The scripting >= 3.02 http://www.php.net/
language used by this
packages
GNU Mailing list >= 2.0 http://www.gnu.org/software/mailman/mailm
Mailman processing software an.html
CVS Version control >= 1.10 http://www.cvshome.org/ Or, in the
software RedHat/RPMS directory on the Building
Linux Clusters CD-ROM
CVSWeb Web interface to >= 1.9 http://www.cs.cornell.edu/nogin/RPM/cvswe
CVS repositories b.html
OpenSSL* HTTP encryption >= 0.9.4 http://rpmfind.net/linux/RPM/openssl.html
*
Only required if you need secure http
connections
Apache+ Apache SSL Module >= 1.37 ftp://ftp.MASTER.pgp.net/pub/crypto/SSL/A
mod_ssl OR *OR*, pache-SSL/ *Only required if you need
ModSSL* >= 2.4.10 secure http connections.

3
We would include them in this online distribution, but its always best to get software from is source. Good places to find any/all of
this software is http://www.freshmeat.net/ or http://www.rpmfind.net/ or http://www.sourceforge.net/ . The PDF version of this
document includes hyperlinks that can be used to download these packages directly; the URL’s are also included in case the PDF
reader you are using doesn’t support hyperlinks or you are reading a hardcopy version of this document.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

There are other packages that are very useful in debugging problems or general poking around,
but not required in order to use the cluster management system – here are some that I have found
useful:

Optional/Useful Software
Package Description Version Download Location
PhpMyAdmin Browser >= 2.1 http://phpwizard.net/projects/phpMyAdmin/inde
interface to x.html
MySQL
Ddd DataDisplay >= 3.2 http://www.gnu.org/software/ddd/
Debugger

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Obtaining the Cluster Administration System

The Building Linux Clusters Administration System may be obtained from two
sources:
O’Reilly & Associates FTP site: ftp://ftp.ora.com/published/oreilly/linux/clusters
The author’s web site: http://www.zeitgeist.com/software/clusteradmin
The Cluster Administration System is distributed as GNU zip’ed tar image; it will have a file
name in the form of BLC-ClusterAdmin-X.Yreltype.tar.gz, where “X.Yreltype” is a
release number and a kind of release. For example:

BLC-ClusterAdmin-2.0beta.tar.gz

This represents the 2.0 beta release of the software. This distribution contains all of the cluster
management software. As new modules are released and/or updated, there will be packages
released that represent individual modules; these will be of the form:

BLC-ClusterAdmin-modulenameX.Yreltype.tar.gz

These will be drop-in replacements for existing modules, or new functionality that will come with
their own upgrade or installation instructions. This document covers the installation of the whole
package, not any of the individual modules.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Installing the Required Software

Before attempting to install the new version of the cluster management software, you should
install (and configure) all of the “required” software packages listed in table 1.
It is strongly recommended that wherever possible that you obtain these packages in the form of
RPM (RedHat Package Manager) files.

The packages should be installed in the following order:

1) MySQL packages
2) Perl packages, starting with the MSQL-MySQL modules, followed by the Perl DBI
package, then the MySQL driver for DBI
3) GNU Mailman
4) CVS and CVSWeb. You should make a symbolic link from the root to wherever you put
your CVS repository on your system. For example, if your CVS repository is on the device
named “/spare” you would make the link by invoking “ln –s /spare /cvsroot” as
root.
If your CVS repository is on its own device, you could simply mount it as “/cvsroot”
5) OpenSSL/ModSSL is required at your site

PHP should have been installed already on your master-node by the Building Linux Clusters CD-
ROM installation software. If, for some reason, it is not installed, the RPM files can be found in
the RedHat/RPMS directory on the CD-ROM.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Installing the Cluster Administration System

Before installing the cluster administration system itself, all of the software listed in the
“required software” table must be installed. Each of the packages come with test-scripts and/or
instructions that will test the installation of the package. Make sure that all of the packages are
successfully installed before attempting installation of the Cluster Administration System itself.
If the required packages are not installed, trying to get the rest of the system to work properly will
be difficult if not impossible.

Once the required software has been installed, work can begin on the installation of the Cluster
Administration System itself.

It is recommended that you unpack the distribution into a temporary directory, such as /tmp .
The distribution kit will unpack into a directory that has the same name as the package you are
unpacking. Inside this directory will be a README/INSTALL file (i.e., this document!) that
describes the software, what is required to use it, and how it should be installed.

The software can be unpacked with the command:

tar zxvpf filename

Where filename is the file you have retrieved from one of the sources listed above. It is
important to include the “p” option in the tar command – this preserves the file permissions that
were recorded in the tar file. Since the files are for the most part executable scripts, if the
permissions are incorrect, the Apache web server will refuse to execute them. This can be time-
consuming and difficult to debug

The tar command will show you all the files it is unpacking as it unpacks them. At the end of the
process you will have a new directory called BLC-ClusterAdmin-X.Yreltype .

This directory will contain installation instructions (this document) in PDF (Adobe Acrobat) and
HTML formats, and four directories, “db” , “etc” , “html” and “utils” .

The db directory contains the SQL schema required to instantiate the Cluster Administration
Database.

The etc directory contains configuration files that define the operation of the system

The html directory contains the code that makes up the bulk of the system.

The utils directory contains perl and other scripts that are run by cron in order to implement
various parts of the administration system and the development environment such as account
creation and node management.

The rest of this section details in a step-by-step fashion how to install the software and get it
running.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 The Cluster Build Process

If you have already built a cluster using the procedure outline in the 1st edition of Building Linux
Clusters, skip ahead to “Installing ClusterForge Step-by-Step” and continue from there. You
already are far enough along that you don’t need to use the ClusterForge for preliminary setup of
your slave/compute nodes.

If you have not yet set up your cluster, you should follow the process outlined in the book, but
stop once you have gotten the master node set up, and when you have made the boot floppies for
the slave/compute nodes.

When you are ready to bootstrap the slave/compute nodes, pick up here in the ClusterForge
installation section (skip chapter 6, Cluster Management, this new software replaces the version
that comes with the book). Using the new software will help you set up all your slave nodes
without having to re-enter any information ones you get the slave nodes set up.

Once you have completed the software installation described in this document, and told the
cluster administration database about your compute nodes you can pick up where you left off in
Chapter 5, Software Installation and Configuration.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Installing ClusterForge Step-by-Step

Step 1: Log in as root

In order to install and set up this software, you will need to log in to the master node of your
system as root. All of the installation and setup work will be done as superuser.

Step 2: Backing up existing materials

If you have installed a previous version of the cluster management tools (for example, from the
first release of the Building Linux Clusters CD-ROM), you should back up that release into a safe
place. This can be done by copying the /home/httpd/html directory to another device and
then recursively removing the html directory, or simply by renaming the directory to another
name. For example:

{root}# cp -Rp /home/httpd/html /anotherdev/html-SAVE; \

rm –rf /home/httpd/html

will copy, and then remove the old directory, or,

{root}# mv /home/httpd/html /home/httpd/html-SAVE

will rename the old directory. If you are at all short on disk space on the device where /home
resides, it is a better idea to take the first course rather than just renaming the directory.

Step 3: Deactivating PostgreSQL

The previous version of the software that shipped with the 1st edition of Building Linux Clusters
made use of the PostgreSQL database. The new version uses a different database that was used in
the development of SourceForge, upon which the new version of these tools is based.
PostgreSQL is therefore no longer necessary (unless you are using it for some other purpose on
your system, in which case it should be left alone).

PostgreSQL can be shut down with the following command:

{root}# /etc/rc.d/init.d/postgresql stop

Next, it would be a good idea to stop PostgreSQL from starting up at system boot-time so that a
process that isn’t being used by anything does not consume CPU time and other resources. This
can be done by deactivating postgresql’s init script with the command:

{root}# /sbin/chkconfig --level 345 postgresql off

Step 4: Installing the new HTML directory

The html directory included with the new distribution contains a complete version of the
administration system and all the manuals described in Building Linux Clusters. To install this in
its proper place, cd to the distribution and then recursively copy the html directory to
/home/httpd/, as in:

% cd /tmp/BLC-ClusterAdmin-X.Yreltype

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

% cp –Rp html /home/httpd/

These commands copy the entire directory structure, to the destination, preserving the file
permissions that were in the source.

Step 5: Instantiating the Cluster Administration Database

In order to function, the scripts and HTML files installed in the previous step work by accessing a
database where information about your cluster is stored. With MySQL n a new database is
created using the mysqladmin program. You can call the name anything you like; the
configuration files supplied with this distribution use the name “alexandria” (the same name as
SourceForge uses) – if you decide on another name, you will have to update the configuration file
which is covered in Step 6. To create the database, execute the following command:

{root}# mysqladmin create alexandria

Mysqladmin will respond with a message indicating the database has been created.

If you want to password protect this database (a good idea in multi-user environments), you can
set the password with the following command:

{root}# mysqladmin password “somepassword” alexandria

Where somepassword should be replaces by the password you wish to apply to this database.
Make sure you remember what this is – there’s no way of recovering it if you lose it! Without
this password, anyone who can make a connection to your machine can connect to your database
and execute any command whatsoever (i.e., they could delete everything without a trace!).

Step 6: Loading the Cluster Administration Database

Once the database has been instantiated, we need to load in its schema and its default values. The
database SQL file is in the “db” directory of the distribution and is named clusterforge.sql. To
load the database tables, execute the command:

{root}# mysql –p somepassword dbname < clusterforge.sql

Dbname is the name of the database when you created with the mysqladmin command and
somepassword is the password that controls access to this database.

The mysql command will digest the contest of this file silently. If you wish to see what database
looks like, you can execute the following command:

{root}# mysql –e “show tables” –p somepassword dbname

MySQL will show you a very long listing of tables that exist in the database that looks like this:

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

+-----------------------------------+
| Tables in alexandria |
+-----------------------------------+
| activity_log |
| activity_log_old |
| activity_log_old_old |
| bug |
| bug_bug_dependencies |
| bug_canned_responses |
| bug_category |
| bug_filter |
| bug_group |
| bug_history |
| bug_resolution |
| bug_status |
| bug_task_dependencies |
| cluster_cpu_types |
| cluster_info |
| cluster_interface_names |
| cluster_node |
| cluster_node_accounting |
| cluster_node_interfaces |
| cluster_node_packages |
| cluster_project |
| cluster_queue_info |
| cluster_queue_node_membership |
:
<rest of very, very long list omitted…>

This is an abbreviated listing of some of the names of tables in the database. Also loaded are the
default values needed to set up the system that are not shown here, but they are stored in the
database tables.

Once the tables have been created and the default values loaded, we can proceed to configuring
ClsuterForge for use.

Step 7: Installing the ClusterForge Configuration File

ClusterForge, like SourceForge upon which is based, uses an extensive configuration file to tell
the system where to find the resources it needs to operate. In the ClusterForge distribution
directory, in the “etc” subdirectory, you will find a file named local.inc,this file needs to be
installed in the master node’s /etc/ directory, and made readable only by root and root’s group.
For example:

{root}# cd /tmp/LC-ClusterAdmin-X.Yreltype
{root}# cp /tmp/LC-ClusterAdmin-X.Yreltype/etc/local.inc /etc/
{root}# chmod 660 /etc/local/inc

This will ensure that unauthorized processes and users cannot read the configuration file.

Step 8: Editing the Configuration File

Once the configuration file is in the right place, we’ll need to set up all of the parameters
correctly for your master node’s configuration. The file is extensively commented, and included
in its entirety as Appendix A of this document. In this section we’ll cover exactly what you need

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

to change in order to get your system running. It is strongly recommended that you do not
change any other flags or settings in this file – doing so can lead to disappointing results.

The local.inc file is divided into 4 sections that control various parts of ClusterForge’s
operations. These are:

• Preliminaries – SSL Security

• Part I – SourceForge Hostnames
• Part II – Databases html/php paths
• Part III – GUI Modifications

Since the Cluster Administration System is based on SourceForge you will see that name used
throughout the configuration file – just pretend it says “ClusterForge.” :-)

We are only concerned with Parts I & II, where we will specify hostnames and paths used by the
system, specifically:

From Part I:

$sys_default_domain = "master.cluster.ny.zeitgeist.com";
$sys_cvs_host = "master.cluster.ny.zeitgeist.com";
$sys_download_host = "master.cluster.ny.zeitgeist.com";
$sys_shell_host = "master.cluster.ny.zeitgeist.com";
$sys_users_host = "master.cluster.ny.zeitgeist.com";
$sys_docs_host = "master.cluster.ny.zeitgeist.com";
$sys_lists_host = "master.cluster.ny.zeitgeist.com";

This collection of host names specifies which hosts perform what functions in the cluster
administration system.

All of these variables should be set to the hostname of the master node.

As you can see from the examples here, the name of the master node of my cluster is
“master.cluster.ny.zeitgeist.com” – although that’s pretty obvious from the text, I am being
purposefully explicit here because it is very important that all host names used in the
configuration of this system use fully-qualified-domain-names (FQDNs). This is so that there is
no ambiguity about what host you are naming, and so that someone accessing the administration
system from a machine with a slightly broken domain name resolver doesn’t get confused and fail
to find a required host. Another reason to use FQDNs is that, if necessary, the parts of the
administration system and many of the SourceForge functions can be run over several hosts.

For example, if your corporate source code repository lives on some big server with a large disk
farm, you could put the name of that server here instead of the cluster’s master node. Just make
sure that CVS is installed properly there otherwise the ClusterForge source code repository will
not function correctly.

The next important host names involve the domain name hosts that will be used by the system:
$sys_dns1_host = "master.cluster.ny.zeitgeist.com";
$sys_dns2_host = "master.cluster.ny.zeitgeist.com";

These hosts specify the DNS servers that know about the cluster. It is usually the master node
itself that is usually configured as a DNS server for the cluster so as not to clutter up the

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

organizational name space with all of the cluster nodes when all the main network needs to know
about is the master node. Conversely, the cluster administration system needs to know about all
of the nodes of the cluster if it’s going to be able to administer them. The second DNS host can
be set to something that is “up a level” that knows about the larger organizational network, if any.
This would allow for a faster name resolution of objects like the hypothetical corporate CVS
server mentioned previously.

From Part II:

The next section deals with the database and the places files are accessed or stored.
$sys_dbhost="localhost";
$sys_dbname="alexandria";
$sys_dbuser="root";
$sys_dbpasswd="";
$sys_server="mysql";

Here in $sys_dbname is where you need to specify the hostname you passed to mysqladmin to
create the database; if you gave the database a password, you should specify that in
$sys_dbpasswd, or the ClusterForge code won’t get access to the database. None of the other
variables should be changed.

Next we’ll specify where users can upload files to be added to the CVS repository. This must be
an absolute path name, for example:
$FTPINCOMING_DIR = "/nfs/ftp/incoming";

The next variable specifies the location where files can be downloaded. This is the root of the
FTP tree for projects stored under ClusterForge.

$FTPFILES_DIR = "/nfs/ftp/sourceforge";

These next two variables tell the ClusterForge code all about accessing itself. If these are not set
properly, ClusterForge will fail in various mysterious ways, which are almost impossible to
debug (it’s a PHP problem that is probably fixed in PHP 4.x with its debugging hooks).

$sys_urlroot="/home/httpd/html/";

This is an absolute pathname to where you copies the HTML directly back in Step 4. Make sure
you leave the trailing slash (“/”); this is used extensively by the code in forming URLs.

$sf_cache_dir = "/tmp/sfcache";

This is an absolute pathname to a directory that is used by ClusterForge to write a set of cache
files that store various chuncks of HTML that don’t often change. This is done because
ClusterForge (and SourceForge) is actually a very large server-side PHP application, and every
time a user accesses the ClusterForge home page, or their personal home page (more on this
later!), large amounts of HTML would have to be rebuilt on the fly. For a large numbers of users
this would have a very negative impact on the performance of the web server.

Set this variable to a device that has free space and is world-readable – the /tmp file system is a
good place.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Finally, save the changes to the file.

Step 9: Creating the Cache Directory

As mentioned in Step 7, ClusterForge needs a directory in which it can place cache files; you will
need to make this directory and set its access permissions so that it can be read by the world, but
only written to by root and the web server, for example:

{root}# mkdir /tmp/SFcache; chmod 775 /tmp/SFcache

{root}# chgrp nobody /tmp/SFcache

Step 10: Setting the PHP Path

In the Apache configuration file, the PHP include path needs to be set so that ClusterForge can
find all of its libraries and utility code; this done by adding a line to the default directory directive
in the Apache configuration file.

Edit the file /etc/httpd/conf/httpd.conf, and look for the default directory, which
should start with a line that looks like:

<Directory "/home/httpd/html">

Scroll down the file until you find the closing tag for this directive, which is a line that looks like:

</Directory>

and, then, insert the following line:

php3_include_path /home/httpd/html/include:.

before the closing tag. Save the addition and restart the web server with the command

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 {root}# /etc/rc/init.d/httpd restart

This will restart the web server and make web portion of the ClusterForge system ready for use.

Step 11: Creating up the clusteradmin User

If you have not already done so as part of the cluster installation process specified in Chapter 6 of
Building Linux Clusters, you should create a clusteradmin user. This can be done with the
command:

{root}# useradd -G clusteradmin –p password clusteradmin

This will create the user “clusteradmin,” along with a new group of the same name, with the
password specified.

Once the clusteradmin user has been created, find out what Unix UID has been assigned to it
during the creation process, this can be done by using the grep command, as follows:

{root}# grep clusteradmin /etc/passwd

the result should come back looking very much like this:

clusteradmin:$1$0RTdMI8d$jq0O7SX6WDxX2tZ6eVhM4.:501:502::/home/clusteradmin:/bin/bash

The “501:502” in the sample output above are the User ID and the Group ID of the clusteradmin
user – the values on your system will probably be different. Make note of the User ID, it will be
needed for the when we set up the background jobs.

Step 12: Copying the clusteradmin Utilities

Beyond the visible portion of the cluster management system there are a large number of
background processes that run at various times to do the work specified by the cluster manager,
and keep the various project-management and collaboration subsystems in proper working order.
From the ClusterForge distribution directory, recursively copy the utils sub-directory into
clustadmin’s home directory. For example:

{root}# cp /tmp/LC-ClusterAdmin-X.Yreltype/utils ~clusteradmin/

Step 13 Installing the Batch Jobs

Now that the utility files are in their final destination, there are just a couple of tasks to complete
before the cluster management system can be brought on-line, all of which need to be done from
the clusteradmin user’s home directory. Go there with:

{root}# cd ~clusteradmin/

In the clusteradmin directory, a subdirectory needs to be created to hold data extracted from the
cluster administration database. Create this directory as follows:

{root}# mkdir dumps

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Make sure that the clusteradmin user owns this directory with the command:

{root}# chown cluster dumps

Next, modifications need to be made to an include file that is common to all of the background
processes, this file is named include.pl in the utils directory that was just copied into the
clusteradmin user’s home directory. Edit this file and look for the line that starts:

$dummy_uid = "503";

The “503” here must be changed to the Unix User ID that you recorded for the clusteradmin user.
This setting tells the background processes what user owns various work files – without it the
background processes cannot run.

Save your changes to this file.

Finally, the commands that run the batch jobs themselves are stored in two different files: one file
for jobs that need to run as the clusteradmin user, and other for jobs that need to run as root, with
full superuser privileges. These batch jobs are represented by the files crontab-clusteradmin.sf
and crontab-root.sf .

Install the cron jobs with the following commands:

{root}# crontab –u clusteradmin crontab-clusteradmin.sf

{root}# crontab crontab-root.sf

For the most part these cron jobs will never need to be changed, except in two instances: First, the
interval for account creation. The default is to scan the database every 6 hours for new accounts
to create. If your installation is very busy, or you think that 6 hours is too long to wait for user
creation the timing can be changed as per the comments in the batch files. The second instance is
the interval for new-node activation. The cron file contains a batch job (dump-dhcp.pl) that scans
the cluster administration database every 6 hours looking for nodes that are either new, or have
had they primary network interfaces updated. When this job finds such hosts in the database, it
writes a new /etc/dhcpd.conf file and restarts the dhcp daemon which will allow these
new/updated hosts to be reconfigured when they are booted. If you have added the information
for a new node into the ClusterForge and the node fails to bootstrap itself when you power it on,
you will most likely want to run this job by hand.

Once these cron jobs have been installed, the Cluster Administration system should be
operational. You should proceed to the next section and log in to start using the system to set up
your cluster.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Using the System

The best way to see if everything successfully configured is to log in to the system and try it out.

This section will get you stared using the system by showing you the basics of adding nodes and
interfaces to your cluster. Once we have these basic tasks out of the way, the rest of this
document will cover very specific tasks involved in the other components of ClusterForge, those
that are part of the SourceForge code base.

&RQQHFWLQJWR&OXVWHU0DQDJHPHQW6\VWHP

Connecting to the management system is as easy as pointing a browser at the master node. Start a
browser, and connect to the master node of your cluster. If the web server is set up correctly, you
should see the web page shown in figure 1:

Figure 1

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Some details, such as the number of cluster hosts shown will differ (because no nodes have yet
been entered into your version of the cluster management database or because your cluster may
contain a different number of nodes).

/RJJLQJLQWRWKH&OXVWHU0DQDJHPHQW6\VWHP

Logging in to the system is quite simple; an account for the cluster administrator has been
included in the initial data that was put into the database when you loaded the tables and other
default information. To start the login process, click the “login” label that will bring up the login
panel, shown in figure 2.

Figure 2

Into the “login name” area, enter the account name of the cluster administrator, in this case
“admin” – the password is the same default password that is set up at cluster build-time:
“mr.linux”. Enter the password, and press the “Login” button.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

2YHUYLHZRIWKH&OXVWHU0DQDJHPHQW6\VWHP

Once logged in, there are a number of options available, as shown in figure 3. As you can see, the
browser window is divided into two areas: The left side is dedicated to action-oriented items
such a commands, and menus, while the right side is used for informational elements, such as
information about tasks that may be assigned to you based on projects you are a member of, and
so on. In this first release, almost all of this information is that same information one would see
in SourceForge – subsequent released will all a large number of cluster monitoring tools and
other information elements useful in the clustered environment.

The items of greatest interest to the cluster administrator are the SourceForge Management and
the Cluster Management Menus. We’ll focus first on the cluster management portion of the
system since that is where information about your cluster will reside.

Figure 3
Select the Cluster Management menu to view the options in the cluster management section. This
will bring up the cluster administration menus, shown in figure 4.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Figure 4
The cluster management screen is where all of the functions related to the management of cluster
nodes can be found At the moment, the modules supplied with ClusterForge handle the creation
and management of cluster nodes only. Upcoming modules will add to this functionality. From
this screen it is possible to list all of the nodes currently defined in the cluster, or add a new node.
Since listing all the nodes of an empty cluster will not get us very far, we’ll start by adding a new
node to the cluster.

$GGLQJD1RGH

Adding a node to the cluster administration database is quite simple, and requires just a few
pieces of information, which you should have ready since you have either built your
slave/compute nodes or are ready to start that process. You will need:

• The names of the slave/compute nodes

• The IP address you wish to assign each node
• The name of the primary Ethernet card (e..g. “eth0”)
• The hardware address of the primary Ethernet interface of each node

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Optionally, you might wish to have the following extra information available:

• System manufacturer name

• System serial number
• Number, kind and speed of CPUs
• Memory configuration
• Amount of available disk space
• Name, hardware address and IP address of secondary network cards

If you have all the information on the slave nodes you wish to install, click on the “Add a node”
link. This will bring up the display shown in figure 5.

Figure 5

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Fill in the required information listed above; the optional information, though not required, will
be used by other reporting modules that will be available in upcoming releases.

The form displayed will look very much like the one displayed here, but may differ slightly
depending upon the fonts installed on your system; the domain name shown in the form will, of
course, be the domain name you set up when you configured your master node.

Once you are satisfied with the information you have entered, press the “Insert” button and the
node will be entered into the cluster administration database. If you want to enter more compute
nodes into the database, you can press the “BACK” button in your browser, modify the form and
insert more nodes. Once you have entered a node or two, we can look at the listing of the cluster
database to see what’s been recorded. Click on the link labeled “Cluster Management Home” to
return to the top level of the cluster management area.

9LHZLQJWKH1RGH'DWDEDVH

To see what the is in the database, click on the link labeled “List cluster nodes…” and a listing
very much like figure 6 will be displayed:

Figure 6
This listing gives a capsule summary of the nodes in the cluster, the kinds of processing power
they have and what the configuration of their primary network interfaces is.

On the right-hand side of the screen there are two additional links next to each node, one allows
you to bring up a form to edit the node’s information and the other allows you to delete the node
from the database. These should be used with case as they cn seriously affect the operation of
your cluster. The delete link will prompt you before taking any action, so there is no possibility
that you will accidentally remove a node.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

6KRZLQJD1RGH·V&KDUDFWHULVWLFV

To see a more complete listing of a node’s facilities (in fact, all the information you entered in the
“Add Node” form), you can click on any of the node names in this list. This will bring up a
display similar to the one shown in figure 7:

Figure 7
Of course, your nodes will have different characteristics (and probably real IP addresses, unlike
the dummies used here!). A useful feature of the “show node” display is that you can see other
network interfaces that installed on the node, edit them, add new interfaces, or even delete
interfaces. Of course, editing or deleting interfaces should be done with case as such actions can
disrupt the cluster (and probably your users, too).

$GGLQJD1HZ1HWZRUN,QWHUIDFH

Adding a new interface is started by clicking on the “Add interface to this node…” link in the
“ShowNode” display. This will bring up a display like the one shown in figure 8:

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Figure 8
Editing an interface, unlike adding a new node, requires only three pieces of information:
• The network device name
• The IP Address
• The Hardware address

The network device name is selected here via a pop-up menu that lists 17 different kinds of
network interface types, the other requisite information is entered directly in the text boxes
provided.

On non-primary interfaces only the IP address and device name are critical since you cannot
normally boot of one of these interfaces. However, the system will not allow you to enter a bogus
IP address, or use the name of a device that already exists on node you are editing. Even though
you can’t boot from the device DHCP information is generated so that the network device can be
automatically configured at system boot-time.

2WKHU)RUPV

There are several other forms in the cluster administration section, but they are all variants of the
forms that have been shown here and are used for editing already entered information. Rather
than go over these forms in detail, we will move on to the process of getting the DHCP
information extracted from the database so you can get your slave/compute nodes up and running.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Generating the DHCP File

The /etc/dhcpd.conf file contains the information needed by the master node in order to
bootstrap the slave/compute nodes and finish the installation of your cluster.

The information that you have entered into the cluster administration database is not very useful
if it can’t be used to get cluster nodes configured. The way that that this information is translated
from the database into the DHCP system is by way of a small perl program that runs as part of a
larger set of utility scripts that you installed when you installed ClusterForge. This Perl script
looks at the existing DHCP configuration file installed on the master node and makes note of its
last modification time. The script then looks at the all of the network interfaces described in the
interfaces portion of the cluster database and if it finds any interfaces that are newer than the
DHCP file it writes out a new version of the file and restarts the DHCP daemon so that nodes can
be initialized.

For your purposes at the moment, if you have just added a new node to the database, there is one
problem with this scenario: the process that checks the database only wakes up every six hours.

In the normal, day-to-day operation of your cluster its is (hopefully) unlikely that you will be
adding new nodes or interfaces several times a day, but the process is there as a utility to catch
new nodes or interfaces as they are added. But, this 6-hour wait does little for you in testing your
cluster:

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 SourceForge Menus and Commands

All of the non-cluster specific components of ClusterForge are part of the SourceForge tool-set
that forms the framework for the clustering tools that share their space. These tools allow a wide
variety of activities, from discussion forums, to source code and release control though bug
tracking and the creation of code libraries that aid in code re-use.

From the perspective of setting up your cluster there are really only two aspects of the
SourceForge tools that really important: The first the SourceForge Administration menu, shown
in figure 9, and then second is the new-user form, shown in the next section in figure 10.

The SourceForge administration menus allow you to examine the state of user accounts and
manage development groups.

Figure 9

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Creating Users

The most important capability, next to being able to get the cluster running, is the ability to get
user accounts set up. Traditionally this required a lot of work on the part of the system admin to
create accounts, set up directories, etc.. Following the SourceForge model, ClusterForge allows
the users to request/set-up account themselves by connecting to the system and selecting the
“register as a new user” link on the home page.

Filling in the new account registration screen will generate a new user account in about 6 hours,
which is the in which interval that the batch-job that monitors accounts is run. This delay exists
so that the password files and NIS maps are not constantly being updated which would disrupt
general use of the cluster.

Figure 10

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

Once the batch process has created a user account, a user may log in and begin using the cluster.
No further action is required for an individual’s use of the cluster.

There is a lot more that can be done with ClusterForge and SourceForge-based project
management and collaboration tools, but this should get you started with the cluster
administration components.

A more complete manual for ClusterForge and all its components will be published in coming
months, along with more modules that implement new pieces of the cluster administration
functionality. These will be made available via the O’Reilly & Associates web site and directly
from the author as mentions in the introduction to this document.

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Appendix A : The ClusterForge Configuration File

<?php
// SourceForge Universal Site Modifications
// by David HM Spector <spector@zeitgeist.com>
//
// Really Important Safety Tip: --> DO NOT LEAVE ANY WHITE
// SPACE AFTER THE CLOSING PHP TAG AT THE END OF THIS FILE!
//
// Doing so will really confuse the software and cause
// 1) cookies to fail and 2) HTML page headers to fail
// which will give you some preally hard-to-debug problems.
// Why? PHP is a *pre-processor* -- anything that’s not PHP gets
// emitted as part of the HTML stream and processed by the browser,
// so white space is meaningful!
//
// Preliminaries: Security
//
// It would be a "good thing" if your web server had a SSL certificate so that
// users’ connections to the cluster administration system were encrpyted.
// However, sensible people realize that not everyone wants to spend a few
// hundred dollars every year on a new certificate and many clusters are on
// private networks where this isn’t an issue. If you have a cert, set this
// variable to "1", otherwise leave it at the default of "0" (zero).
//If you turn this on an you have no ssl-enabled http server running you
// won’t be able to login to the cluster adminstration system and you’ll
// get very frustrated. You have been warned!
//
$sys_use_ssl = 0;
//
//
// This flag controls whether or now users’ can create their own SF accounts, or
// if account creation MUST be done by an administrator.
//
$sys_user_created_accounts = 1;
//
//
// This flag control whether the background cron jobs use the “/etc/passwd”
// and shadow password files, or if they put user name entries into
// “/var/yp/src/passwd”
// The default is to use the YP files.
// **DON'T CHNAGE THIS UNLESS YOU KNOW WHAT YOU'RE DOING (If this is
// changed, users will be created in /etc/passwd which will stop them from
// using the cluster, which depends on NIS)
$sys_use_yppasswd = 1;
//
// PART I - SourceForge hostnames
//
// Hostnames should be fully qualified domain names (FQDNs); using short names
// would be prettier but would stop you from distributing your SourceForge
// implementation across multiple domains.
//
// Of course, if you have a lot of machines serving a particular purpose
// such as FTP or for shell accounts, the "hostname" here might be in
// reality an addr_list of machines that is serviced by a round-robin
// mechanism or something fancy like a local-director.
//
// The default SourceForge domain
// this is used where ever the "naked" form of the SourceForge domain
// might be used. E.g., "mailto:admin@sourceforge.net"

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

$sys_default_domain = "master.cluster.ny.zeitgeist.com";

// Machine that hosts CVS

$sys_cvs_host = "master.cluster.ny.zeitgeist.com";

// Machine used for downloading sources/packages

$sys_download_host = "master.cluster.ny.zeitgeist.com";

// Machine(s) that host users’ shell accounts

// N.B. to the SourceForge Crew: What’s the difference between the user
// host and the shell host? They are clearly two different hostnames
// in the source code, but they seem to serve the same purpose..?
$sys_shell_host = "master.cluster.ny.zeitgeist.com";
$sys_users_host = "master.cluster.ny.zeitgeist.com";

// Machine that hosts docs (such as the FAQs and the various software
// licenses (*BSD, [L]GPL, etc.). You REALLY want this to be the same
// machine that the SourceForge code is running on because all of the
// PHP makes reference to these documents in terms of relative paths that
// are part of the SourceForge code tree.
$sys_docs_host = "master.cluster.ny.zeitgeist.com";

// Machine that hosts the SourceForge mailing lists (This could also be
// the mail host if you have enough horsepower & bandwidth)
$sys_lists_host = "master.cluster.ny.zeitgeist.com";

// Domain Name Servers

// N.B.: Use terminated FQDNs here (with the final ".") so the resolver
// doesn’t attempt to recurse in the case of a slightly broken DNS
// configuration
$sys_dns1_host = "master.cluster.ny.zeitgeist.com";
$sys_dns2_host = "master.cluster.ny.zeitgeist.com";

// Part II - Databases, html/php/other paths

$sys_dbhost="localhost";
$sys_dbname="alexandria";
$sys_dbuser="root";
$sys_dbpasswd="";
$sys_server="mysql";

// Where files are placed when uploaded

$FTPINCOMING_DIR = "/nfs/remission/u7/ftp/incoming";

// Where the released files are located

$FTPFILES_DIR = "/nfs/ftp/sourceforge";

// Where the SourceForge files are placed

// *** IMPORTANT: sys_urlroot *MUST* be an ABSOLUTE FILEYSTEM PATH NAME
// that points to the www directory of the SourceForge
// installation. If you use ANY form of relative path
// you will break the html_image function in include/html.php
//
$sys_urlroot="/home/httpd/html/";

// Cache location -- this is needed by include/cache.php

// This directory must be world reabable, but writable only by the web-server
$sf_cache_dir = "/tmp/sfcache";
// Name of the system as a whole (needed by various utils and titles)
$sys_name = "Building Linux Clusters";
$sys_shortname = "BLC";

// Part III - GUI modifications (menu colors, etc.)

// See the top of the file include/html.php, this is where the menu colors

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

// and colors used throughout SourceForge are defined.

// Themeing related vars... Some of this needs to change in the session stuff
// The theme base directory, everything else is handled by theme_sysinit()
$sys_themeroot=$sys_urlroot."themes/";

// End of customizations -- place nothing after the closing PHP tag!

?>

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

 Appendix B : The ClusterForge Crontab Entries

5RRWWDVNV

# periodic clusterforge tasks for root

# NB: If you have more crontab files for root add this file to
# the worklist (i.e., a master crontab) for root, otherwise you
# will REPLACE the exiting crontab with this one -- probably
# not what you want to do...
#
# $Id: crontab-root.sf,v 1.6 2000/12/20 05:17:05 spector Exp $
#
# crontab times are written as follows [see also crontab(5) ]
# field allowed values
# ----- --------------
# minute 0-59
# hour 0-23
# day of month 1-31
# month 1-12 (or names, see below)
# day of week 0-7 (0 or 7 is Sun, or use names)
#
#
# this first entry is really important -- it’s what makes the users that
# are created inside clusterforge.
#
10 */6 * * * cd /home/clusteradmin/utils && ./new_parse.pl ; cd /var/yp && \
make >/dev/null
#
# Look for new or updated nodes in the clusteradmin database and write them
# out into the /etc/dhcpd.conf file. Really this should only be done when
# new nodes are added, but it costs very little to run this job since it only
# runs if there is work to do...
0 */6 * * * cd /home/clusteradmin/utils/underworld-root && ./dump-dhcp.pl

# These are the nightly jobs that take care of updating the counts
# for items in the code snippets (called "the trove" in the code) and check
# on the statis
# of jobs in various priject "to do: lists
0 * * * * cd /home/clusteradmin/utils/underworld-root && ./db_trove_treesums.pl
0 2 * * * cd /home/clusteradmin/utils/underworld-root && ./stats_nightly.sh >/dev/null
0 2 * * * cd /home/clusteradmin/utils/underworld-root && ./db_jobs_close.pl

# These are the daily jobs that take care of all of the various project stats
0 4 * * * cd /home/clusteradmin/utils/underworld-root/ && \
(./db_project_metric.pl ;./db_project_cleanup.pl)
0 4 * * * cd /home/clusteradmin/utils/underworld-root && \
db_project_weekly_metric.pl >/dev/null
0 4 * * * cd /home/clusteradmin/utils/underworld-root && db_rating_stats.pl
0 4 * * * cd /home/clusteradmin/utils/underworld-root &&
db_top_groups_calc.pl>/dev/null
0 4 * * * cd /home/clusteradmin/utils/underworld-root && db_site_stats.pl

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA 

7DVNVIRUFOXVWHUDGPLQ

# periodic tasks for the clsuteradmin user

# $Id: crontab-clusteradmin.sf,v 1.6 2000/12/10 22:38:45 spector Exp $
#
# crontab times are written as follows [see also crontab(5) ]
#
# field allowed values
# ----- --------------
# minute 0-59
# hour 0-23
# day of month 1-31
# month 1-12 (or names, see below)
# day of week 0-7 (0 or 7 is Sun, or use names)
#
# This needs to run in concert with the root batch jobs that
# actually create user accounts. This jobs runs first, then
# 10 minutes later the new_parse.pl script runs which actually creates
# the accounts.
#
* */6 * * * cd /home/clusteradmin/utils/underworld-dummy && ./dump_database.pl

BUILDING LINUX CLUSTERS – CLUSTER ADMINISTRATION SYSTEM VERSION 2.0BETA