Rescue CD System Management

ADMIN MAGAZINE

NEW MAGAZINE!
Smart tools for better networks

ADMIN
ANALYZE AND REPAIR PC SYSTEMS
Four Live systems on one disc
ADMIN
✚ SystemRescueCd ✚ Parted Magic ✚
Network & Security

✚ Clonezilla Live ✚ Redo Backup ✚

ISSUE 01

Network & Security

System 5Free
Fantastic
System Management

Back-up Tools

Management
Spacewalk
Icinga
MySQL Forks

Spacewalk: Corral your Red Hat Servers Teamviewer

Easy remote control
Optimizing with SystemTap
Exchange 2010

Mastering Microsoft’s virtual ModSecurity

machine manager Protect your Apache
web server

Icinga
Free Backup Tools

Is this nascent network

monitor nattier than
Nagios?
SystemTap

Oracle Clusters EXCHANGE 2010

Efficient data access What’s new with
with OCFS2 Microsoft messaging?
OpenVZ

MySQL Forks
A perfect tool for
Teamviewer

every task
ADMIN Magazine
Issue 01 £7.99

Windows • Linux • Unix • OpenSolaris

01
Chef

9 772045 070003
WWW.ADMIN-MAGAZINE.COM
 “You even get six-star customer support thrown in."
- PC Pro magazine

HALVE YOUR IT COSTS WITH

CLOUD COMPUTING
RENT YOUR SERVERS FROM

Scalable, affordable computing instances

from virtual machines to performance dedicated servers

We’ve been voted the UK’s best hosting company for 4 years running.
Isn’t it time you found out why?

ISO 9001: Quality ISO 14001: Environmental ISO 27001: Security

www.memset.com
0800 634 9270 hosting
 Welcome to Admin W E LCO M E

ALL FOR ADMINS

Our Admin special edition was so popular we’re back, with a new quarterly magazine that is all for
admins. Welcome to the first issue of Admin: Network and Security – a magazine for administrators
of heterogeneous networks.
In these pages, you’ll learn about tools for configuring, managing, and troubleshooting your net-
works. You’ll hear about cloud infrastructures, database systems, server plugins, and enterprise man-
agement applications. We’ll tune in to security and network protocols, and we’ll show you the latest
interoperability techniques.
This issue is packed with practical information for real networks. Red Hat published the source code
for their popular Network Satellite management tool in 2008, and a community version known as
Spacewalk soon followed. Our first article takes you on a walk with Spacewalk. Other special fea-
tures include a study of Icinga – a GPLed fork of the popular Nagios network monitoring
system, as well as an article on the forks and patches of MySQL and a roundup of
open source backup tools.
Farther in, we’ll show you the OCFS2 filesystem for Oracle clusters.
We’ll also look at some virtualization tools, including Micro-
soft’s vm2008 and OpenVZ – a container-based virtual-
ization alternative built on Linux. We’ll investi-
gate some security and system monitoring
tools, and we’ll cook up some recipe’s
for fast client configuration with the
Chef configuration management
app.
If you’re an IT professional,
and you’re looking for a
magazine with detailed,
technical articles that
are relevant to the
real-life world you live
and work in, read on.
And if you would like
to receive future issues
of Admin delivered to
your door, see page 58
for information on sub-
scribing.

W W W. A D M I N - M AGA Z I N E .CO M ADMIN 01 3

S E RV I C E Table of Contents

ADMIN Network & Security

Features Tools Virtualization

Management, monitoring, database Save time and simplify your workday Still finding your way through the
forks, and other pertinent glimpses at with these useful tools for real-world cloud? Keep on course with these cool
innovations across the IT landscape. networks. tools for virtual environments.

8 Spacewalk 36 OCFS2 48 Package Your Scripts

Walk on air with this free version of Build a database cluster with Oracle’s We show you how to use Debian’s
Red Hat's enterprise-ready Satellite Cluster Filesystem. packaging tools to deploy and manage
management server. scripts in the cloud.

14 Icinga
This is not your father's Nagios fork.

52 OpenVZ
42 Synergy Container-based virtualization tools like
20 MySQL Forks Too many monitors on your desk? OpenVZ are sometimes more efficient
We investigate some invaluable This handy tool lets you control your than hypervisor systems.
variations on the MySQL theme. servers from a single desktop.
Applications
Container Context

25 Exchange 2010 Resource Resource Resource

Container 1 Container 2 Container 3
What's new in Microsoft's email and Applications

messaging system? OpenVZ

Host Context

Abstraction Layer

28 Backup Tools Host System Kernel

We round up some of the best open

source backup utilities. 44 SystemTap
Optimize and troubleshoot your home- 60 SCVMM 2008
grown apps with this powerful profiling Exploring Microsoft's Service Center
tool. Virtual Machine Manager 2008.

34 BlackHat USA 2010

Learn the latest tricks of network
intruders at the BlackHat conference.

4 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

 Table of Contents S E RV I C E

86 ModSecurity

Rescue CD
8 Spacewalk Careful admins know new exploits
Red Hat released the source code appear every day. Keep the intruders
for the Satellite Server management off your pages with this web

Toolbox
tool in 2008. Spacewalk is a new free application firewall for Apache web
version. servers.

ANALYZE AND REPAIR PC SYSTEMS

Management Nuts and Bolts

Use these practical apps to extend, Timely tutorials on fundamental ✚ Four fabulous Live
simplify, and automate routine admin techniques for system Linux systems
tasks. administrators.
✚ SystemRescueCd –
64 Teamviewer 78 PAM
This popular remote control tool isn't The powerful Pluggable Authentication
A handy collec-
just for Mac and Windows anymore. System offers centralized authentication tion of advanced
for Unix and Linux systems.
rescue utilities
68 Chef
This snappy configuration manager lets ✚ Parted Magic –
you roll out Linux systems with a couple
of mouse clicks.
Partition your hard disk
✚ Clonezilla Live – Clone and
restore Linux systems
✚ Redo Backup – World‘s
easiest backup distro
86 ModSecurity
How safe is your web server? This
powerful Apache extension will help
d etails
74 Sysinternals
Get the pulse of your Windows network
keep intruders from getting control.
See p 6 for full
with this convenient collection of 92 Monitoring Daemons
management tools. Why write a custom script? A few simple
shell commands might be all you need
to monitor system daemons.

94 VPNs with SSTP

Build a virtual private network with
the Secure Sockets Tunneling Protocol
(SSTP).

Service

3 Welcome
4 Table of Contents
6 On the CD
98 Call for Papers

W W W. A D M I N - M AGA Z I N E .CO M ADMIN 01 5

S E RV I C E On the CD

Rescue CD Toolbox

On the CD
The CD included with this issue lets you boot to four
special-purpose Live Linux systems:
■ SystemRescueCd – This versatile rescue distro in-
cludes a copious collection of networking and trou-
bleshooting tools, including utilities for accessing and
repairing Windows systems.
■ Parted Magic – A little Linux that specializes in for-
matting, resizing, and recovering partitions.
■ Clonezilla Live – Clone systems for fast and easy
backup and restore. The bare-metal backup and re-
covery approach will bring back your system in a
fraction of the time.
■ Redo Backup – Another bare-metal backup tool with
an emphasis on simplicity.
Place the CD in the drive and reboot your system. (Make
sure your computer is configured to boot to the CD
CD?
DEFECTIVE ill be replaced. drive.) Choose a distro in the handy boot menu. See the
ective C w
Def Ds box titled “Resources” for links to more information on
email to
Please send an m.
the tools in our Rescue CD Toolbox. ■
-magazine.co
subs@admin

Resources
[1] SystemRescueCd: [http://www.sysresccd.org/]
[2] Parted Magic: [http://partedmagic.com/]
[3] Clonzilla: [http://www.clonezilla.org/]
[4] Redo Backup: [http://redobackup.org/]

6 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

POWERHOUSE
PERL
11 cool projects!

■ Math Tricks: Solve math problems with Perl

■ Daily Tip: Perl with an SQLite database
■ AJAX: Add dynamic updates to web pages
■ isp-switch: Switch your computer to
another ISP if your connection goes down
■ MAC Addresses: Monitor your network for
unfamiliar MAC addresses
■ Multimeter: Read and interpret data from
an external device
■ Google Chart: Create custom graphs
■ Twitter: Help your scripts send Tweets
■ Webcam: Control a web camera with Perl
■ Perl Gardening: Water your house plants
with a little help from Perl
■ GPS: Extract data from a GPS device and
plot it on a Yahoo! Map
FREE DVD inside:
openSUSE 11.2
Linux Mint 8.0

New to Perl? Perl expert Randal L. Schwartz provides an in-depth

introduction to the principles of the versatile Perl language.
Then Perlmeister Mike Schilli explains how to speed up and debug your
scripts. Also inside: Get hands-on with a collection of some of the
Perlmeister’s best columns!

N E A R Y OU!
ND
R L O NA NEWSSTA S l
pecia
H O US E P E azin e . c o m /
R
!
g copy today
W E - m a
IND PO x order your
F AT lin u m it e d so
INE
li
Supplies are
O R D E R ONL
OR
 F e at u r e s Spacewalk
© patrimonio designs, Fotolia.com
Managing Linux systems with Spacewalk
Moon Landing
As your network grows, managing Linux systems manually becomes time consuming and impractical.
Enter Spacewalk: an open source tool that takes the footwork out of network management.
By Thorsten Scherf
Spacewalk [1] is the open source
derivative of the popular Red Hat Net-
work Satellite Server. Red Hat pub-
lished the source code for the server
in the summer of 2008, and the com-
munity has now released version 1.0.
The application’s core tasks include
RPM package software provisioning,
managing configuration files, and
kickstart trees, thus supporting the
installation of bare-metal systems.
The approach that Spacewalk uses
is quite simple: Before a system can
access Spacewalk’s resources, it
8 Admin 01
first has to register with the server.
Registration can be based either on
a username/​password combination
or an activation key that is pregener-
ated by the Spacewalk server. After
registration, the system appears in the
server’s web GUI.
If the server has more resources, you
can assign them to the system at this
point. Resources include software
packages or configuration files that
are normally organized in channels.
A system always has exactly one
base channel with optional subchan-
nels. The base channel contains the
RPM-based operating system, such
as Red Hat Enterprise Linux, Fedora,
or CentOS. The subchannels contain
additional software packages that are
independent of the operating system,
such as the Red Hat Cluster Suite or
the 389 Directory Server.
Spacewalk can clone existing chan-
nels and create new channels from
scratch. This feature gives you full
control of the software stack that you
provide via Spacewalk. Configura-
tion channels help you distribute the
w w w. a d m i n - m aga z i n e .co m

configuration files for the software
packages. Spacewalk also keeps older
versions of the files to let you roll
back to a previous version at any time
if the need arises.
The software packages or configura-
tion files can be installed either via
the target system or centrally in the
Spacewalk web front end. To avoid
spending too much time on the instal-
lation of a large number of systems,
you can assign systems to logical
groups and apply the installation of
a resource to a group. For example, it
might make sense to assign all your
web servers to a WWW-Server group
in Spacewalk. When a new version of
the web server software is released,
you would simply tell Spacewalk to
apply the update to the group, au-
tomatically updating all the systems
belonging to the group.
The installation uses polling by
default; in other words, the client
systems query the server at a pre-
defined interval (which defaults to
four hours) to see if new actions have
been defined since the last poll. If so,
Spacewalk then runs these actions.
As an alternative, you can trigger the
installation of software packages and
other actions using a push approach.
The client system and the Spacewalk
server talk to each other constantly
using the Jabber protocol. Any new
actions you define are immediately
run on the client by Spacewalk.
Ground Control
Communications are always from the
client to the server; this is important
with respect to firewall rules. A list
of the network ports you need to en-
able can be found online [2]. Besides
software package or configuration
file installation, actions can also run
arbitrary commands on the individual
systems via the Spacewalk server.
For example, after creating a new
configuration file for your web serv-
ers and distributing it to the systems,
you need to restart the web server
process to parse the new configura-
tion instructions. Instead of logging
in to each individual system or using
a for loop, simply issue the restart
w w w. a d m i n - m aga z i n e .co m
command centrally on the Spacewalk
server.
Installing new systems is also quite
simple. Spacewalk has the installation
files you need in the form of kickstart
trees. The installation candidate uses
a boot medium such as a CD, a USB
stick, or a PXE-capable network card
to contact the server. The First-Stage
Installer, which is part of the instal-
lation medium, defines which server
will handle the installation.
The remaining installation steps are
handled by the Second-Stage Installer,
located on the Spacewalk server and
transferred to the client system when
the installation starts. If you want to
automate the installation fully, define
the kickstart file location in the boot
medium. The kickstart file is a kind of
answer file that describes the proper-
ties of the installation candidate, such
as partitioning, software, language,
and firewall settings. Of course, you
can create a kickstart file on the
Spacewalk server and just include a
link to the file on the boot medium.
Spacewalk can manage any RPM-
based distribution. You even have
the option of operating client systems
across multiple organizations. Using
the web interface, the administrator
creates various organizations and as-
signs a certain number of system en-
titlements to them. Entitlements are
linked to certificates that Spacewalk
automatically generates during the in-
stallation. You can then add users to
the organizations.
If a client is registered with a user
account from a specific organiza-
tion, the system is assigned to this
organization. When users from the
organization logs into the Spacewalk
server, they will only see the systems
in their own organization. This fea-
ture is useful if you manage multiple
departments and prefer to manage the
systems in the individual departments
separately. You just assign them to
different organizations, which, of
course, you need to create up front.
Installation
Spacewalk can be installed on Red
Hat Enterprise (RHEL) [3], Fedora
Spacewalk F e at u r e s
[4], or CentOS [3] Linux. Note that
Spacewalk does need a current Java
Runtime Version 1.6.0 or newer. You
can use the Open JDK for this; Fedora
includes it out of the box. Admins
on RHEL or CentOS can retrieve the
package via the additional EPEL (Ex-
tra Packages for Enterprise Linux)
software repository.
Besides the Java package, an Oracle
10g database is also required for
installing Spacewalk. Oracle XE pro-
vides a free version of the database.
The developers are currently working
hard on implementing support for an
open source database after identify-
ing PostgreSQL as the best alterna-
tive to Oracle. As of this writing it is
hard to say when official support for
PostgreSQL will be available, but it
makes sense to check the roadmap
[5] or the mailing lists [6] at regular
intervals.
Oracle XE
After installing the repository RPM
for your distribution, the first step is
to install Oracle Express, which you
can download for free [7]. You will
need version 10.2.0.1. Besides the
database, you also need the oracle‑in-
stantclient‑basic and oracle‑instant-
client‑sqlplus, which you can then
install with Yum:
yum localinstall ‑‑nogpgcheck U
oracle‑xe‑univ*.rpm
oracle‑instantclient‑basic*.rpm
oracle‑instantclient‑sqlplus*.rpm
Before configuring the database, you
should make sure that your hostname
points to the correct IP address in
your /etc/hosts to avoid problems
Listing 1: Oracle Listener Configuration
cat >> /etc/tnsnames.ora << 'EOF'
XE =
(DESCRIPTION =
(ADDRESS_LIST =
(
ADDRESS = (PROTOCOL = TCP)(HOST = localhost)
(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = xe)
)
)
EOF
Admin 01 9
 F e at u r e s Spacewalk
with the Oracle Listener configuration
later on. Use the following parameters
for the configuration:
HTTP port for Oracle Application U
Express: 9055
Database listener port: 1521
Password for SYS/SYSTEM: Password
Start at boot: y
The default HTTP port for the Oracle
Express application (8080) is already
occupied by the Tomcat application
server, so you need to choose an al-
ternative port to avoid conflicts.
To talk to the database, you need to
configure the listener in the /etc/
tnsnames.ora file (Listing 1).
Now you just need to make a few
changes to the database. To do this,
log in to the database with sqlplus
and create a spacewalk user, to which
you could assign a password of
spacewalk (Listing 2).
The standard configuration of Oracle
Express supports a maximum of 40
simultaneous connections, which is
not enough for Spacewalk operations.
The instructions in Listing 3 change
the limit to a maximum of 400 con-
nections.
Now you need to restart the database
by giving the /sbin/service oracle‑
xe command.
Spacewalk Setup
The next step is to install the Space-
walk server. To do so, you need to
include the Spacewalk repository as
described previously. You should have
a spacewalk.repo file that points to
Listing 2: Creating the Spacewalk User
sqlplus 'sys@xe as sysdba'
SQL> 
create user spacewalk identified by spacewalk
default tablespace users;
SQL> grant dba to spacewalk;
SQL> quit
Listing 3: Oracle Tuning
sqlplus spacewalk/spacewalk@xe
SQL> alter system set processes = 400 scope=spfile;
SQL> alter system set "_optimizer_filter_pred_pullup"
=false scope=spfile;
SQL> 
alter system set "_optimizer_cost_based_
transformation"=off scope=spfile;
SQL> quit
10 Admin 01
the appropriate repository in /etc/
yum.repos.d/. The following com-
mand starts the installation:
yum install spacewalk‑oracle
Because this package depends on all
the other Spacewalk packages, the
package manager will automatically
download and install the dependen-
cies in the next step. Then you can
configure the application interactively
with the setup tool or with the use of
an answer file (Listing 4).
Pass the file in to the setup tool as
follows:
spacewalk‑setup ‑‑disconnected U
‑‑answer‑file=answerfile
The configuration can take some time
to complete as the process sets up
the database tables. The setup tool
then launches all the required ser-
vices. You can manually restart using
the /usr/sbin/rhn‑satellite tool.
To configure the system, launch the
Spacewalk web interface via its URL
(http://​spacewalk.server.tld). Besides
contact information, you can also set
the password for the Spacewalk ad-
ministrator here.
Software Channels
The next step is to set up an initial
software channel for the client sys-
tems. When you register a client, you
must specify exactly one base channel
for the client; it will use this channel
to retrieve its operating system pack-
ages and their updates. Of course,
Figure 1: The easiest approach to setting up a software channel is to use the web graphical interface.
you can set up subchannels for the
base channel and assign the subchan-
nels to clients as needed. After doing
so, you can use the subchannels to
distribute more RPM packages to the
systems. The packages can be your
own creations or RPMs from other
repositories.
The easiest approach to setting up a
software channel is to use the web in-
terface (Channels | Manage Software
Channels | Create; Figure 1). Thanks
to the Spacewalk API, you can also
script this process [8]. Call the script
as follows:
create_channel.py ‑‑label=fedora‑12‑i386 U
‑‑name "Fedora 12 32‑bit" U
‑‑summary "32‑bit Fedora 12 channel"
In the script, you need to provide
the Fully Qualified Domain Name
(FQDN) for the Spacewalk server
and the user account for creating
the channels, such as the Spacewalk
administrator account created previ-
ously. The Users tab also gives you
the option of creating more users with
specific privileges (Figure 2).
The channel you set up should now
be visible in the Channels tab of the
web interface but will not contain
any software packages. Although you
can upload software packages to the
server in several ways, the method
you choose will depend on whether
the packages are available locally
(e.g., DVD) or you want to synchro-
nize a remote Yum repository with
the Spacewalk server. If you choose
the local upload, you can use the
w w w. a d m i n - m aga z i n e .co m

Figure 2: Assigning individual users different privileges on the Spacewalk server.
rhnpush tool, which you launch as
follows:
rhnpush ‑v ‑‑channel=fedora‑13‑i386 U
‑‑server=http://localhost/APP U
‑‑dir=/path/to/the/packages
To synchronize with a remote soft-
ware repository, you simply need to
specify the URL for the remote reposi-
tory in the software channel proper-
ties in the web interface (Channels |
Manage Software Channels | Fedora
12 32-bit). Synchronization can take
a while to happen. Your other op-
tion here is the spacewalk‑repo‑sync
command-line tool that downloads
software packages from a Yum reposi-
tory to your own Spacewalk server.
To keep the server up to date, you can
use cron to run a script [9] at regular
intervals. This script will check your
configured software sources and au-
tomatically download any new pack-
ages. This approach removes the need
for manual synchronization.
Incidentally, you can use the method
discussed here to set up subchannels,
Figure 3: Various resources can be bound to the registration key. Systems that use the key are given access
to the associated resources.
w w w. a d m i n - m aga z i n e .co m
too. Note that any RPM packages
you build yourself must be digitally
signed. Both the Spacewalk server
and the Yum client application will
reject unsigned packages by default.
Although you can disable this feature,
it makes more sense to work with
digital signatures for security reasons.
The rpm ‑‑resign RPM package com-
mand will sign the package for you;
you must have GPG keys in place for
the RPM tool. The ~/.rpmmacros file
tells you the name and location of the
key (Listing 5).
To allow client systems to verify pack-
ages signed with this key, you need to
deposit the public key on the Space-
walk server, preferably in /var/www/
html/pub, which any client can ac-
cess. The following command exports
the public key from the GPG keyring:
gpg ‑‑armor ‑‑export tscherf@redhat.com > U
/var/www/html/pub/rpm‑gpg‑key
To allow the existing client systems
to access the software packages you
just uploaded, you need to register
Spacewalk F e at u r e s
them with the Spacewalk server.
Start by installing the Spacewalk Cli-
ent Repository RPM on the clients.
Fedora 12 systems have a matching
RPM [10], as do RHEL5 and CentOS5
[11]. On RHEL and CentOS, you also
need to install the RPM for the EPEL
repository [12] because the client tool
dependencies might not resolve cor-
rectly otherwise. The following com-
mand installs the Yum file on a 32-bit
Fedora 12 system:
rpm ‑Uvh http://spacewalk.redhat.com/ U
yum/1.0/Fedora/12/i386/spacewalk‑ U
client‑repo‑1.0‑2.fc12.noarch.rpm
Then, use Yum to install the client
tools:
yum install rhn‑client‑tools U
rhn‑check rhn‑setup rhnsd m2crypto U
yum‑rhn‑plugin
The easiest approach to registering
a system on the server is to run the
rhnreg_ks tool, which expects a reg-
istration key. You need to create the
key up front on the Spacewalk server
(Systems | Activation Key | Create
Key). When you create a key, you can
bind various resources to it, such as
the Fedora 12 software channel just
created here, or various configuration
channels, if you have created some
(Figure 3). Also, you can assign sys-
tem groups to the key. Systems that
use this key to register are granted
access to the associated resources.
To do so, specify the key you created
during the registration process: E
Listing 4: Answer File
admin‑email = root@localhost
ssl‑set‑org = Tuxgeek Org
ssl‑set‑org‑unit = Tuxgeek OU
ssl‑set‑city = Essen
ssl‑set‑state = NRW
ssl‑set‑country = DE
ssl‑password = spacewalk
ssl‑set‑email = root@localhost
ssl‑config‑sslvhost = Y
db‑backend=oracle
db‑user=spacewalk
db‑password=spacewalk
db‑sid=xe
db‑host=localhost
db‑port=1521
db‑protocol=TCP
enable‑tftp=Y
Admin 01 11
 F e at u r e s Spacewalk

rhnreg_ks ‑‑serverUrl=U

http://spacewalk.server.tld/XMLRPC U

‑‑activationkey=key

If all of this worked correctly, you will

see the system in the Systems tab of
the server web interface. Viewing the
system’s properties should also show
you the configured software chan-
nel. The easiest approach to check-
ing whether access to the channel is
working is to install a package from
the channel. If this doesn’t work, one
possible issue could be that the client
system is not using the Spacewalk
server’s CA certificate. The certificate
is stored in http://spacewalk.server. Figure 4: After completing the registration, the system appears in the Spacewalk server’s web interface.
tld/pub/ on the server and must be
stored in /usr/share/rhn on the cli- you want to install, such as Fedora distro‑trees/Fedora‑12 directory. If
ent side. The /etc/sysconfig/rhn/ 12, but the basic installation files, like all of this works out, just point to the
up2date file needs a reference to the the Anaconda tool. distribution you created when you
certificate. The software repositories you syn- made the kickstart file. When a cli-
As before, you need to enter the chronized earlier will not normally ent system is installed from scratch,
name of the Spacewalk server. You provide a kickstart distribution, and it will automatically pick up the right
only need to perform these steps on this means creating the distribution files from this source.
systems you have already installed. on the Spacewalk server. Again, just Although there are a number of ways
Any that you install from scratch via navigate to Systems | Kickstart | Dis- to install a Fedora 12 system from
the Spacewalk server are automati- tributions in the web interface and scratch, the easiest approach is to
cally registered with the server as part point to the required files. The easiest point any client PXE requests by your
of the installation process and can way to provide the files is to mount clients to the Spacewalk server with
thus access the server immediately an installation CD/​DVD for your pre- the next‑server command. Thanks to
(Figure 4). ferred distribution via the loopback Cobbler [13] integration, the Space-
device: walk server has a TFTP server and
Kickstart Installation mount ‑o loop U
any kickstart profiles that you have
/var/iso‑images/Fedora‑23‑i386‑DVD.iso U
set up. To confirm this, you can type
To automate the installation of new /var/distro‑trees/Fedora‑12 cobbler profile list at the com-
client systems, you need two pieces mand line.
of information on the Spacewalk When you create a Fedora 12 kick- When you boot a client system via a
server. One of them is a kickstart file start distribution, you simply point PXE-capable network card, you will
with details of how to install the new the Spacewalk server to the /var/ automatically see a list of the existing
system, including partitioning, the
software selection, and other settings
that you would need to provide for
a manual install. The easiest way to
create a kickstart file is to select Sys-
tems | Kickstart | Profiles in the web
front end.
After checking out the overview of
existing profiles, you can also create a
new profile. The kickstart distribution
must be specified as part of the pro-
file file. This does not mean the RPM
files that belong to the distribution

Listing 5: GPG Configuration for RPM

cat .rpmmacros
%_signature gpg
Figure 5: The system properties give you a neat option for handling a variety of administrative tasks for a
%_gpg_name Thorsten Scherf <tscherf@redhat.com>
system via the Spacewalk server.

12 Admin 01 w w w. a d m i n - m aga z i n e .co m


kickstart profiles. To install the cli-
ent, simply select the required profile
from the list. The client is then auto-
matically registered on the Spacewalk
server. Existing systems can easily be
reinstalled using:
koan ‑‑replace‑self U
‑‑server=Spacewalk‑Server U
‑‑profile=Kickstart‑Profile
This creates an entry in the system’s
bootloader menu and automatically
selects the entry when the system
reboots.
System Management
All of the systems registered on the
Spacewalk server retrieve their soft-
ware packages from this source, with
no need to access external reposito-
ries. This method not only improves
your security posture but also saves
network bandwidth. With a registered
system, you can customize various
settings in the System Properties sec-
tion (Figure 5).
For example, you can assign new
software or configuration channels,
compare the installed software with
profiles on other systems, or create
snapshots as a backup that you can
roll back later. Additionally, you can
install new software or distribute
configuration files from a centralized
location.
Thanks to the ability to assign reg-
istered systems to groups, you can
point and click to do this for a large
Figure 6: An XML-RPC interface opens up a huge selection of Spacewalk server functions via the
programmable API.
w w w. a d m i n - m aga z i n e .co m
number of systems. The rhnsd service
on the systems queries the Spacewalk
server at predefined intervals to check
for new actions, such as software in-
stallations.
When a system finds an action, it
then executes it. If the osad service
is enabled on the system, you can
even run actions immediately with-
out waiting for the polling interval to
elapse. The client and the server then
use the Jabber protocol for a continu-
ous exchange.
Finally, don’t forget the feature-rich
Spacewalk API, which is accessible at
http://Servername/rhn/apidoc/index.
jsp on the installed server. This tool
gives you access to a plethora of func-
tions that are not available in the web
interface.
The API can be accessed with XML-
RPC, which makes it perfect for your
own Perl or Python scripts. A Python
script [8] for creating a software
channel is just one example of access-
ing the Spacewalk server via the API
(Figure 6).
Conclusions
Spacewalk gives administrators a very
powerful tool for managing large-
scale Linux landscapes. It facilitates
many daily tasks, such as the instal-
lation of software updates or upload-
ing of configuration files. Advanced
features, such as channel cloning,
make it possible to put any software
through a quality assurance process
Spacewalk F e at u r e s
before rolling it out to your produc-
tion systems. Thanks to the compre-
hensive API, many tasks can also be
scripted. n
Info
[1] Spacewalk project homepage:
[https://​­fedorahosted.​­org/​­spacewalk]
[2] Spacewalk network ports:
[http://​­magazine.​­redhat.​­com/​­2008/​­09/​
­30/​­tips‑and‑tricks‑what‑tcpip‑ports‑are‑r
equired‑to‑be‑open‑on‑an‑rhn‑satellite‑pr
oxy‑or‑client‑system/]
[3] RHEL5, CentOS5 Spacewalk Server Repos‑
itory RPM: [http://​­spacewalk.​­redhat.​­com/​
­yum/​­1.​­0/​­RHEL/​­5/​­i386/​­spacewalk‑repo‑1.​
­0‑2.​­el5.​­noarch.​­rpm]
[4] Fedora12 Spacewalk Server Repository
RPM: [http://​­spacewalk.​­redhat.​­com/​­yum/​
­1.​­0/​­Fedora/​­12/​­i386/​­spacewalk‑repo‑1.​­0‑2.​
­fc12.​­noarch.​­rpm]
[5] Spacewalk Roadmap: [http://​
­fedorahosted.​­org/​­spacewalk/​­roadmap]
[6] Spacewalk mailing list:
[http://​­www.​­redhat.​­com/​­spacewalk/​
­communicate.​­html#​­lists]
[7] Oracle XE: [http://​­www.​­oracle.​­com/​
­technology/​­software/​­products/​­database/​
­xe/​­htdocs/​­102xelinsoft.​­html]
[8] Spacewalk API script for creating a
software channel: [http://​­fedorahosted.​
­org/​­spacewalk/​­attachment/​­wiki/​
­UploadFedoraContent/​­create_channel.​­py]
[9] Repository sync: [http://​­fedorahosted.​
­org/​­spacewalk/​­attachment/​­wiki/​
­UploadFedoraContent/​­sync_repos.​­py]
[10] Fedora12 Spacewalk Client Reposi‑
tory RPM: [http://​­spacewalk.​­redhat.​
c­ om/​­yum/​­1.​­0/​­Fedora/​­12/​­i386/​
­spacewalk‑client‑repo‑1.​­0‑2.​­fc12.​­noarch.​
­rpm]
[11] RHEL5 and CentOS5 Client Repository
RPM: [http://​­spacewalk.​­redhat.​­com/​­yum/​
­1.​­0/​­RHEL/​­5/​­i386/​­spacewalk‑client‑repo‑1.​
­0‑2.​­el5.​­noarch.​­rpm]
[12] EPEL Repository: [http://​­download.​
­fedora.​­redhat.​­com/​­pub/​­epel/​­5/​­i386/​
­epel‑release‑5‑3.​­noarch.​­rpm]
[13] Cobbler:
[https://​­fedorahosted.​­org/​­cobbler/]
The Author
Thorsten Scherf is a Senior Consultant for Red
Hat EMEA. You can meet him as a speaker at
conferences. He is also a keen marathon runner
whenever time permits.
Admin 01 13
 F E AT U R E S Icinga

Monitoring network computers with the Icinga Nagios fork

Server Observer
© Alterfalter, 123RF.com

Icinga’s developers grew weary of waiting for updates to the popular Nagios monitoring tool, so they
started their own project. By Falko Benthin

A server can struggle for many
reasons: System resources like the
CPU, RAM, or hard disk space could
be overloaded, or network services
might have crashed. Depending on
the applications that run on a server,
consequences can be dire – from
irked users to massive financial impli-
cations.
Therefore, it is more important than
ever in a highly networked world to
be able to monitor the state of your
server and take action immediately.
Of course, you could check every
server and service individually, but it
is far more convenient to use a moni-
toring tool like Icinga.
Nagios Fork
Icinga [1] is a relatively young project
that was forked from Nagios [2] be-
cause of disagreements regarding the
pace and direction of development.
Icinga delivers improved database
connectors (for MySQL, Oracle, and
PostgreSQL), a more user-friendly
web interface, and an API that lets
administrators integrate numerous
extensions without complicated
modification of the Icinga core. The
Icinga developers also seek to reflect
community needs more closely and to
integrate patches more quickly. The
first stable version, 1.0, was released
in December 2009, and the version

Listing 1: my_hosts.cfg
01 # Webserver 19 # Fileserver
02 define host{ 20 define host{
03 host_name webserver 21 host_name fileserver
04 alias languagecenter 22 alias Fileserver
05 display_name Server at language center 23 display_name Fileserver
06 address 141.20.108.124
24 address 192.168.10.127
07 active_checks_enabled 1
25 active_checks_enabled 1
08 passive_checks_enabled 0
26 passive_checks_enabled 0
09 max_check_attempts 3
27 max_check_attempts 3
10 check_command check-host-alive
28 check_command check-host-alive
11 check_interval 5
29 check_interval 5
12 retry_interval 1
30 retry_interval 1
13 contacts spz_admin
14 notification_period 24x7 31 contacts admin

15 notification_interval 60 32 notification_period 24x7

16 notification_options d 33 notification_interval 60
17 } 34 notification_options d,u,r
18 35 }

14 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

 Icinga F e at u r e s

Table 1: States
Option Status
Server
o OK
d Down
u Unreachable
r Recovered
Services
o OK
w Warning
c Critical
r Recovered
u Unknown

counter has risen every couple of

months ever since.
Icinga comprises three components:
the core, the API, and the optional
web interface. The core collects sys-
tem health information generated by
plugins and passes it via the IDOMOD
interface to the Icinga Data Out Data-
base (IDODB) or the IDO2DB service
daemon. The PHP-based API accepts
information from the IDODB and
displays it in a web-based interface. Figure 1: If the hosts are healthy, the admin is happy.
Additionally, the API facilitates the
development of add-ons and plugins. distributions offer binaries in their re- Icinga can monitor the private ser-
Icinga Web is designed to be a state- positories, but if not, or if you prefer vices on a computer, including CPU
of-the-art web interface that is easily to use the latest version, the easy-to- load, RAM, and disk usage, as well as
customized for administrators to keep understand documentation includes public services like web, SSH, mail,
an eye on the state of the systems a quick-start guide (for the database and so on. The lab network environ-
they manage. At the time of writing, via libdbi with IDOUtils), which can ment consists of three computers, one
Icinga Web was in beta, and it has a help you set up the network monitor of which acts as the Icinga server; the
couple of bugs that make it difficult in next to no time for access at http://​ other two are a web server and a file
to recommend for production use. Server/​icinga. The challenges come server that send information to the
If you only need to monitor a single when you want to monitor a larger monitoring server. Because no native
host, Icinga is installed easily. Some number of computers. approach lets you request information

Listing 2: my_services.cfg (Excerpt)

01 #
 SERVICE DEFINITIONS 15  notification_interval 60
02 d
efine service{ 16  notification_options w,c,u,r
03  host_name webserver 17  }
04  service_description HTTP 18 d
efine service{
05  active_checks_enabled 1 19  host_name fileserver, webserver
06  passive_checks_enabled 0 20  service_description SSH
07  check_command check_http 21  active_checks_enabled 1
08  max_check_attempts 3 ;
how often to perform 22  passive_checks_enabled 0
the check before 23  check_command check_ssh
Icinga notifies 24  max_check_attempts 3
09  check_interval 5 25  check_interval 15
10  retry_interval 1 26  retry_interval 1
11  check_period 24x7 27  check_period 24x7
12  contacts spz_admin 28  contacts admin
13  notifications_enabled 1 29  notifications_enabled 0
14  notification_period weekdays 30  }

w w w. a d m i n - m aga z i n e .co m Admin 01 15

 F e at u r e s Icinga

monitoring environments, notification

Figure 2: Everything is working, but the NRPE plugin is causing problems.
externally about CPU load, RAM, or
disk space usage, you need to install
a verbose add-on, such as NRPE
[3], on each machine. The remote
Icinga server will tell it to execute
the plugins on the local machine and
transmit the required information.
Icinga sends the system administrator
all the information needed and alerts
the admin of emergencies. Advanced
features that are a genuine help in
daily work include groups, redundant
escalation, and check schedules.
Icinga differentiates between active
and passive checks. Active checks are
initiated by the Icinga service and run
at times specified by the administra-
tor. For a passive check, an external
application does the work and for-
wards the results to the Icinga server,
which is useful if you can’t actively
check the computer (e.g., it resides
behind a firewall). A large number of
plugins [4] already exist for various
styles in Nagios and Icinga. But be-
fore the first check, the administrator
needs to configure the computers and
the services to monitor in Icinga.
The individual elements involved in
a check are referred to as objects in
Icinga. Objects include hosts, ser-
vices, contacts, commands, and time
slots. To facilitate daily work, you can
group hosts, services, and contacts.
The individual objects are defined in
CFG files, which reside below Icinga’s
etc/objects directory. The network
monitor includes a number of sample
definitions of various objects that ad-

Listing 3: commands.cfg (Excerpt)

01 
# 'notify‑service‑by‑email' command definition
02 
define command{
03  command_name notify‑service‑by‑email
04  command_line /
usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService:
$SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time:
$LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" |
/usr/bin/mail ‑s "**$NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTA
05 T
E$ **" $CONTACTEMAIL$
06  }
07 
08 #
 'check‑host‑alive' command definition
09 d
efine command{
10  command_name check‑host‑alive
11  command_line $USER1$/check_ping ‑H $HOSTADDRESS$ ‑w 3000.0,80% ‑c 5000.0,100% ‑p
12 5
13  }

Listing 4: timeperiods.cfg (Excerpt)

01 
define timeperiod{ 12 
02  timeperiod_name 24x7 13 d
efine timeperiod{
03  alias 24 Hours A Day, 7 Days A Week 14  timeperiod_name wochentags
04  sunday 00:00‑24:00
15  alias Robot Robot
05  monday 00:00‑24:00
16  monday 07:00‑17:00
06  tuesday 00:00‑24:00
17  tuesday 07:00‑17:00
07  wednesday 00:00‑24:00
18  wednesday 07:00‑17:00
08  thursday 00:00‑24:00
09  friday 00:00‑24:00 19  thursday 07:00‑17:00

10  saturday 00:00‑24:00 20  friday 07:00‑17:00

11  } 21  }

16 Admin 01 w w w. a d m i n - m aga z i n e .co m

 Icinga F e at u r e s

ministrators only need to customize.
In principle, you can define multiple
objects in a CFG file, but you can just
as easily create separate files for each
object in a directory below /path‑​
to‑Icinga/etc/objects. Lines that
start with a hash mark within an
object definition are regarded as com-
ments, as is everything within a line
to the right of a semicolon.
Defining Hosts and Services
Listing 1 provides a sample host defi-
nition. The host is the web server at
a language center (display_name) and
is displayed accordingly in the web
interface.
To inform the administrator (con‑
tacts) when the server goes down
(notification_options), I want Icinga
to ping (check_command) the server ev-
ery 5 minutes (check_interval). If the
server is still down 60 minutes (noti‑
fication_interval) after notifying the
administrator, I want to send another
message.
Icinga is capable of deciding whether
a host is down or unreachable (see
Table 1). However, to determine that
a host is unreachable, you have to de-
fine the nodes passed along the route
to the host as parents – and this will
only work if the routes for outgoing
packets are known. The file server
definition looks similar.
Once the servers are defined, the
administrator configures the respec-
tive services that Icinga will monitor
(Listing 2), along with the matching
commands (Listing 3), the intervals
(Listing 4), and the stakeholding
administrators (Listing 5). The indi-
vidual configuration files have a simi-
lar structure. For each service, you

Listing 5: contacts.cfg (Excerpt)

01 d
efine contact{ 08  host_notification_options d,u,r

02  contact_name icingaadmin 09  service_notification_options w,u,c,r

03  alias Falko Benthin
10  host_notification_commands notify‑host‑by‑email
04  host_notifications_enabled 1
11  service_notification_commands notify‑service‑by‑email
05  service_notifications_enabled 1

06  host_notification_period 24x7 12  email root@localhost

07  service_notification_period 24x7 13  }

Open Source Monitoring Conference 2010

Live at your PC
You can watch high quality speeches about monitoring topics from a Live-Stream on the
6th and 7th of October at your PC. You will be able to see all the slides as well.

Conference Topics:
• NSClient++ (Michael Medin)
• Clientless Windows Monitoring about
WMI with Samba4 (Thomas Sesselmann)
• The social seismograph at XING
(Dr Johannes Mainusch)
• RRDCacheD - how to escape the I/O hell
(Sebastian Harl)
• Monitoring at Thales Hengelo
using Nagios (Pieter van Emmerik)

Register at: ss to the

streaming.linux-magazin.de/en including acce
video archive
CONTACT: Phone: +49 (0) 89 / 99 34 11 - 0 • Fax: +49 (0) 89 / 99 34 11 - 99 • E-Mail: streaming@linux-magazin.de
 F e at u r e s Icinga

Figure 3: A manual check of commands in commands.cfg reveals the culprit. Figure 4: Mail dispatched by Icinga is short and to the point.

need to consider the interval between Icinga with /etc/init.d/icinga re‑ are typically linked, so that clicking
checks. One useful feature is the abil- start. one takes you to more detailed infor-
ity to define time slots, within which mation.
Icinga will perform checks and, if GUI and Messages If something is so drastically wrong
necessary, notify the administrator. that a message is necessary, Icinga
Here, time limitations or holidays can Icinga works without a graphical will check its complex ruleset to see
be defined. interface, but it’s much nicer to have whether it should send a message
The contact configuration can include one. The standard interface can’t and, if so, to whom (Figure 4). The
email addresses or cell phone num- deny its Nagios ancestry, but it is filters through which the message
bers, but to integrate each contact clear-cut and intuitive. passes check the following: whether
with, for example, an Email2SMS If everything is working, you’ll see a notifications are required, if the
gateway or a Text2Speech system lot of green in the user interface (Fig- problem occurred at a time when the
(e.g., Festival), you need a matching ure 1), but if something goes wrong host and service should be running,
command. somewhere, the color will change and if messages should be sent for this
Icinga can use macros, which notice- move closer and closer to red to re- service in the current time slot, and
ably simplifies and accelerates many flect the status of the hosts or services what the contacts linked to the ser-
tasks because you can use a single (Figures 2 and 3). Status messages vice actually want. Each contact can
command for multiple hosts and ser-
vices. Listings 2 and 3 give examples
of macros.
All services defined for monitoring
the file server include a check_nrpe
instruction with an exclamation
mark. Each exclamation mark can
be followed by an argument, which
in turn is evaluated by the macros in
other definitions. Macros are nested
in $ signs.
After creating the configuration files
and storing them in etc/objects, you
still need to tell Icinga by adding a
new

cfg_file=/
usr/local/icinga/etc/objects/
object.cfg

to the main configuration file, /etc/

icinga.cfg. After doing so, you
should verify the configuration,
/path‑to‑Icinga/bin/icinga ‑v
/path‑to‑Icinga/etc/icinga.cfg;
assuming there are no errors, restart Figure 5: Icinga Web beta was not entirely convincing. Version 1.0.3 is out now.

18 Admin 01 w w w. a d m i n - m aga z i n e .co m


Figure 6: Network overview. If you need to monitor a large number of machines
and have defined “parents,” you can also visualize the intermediate nodes.
define its own rules to stipulate when
it wants to receive messages and for
what status. If multiple administrators
exist and belong to a single group,
Icinga will notify all of them. Again,
you can define individual notification
periods so that each admin will be
responsible for one period.
Interesting Features
Icinga contains several interesting
features that allow administrators to
customize the network monitor to
reflect their needs and system envi-
ronment. For example, you can define
distributed monitoring environments.
If you need to monitor several hun-
dred or thousand hosts, the Icinga
server might conceivably run out of
resources because every active check
requires system resources. To take
some of the load off the main server,
Icinga can delegate individual tasks
to auxiliary servers which, in turn,
forward the results to a central server.
Scheduling the checks can also help
reduce this load. Instead of running
all your active checks in parallel, you
can let Icinga stagger them.
Another interesting feature is the abil-
ity to escalate notifications. Not every
administrator can be available and
w w w. a d m i n - m aga z i n e .co m
Figure 7: The alert histogram, another useful gadget Icinga offers, shows peak
trouble times.
ready for action 24/​7. If the contact
that Icinga notifies does not respond
within a defined period, Icinga can at-
tempt to establish contact on another
channel (e.g., a cell phone instead of
email). If this notification fails as well,
the case can be escalated to someone
higher up the chain of responsibility –
the team leader, for example.
Conclusions
Icinga is a complex tool that provides
valuable services whenever an ad-
ministrator needs to monitor comput-
ers on a network. But don’t expect to
be able to set up the network monitor
in a couple of minutes of spare time;
if all goes well, the installation and
configuration will take at least a cou-
ple of hours. Once you have battled
through the extensive configuration,
you can reward yourself with an
extended lunch break: If something
happens that requires your attention,
Icinga will tell you all about it.
The traditional web interface is clear
cut and packed with information;
when this article went to print, how-
ever, the new interface wasn’t entirely
convincing (Figure 5). The installa-
tion was tricky, the documentation
required some imagination at times,
Icinga F e at u r e s
and the final results were disappoint-
ing. The interface was buggy and
very slow under my, admittedly, not
very powerful Icinga test server (Via
C3, 800MHz, 256MB RAM). As a de-
fault, you need a new username and
password for Icinga Web. That said,
however, the current status does re-
veal some potential; it makes sense to
check how the new interface is devel-
oping from time to time.
The Icinga kernel is well and com-
prehensively documented and leaves
no questions unanswered. Icinga also
offers a plethora of useful gadgets,
such as the status map (Figure 6) or
the alert histogram (Figure 7), mak-
ing the job of monitoring hosts less
boring – at least initially. The depth
of information that Icinga provides is
impressive and promises an escape
route for avoiding calls from end us-
ers. In short, Icinga is a useful tool
that makes the administrator’s life
more pleasant. n
Info
[1] Icinga: [http://​­www.​­icinga.​­org/]
[2] Nagios: [http://​­www.​­nagios.​­org/]
[3] NRPE: [https://​­git.​­icinga.​­org/]
[4] Nagios plugins: [http://​­sourceforge.​­net/​
­projects/​­nagiosplug/]
Admin 01 19
 F e At u r e s mysQL Forks and Patches
Modern MySQL Forks and Patches
Spoiled for
Choice
© Dmitry Karasew, 123RF.com
mysQL is the standard solution for free relational database systems in web applica-
tions, but new forks, more storage engines, and patched versions muddy the water.
now’s the time to take stock of the most important offerings. By caspar clemens mierau
If your MySQL server is too slow,
you have various approaches to solv-
ing the problem. Besides optimizing
queries and indexes, reworking the
configuration, and upgrading your
hardware, moving to a customized
version of the MySQL server can be
a good idea. In recent years, so many
patches, forks, and new storage en-
gines have been released that it is
hard to keep track of them. For hard-
working developers and database
administrators, this means a change
from the simple choice of a standard
MySQL distribution.
Little Band-aids
Many enhancements for MySQL come
from major corporations like Face-
book [1] and Google [2], who run
their ad services on top of MySQL
or from MySQL specialists like Per-
20 Admin 01
cona [3], whose claim to fame is
the “MySQL Performance Blog” [4]
(standard reading for anyone inter-
ested in MySQL). The patches can be
grouped into three categories: (1) re-
porting enhancements, (2) functional
enhancements of the MySQL kernel
and database engines, and (3) perfor-
mance optimizations. In most cases, a
combination of patches from all three
categories will make the most sense.
Moving to a database server that you
patched and compiled yourself can
be a daunting prospect. Thankfully,
projects such as OurDelta [5] offer re-
positories with meaningfully patched
and prebuilt MySQL packages for
popular distributions like Debian,
Ubuntu, and CentOS/RHEL. The
patches I will be looking at in the rest
of this article are a cross-section of
the current OurDelta versions of the
MySQL server; see their website for a
complete list of patches and notes on
how to use them.
Reporting
Extended reporting allows adminis-
trators to collect more granular in-
formation about the MySQL server’s
behavior under load. Thus far, slow.
log, which offers very little in the
line of configuration options, might
be your first port of call. However, its
utility value is restricted to identifying
individual, computationally intensive
queries on the basis of the time they
use – and non-used indexes. The
MicroSlow patch offers new filters
for a more targeted search for poorly
formulated queries. Thus, it logs que-
ries that are responsible for writing
temporary tables to disk, performing
complete table scans, or reading a
freely defined minimum number of
lines in a table. The mysqldumpslow
Slow.log statistics tool, which is not
very well known but is part of the
MySQL standard distribution, has
been modified to be able to read and
evaluate the extended entries.
Aggregated run-time statistics on us-
age behavior are equally as useful.
The UserStats patch extends MySQL
by adding statistics for users, clients,
tables, and indexes. After enabling
data collection in my.cnf or issuing
the SQL SET GLOBAL userstat_running
= 1 command at run time, four tables
in the information_schema, USER_STA‑
TISTICS, CLIENT_STATISTICS, INDEX_
STATISTICS, and TABLE_STATISTICS,
are continually populated with data.
The statistics can be accessed via the
SHOW command. For example, SHOW
TABLE_STATISTICS will give you a
table-by-table evaluation of lines read
and modified and indexes updated.
Direct access to the statistics tables in
information_schema is useful because
they are accessed as normal tables,
and you can target the results to
manipulate. Listing 1 shows a query
for the five tables with the most
frequently read lines. This example
taken from Live operations of the
Rails-based Moviepilot movie com-
munity and the underlying movie da-
tabase OMDB (both anonymized for
w w w. A d m i n - m AgA z i n e .co m
 MySQL Forks and Patches F e at u r e s

this evaluation) shows that read ac-
cess to the images table is particularly
frequent. The next step would be for
the database user to experiment with
code optimization or other changes to
the table format to reduce access in-
cidence and save time. The relatively
write-intensive movies table has a
suspiciously high write-access count
and, at the same time, a large number
of index updates.
Because the statistics can be reset
easily with FLUSH TABLE_STATISTICS,
interval-based evaluation by means of
a Munin plugin that you write your-
self, or some similar method, would
be your best bet. Retrospectively, you
could investigate load peaks in rela-
tion to table access and modification.
New Functions
New functions in the MySQL kernel
give administrators additional options
so that maintenance of the MySQL
server is more secure and conve-
nient. A typical task is to stop MySQL
processes with the KILL command.
Under load, you might see a process
listed as Idle by SHOW PROCESSLIST
start to handle a new query just at the
moment you kill it. The “Kill if Idle”
patch adds an option to kill a process
only if it is doing nothing: KILL IF_
IDLE Process_Id. This saves you the
embarrassment of accidentally killing
a process while it is handling a query.
LVM and ZFS snapshots are com-
monly regarded as the simplest
methods for backing up an InnoDB
database on the fly without inter-
rupting operations. If this method is
not an option for you and you are
forced to rely on a legacy dump file,
you need to make sure that the data
on your MySQL server do not change
while a file is being dumped. The
legacy approach to doing this is FLUSH
nodb_disallow_writes = 0 disables
the freeze.
Performance Enhancements
Thanks to its support for transactions
and line-based locking, InnoDB has
developed into a modern alternative
for the now fairly ancient MyISAM
engine. Despite performance gains
in write access thanks to line-based
locking, the overhead for support-
ing transactions, foreign keys, and
other functions (even if you don’t
use them) costs valuable CPU cycles
and hardware I/​O resources. MySQL
systems under heavy load thus need
a perfectly configured and powerful
InnoDB engine.
A variety of performance-boosting
patches are available for the version
of InnoDB that ships with MySQLM;
some of them are included in the
OurDelta version. One that is worthy
of mention is a reworked RW lock
that improves locking behavior on
multi-processor systems in particular.
A description of all the improvements
is beyond the scope of this article, but
one thing is clear; patches for InnoDB
exist that retain compatibility and
offer transparent optimization of the
engine. Typically, it is difficult to mea-
sure the performance gain in an ob-
jective way because the performance
of the MySQL server will depend to
a great extent on the hardware, con-
figuration, and data it uses. The most
exhaustive and reliable source is the
MySQL Performance Blog [4], which
regularly publishes test results.
Improvements by selectively installing
patches for the legacy InnoDB engine
are regarded as a fairly conservative
approach. The use of an alternative
Listing 1: Userstats Patch in Action
mysql> select * from information_schema.TABLE_STATISTICS ORDER BY ROWS_READ DESC LIMIT 0,5;
database engine holds more ­promise.
The InnoDB plug­in and Percona
XtraDB engines are becoming increas-
ingly widespread.
The InnoDB Plugin
The InnoBase InnoDB plugin is an
ongoing development of the InnoDB
engine that ships with MySQL [6].
Improvements include general optimi-
zation of CPU load and I/​O access, a
faster locking mechanism, extended
configuration and reporting options,
and optional table compression.
MySQL 5.1 introduced the option of
unloading the standard engine and
replacing it with a different version.
Starting with MySQL 5.1.38, MySQL
additionally supplies the InnoDB
plugin. As MySQL describes this as
a release candidate, administrators
do need to enable it manually. The
official MySQL documentation de-
scribes the steps required to do so
[7]. To benefit from the combination
of distribution updates for the MySQL
kernel and the latest functions and
optimizations of the InnoDB plugin, it
is a good idea to install the latest In-
noDB plugin version from the InnoDB
website.
Ubuntu 10.04 comes with MySQL
server version 5.1.41. The /usr/lib/
mysql/plugin/ houses an InnoDB pl-
ugin version 1.0.4 that is disabled by
default. The InnoDB website has the
current version, 1.0.6, which you can
download and unpack. Then copy
ha_innodb.so to the /usr/lib/mysql/
plugin/ directory. Because Ubuntu
uses AppArmor to protect services by
default, you need to disable or modify
AppArmor to let you load content
from the plugin directory by adding

TABLES WITH READ LOCK. However, this +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

might not be sufficient in the InnoDB | TABLE_SCHEMA | TABLE_NAME | ROWS_READ | ROWS_CHANGED | ROWS_CHANGED_X_INDEXES |

case because background processes +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

also write to the database. The In- | moviepilot | images | 13138219791 | 14778 | 118224 |
| moviepilot | events | 3957858216 | 59964 | 359784 |
noDB Freeze patch executes SET
| moviepilot | comments | 2650553183 | 3408 | 20448 |
GLOBAL innodb_disallow_writes = 1
| moviepilot | movies | 2013076357 | 598505 | 7780565 |
and then freezes all processes that
| omdb | log_entries | 1106683022 | 2737 | 5474 |
write InnoDB data so you can create
+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
a backup. Afterward, SET GLOBAL in‑

w w w. a d m i n - m aga z i n e .co m Admin 01 21

 F e at u r e s MySQL Forks and Patches

these lines to the /etc/apparmor.d/
usr.sbin.mysqld ruleset:
/usr/lib/mysql/plugin/ r,
/usr/lib/mysql/plugin/* mr,
Then restart Apparmor with the ser‑
vice apparmor restart command
to enable the new rules. In my.cnf,
you then need to enable the InnoDB
engine and load the InnoDB plugin
as shown in Listing 2. The InnoDB
plugin documentation contains de-
tailed information on this procedure.
The MySQL server error log contains
the message shown in Listing 3 after
loading the InnoDB plugin.
The documentation provides detailed
information on the optimizations
and new features offered in the In-
noDB plugin. One thing that stands
out against the rest is the new “Bar-
Listing 2: Loading the InnoDB Plugin into my.cnf
[mysql]
ignore_builtin_innodb
plugin_load=innodb=ha_innodb.so;
innodb_trx=ha_innodb.so;innodb_locks=ha_innodb.so;
innodb_lock_waits=ha_innodb.so;
innodb_cmp=ha_innodb.so;innodb_cmp_reset=ha_innodb.so;
innodb_cmpmem=ha_innodb.so;
innodb_cmpmem_reset=ha_innodb.so
Listing 3: With and Without the InnoDB Plugin
MySQL server without InnoDB plugin:
InnoDB: Started; log sequence number 0 44233
MySQL server with current InnoDB plugin:
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: highest supported file format is Barracuda.
InnoDB Plugin 1.0.6 started; log sequence number 44233
racuda” file format. In contrast to
the standard “Antelope” format, it
stores the InnoDB tables in a com-
pressed format. Although this could
cost additional CPU cycles, it will
give you huge I/​O performance sav-
ings – depending on your data struc-
ture – because far fewer operations
are required on disk/​SSD. To discover
whether it is worthwhile changing to
Barracuda, you will need to measure
performance. Tables with large text
and blob fields in particular will ben-
efit from compression. Incidentally,
MyISAM has supported “compressed”
tables for some time; however, you
cannot modify compressed MyISAM
tables in ongoing operations.
Percona XtraDB
Percona XtraDB [8] takes things one
step further than the InnoDB plugin.
This storage engine is a merge of the
current InnoDB plugin version with
additional performance and feature
patches. From a codebase point of
view, XtraDB is thus the most innova-
tive version of InnoDB. But don’t let
the name worry you: XtraDB is an In-
noDB engine. The new name simply
serves to underline the major differ-
ences between it and the version of
InnoDB that ships with MySQL.
Your easiest approach to installing
XtraDB is to resort to the MariaDB
packages created by OurDelta.
Maria­DB [9] itself is a MySQL fork
by the well-known MySQL developer
Michael “Monty” Widenius. Widenius
is working on a transactional alterna-
tive to MyISAM – the new Maria en-
gine. At the same time, the MariaDB
fork happily integrates XtraDB as a
high-performance update to InnoDB.
For the administrator, this means a
whole lot more optimizations: a state-
of-the-art MySQL version reworked
by the MariaDB project and extended
to include XtraDB (and Maria), along
with additional patches courtesy of
the OurDelta project.
A conversion from InnoDB to XtraDB
tables is not needed because XtraDB
replaces the standard InnoDB engine,
just as the InnoDB plugin does. Exist-
ing or new tables are automatically
managed by XtraDB. A downgrade to
the InnoDB plugin and the standard
InnoDB engine is also possible. To
see that XtraDB still refers to itself as
“InnoDB,” you can call SHOW ENGINES
– also, you will see the other modern
engines, such as Maria and PBXT,
here. Listing 4 shows the engines on
a current MariaDB server.
Tests on Live systems demonstrates
that migrating to the Maria­DB pack-
age is unproblematic for the most
part. MyISAM tables are left un-
changed; InnoDB tables continue to
work. However, MySQL-specific con-
figurations in my.cnf are interpreted
in a fairly strict manner. sql‑mode=NO_
ENGINE_SUBSTITUTION,TRADITIONAL
will put MySQL in traditional mode,
which handles many warnings as er-
rors. For example, a typical Rails-style
database migration failed because
it did not use completely standards-
compliant queries, such as setting de-
fault values for text and blob fields.

Listing 4: Engines on a MariaDB Server

MariaDB [(none)]> show engines;
+‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑+
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| FEDERATED | YES | FederatedX pluggable storage engine | YES | NO | YES |
| MARIA | YES | Crash‑safe tables with MyISAM heritage | YES | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| MyISAM | YES | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
| InnoDB | DEFAULT | Supports transactions, row‑level locking, and foreign keys | YES | YES | YES |
| PBXT | YES | High performance, multi‑versioning transactional engine | YES | YES | NO |
+‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑+

22 Admin 01 w w w. a d m i n - m aga z i n e .co m

 MySQL Forks and Patches F e at u r e s

In non-traditional mode, MySQL ig-
nores the default use and still runs
the queries; in traditional mode, the
query quits with an error. Thus, it is
a good idea in many cases to change
the line to sql‑mode=NO_ENGINE_SUB‑
STITUTION and then restart the
server. This problem is not specific
to MariaDB, simply a very restrictive
write out the InnoDB buffer to disk.
If you do need to restart the MySQL
or MariaDB server, the InnoDB engine
loses its valuable buffer pool in RAM.
Depending on your configuration and
the application, the pool can be sev-
eral gigabytes and might be filled in
the course of hours. Storing the buffer
pool before quitting and loading it
on the code of the not-yet-released
MySQL 6.0 and mainly pursues the
goals of removing unnecessary func-
tions and reducing complexity.
The developers really have been radi-
cal in the features they eradicated: E
Listing 5: Storing and Loading the Buffer Pool
// storing the buffer pool

configuration in line with the MySQL again after restarting will save valu- MariaDB [(none)]> select * from

standard. Additionally, it makes sense
to check whether programs compiled
against Libmysqlclient-dev need to be
recompiled against the current Lib-
mariadbclient-device. In a Rails envi-
ronment, this will affect Mysql-Gem.
Besides the benefits of the InnoDB
plugin described thus far, XtraDB also
offers a considerable performance
able warmup time, which you would
notice as slow response on the part of
the database server. Listing 5 shows
the commands and returns for storing
and loading the buffer pool.
Drizzle
At this point, I’ll take a quick look
information_schema.XTRADB_ADMIN_COMMAND
/*!XTRA_LRU_DUMP*/;
+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
| result_message |
+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
| XTRA_LRU_DUMP was succeeded. |
+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
// loading buffer pool and
MariaDB [(none)]> select * from information_schema.
XTRADB_ADMIN_COMMAND /*!XTRA_LRU_RESTORE*/;

boost, as a variety of benchmarks at a new development, Drizzle [11]. +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

with various setups proves [10]. Nu- According to the project, the fork is a | result_message |

merous additional functions make life
easier for developers and administra-
tors – and don’t forget the ability to
return to the original MySQL values:
simplicity, reliability, and perfor-
mance. Drizzle was originally based
+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
| XTRA_LRU_RESTORE was succeeded. |
+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

Make the most of your Data Center

Watch our video archiv of the Open Source Data Center
Conference – each session complete with slides.

Hot Topics
• High Availability
• Computer Clusters
• Load Balancing
• Configuration Management
• Security Management

Register here:
streaming.linux-magazin.de/en

w w w. a d m i n - m aga z i n e .co m Admin 01 23

Contact: Phone: +49 (0) 89 / 99 34 11 - 0 • Fax: +49 (0) 89 / 99 34 11 - 99 • E-Mail: streaming@linux-magazin.de
F e at u r e s MySQL Forks and Patches
MySQL Community Edition
MySQL 5.0 MySQL 5.1
Percona, Google,
and other patches MariaDB 5.1
Percona, Google,
and other patches
OurDelta MySQL 5.0 OurDelta MariaDB 5.1
Figure 1: The MySQL development community now has many forks.
storage engines such as Federated and
Merged have been removed; others,
such as CSV and MyISAM, have been
demoted to temporary engines. Mod-
ern engines such as XtraDB are main-
tained in a separate branch. The stan-
dard engine for Drizzle is InnoDB.
However, this does not mean that
data dumped from a classical MySQL
server with InnoDB tables can be
integrated without problems, because
Drizzle has also eradicated many field
types, such as TINYINT, TINYTEXT, and
YEAR. Migrating to Drizzle thus means
architectural changes to your data-
base design. Although a change from
TINYINT to INT could simply mean
searching and replacing occurrences
in a dump file, the lack of a YEAR field
can have a more serious effect on ex-
isting applications. A generic solution
for the migration does not exist.
On a more positive note, Drizzle of-
fers totally new replication mecha-
nisms. One feature that stands out
is the ability to perform rabbit repli-
cation to NoSQL databases such as
Voldemort [12] or services such as
Memcached [13]; thus, you would
be able to provision a variety of back
ends automatically from a central lo-
cation. As a state-of-the-art, high-per-
formance, non-transactional database
engine, the Drizzle project is working
on BlitzDB, which will be positioned
as an alternative to MyISAM.
Conclusions
The community’s response to the le-
thargic integration of patches into the
24 Admin 01
InnoDB
plugin MySQL 6.0
Percona
XtraDB Drizzle
Maria
PBXT
BlitzDB
FederatedX
Community Edition of MySQL server
is to launch new and active projects
(Figure 1). Existing MySQL 5.0 instal-
lations can be replaced easily by the
OurDelta MySQL 5.0 build, which ac-
celerates the server, thanks to perfor-
mance patches, and offers advanced
reporting functionality so the admin-
istrator can plan further steps on the
basis of run-time statistics. Installa-
tions of version 5.1 can benefit from
the latest optimizations by installing
the current InnoDB plugin – ideally
without needing to rebuilding. Migrat-
ing to the state-of-the-art MariaDB,
which outperforms the InnoDB plugin
in performance tests, turns out to be
more effective. Luckily, MariaDB is
packetized by the OurDelta project,
which also adds a number of addi-
tional patches.
The Drizzle database, with its simpli-
fied variant of InnoDB, is still at a
very early stage. The Maria engine
also represents a possible fast future
alternative to the classical combina-
tion of MyISAM/​InnoDB; however,
you need to perform extensive checks
before using it. Both projects’ engines
require architectural changes to the
database system and the program
code that accesses it, in contrast to
the InnoDB plugin, XtraDB, and pop-
ular MySQL patches.
Administrators and developers are
put in an ambivalent situation: Al-
though it has become inevitable for
administrators to concern themselves
with alternative MySQL patches,
engines, and forks and, ideally, to
deploy benchmarks to discover the
perfect solution for their require-
ments profile, this does involve a
considerable amount of overhead. At
the same time, the installation of the
server is no more complex than using
distribution packages thanks to pre-
packetized software.
Work is already in progress on inte-
grating modern forks into distribution
repositories [14]. The reward for all
this effort will ideally be a notice-
ably faster database server that helps
reduce hardware and development
costs. n
Info
[1] MySQL on Facebook:
[https://​­launchpad.​­net/​­mysqlatfacebook]
[2] Google patches: [http://​­code.​­google.​
­com/​­p/​­google‑mysql‑tools/​­wiki/​
­Mysql5Patches]
[3] Percona patches: [http://​­www.​­percona.​
­com/​­docs/​­wiki/​­patches:start]
[4] MySQL performance blog: [http://​­www.​
­mysqlperformanceblog.​­com/]
[5] Patches from OurDelta:
[http://​­ourdelta.​­org/​­patches]
[6] InnoDB plugin: [http://​­www.​­innodb.​­com/​
­products/​­innodb_plugin/​­features/]
[7] Installing the InnoDB plugin:
[http://​­dev.​­mysql.​­com/​­doc/​­refman/​­5.​­1/​­en/​
­innodb.​­html]
[8] Percona XtraDB: [http://​­www.​­percona.​
­com/​­docs/​­wiki/​­percona‑xtradb:start]
[9] MariaDB:
[http://​­askmonty.​­org/​­wiki/​­MariaDB]
[10] Benchmarks: InnoDB plugin and XtraDB vs.
InnoDB:
[http://​­www.​­mysqlperformanceblog.​­com/​
­2010/​­01/​­13/​­innodb‑innodb‑plugin‑vs‑xtrad
b‑on‑fast‑storage/]
[11] Drizzle: [http://​­drizzle.​­org/]
[12] Project Voldemort:
[http://​­project‑voldemort.​­com]
[13] “Memcached” by Tim Schürmann, Linux
Magazine, November 2009, pg. 28
[14] MariaDB in Ubuntu: [https://​­wiki.​­ubuntu.​
­com/​­Lucid‑MariaDB‑Inclusion]
The Author
Caspar Clemens Mierau’s “Screenage” project
provides consultancy services to Rails and PHP
portals such as moviepilot.de, omdb.org, and
Artfacts.Net. Caspar Clemens works as a free‑
lance author and is collecting literature for his
thesis on development environments.
w w w. a d m i n - m aga z i n e .co m

Microsoft Exchange 2010: The Highlights
Message Exchange
For years, Exchange has been the standard in-house server solution for all messaging tasks on Windows. This
article introduces the highlights of the new Exchange version 2010. By Björn Bürstinghaus
Exchange 2010 [1] is the latest gen-
eration of the Microsoft email server,
and it comes with a whole bunch
of new and useful functions for
administrators and users alike [2].
The version is even interesting for
companies currently using Exchange
2007: Besides an archiving function,
the new Exchange also integrates an
intelligent SMS gateway architecture,
thus removing the need for expensive
third-party add-ons.
Improved Management for
Admins
Many functions that posed tedious
Exchange Management Shell tasks for
administrators in Exchange 2007 have
now been integrated into the console.
For example, you can create new
certificates or view the current crop
of certificates at the console, without
needing to compose your own Power-
Shell request (Figure 1).
This version of Exchange also of-
fers a console view of the number of
licenses used in the company. This
new transparency allows administra-
tors to identify cases in which a com-
w w w. a d m i n - m aga z i n e .co m
pany is using more licenses than it
has purchased.
Giving users permission to create dis-
tribution groups and their members is
also new; this task can be handled in
the new, web-based Exchange Control
Panel (ECP). In ECP, users can modify
their own Active Directory informa-
tion, such as cellphone numbers or
addresses, without needing to contact
IT to do so. In this way, ECP will
probably make a major contribution
to reducing IT costs in the enterprise.
Whereas Exchange 2007 restricted
bulk changes to the PowerShell, Ex-
change 2010 finally lets administrators
make bulk changes in the manage-
ment console, so you can run a task
simultaneously against multiple mail-
boxes, which wasn’t possible in previ-
ous versions (Figure 2).
The new transport cache prevents
email messages transmitted via SMTP
from being deleted until the down-
stream node confirms that they have
been forwarded successfully. In other
words, if a hub transport server in
your company goes down, the trans-
port cache will retain the messages
until the server becomes available
Exchange 2010 F e at u r e s
© peapop, Fotolia.com
again or the transport rules are modi-
fied.
New Features from the
User’s Point of View
Exchange 2010 lets end users search
multiple mailboxes and send the re-
sults as a PST export to another per-
son. The Unified Messaging function
integrates a voicemail function for
each user in Exchange and supports
speech-to-text conversion, making it
possible to display a voicemail as well
as listen to the attached audio file.
The ability to display messages as
conversations both in Microsoft Of-
fice Outlook 2010 and in the Outlook
Web App gives users a clearer view of
their email folders. Users do not need
to check the sent items for email they
sent to the same person on the same
subject, which can save time.
The Outlook Wep App’s premium
functions were previously restricted
to Internet Explorer. Exchange 2010
lifts this restriction. The new Outlook
Web App now supports premium
functions in Mozilla Firefox and Ap-
ple Safari (Figure 3). E
Admin 01 25
F e at u r e s Exchange 2010

G Figure 2: Bulk Edit lets admins make changes to multiple recipients.

F Figure 1: Wizard for creating new certificates

Another new feature is the approach
to migrating mailboxes. Previously, a
user’s mailbox was switched offline
during migration to another mailbox
database, which meant delays of a
few minutes to several hours. Now,
while the mailbox is in transit, the
user can stay online. Exchange copies
the total content of the mailbox and
then synchronizes any changes that
occurred during the migration.
Archive Mailbox
Centralized archiving of email previ-
ously relied on third-party add-ons.
Again, Exchange 2010 puts an end
to this. In addition to a normal mail-
box, the administrator can create an
archive for each member of staff, al-
though this is a premium feature and
does require an Exchange Premium
Client Access License for each user
(Figure 4). Once enabled, the archive
is automatically displayed in the Out-
look 2010 and the Outlook Web App
folder navigation, in addition to the
normal mailbox.
This solution allows administrators to
create an Exchange solution quickly
that lets end users restore messages
they deleted from their mailboxes
without having to call the help desk.
All you need for this is the journaling
function, the enterprise-wide policies
to use for automatically synchroniz-
ing or moving email from the mailbox
to the archive mailbox, and a match-
ing retention policy that prevents
deletion.
Exchange 2010 currently does not of-
fer the same feature set as archiving
solutions by third parties such as
MailStore Server or GFI Mail Ar-
chiver; however, the functionality that
is currently available does relieve the
burden on mailboxes, and it puts an
end to the “PST Hell” that some ad-
ministrators rightly complained of.
Text Messages
If a company’s employees use smart-
phones with Windows Mobile 6.5,

Figure 4: The mailbox configuration now supports

Figure 3: The Outlook Web App implements a mail client in the web browser. Archiving for email.

26 Admin 01 w w w. a d m i n - m aga z i n e .co m

they can now synchronize
text messages with their Ex-
change mailbox, along with
email messages, contacts,
appointments, and tasks.
Exchange previously did
not offer an option for send-
ing or receiving text mes-
sages (unless you relied on
a third-party product), and
SMS gateways were the only
option for integrating text
message functionality. Un-
fortunately, this meant that
the SMS gateway number
was the same for the whole
company, rather than having
extensions provided for indi-
vidual users.
Exchange 2010, in combi-
nation with the Windows
Mobile 6.5 platform, uses
the smartphone as the SMS
gateway and thus supports
composing or reading per-
sonalized text messages with
Outlook 2010 or Outlook Web
App.
When you send a text mes-
sage via Exchange, the mes-
sage is forwarded by the
server via an ActiveSync
interface to the Smartphone,
which then uses the em-
ployee’s cellphone number
to send the message; this
approach is far more flex-
ible than using a centralized
gateway.
Improved Availability
The 24/7 availability of ser-
vices plays an important role
in many large corporations
that need permanent avail-
ability across multiple time
zones because of globaliza-
tion.
Exchange 2010 takes this one
step further than the previ-
ous versions: Windows Clus-
ter lets you group up to 16
mailbox servers in a cluster,
which means databases can
be replicated on up to 15 ad-
ditional servers.
w w w. A d m i n - m AgA z i n e .co m
Also, it provides an option of
integrating the logfiles for a
database at a later stage. You
can refer to the help [3] or
forum [4] websites for addi-
tional information.
Is It Worthwhile?
If you are planning to invest
in an email archiving solu-
tion or an SMS gateway,
you might consider moving
to Exchange 2010, instead,
because it already integrates
many of these functions.
The new version also im-
proves and facilitates the
experience of integrating
subsidiaries, so a change
could also help reduce your
company’s IT costs.
The test version of Exchange
supports virtualization, thus
giving those who are inter-
ested in migrating a chance
to familiarize themselves
with the new functions and
options before they invest in
new hardware. ■
Info
[1] Exchange Server 2010 product
page: [http://technet.microsoft.
com/en-us/exchange/dd203064.
aspx]
[2] Exchange Server 2010 new
features: [http://technet.
microsoft.com/en-us/library/
dd298136.aspx]
[3] Exchange Server 2010 help:
[http://technet.microsoft.com/
en-us/library/bb124558.aspx]
[4] Exchange Server 2010 Fo-
rum: [http://social.technet.
microsoft.com/Forums/en-US/
exchange2010/threads]
The Author
Björn Bürstinghaus is a systems
administrator with simyo GmbH in
Düsseldorf, Germany. In his leisure
time, he runs Björn’s Windows blog,
a blog on Microsoft Windows topics.
You can find his blog at [http://blog.
buerstinghaus.net] (in German).
Admin 01 27
 F e at u r e s Backup Software
Backup tools to save you time and money
Safety Net
© Melissa King, 123RF.com
One small oversight can cost you hours of extra work and your
company thousands of dollars. Here are a few backup tools to
help you recover gracefully. By James Mohr
Computers are of little value if they
are not doing what they’re supposed
to, whether they’ve stopped work-
ing or they’re not configured cor-
rectly. Redundant systems or standby
machines are common methods of
quickly getting back to business, but
they are of little help if the problem is
caused by incorrect configuration and
those configuration files get copied
to the standby machines. Sometimes,
the only solution is to restore from
backup.
Backing up all of your data every day
is not always the best approach. The
amount of time and the amount of
storage space required can be limiting
factors. On workstations with only a
handful of configuration files and few
data files, it might be enough to store
these files on external media (such
as USB drives). Then, if the system
should crash, it could be simpler to
reinstall and restore the little data you
have. Doing a full system restore of-
ten takes longer.
Because each system is different, no
single solution is ideal. Even if you
find a product that has all the fea-
tures you could imagine, the time and
effort needed to administer the soft-
ware might be restrictive. Therefore,
knowing the most significant aspects
of backups and how these can be
28 Admin 01
addressed in the software are key is-
sues when deciding which product
to implement (see also the “Support”
box).
How Strong Is Your
Parachute?
A simple copy of your data to a lo-
cal machine with rsync started from
a cron job once a day is an effective
way to back up data when everything
comes from a single directory. This
method is too simple for businesses
that have much more and many dif-
ferent kinds of data – sometimes
so much data that it is impractical
to back it all up every day. In such
cases, you need to make decisions
about what to back up and when (see
the “Backup Alternatives” box).
In terms of what to back up and how
often, files increase in importance
on the basis of how difficult they are
to recreate. The most important files
are your “data,” such as database
files, word-processing documents,
spreadsheets, media files, email, and
so forth. Such files would be very
difficult to recreate from scratch, so
they must be protected. Configuration
information for system software, such
as Apache or your email server, typi-
cally changes frequently, but these
files still need to be backed up.
Sometimes, reinstalling is not prac-
tical if you have to install a lot of
patches, or it might be impossible if
you have software that is licensed for
a particular machine and you cannot
simply reinstall the OS without ob-
taining a new license key.
How often you should back up your
files is the next thing to determine.
For example, database files could be
backed up to remote machines every
15 minutes, even if the files are on
cluster machines with a hardware
RAID. Local machines could be
backed up twice a day: The first time
to copy the backup from the previ-
ous day to an external hard disk, and
the second time to create a new full
backup. Then, once a month, the
most important data could be burned
to DVD (see also the “Incremental vs.
Differential” box).
What Color Is your
Parachute?
Individuals and many businesses are
not concerned with storing various
versions of files over long periods
of time. However, in some cases, it
might be necessary to store months
or even years worth of backups, and
the only effective means to store them
is on removable media (e.g., a tape
drive). Further requirements to store
backups off-site often compound
the problem. Although archives can
be kept on external hard disks, it
becomes cumbersome when you get
into terabytes of data.
In deciding which type of backup
medium you need to use, you have to
consider many things. For example,
you need to consider not only the
total amount of data but also how
many machines you need to back
up. Part of this involves the ability to
distinguish quickly between backups
from different machines. Moving an
external hard disk between two ma-
chines might be easiest, but with 20
machines, you should definitely con-
sider a centralized system.
Here, too, you must consider the
speed at which you can make a
backup and possibly recover your
w w w. a d m i n - m aga z i n e .co m

data. One company I worked for had
so much data it took more than a
day to back up all of the machines.
Thus, full backups were done over
the weekend, with incrementals in
between. Also, in a business envi-
ronment with dozens of machines,
trying to figure out exactly where the
specific version of the data resides
increases the recovery time consider-
ably.
Finally, you must also consider the
cost. Although you might be tempted
to get a larger single drive because
it is less expensive than two drives
that are only half as big, being able to
switch between two drives (or more)
adds an extra level of safety if one
fails. Furthermore, you could poten-
tially take one home every night. If
you are writing to tape, an extra tape
drive also increases safety; it can also
speed up backups and recovery.
Which Tape?
Some companies remove all of the
tapes after the backup is completed
and store them in a fireproof safe or
somewhere off-site. This means that
when doing incremental backups,
the most recent copy of a specific
file might be on any one of a dozen
tapes. Naturally, the question be-
comes, “Which tape?” (see also the
“Whose Data” box). To solve this
problem, the backup software must
be able keep track of which version of
Incremental vs. Differential
Because of the amount of data, businesses
frequently have a two-tiered backup scheme.
Once a week, a full backup is done (of every
single file); on subsequent days, backups are
done of only those files that have changed.
This approach is referred to as an incre-
mental backup. Although it saves media, it
potentially takes more time to recover. With
this method, you first need to restore the
full backup and, depending on which files
have changed, you might need to access
every single incremental backup.
One alternative is a differential backup,
which stores only files that have been
changed since the last full backup. This has
the advantage of saving time compared with
an incremental backup, because you need to
restore from, at most, two backups.
w w w. a d m i n - m aga z i n e .co m
Backup Software F e at u r e s
Backup Alternatives
If you are running Linux and your software repositories are configured properly, a number of
backup applications are available through your respective installation tool (e.g., YaST, Synaptic).
In fact, I found more than two dozen products that have defined themselves in one way or an-
other as a backup tool (not counting those explicitly for backing up databases).
Here are a few important questions to ask about your backup software:
n Is your hardware supported?
n How does the software deal with database backups?
n Can you do a directed recovery (i.e., to a different directory)?
n Can the software verify the data after a backup and restore?
n Can the software write to multiple volumes?
n Do you really need all of the features?
n Can the software do a backup of a remote system?
which file is stored where (i.e., which Many of the products I looked at have
tape or disk). the ability to define “profiles” (or
Once a software product has reached use a similar term). For example, you
this level, it will typically also be define a Linux MySQL profile, assign
able to manage multiple versions of it to a subset of your machines, and
a given file. Sometimes you will need the backup software automatically
to make monthly or even yearly back- knows which directories to include
ups, which are then stored for longer and which to ignore. The Apache pro-
periods of time. (This setup is com- file, for example, has a different set
mon when you have sensitive data of directories. This might also include
like credit card or bank information.) a pre-command that is run immedi-
To prevent the software from over- ately before the backup, then a post-
writing tapes that it shouldn’t, you command that is run immediately
should be able to define a “recycle afterward.
time” that specifies the minimum
amount of time before the media can Storage
be reused.
Because not all backups are the same How is the backup information
and not all companies are the same, stored? Does the backup software
you should consider the ability of have its own internal format or does
the software to be configured to your it use a database such as my SQL?
needs. If you have enough time and The more systems you back up, the
space, software that can only do a more you need a product that indexes
full backup might be sufficient. On which files are saved and where they
the other hand, you might want to be are saved as well. Unless you are
able to pick and choose just specific simply doing a complete backup ev-
directories, even when doing a “full” ery night to one destination for one
backup. machine (i.e., one tape or remote di-
Support Whose Data?
One consideration that is often overlooked One important aspect is the ability to write
is the amount of support available for your data from different sources to specific
product. Commercial support might be nec- media. For example, where I work, each
essary if implementing the backup solution customer is assigned specific tapes (often
for a company. However, the amount of free referred to as a “pool”). With the use of
support (forums, mailing lists) can be an is-
labels written to the tape, the software can
sue. When considering open source software
tell which tape belongs to which pool, so
of any kind for a business, I always suggest
that data from different environments is
taking a good look at the product’s website.
If the product has not been updated in three not mixed. This scheme is very useful if, for
years, you might want to look elsewhere. If example, one customer wants weekly back-
forums have few posts and most are unan- ups stored off-site and another customer
swered, you likely won’t get your questions frequently requests the backup tapes to load
answered either. them into a local test system.
Admin 01 29
F e at u r e s Backup Software
rectory), finding the right location for
a given file can be a nightmare. Even
if you are dealing with just a few sys-
tems, administration of the backups
can become a burden.
This leads into the question of how
easy it is to recover your data. Can
you easily find files from a specific
date if there are multiple copies? How
easy is it to restore individual files?
What about all files changed on a
specific date?
Depending on your business, you
might have legal obligations in terms
of how long you are required to keep
certain kinds of data. In some cases,
it might be a matter of weeks; in
other cases, it can be 10 years or lon-
ger. Can you recover data from that
long ago? Even if it’s not required by
law, having long-term backups is a
good idea. If you accidentally delete
something and don’t notice it has
happened for a period longer than
your backup cycle, you will probably
never get your data back. How easy is
it for your backup software to make
full backups at the end of each month
– for example, to ensure that the me-
dia does not get overwritten?
Scheduling
If your situation prevents you from
doing complete backups all the time,
consider how easy it is to schedule
them. Can you ensure that a complete
backup is done every weekend, for
example?
Also, you need to consider the sched-
uling options for the respective tool.
Can it start backups automatically? Is
it dependent on some command? Is it
simply a GUI for an existing tool, and
all the operations need to be started
manually? Just because a particular
operating system has no client does
not mean you are out of luck: You
can mount filesystems using Samba
or NFS and then back up the files.
rsync
Sometimes you do not need to look
farther than your own backyard.
Rsync is available for all Linux distri-
butions, all major Unix versions, Mac
30 Admin 01
OS X, and Windows. With a handful
of machines, configuring rsync by
hand might be a viable solution. If
you prefer a graphical interface, sev-
eral different graphical interfaces are
available. In fact, many different ap-
plications rely on it to do the backup.
The rsync tool can be used to copy
files either from a local machine to
a remote machine or the other way
around. A number of features also
make rsync a useful tool for synchro-
nizing directories (which is part of its
name). For example, rsync can ignore
files that have not been changed since
the last backup, and it can delete files
on the target system that no longer
exist on the source. If you don’t want
existing files to be overwritten but
still want all of the files to be copied,
you can tell rsync to add a suffix to
files that already exist on the target.
The ability to specify files and direc-
tories to include or exclude is very
useful when doing backups. This can
be done by full name or with wild-
cards, and rsync allows you to specify
a file that it reads to determine what
to include or exclude. When deter-
mining whether a file is a new ver-
sion or not, rsync can look at the size
and modification date, but it can also
compare a checksum of the files.
A “batch mode” can be used to up-
date multiple destinations from a
single source machine. For example,
changes to the configuration files can
be propagated to all of your machines
without having to specify the change
files for each target. Rsync also has a
GUI, Grsync [Figure 1].
luckyBackup
At first, I was hesitant to go into de-
tails about luckyBackup [1], because
it is still a 0.X version and it has a
somewhat “amateurish” appearance.
However, my skepticism quickly
faded as I began working with it.
luckyBackup is very easy to use and
provides a surprising number of op-
tions. Despite its simplicity, lucky-
Backup had the distinction of winning
third place in the 2009 SourceForge
Community Choice Awards as a “Best
New Project.”
The repository I used had version
0.33, so I downloaded and installed
that (although v0.4.1 is current). The
source code is available, but various
Linux distributions have compiled
packages.
Describing itself as a backup and
synchronization tool, luckyBackup
uses rsync, to which it passes various
configuration options. It provides the
ability to pass any option to rsync, if
necessary. Although it’s not a client-
server application, all it needs is an
rsync connection to back up data
from a remote system.
When you define which files and
directories to back up, you create
a “profile” that is stored under the
user’s home directory. Profiles can be
imported and exported, so it is pos-
sible to create backup templates that
are copied to remote machines. (You
still need the luckyBackup binary to
run the commands.)
Each profile contains one or more
tasks, each with a specific source
and target directory, and includes the
configuration options you select [Fig-
ure 2]. Thus, it is possible to have
different options for different directo-
ries (tasks), all for a single machine
(profile).
Within a profile, the tool makes it
easy to define a restore task on the
basis of a given backup task. Essen-
tially, this is the reverse of what you
defined for the backup task, but it is
very straightforward to change op-
tions for the restores, such as restor-
ing to a different directory.
Scheduling of the backup profiles is
done by cron, but the tool provides a
simple interface. The cron parameters
are selected in the GUI; you click a
button, and the job is submitted to
cron.
A console, or command-line mode,
allows you to manage and configure
your backups, even when a GUI is
not available, such as when connect-
ing via ssh. Because the profiles are
stored in the user’s home directory, it
would be possible for users to create
their own profile and make their own
backups.
Although I would not recommend
it for large companies (no insult in-
w w w. a d m i n - m aga z i n e .co m

tended), luckyBackup does provide a
basic set of features that can satisfy
home users and small companies.
Amanda
Initially developed internally at the
University of Maryland, the Advanced
Maryland Automatic Network Disk
Archiver (Amanda) [2] is one of the
most widely used open source backup
tools. The software development is
“sponsored” by the company Zmanda
[3], which provides an “enterprise”
version of Amanda that you can pur-
chase from the Zmanda website. The
server only runs on Linux and Solaris
(including OpenSolaris), but Mac OS
X and various Windows versions also
have clients.
The documentation describes
Amanda has having been designed to
work in “moderately sized computer
centers.” This and other parts of the
product description seem to indicate
the free, community version might
have problems with larger computer
centers. Perhaps this is one reason for
selling an “enterprise” version. The
latest version is 3.1.1, which came
out in June 2010, but it just provided
bug fixes. Version 3.1.0 was released
in May 2010.
Amanda stores the index of the files
and their locations in a text file. This
naturally has the potential to slow
Figure 1: Grsync – A simple front end to rsync.
w w w. a d m i n - m aga z i n e .co m
Backup Software
down searches when you need to
recover specific files. However, the
commercial version uses MySQL to
store the information.
Backups from multiple machines
can be configured to run in parallel,
even if you only have one tape drive.
Data are written to a “holding disk”
and from there go onto tape. Data
are written with the use of standard
(“built-in”) tools like tar, which
means data can be recovered inde-
pendently from Amanda. Proprietary
tools typically have a proprietary
format, which often means you can-
not access your data if the server is
down.
Scheduling is also done with a local
tool: cron. Commands are simply
started at the desired time with the
respective configuration file as an
argument.
Amanda supports the concept of “vir-
tual tapes,” which are stored on your
disk. These can be of any size smaller
than the physical hard disk. This ap-
proach is useful for splitting up your
files into small enough chunks to be
written to DVD, or even CD.
Backups are defined by “levels,” with
0 level indicating a full backup; the
subsequent levels are backups of the
changes made since the last n - 1 or
less. The wiki indicates that Aman-
da’s scheduling mechanism uses
these levels to implement a strategy
Figure 2: luckyBackup profile configuration.
F e at u r e s
of “optimization” in your backups.
Although optimization can be useful
in many situations, the explanation
is somewhat vague about how this is
accomplished – and vague descrip-
tions of how a system makes deci-
sions on its own always annoy me.
One important caveat is that Amanda
was developed with a particular envi-
ronment in mind, and it is possible (if
not likely) that you will need to jump
through hoops to get it to behave
the way you want it to. The default
should always be to trust the admin-
istrator, in my opinion. If the admin
wants to configure it a certain way,
the product shouldn’t think it knows
better.
For example, you should determine
whether the scheduling mechanism is
doing full backups at times other than
when you expect or even want. In
many cases, large data centers do full
backups on the weekend when there
is less traffic and not simply “every
five days.” If your installation has
sudden spikes in data, Amanda might
think it knows better and change the
schedule.
Although such situations can be ad-
dressed by tweaking the system, I
have a bad feeling when software
has the potential for doing something
unexpected. After all, as a sys admin,
I was hired to think, not simply to
push buttons. To make things easier
Admin 01 31
F e at u r e s Backup Software
in this regard, Zmanda recommends
their commercial enterprise product.
Although Amanda has been around
for years and is used by many organi-
zations, I was left with a bad taste in
my mouth. Much of the information
on their website was outdated, and
many links went to the commercial
Zmanda website, where you could
purchase their products. Additionally,
a page with the wish list and planned
features is as old as 2004. Although a
note states that the page is old, there
is no mention of why the page is still
online or any explanation of what
items are still valid. Half of the pages
on the administration table of con-
tents (last updated in 2007) simply
list the title with no link to another
page.
Also, I must admit I was shocked
when I read the “Zmanda Contribu-
tor License Agreement.” Amanda is
an open source tool, which is freely
available to everyone. However, in the
agreement “you assign and transfer
the copyrights of your contribution
to Zmanda.” In return, you receive
a broad license to use and distribute
your contribution. Translated, this
means you give up your copyright
and not simply give Zmanda the right
to use it, which also means Zmanda
is free to add your changes to their
commercial product and make money
off of it – and all you get is a T-shirt!
Areca Backup
Sitting in the middle of the features
spectrum and somewhat less well
known is Areca Backup [4]. Run-
ning from either a GUI [Figure 3]
or a command-line interface, Areca
provides a simple design and a wide
range of features. The documentation
says it runs on all operating systems
with Java 1.4.2 or later, but only Li-
nux and Windows packages are avail-
able for download. Installing it on
my Ubuntu systems was no problem,
and I could not find any references to
limitations with specific distributions
or other operating systems.
Areca is not a client-server applica-
tion, but rather a local filesystem
backup. The Areca website explicitly
32 Admin 01
states that it cannot do filesystem or
disk images, nor can it write to CDs
or DVDs. Backups can be stored on
remote machines with FTP or FTPS,
and you can back up from remotely
mounted filesystems, but with no re-
mote agent. Areca provides no sched-
uler, so it expects you to use some
other “task-scheduling software”
(e.g., cron) to start your backup auto-
matically.
In my opinion, the interface is not
as intuitive as others, and it uses ter-
minology that is different from other
backup tools, making for slower prog-
ress at the beginning. For example,
the configuration directory is called
a “workspace” and a collection of
configurations (which can be started
at once) is a “group,” as opposed to a
collection of machines.
Areca provides three “modes,” which
determine how the files are saved:
standard, delta, and image. The
standard mode is more or less an
incremental backup, storing all new
files and those modified since the
last backup. The delta mode stores
the modified parts of files. The image
mode is explicitly not a disk image;
basically, it is a snapshot that stores
a unique archive of all your files with
each backup. The standard backups
(differential, incremental, or full) de-
termine which files to include.
The GUI provides two views of your
backups. The physical view lists the
archives created by a given target.
Figure 4: Bacula admin tool.
The logical view is a consolidated
view of the files and directories in the
archive.
Areca is able to trigger pre- and post-
actions, like sending a backup report
by email, launching shell scripts
before or after your backup, and so
forth. It also provides a number of
variables, such as the archive and
computer name, which you can pass
to a script. Additionally, you can de-
fine multiple scripts and specify when
they should run. For example, you
can start one script when the backup
is successful but start a different one
if it returns errors.
Areca provides a number of interest-
ing options when creating backups. It
allows you to compress the individual
files as well as create a single com-
pressed file. To avoid problems with
very large files, you can configure
the backup to split the compressed
archive into files of a given size. Also,
you can encrypt the archives with ei-
ther AES 128 or AES 256. One aspect
I liked was the ability to drop directo-
ries from the file explorer directly into
Areca.
The Areca forum has relatively low
traffic, but posts are fairly current.
However, I did see a number of recent
posts remain unanswered for a month
or longer. The wiki is pretty limited,
so you should probably look through
the user documentation, which I
found to be very extensive and easy
to understand.
w w w. a d m i n - m aga z i n e .co m

Two “wizards” also ease the creation
of backups. The Backup Shortcut
wizard simplifies the process of creat-
ing the necessary Areca commands,
which are then stored in a script that
you can execute from the command
line or with cron.
The Backup Strategy wizard generates
a script containing a set of backup
commands to implement a specific
strategy for the given target. For ex-
ample, you can create a backup every
day for a week, a weekly backup for
three weeks, and a monthly backup
for six months.
Bacula
The Backup Dracula “comes by
night and sucks the vital essence
from your computers.” Despite this
somewhat cheesy tag line, Bacula
[5] is an amazing product. Although
it’s a newer product than Amanda, I
definitely think it surpasses Amanda
in both features and quality. To be
honest, the setup is not the point-
and-click type that you get with other
products, but that is not really to be
expected considering the range of fea-
tures Bacula offers.
Although Bacula uses local tools to
do the backup, it is a true client-
server product with five major com-
ponents that use authenticated com-
munication: Director, Console, File,
Storage, and Catalog. These elements
are deployed individually on the ba-
Figure 3: Areca backup.
w w w. a d m i n - m aga z i n e .co m
Backup Software
sis of the function of the respective
machine.
The Director supervises all backup,
restore, and other operations, includ-
ing scheduling backup jobs. Backup
jobs can start simultaneously as well
as on a priority basis. The Director
also provides the centralized control
and administration and is responsible
for maintaining the file catalog. The
Console is used for interaction with
the Bacula director and is available as
a GUI or command-line tool.
The File component is also referred
to as the client program, which is the
software that is installed on the ma-
chines to be backed up. As its name
implies, the Storage component is
responsible for the storage and recov-
ery of data to and from the physical
media. It receives instructions from
the Director and then transfers data to
or from a file daemon as appropriate.
It then updates the catalog by send-
ing file location information to the
Director.
The Catalog is responsible for main-
taining the file indexes and volume
database, allowing the user to locate
and restore files quickly. The Catalog
maintains a record of not only the
files but also the jobs run. Currently,
Bacula supports MySQL, PostgreSQL,
and SQLite. As of this writing, the
Directory and Storage daemons on
Windows are not directly supported
by Bacula, although they are reported
to work.
F e at u r e s
One interesting aspect of Bacula is the
built-in Python interpreter for script-
ing that can be used, for example,
before starting a job, on errors, when
the job ends, and so on. Addition-
ally, you can create a rescue CD for a
“bare metal” recovery, which avoids
the necessity of reinstalling your sys-
tem manually and then recovering
your data. This process is supported
by a “bootstrap file” that contains a
compact form of Bacula commands,
thus allowing you to store your sys-
tem without having access to the
Catalog.
The basic unit is called a “job,”
which consists of one client and one
set of files, the level of backup, what
is being done (backing up, migrating,
restoring), and so forth.
Bacula supports the concept of a “me-
dia pool,” which is a set of volumes
(i.e., disk, tape). With labeled vol-
umes, it can easily match the external
labels on the medium (e.g., tape) as
well as prevent accidental overwrit-
ing of that medium. It also supports
backing up to a single medium from
multiple clients, even if they are on
different operating systems.
The Bacula website is not as fancy as
Amanda, but I found it more useful
because the details about how the
program works are much more acces-
sible, and the information is more up
to date.
The Right Fit
Although I only skimmed the surfaces
of these products, this article should
give you a good idea what is possible
in a backup application. Naturally,
each product has many more features
than I looked at, so if any of these
products piqued your interest, take a
look at the website to see everything
that product has to offer. n
Info
[1] luckyBackup:
[http://​­luckybackup.​­sourceforge.​­net]
[2] Amanda: [http://​­www.​­amanda.​­org]
[3] Zmanda: [http://​­www.​­zmanda.​­com]
[4] Areca Backup: [http://​­www.​­areca‑backup.​­org]
[5] Bacula: [http://​­www.​­bacula.​­org]
Admin 01 33
F e at u r e s BlackHat 2010
BlackHat USA 2010
Learning
from the Best
The latest and greatest security issues By Kurt Seifried
I’ve been to BlackHat twice now,
and both times I have taken the same
lesson home: If you think things are
getting better in the field of computer
security, you’re probably wrong. Over
the years, progress has been made
identifying bug types – currently the
CWE lists 668 weaknesses in 120
categories – and some progress has
been made with projects to identify
and remove them systematically (e.g.,
OpenBSD has had remarkable suc-
cess). However, you then come to
the BlackHat conference and see a
presentation like “HTTPS Can Byte
Me,” in which Robert Hansen and
Josh Sokol disclosed 24 vulnerabili-
ties (Figure 1) that can compromise
the integrity and security of SSL-
encrypted web traffic [2].
The problem is not so much a failing
within SSL, but unless you’re taking
extreme measures to protect network
traffic against analysis (e.g., padding
traffic out, introducing time delays,
etc.), chances are, attackers will be
able to glean information even if they
can’t read the traffic directly.
Also, consider the case of the well-
meaning web browser that attempts
to be helpful. I guess people hate typ-
Figure 1: Final HTTPS slide – 24 issues in all.
34 Admin 01
ing in personal information, so almost
all browsers support “auto-complete,”
which automatically fills out form
fields (e.g., name, address, and credit
card number). Unfortunately, this
feature can be abused by attackers
(imagine that), allowing them to steal
personal information saved within
your web browser if you visit a web
page. Using JavaScript, they can set
it up so you don’t even have to type
anything in – combined with a hid-
den IFRAME, you might never realize
that it happened.
The security talks are especially wor-
rying – the ones in which research-
ers don’t find new vulnerabilities
but simply quantify existing ones.
In the case of SSL certificates, they
scanned the Internet and found 1.2
million SSL-enabled websites [3] [4].
Among the problems found were cer-
tificates for reserved addresses (e.g.,
192.168.1.2, a reserved IP address
used by multiple sites) that never
should have been allowed. Also, they
found 50 percent of servers config-
ured to allow SSLv2 (known to be
insecure for 14 years).
Now, I’m not a glass half empty kind
of guy, but seeing 50 percent of serv-
ers configured insecurely is
a bit depressing (which is
probably why most security
people buy beer in pitchers,
not glasses).
BlackHat isn’t an unend-
ing stream of bad news,
however. Many of the pre-
sentations not only present
problems but also discuss
the solutions. The perfect
example was a presenta-
tion called “Lifting the Fog” [5], in
which Marco Slaviero scanned for
memcached (a memory-caching
program widely used to speed up
web-based applications). He found
many memcached servers open to
the world, and by using two poorly
documented commands, stat detail
on (which enables debugging) and
stats cachedump (which lists all the
key names), he was able to retrieve
all the items stored in the memcached
server. And by “all” I mean every-
thing; according to his presentation,
he retrieved 136TB of data from 229
memcached servers.
The good news is that securing your
memcached is simple: Firewall it so
that only local trusted systems can
connect to it (and if you must use
it over the Internet, set up a VPN to
connect systems to it). This solution
is not magical, but it drives home the
point that you need to test and verify
security measures using tools like
Nmap [6] (which is how he found all
the memcached instances).
So, if you need an excuse (well, a
work-related excuse) to go to Las Ve-
gas, BlackHat, and Defcon afterward,
they’re not only a lot of fun, but very
educational. My only complaint is
that with 10 tracks, chances are you’ll
have to choose between two or more
interesting talks, which is definitely a
glass half full type of problem. n
Info
[1] Common Weakness Enumeration:
[http://​­cwe.​­mitre.​­org/]
[2] HTTPS Can Byte Me:
[https://​­media.​­blackhat.​­com/​­bh‑us‑10/​
­whitepapers/​­Hansen_Sokol/​­Blackhat‑USA
‑2010‑Hansen‑Sokol‑HTTPS‑Can‑Byte‑Me
‑wp.​­pdf]
[3] SSL Observatory:
[http://​­www.​­eff.​­org/​­observatory]
[4] Internet SSL Survey 2010:
[http://​­blog.​­ivanristic.​­com/​­Qualys_SSL_
Labs‑State_of_SSL_2010‑v1.​­6.​­pdf]
[5] Lifting the Fog:
[https://​­media.​­blackhat.​­com/​­bh‑us‑10/​
­presentations/​­Slaviero/​­BlackHat‑USA‑2010
‑Slaviero‑Lifting‑the‑Fog‑slides.​­pdf]
[6] “Nmap scripting” by Eric Amberg, Linux
Magazine, February 2008, pg. 68
w w w. a d m i n - m aga z i n e .co m
Backup & Disaster Recovery
Georgetown University Chooses SEP sesam Backup Solution
The Situation
Georgetown University’s McDonough School of Business (MSB),
one of the most renowned business schools in the United States,
called SEP sesam to replace their under-performing backup
software. MSB is experiencing a period of strong program growth.
The MSB Technology Canter has been tasked with keeping their IT
systems up to date and required a state of the art backup system
to ensure continuity of operations. After a brief discussion,
SEP was able to analyze current problems and provide reliable
backups for critical data.
During the course of using the old solution, administrators were
continually asked to reconfigure and restart their backup systems
even though changes were not being made to the environment or
the network infrastructure.
The Challenge
MSB’s new data and applications services requirements had
outstripped its legacy backup software. The old solution was
not flexible enough to meet new demands without continuous
monitoring. The old system continually failed during overnight
backup tasks. Each error and failed backup required a lengthy
call to vendor tech support and often required custom code
changes and hot patches. The situation finally became untenable
when the software could not work with a newly purchased EMC
DL3D1500 Disk Library. This final straw initiated an active search
for a better and more effective backup solution.
Backup & Disaster Recovery
For more information visit:
www.sepsoftware.com | www.sepusa.com
The Solution - SEP sesam
Mike Yandrischovitz, Data Systems and Security Manager for the business
school, consulted with other members of the user community and
discovered SEP sesam. After contacting SEP, he downloaded and installed
the software. In less than two hours, Mike, along with SEP assistance,
was able to get backups for McDonough’s most critical applications.
The decision to move from the old vendor was still difficult. Business
school staff had invested a great deal financially and even more in time
and lost productivity. Nevertheless it was decided to make a change to
SEP sesam.
“James Delmonico, at SEP, had us up and running in hours. I was
getting substantial ‘heat’ from our user community because our backup
solution was unstable. We were not able to get reliable backups using
the old backup software. Thanks to the SEP sesam solution we had
reliable backups almost immediately and restores of critical data for
our customers’ everyday requirements were fast, easy and accurate.
We are now a great fan of the software and the team at SEP,” said
Yandrishovitz.
“SEP engineers were even instrumental in using the SEP software to
help diagnose a configuration problem with our new SAN. Isolating the
problem, we were able to pinpoint an issue with the SAN Switch. The
vendor reconfigured the switch and now, with the new software and new
hardware, our backups are completed within time windows previously
considered unattainable.”
According to John Carpenter, McDonough Chief Technology Officer, “SEP
sesam and their helpful engineers took a major worry off our plates. The
new implementation has performed better than we expected. Our staff
can now go home on time and I’vesaved the cost of acquisition of SEP
sesam by returning scheduled overtime back to the operating budget.
Backups that used to take a whole weekend are now complete in under
eight hours.”
Results
“Implementing SEP sesam has been truly instrumental in easing our
workload and providing a quality backup solution for all of our customers.
The implementation has allowed us to use other equipment including our
hundred-slot ADIC tape library, which was not available to us when using
the previous solution. The time we spend working on backup related
issues has been reduced by a factor of 90%. The acquisition cost for SEP
sesam was less than our annual maintenance fee for the old back up
solution. Call us one satisfied customer,” stated Carpenter.
 To o l s OCFS2
© Kheng Ho Toh, 123RF.com

A simple approach to the OCFS2 cluster filesystem

Divide and Conquer

The vanilla kernel includes two cluster filesystems: OCFS2 has been around since 2.6.16 and is thus senior to
GFS2. Although OCFS2 is non-trivial under the hood, it is fairly simple to deploy. By Udo Seidel

Wherever two or more computers
need to access the same set of data,
Linux and Unix systems will have
multiple competing approaches. (For
an overview of the various technolo-
gies, see the “Shared Filesystems”
box.) In this article, I take a close
look at OCFS2, the Oracle Cluster File
System shared disk filesystem [1]. As
the name suggests, this filesystem
is mainly suitable for cluster setups
with multiple servers.
Before you can set up a cluster file-
system based on shared disks, you
need to look out for a couple of
things. First, the administrator needs
to establish the basic framework of a
cluster, including stipulating the com-
puters that belong to the cluster, how
to access it via TCP/​IP, and the clus-
ter name. In OCFS2’s case, a single
ASCII file is all it takes (Listing 1).
The second task to tackle with a clus-
ter filesystem is that of controlled and
orderly access to the data with the
use of file locking to avoid conflict sit-
uations. In OCFS2’s case, the Distrib-
uted Lock Manager (DLM) prevents
filesystem inconsistencies. Initializing
the OCFS2 cluster automatically
launches DLM, so you don’t need to
configure this separately. However,
the best file locking is worthless if the
computer writing to the filesystem
goes haywire. The only way to pre-
vent computers from writing is fenc-

36 Admin 01 w w w. a d m i n - m aga z i n e .co m

 OCFS2 To o l s

ing. OCFS2 is fairly simplistic in its

approach and only uses self-fencing.
If a node notices that it is no longer
cleanly integrated with the cluster, it
throws a kernel panic and locks itself
out. Just like the DLM, self-fencing
in OCFS2 does not require a separate
configuration. Once the cluster con-
figuration is complete and has been
distributed to all the nodes, the brunt
of the work has been done for a func-
tional OCFS2.
Things are seemingly quite simple
at this point: Start the cluster, create
OCFS2 if needed, mount the filesys-
tem, and you’re done.

Getting Started
Figure 1: Cluster configuration with the ocfs2console GUI tool.
As I mentioned earlier, OCFS2 is a
cluster filesystem based on shared man page for mkfs.ocfs provides a OCFS2 volumes, you can expect a
disks. The range of technologies full list of options, the most important short delay: During mounting, the
that can provide a shared disk spans of which are covered by Table 2. executing machine needs to register
expensive SAN over Fibre Channel, Once you have created the filesys- with the DLM. In a similar fashion,
from iSCSI to low-budget DRBD [7]. tem, you need to mount it. The mount the DLM resolves any existing locks
In this article, I will use iSCSI and command works much like that on or manages them on the remaining
NDAS (Network Direct Attached Stor- unclustered filesystems (Figure 2). systems in case of a umount. The doc-
age). The second ingredient in the When mounting and unmounting umentation points to various options
OCFS2 setup is computers with an
OCFS2-capable operating system. The Table 1: Directories in a Repository
best choices here are Oracle’s Enter- Filesystem Function GUI Menu CLI Tool
prise Linux, SUSE Linux Enterprise Mount Mount mount.ocfs2
Server, openSUSE, Red Hat Enterprise Unmount Unmount umount
Linux, and Fedora. Create Format mkfs.ocfs2
The software suite for OCFS2 com- Repair Check fsck.ocfs2
prises the ocfs2-tools and ocfs2con- Repair Repair fsck.ocfs2
sole packages and the ocfs2‑`uname Change name Change Label tunefs.ocfs2
‑r` kernel modules. Typing ocfs2con‑ Maximum number of nodes Edit Node Slot Count tunefs.ocfs2
sole launches a graphical interface in
which you can create the cluster con- Table 2: Important Options for mkfs.ocfs2
figuration and distribute it over the Option Purpose
nodes involved (Figure 1). However, b Block size
you can just as easily do this with vi C Cluster size
and scp. Table 1 lists the actions the L Label
graphical front end supports and the
N Maximum number of computers with simultaneous access
equivalent command-line tools.
J Journal options
After creating the cluster configu-
T Filesystem type (optimization for many small files or a few large ones)
ration, /etc/init.d/o2cb online
launches the subsystem (Listing 2).
The init script loads the kernel mod- Listing 1: /etc/​o cfs2/​c luster.conf
ules and sets a couple of defaults for node: ip_port = 7777

the heartbeat and fencing. ip_port = 7777 i

p_address = 192.168.0.2

Once the OCFS2 framework is run- ip_address = 192.168.0.1

 number = 1
number = 0 name = node1
ning, the administrator can create
name = node0 cluster = ocfs2
the cluster filesystem. In the sim-
cluster = ocfs2 cluster:
plest case, you can use mkfs.ocfs2
node: node_count = 2
devicefile (Listing 3) to do this. The
­

w w w. a d m i n - m aga z i n e .co m Admin 01 37

 To o l s OCFS2
you can set for the mount operation.
If OCFS2 detects an error in the data
structure, it will default to read-
only. In certain situations, a reboot
can clear this up. The errors=panic
mount option handles this. Another
interesting option is commit=seconds.
The default value is 5, which means
that OCFS2 writes the data out to disk
every five seconds. If a crash occurs,
a consistent filesystem can be guaran-
teed – thanks to journaling – and only
the work from the last five seconds
will be lost. The mount option that
specifies the way data are handled
Listing 2: Starting the OCFS2 Subsystem
# /etc/init.d/o2cb online
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading filesystem "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
#
# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Not active
#
Listing 3: OCFS2 Optimized for Mail Server
# mkfs.ocfs2 ‑T mail ‑L data /dev/sda1
mkfs.ocfs2 1.4.2
Cluster stack: classic o2cb
Filesystem Type of mail
Filesystem label=data
Block size=2048 (bits=11)
Cluster size=4096 (bits=12)
Volume size=1011675136 (246991 clusters) (493982 blocks)
16 cluster groups (tail covers 8911 clusters, rest cover
15872 clusters)
Journal size=67108864
Initial number of node slots: 2
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Formatting slot map: done
Writing lost+found: done
mkfs.ocfs2 successful
38 Admin 01
for journaling is also important here.
The latest version lets OCFS2 write all
the data out to disk before updating
the journal. date=writeback forces the
predecessor’s mode.
Inexperienced OCFS2 admins might
wonder why the OCFS2 volume is
not available after a reboot despite an
entry in /etc/fstab. The init script
that comes with the distribution,
/etc/init.d/ocfs2 makes the OCFS2
mount resistant to reboots. Once en-
abled, this script scans /etc/fstab for
OCFS2 entries and integrates these
filesystems.
Just as with ext3/​4, the administra-
tor can modify a couple of filesystem
properties after mounting the file-
system without destroying data. The
tunefs.ocfs2 tool helps with this. If
the cluster grows unexpectedly and
you want more computers to access
OCFS2 at the same time, too small a
Shared Filesystems
The shared filesystem family is a fairly colorful
bunch. By definition, they all share the ability
to grant multiple computers simultaneous ac-
cess to certain data. The differences are in the
way they implement these requirements.
On the one hand are network filesystems, in
which the most popular representative in the
Unix/​Linux camp is Network Filesystem (NFS)
[2]. NFS is available for more or less any op-
erating system and to all intents and purposes
only asks the operating system to provide a
TCP/​IP stack. The setup is also fairly simple.
The Andrew filesystem (AFS) is another
network filesystem that is available in a free
implementation, OpenAFS [3].
On the other hand are cluster filesystems. Be-
fore computers can access “distributed” data,
History
OCFS2 is a fairly young filesystem. As the “2”
in the name suggests, the current version
is an enhancement. Oracle developed the
predecessor, OCFS, for use in Real Application
Cluster for Oracle databases. The new OCFS2
is designed to fulfill the requirements placed
on a mature filesystem capable of storing ar-
bitrary data. POSIX compatibility and the typi-
cal – and necessary – performance required
for databases were further criteria.
After two years of development, the program-
mers released version 1.0 of OCFS2, and it
made its way into the vanilla kernel (2.6.16)
value for the N option in mkfs.ocfs2
can become a problem. The tunefs.
ocfs2 tool lets you change this in next
to no time. The same thing applies to
the journal size (Listing 4).
Also, you can use this tool to modify
the filesystem label and enable or
disable certain features (see also List-
ing 8). Unfortunately, the man page
doesn’t tell you which changes are
permitted on the fly and which aren’t.
Thus, you could experience a tunefs.
ocfs2: Trylock failed while opening de-
vice "/dev/sda1" message when you
try to run some commands on OCFS2.
More Detail
As I mentioned earlier, you do not
need to preconfigure the cluster
heartbeat or fencing. When the
cluster stack is initialized, default
values are set for both. However, you
they first need to enter the cluster. The clus-
ter setup requires additional infrastructure,
such as additional I/​O cards, cluster software,
and, of course, a configuration. Cluster file-
systems are also categorized by the way they
store data. Those based on shared disks allow
multiple computers to read and write to the
same medium. I/​O is handled via Fibre Channel
(“classical SAN”) or TCP/​IP (iSCSI). The most
popular representatives in the Linux camp
here are OCFS2 and the Global Filesystem
(GFS2) [4].
Parallel cluster filesystems are a more recent
invention. They distribute data over computers
in the cluster by striping single files across
multiple storage nodes. Lustre [5] and Ceph
[6] are popular examples of this technology.
just a year later. Version 1.2 became more
widespread, with a great deal of support from
various Enterprise Linux distributions.
OCFS2 has been available for the major play-
ers in the Linux world for some time. This
applies to commercial variants, such as SLES,
RHEL, or Oracle EL, and to the free Debian,
Fedora, and openSUSE systems. Depending on
the kernel version, users either get version
1.4, which was released in 2008, or version 1.2,
which is two years older. The “OCFS2 Choices”
box and Table 3 show you what you need to
watch out for.
w w w. a d m i n - m aga z i n e .co m
 OCFS2 To o l s

F Figure 2: Unspecta­
cular: the OCFS2 mount
process.

E Figure 3: Automatic
reboot after 20 sec­
onds on OCFS2 cluster
­errors.

can modify the defaults to suit your
needs. The easiest approach here is
via the /etc/init.d/o2cb configure
script, which prompts you for the
required values – for example, when
the OCFS2 cluster should regard a
node or network connection as down.
At the same time, you can specify
when the cluster stack should try to
reconnect and when it should send a
keep-alive packet.
Apart from the heartbeat timeout,
all of these values are given in mil-
liseconds. However, for the heartbeat
timeout, you need a little bit of math
to determine when the cluster should
consider that a computer is down.
The value represents the number of
two-second iterations plus one for
the heartbeat. The default value of 31
is thus equivalent to 60 seconds. On
larger networks, you might need to
increase all these values to avoid false
alarms.
In production use, you will probably
want to remedy this state without
in-depth error analysis (i.e., reboot
the cluster node). For this to happen,
you need to modify the underlying
operating system so that it automati-
cally reboots in case of a kernel oops
or panic (Figure 3). Your best bet for
this on Linux is the /proc filesystem
for temporary changes, or sysctl if
you want the change to survive a
reboot.
Just like any other filesystem, OCFS2
has a couple of internal limits you
need to take into account when de-
signing your storage. The number
of subdirectories in a directory is
restricted to 32,000. OCFS2 stores
data in clusters of between 4 and
1,024Kb. Because the number of
cluster addresses is restricted to 232,
the maximum file size is 4PB. This
limit is more or less irrelevant be-
cause another restriction – the use
An active OCFS2 cluster uses a
handful of processes to handle
its work (Listing 5). DLM-related
tasks are handled by dlm_thread,
dlm_reco_thread, and dlm_wq. The
ocfs2dc, ocfs2cmt, ocfs2_wq, and
ocfs2rec processes are responsible
for access to the filesystem. o2net
and o2hb‑XXXXXXXXXX handle cluster
communications and the heartbeat.
All of these processes are started and
stopped by init scripts for the cluster
framework and OCFS2.
OCFS2 stores its management files
in the filesystem’s system directory,
which is invisible to normal com-
mands such as ls. The debugfs.ocfs2
command lets you make the system
directory visible (Figure 4). The
objects in the system directory are
divided into two groups: global and
Listing 4: Maintenance with tunefs.ocfs2
# tunefs.ocfs2 ‑Q "NumSlots = %N\n" /dev/sda1

If OCFS2 stumbles across a criti-
cal error, it switches the filesystem
to read-only mode and generates a
kernel oops or even a kernel panic.
of JBD journaling – limits the maxi- NumSlots = 2
# tunefs.ocfs2 ‑N 4 /dev/sda1
mum OCFS2 filesystem size to 16TB,
# tunefs.ocfs2 ‑Q "NumSlots = %N\n" /dev/sda1
which can address a maximum of 232
NumSlots = 4
blocks of 4KB. #
# tunefs.ocfs2 ‑Q "Label = %V\n" /dev/sda1
Label = data
# tunefs.ocfs2 ‑L oldata /dev/sda1
# tunefs.ocfs2 ‑Q "Label = %V\n" /dev/sda1
Label = oldata
#

Listing 5: OCFS2 Processes

# ps ‑ef|egrep '[d]lm|[o]cf|[o]2'
root 3460 7 0 20:07 ? 00:00:00 [user_dlm]
root 3467 7 0 20:07 ? 00:00:00 [o2net]
root 3965 7 0 20:24 ? 00:00:00 [ocfs2_wq]
root 7921 7 0 22:40 ? 
00:00:00
[o2hb‑BD5A574EC8]
root 7935 7 0 22:40 ? 00:00:00 [ocfs2dc]
root 7936 7 0 22:40 ? 00:00:00 [dlm_thread]
root 7937 7 0 22:40 ? 
00:00:00 [dlm_reco_
thread]
root 7938 7 0 22:40 ? 00:00:00 [dlm_wq]

Figure 4: The metadata for OCSFS2 are stored in files that are invisible to the ls command. They can be listed root 7940 7 0 22:40 ? 00:00:00 [ocfs2cmt]
#
with the debugfs.ocfs2 command.

w w w. a d m i n - m aga z i n e .co m Admin 01 39

 To o l s OCFS2

local (i.e., node-specific) files. The is necessary to create further node- complexity in a cluster filesystem
first of these groups includes global_ specific system files. doesn’t help. From the viewpoint of
inode_alloc, slot_map, heartbeat, OCFS2, things can go wrong in three
and global_bitmap. They have access What Else? different layers – the filesystem struc-
to each node on the cluster; incon- ture on the disk, the cluster configu-
sistencies are prevented by a locking When you make plans to install ration, or the cluster infrastructure –
mechanism. The only programs that OCFS2, you need to know which ver- or even a combination of the three.
access global_inode_alloc are those sion you will putting on the new ma- The cluster infrastructure includes
for creating and tuning the filesystem. chine. Although the filesystem itself – the network stack for the heartbeat,
To increase the number of slots, it that is, the structure on the medium cluster communications, and possibly
– is downward compatible, mixed op- media access. Problems with Fibre
Listing 6: Debugging with mounted.ocfs2 erations with OCFS2 v1.2 and OCFS2 Channel (FC) and iSCSI also belong
# grep ‑i ocfs /proc/mounts |grep ‑v dlm v1.4 are not supported. The network to this group.
# hostname protocol is to blame for this. The de- For problems with the cluster infra-
testvm2.seidelnet.de
velopers enabled a tag in the active structure, you can troubleshoot just
# tunefs.ocfs2 ‑L olddata /dev/sda1
tunefs.ocfs2: Trylock failed while opening device "/dev/
protocol version so that future OCFS2 as you would for a normal network,
sda1" versions would be downward compat- FC, or iSCSI problems. Problems can
# mounted.ocfs2 ‑f ible through the network stack. This also occur if the cluster configuration
Device FS Nodes comes at the price of incompatibility is not identical on all nodes. Armed
/dev/sda1 ocfs2 testvm
with v1.2. Otherwise, administrators with vi, scp, and md5sum, you can
#
have a certain degree of flexibility check this and resolve the problem.
when mounting OCFS2 media. OCFS2 The alternative – assuming the cluster
Listing 7: Restoring Corrupted OCFS2 Superblocks v1.4 computers will understand the infrastructure is up and running – is
# mount /dev/sda1 /cluster/ data structure of v1.2 and mount to synchronize the cluster configu-
mount: you must specify the filesystem type them without any trouble. This even ration on all of your computers by
# fsck.ocfs2 /dev/sda1
works the other way around: If the updating the configuration with ocfs­
fsck.ocfs2: Bad magic number in superblock while opening
OCFS2 v1.4 volume does not use the 2console.
"/dev/sda1"
# fsck.ocfs2 ‑r1 /dev/sda1
newer features in this version, you It can be useful to take the problem-
[RECOVER_BACKUP_SUPERBLOCK] Recover superblock can use an OCFS2 v1.2 computer to atic OCFS2 volume offline – that is,
information from backup block#262144? <n> y access the data. to unmount it and restart the cluster
Checking OCFS2 filesystem in /dev/sda1: service on all of your computers by
label:
uuid:
backup
31 18 de 29 69 f3 4d 95 a0 99 a7
Debugging giving the /etc/init.d/o2cb restart
command. You can even switch the
23 ab 27 f5 04
A filesystem has a number of poten- filesystem to a kind of single-user
number of blocks: 367486
bytes per block: 4096
tial issues, and the added degree of mode with tunefs.ocfs2.
number of clusters: 367486
bytes per cluster: 4096
Table 3: New Features in OCFS2 v1.4
max slots: 2 Feature Description
Ordered journal mode OCFS2 writes data before metadata.
/dev/sda1 is clean. It will be checked after 20
additional mounts. Flexible allocation OCFS2 now supports sparse files – that is, gaps in files. Additionally,
# preallocation of extents is possible.
Inline data OCFS2 stores the data from small files directly in the inode and not in
extents.
Listing 8: Enabling/​Disabling OCFS2 v1.4 Features
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1
Clustered flock() The flock() system call is cluster capable.
Incompatibel: sparse inline‑data
# tunefs.ocfs2 ‑‑fs‑features=nosparse /dev/sda1 OCFS2 Choices
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1
Administrators will basically come across two The mkfs.ocfs2 supplied with version 1.4
Incompatibel: inline‑data
versions of OCFS2: version 1.2 or 1.4. As re- automatically enables all the new features,
# tunefs.ocfs2 ‑‑fs‑features=noinline‑data /dev/sda1
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1
gards the data structure on disk, the two ver- thus effectively preventing OCFS v1.2 ma-
Incompatibel: None
sions are compatible; however, this does mean chines from accessing the filesystem. To
# doing without the newer features in v1.4. The change this, use tunefs.ocfs2 to disable
# tunefs.ocfs2 ‑‑fs‑features=sparse,inline‑data /dev/ documentation lists 10 significant differences the new functions (Listing 8). An easier ap-
sda1 between versions 1.2 and 1.4. Table 3 lists the proach is to create the filesystem with the
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1 most interesting of these. No matter which ‑‑fs‑feature‑level=max‑compat
Incompatibel: sparse inline‑data version you decide on, you should always option set. tunefs.ocfs2 will help you mi-
# watch for a couple of things. grate from version 1.2 to 1.4.

40 Admin 01 w w w. a d m i n - m aga z i n e .co m

To do this, you need to
change the mount type from
cluster to local. After doing
so, only a single computer
can mount the filesystem,
and it doesn’t need the clus-
ter stack to do so.
In all of these actions, you
need to be aware that the
filesystem can be mounted
by more than one computer.
Certain actions that involve,
say, tunefs.ocfs2, will not
work if another computer ac-
cesses the filesystem at the
same time.
The example in Listing 6
shows the user attempting
to modify the label. This
process fails, although the
filesystem is offline (on this
computer). In this case,
mounted.ocfs2 will help: It
checks the OCFS2 header to
identify the computer that is
online with the filesystem.
The most important filesys-
tem structure data are con-
tained in the superblock. Just
like other Linux filesystems,
OCFS2 creates backup copies
of the superblock; however,
the approach the OCFS2
developers took is slightly
unusual.
OCFS2 creates a maximum
of six copies at non-config-
urable offsets: 1, 4, 16, 64,
and 256GB and 1TB. Need-
less to say, OCFS2 volumes
smaller than 1GB (!) don’t
have a copy of the super-
block. To be fair, mkfs.ocfs2
does tell you this when you
generate the filesystem. You
need to watch out for the
Writing backup superblock:
... line.
A neat side effect of these
static backup superblocks is
that you can reference them
by number during a filesys-
tem check. The example in
Listing 7 shows a damaged
primary superblock that is
preventing mounting and
a simple fsck.ocfs2 from
w w w. a d m i n - m aga z i n e .co m
working. The first backup
makes it possible to restore.
Basically, Yes, but …
On the whole, it is easy to
set up an OCFS2 cluster. The
software is available for a
number of Linux distribu-
tions. Because OCFS2 works
just as well with iSCSI and
Fibre Channel, the hardware
side is not too difficult either.
Setting up the cluster frame-
work is a fairly simple task
that you can handle with
simple tools like Vi.
Although OCFS2 doesn’t
include sophisticated fenc-
ing technologies, in contrast
to other cluster filesystems,
fencing is not necessary in
many areas. The lack of a
cluster-capable volume man-
ager makes it easier for the
Linux Pros read
user to become immersed in
the world of OCFS2. Because
LINUX PRO
OCFS2 is simpler and less
complex than other cluster
filesystems, it is well worth
investigating. n
Info Enjoy a rich blend of tutorials, reviews,
[1] OCFS2: [http://​­oss.​­oracle.​­com/​
­projects/​­ocfs2/] international news, and practical
[2] First NFS RFC: [http://​­tools.​­ietf.​ solutions for the technical reader.
­org/​­html/​­rfc1094]
[3] OpenAFS:
[http://​­www.​­openafs.​­org/]
[4] GFS: [http://​­sources.​­redhat.​
­com/​­cluster/​­gfs/]
Subscribe now
[5] Lustre: [http://​­wiki.​­lustre.​­org]
[6] Ceph:
to receive:
3 issues
[http://​­ceph.​­newdream.​­net/]
[7] DRBD: [http://​­www.​­drbd.​­org/]
+ 3 DVDs
The Author
for only
Udo Seidel is a teacher of math and
physics and has been an avid sup-
porter of Linux since 1996. After
completing his PhD, he worked as a
Linux/​Unix trainer, system adminis-
trator, and senior solutions engineer.
$
3.00!
He now works as the head of a www.linuxpromagazine.com/trial
Linux/​Unix team for Amadeus Data
Processing GmbH in Erding, Germany.
Admin 01 41
 To o l s Synergy
©Kheng Ho Toh, 123RF.com

Controlling multiple systems simultaneously with Synergy

Side Effect
The many approaches to managing remote computers include VNC, No-
machine, and SSH. Synergy is a clever tool that does a bit of lateral think-
ing and connects multiple PCs to create a virtual desktop. By Florian Effenberger
To operate Synergy, you need at least
two PCs, each with its own operat-
ing system, monitor, and functional
network card. The software supports
Windows 95 through Windows 7,
Mac OS X as of version 10.2, and
Linux with the current X server. Pre-
built packages for Windows and Mac
OS X are available from the Synergy
homepage [1]. An RPM file is avail-
able for Linux and can be installed on
most popular distributions with tools
such as Alien [2], if needed. Some
distributions also offer prebuilt pack-
ages; for example, Ubuntu Universe
contains a package called Synergy.
Test Case
The administrator’s workplace com-
prises a large desktop system running
Ubuntu and a small notebook run-
ning Vista on the right. To avoid con-
stantly switching between keyboards,
the administrator has decided to use
Synergy. The admin will work mainly
on the large PC, which is the Ubuntu
system. In Synergy-speak, this is re-
ferred to as the control system; the
administrator will use the keyboard
and mouse on this server. The other
devices are clients.
Configuration
Before you start using Synergy, you
need to configure it by editing the
/etc/synergy.conf or ~/.synergy.
conf text file. The elementary unit is
a screen: Each computer belonging to
a group, whether server or client, is a
screen with a precisely defined posi-
tion – just like the display arrange-
ment in a configuration with multiple
monitors. For each computer, you
need to enter into the configuration
file the name of the screen, its aliases,
and its position relative to other de-
vices – in both directions. Listing 1
contains an example with comments
for the test case. The Synergy homep-
age documents many additional op-
tions [3].
All options in the configuration file
should be in lowercase. Also, make
sure you use the correct line breaks,
because Synergy is fussy about them
and will not use the file if they are
wrong. After completing all this work,
you can launch the Synergy server on
Ubuntu as a normal user by typing
synergys. The ‑f parameter will pre-
vent Synergy from disappearing into
the background.
QuickSynergy gives you an even more
convenient approach to configuration.
On Ubuntu, you can download the
package from the Universe reposi-
tory and launch it with Applications |
Tools | QuickSynergy after the install.
Unfortunately, the program failed to
launch a working server during my
test.
The Vista client, which I want to con-
trol remotely with the Ubuntu system,
is even easier to configure. After the
installation, you can launch Synergy

42 Admin 01 w w w. a d m i n - m aga z i n e .co m

 Synergy To o l s

ent while the PCs. And the cross-operating system

Figure 1: Synergy as a client on Windows Vista.
directly via the Start menu, and it will
come up with a neat graphical inter-
face (Figure 1). To open a connection
to the server, select the Use another
computer’s shared keyboard and
mouse (client) option and enter the
name of the computer you require.
Power users might also want to con-
figure extras like the Logging Level,
AutoStart, and network details.
Connecting the Screens
After configuring all the clients, you
can simply run synergyc server IP
(or click Start on Windows) to com-
bine the screens. The whole thing is
unspectacular at first, and you can
focus is on its
screen (i.e., the
mouse cursor is
displayed there).
But, that’s not
all – Synergy
also coordinates
the clipboard
between the two
systems. Accord-
ing to the devel-
opers, Synergy
automatically
identifies the cor-
rect character set
and converts line
breaks between operating systems,
which is perfect for centralized copy-
ing of long text blocks and configura-
tion files. Pressing the Scroll key dis-
ables Synergy temporarily if needed.
In the configuration file, you can set a
number of additional options. Among
other things, Synergy can map keys
between the server and the client,
configure screen areas when you do
not want to be able to toggle between
screens, and perform certain actions
at a key press. Synergy messes up
some functions, however: In our
lab, the tool failed to synchronize
screen savers and failed to lock all
the screens centrally. According to
the homepage, the Mac OS X variant
clipboard, which removes the need to
copy text files, is really convenient.
One item on my Synergy wish list,
however, is easier configuration on
Linux. n
Info
[1] Synergy homepage:
[http://​­synergy-foss.org]
[2] Alien:
[http://​­kitenet.​­net/​­~joey/​­code/​­alien/]
[3] Configuration options: [http://synergy-
foss.org/pm/projects/synergy/wiki/Setup]
[4] Security notes: [http://synergy-foss.org/
pm/projects/synergy/wiki/UserFAQ#Q-
How-secure-is-your-application]
[5] Tunneling with SSH: [http://www.revsys.
com/writings/quicktips/ssh-tunnel.html]
The Author
Florian Effenberger has been a free software
evangelist for many years. He is the Lead of the
Marketing Project for OpenOffice.org interna-
tional and a member of the OpenOffice.org Ger-
many board. Other work involves designing and
implementing enterprise and school networks,
including software distribution solutions based
on free software. He also writes for numerous
German and English language publications, in
which he mainly focuses on legal issues.
Listing 1: Configuration
# Define screens

continue to work on each system in of the program, in particular, is not section: screens

the normal way. as mature as the Linux and Windows ubuntu:

vista:
However, if you move the mouse on versions.
end
the server beyond the right edge of
the screen, the mouse pointer moves Conclusions # Alternative names
onto the Vista desktop just as on a section: aliases
single computer with multiple dis- Synergy offers an interesting ap- # ubuntu ‑> desktop
plays, but here, it moves between two proach to controlling multiple com- ubuntu:
computers with different platforms. puters centrally without investing in desktop

Keyboard input also reaches the cli- additional hardware. In contrast to # vista ‑> notebook

legacy approaches, each system keeps vista:

Enabling Universe its own display. The program is defi- notebook

end
Ubuntu organizes its software packages nitely useful for owners of multiple
into a number of repositories. The Ubuntu
# Screen arrangement
Universe repository contains packages that Security Note
section: left
have less comprehensive support and main- The authors of Synergy point out on their # vista: right of ubuntu
tenance than others. To use Universe, you homepage that Synergy does not provide ubuntu:
first need to enable the corresponding line right = vista
anything in the line of authentication or
in the /etc/apt/sources.list file by # ubuntu: left of vista
removing the hash sign. After an apt‑get encryption [4]. To be on the safe side, you
vista:
update, you can install the new packages, might want to set up an SSH tunnel to en- left = ubuntu
including Synergy. crypt all your data [5]. end

w w w. a d m i n - m aga z i n e .co m Admin 01 43

 TO O L S SystemTap

Tracing applications with OProfile and SystemTap

Data on Tap
© boing, Photocase.com

Does your application data take ages to creep off your disk or your network card, even if no noticeable activity
is taking place? Tools such as OProfile and SystemTap help you find out why. By Thorsten Scherf

Experienced administrators tend to
use tools such as ps, vmstat, or the
like when they need statistics for in-
dividual subsystems such as the net-
work, memory, or block I/O. These
tools can help identify hardware or
software bottlenecks, and they are
indisputably useful for a general
appraisal, but if you want to delve
deeper, you need something with
more punch.
Again, the standard toolbox offers a
couple of utilities. For example, the
popular strace traces applications.
In the simplest cases, the tool lists
all the system calls (syscalls) with
their arguments and return codes for
a specific application. Setting options
allows for highly selective Strace
output. For example, if you need to
investigate whether an application is
parsing the configuration file that you
painstakingly put together, you can
call Strace:
strace -e trace=open -o mutt.trace mutt
This command line sends all open sys-
calls for the Mutt application to the
/tmp/mutt.trace output file. Then,
you can easily grep the configuration
file from the results.
Profiling applications, including the
popular OProfile tool [1], take this a
step further by giving you details of
the performance of individual appli-
cations, the kernel, or the complete
system (see Figure 1). For this to hap-
pen, OProfile accesses the CPU per-
formance counters on state-of-the-art

Listing 1: opcontrol Events

opcontrol --list-events 0x30: prefetch: all inclusive
oprofile: available events for CPU type "Core 2" 0x10: prefetch: Hardware prefetch only
0x00: prefetch: exclude hardware prefetch
See Intel Architecture Developer's Manual Volume 3B, Appendix A and 0x08: (M)ESI: Modified
Intel Architecture Optimization Reference Manual (730795-001) 0x04: M(E)SI: Exclusive
0x02: ME(S)I: Shared
INST_RETIRED_ANY_P: (counter: all)) 0x01: MES(I): Invalid
number of instructions retired (min count: 6000) LLC_MISSES: (counter: all))
L2_RQSTS: (counter: all)) L2 cache demand requests from this core that missed the
number of L2 cache requests (min count: 500) L2 (min count:6000)
Unit masks (default 0x7f) Unit masks (default 0x41)
---------- ----------
0xc0: core: all cores 0x41: No unit mask
0x40: core: this core [...]

44 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

 SystemTap To o l s

hardware. The counters have informa-

tion on how often a specific event has
occurred. In this context, an event
can be RAM access or the number of
interrupts. This information is very
useful for identifying bottlenecks or
debugging the system.
To install OProfile, you need the
kernel-debuginfo [2] package, which
provides the symbol to machine
code mappings. Note that the kernel-
debuginfo version must match your
kernel version. To install the pack-
age easily, use your distribution’s
standard repository. On Fedora, you
would do this with Yum:
yum install oprofile kernel‑debuginfo
A call to RPM should confirm that the
kernel and kernel-debuginfo packages
have the same version number:
rpm ‑q k
ernel‑PAE kernel‑PAE‑debuginfo U
oprofile
kernel‑PAE‑2.6.32.10‑90.fc12.i686
kernel‑PAE‑debuginfo‑2.6.32.10‑90.fc12.i686
oprofile‑0.9.5‑4.fc12.i686
To profile the kernel, you need to tell
OProfile where the kernel image is lo-
cated with the --vmlinux option:
opcontrol ‑‑setup U
‑‑vmlinux=/usr/lib/debug/lib/ U
modules/`uname ‑r`/vmlinux
In normal use, you can omit the im-
age details. The following command
gives you an overview of the events
that OProfile can enumerate (see
Listing 1):
opcontrol ‑‑list‑events
Figure 1: OProfile architecture.
CPUs; if you set 0x40, you get the
value only for the CPU actually run-
ning the OProfile process. The com-
mand in Listing 2 monitors a specific
event.
The --event option can occur mul-
tiple times to monitor more than
one event. To make sure that the
results are not distorted by histori-
cal data, the --reset option deletes
them before collecting fresh data with
--start. After a while, the --stop
option stops monitoring the system.
The data collected in this way are
now available in the /​var/​lib/​opro-
file/​ samples directory. For a general
overview, you can access the data
with opreport – either the data for
the complete system or the data for a
specific application (Listing 3).
Depending on the event selected, this
information will give you a pretty
clear picture of what is happening
on your system. For more details of
OProfile, check out the highly infor-
mative website for the tool [1].
ample, for kernel functions (kernel.
function("function")) inside of
kernel modules (module("module").
function("function")) or system calls
(syscall.system_call). The injected
program monitors the event and
collects information in the process.
Thus, far more precise results can
be achieved here than with OProfile,
which only queries an event periodi-
cally.
SystemTap has also supported query-
ing static tracepoints in the kernel
(kernel.trace("tracepoint")) and,
more recently, in userspace ap-
plications. Developers build static
tracepoints into the program code at
Listing 2: opcontrol Events
opcontrol ‑
‑vmlinux=/usr/lib/debug/lib/modules/`uname
‑r`/vmlinux ‑‑event L2_RQSTS:500
# opcontrol ‑‑reset
# opcontrol ‑‑start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log

The events will differ depending on Daemon started.

the CPU you use. The /​usr/​share/​
oprofile directory has lists for the
various architectures. An event com-
prises a symbolic name (L2_RQSTS), a
Profiler running.
SystemTap
The SystemTap [3] tool aims to com- Listing 3: opreport for Mutt
bine the functionality of classical trac- opreport ‑l /usr/bin/mutt

counter (500), and an optional mask ing and profile tools such as Strace CPU: Core 2, speed 2401 MHz (estimated)
Cou
nted L2_RQSTS events (number of L2 cache requests)
(0xc0). The counter defines the accu- and OProfile while providing a simple
with a unit mask of 0x7f
racy of a profile. The lower the value, but powerful interface for the user.
(multiple flags) count 500
the more often the event will be que- SystemTap was originally developed samples % image name symbol name
ried. Special properties of an event for monitoring the Linux kernel, al- 414 10.8377 mutt imap_exec_msgset
are available by query with the mask. though more recent versions also let 162 4.2408 mutt parse_set
For example, the L2_RQSTS event tells you monitor userspace applications. 161 4.2147 mutt mutt_buffer_add
you how many requests have been SystemTap builds on the kprobes 145 3.7958 mutt mutt_extract_token

made to the CPU’s L2 cache. When kernel subsystem. It lets the user 126 3.2984 mutt ascii_strncasecmp

called with a mask of 0xc0, OProfile insert arbitrary program code before 124 3.2461 mutt imap_read_headers
[...]
returns the value for all the available any event in kernel space – for ex-

w w w. a d m i n - m aga z i n e .co m Admin 01 45

 To o l s SystemTap

important locations. Because develop- the module on another system that Tapsets are easy to integrate with
ers know their program better than doesn’t have a compiler is not a prob- your own scripts. The templates are
anybody else, this kind of information lem, though. SystemTap lets you build typically located below /​usr/​share/​
is a big help. modules for kernel versions besides systemtap/​ tapsets. Besides these syn-
SystemTap programs are written in the kernel on your own system. Then chronous events, other asynchronous
a language that is similar to Awk. A you can copy the module to the target events are not bound to a specific
parser checks the script for syntax er- system and run it with staprun – event in the kernel or program code,
rors before converting it to the faster more on this subject later. and typically they are used when you
C language, which is then loaded as Because all the major Linux distribu- need to create a header or footer for
a kernel module (Figure 2). Using tions support SystemTap, you can eas- your script. They are also suitable
ily install from the standard software for running specific events multiple
Listing 4: “Hello world” in SystemTap repository. The important thing is to times.
01 
#!/usr/bin/stap install the kernel-debuginfo package Listing 5 shows a simple example
02 
probe begin {printf("Hello, world!\n");} along with kernel-devel: with two probes, each with an asyn-
03 
probe timer.sec(5) {exit();}
chronous and a synchronous event.
04 
probe end {printf("Good‑bye, world!\n");} yum install kernel‑debuginfo U
kernel‑devel systemtap U
The first outputs a header at one-
05 
systemtap‑runtime second intervals, the second calls the
06 
# stap helloword.stp
prebuilt tcp.receive tapset, which is
07 
Hello, world!
The latest versions are available from defined in Listing 6. This example
08 
<5 seconds later>
the project’s Git repository: shows the extent to which the use
09 
Good‑bye, world!
of tapsets reduces the complexity of
git clone U
http://sources.redhat.com/git/U
your own scripting. When you launch
Listing 5: tcpdump via SystemTap systemtap.git systemtap the script from Listing 1, typing stap
01 
#!/usr/bin/stap tcpdump.stp lets you see the network
02 
Assuming the installation is success- packets arriving at one-second in-
03 
// A TCP dump like example
ful, you can use the following one- tervals with various other pieces of
04 
liner to check that SystemTap is work- information. If you omit timer.s(1)
05 
probe begin, timer.s(1) {
ing properly: in the first event, the header is only
06  printf("‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑
‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑\n")
output before outputting the first net-
stap ‑v ‑e 'probe vfs.read U
07  printf(
" Source IP Dest IP SPort {printf("Reading data from U
work packet.
DPort U A P R S F\n") disk.\n"); exit()}' The handler, also known as a body,
08  printf("‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑ supports instructions that will be
‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑\n") If this accesses the kernel’s VFS sub- familiar from various programming
09 
} system, stap will send a message to languages. For example, you can
10  standard output and terminate. The initialize variables and arrays, call
11 p
robe tcp.receive {
ubiquitous “Hello World” program for functions, and query positioning pa-
12  printf(" %15s %15s %5d %5d %d %d %d %d %d
SystemTap is shown in Listing 4. rameters $ (integer) or @ (string).
13  %d\n", saddr, daddr, sport, dport, urg,
The “Hello World” example is a great Of course, you wouldn’t want to do
14 ack, psh, rst, syn, fin)
demonstration of the generic struc- without loops (while, until, for, if/​
15 
}
ture of a SystemTap script. A script else), which give you useful flow
always comprises two parts: an event control options for the script.
Listing 6: Tapset tcp.stp and a handler – typically
01 
probe tcp.receive = kernel.function("tcp_v4_rcv") { preceded by a probe in-
02  iphdr = __get_skb_iphdr($skb)
struction. In this example,
03  saddr = ip_ntop(__ip_skb_saddr(iphdr))
the event is the read.vfs
04  daddr = ip_ntop(__ip_skb_daddr(iphdr))
function and the handler
05  protocol = __ip_skb_proto(iphdr)
is the printf command
06 
07  tcphdr = __get_skb_tcphdr($skb)
that outputs text to stdout.
08  dport = __tcp_skb_dport(tcphdr)
The handler is always ex-
09  sport = __tcp_skb_sport(tcphdr) ecuted when the specified
10  urg = __tcp_skb_urg(tcphdr) event occurs. Events can
11  ack = __tcp_skb_ack(tcphdr) be kernel functions, sys-
12  psh = __tcp_skb_psh(tcphdr) calls, or, as in this exam-
13  rst = __tcp_skb_rst(tcphdr) ple, prebuilt tapsets – that
14  syn = __tcp_skb_syn(tcphdr)
is, prebuilt code blocks for
15  fin = __tcp_skb_fin(tcphdr)
specific kernel functions Figure 2: After the syntax of the SystemTap script is checked, the
16 
}
and system calls. script is converted to C and loaded as a kernel module.

46 Admin 01 w w w. a d m i n - m aga z i n e .co m

 SystemTap To o l s

Instead of searching through a mass
of data for the required information,
which is the case with Strace, System-
Tap lets you output the information
after exceeding a specific threshold
value, or when a specific event oc-
curs. Thanks to the functional scope
of the language, the choice of lan-
guage constructs is more than ad-
equate.
Listing 7 shows another example that
uses the vfs.read tapset. The global
variable totals is an associative array
in this case. It contains the process
names and process IDs for all the
applications that access the VFS sub-
system to read data from disk. The
counter is incremented each time it’s
accessed.
If you are interested in a specific user-
space program, you’ll need to install
the matching debuginfo package for
the application. To make things easy,
I will look at the ls tool as an ex-
build system. The target systems only
require the systemtap-runtime RPM
and the staprun program it contains.
The following command creates a
prebuilt binary kernel module for the
target system:
stap ‑r kernel‑PAE‑2.6.31.12‑174.2.22 U
capt‑io.stp ‑m read‑io
The build system also needs the
kernel-debuginfo package to match
the target system version, and you
must ensure that the build and tar-
get systems have the same hardware
architecture. After creating a kernel
module, copy it to the target system
and launch it with staprun:
staprun capt‑io.ko
If you want non-root users to load
this kernel module, they need to be
members of the stapusr group; mem-
bers of the stapdev group can addi-
tionally compile their own scripts.
comprehensive tapset library, this can
be done without serious programming
skills. Advanced users will enjoy the
flexible, Awk-like scripting language
that gives them the freedom to create
highly complex tracing and profiling
scripts. The SystemTap FAQ [4] and
the language reference [5] are useful
ports of call for more help. n
The Author
Thorsten Scherf is a Senior Consultant for Red
Hat EMEA. You can meet him as a speaker at
conferences. He is also a keen marathon runner
whenever time permits.
Listing 7: Finding I/​O-Intensive Apps
01 
#!/usr/bin/stap
02 
03 
global totals;
04 
probe vfs.read
05 
06 
{
07  totals[execname(), pid()]++

ample. To perform a trace here, you’ll 08 

}
09 
need the coreutils-debuginfo package. Conclusions 10 
probe end
Calling stap as in Listing 8 gives you
11 
{
an overview of the functions in a spe- The SystemTap tracing and profil- 12  printf("** Summary ** \n")
cific process. ing tool lets regular users perform 13  foreach ([name,pid] in totals‑)
If the parameters of a specific func- detailed analyses of kernel and us- 14  printf(
"%s (%d): %d \n", name, pid,

tion are also of interest, you can erspace programs without rebooting totals[name,pid])
15 
}
change the call to stap as shown in the whole system. Thanks to the
Listing 9.
Listing 8: SystemTap Tracing a Userspace App
Cross-Compiling stap ‑e 'probe process("ls").function("*").call {log (pp())}' ‑c 'ls ‑l'
total 20

If you want to run a SystemTap script
on multiple systems, you will prob-
ably prefer not to have to install the
compiler and the kernel debug in-
formation on all of these machines.
In fact, you only need to do so on a
Info
‑rw‑rw‑r‑‑. 1 tscherf tscherf 17347 2010‑04‑12 08:43 systemtap.txt
process("/bin/ls").function("main@/usr/src/debug/coreutils‑7.6/src/ls.c:1225").call
process("/bin/ls").function("set_program_name@/usr/src/debug/coreutils‑7.6/lib/progname.c:35").call
process("/bin/ls").function("human_options@/usr/src/debug/coreutils‑7.6/lib/human.c:462").call
process("/bin/ls").function("clone_quoting_options@/usr/src/debug/coreutils‑7.6/lib/quotearg.c:99").call
process("/bin/ls").function("xmemdup@/usr/src/debug/coreutils‑7.6/lib/xmalloc.c:107").call
process("/bin/ls").function("xmalloc@/usr/src/debug/coreutils‑7.6/lib/xmalloc.c:43").call
process("/bin/ls").function("get_quoting_style@/usr/src/debug/coreutils‑7.6/lib/quotearg.c:110").call
process("/bin/ls").function("clone_quoting_options@/usr/src/debug/coreutils‑7.6/lib/quotearg.c:99").call
process("/bin/ls").function("free_pending_ent@/usr/src/debug/coreutils‑7.6/src/ls.c:1132").call

[1] OProfile project homepage: [...]

[http://​­oprofile.​­sourceforge.​­net/​­news]
[2] Kernel debuginfo information:
[http://​­fedoraproject.​­org/​­wiki/​
­StackTraces#​­What_are_debuginfo_rpms.​
­2C_and_how_do_I_get_them.​­3F]
[3] SystemTap project homepage:
process("/bin/ls").function("close_stdout@/usr/src/debug/coreutils‑7.6/lib/closeout.c:107").call
process("/bin/ls").function("close_stream@/usr/src/debug/coreutils‑7.6/lib/close‑stream.c:57").call
process("/bin/ls").function("close_stream@/usr/src/debug/coreutils‑7.6/lib/close‑stream.c:57").call
Listing 9: Parameter Tracing
stap 
‑e 'probe process("ls").function("clone_quoting_options").call {log (probefunc() . " " . $$parms) }'

[http://​­sourceware.​­org/​­systemtap/] ‑c '/bin/ls ‑l'

[4] SystemTap FAQ: [http://​­sourceware.​­org/​ total 20

‑rw‑rw‑r‑‑. 1 tscherf tscherf 18216 2010‑04‑12 09:02 systemtap.txt
­systemtap/​­wiki/​­SystemTapFAQ]
clone_quoting_options o=0x0
[5] SystemTap language reference: [http://​
clone_quoting_options o=0x0
­sourceware.​­org/​­systemtap/​­langref/]

w w w. a d m i n - m aga z i n e .co m Admin 01 47

© Cathy Yeulet, 123RF.com V i rt ua l i z at i o n Package Your Scripts
Bundle your custom apps in a Debian package
It’s a Wrap
Get standard scripts and custom applications into the cloud with the
­Debian packaging system. By Dan Frost
You’ve got a cloud. It’s great. You
can scale, and you’ve got redundancy.
But you have about 20 scripts for a
bunch of tasks (e.g., one for when
an instance is booted up and another
for when its IP changes) and these
scripts aren’t getting any shorter,
they’re getting better and longer. If
you want to manage them in your
favorite versioning software (which I
hope is Git, but might be something
else), how do you get that onto the
new instances simply?
Enter the not-new-at-all technology of
Debian packages. They are straight-
forward to use across any Debian-
based Linux and simple to create, and
they provide an ideal way of contain-
48 Admin 01
ing and releasing your scripts.
In this article, I’ll show how to create
Debian packages and how to install
them (which you probably already
know). And, I’ll explain how the
process will make you feel more com-
fortable about pushing changes live
across your cloud. I talked to cloudy
people about how to get code onto
new instances, and I tried lots of dif-
ferent things, but the Debian package
is such a solid, reliable format that I
just had to share it.
Debian Packages
Debian packages are simply archives
that are very easy to install, usually in
one line. If you’ve ever worked with
packages in Ubuntu or any other De-
bian-related Linux, you’ve probably
needed to download a package from
an online source and install it:
apt‑get install blarblar.deb
On the inside, a Debian package is
an archive of binary files, scripts, and
any other resources an application
needs, plus a handful of control files
that the various command-line tools
use to install the package.
Because this standard package format
is so easy to install on any Debian-
based Linux, it’s a great way to get
standard scripts and custom applica-
tions into the cloud. Often you need a
few lines to configure a new instance
or to connect the instance to the rest
of your cloud, and storing all those
scripts in a package keeps things very
neat.
The parts of a Debian package I’m
most interested in are the control
files, which live in a directory called
DEBIAN. Control files tell Debian all
about what the package contains,
w w w. a d m i n - m aga z i n e .co m
 Package Your Scripts V i rt ua l i z at i o n

what it’s called, the version, and
so on. The second part is the code,
which I’ll look at in detail later.
Creating the Package
To show this process at work, I’ll
create a simple package. Suppose I
need a simple page that can sit on
your server until an application is
installed. Good practice dictates that
your instances should fail nicely, so
if you start 10 instances when your
app gets 6 million tweets, you at least
want them to deliver a nice page be-
fore they’re ready to do business.
To begin,
n Create a directory called myserver
ing and development servers all sit
inside one Debian package.
A Cloudy Package
A tiny HTML file isn’t all that useful
in the cloud, so I’ll look at something
a bit more useful. Server configura-
tion can be set from the Debian pack-
age simply by placing your preferred
configuration file in the package:
./etc/apache2/conf.d/ our‑config.conf
As long as Apache is configured to
include this file (which, in Ubuntu, it
often is), it will take effect right away.
Although you might want to control
this with tools such as Puppet after
interaction. Everything has to run
automatically when new instances
start up, and you really don’t want
your script waiting for a human that
doesn’t exist.
Next, package your project into a .deb
file and place it somewhere public
from which you can download. This
might be where you host, but it is
much better to put it somewhere re-
silient, like S3 [2].
Next, log in to Scalr and add the fol-
lowing lines to a new script that will
run on the event OnHostUp (Figure 1).
# install the package
wget ‑O myserver.deb http://mybucket. U
s3.amazonaws.com/myserver.deb
deb ‑i myserver.deb

and the directories and files inside
it:
./DEBIAN/
./var/www/index.html
n Put the code shown in Listing 1 in
./DEBIAN/control.
n Put the file shown in Listing 2 in
./var/www/index.html.
If the path to the index.html file looks
familiar, it’s because the file structure
deployment, starting the instance
with a good configuration will help
keep the environment sane from the
outset.
Cloud hosting becomes difficult when
you use strange configurations – cre-
ating exceptions for some apps or
generally working against the grain
(e.g., using Tomcat’s configuration
style and Apache’s config directories).
Avoid customizing the environment
# run the script
/usr/local/myserver/bin/on‑host‑up.sh
Save the Scalr script under the name
of the event that you want to trigger
it and go to the farm configuration.
Now you can add your neatly orga-
nized scripts without having to edit
Listing 1: Control Files
Package: myserver
Version: 0.0.1

inside your package mirrors exactly too much because it will mean extra Section: server

the structure in the target instance. If maintenance in the future and could Priority: optional

you want a file in /var/some/where/
here, just create that path inside your
package project.
Once you’ve created this amazing
page, package it up with:
$ dpkg‑deb ‑‑build myserver
dpkg‑deb: building package U
`myserver' in `myserver.deb'
When you look in the directory, you’ll
see a file called myserver.deb. Now
that your project is all packaged up,
limit how you can scale. Architecture: all
Essential: no
Another common script for cloud
Installed‑Size: 1024
servers will run tasks at certain points
Maintainer: Dan Frost [dan@3ev.com]
in the instance’s life cycle. For in-
Description: My scripts for running stuff in the cloud
stance, a service such as Scalr will
run scripts on various events, such
as OnHostUp, OnHostInit, and OnIPAd‑
dressChanged. You can create some Listing 2: Message Page
scripts for these events in your De- 01  <html>
bian package: 02  <head>
03  <title>We're getting there...</title>
./usr/local/myserver/bin/on‑host‑up.sh
./usr/local/myserver/bin/on‑ip‑ U
04  </head>

you can install it. But, before you do, address‑changed.sh

05  <body>
06  <h1>Give us a moment.</h1>
take a look at what’s inside:
07  <p>We're just getting some more machines
The first script should download an
$ dpkg‑deb ‑‑contents myserver.deb plugged in ...</p>
HTML file or a PHP file from either S3
08  </body>
then install: or your existing repository and place
09  </html>
it in the document root:
$ sudo dpkg‑deb ‑i myproject.deb

cd /var/www/
After running this command, you’ll Listing 3: HTML File
wget ‑O tmp.tgz http://mybucket.s3. U
find the HTML file sitting there for amazonaws.com/website.tgz 01 
$ cat /var/www/index.html

when Apache starts (Listing 3): tar xzf tmp.tgz 02 

<html>

The next step is to create a skeleton service apache2 restart 03  <head>

04  <title>We're getting there ...</title>
set of scripts for a cloud provisioning These scripts will be like any other
05  </head>
service, such as Scalr or RightScale script you would write, with one im-
06  ...
[1]. Indeed, the scripts I use for host- portant difference: no more human

w w w. a d m i n - m aga z i n e .co m Admin 01 49

V i rt ua l i z at i o n Package Your Scripts
Figure 1: Creating a new script.
scripts via a web interface (Figure 2).
If you have a team working on your
cloud hosting, you can even start us-
ing standard code management, such
as Git or SVN, to version your cloud
environment’s bootup and configura-
tion scripts.
A second event script, which is called
each time the IP address changes,
would typically update Dynamic DNS
with a one-liner (you’ll need to set up
your Dynamic DNS account first):
curl 'http://www.dnsmadeeasy.com/ U
servlet/updateip?username= U
myuser&password=mypassword&id= U
99999999&ip=123.231.123.231'
Once you’ve placed this code in the
script on‑ip‑address‑changed.sh, sim-
ply package it up into your .deb file,
upload it to S3 again, and start a new
instance. With this approach, testing
small changes takes a little longer, but
because the scripts are all in a .deb,
Figure 2: Adding a script.
50 Admin 01
you can test them more easily outside
the cloud.
The Package in Production
Everything thus far might feel a bit
heavy-handed. I put a lot of effort
into getting a short script up onto a
cloud instance. But suppose you have
a running server farm, and you need
to update some scripts across the
farm.
Several cloud services let you edit
scripts via a web interface, which is
fine up to a point, but beyond a few
lines, you will start pining for Emacs
or your favorite editor. A custom .deb
package makes it easy to create and
test the script on local machines or a
development cloud before uploading
the final version to the production
environment.
Installing the script on instances is
simply a matter of deb ‑i myserver.
deb, so you can install, run arbitrary
tests, and repeatedly verify that your
cloud scripts are stable before they
hit the live environment. And you
should!
When everything is stable, upload to
S3, which you might want to script
as well:
s3cmd put myserver.deb U
s3://mybucket/myserver.deb
Then, all you need is a corresponding
script to run on your instances. Cre-
ate a new script in Scalr or RightScale
that downloads and installs the latest
version:
wget ‑O myserver.deb U
http://mybucket.s3.amazonaws.com/ U
myserver.deb
deb ‑i myserver.deb
The ability to test server configuration
in the cloud, for the cloud, is really
important.
If you’ve been running nice chunky
servers for years, you wouldn’t make
changes to them unless you were 100
percent sure, but with cloud comput-
ing, you can prototype your configu-
rations and settle your nerves before
putting things live.
When your cloud is running, you will
want every opportunity to test the
scripts, so being able to install, run,
and test them on any instances is
valuable.
After Installation: Uninstall
So far, Debian packages might just
look like glorified tarballs, so why
not just tar up your scripts? Well …
they’re better than that. Two hooks
are provided: post-install and pre-
uninstall. Once your Debian pack-
age’s files have been copied to the
filesystem, the post-install script,
./DEBIAN/postinst, is run, and when
you uninstall, Debian removes your
files before running ./DEBIAN/prerm.
With these scripts, you can install
software, start services, and call a
monitoring system to tell it exactly
what’s going on with the new in-
stance.
For example, open ./DEBIAN/postinst
and add something like:
w w w. a d m i n - m aga z i n e .co m

curl U
http://my‑monitor.example.com/U
?event=installed‑apache&server=U
$SERVER_NAME
How you keep your moni-
toring systems informed
depends on what you’re
running, but you can add
arbitrary commands here to
keep yourself happy. A more
typical post-install task is
sym-linking your scripts into
the standard path:
ln ‑s /usr/local/myserver/bin/ U
on‑host‑up.sh/usr/bin/
on‑host‑up.sh
Anything that gets your
scripts working, such as
starting any services that are
provided or used by your
scripts, should be done in
the post-install script:
service apache2 start
service my‑monitor start
However, this is not where
you install your web app’s
code, nor is it where you
grab the latest data. Stick
to getting the helper scripts
running in the Debian hooks
and installing your site from
the scripts inside your pack-
age.
Remember that the key to
cloud computing is scaling
without friction. Your scripts
must install themselves with-
out the need for checking
the OS afterward, so use the
hooks to leave everything
ready to go live.
Why It All Makes
Sense
To finish, I’ll look at a real-
world example. Suppose you
want to change your proxy
from Apache to HAProxy,
and you want your web serv-
ers to host some extra code
because this makes your
app more scalable. Instead
of changing to HAProxy on
the instance, you create the
w w w. a d m i n - m aga z i n e .co m
Linux Magazine
ACADEMY
script that installs and con-
figures HAProxy, but you do
Online Training with
this on your local machine.
When you’re happy, you
Linux Magazine Academy
commit this into your De-
bian package and install it Preparing for the LPIC
on some cloud instances for
testing.
exam - the easy way:
When your HAProxy scripts
Linux+
all work fine, simply push
to Com pT IA
Equivalent
your Debian package, along
PI
powered by L
with the new script, up to
S3. Next, just terminate your
HAProxy instance and wait
for a new one to replace it
that will run the new scripts,
installing and running
HAProxy instead of whatever
you had before.
GET YOUR LINUX KNOWHOW
To get extra code onto an CERTIFIED WITH LPIC:
instance, just pull it by us-
ing SVN, Git, or Wget; put it Professional training
into place; and the work is
done. So, if you have a huge
for the exams
repository of PDFs that never LPI 101 and 102
change or a massive archive
database, your scripts can
copy this down to instances
so that each runs indepen-
dently.
Anything you can do on
the command line can be
scripted, and packing up
your common tasks into a
❚ hardware settings
Debian package means your
best scripts and best config
❚ package management
will be used on all of your
instances.
❚ partitioning and file systems
Finally, remember that be-
ing scalable means being
❚ shell environment
friction-free. The people I
work with use Debian pack-
❚ automate system administration
ages, because if the package
installs, we’ve won half the
❚ network configuration
battle: Our scripts are on the
❚ security administration tasks
instance. It’s a standard and
convenient way of deploying,
and it works every time. n ❚ troubleshooting
20%
Info
rrent
[1] “Scaling the Cloud with Scalr”
by Dan Frost, Linux Magazine,
o r c u
August 2010, pg. 20 off f srcibers
[2] Amazon Simple Storage sub
Service (S3):
[http://​­aws.​­amazon.​­com/​­s3/]
For more information and to order:
academy.linux-magazine.com/LPIC
 V i rt uA l i z At i o n openVz

Operating system virtualization with OpenVZ

Container Service
© sculpies, Fotolia.com

the virtualization technology market is currently concentrating on hypervisor-based systems, but hosting pro-
viders often use an alternative technology. container-based solutions such as openVz/Virtuozzo are the most
efficient way to go if the guest and host systems are both linux. By thomas drilling

Hypervisor-based virtualization
solutions are all the rage. Many com-
panies use Xen, KVM, or VMware
to gradually abstract their hardware
landscape from its physical underpin-
nings. The situation is different if
you look at leased servers, however.
People who decide to lease a virtual
server are not typically given a fully
virtualized system based on Xen or
ESXi, and definitely not a root server.
Instead, they might be given a re-
source container, which is several
magnitudes more efficient for Linux
guest systems and also easier to set
up and manage. A resource container
can be implemented with the use of
Linux VServer [1], OpenVZ [2], or
Virtuozzo [3].
Benefits
Hypervisor-based virtualization solu-
tions emulate a complete hardware
layer for the guest system. Ideally,
any operating system including appli-
cations can be installed on the guest,
which will seem to have total control
of the CPU, chipset, and peripherals.
If you have state-of-the-art hardware
(a CPU with a virtualization extension
– VT), the performance is good.
However, hypervisor-based systems
do have some disadvantages. Because
each guest installs its own operating
system, it will perform many tasks in
its own context just like the host sys-
tem does, meaning that some services
might run multiple times. This can
affect performance because of over-
lapping – one example of this being
cache strategies for the hard disk sub-
system. Caching the emulated disks
on the guest system is a waste of time
because the host system already does
this, and emulated hard disks are ac-
tually just files on the filesystem.
Parallel Universes
Resource containers use a different
principle on the basis that – from the
application’s point of view – every
operating system comprises a filesys-
tem with installed software, space for
data, and a number of functions for
accessing devices. For the application,
all of this appears to be a separate
universe. A container has to be de-
signed so that the application thinks
it has access to a complete operating
system with a run-time environment.
From the host’s point of view, con-
tainers are simply directories. Because
all the guests share the same kernel,
they can only be of the same type as
the host operating system or its ker-
nel. This means a Linux-based con-
tainer solution like OpenVZ can only
host Linux guests. From a technical
point of view, resource containers ex-
tend the host system’s kernel. Adding
an abstraction layer then isolates the
containers from one another and pro-
vides resources, such as CPU cycles,
memory, and disk capacity (Figure 1).
Installing a container means creating
a sub-filesystem in a directory on the
host system, such as /var/lib/vz/
gast1; this is the root directory for the
guest. Below /var/lib/vz/gast1 is a
regular Linux filesystem hierarchy but
without a kernel, just as in a normal
chroot environment.

52 Admin 01 w w w. A d m i n - m AgA z i n e .co m

 openVz V i rt uA l i z At i o n

live migration, checkpointing and restoring (see the box “Live Migration, Check-
pointing and Restoring”). Incidentally,
OpenVZ containers can be shifted from one physical host to another during operations (Live
the host is referred to as the hardware
migration). Ideally, the user will not even notice this process. However, the host environment
must be configured to support live migration from a technical point of view. In other words, both
node in OpenVZ-speak.
virtual environments must reside on the same subnet, and data transmission rate must be high To be able to use OpenVZ, you will
enough. Additionally, the target virtual environment (VE) must have sufficient hard disk space. If need a kernel with OpenVZ patches.
these conditions are fulfilled, the following command starts the migration: One problem is that the current stable
release of OpenVZ is still based on
vzmigrate ‑online target IP VEID
kernel 2.6.18, and what is known as
Target IP is the network address of the VE into which you want to migrate to the VE with the ID the super stable version is based on
of VEID. Of course, the vzmigrate tool supports a plethora of different options (e.g., for migrat- 2.6.9. It looks like the OpenVZ de-
ing over secure connections). The exact syntax and other examples of applications are discussed velopers can’t keep pace with official
[12]. Additionally, OpenVZ can create what it refers to as checkpoints (snapshots) of VEs: A kernel development. Various distribu-
checkpoint freezes the current state of the VE and saves it in a file. The checkpoint can be cre- tions have had an OpenVZ kernel,
ated from within the host context with the vzctl chkpnt VEID command. The checkpoint file such as the last LTS release (v8.04) of
can be used later to restore the VE on another OpenVZ host with vzctlrestore VEID.
Ubuntu, on which this article is based
(Figure 2).
The container abstraction layer makes ever, you can’t load any drivers or Ubuntu 9.04 and 9.10 no longer fea-
sure that the guest system sees its kernels from within a container. The ture OpenVZ, apart from the VZ tools;
own process namespace with separate predecessors of container virtualiza- this also applies to Ubuntu 10.04. If
process IDs. On top of this, the kernel tion in the Unix world are technolo- you really need a current kernel on
extension that provides the interface gies that have been used for decades, your host system, your only option is
is required to create, delete, shut such as chroot (Linux), jails (BSD), or to download the beta release, which
down and assign resources to con- Zones (Solaris). With the exception uses kernel 2.6.32. The option of us-
tainers. Because the container data of (container) virtualization in User- ing OpenVZ and KVM on the same
are extensible on the host file system, Mode Linux [4], only a single host host system opens up interesting pos-
resource containers are easy to man- kernel runs with resource containers. sibilities for a free super virtualization
age from within the host context. solution with which administrators
OpenVZ can experiment.
Efficiency If you are planning to deploy OpenVZ
OpenVZ is the free variant of a com- in a production environment, I sug-
Resource containers are magnitudes mercial product called Parallels gest you keep to the following rec-
more efficient than hypervisor sys- Virtuozzo. The kernel component is ommendations: You must disable
tems because each container uses available under the GPL; the source SELinux because OpenVZ will not
only as many CPU cycles and as code for the matching tools under the work correctly otherwise. Addition-
much memory as its active applica- QPL. OpenVZ runs on any CPU type, ally, the host system should only be
tions need. The resources the abstrac- including CPUs without VT exten- a minimal system. You will probably
tion layer itself needs are negligible. sions. It supports snapshots of active want to dedicate a separate partition
The Linux installation on the guest containers as well as the Live migra- to OpenVZ and to mount this below,
only consumes hard disk space. How- tion of containers to a different host /ovz, for example Besides this, you

Applications
Container Context

Resource Resource Resource

Container 1 Container 2 Container 3

Applications
Host Context
OpenVZ
Abstraction Layer

Host System Kernel

Figure 1: In virtualization based on resource containers, the host and guest use Figure 2: openSUSE with Ubuntu: System virtualization with resource containers
the same kernel; therefore, they must be of the same type. This means that a is an interesting option if you need to host (multiple) Linux guest systems as
Linux host can only support Linux guests. efficiently as possible on a Linux host system.

w w w. A d m i n - m AgA z i n e .co m Admin 01 53

V i rt ua l i z at i o n OpenVZ

Figure 3: I installing OpenVZ from the package sources for Ubuntu 8.04 – the Figure 4: The OpenVZ developers provide container templates for various guest
last version of Ubuntu to officially include an OpenVZ kernel. The only package systems; this makes installing a guest system a quick and easy experience.
needed for this was the linux‑openvz meta-package. Templates from the community are also available.

should have at least 5GB of hard disk
space, a fair amount of RAM (at least
4GB), and enough swap space.
Starting OpenVZ
Installing OpenVZ is simple. Users on
RPM-based operating systems such as
RHEL or CentOS can simply include
the Yum repository specified in the
quick install manual on the project
homepage. Ubuntu 8.04 users will
find a linux‑openvz meta-package
in the multiverse repository, which
installs the required OpenVZ kernel,
including the kernel modules and
header files (Figure 3). At the time of
writing, no OpenVZ kernel was avail-
able for Ubuntu 10.04. If you are in-
terested in using OpenVZ with a cur-
rent version of Ubuntu, you will find
a prebuilt deb package in Debian’s
unstable branch. To install type:
detailed information on this, refer to
the sysctl section in the quick install
guide, which covers providing net-
work access to the guest systems, in-
volving setting up packet forwarding
for IPv4 [5]. Then, you need to re-
boot with the new kernel. If you edit
sysctl after rebooting, you can reload
by typing sudo sysctl ‑p. Typing
sudo /etc/init.d/vz start
wakes up the virtualization machine.
Next, you should make sure all the
OpenVZ services are running; this is
done easily (on Ubuntu) by issuing:
sudo sysv‑rc‑conf ‑list vz
If the tool is missing, you can type
sudo apt‑get install sysconfig
to install it. Debian and Red Hat users
can run the legacy chkconfig tool. A
check of service vz status should
now tell you that OpenVZ is running.
Container Templates
OpenVZ users don’t need to install
an operating system in the traditional
sense of the word. The most con-
venient approach to set up OpenVZ
containers is with templates (i.e.,
tarballs with a minimal version of the
distribution you want to use in the
container). Administrators can create
templates themselves, although it’s
not exactly trivial [6]. Downloading
prebuilt templates [7] and copying
them to the template folder is easier:
sudo cp path_to_template U
/var/lib/vz/template/cache
Besides templates provided by the
OpenVZ team, the page also offers

sudo dpkg ‑i linux‑base_2.6.32‑10_all.debU

linux‑image‑2.6.32‑4‑openvz‑686_U
2.6.32‑10_i386.deb

The sudo apt‑get ‑f install com-

mand will automatically retrieve any
missing packages. You will also need
to install the vzctl tool, which has a
dependency for vzquota.
Before setting up the containers and
configuring the OpenVZ host envi-
ronment, you need to modify a few
kernel parameters that are necessary
to run OpenVZ in the /etc/sysctl.
conf file on the host system. For more Figure 5: A couple of clear-cut commands are used for creating and starting a VE and for entering the VE.

54 Admin 01 w w w. a d m i n - m aga z i n e .co m

 OpenVZ V i rt ua l i z at i o n

a number of community templates
(Figure 4).
Configuring the Host
Environment
The /etc/vz/vz.conf file lets you
configure the host environment. This
is where you specify the path to the
container and template data on the
host filesystem. If you prefer not to
use the defaults of
subnet and tell them the DNS server
address, which lets OpenVZ create
venet devices. All of the following
commands must be given in the host
context. To do this, you first need to
stop the VE and then set all the basic
parameters. For example, you can set
the hostname for the VE as follows:
sudo vzctl set VEID U
‑‑hostname Hostname ‑‑save
The ‑‑ipadd option lets you assign a
local IP address. If you need to install
a large number of VEs, use VEID as
the host part of the numeric address.
sudo vzctl set VEID ‑‑ipadd U
IP-Address ‑‑save
The DNS server can be configured us-
ing the ‑‑nameserver option:
sudo vzctl set VEID U
‑‑nameserver Nameserver-address ‑‑save E

TEMPLATE=/var/lib/vz/template
VE_ROOT=/var/lib/vz/root/$VEID
VE_PRIVATE=/var/lib/vz/private/$VEID

you can set your own paths. VE_ROOT

is the mountpoint for the root direc-
tory of the container. The private
data for the container are mounted in
VE_PRIVATE. VEID is a unique ID that
identifies an instance of the virtual
environment. All OpenVZ tools use Figure 6: The virtual environment uses venet devices to communicate with the outside world.
this container ID to address the re-
quired container.

Creating Containers
The vzctl, which is only available
in the host context, creates contain-
ers and handles most management Figure 7: The vzlist command outputs a list of active VEs.
tasks, too. In the following example,
I used it to create a new VE based on
a template for openSUSE 11.1 that I
downloaded:

sudo vzctl create VEID U

‑‑ostemplate suse‑11.1‑x86_64

The template name is specified with-

out the path and file extension. The
sudo vzctl start VEID starts the
VE, and sudo vzctl stop VEID stops
it again (Figure 5). The commands
sudo vzctl enter VEID and exit let
you enter and exit the VE.
Entering the VE gives you a working
root shell without prompting you for
a password. Unfortunately, you can’t
deny root access in the host context.

Network Configuration
The next step is to configure network
access for the container. OpenVZ
supports various network modes for
this. The easiest option is to assign Figure 8: User Bean Counters, a set of configuration parameters, allow the administrator to limit resources
the VEs an IP on the local network/​ for each virtual environment.

w w w. a d m i n - m aga z i n e .co m Admin 01 55

V i rt ua l i z at i o n OpenVZ
After restarting the VE, you should be
able to ping it from within the host
context. After entering the VE, you
should also be able to ping the host
or another client (Figure 6). For more
details on the network configuration,
see the “Network Modes” box.
OpenVZ Administration
The vzctl tool handles a number
of additional configuration tasks.
Besides starting, stopping, entering,
and exiting VEs, you can use the
‑set option to set a number of op-
erational parameters. Running vzlist
in the host context displays a list of
the currently active VEs, including
some additional information such as
the network parameter configuration
(Figure 7).
In the VE, you can display the pro-
cess list in the usual way by typing
ps. And, if the package sources are
configured correctly, patches and
software updates should work in the
normal way using apt, yum, or yast
depending on your guest system.
For the next step, it is a good idea to
enter the VE by typing vzctl enter
VEID. Then, you can set the root pass-
word, create more users, and assign
the privileges you wish to give them;
otherwise, you can only use the VEs
in trusted environments.
Network Modes
In many cases, a venet device is all you need
in the line of network interfaces in a VE. Each
venet device sets up a point-to-point connec-
tion to the host context and can be addressed
using an IP address from the host context.
Venet devices have a couple of drawbacks,
however: They don’t have a MAC address and
thus don’t support ARP or broadcasting, which
makes it impossible to use DHCP to assign IP
addresses. Also, network services like Samba
rely on functional broadcasting.
A virtual Ethernet (veth) device solves this
problem (Figure 9). These devices are sup-
ported by a kernel module that uses vzctl to
present a virtual network card to the VE. The
vzethdev sets up two Ethernet devices: one in
the host context and one in the VE. The devices
can be named individually, and you can manu-
ally or automatically assign a MAC address to
them. The host-side device can also reside on
a bridge to give the VE a network environment
56 Admin 01
Figure 9: Virtual Ethernet devices make the VE a full-fledged member of the network with all its advantages
and disadvantages.
Without additional configuration, the
use of VEs is a security risk because
only one host kernel exists, and each
container has a superuser. Besides
this, you need to be able to restrict
the resources available to each VE,
such as the disk and memory and
the CPU cycles in the host context.
OpenVZ has a set of configuration pa-
rameters for defining resource limits
known as User Bean Counters (UBCs)
[8]. The parameters are classified by
importance in Primary, Secondary,
and Auxiliary. Most of these param-
eters can also be set with vzctl set
(Figure 8).
For example, you can enter
sudo vzctl set 100 ‑‑cpus 2
to set the maximum number of CPUs
for the VE. The maximum permitted
disk space is set by the ‑‑diskspace
parameter. A colon between two val-
that is visible in the host context. Within the
container, the administrator can then use Linux
tools to configure the network interface with a
static address or even use DHCP.
The kernel module is loaded when the OpenVZ
kernel boots. You can check that you have it by
issuing the sudo lsmod | grep vzethdev
command. To configure a veth device in the
container, run
sudo vzctl set 101 ‑‑netif_add eth0 ‑‑save
where eth0 is the interface name in the con-
tainer context. The device name in the host
context defaults to vethVEID.
If needed, you can assign MAC addresses and
device names explicitly. The device can be
listed in the host context in the normal way
with ifconfig. The number following the dot
(0) refers to the device number; here, this is
the first veth device in the container with the
VEID of 100:
ues lets you specify a minimum and
maximum limit:
sudo vzctl set 100 ‑‑diskspace 8G:10G U
‑‑quotatime 300
Incidentally, sudo vzlist ‑o lists all
the set UBC parameters. Note that
some UBC parameters can clash, so
you will need to familiarize your-
self with the details by reading the
exhaustive documentation. To com-
pletely remove a container from the
system, just type the
sudo vzctl destroy
command.
Conclusions
Resource containers with OpenVZ
offer a simple approach to running
virtual Linux machines on a Linux
host. According to the developers, the
sudo ifconfig veth100.0
A bridge device is the only thing missing for
host network access; to set this up host-side,
give the sudo brtcl addbr vmbr0 com-
mand, then sudo brctl addif vmbr0
verth100.0 to bind it to the veth device, as-
suming bridge‑utils is installed. Host-side you
now have the interfaces l0, eth0, venet0,
and veth100.0. If the bridge device is set up
correctly, brctl show gives you a listing simi-
lar to Figure 10. The additional veth device set
up here, 100.1, is for test purposes only and is
not important to further steps.
Virtual network devices are slightly slower
than venet devices. Also, security might be
an issue with a veth device – this places a
full-fledged Ethernet device in the container
context, which the container owner could
theoretically use to sniff all the traffic outside
the container.
w w w. a d m i n - m aga z i n e .co m

virtualization overhead with OpenVZ
is only two to three percent more CPU
and disk load: These numbers com-
pare with the approximately five per-
cent quoted by the Xen developers.
The excellent values for OpenVZ
are the result of the use of only one
kernel. The host and guest kernels
don’t need to run identical services,
and caching effects for the host and
guest kernels do not interfere with
each other. The containers themselves
provide a complete Linux environ-
ment without installing an operating
system. The environment only uses
the resources that the applications
running in it actually need.
The only disadvantage of operating
system virtualization compared with
paravirtualization or hardware virtu-
alization is that, apart from the net-
work interfaces, it is not possible to
assign physical resources exclusively
to a single guest.
Otherwise, you can do just about
anything in the containers, includ-
ing installing packages and providing
services. Additionally, setting up the
Figure 10: This example includes one venet and one veth device in the host context. The latter is physically
connected to the host network via a bridge device. The host-side veth bridge looks like a normal Ethernet
device (eth0) from the container context.
w w w. a d m i n - m aga z i n e .co m
OpenVZ kernel requires just a couple
of simple steps, and the template
system gives you everything you need
to set up guest Linux distributions
quickly.
OpenVZ has a head start of several
years development compared with
modern hypervisor solutions such as
KVM and is thus regarded as mature.
Unfortunately, the OpenVZ kernel
lags behind vanilla kernel develop-
ment.
However, if you are thinking of de-
ploying OpenVZ commercially, you
might consider its commercial coun-
terpart Virtuozzo. Besides support,
there are a number of aspects to take
into consideration when using re-
source containers. For example, host-
ing providers need to offer customers
seamless administration via a web
interface, with SSH and FTP, or by
both methods; of course, the security
concerns mentioned previously can-
not be overlooked.
Parallels offers seamless integration
of OpenVZ with Plesk and convenient
administrations tools for, say, impos-
OpenVZ V i rt ua l i z at i o n
ing resource limits in the form of
the GUI-based Parallels Management
Console [9] or Parallels Infrastructure
Manager [10]. The excellent OpenVZ
wiki covers many topics, such as the
installation of Plesk in a VE or set-
ting up an X11 system [11]. OpenVZ is
the only system that currently offers
Linux guest systems a level of perfor-
mance that can compete with that of
a physical system without sacrificing
performance to the implementation
itself. This makes OpenVZ a good
choice for virtualized Linux servers of
any kind. n
Info
[1] Linux VServer: [http://​­linux‑vserver.​­org/​
­Welcome_to_Linux‑VServer.​­org]
[2] OpenVZ:
[http://​­wiki.​­openvz.​­org/​­Main_Page]
[3] Virtuozzo: [http://​­www.​­parallels.​­com/​­de/​
­products/​­pvc45]
[4] User-Mode Linux: [http://​
­user‑mode‑linux.​­sourceforge.​­net]
[5] OpenVZ quick install guide: [http://​­wiki.​
­openvz.​­org/​­Quick_installation]
[6] Creating your own OpenVZ tem-
plates: [http://​­wiki.​­openvz.​­org/​
­Category:Templates]
[7] Prebuilt OpenVZ templates:
[http://​­wiki.​­openvz.​­org/​­Download/​
­template/​­precreated]
[8] OpenVZ User Bean Counters: [http://​­wiki.​
­openvz.​­org/​­UBC_parameters_table]
[9] Parallels Management Console:
[http://​­www.​­parallels.​­com/​­de/​­products/​
­virtuozzo/​­tools/​­vzmc]
[10] Parallels Infrastructure Manager: [http://​
­www.​­parallels.​­com/​­de/​­products/​­pva45]
[11] X11 forwarding:
[http://​­wiki.​­openvz.​­org/​­X_inside_VE]
[12] Live migration: [http://​­openvz.​­org/​
­documentation/​­mans/​­vzmigrate.​­8]
The Author
Thomas Drilling has been a freelance journalist
and editor for scientific and IT magazines for
more than 10 years. With his editorial office
team, he regularly writes on the subject of open
source, Linux, servers, IT administration, and
Mac OS X. In addition to this, Thomas Drilling is
also a book author and publisher, a consultant
to small and medium-sized companies, and a
regular speaker on Linux, open source and IT
security.
Admin 01 57
SUBSCRIBE NOW
Save
30%
or more!

The New IT
New tools, new threats, new technologies...
Looking for a guide to the changing world
of system administration?

www.admin-magazine.com/subs
AND SAVE 30%!
ADMIN Network & Security
Explore the new world
of system administration

It isn’t all Windows anymore – and it isn’t all Linux. A router is more than a router.
A storage device is more than a disk. And the potential intruder who is looking for a way
around your security system might have some tricks that even you don’t know. Keep your
network tuned and ready for the challenges with the one magazine that is all for admins.

Each issue delivers technical solutions to the real-world problems you face every
day. Learn the latest techniques for better:

• network security

ORLD
• system management
R E A L - W
• troubleshooting
B L E M S
PR O
!
• performance tuning
SOLVED
• virtualization

• cloud computing

on Windows, Linux, Solaris, and popular varieties of Unix.

Special introductory offer!

Order by December 31st to save 10%
off the regular subscription price!
Subscription prices for 1 year of ADMIN Magazine (4 issues + 4 DVDs)
are $39.95 for the USA and £24.90/29.90 for UK/Europe
(other regions please see our website).

www.admin-magazine.com/subs
 V i rt uA l i z At i o n Virtual machine manager
Microsoft System Center Virtual Machine Manager 2008 R2
© Tono Balaguer, 123RF.com
Virtual Windows
in theory, virtualizing all your old servers is a good idea, but managing them won’t necessarily become any
easier. Virtual machine manager gives windows administrators an easy option. By Björn Bürstinghaus
Service virtualization is no longer
just an interesting topic for large
corporations and data centers. In
fact, virtualization of multiple server
systems on a single physical machine
has become an affordable option for
small and medium-sized businesses,
too. With virtualization and the
consolidation benefits that it offers,
system management also changes.
The machines you are managing are
Figure 1: The Virtual Machine Manager startup screen shows a selection of the version is a cost-
components to install.
60 Admin 01
only available as virtual entities and managing an unlimited number of
as the number of virtual machines virtual machines.
and virtualization hosts continues to
rise, administrators need to consider System Requirements
centralizing their management.
Microsoft System Center Virtual Ma- To install SCVMM, you need a 64-
chine Manager 2008 R2 (SCVMM) is bit version of Windows Server 2008
a management solution for Hyper-V (R2), which you can run as a virtual
(R2) hosts, Virtual Server 2005 R2 machine in smaller environments
hosts, and VMware ESX hosts that (with a maximum of five hosts). The
use the VMware VCenter. SCVMM [1] system on which you install SCVMM
offers excellent must be a member of an Active Direc-
scalability, easy tory domain, but you can use it to
management of manage host systems in non-member
hosts and virtual networks. In this case, you’ll need to
machines and install the agents manually because
many benefits automatic installation only works
for the adminis- inside the domain. SCVMM relies on
trator. A Work- Microsoft SQL Server to store data.
group Edition Depending on the size of your envi-
is available for ronment, you can use the free SQL
deployment in Server 2005 Express Edition supplied
small and me- with the bundle, which has a data-
dium-sized busi- base size limit of 4GB, or you can use
nesses: If you an instance of SQL Server 2005 or
use a maximum SQL Server 2008.
of five hosts, this It makes sense to use a separate
server for the database in larger envi-
effective way of ronments. You can install the SCVMM
w w w. A d m i n - m AgA z i n e .co m
 Virtual Machine Manager V i rt ua l i z at i o n

database component on a separate

system; it doesn’t need to reside on
the management server. To install the
management component in SCVMM,
you also need the Windows Auto-
mated Installation Kit (WAIK) 1.1,
which is automatically installed when
you install the management server;
Windows PowerShell 1.0 or 2.0; Win-
dows Remote Management (WinRM)
1.1 or 2.0; and the .NET framework
3.0 (SP1).
The web-based SCVMM Self-Service
Portal additionally requires Internet
Information Server (IIS) version 7.0 or
7.5. The SCVMM Administrator Con-
sole can also be installed on client
systems such as Windows Vista and
Windows 7. If you also want to send
command-line jobs to SCVMM via the
client, you additionally need to install
Windows PowerShell client-side.
Figure 2: The management console gives you an overview of the virtual machines in the central panel and the

Installation system load for the selected VM below.

After starting the installation, you’ll The agents for managing hosts via Microsoft offers a free configuration
be given a choice of components to SCVMM can be installed through the analyzer for SCVMM; after the install,
install for SCVMM (Figure 1). The management console or manually. the analyzer checks whether all the
management server, management If you use the management console components are installed optimally.
console, and self-service portal are for the install, an automatic check is Also, you can use the configuration
all installed separately. When you performed to see whether the host analyzer to check the configuration
install the management server, you has a hypervisor. If not, the Hyper-V on remote systems that you will be
can place the individual modules, role will be installed automatically on deploying as hosts and systems in-
such as the database and the library Windows Server 2008 (R2), and Vir- tended for P2V conversion. If any
server, on different systems. This ar- tual Server 2005 R2 will be installed issues are detected, you’ll be given
rangement improves performance if on Windows Server 2003 (R2). a detailed description and possible
you have a large number of hosts and
virtual machines.
Before the installation starts, an au-
tomatic check is performed to make
sure that the hardware and software
requirements are fulfilled. If this is
the case for all the components, pro-
visioning SCVMM will take less than
20 minutes.
Before installing the self-service por-
tal, you must enable the web server
role on Windows Server 2008 (IIS 7.0)
or Windows Server 2008 R2 (IIS 7.5),
as well as the ASP.NET, IIS-6 meta-
compatibility and IIS 6 WMI compat-
ibility role services. If the role or one
of the additional services is not in-
stalled, you will see an error message
to this effect. Portal access privileges
are configured in the management
console preferences. Figure 3: The Job queue shows modified and outstanding jobs.

w w w. a d m i n - m aga z i n e .co m Admin 01 61

V i rt ua l i z at i o n Virtual Machine Manager
Figure 4: Privileges in the self-service portal: The Web
allowed to manage its own virtual machines.
solutions. To use the configuration
analyzer for SCVMM, you also need
the free Microsoft Baseline Security
Analyzer.
The Management Console
The SCVMM management console
allows administrators to handle a full
set of host and virtual machine ad-
ministrative tasks. The management
console is clear-cut and configurable
in many places.
The console is divided into three ar-
eas (Figure 2). The left-hand panel
contains the navigation aids for the
various SCVMM subsections (Hosts,
Virtual Machines, Library, Jobs, and
preferences). After selecting a subsec-
tion, its objects are listed in the cen-
tral panel of the console. The right-
hand panel shows you the actions
available for the selected subsection.
The Hosts and Virtual Machines sub-
sections give you a perfect overview
of the status of all your systems. If
you manage a large number of sys-
tems, you can use the filters in the
navigation area on the left to restrict
the number of systems shown.
The Jobs subsection is used to check
all active SCVMM jobs (Figure 3);
Again, you can use the navigation
aids to filter on various criteria. This
helps you quickly identify and resolve
errors and issues.
The Self-Service Portal
With the SCVMM self-service portal,
you can grant individual users or
62 Admin 01
groups access
to virtual ma-
chines or provide
templates for
creating new
systems by way
of a web-based
interface. This
system means
you can allow
developers to
restart a test sys-
tem themselves
Admins user group is or allow them
to create a new
virtual machine
based on templates from the library,
without requiring that they access the
management console. Additionally,
you can assign privileges for the por-
tal individually for various groups or
users, thus allowing certain users to
manage or access a virtual machine
but not restart it or switch it off (Fig-
ures 4 and 5).
After logging in to the self-service
portal, the user sees all the virtual
machine assignments. To open a
connection and manage a virtual ma-
chine, an ActiveX control is installed
on the client side; the control requires
Internet Explorer.
Using shared templates, users can
create new virtual machines based on
the template in the portal. The host
is then assigned by the built-in intel-
ligent placement function in SCVMM.
Also note that to control virtual ma-
Figure 5: The self-service portal lets non-privileged users manage virtual machines.
chines on VMware ESX hosts, SSL
authentication must be enabled on
the host side or the VMware ActiveX
control must be installed on the client
from which you will be managing the
host.
Command Line-Based
Controls
PowerShell [2] is an extension of the
well-known Windows command line;
it offers a plethora of administrative
commands for script-based Windows
management.
SCVMM includes more than 150
PowerShell commands, or Cmdlets,
which you can use for command
line-based management and admin-
istration without having to launch
the management console (Figure 6).
Thus, you can use scheduled tasks on
Windows to run tasks at a predefined
time on the management server – for
example, to store the status of a vir-
tual machine.
Intelligent Placement
One of SCVMM’s most practical
features is intelligent placement of
virtual machines. Because the man-
agement server monitors the load on
the hosts, it automatically displays
a host statistic when you add a new
machine so that you can easily see
which host is best suited to the task.
You can customize the automatic host
w w w. a d m i n - m aga z i n e .co m

evaluation function for intelligent
placement.
Libraries and Templates
The library component in SCVMM
is a shared directory of virtual hard
disks, ISO images, hardware, and
guest operating system profiles.
Templates automatically provision
Windows client and server systems
quickly. A template comprises a vir-
tual hard disk and predefined hard-
ware and operating system profiles.
The hardware profile lets you specify
the minimum requirements for the
CPU type or the amount of RAM the
virtual machine needs. When a new
virtual machine with the specified
CPU type is created, SCVMM auto-
matically searches for a host with re-
sources to match the hardware profile
requirements. The operating system
profile helps automate operating sys-
tem provisioning. Besides selecting
the operating system, you can also
configure the administrator password,
a license key, the computer name,
and the domain membership.
P2V Conversion
SCVMM also converts physical sys-
tems to virtual machines on the fly
with physical-to-virtual (P2V) migra-
tion. For this, simply install a small
client on the machine; the client
checks the system and displays po-
Figure 6: Using the PowerShell to move virtual machines between hosts.
w w w. a d m i n - m aga z i n e .co m
Virtual Machine Manager
tential issues before using the volume
shadow copy service to create an
image. On-the-fly conversion works
with client systems as of Windows XP
and for server systems as of Windows
Server 2003. For older systems, you
have only an offline conversion op-
tion. After conversion, you can shut
down the physical system and boot
the system as a virtual machine.
Higher Availability with
Clustering
Host clustering is a useful way to
guarantee system availability. Instead
of expensive SAN memory, the data is
provided by cheaper iSCSI solutions.
To create a Hyper-V cluster, you need
two host systems, both of which ac-
cess the same SAN or iSCSI storage.
Live migration introduced in Hyper-V
R2 means you can move a virtual ma-
chine between clusters without taking
the virtual machine offline. The previ-
ous version only supported virtual
machine migration if you used the
same processor type in both clusters.
Although this restriction has not been
completely lifted, it only applies to
the CPU vendor, thus improving sup-
port for a variety of hardware in the
cluster and offering more flexibility.
Resource Monitoring
SCVMM can be combined with the
System Center Operations Manager
V i rt ua l i z at i o n
2007 R2 [3] for monitoring host and
virtual machine availability. In this
case, SCVMM not only uses its own
agent to monitor the systems but also
provides performance analysis and
reporting for a host or virtual ma-
chine. The performance and resource
optimization (PRO) function built into
SCVMM can use Operations Manager
2007 R2 to collect performance data
down to the application layer and
thus suggest optimization strategies,
which are displayed as PRO tips in
the management console.
Conclusions
Microsoft System Center Virtual
Machine Manager 2008 R2 greatly fa-
cilitates the management and admin-
istration of homogeneous or hetero-
geneous virtual infrastructures under
Windows. Automated provisioning
of new client and server systems can
be done in minutes with SCVMM.
Thanks to the integration of System
Center Operations Manager 2007 R2,
SCVMM also directly supports perfor-
mance and availability monitoring for
hosts and virtual machines.
Because system management al-
ways takes a great deal of your time,
whether you have five or 50 host
systems, it makes sense to plan a
centralized solution for all aspects of
virtualization, which is exactly what
SCVMM offers. n
Info
[1] Microsoft System Center Virtual Machine
Manager 2008 R2: [http://​­support.​
­microsoft.​­com/​­kb/​­974722]
[2] PowerShell: [http://​­www.​­microsoft.​
­com/​­windowsserver2003/​­technologies/​
­management/​­powershell/​­default.​­mspx]
[3] Microsoft System Center Operations Man‑
ager 2007 R2:
[http://​­www.​­microsoft.​­com/​­systemcenter/​
­en/​­us/​­operations‑manager.​­aspx]
The Author
Björn Bürstinghaus is a system administrator
with simyo GmbH in Düsseldorf, Germany. In his
leisure time, he runs Björn’s Windows Blog, a
blog on Microsoft Windows topics located at
[http://​­blog.​­buerstinghaus.​­net].
Admin 01 63
 Ma n ag e m e n t Teamviewer
© Thor Jorgen Udvang, 123RF.com
Convenient graphical remote control
Remotely
Controlled
Teamviewer is an impressive demonstration of how easy remote control
across routers and firewalls can be. The popular software is now available
for Linux. By Daniel Kottmair
Some 60 million users already have
the Teamviewer [1] commercial re-
mote control solution running on
Windows and Mac OS X. Because of
the many requests from customers,
Teamviewer’s manufacturer now pro-
vides a variant for Linux in version 5.
Teamviewer facilitates remote access
to other computers across a network.
The only requirement is that the ma-
chine at the other end is also running
Teamviewer. Teamviewer provides all
this functionality in a standalone pro-
gram; special client or server versions
are not available.
Teamviewer automatically generates a
globally unique ID on each machine.
When it is launched, Teamviewer
generates a new password that the
64 Admin 01
computer on the opposite end of the
connection can use to access the local
machine. This scheme prevents any-
body who has ever logged in to that
machine from doing so again without
the owner’s authorization. You can
keep the newly generated password
or define one yourself.
Connections
Remote access without port forward-
ing works across routers and firewalls
thanks to one of the globally distrib-
uted Teamviewer servers on the web,
which initiates a 256-bit encrypted
UDP connection between the two
parties. If a proxy server or a firewall
with content filtering makes this con-
nection impossible, the transfer is
handled directly by the Teamviewer
server. The HTTP label in the window
header, rather than the UDP label,
identifies this kind of connection. If
you are worried about using a third-
party server, Teamviewer will sell you
your own authentication server on
request.
Teamviewer will even let you re-
motely control computers that only
have a modem connection. The soft-
ware vendor improved compression
in version 5 to reduce the amount of
data crossing the wire. Video, Flash
banners, and other applications that
permanently change screen content
are problematic, but a fast DSL con-
nection will let even those types of
applications run at an acceptable
speed.
Private users can run the program
free of charge, and the vendor offers
commercial licenses for commercial
use. Teamviewer is available for Win-
dows, Mac OS X, and Linux; any plat-
form can remotely control any other
platform. An iPhone client, available
after registering for free online, lets
you remotely control a computer as
well.
The web and iPhone clients are the
only versions that can only control,
rather than work in both directions.
The other variants let you control
w w w. a d m i n - m aga z i n e .co m
 Teamviewer Ma n ag e m e n t

and be controlled, and you can even

change directions mid-flow.

Linux Specifics
Teamviewer offers downloads of deb
and RPM versions 5.0.8206 pack-
ages for Linux, along with an X64
deb package and a simple tarball that
you don’t even need to install. Team-
viewer for Linux is still beta, and the
vendor asks for feedback and bug
reports.
The program is based on a modified
version of Wine, although the ven-
dor has made some Linux-specific
changes (e.g., to accelerate reading of
X server graphics). Although it uses
Wine, it is not just a copy of the Win- Figure 1: The Teamviewer window at program launch.
dows version. A native Linux version
is not available right now, but the ing the wallpaper (to avoid unneces- a connection between a v4 Mac client
vendor is considering creating one de- sary data traffic). and a v5 Linux client worked without
pending on acceptance and popularity problems.
of the Wine-based version. In the Lab The program offers three operating
The program offers a plethora of modes when it launches (Figure 1):
built-in remote control solutions, such The Linux version works fairly well Remote support, Presentation, and
as the ability to change direction, despite its beta status. One thing that File transfer. Remote support lets
reboot, simulate a Ctrl+Alt+Del key always worked during testing – no you remotely control a system, and
press, and transfer files conveniently matter what network the comput- presentation mode lets you demon-
between two machines. Multiple ers used or what firewalls they were strate an action to one or multiple
logins on a single machine are also hiding behind – was the connection users on their own machines. File
supported (e.g., for training purposes setup. Teamviewer has thus mas- transfer leaves out the administrative
in which you need to demonstrate tered the most important discipline and graphic functions and simply
something to a group of users). in remote control with flying colors. sends files to, or retrieves them from,
The VoIP and video chat function Also, no version problems emerged; a remote machine. This mode is ac-
introduced in version 5 is also use-
ful, and the application relies on free
codecs: Speex for audio and Theora
for video. Video on Linux only works
in the receiving direction right now.
V4L-connected Linux webcams are
not currently supported by Team-
viewer.
The Linux version has a couple of
other restrictions: The whiteboard
function, which lets users draw on
a whiteboard at the same time, and
VPN support both fail to provide the
goods. The program does not transmit
virtual consoles, so you need an X
server. The reboot, Ctrl+Alt+Delete
key sequence, and Disable Input/​
Display on remote computer functions
all require the remote machine to run
Windows – Mac users have a similar
problem. And, the same thing applies
to changing the resolution and remov- Figure 2: File transfers between computers are convenient and easy to keep track of.

w w w. a d m i n - m aga z i n e .co m Admin 01 65

Ma n ag e m e n t Teamviewer
cessible at any time during normal
remote support.
From the Teamviewer startup win-
dow, use Extras | Options to change
various settings, such as your own
computer name, or to assign a fixed
password for remote logins. Also, you
can specify which privileges you want
to grant a remote user accessing a
system across the wire. Teamviewer
will also accept a whitelist or blacklist
of computers that are allowed or not
allowed to access your computer.
File Transfers
File transfers occur in a separate win-
dow (Figure 2) that features a two-
column view, with your own com-
puter on the left and the remote com-
puter on the right. To transfer files,
just select them and click the button
above the column. This seems a little
convoluted; drag-and-drop, or at
least double-clicking, to transfer files
would make more sense. The vendor
aims to change this soon. Also, the
Windows-style Wine symlinks and
drive letters are a little irritating for
Linux users.
In normal remote support mode, a
hideable Teamviewer function bar is
displayed on the remote desktop, and
you can use it to access a full set of
important remote control functions.
Figure 3: View of the remote desktop in Teamviewer.¡
66 Admin 01
On the bottom right is the connection
monitor, which you can also hide and
which tells you who is accessing the
computer across the wire (Figure 3).
If you so desired – again this could be
useful for training purposes – the pro-
gram will use screencasting to moni-
tor activities on the remote computer.
The screencast files you can save only
contain the data stream transmitted
across the wire by Teamviewer and
are thus quite compact. Administra-
tors can create a list of computers
for single-click access to remote ma-
chines.
If needed, you can change the trans-
mission focus. The High speed option
reduces the color depth to speed up
the transmission. If you prefer perfect
image quality at the cost of smoother
operations, you can opt for this in
the View menu. The Automatic set-
ting changes the mode to reflect the
connection. Unfortunately, Team-
viewer changes the standard gray of
some desktops (including Gnome on
Ubuntu) to a horrible pink. Because
I didn’t notice any speed boosts in
reduced color mode, you might pre-
fer to keep a high-quality mode, for
Linux-to-Linux connections, at least.
A couple of options make the remote
support experience smoother. To
begin with, you will want to disable
desktop effects, such as those pro-
vided by Compiz. Teamviewer only
transfers the window content and not
the windows, so window zooms and
soft fades just create unnecessary traf-
fic. A Flash blocker for the browser is
also a good idea for the same reason.
Animated ads cause an unnecessarily
high level of data – and the bigger the
ad space, the worse the problem.
Beta Blockers
The older the distribution I tried, the
more difficulties I had in testing. The
program performs better and is more
stable on more recent versions; for ex-
ample, virtually no problems occurred
in connections between Ubuntu 9.04
and 9.10. I definitely advise against
changing the resolution on remote
Linux clients, because it typically
caused Teamviewer to crash on the
client. Also, you shouldn’t change the
native system settings during remote
access, because that can interrupt the
data stream from the remote machine.
Another failing: The mouse cursor
doesn’t change its appearance from
an arrow to a hand, for example, if
it moves outside of a window or title
bar. The vendor has promised to fix
this before the final release.
Conclusions
Teamviewer is an easy-to-use and
practical piece of software. Even if
you aren’t an administrator or con-
sole jockey, it gives you a really sim-
ple approach to managing machines
remotely – or for accessing your com-
puter when you’re on the road. Team-
viewer works quite well on Linux,
even though the beta had a few bugs.
The vendor promises to have every-
thing resolved by the time of the final
release. One of Teamviewer’s main
strengths is its cross-platform compat-
ibility between Linux, Windows, Mac
OS X, and even web browsers and the
iPhone. Also, the connection always
works. n
Info
[1] Teamviewer:
[http://​­www.​­teamviewer.​­com/​­index.​­aspx]
w w w. a d m i n - m aga z i n e .co m
NOV. 7–12 24th Large Installation San Jose
2010 System Administration Conference california

Uncovering the Secrets

of
System Administration

s e 2 0 10 re
San J o Baltimo 9
200

8
a n D ie go 200
S Dallas
2007

Program Includes:
Unraveling the Mysteries oF Twitter Infrastructure,
Legal issues in the Cloud, and Huge NFS at Dreamworks

SPONSORED KEYNOTE AD
STAPLE

BY DRESS BY
HERE

Tony Cass,
CERN
in cooperat
ion with LO
PSA & SNIA
**JOIN US
TRAINING O FOR 6 DAYS OF PRACTI
N TOPICS I CAL **PLUS A 3
* 6-day Virt
N CL UDI NG: -DAY
ualization
Track by in TECHNICAL
John Arrasj
id and Rich structors PROGRAM
ard McDoug including
* Advanced Ti all
me Manageme Invited Tal
INSERT

* Dovecot an nt: Team Ef

ficiency by ks
d Postfix Tom Limonc Refereed Pa
* 5-day Linu by Patrick elli pers
x Security Be n Koetter an
and Admini d Ralf Hildeb
randt
Workshops
stration Tr
ack Vendor Exhi
**REGISTER bition
BY OCTOBER
www.usenix 18 AND SAV
.org/lisa1 E**
0/lp
 m A n Ag e m e n t chef
Configuration management with Chef
Chef de Config
© Alistair Cotton, 123RF.com
ever dream of rolling out a complete computer farm with a single mouse
click? if you stick to Linux computers and you speak a little Ruby, chef can
go a long way toward making that dream come true. By tim Schürmann
Chef is basically a server that stores
customized configuration guides for
software. Clients connected to the
server access the recipes and auto-
matically configure their systems on
the basis of the rulesets the recipes
contain.
To do so, the clients not only modify
their configuration files, but – if
needed – launch their package man-
agers. If the recipes change, or new
ones are added at a later date, the
clients affected automatically update
to reflect the changes. In an ideal en-
vironment, this just leaves it up to the
administrator to manage the recipes
on the server.
Bulk Shopping
Before you can enjoy the benefits,
the developers behind Chef expect
you to put in a modicum of work. For
example, recipes are made up of one
or multiple standard Ruby scripts. If
you need anything beyond the fairly
generic recipes available on the web,
you need to have a good command of
the Ruby scripting language. In other
words, your mileage will vary before
68 Admin 01
you deploy a home-grown and home-
tested solution.
The installation is another obstacle
– and a fairly complex one, too, be-
cause the Chef server depends on sev-
eral other components, each of which
in turn requires even more software
packages. The Chef server itself is
written in Ruby but relies on the Rab-
bitMQ server and on a Java-based
full-text search engine, at the same
time storing its data in a CouchDB
database.
Finally, your choice of operating sys-
tem is also important. Chef prefers
Linux underpinnings, but it will also
run on other Unix-flavored operat-
ing systems such as Mac OS X, Open
Solaris, FreeBSD, and OpenBSD, ac-
cording to the wiki [1]. The fastest
approach today is offered by Debian
5, Ubuntu 8.10 or later, or CentOS
5.x. Setting up the server on any
other system can be an adventure.
This article mainly relates to Debian
and Ubuntu for this reason. If this is
the first time you have ever cooked
one of Chef’s recipes, it is also a good
idea to run your kitchen on a virtual
machine. This prevents things boiling
over and making a mess on the server
room floor.
Valuable Ingredients
A full-fledged Chef installation com-
prises the systems you want to config-
ure (nodes) and the server that man-
ages and stores the recipes. Chef cli-
ents do all the hard work, picking up
the recipes from the server via a REST
interface and running the scripts.
Each client runs on one node but can
apply recipes to multiple nodes. [Fig-
ure 1] shows you how this works.
For simplicity’s sake, the following
examples just use the Chef server and
a single client. The latter only config-
ures the computer on which it is run-
ning. The first thing you need to have
in place is Ruby version 1.8.5 through
1.9.2 (with SSL bindings). Add to
this, RubyGems, which will want to
build various extensions and libraries
later on, thus necessitating the exis-
tence of make, gcc, g++, and the Ruby
developer packages. Additionally, you
need the wget tool for various down-
loads. The following command in-
stalls the whole enchilada on Debian
and Ubuntu Linux:
sudo apt‑get install ruby U
ruby1.8‑dev libopenssl‑ruby1.8 U
rdoc ri irb build‑essential U
wget ssl‑cert
w w w. A d m i n - m AgA z i n e .co m
 Chef Ma n ag e m e n t

you need to concentrate on the instal-

Server
Provides recipes
Figure 1: Overview of the Chef landscape with the server, clients, and nodes.
The packages for openSUSE are called
ruby, ruby-devel, wget, openssl-
certs, make, gcc, and g++. The certifi-
cates from ssl-cert will be required
later.
According to the how-to [1], Chef
prefers RubyGems version 1.3.6 or
newer, but not 1.3.7. This version
contains a bug that kills the following
installation mid-way. Because most
distributions have an older version of
RubyGems, your best bet is to head
for the source code archive:
cd /tmp
wget http://rubyforge.org/frs/U
download.php/69365/rubygems‑1.3.6.tgz
tar zxf rubygems‑1.3.6.tgz
cd rubygems‑1.3.6
sudo ruby setup.rb
If the last command installs the Gems
executable as /​usr/​bin/​gem1.8 (as is
Collects recipes
Client
and executes scripts
the case with Debian and Ubuntu), a
symbolic link will improve things:
sudo ln ‑sfv /usr/bin/gem1.8 /usr/bin/gem
Now you can issue the following
Gems command to install the Chef
package:
sudo gem install chef
When you run a Gem update, keep
an eye on the JSON Gem. The ver-
sion that now comes with RubyGems,
1.4.3, causes an error in Chef. If gem
update installs the offending JSON
package on your disk, these com-
mands revert to the original version:
sudo gem uninstall ‑aIx json
sudo gem install ‑v1.4.2 json
The steps thus far provide the under-
pinnings for Chef operations. Now,
lation, particularly server-side.
Nodes
Who’s the Chef?
Chef can automate the process of in-
stalling and configuring software, so
it only seems logical to let Chef install
itself. The developers refer to this
process as bootstrapping. Having said
this, recipes that install the server
in this way only exist for Debian 5,
Ubuntu 8.10 or later, and CentOS 5.x.
On any other distribution, you need
to perform all of the steps manually
as described in the [Manual Server
Installation] boxout.
Life is a little easier with one of the
operating systems officially supported
by Chef. To begin, make sure the
computers involved have Fully Quali-
fied Domain Names (FQDNs), such
as chefserver.example.com. If you
don’t, you will be bombarded with
error messages like Attribute domain
is not defined! (ArgumentError)
later on. Additionally, the repositories
need to provide the runit program in

Manual Server Installation

If you need to set up the Chef server manually, start by installing the openSUSE goes for zlib-devel and libxml-devel. Now, finally, you
RabbitMQ messaging server [2]. openSUSE users should use the open- can install the Chef server
SUSE Build Service to install rabbit-mq [3]. Doing so means that YaST
sudo gem install chef‑server chef‑server‑api chef‑server chef‑solr
automatically adds repositories that you need later on.
Once you have RabbitMQ in place, it’s time to start the Chef configura- and add the really practical web front end:
tion:
sudo gem install chef‑server‑webui
sudo rabbitmqctl add_vhost /chef
After completing this work, create the /​etc/​chef/​server.rb con-
sudo rabbitmqctl add_user chef testing
figuration file. [Listing 1] gives you a template. As a minimum, you need
sudo rabbitmqctl set_permissions ‑p /chef chef ".*" ".*" ".*"
to replace the domain name that follows chef_server_url with the
The next task on the list concerns the CouchDB database from the output from hostname ‑f and add a password of your choice after
CouchDB package. If needed, you can start the service manually on web_ui_admin_default_password. All the other defaults you can
openSUSE by typing rccouchdb start. The Chef server also requires keep, particularly the paths, which the server automatically creates
Sun Java SDK version 1.6.0. Some distributions keep this package in an later, should the need arise.
external or special repository. On Debian, you need to enable the non- In the next step, the script shown in [Listing 2] creates a pair of SSL
free package source; on Ubuntu 10.04, you can add the partner reposi- certificates, which you will need. The following command line creates
tory like this: the search index:
sudo add‑apt‑repository "deb http://archive.canonical.com/ U sudo chef‑solr‑indexer
lucid partner"
Another command launches the Chef SOLR Server
sudo apt‑get update
sudo chef‑solr
Now install the JDK. On Debian and Ubuntu, the JDK is hidden away
in the sun-java6-jdk package, whereas openSUSE calls it java- and the Chef server itself,
1_6_0-sun-devel. Users on openSUSE will probably want to delete the
sudo chef‑server ‑N ‑e production
OpenJDK packages java-1_6_0-openjdk and java-1_6_0-openjdk-
devel to be on the safe side.
including the graphical web interface:
Then, you just need to install the developer packages for zlib and
libxml. Debian and Ubuntu call them zlib1g-dev and libxml2-dev; sudo chef‑server‑webui ‑p 4040 ‑e production

w w w. a d m i n - m aga z i n e .co m Admin 01 69

 Ma n ag e m e n t Chef

Table 1: Directories in a Repository If the command terminates with a

Directory Content
certificates/​ SSL certificates (typically created by rake ssl_cert)
config/​ General configuration files for the repository
cookbooks/​ Complete cookbooks
roles/​ Role definitions
site-cookbooks/​ Modified cookbooks; any cookbooks stored here will overwrite or
modify the ones stored in cookbooks
a package named runit (don’t install
this yourself!).
The Chef server also requires Sun
Java SDK version 1.6.0, which the
distributions love to hide in a special
repository. Debian users need to en-
able the non-free package source for
this, whereas Ubuntu users can add
the partner repository with the fol-
lowing two lines:
sudo add‑apt‑repository U
"deb http://archive.canonical.com/ U
lucid partner"
sudo apt‑get update
Theoretically, the Chef server will
run with the OpenJDK, although the
developers do not give you any guar-
antees.
Lonely Kitchen Helper
After fulfilling all the requirements,
you can create configuration files
for Chef Solo on the server and the
client. This Chef variant runs the
recipes directly on the client without
involving the server. Without the
aid to creating simple scripts – for
installing the full-fledged server and
clients. To do this, create a ~/​solo.rb
file with the following three lines on
each of the systems involved:
file_cache_path "/tmp/chef‑solo"
cookbook_path U
"/tmp/chef‑solo/cookbooks"
recipe_url U
"http://s3.amazonaws.com/U
chef‑solo/bootstrap‑latest.tar.gz"
This tells Chef Solo where the instal-
lation recipes are located.
Chef on Call
Now it’s time to move on to the
server candidate. On this machine,
create a JSON configuration file
named ~/​chef.json to provide infor-
mation about the node. See [Listing
3] for the file content.
To match your local environment, you
need to modify the server name for
server_fqdn. To set up the full-fledged
Chef server, give the command:
sudo chef‑solo ‑c ~/solo.rb ‑j U
cryptic error message, try running it
again. During testing, the installation
ran without any errors. First, the com-
mand installs a Chef client, then Rab-
bitMQ, CouchDB, the developer pack-
ages for zlib and xml, and the Chef
server, including the indexer and a
web GUI you can use later to manage
the Chef server (WebUI). It then goes
on to create matching configuration
files and the required directories and
adds init script entries for the server
to round off the process.
At the end of this procedure, the Chef
server should be listening on port
4000; the web GUI is accessible on
port 4040. Java and RabbitMQ use the
Apache SOLR-based full-text search
engine built into the Chef server.
Among other things, it provides infor-
mation about the existing infrastruc-
ture, which in turn can be referenced
for recipes. For details of the search
function, see the wiki page [4].
Workers
Once you have the server up and run-
ning, it’s time to turn to the client.
Start by creating a ~/​chef.json JSON
configuration file. [Listing 4] gives
you the content. The server_fqdn
entry here must contain the server
name, not the client’s.
Now you can launch Chef Solo:
sudo chef‑solo ‑c ~/solo.rb U

server, Chef Solo is only useful as an ~/chef.json ‑j ~/chef.json U

Listing 1: Template for server.rb

01 
log_level :info 19 v
alidation_key "/etc/chef/validation.pem"
02 
log_location STDOUT 20 c
lient_key "/etc/chef/client.pem"
03 
ssl_verify_mode :verify_none 21 w
eb_ui_client_name "chef‑webui"
04 
chef_server_url "http://chef.example.com:4000" 22 w
eb_ui_key "/etc/chef/webui.pem"
05  23 
06 
signing_ca_path "/var/chef/ca"
24 w
eb_ui_admin_user_name "admin"
07 
couchdb_database 'chef'
25 w
eb_ui_admin_default_password "somerandompasswordhere"
08 
26 
09 
cookbook_path [ "/var/chef/cookbooks", "/var/chef/site‑cookbooks" ]
27 s
upportdir = "/srv/chef/support"
10 
28 s
olr_jetty_path File.join(supportdir, "solr", "jetty")
11 
file_cache_path "/var/chef/cache"
29 s
olr_data_path File.join(supportdir, "solr", "data")
12 
node_path "/var/chef/nodes"
30 s
olr_home_path File.join(supportdir, "solr", "home")
13 
openid_store_path "/var/chef/openid/store"
14 
openid_cstore_path "/var/chef/openid/cstore" 31 s
olr_heap_size "256M"

15 
search_index_path "/var/chef/search_index" 32 

16 
role_path "/var/chef/roles" 33 u
mask 0022

17  34 
18 
validation_client_name "validator" 35 M
ixlib::Log::Formatter.show_time = false

70 Admin 01 w w w. a d m i n - m aga z i n e .co m


‑r http://s3.amazonaws.com/U
chef‑solo/bootstrap‑latest.tar.gz
The tool creates a couple of directo-
ries, corrects the configuration files,
and adds chef-client to the init
scripts. The latter ensures that the cli-
ent will talk to the server on booting
and execute any recipe changes that
have occurred in the meantime.
After this, the client has to register
with the server. To allow this to hap-
pen, copy the /​etc/​chef/​validation.
pem file from the server to the /​etc/​
chef/​directory client-side and then
restart the client manually:
sudo chef‑client
The client automatically creates a key,
which you need to add to the /​etc/​
chef/​client.pem file and which will
sign every transaction with the server
from this point on. Then you want
to delete the validation.pem file for
security reasons.
Librarian
Now that you have the server and the
client running, the next step is to cre-
ate a repository server-side for your
recipes: This is simply a hierarchy of
multiple, standardized (sub-)direc-
tories. Of course, you could create
them all manually, but the template
provided by Opscode does a quicker
job; you just need to download and
unpack:
wget http://github.com/opscode/U
chef‑repo/tarball/master
tar ‑zxf opscode‑chef‑repo‑U
123454567878.tar.gz
Because this cryptic number is dif-
ficult to remember in the daily grind,
you might want to rename the direc-
tory (incidentally, the number comes
from the versioning system and repre-
sents the Commit ID):
mv opscode‑chef‑repo‑123454567878 U
chef‑repo
cd chef‑repo
[Table 1] explains the directory hierar-
chy in chef-repo.
The recipes stored here are injected
into the server by a tool named knife.
w w w. a d m i n - m aga z i n e .co m
Chef Ma n ag e m e n t
Listing 2: SSL Certificates for the Chef Server
01 
server_ssl_req="/C=US/ST=Several/L=Locality/O=Example/OU=Operations/CN=chef.example.com/
emailAddress=ops@example.com"
02 
openssl genrsa 2048 > /etc/chef/validation.key
03 
openssl req ‑subj "${server_ssl_req}" ‑new ‑x509 ‑nodes ‑sha1 ‑days 3650 ‑key /etc/chef/validation.key
> /etc/chef/validation.crt
04 
cat /etc/chef/validation.key /etc/chef/validation.crt > /etc/chef/validation.pem
05 
openssl genrsa 2048 > /etc/chef/webui.key
06 
openssl req ‑subj "${server_ssl_req}" ‑new ‑x509 ‑nodes ‑sha1 ‑days 3650 ‑key /etc/chef/webui.key > /
etc/chef/webui.crt
07 
cat /etc/chef/webui.key /etc/chef/webui.crt > /etc/chef/webui.pem
To prepare a recipe for action, run the The target, upload_cookbook, is de-
command fined in the Rakefile provided by the
repository.
knife configure ‑i
and confirm the default responses by GUI Management
pressing Enter – except, enter your
own username when asked Your cli- The server now knows the emacs
ent user name?, and type . (dot) in cookbook, but the clients don’t. To
response to the Path to a chef reposi- change this, launch a browser and ac-
tory (or leave blank)? query. Knife cess the web front end with http://​
then registers a new client on the chefserver.example.com:4040. Chef
Chef server, creates the above-men- does not offer SSL encryption here. If
tioned certificate in /​
.chef/​ you prefer a more secure approach,
my-knife.pem, and finally creates the you could use Apache as a proxy.
/​.chef/​
knife.rb configuration file. In the form that then appears, log in
by typing the admin username [Figure
Convenience Food 2]. The matching password is stored
in the web_ui_admin_default_password
Multiple recipes with the same objec- line of the /​etc/​chef/​server.rb file.
tive can be grouped in a cookbook.
For example, the mysql cookbook con- Listing 3: ~/​c hef.json for the Server
tains all the recipes required to install 01 
{
and set up the free database. For an 02  "bootstrap": {
03  "chef": {
initial test, it is a good idea to look for
04  "url_type": "http",
a simple cookbook [5].
05  "init_style": "runit",
In the section that follows, I will use
06  "path": "/srv/chef",
the cookbook for emacs from the ap- 07  "serve_path": "/srv/chef",
plications group as an example. In 08  "server_fqdn": "chefserver.example.com",
this example, I’ll use the package 09  "webui_enabled": true
manager to install the popular Emacs 10  }
text editor. 11  },
12  "run_list": [ "recipe[bootstrap::server]" ]
After downloading the Cookbook ar-
13 
}
chive, unpack it in the cookbooks sub-
directory, then introduce the server to
the new recipes: Listing 4: ~/​c hef.json for the Client
01 
{
rake upload_cookbooks 02  "bootstrap": {
03  "chef": {
The rake command automatically 04  "url_type": "http",
calls knife with the correct param- 05  "init_style": "runit",
eters, and knife then uploads all the 06  "path": "/srv/chef",
07  "serve_path": "/srv/chef",
cookbooks from the corresponding di-
08  "server_fqdn": "chefserver.example.com"
rectory. To upload a single cookbook 09  }
to the server, do this: 10  },
11  "run_list": [ "recipe[bootstrap::client]" ]
12 
}
rake upload_cookbook[emacs]
Admin 01 71
 Ma n ag e m e n t Chef

at the same time and get in each

other’s way.

Role-Out
To group multiple cookbooks in a
role, create a new file below Roles in
the repository, say, beispiel.rb, with
the following content:

name "beispiel"
description "Example of a role"
run_list("recipe[emacs]", U
"recipe[zsh]", "recipe[git]")

This groups the emacs, zsh and git

recipes under the beispiel role name.
Then send the role to the server like
this:

Figure 2: The web front end login page: the default password specified on the right is incorrect.
Changing the slightly cryptic default
after logging in the first time is a good
idea.
Now go to the Nodes menu. When
you get there, click the client name,
change to the Edit tab, and finally
drag the recipe you want to use from
Available Recipes and drop it into
the Run List (the recipe will slot into
the top position in the list). In the
example, you would now see emacs
at the top [Figure 3]. To store this as-
signment, press the Save Node button
bottom left on the page.
Client-side now, manually launch the
chef-client tool:
sudo chef‑client
This command line immediately
opens a server connection, picks
up the recipes assigned to the client
(only emacs for the time being) and
executes the recipes [Figure 5]. To al-
low this to happen on a regular basis,
you should run the client at regular
intervals as a daemon:
chef‑client ‑i 3600 ‑s 600
In this example, the client contacts
the server every 3,600 seconds. The
-s parameter lets you vary the period
slightly. If you don’t set this, all of
your clients might query the server
rake roles
In the web front end, you can assign
roles to a node just like cookbooks
using drag and drop.
Freshly Stirred
Ready-made recipes and cookbooks
off the Internet will only cover stan-
dard application cases. For special
cases, or individual configurations,
you will typically need to create your
own cookbook.
The following, extremely simple
example, creates a text file on the cli-
ent called /​tmp/​thoughts.txt that is
based on the quick_start cookbook
[6], and it adds a sentence that is

G Figure 4: The Status tab lists all the nodes with their last contact attempts
and recipe assignments (following Run List).

F Figure 3: Using drag and drop to assign recipes to a node. In this example, the
client runs the beispiel recipe first, followed by emacs.

72 Admin 01 w w w. a d m i n - m aga z i n e .co m


Figure 5: The client has picked up its assigned recipe from the server and executed it. This puts a pre-
configured version of Emacs on its disk.
generated dynamically in part. Start
by creating a new cookbook called
beispiel in chef-repo:
rake new_cookbook COOKBOOK=beispiel
The command creates a beispiel
folder below cookbooks, populates it
with the required subdirectories, then
creates an empty recipe named de-
fault.rb.
Before you start filling this with con-
tent, first create a template for the file
you want to create, /​tmp/​thoughts.
txt. This will later contain the sen-
tence
Thought for the day:
and the recipe will append an inge-
nious thought on a daily basis. The
complete template is thus:
Thought for the day: <%= @thought %>
The recipe will replace the wildcard
with text later on. The new template
needs to be in templates/​default/​;
you can save it as thoughts.txt.erb.
Most recipes use templates like this,
or, to quote the developers: “We love
templates.”
Hand Mixer
Now, compose a matching recipe that
picks up the template and uses it to
generate the /​tmp/​thoughts.txt file.
To save work here, you can extend
the existing, but empty, default.rb
recipe in the recipes subdirectory.
The recipe for this example looks like:
template "/tmp/thoughts.txt" do
source "thoughts.txt.erb"
variables :thought => U
node[:thought]
action :create
end
w w w. a d m i n - m aga z i n e .co m
This should be fairly self-explanatory
for Ruby aficionados: It creates the
/​
tmp/​thoughts.txt file from the
thoughts.txt.erb template and then
replaces the wildcard with the con-
tent of the thought variable. Now
you just need to think about what
thoughts to use here.
Spice
In this example, thought will be an at-
tribute. Attributes store node-specific
settings in a cookbook for recipes to
evaluate and use. A typical attribute
would be, say, a command-line pa-
rameter for a program launched auto-
matically by a recipe. The attributes
are identical for each call to the recipe
and, thus, no more than constants
provided by the recipe author.
In contrast to genuine Ruby con-
stants, attributes can be modified via
the web interface (in the window
used to assign cookbooks to nodes).
A cookbook groups all of the attri-
butes in its attributes subdirectory.
For the example here, you need to
create a beispiel.rb file with the fol-
lowing content:
thought "Silence is golden ..."
Now you just need to register the new
cookbook with the server
rake upload_cookbooks
and assign it to one or multiple nodes
in the web front end. After running
chef-client, the /​tmp/​thoughts.txt
file should appear.
This recipe leaves much scope for
improvement. For example, you could
randomly choose the thought of the
day, which Ruby programmers should
handle easily. Because recipes are
full-fledged Ruby scripts, you can
Chef Ma n ag e m e n t
draw from the full scope of the lan-
guage and on RubyGems. In the case
of the latter, the recipe should first
check to see whether the Gem exists
on the client and, if not, install it.
The beispiel/​metadata.json file
stores metadata on the new cook-
book. Before you roll out the cook-
book in a production environment,
you might want to add some details.
As the file extension suggests, the file
uses the JSON format.
Conclusions
Chef is a complex piece of software,
and once you have it running and
have finished modifying or creating
your recipes, it does make the admin-
istrator’s life much easier – at least
on Linux systems. Unfortunately, the
learning curve is very hard going for
newcomers. The online documenta-
tion for Chef is fairly chaotic and
incomplete [7]. If you need to know
more about writing cookbooks, it
is a good idea to download prebuilt
examples and investigate them. The
cookbook for emacs shows you how
to use action :upgrade to install a
package for example.
Additionally, it is hard to find help or
how-tos, even on the web, if you have
a problem. Your best option here is
to post your questions on the mailing
list [8]. n
Info
[1] Installation guide: [http://​­wiki.​­opscode.​
­com/​­display/​­chef/​­Installation]
[2] RabbitMQ:
[http://​­www.​­rabbitmq.​­com/​­server.​­html]
[3] openSUSE build service for Rab-
bitMQ: [http://​­software.​­opensuse.​
­org/​­search?​­q=rabbitmq‑server&​
­baseproject=openSUSE%3A11.​­2]
[4] Information on full-text searches in Chef:
[http://​­wiki.​­opscode.​­com/​­display/​­chef/​
­Search]
[5] Repository with prebuilt cookbooks:
[http://​­cookbooks.​­opscode.​­com/]
[6] Cookbook quick start: [http://​­wiki.​­opscode.​
­com/​­display/​­chef/​­Cookbook+Quick+Start]
[7] Chef wiki:
[http://​­wiki.​­opscode.​­com/​­display/​­chef/]
[8] Chef mailing list: [http://​­lists.​­opscode.​
­com/​­sympa/​­lists/​­opensource/​­chef]
Admin 01 73
 Ma n ag e m e n t Sysinternals
System monitoring with Sysinternals
Health Check
© Denis Tevekov, 123RF.com
Administrators don’t need a massive arsenal of tools just to monitor a couple of
systems. With Microsoft’s free Sysinternals suite, admins can handle all sorts of
tasks. By Thomas Joos
The Sysinternal tools are free tools
from Microsoft that can help Win-
dows administrators handle many
tasks. This article introduces the
Sysinternal tools that are useful for
system monitoring. All of the tools
described here can be downloaded
free of charge from the Microsoft site
[1], either as individual downloads or
as part of the Sysinternals suite.
One advantage of the Sysinternals
utilities is that you don’t need to in-
stall them, so they can be launched
conveniently from a USB stick. When
launched for the first time, the pro-
grams display a license dialog; you
can suppress this dialog with the
/accepteula option, which can be
useful in scripting. Unfortunately, this
option does not work for all of the
Sysinternal tools.
The programs only run on a Windows
system as of Windows 2000 Server.
For this article, I used Windows
Server 2008 R2 and Windows 7.
Windows Server 2008 R2, Windows
Server 2008, Windows Vista, and
Windows 7 do not support access to
the hidden System $ shares such as
C$, or admin$ as easily as Windows
XP or Windows Server 2003; the com-
puters do not belong to a Windows
74 Admin 01
domain because the new operating
systems block access to administra-
tive shares by authentication of local
user accounts.
Some Sysinternal tools, such as
PSInfo.exe, require access to the
admin share and thus will not work
at first. To allow access, you must
enable local logins to administrative
shares in the Registry of standalone
computers. To do so, launch the Reg-
istry Editor by typing regedit, then
navigate to HKEY_LOCAL_MACHINE\SOFT‑
WARE\Microsoft\Windows\CurrentVer‑
sion\Policies\System. Create a new
Dword entry with the label LocalAc‑
countTokenFilterPolicy, set the value
to 1, then restart the computer.
LDAP Microscope
Insight for Active Directory, also
known as AdInsight, lets you monitor
the LDAP connections on a domain
controller in real time with a GUI.
The user interface is similar to the
Sysinternal tools Regmon and File-
mon. The tool investigates calls to
the wldap32.dll file, which most pro-
grams, including Exchange, use for
LDAP-based access to Active Direc-
tory per LDAP.
AdInsight lists all requests including
those that are blocked. This gives
administrators an easy option for ana-
lyzing authentication problems with
Active Directory-aware programs and
identifying clients and servers that set
up a connection to the domain con-
troller. AdInsight logs all requests is-
sued to domain controllers and stores
them as an HTML report or text file
for troubleshooting purposes. The
logfile contains the client request and
responses that the client received via
LDAP.
AdInsight also logs access to system
services (Figure 1). When a program
such as Exchange accesses the do-
main controller, the window fills with
information; then, you can right-click
to display details of the individual en-
tries, as well as filter the display via
the menu.
The display also includes the name
of the accessing user. Unfortunately,
AdInsight only lets you monitor lo-
cal access; over-the-wire diagnostics
via remote access are not supported.
However, AdInsight’s search function
does let you filter by process, error, or
request response. The tool selects the
response to let you perform specific
monitoring.
w w w. a d m i n - m aga z i n e .co m
 Sysinternals Ma n ag e m e n t

AdInsight also supports automated

deployment and offers a variety of
options for this. One useful automa-
tion feature is the ability to write the
log to a file without displaying the
events in the GUI. AdInsight runs on
Windows 2000 Server or newer and
includes a help file, which can be a
useful aid for tasks that require more
thorough analysis.

Filesystem, Registry,
Processes
The Process Monitor provides a
graphical user interface for monitor-
ing and color-tagging the filesystem, Figure 1: AdInsight helps you investigate LDAP access to domain controllers.
registry, and process/​thread activity
in real time. The tool combines two entries that contain the same string. to use and does not have an extended
programs standard to Sysinternals: Additionally, you can enable multiple learning curve.
Filemon and Regmon. With a click filters at the same time and save the Process Explorer takes things a step
of the button, you can enable and configuration. further than Process Monitor, in that
disable the individual monitoring For more efficient diagnostics, you Process Explorer lists all the processes
options and restrict your monitoring might want to save the logfile and in a window and includes more
of registry and filesystem access and then load it for analysis, applying ad- detailed information on the current
process calls to an area in which you ditional filters as needed. The Tools process, such as access to directories
are interested. menu gives you a selection of precon- (Figure 3).
Process Monitor is a valuable aid for figured views. In DLL View mode, the tool shows
monitoring stability and identifying Process Monitor can also monitor the which libraries are used and where
bottlenecks; it is capable of logging boot process on a server because it they originated. The process menu
all read and write access to the sys- launches at a very early stage. You contains a Restart entry that first kills
tem and its media. Additionally, it can redirect all the output to a file. a process and then restarts it. Also,
displays comprehensive data on any If Windows fails to boot, analysis of you can temporarily stop individual
process or thread that is launched the output file gives you a fast way to threads and highlight processes in dif-
or terminated. Also, you can moni- identify the issue. Just like all Sysin- ferent colors. Process Explorer nests
tor any TCP/​IP connections that are ternal tools, Process Monitor is easy in the taskbar, which provides an at-
opened, as well as UDP traffic. Note
that Process Monitor does not save
the content of the TCP packets or
payload data; it is not specifically de-
signed for network monitoring. If that
is what you need, you might prefer a
tool such as Wireshark [2].
The Process Monitor tool is also ex-
tremely useful for troubleshooting lo-
cal connection and privilege problems
(Figure 2). If needed, it can display
additional information on active
processes (e.g., their DLL files or the
parameters set when the process was
launched).
The filter function allows you to
reduce the volume of data output
generated by Process Monitor. For
example, it lets you hide any pro-
cesses with a specific string in their
names, without filtering out registry Figure 2: Monitoring processes, the registry, and filesystem access with Process Monitor.

w w w. a d m i n - m aga z i n e .co m Admin 01 75

 Ma n ag e m e n t Sysinternals

a-glance overview of the current CPU

load and disk utilization.

Logging into the Domain

The LogonSessions tool lists all the
active sessions on a computer at the
command line (Figure 4). If you run
this command without setting op-
tions, the prompt buffer size might
not be sufficient to display all of the
information. In this case, you can use
the logonsessions | more command
or modify the command-line prompt
properties to increase the buffer size.
Alternatively, you can redirect the
output to a file by specifying > logon.
txt. The ‑p option tells LogonSes-
sions to display the active processes
in the individual sessions for login
users, allowing you to monitor who
is logged in to a terminal server and
which applications the users are run-
ning. The tool is also really useful in
Active Directory environments.
Monitoring Local Logins
Just like LogonSessions, PsLoggedOn
is a tool for monitoring logged in
users. If you type psloggedon at the
command-line prompt, the tool will
display all the logged in users on the
local system with their login times
(Figure 5). Also, you can see who is
accessing a share on the server. If you
launch the tool with the username as
an argument, it will investigate all the
computers in the network environ-
ment or domain and show you where
the user is logged in. Unfortunately,
this is not an entirely error-free pro-
cess in Windows DOS. Although you
can identify domain users logged in
Figure 3: Monitoring and controlling processes with Process Explorer.
to the local computer, you can’t actu-
ally find out where else the user is
logged on in the domain. If you use
psloggedon \\computername, you can
also display domain users logged in
remotely.
To access a computer on the network,
the user account with which you
launch PsLoggedOn must have ad-
ministrative privileges on the remote
machine. Although you can check
who has an active session on a com-
puter without add-on tools, with net
session, for example, you can only
monitor local logins, not network log-
ins. The tool accesses the HKEY_USERS
registry key for this check (Windows
creates a separate key for each logged
in user by default).
The ‑l option lists locally logged in
users only; ‑x leaves out the login
times. The PsLoggedOn tool is useful
when you need to investigate unau-
thorized access to your network and
see the computers on which a user is
logged in.
Psinfo is another tool in the collec-
tion that lets you retrieve a variety of
information for the local system, such
as the operating system, the CPU, the
build number, and whether the com-
puter is a member server or a domain
controller. Calling psinfo /? lists
more options. For example, psinfo ‑s
lists the software packages installed
on the machine. Although Microsoft
provides programs such as system‑
info.exe and msinfo32.exe with
Windows, Psinfo can also retrieve
information from remote computers
across the wire.
Event Lister
The PsLogList utility is a command-
line tool that retrieves the event

G Figure 5: Listing logged in users with PsLoggedOn.

F Figure 4: Logged in users and services.

76 Admin 01 w w w. a d m i n - m aga z i n e .co m

 Sysinternals Ma n ag e m e n t

Table 1: Selected PsLogList Options each file, you are told whether the
user has read (R), write (W), or both
Name Function
(RW) types of access. If you use |
File Runs the command on all the computers listed in the file. Each com-
more to redirect the output from this
puter needs a separate column in the text file.
command, the display will pause on
‑a dd/mm/yy Lists the entries after the specified date.
each page, and you can continue by
‑b dd/mm/yy Lists entries before the specified date.
pressing any key. In a similar fashion,
‑c Deletes the event logs after displaying them in PsLogList, which is
>file.txt redirects the output to a
useful in the case of batch-controlled retrieval.
file. With this approach, you do not
‑d n Displays the entries for the past n days.
see any command-line output.
‑e id1, id3, ... Filters entries with defined IDs.
The accesschk user ‑cw * command
‑f Filters entries with specific types (e.g., ‑f w filters warnings). You
can use arbitrary strings here.
shows you the Windows services to
which a group or user has write ac-
‑h n Lists entries for the past n hours.
cess. If you want to see a user’s ac-
‑i id1, id3, ... Shows entries with IDs defined in a comma-separated list of IDs.
cess privileges for a specific registry
‑l event_log_file Stores entries for the defined event log.
key, the accesschk ‑kns contoso\tami
‑m n Lists entries for the past n minutes.
hklm\software command is your best
‑n n Only shows the n latest defined entries. bet. The AccessChk tool is excellent
for checking computers for vulnera-
display from various computers and is the only account with privileges on bilities, and it also supports scripting.
then displays and compares events. If all the PCs and servers in the domain. AccessEnum gives you a GUI that
you run the tool without any options, In networks without a domain, you presents a full directory tree of user
it will output the entries in the local can use the administrator account for privileges. In other words, AccessE-
system event log. The program has the login; however, you need to set num is the graphical front end for Ac-
numerous options that give you vari- the password to be the same on all cessChk. The download contains both
ous ways of comparing the event logs the computers you want to monitor. files because AccessEnum relies on
that you retrieve. ShareEnum shows you not only the the AccessChk program for perform-
Table 1 lists some of PsLogList’s com- shares but also the local paths for the ing scans. In the GUI, you can select
mand-line options. By default, the shares on the computer. a directory and scan it for privilege
tool uses the system event log; you The Refresh button tells ShareEnum assignments. The tools also show de-
can select the event log by entering to launch a scan. If you want to scan nied privileges.
the first letter or by entering an ab- an individual computer, enter the
breviation, such as sec for security. same IP address as the start and end Conclusions
address of the IP range. The tool will
Monitoring Shares show you all of the shares on the net- The free command-line and GUI tools
work in a single window and list the in the Sysinternals suite are a feature-
The ShareEnum tool lets you monitor access privileges to boot. If you only rich addition to any Windows ad-
shares and their security settings by want to see local privileges, Sysinter- ministrator’s toolbox. The individual
scanning either an IP range or all the nals gives you a choice of two tools: programs are useful for monitoring
PCs and servers in a domain (or all AccessChk and AccessEnum. processes, users, and network con-
the domains in a network) for shares AccessChk outputs an exhaustive list nections, and tools like AdInsight
(Figure 6). To display all of this in- of a user’s rights at file, service, or are indispensable aids if you need to
formation reliably, you need to log in registry level at the command line, troubleshoot Active Directory logins.n
as the domain administrator, which giving you a quick overview of how
access privileges
are defined for a Info
specific user. [1] Sysinternals tools:
To see the privi- [http://​­www.​­sysinternals.​­com]
leges for the ad- [2] Wireshark: [http://​­www.​­wireshark.​­org]
ministrator in
the C:\Windows\ The Author
System32 direc- Thomas Joos is a freelance IT consultant who
tory, you would has worked in the IT industry for more than 20
type accesschk years. Among his many projects, Joos writes
administra‑ practical guides and articles on Windows and
Figure 6: ShareEnum gives you a neat list of all the shares and assigned tor c:\windows\ other Microsoft topics. You can meet him online
privileges on the network. system32. For at [http://​­thomasjoos.​­spaces.​­live.​­com].

w w w. a d m i n - m aga z i n e .co m Admin 01 77

 N u ts a n d B o lts PAM and Hardware

© An
a Vas
ileva
, Foto
lia.co
m

Flexible user authentication with PAM

Turnkey Solution
PAM is a very powerful framework for handling software- and hardware-based user authentication, giving ad-
ministrators a choice of implementation methods. By Thorsten Scherf

Hardware innovations are daily busi-
ness in user account authentication.
Pluggable Authentication Modules
(PAM) help transparently integrate
these new devices into a system.
This gives experienced administra-
tors the option of offering a variety
of different authentication methods
to their users while providing scope
for controlling the total user session
workflow.
Old School
User logins on Linux systems are
traditionally handled by the /etc/
passwd and /etc/shadow files. When
a user runs the login command to
log in to the system with a name and
password, the program creates a cryp-
tographic checksum of the password
and compares the results with the
checksum stored for this user in the
/etc/shadow file. If the checksums
match, the user is authenticated; if
not, the login will fail.
This approach doesn’t scale well. In
larger environments, user credentials
are typically stored centrally on an
LDAP server, for example. In this
case, the login program doesn’t re-
trieve the password checksum from
the /etc/shadow file but from a direc-
tory service. This task can be simpli-
fied by deploying PAM [1].
Modular Authentication
Originally developed in the mid-
1990s by Sun Microsystems, PAM is
available on most Unix-style systems
today. PAM offloads the whole au-
thentication process from the appli-
cation itself to a central framework
comprising an extensive collection
of modules (Figure 1). Each of these
modules handles a specific task;
however, the application only gets to

78 Admin 01 w w w. a d m i n - m aga z i n e .co m


Figure 1: PAM provides a centralized user management framework for the
application.
know whether or not the user logged
in successfully. In other words, it is
PAM’s job to find a suitable method
for authenticating the user. The PAM
framework defines what this method
looks like, and the application re-
mains blissfully unaware of it.
PAM can use various authentication
methods. Besides popular network-
based methods like LDAP, NIS, or
Winbind, PAM can use more recent
libraries to access a variety of hard-
ware devices, thus supporting logins
based on smartcards or the user’s
digital fingerprint. One-time password
systems, such as S/​Key or SecurID,
are also supported by PAM, and some
methods even require a specific Blue-
tooth device to log in the user.
The way PAM works is fairly simple.
Each PAM-aware application (the ap-
plication must be linked against the
libpam library) has a separate config-
uration file in the /etc/pam.d/ folder.
The file will typically be named after
the application itself – login, for
example. Within the file, modules dis-
tribute PAM tasks among themselves.
Numerous libraries are available in
each group, and they handle a variety
of tasks within the group (Figure 2).
Control flags let you manage PAM’s
behavior in case of error – for ex-
ample, if a user fails to provide the
correct password or if the system is
unable to verify a fingerprint.
Fingerprints
More recent PAM libraries allow ad-
ministrators to authenticate users by
means of smartcards, USB tokens, or
biometric features. State-of-the-art
w w w. a d m i n - m aga z i n e .co m
PAM and Hardware
Figure 2: A classic PAM configuration file contains modules and libraries that the
administrator can use to customize PAM.
notebooks often include a fingerprint
reader that allows the owner to use a
digital fingerprint when logging into
the system. The PAM ThinkFinger li-
brary [2] provides the necessary sup-
port. According to the documentation,
the module will support the UPEK/​
SGS Thomson Microelectronics fin-
gerprint reader used by most recent
Lenovo notebooks and many external
devices.
Most major Linux distributions offer
prebuilt packages for the PAM librar-
ies. You can use your distribution’s
package manager to install the soft-
ware from the repositories. To install
the required packages on your hard
disk, you would give the
yum install thinkfinger
command on a Fedora system and
apt‑get install thinkfinger‑tools U
libpam‑thinkfinger
on Ubuntu Hardy. Gentoo admins can
issue a compact command:
emerge sys‑auth/thinkfinger
If you’re using openSUSE, you’ll need
the libthinkfinger and pam_thinkfin-
ger packages, the repository versions
of which are not up to date.
You might prefer a manual install
with the typical ./configure, make,
make install steps and files from the
current source code archive. Debian
users on Lenny will need to access
the Experimental repository and then
type
aptitude install libthinkfinger0 U
libpam‑thinkfinger thinkfinger‑tools
for the install.
N u ts a n d B o lts
Before you modify the existing PAM
configuration, you might want to test
the device itself. To do so, scan a fin-
gerprint by giving the
tf‑tool ‑‑acquire
command (Figure 3). Then you can
use
tf‑tool ‑‑verify
to verify the results. You might see
a Fingerprint does *not* match mes-
sage at this point; initial attempts can
be fairly inaccurate because you will
need to familiarize yourself with the
device.
If you drag your finger too quickly
or too slowly across the scanner, the
device could fail to identify the fin-
gerprint correctly. In this case, it will
output an error message and quit.
When you achieve reliable results
from fingerprint scans, you can delete
the temporary file with the test scan
in /tmp and create an individual file
for each user on the system that will
contain the user’s fingerprint. The
command is
tf‑tool ‑‑add‑user username
(Figure 4). Users must scan their fin-
gerprints three times for this to work.
If the fingerprint is identified correctly
each time, the tool will store it in a
separate file below /etc/pam_think‑
finger/.
Once everything is working, you can
begin the PAM configuration. Figure
2 shows a PAM configuration for
the login program that lists just one
authentication module: pam_unix. If
you want to authenticate against the
fingerprint scanner first, you need to
Admin 01 79
N u ts a n d B o lts PAM and Hardware
Figure 3: tf-tool gives you an option for testing your fingerprint scanner …
call the pam_thinkfinger PAM module
before pam_unix.
To prevent PAM from prompting us-
ers to enter their password despite
passing the fingerprint test, you need
add a sufficient control flag. This
tells PAM not to call any more librar-
ies once an authentication test has
succeeded and to return PAM_SUCESS
to the calling program – login in this
example. If the fingerprint-based login
fails, pam_unix is called as a last
resort and will prompt for the user’s
regular password.
Manually entering the PAM libraries
for all of your PAM-aware applica-
tions in every single PAM configura-
tion file would be fairly tedious. A
centralized PAM configuration file
gives you an alternative. On Fedora
or Red Hat, this file is named /etc/
Figure 5: On Fedora, system‑config‑authentication provides a ba‑
sic PAM configuration tool.
80 Admin 01
Figure 4: … and then creating a fingerprint for each user.
pam.d/system‑auth, although other
Linux distributions call it /etc/pam/
common‑auth. You can enter all the
libraries against which you want to
authenticate your users in the file
(Figure 5).
The include control flag then includes
the file in all your other PAM configu-
ration files. From now on, this makes
all the programs in the PAM libraries
listed by the centralized configuration
file available in the individual PAM
files, including the pam_thinkfinger
module.
USB Tokens
The pam_usb library supports an-
other hardware-based approach, in
which PAM checks to see whether a
specific USB device is plugged into
Figure 6: The USB device is identified by its properties. If the user tries to log in without the
device, it will not work.
the machine. If so, the user is logged
in; if not, access to the system is de-
nied. The plugged in device is identi-
fied by its unique serial number and
model and vendor names. Addition-
ally, a random number is stored on
the USB device and on the computer;
the number changes after each suc-
cessful login attempt.
When a user logs in, PAM checks
both the specified USB device proper-
ties and the random number. If the
number stored on the USB does not
match the number on the disk, the
login fails. This prevents attackers
from stealing the random number,
placing it on their own USB device,
and then modifying the properties of
their own device to access the sys-
tem. Because the random number on
the system changes after each login,
w w w. a d m i n - m aga z i n e .co m
November 13-19, 2010 • Ernest N. Morial Convention Center • New Orleans, Louisiana
Conference Dates: November 13-19, 2010 • Exhibition Dates: November 15-18, 2010

The Future of Discovery

Sponsors:
IEEE Computer Society
ACM SIGARCH

T h e I n t e r n a t i o n a l C o n f e re n c e fo r H i g h Pe r fo r m a n c e C o m p u t i n g , N e t wo r k i n g , S t o ra g e, a n d A n a l ys i s
 N u ts a n d B o lts PAM and Hardware
the stolen number will not match the
number on the system.
Gentoo and Debian Linux offer pre-
built packages of this PAM library. In
both cases, you can use the package
manager to install, as described for
pam_thinkfinger. Users on any other
Linux distribution can download the
current source code archive [3] and
run make; make install to compile
the required files and install them on
the local system. Then you need to
connect any USB device – it can be
a cellphone with an SD card if you
like – and store its properties in the
/etc/pamusb.conf file. The command
for this is
pamusb‑conf ‑‑add‑device USB‑device‑name
(Figure 6). The command
pamusb‑conf ‑‑add‑user user
lets you add more users to the con-
figuration and generates a matching
random number. The number for
each user is stored both on the USB
device and on the system. Also, the
tool adds each user to the XML-based
/etc/pamusb.conf configuration file.
You can use the file to define ac-
tions for each user; these actions will
run when the USB is plugged in or
unplugged. For example, the entry
in Listing 1 of the configuration file
Listing 1: Configuration File for pam_usb
01 
<user id="tscherf">
02  <device>
03  /dev/sdb1
04  </device>
05  <
agent event="lock">gnome‑screensaver‑command
‑‑lock</agent>
06  <
agent event="unlock">gnome‑screensaver‑command
‑‑deactivate</agent>
07 
</user>
Listing 2: USB Device-Based Authentication
[tscherf@tiffy ~]$ id ‑u
500
[tscherf@tiffy ~]$ su ‑
* pam_usb v0.4.2
* Authentication request for user "root" (su‑l)
* Device "/dev/sdb1" is connected (good).
* Performing one time pad verification...
* Verification match, updating one time pads...
* Access granted.
[root@tiffy ~]# id ‑u
0
82 Admin 01
automatically blocks the screen if the
USB device is unplugged: Then You
need to add the pam_usb PAM library
to the corresponding PAM configura-
tion file, /etc/pam.d/system‑auth or
/etc/pam.d/common‑auth. If you use
the sufficient control flag, users can
log in to the system by plugging in
the USB device, assuming the random
number for the user matches on both
devices (Listing 2).
To enhance security, you can replace
the sufficient control flag with re‑
quired. This setting first looks for the
USB device, but even if the device is
identified correctly, PAM still prompts
the user for a password in the next
stage of the login process. Both of
these tests have to complete success-
fully for the user to log in.
All of the hardware-based login
methods I have looked at thus far
are easily set up, but they all have
vulnerabilities, and it is easy to fake
fingerprints. Also, USB sticks can be
stolen, thereby putting an end to any
security they offered. If you take your
security seriously, you will probably
want to use two-factor authentication.
This method inevitably involves using
chip cards with readers or USB tokens
with one-time passwords and PINs.
Yubikey
A small company from Sweden,
Yubico [4], recently started selling
Yubikeys (Figure 7), which are small
USB tokens that emulate a regular
USB keyboard. The key has a button
on top which, when pressed, tells the
token to send a one-time password
(OTP) to the active application, such
as a login prompt on an SSH server
or the login window of a web service.
The OTP is verified in real time by a
Yubico authentication server. Because
the software was released under an
open source license, you could theo-
retically set up your own authentica-
tion server on your LAN. This would
remove the need for an Internet con-
nection.
The way the token works is quite
simple. In contrast to popular RSA to-
kens, Yubikey doesn’t need a battery
because the OTP is not generated on
the fly; instead, one-time passwords
are defined in advance. The pass-
words are stored on the token and
in a database on the authentication
server.
When you press the Yubikey button,
the key sends one of these OTPs to
the active application, which then
uses an API to access the server and
verify the password. If this fails (Un-
known Key) or if the password has
already been used (Replayed Key), an
error message is output and the login
fails. If the server identifies the key
as valid, it sets usage‑count to 1 and
the user is authenticated. The user
cannot login with this key anymore
times.
Because of the simple API, more
and more applications are relying
on authentication against the Yubico
server. One example is the plugin for
the popular WordPress blog, which
allows users with a Yubikey to log in
to the blog. A project from Google’s
Summer of Code produced a PAM
module that supports logging in to an
SSH server [5].
Instead of typing your user password
at the login prompt, you simply press
the button on the Yubikey to send a
44-character, modhex-encoded pass-
word string to the SSH server. The
server then verifies the string by que-
rying the Yubico server. The first 12
characters uniquely identify the user
on the Yubikey server; the remaining
32 characters represent the one-time
password.
You can define a central file on the
SSH server to specify users permitted
to log in by producing a Yubikey. To
do so, first create a /etc/yubikey‑us‑
ers.txt file with a username, a colon
separator, and the matching Yubikey
ID (i.e., the first 12 characters of the
user’s OTP) for each user. Alterna-
tively, users can create a file (~/.
yubico/authorized_yubikeys) with
the same information in their home
directory.
You need to configure PAM to verify
the OTP against the Yubico server. To
do so, add a line for the Yubikey to
your /etc/pam.d/sshd file (Listing 3).
The configuration shown in Listing
3 runs this authentication in addition
w w w. a d m i n - m aga z i n e .co m
 PAM and Hardware N u ts a n d B o lts

to the regular, system‑auth-driven
authentication method. But if you
replace the required flag with suffi‑
cient, there is no need for the user to
log in after the Yubikey OTP has been
validated. Unfortunately, the Yubikey
is not protected by an additional PIN,
and the system is vulnerable if the
token is stolen. An unauthorized user
in possession of a token would be
able to spoof a third party’s identity.
The developers are working on add-
ing PIN protection for OTPs, and an
unofficial patch is already available.
X.509 Certificates and PAM
Classic two-factor authentication
typically relies on chip cards. The
cards typically contain a certificate
protected by a PIN. The PAM pam_
pkcs11 library allows users to log in
to the system via an X.509 certificate.
The certificate contains a private/​
public key pair. Both can be stored on
a suitable chip card, with the private
key protected by an additional PIN
to prevent identity spoofing simply
by stealing a chip card. To log in,
you need both the chip card and the
matching PIN. If the PIN is unknown,
the login fails.
The details of the login process are as
follows: The user inserts the chip card
into the reader and enters the PIN.
The system searches for the certificate
on the card. If the certificate is valid,
the user is mapped onto the system.
The mapping process can retrieve a
variety of information from the certifi-
cate, typically the Common Name or
the UID stored in the certificate.
To make sure the user really is who
they claim to be, the system generates
a random 128-bit number. A function
on the chip card then encrypts the
number using the private key, which
is also stored on the card. The user
needs to enter the right PIN to be able
to access the private key. The system
then uses the freely available public
key to decrypt the encrypted number.
If the results match the random num-
ber, the user is correctly authenticated
because the two keys match.
The hardware required for this setup
is a chip card with a matching reader
– for example, the Gemalto e-Gate or
SCR USB device by SCM. You can use
any Java Card 2.1.1 or Global Plat-
form 2.0.1-compatible token: Gemalto
Cyberflex tokens are widely available.
Various software solutions are also
available: The approach described in
this article relies on the pcsc-lite and
pcsc-lite-libs packages for accessing
the reader.
Public Key Infrastructure
It makes sense to use X.509 certifi-
cates, but only if you have a complete
In this example, I’ll use Dogtag [6]
from the Fedora project as a PKI solu-
tion. Users with other distributions
might prefer OpenSC [7]. The PAM
library is the same for both variants,
pam_pkcs11.
Dogtag consists of various compo-
nents. For this setup, you’ll also need
a Certificate Authority (CA) to create
the X.509 certificates. Online Certifi-
cate Status Protocol (OCSP) is used
for online validation of the certificates
on the chip cards. For offline valida-
tion, you just need the latest version
of the Certificate Revocation List
(CRL) on the client system. Of course,
you also need a way of moving the
user certificate from the certificate au-
thority to the chip card. You can use
the Enterprise Security Client (ESC)
to open a connection to another PKI
component, the Token Processing Sys-
tem, for this.
Assuming correct authentication, the
user certificate is then copied to the
chip card in the enrollment process.
The ESC tool then gives the user a
convenient approach to managing the
card. If the user needs to request a
new certificate from the CA or needs
a new PIN for the private key on the
card, it’s no problem with ESC.
If you use OpenSC to manage your
Listing 3: PAM Configuration for a Yubikey
auth required pam_yubico.so authfile=/etc/yubikey‑users.

with the public key and private key Public Key Infrastructure (PKI) set up. txt

auth include system‑auth

account required pam_nologin.so

account include system‑auth

password include system‑auth

session required pam_selinux.so close

session required pam_loginuid.so

session required pam_selinux.so open env_params

session optional pam_keyinit.so force revoke

session include password‑auth

Listing 4: Configuration File for pam_pkcs11

01 
pkcs11_module coolkey {
02  module = libcoolkeypk11.so;
03  description = "Cool Key"
04  slot_num = 0;
05  ca_dir = /etc/pam_pkcs11/cacerts;
06  nss_dir = /etc/pki/nssdb;
07  crl_dir = /etc/pam_pkcs11/crls;
08  crl_policy = auto;
Figure 7: USB keyboard emulation means that the Yubikey for one-time passwords doesn’t need special
09 
}
drivers. The token works with the press of a button.

w w w. a d m i n - m aga z i n e .co m Admin 01 83

N u ts a n d B o lts PAM and Hardware
Figure 8: The Security Client gives you an easy option for managing chip cards.
chip card, you can transfer a prebuilt
PKCS#12 file [8] to the card using:
pkcs15‑init U
‑‑store‑private‑key tscherf.p12 U
‑‑format pkcs12 U
‑‑auth‑id 01
The PKCS#12 file contains both the
public key and private key. If you
have a user certificate from a public
certification authority like CACert [9],
you can use your browser’s certificate
management facility to export the cer-
tificate to a file and then transfer it to
the chip card as described.
If you don’t have a certificate, you
can create a request and send it to
the appropriate certificate authority.
Once the authority has verified your
request, it will return a certificate to
you.
For both approaches, you can use the
/etc/pam_pkcs11/pam_pkcs11.conf file
to define the driver for access to the
chip card. The driver can be modified
in the configuration file, as shown in
Listing 4.
Here, you must specify the correct
paths to the local CRL and CA certifi-
cate repository. The CRL database is
necessary to check that the certificate
on a user’s chip card is still valid and
has not been revoked by the certifi-
cate authority. You need the certifi-
cate for the certificate authority that
issued the user’s certificate from the
CA certificate repository. This makes
84 Admin 01
it possible to validate the user’s au-
thenticity.
Certificates for
Thunderbird and Co.
Applications that rely on Network
Security Services (NSS) for signing or
encrypting email with S/​MIME, such
as Thunderbird, use a file in the nss_
dir as the CA database; applications
based on the OpenSSL libraries use
the database in the ca_dir directory.
The certutil tool can import the
CA certificate into the NSS database;
OpenSSL-based certificates can sim-
ply be appended to the existing file.
Finally, you can define the mappings
between user certificates and Linux
users in the pam_pkcs11 configuration
file. Various mapping tools are avail-
able for this, specified as follows:
use_mappers = cn, uid
Next, you still need to add the PAM
pam_pkcs11 library to the correct
PAM configuration file – that is, /etc/
pam.d/login or /etc/pam.d/gdm. You
can edit the file manually or use the
system‑config‑authentication tool
referred to previously.
When you insert the chip card into
the reader and launch the ESC tool,
you should be able to see the certifi-
cate (Figure 8). If you now attempt
to log in via the console or a new
GDM session, the authentication pro-
cess should be completely transpar-
ent. The Thunderbird email program
can use the card to sign and encrypt
email; Firefox can use the certificate
for client-side authentication against
a web server. The reward for all this
configuration is a large choice of de-
ployment scenarios. The ESC Guide
[10] has a more detailed description
of the tool and its configuration.
Conclusions
PAM is a very powerful framework
for handling authentication. As
you can see from the PAM libraries
introduced in this article, the func-
tional scope is not just restricted to
authenticating users but also covers
tasks such as authorization, password
management, and session manage-
ment. Administrators who take the
time to familiarize themselves with
configuring PAM, which isn’t always
trivial, will be rewarded with a feast
of feature-rich and flexible options for
password- and hardware-based au-
thentication and authorization. n
Info
[1] Linux PAM: [http://​­www.​­kernel.​­org/​­pub/​
­linux/​­libs/​­pam/]
[2] PAM ThinkFinger:
[http://​­thinkfinger.​­sourceforge.​­net]
[3] pam_usb:
[http://​­downloads.​­sourceforge.​­net/​
­pamusb/​­pam_usb‑0.​­4.​­2.​­tar.​­gz?​­download]
[4] Yubico website: [http://​­www.​­yubico.​­com/​
­products/​­yubikey/]
[5] SSH server for Yubikey: [http://​­code.​
­google.​­com/​­p/​­yubico‑pam/​­downloads/]
[6] Dogtag PKI: [http://​­pki‑svn.​­fedora.​­redhat.​
­com/​­wiki/​­PKI_Main_Page]
[7] OpenSC: [http://​­www.​­opensc‑project.​­org]
[8] PKCS specifications:
[http://​­en.​­wikipedia.​­org/​­wiki/​­PKCS]
[9] CACert certificate authority:
[http://​­cacert.​­com]
[10] ESC Guide: [http://​­directory.​
­fedoraproject.​­org/​­wiki/​­ESC_Guide]
The Author
Thorsten Scherf is a Senior Consultant for Red
Hat EMEA. You can meet him as a speaker at
conferences. He is also a keen marathon runner
whenever time permits.
w w w. a d m i n - m aga z i n e .co m
ON NEWSSTANDS NOW !
UBUNTU USER MAGAZINE
Includes a comprehensive Discovery
Guide to help new users install,
configure, and explore clu d e s f r ee Ubuntu
Also in
Ubuntu! x” DVD!
“Lucid Lyn
Find out more on
www.ubuntu-user.com
 n u tS A n d b o ltS modSecurity
Protecting web servers with ModSecurity
Apache Protector
© KrishnaKumar Sivaraman, 123RF.com
even securely configured and patched web servers can be compromised
because of vulnerabilities in a web application. modSecurity is an Apache
extension that acts as a web application firewall to protect the web server
against attacks. by Sebastian wolfgarten
Security issues on the web are no
longer typically a result of poor con-
figuration or the lack of up-to-date
server software. Tomcat, Apache,
and even IIS have become extremely
mature over the past few years – so
much so that they don’t have any no-
ticeable vulnerabilities, although ex-
ceptions can always turn up to prove
the rule. Thus, hackers have turned
their attention to the web applications
and scripts running on the servers.
Increasingly complex user require-
ments are making web applications
more complex, too: Ajax, interaction
with external databases, back-end
interfaces, and directory services are
just part of the package for a modern
application. And, attack vectors grow
to match this development (see the
“Attacks on Web Servers” box).
Firewalls for the Web
In contrast to legacy packet filters,
Web Application Firewalls (WAFs)
don’t inspect data in the network
or transport layer, but rather at the
HTTP protocol level (i.e., in OSI Layer
7) [1]. They actually speak HTTP. For
86 Admin 01
this to happen, these firewalls ana-
lyze incoming and outgoing client re-
quests and server responses to distin-
guish between benevolent and malev-
olent requests on the basis of rules.
If necessary, they can even launch
countermeasures; if configured to do
so, the software will also inspect en-
crypted HTTPS connections.
Accessories en Masse
Where classical network-based fire-
walls – I’m exaggerating slightly
here – either permit any or no HTTP
connections, WAFs target individual
HTTP connections based on their
content. ModSecurity is a high-
performance WAF for Apache and a
complex module for the Apache web
server. Originally developed by Ivan
Ristic, Breach Security handles its dis-
tribution and development [2].
Two variants of the software are avail-
able: the open source variant released
under the GPLv2, and a commercial
version with professional support,
pre-configured appliances, and man-
agement consoles. ModSecurity runs
on Linux, Solaris, FreeBSD, OpenBSD,
NetBSD, AIX, and Windows, with
the later versions only available for
Apache 2.x. This article discusses
version 2.5.10; the successor 2.5.11 is
merely a bugfix.
The software’s functional scope is
enormous but comprehensively docu-
mented [3]. It logs HTTP requests
and gives administrators unrestricted
access to the individual elements of
a request, such as the content of a
POST request. It also identifies at-
tacks in real time based on positive or
negative security models and detects
anomalies based on supplied patterns
for known vulnerabilities.
The powerful rules discover whether
credit cards are in the data stream or
use GeoIP to prevent access from cer-
tain regions. ModSecurity checks not
only incoming requests but also the
server’s outgoing responses. The soft-
ware can implement chroot environ-
ments. As a reverse proxy, it protects
web applications on other web serv-
ers, such as Tomcat or IIS.
Breach also provides a collection of
core rules that guarantees the basic
security of the web server. Com-
prehensive documentation, many
examples, and a mailing list provide
support for the user. This makes Mod-
Security a good choice for protecting
web servers and their applications
against vulnerabilities. But before you
can even consider tackling the highly
w w w. A d m i n - m AgA z i n e .co m

complex configuration, you first need
to install the third-party module.
Packages for any
Distribution
If you prefer not to build the pack-
age for Apache 2 yourself, you
can pick up pre-built packages for
Debian, RHEL, CentOS, Fedora,
FreeBSD, Gentoo, and Windows. The
manual install requires the Apache
mod_unique_id module, which is not
automatically provided by some dis-
tributions. You can use the
LoadModule security2_module U
modules/mod_security2.so
directive to integrate the module into
the Apache configuration file, httpd.
conf, if your distribution doesn’t
do this for you. After restarting, the
web server lists the module in its er-
ror_log.
Filter Rules
ModSecurity has a mass of configura-
tion options, but to understand how it
works, all you need is the basic con-
figuration. The SecRuleEngine option
activates the module’s filter mecha-
nism and allows it to process filter
rules. The settings here are On, Off,
and DetectionOnly, which tells the
module to monitor, but not become
actively involved with, the client-
server connection, even if individual
rules are configured to let it do so.
This setting is useful for testing the
module and your own rules.
To help you get started, or for de-
bugging, you also want to enable
the SecDebugLog option to define a
troubleshooting log (e.g., SecDebugLog
/var/log/httpd/modsec_debug.log).
Additionally, you can set the SecDe-
bugLogLevel parameter to specify
verbosity, on a scale of 0 to 9, which
issues comments on its own activities
and the way it processes user-defined
rules. Levels 4 or 5 are useful for fine
tuning or troubleshooting. In produc-
tion, set this parameter to 0.
The SecDefaultAction parameter de-
fines ModSecurity’s default behavior
for requests that match a filter rule
w w w. A d m i n - m AgA z i n e .co m
modSecurity n u tS A n d b o ltS
Attacks on web Servers
Compared with local applications, web applications are more vulnerable because they involve so
many different components – from the browser and the Internet infrastructure to the web server
and the back ends beyond. Vulnerabilities can occur anywhere, but the server is always at the
center of this environment.
If the web application doesn’t sufficiently validate user input and instead passes it to a database
running in the background, attackers could use SQL injection to inject their own commands into
the command chain. Thus, the attacker would be able to read, modify, or delete data and thereby
exert a major influence on the application.
If an application also allows attackers to store files on the web server and execute them over the
web, the intruder could set up a web shell. Because the server will execute the attacker’s files,
the attacker can run operating system commands on the web server and finally escalate their
privileges to interactive shell access. Although the architecture of a carefully configured Apache
will not give the attacker root access, this is often unnecessary to access sensitive data. And the
more services that are installed on the server, the more likely it is that the attacker will find one
that is vulnerable.
without a corresponding action. The filters and actions (see also the “In-
option also expects you to specify a sider Attacks” box). The software of-
processing phase for the standard ac- fers a number of prebuilt alternatives
tion, as listed in Table 1. ModSecurity here, including converting request
applies filter rules in five different parameters, running external scripts
phases of processing and respond- (e.g., to perform an antivirus scan),
ing to a client request [4]. In real or forwarding malevolent requests.
life, only phase 2, in which the client The latter is useful if you are trying to
requests that content (i.e., incoming investigate attacks or want to forward
data) is filtered, and phase 4, which them to a honeypot.
handles the server response (i.e., The basic configuration (Listing 1)
outgoing data), are relevant. Addi- also logs the contents of incoming
tionally, you can use the option to de- requests and the responses given in
fine one or multiple actions that the return. It enters the information in the
software will perform when a match auditlog as mentioned previously. The
occurs [5]. To log incoming client re- first directive, SecAuditEngine, en-
quests in the auditlog in phase 2 and ables this. The option for the second
respond to the access attempt with a directive defines whether the software
HTTP 403 (Forbidden) error message, stores the auditlog entries in a single
you would do this: file (Serial) or writes a file for each
transaction (Concurrent). Concur-
SecDefaultAction phase:2,log,U
auditlog,deny,status:403
rency is necessary if you intend to
deploy the ModSecurity Console add-
A massively negative default func- on product. Breach Security offers
tion like the default deny is highly the software for managing multiple
restrictive, but it does offer maximum instances, provided you don’t need to
protection if you additionally define monitor more than three servers. E
Table 1: modSecurity Processing Phases
Number Phase Designator Activities
1 Preview phase REQUEST_HEADERS Earliest possible filtering of incoming
requests before access control, authentica‑
tion, authorization, and MIME detection have
taken place Apache‑side
REQUEST_BODY
2 Client request Full access to the content of a client request
(normal case)
3 POST request RESPONSE_HEADERS Initial option for filtering server responses
4 Server response RESPONSE_BODY Full access to the content of the server re‑
sponse to an incoming client request
5 Logging LOGGING Access to all relevant information before it
is written to the Apache logfiles
Admin 01 87
 N u ts a n d b o lts ModSecurity

Figure 1: Once ModSecurity has been enabled, it will log suspicious activity at the detail level specified in SecAuditLogParts – in this case, an SQL injection attack.

The third instruction defines the au-
ditlog storage location relative to the
Apache installation path. Finally, the
SecAuditLogParts instruction defines
the information that ModSecurity logs
in the auditlog (Table 2). In this case,
this is the header and the content of
the request, along with the ModSecu-
rity reaction. The results are shown in
Figure 1.
After this preparation, you can add
the most important directive to your
configuration: SecRule. This defines
a filter rule and optionally an action
that the module will perform if it
discovers a match for the rule. If you
don’t define an action, the tool will
run the standard command defined in
the SecDefaultAction directive. Rules
always follow this pattern:
SecRule Variable Operator [Action]
The number of variables is huge, and
they cover every single element of the
client request (both for POST and for
GET), as well as the most important
server environment details [6]. Ad-
ditionally, you can use regular expres-
sions. For example, to investigate an
HTTP request to find out whether the
client requests the /etc/passwd string
in a GET method, you would use this
rule:
SecRule REQUEST_HEADERS: User‑Agent "nikto"
This example tells the web server to
refuse requests from the Nikto secu-
rity scanner [7]. If you want Mod-
Security to run a specific action for
a rule, you can overwrite the default
action:
SecRule REQUEST_HEADERS:User‑Agent "nikto" U
"phase:2,pass,msg:'Nikto‑Scan logged'"
This rule tells the module to write
a Nikto scan logged message to the
logfile when it detects the Nikto user
agent in a client request in phase 2.
The rule then overwrites the drop
default action, which is defined by
SecDefaultAction with the pass ac-
tion. This allows the client request
to pass. To test ModSecurity, Listing
1 gives you an overview of the basic
configuration discussed thus far.
Practice Session
After restarting Apache, a request
like http://www.example.com/index.
html?file=/etc/passwd would trigger
the sample rule in line 8. Then the
action defined in line 9 would block
the request. The client sees an HTTP
403 Forbidden error. At the same time,
lines 3 through 7 tell ModSecurity to
log the transaction in the auditlog and
send a detailed overview of the way
the client request was processed to
the debuglog. This means that your
Apache error_log file will contain a
note to the effect that a client request
was effectively blocked, as shown in
Listing 2. Once ModSecurity is work-
ing correctly, you can start adding
rules and modifying them for the web
applications you want to protect.
The Art of Detecting
Attacks
To provide effective protection against
a huge assortment of attacks, system
administrators need to set up a robust
ruleset for ModSecurity. To formulate
rules that protect you against SQL
injection, cross-site scripting, or local
and remote file inclusion attacks, for
example, you need in-depth knowl-
edge of how attacks on web servers
work.
Of course, not every administrator
has this knowledge or the time to
re-invent the wheel. To address this,
the Open Web Application Security
Project (OWASP) offers a predefined
ruleset for ModSecurity [8] that re-
lies on anomaly detection to protect
web servers against a number of

SecRule REQUEST_URI "/etc/passwd"

Table 2: SecAuditLogParts Arguments
If the request matches the rule, Mod- Abbreviation Description
Security runs the default deny action. A Header for the entry (mandatory)
To filter by browser type, you would B Request header
do this: C Request content; only available if content exists and ModSecurity is configured
to store it
Listing 1: Basic Configuration for ModSecurity D Reserved
E Temporary response content; only available if ModSecurity is configured for this
01 SecRuleEngine On
F Final response header after possible manipulation by ModSecurity; Apache itself
02 SecAuditEngine On
writes the Date and Server headers
03 SecAuditLogType Serial
G Reserved
04 SecAuditLog logs/audit.log
H Auditlog trailer equivalent to C, except where the request contains form data; in
05 SecAuditLogParts ABCFHZ
this case, the software constructs a suitable request that excludes file content to
06 SecDebugLog logs/debug.log simplify matches
07 SecDebugLogLevel 5 J Reserved
08 SecRule REQUEST_URI "/etc/passwd" K Line by line list of all matching rules in the order of their application
09 SecDefaultAction phase:2,log,auditlog,deny,status:403 Z End of entry (mandatory)

88 Admin 01 w w w. a d m i n - m aga z i n e .co m

 ModSecurity N u ts a n d b o lts

Insider Attacks Security will mean about a 5 percent

performance overhead for your web
A vulnerability in a web application can thus expose an Apache web server despite its being
server.
hardened and up to date. This is particularly dangerous if the server is deployed in a hosting
environment where many customers share the resources of a physical web server. Protection
mechanisms, such as virtualization or jails, will separate the individual instances; however, the Preventing Attacks on the
security of the whole system depends on its weakest link. With a sniffer, for example, attackers United Nations
can simply sniff password authentication conversations if they’re not encrypted. And, this gives
attackers an inside vector. In some situations, you can’t fix a
Scripting languages and frameworks like PHP or Ruby on Rails help web developers achieve web application vulnerability imme-
results quickly, but they often conceal dangers that occur when security is not given sufficient diately. Imagine a major online store
attention. More complex environments, such as Java, Tomcat, and JBoss, are not necessarily the discovering a security hole a week
answer because they hide many aspects from the developer.
before Christmas and needing several
days to fix the problem, meaning the
standard attacks, such as invalid cli- logging to see exactly what the mod- shop would be offline for that time.
ent requests, SQL injection, cross-site ule is doing. To do so, you need to The owner has to make a decision:
scripting, and email or command include the core rules in httpd.conf Live with the risk and keep the shop,
injection. This gives you basic, fairly as follows: including the vulnerability, online so
robust protection, which you can then you can benefit from lucrative pre-
Include conf/modsecurity/*.conf
modify to match your own applica- Include conf/modsecurity/ base_rules/*.conf
Christmas shopping, or protect the
tion environment as needed. company and its customers by taking
To install these core rules, download After restarting, your Apache web the website down and fixing the vul-
the package and unpack it in your server will have a solid suit of armor nerability.
Apache’s conf configuration direc- that responds with an HTTP 403 to ModSecurity offers a technical work-
tory. Then, move the rules, including any client requests classified as at- around in the form of virtual patch-
the base_rules subdirectory, to the tacks. ing that allows you to define one or
ModSecurity directory. You can look A word of caution: Web masters multiple rules that prevent the vulner-
at the configuration in the modsecu- should first test the core rules exten- ability from being exploited without
rity_crs_10_global_config.conf and sively in a lab environment before actually removing it.
modsecurity_crs_10_config.conf files letting them loose on a production The ModSecurity documentation re-
and modify it to suit your needs. The system. Otherwise, the danger of fers to a case that dates back to 2007,
rules are well documented. Also, you preventing legitimate user access is when attackers were trying to hack
might want to enable audit and debug possible. Also, remember that Mod- the United Nations website [9]. The
sub-page with talks by Secretary-Gen-
eral Ban Ki-moon [10] had a statID
parameter that exposed an SQL injec-
tion vulnerability (Figure 2).
If you discover a vulnerability of this
kind, you can temporarily define a
rule for ModSecurity that will prevent
hackers from exploiting the vulner-
ability even though it still exists.
Although this isn’t a good long-term
solution, it does prevent a disaster
until the web developers can remove
the vulnerability from the source code
on the page.
The following solution would work
for the UN bug: E

Listing 2: Rule Match

Sec
AuditLogType Serial [Wed Nov 04 05:39:19 2009]
[error] [client 192.168.209.1] ModSecurity: Access
denied with code 403 (phase 2). Pattern match
"/etc/passwd" at REQUEST_URI. [file "/usr/local/
httpd‑2.2.14/conf/httpd.conf"] [line "420"] [hostname
Figure 2: The website of UN Secretary-General Ban Ki-moon was affected by an input parameter variable
"www.example.com"] [uri "/index.html"] [unique_id
validation vulnerability. A ModSecurity virtual patching rule protected the website temporarily until the UN
"SvFZ138AAQEAAAc4AgQAAAAA"]
system administrator fixed the problem.

w w w. a d m i n - m aga z i n e .co m Admin 01 89

 N u ts a n d b o lts ModSecurity
<Location /apps/news/infocus/ U
sgspeeches/statments_full.asp>
SecRule &ARGS "!@eq 1"
SecRule ARGS_NAMES "!^statid$"
SecRule ARGS:statID "!^\d{1,3}$"
</Location>
Three lines embedded in an Apache
location container state that valid
user requests for the statements_
full.asp file are only allowed to
have one argument (first rule) called
statid (second rule) with numbers
of one to three digits (third rule) as
their parameters. Any requests that
do not follow this pattern are cleaned
up by the default action, as defined in
SecDefaultAction. This would effec-
tively prevent an attacker exploiting
the SQL injection vulnerability.
No Inside Information
ModSecurity also filters outgoing
data, especially server responses to
incoming requests. The PHP program-
ming language throws error messages
such as this:
Fatal error: Connecting to MySQL server
'dbserv.example.com' failed
Although you can disable PHP error
messages in responses, Google still
lists a bunch of websites where PHP
error messages reveal many juicy de-
tails of applications. This information
is very useful to an attacker, because
it can help them understand the inter-
nal workings and structure of a web
application and attack it in a more
targeted way. To tell ModSecurity to
catch PHP error messages and pre-
vent them from being sent to users,
you can define a rule like this:
SecRule RESPONSE_BODY "Fatal error:"
RESPONSE_BODY refers to the content
of the server response to the client,
Listing 3: GeoIP Access
01 LoadModule geoip_module modules/mod_geoip.so
02 LoadModule security2_module modules/mod_security2.so
03 GeoIPEnable On
04 GeoIPDBFile /usr/tmp/GeoLiteCity.dat
05 SecRuleEngine On
06 SecGeoLookupDb /usr/tmp/GeoLiteCity.dat
07 Sec
Rule REMOTE_ADDR "@geoLookup"
"chain,drop,msg:'Connection attempt from .CN!'"
08 SecRule GEO:COUNTRY_CODE "@streq CN" "t:none"
90 Admin 01
and although it is not particularly el-
egant, it does indicate what potential
you have. With a carefully crafted
regular expression, you can use the
same technique to prevent credit card
numbers from being revealed by, for
example, a compromised application
in the aftermath of a successful SQL
injection attack.
The Chinese Wall in Reverse
Another advanced scenario for Mod-
Security involves cooperating with
the GeoIP provider, Maxmind. GeoIP
locates users geographically on the
basis of their IP address, which
means you can restrict access to a
website to a specific region, such as
Pennsylvania – if you have a site in
Pennsylvanian Dutch that nobody
else would understand – or block
a country entirely. To do this, you
would install the mod_geoip2 module
on Apache 2, along with the GeoIP
software and GeoLiteCity.dat geo-
graphical database [11].
Imagine a mechanical engineering
company in Germany’s Swabian
region that is afraid of industrial
espionage from the Far East; in this
case, they could use the configura-
tion in Listing 3 to prevent access
from China – if the people in China
didn’t spoof their origins. The last
two lines form a filter rule chain. Line
6 locates the geographical region for
the requesting IP address, then line
7 dumps the request and a message
into the logfile if the request comes
from China. This might not be politi-
cally correct, but it is technically ef-
fective.
Full Insurance Coverage
ModSecurity has an enormous feature
scope, and it can take some time to
understand it completely. But if you
go to the trouble to plum the depths
of the module, it will pay dividends
with comprehensive methods that
give you additional protection against
attacks on web applications.
Thankfully, the prebuilt rulesets
make it easier to get started. And, the
vendor behind the project offers com-
mercial products and services such
as training. Armed with ModSecurity,
administrators can sit up tall in their
saddles, even if attackers are trying to
make their horses bolt. n
Info
[1] OWASP Best Practices: Use of Web Appli‑
cation Firewalls: [http://​­www.​­owasp.​­org/​
­index.​­php/​­Category:OWASP_Best_Prac‑
tices:_Use_of_Web_Application_Firewalls]
[2] ModSecurity: [http://​­modsecurity.​­org]
[3] ModSecurity Reference Manual:
[http://​­modsecurity.​­org/​­documentation/​
­modsecurity‑apache/​­2.​­5.​­10/​
­html‑multipage]
[4] ModSecurity Processing Phases: [http://​
­www.​­modsecurity.​­org/​­documentation/​
­modsecurity‑apache/​­2.​­5.​­0/​
­html‑multipage/​­processing‑phases.​­html]
[5] ModSecurity Actions: [http://​­www.​
­modsecurity.​­org/​­documentation/​
­modsecurity‑apache/​­1.​­9.​­3/​
­html‑multipage/​­05‑actions.​­html]
[6] ModSecurity Variables: [http://​­www.​
­modsecurity.​­org/​­documentation/​
­modsecurity‑apache/​­2.​­1.​­0/​
­html‑multipage/​­05‑variables.​­html]
[7] Nikto: [http://​­cirt.​­net/​­nikto2]
[8] OWASP ModSecurity Core Rule Set Pro‑
ject: [http://​­www.​­owasp.​­org/​­index.​­php/​
­Category:OWASP_ModSecurity_Core_Rule_
Set_Project]
[9] ModSecurity blog, “Virtual Patching Dur‑
ing Incident Response: United Nations
Defacement”: [http://​­blog.​­modsecurity.​
­org/​­2007/​­08/​­27/]
[10] Talks by the UN General Secretary:
[http://​­www.​­un.​­org/​­apps/​­news/​­infocus/​
­sgspeeches/]
[11] “Apache ModSecurity with GeoIP blocking
country specific traffic: ModSecurity +
GeoIP” by Suvabrata Mukherjee: [http://​
­linuxhelp123.​­wordpress.​­com/​­2008/​­12/​­11/​
­apache]
The Author
Sebastian Wolfgarten works as an IT Security
Expert with the European Central Bank as an
advisor, manager, and auditor of internal proj‑
ects designed to improve the security of the
IT infrastructure. Before this, he spent several
years working for Ernst & Young AG in Germany,
and as an Advisor for Information Security in
Ireland. He has also worked as an IT security
expert with T-Mobile Germany.
w w w. a d m i n - m aga z i n e .co m
 REAL SOLUTIONS
FOR REAL NETWORKS
Each issue delivers
technical solutions
to the real-world
problems you face
every day.

Learn the latest

techniques for better:

• network security

• system management

• troubleshooting

• performance tuning

• virtualization

• cloud computing

FREE DVD on Windows, Linux,

Solaris, and popular
Inside: varieties of Unix.
Knoppix 6.3 +
Backtrack

. c o m/su bs
aga z i n e
w. a d m in-m
N L I N E AT ww
ORDER O
 N U TS A N D B O LTS Daemon Monitoring

Monitoring daemons with shell tools

Watching the
© Shariff Che'Lah, 123RF.com

Daemons
Administrators often write custom monitoring programs to make sure for accessing the exit code. Error logs
are obtainable by redirecting the er-
their daemons are providing the intended functionality. But simple shell ror output to a file or, if available, by
tools are just as well suited to this task, and not just for systems that are setting the corresponding program
low on resources. By Harald Zisler option.
The only thing left to do is to find
Unix daemons typically go about Because almost every program out- the matching client program test the
their work discreetly in the back- puts standardized exit codes when it functionality of each service.
ground. The process table, which is terminates, you can use Unix conven-
output with the ps command, only tions. 0 stands for error-free process- Web Servers
shows you that these willing helpers ing, whereas 1 indicates some prob-
have been launched, although in the lems were encountered. This value is To check a web server, you could use
worst case they could just be hang- stored in the $? shell variable, which wget. The shell script command line
ing around as zombies. Whether or a shell script evaluates immediately for this would be:
not a daemon is actually working is after launching the sensor.
wget --spider -q ip-address
not something that the process table Various programs are suitable for
will tell you. In other words, you automated, “unmanned” access to The --spider option tells wget to
need more granular diagnostics. The the service provided by a given dae- check that the page exists but not to
underlying idea is to write a “sensor” mon; all of them will run in the shell load it. Defining the IP address in-
script for each service that performs a without a GUI. These programs often stead of the hostname avoids a false
tangible check of its individual func- provide an option (typically -q) that positive if DNS-based name resolution
tionality. suppresses output, and this is fine fails for some reason.

Listing 1: Database Monitoring

01 #! /bin/sh 18 34
02 19 if [ $? -eq 0 ]; 35 then
03 while true 20 36
04 do 21 then
37 time=$(date +%d.%m.%y\ %H:%M\ )
05 22
38 echo "$time: Database online!
06 zeit=$(date +%d.%m.%y\ %H:%M\ ) 23 echo "$time: Database online!
+++++++++" >> dba.log
07 +++++++++" >> dba.log
39 break
08 psql -U monitor -d monitor 24
-c "select * from watch;" 25 else 40

09 26 41 fi
10 if [ $? -eq 2 ]; 27 echo "$time: Database: serious error! 42 sleep 15
11 ***************" >> dba.log 43 done
12 then 28 echo "$time: Unable to restart!
44
13 ****************" >> dba.log
45 fi
14 echo "$time: Database is not accessible! 29 while true
46
****************" >> dba.log 30 do
47 fi
15 /usr/local/etc/rc.d/002pgsql.sh start 31 psql -U monitor -d monitor
16 sleep 15 -c "select * from watch;" 48 sleep 15

17 psql -U monitor -d monitor 32 49

-c "select * from watch;" 33 if [ $? -eq 0 ]; 50 done

92 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M


Figure 1: After starting, the script outputs the log at the console: availability,
error, restart, database running.
Almost all known databases include
a client program for the shell – for
example, mysql for MySQL or psql for
PostgreSQL. Alternatively, you can
use ODBC to access the database in
your scripted monitoring, such as the
isql tool provided by the Unix ODBC
project.
For ease of access, you might need to
set up a (non-privileged) user, a da-
tabase, and a table for the test query
on the database server. If you choose
the ODBC option, you also need a
.odbc.ini file with the right access
credentials.
The psql shell client for the Postgres
database also poses the problem of
non-standard exit codes. 1 stands for
an error in the query, although the
connect attempt has been successful;
2 indicates a connection error.
A connection test with psql would
look like this:
psql ‑U User ‑d Database ‑c U
"select * from test_table;"
For ODBC access, you would need to
pipe the SQL query to the client:
echo "select * from test_table;" | U
isql ODBC_data_source user
For the cups printer daemon, lpq
gives you a simple method of check-
ing whether the daemon is alive. If
you need to check access to indi-
vidual printers, you additionally need
to provide the print queue name and
then grep the exit code. To make sure
the exit code complies with this be-
havior, Grep checks the output that
you receive if the printer is active:
lpq ‑Pprinter | grep ‑q U
"printer is ready"
To match the output from lpq, you
need to modify the search string for
grep.
w w w. a d m i n - m aga z i n e .co m
Daemon Monitoring
The ping com-
mand checks
network con-
nections. The
exit error codes
differ, depending
on your operat-
ing system. The
FreeBSD ping
uses 2, the Linux
ping uses 1.
The number of test packets is re-
stricted by the ‑c packets option;
this improves the script run time and
avoids unnecessary network traffic. If
you use the IP address as the target,
you avoid the risk of false positives
from buggy name resolution.
ping ‑c1 ip_address
Sensor scripts can obviously be ex-
tended to cover many other system
parameters, such as disk space usage
(df), logged in users (who), and much,
much more.
If an error or threshold value infringe-
ment occurs, the script can use this
information to generate a message
and notify the system administrator.
The message text should include the
hostname, date, and time. Messages
can be stored in a file to which the
administrator has permanent access.
To allow this to happen, you simply
have to display the logfile in a termi-
nal and use tail ‑f, but other forms
of communication are also possible –
texting, for example.
If the shell script has the correct
privileges, it can become involved
and restart a daemon, remove block
files, or even reboot the whole sys-
tem. Because you should avoid run-
ning this kind of script as root, you
can instead set up special users and
groups to own the script and the
process (which is the case with many
daemons).
Database Restart
The sample script in Listing 1 moni-
tors an active database instance and
notifies the administrator if the da-
tabase happens to fail and then is
successfully restarted (Figure 1). If
it can’t start the daemon, it waits
N u ts a n d B o lts
for the administrator to step in and
handle the situation.
Printer Restart
The second sample script relates to
the printing service. The one shown
here is taken from a production ex-
ample, in which the cupsd server has
an unknown problem with a network
printer. The printer was disabled time
and time again, causing no end of
frustration to users and unnecessary
work for the system admins. The shell
script shown in Listing 2 doesn’t
output messages; instead, it simply
restarts the service. Either run these
scripts manually (for a temporary fix
or quick check) or as RC scripts.
Conclusions
Administrators don’t need a complex
monitoring framework that covers
every aspect of the environment and
has a multi-week learning curve.
With some scripting know-how, you
can easily create your own shell
scripts to monitor server daemon pro-
cesses and restart them autonomously
if so desired. The use of shell scripts
to monitor daemons and other system
functions is by no means restricted to
small embedded systems. With scripts
tailored to match your requirements,
you can establish your own trouble-
shooting arsenal. n
The Author
Harald Zisler has worked with Unix-flavored
operating systems since the early 1990s.
Listing 2: CUPS Monitoring
01 
#! /bin/sh
02 
03 
while true
04 
do
05 
06 
lpq ‑Plp | grep ‑q "lp is ready"
07 
08 
if [ $? ‑gt 0 ]
09  then
10  cupsenable lp
11 
fi
12 
13 
sleep 15
14 
15 
done
Admin 01 93
N u ts a n d B o lts VPNs with SSTP

©
Ma
xim
Ka
z mi
n,
12
3R
F.c
om

State-of-the-art virtual private networks

Private Affair
Because Microsoft’s legacy VPN protocol, PPTP, has a couple of vulner- connection through a firewall, a NAT
device, or a proxy. Like all SSL VPNs,
abilities, SSTP, which routes data via an SSL connection, was introduced SSTP uses TCP port 443 (HTTPS) for
as the new VPN protocol with Vista, Windows Server 2008, and Windows 7. data transfer. Compared with other
By Thomas Drilling
commercial or proprietary solutions
(e.g., IPsec, L2TP, or PPTP), the ad-
Virtual private networks (VPNs) access. The Secure Socket Tunneling vantage is that port 443 is open in
have established themselves as a Protocol (SSTP), which was intro- almost any router or server configu-
standard solution for convenient re- duced with Microsoft Windows Server ration, and SSTP packets can thus
mote access to enterprise networks. 2008, provides a solution by setting pass through without any additional
However, they can cause some is- up a VPN tunnel that encapsulates configuration overhead. The “Hand-
sues in combination with standard PPP or L2TP traffic on a Secure Sock- shake” box explains what Windows
tunneling protocols like PPTP if, for ets Layer (SSL) channel (Figure 1). does with all of these protocols.
example, NAT routers are involved For administrators, this means that Strong encryption in SSL 3.0 ensures
or you need to work around the lo- SSTP is a new VPN tunnel type in maximum security and performance.
cal firewall. Typically, it is not in the the Windows Server 2008 routing SSTP VPNs are thus a class of SSL
administrator’s best interest to modify and RAS server role. It encapsulates VPNs, like Cisco’s WebVPN or the
the firewall, NAT, or proxy configura- PPP (point-to-point protocol) packets Vigor Router by Draytek, that basi-
tion to suit requirements for remote in HTTPS, thus supporting the VPN cally work in the same way as IPsec,

94 Admin 01 w w w. a d m i n - m aga z i n e .co m

 VPns with SSTP n u TS A n d B o LTS

Handshake
Microsoft’s SSTP is basically another proprietary implementation of an
SSL VPN. SSTP relies on standards such as SSL and TLS for encryption
and authentication, but Microsoft has modified the tried-and-trusted
SSL handshake by introducing proprietary extensions that bind the pro-
prietary PPP protocol. If you look at the basic handshake, SSTP at first
keeps to the standard SSL handshake procedure:
1. The client opens a connection to TCP port 443 on the server.
2. The client sends an “SSL Session Setup Message” to indicate that it
wants to set up an SSL connection to the server.
3. The server sends an SSL certificate to the client.
4. The client validates the server certificate, identifies the correct en-
cryption method for the SSL session, and creates a session key, which it
encrypts using the public key from the server certificate.
5. The client sends the encrypted SSL session key to the server.
6. The server decrypts the client’s SSL session key using its own private
key and encrypts the communications with the session key. Up to this
point, the procedure is no different from standard SSL communication.
However, Microsoft then implements additional handshake steps that
build on what has happened thus far.
7. The client sends an HTTP-over-SSL request message to the server and
negotiates an SSTP tunnel with the server.
8. After this, the client negotiates a PPP connection with the SSTP
server, which includes authenticating the user’s login credentials with
a PPP authentication method and configuring the settings for the data
traffic.
9. The client starts to transfer data via the PPP connection.

L2TP, or PPTP, but use SSL to handle
the data transfer. Because SSTP en-
capsulates complete IP packets, the
connections act just like a PPTP or
IPsec tunnel on the client side. Ac-
cording to the Microsoft definition,
SSTP is a protocol mainly intended
for dialup connections in the applica-
tion layer that guarantees the confi-
dentiality, authenticity, and integrity
of the data to be transferred. A public
key infrastructure (PKI) is used for
authentication purposes. Microsoft in-
troduced SSTP with Windows Server
2008 and Vista SP1. Today, Windows
Server 2008 R2 and Windows 7 also
support SSTP [1]. But what does SSTP
offer the administrator, and how do
you set up a VPN server with SSTP?
Sample Scenario
In this example, I am using Windows
Server 2008 R2 to provide an SSTP-
based VPN server behind a NAT
device. The server is configured as
the domain controller and needs two
network cards for the VPN setup. At
SSTP VPN connection (TCP port 443)
1. Open TCP connection
2. Set up SSL connection and validate certificate
3. HTTPS request
4. Initiate SSL tunnel
5. Data communication via PPP
SSTP gateway server
Figure 1: The SSTP handshake is not much different from a standard SSL handshake. In contrast to IPsec, SSTP
sends PPP packets (not IP packets) through the tunnel.
least one card (preferably both) needs
a static IP address. The client will be
MS Windows 7.
The server and clients are both mem-
bers of an Active Directory (AD)
domain. Additionally, the server and
clients should have all the current
updates in place, such as the current
Service Pack 2 for Windows Server
2008 R2. After installing a Windows
server, you need to install the Ac-
tive Directory Domain Services to
make the server a domain controller.
The easiest way to do this is to run
dcpromo at the command line and
then follow the wizard.
The server also needs to provide
DNS, DHCP, and certificate services,
which you can achieve by configuring
the matching server roles in the role
wizard. VPN functionality is also pro-
vided via a server role Network policy
and access services.
PKI
Because SSTP uses HTTPS (port 443)
to handle all the data traffic, there is
SSTP VPN client
no alternative to configuring a public
key infrastructure. At a minimum,
this means installing at least one cer-
tificate on the SSTP server and a root
certificate authority certificate on all
SSTP VPN clients. You might have
to modify the packet filter rules, too,
even though SSTP doesn’t actually
need any additional NAT configura-
tion because port 443 is typically
open. The “Port Customizer” box ex-
plains how you can use a port other
than 443 for SSTP.
Next, you’ll need to configure an
Active Directory-integrated root certi-
fication authority on the domain con-
troller. In combination with a group
policy, this causes clients that are do-
main members to request certificates
automatically when they open a con-
nection. Certificates are then issued
Port customizer
SSTP normally uses TCP port 443, which is
open in most router and NAT configurations.
Security-conscious Windows administrators
might prefer to modify the standard port
used by SSTP. To do so, you need to edit the
following registry key
HKEY_LOCAL_MACHINE\SYSTEM\ U
CurrentControlSet\Services\SstpSvc\ U
Parameters
in the Registry Editor. Look for the Lis-
tenerPort parameter. Changing the view
to Decimal (right-click) lets you specify a
different port. Then you need to restart the
Routing and RAS service. If you change the
ListenerPort, you need to reconfigure
your NAT device to match and forward all
incoming traffic addressed to port 443 to
the newly configured port on the SSTP-based
VPN server.

w w w. A d m i n - m AgA z i n e .co m Admin 01 95

N u ts a n d B o lts VPNs with SSTP

Setting Up a Certification Authority ment console. Note that you cannot

install the Forefront 2010 server com-
To set up your own (private) certification authority in Active Directory, you need to launch the
ponent on a domain controller. Ad-
MMC console on your Windows server and add the certificate snap-in. Then, in the Certificate
ditionally, you will want to publish a
Snap‑In dialog box, select Computer account and Local computer for Select computer.
n To begin, right-click on Own certificate in the MMC, followed by All tasks | Request new cer- certificate revocation list in a produc-
tificate. tion scenario.
n You will see a selection of templates: Web Server will do the trick here. Now you need to
provide a name, such as fw.example.local. For this certificate, the name of the certificate Revocation
requestor must match the hostname of the device with which the VPN client will open the
connection. This step is essential for a successful SSL handshake. A Certificate Revocation List (CRL)
n If the request worked as intended, the certificate will install automatically and should be details any certificates revoked before
visible in the MMC console: Certificates snap-in below Certificates (local computer) | Own their expiry date. Revocation lists are
certificates. The CA certificate must also be visible in Certificates (local computer) | Trusted always available at CLR Distribution
root certificate authorities; you can check this if you experience difficulty with the certificate
Points (CDPs). A CDP can be set in
request.
the Extensions tab of the CA Proper-
ties dialog box (Figure 3).
by the domain controller and placed this to happen, the firewall must also Additionally, the certificate revocation
in the client’s local certificate store be authorized to request certificates list must be accessible to all clients at
along with the certification authority (Figure 2). You can set this up under all times via the Internet, which will
certificate. If you are unable or prefer the Web Server Properties Security mean configuring the packet filter on
not to purchase a commercial certifi- tab; Read and Enroll privileges are the local firewall. In Forefront TMG
cate, you can use a private certificate required at minimum. 2010, you can use the website pub-
issued by Active Directory’s built-in In production, it often makes more lishing wizard for this.
CA. The “Setting up a Certification sense to purchase a commercial cer-
Authority” box explains how to do tificate and install a dedicated firewall VPN Server
this. server with Microsoft’s new Forefront
You need to configure the firewall to Threat Management Gateway (TMG) Setting up the VPN server itself is
allow network traffic to pass through 2010 [2]. easily done. Once you have added
to the certificate authority in order for The gateway offers a wizard-based the network policy and access ser-
the certificate request to work. To get configuration in the TMG manage- vices role by clicking Add roles in

Figure 2: If you are unable to request a certificate, you can check the privileges Figure 3: A CLR distribution point controls the availability of the certificate
in the certificate template for web servers to see whether Enroll is allowed. revocation list, which all clients need to be able to access at any time.

96 Admin 01 w w w. a d m i n - m aga z i n e .co m


Figure 4: You need to set up the RAS services to run a VPN server.
the Customize this server section of
the Server Manager, the Routing and
Remote Access tool is available below
Management. The Action | Configure
and enable routing and RAS button
takes you to the Routing and Remote
Access Server Setup Wizard.
At the second Configuration step
(Figure 4), you’ll want to enable
Virtual private network (VPN) access
and NAT, then click Next and select
the required network interface. At the
following step, Address assignment
defines how the VPN server assigns
IP addresses to remote clients. If you
have the DHCP service running, Au-
tomatic is the quickest and cleanest
option. Then you can define an IP
address pool for the DHCP server to
use in the next step, Address range
assignment.
In the final step, you can choose
whether or not to use a Radius server
to authenticate clients on a large-scale
network; this is disabled by default.
The wizard will then instruct you to
set up the DHCP relay agent for Win-
dows to support the forwarding of
IPv4 or IPv6 TCP SSTP
Figure 6: The SSTP encapsulation structure is like a Russian doll. Microsoft has gone to considerable trouble
to make something proprietary from what are basically open protocols. From a technical point of view, there
seems to be no real reason to use PPP.
w w w. a d m i n - m aga z i n e .co m
DHCP messages
to RAS clients.
To do this, you
need to enable
the Relay DHCP Figure 5: Configuring Windows to forward DHCP messages to the RAS clients.
packets option in
the DHCP Relay Properties dialog box
(Figure 5).
Conclusions
Microsoft’s SSTP encapsulation struc-
ture is like a Russian doll. Just as
with PPTP, Microsoft uses the PPP
protocol with SSTP, which leads to a
fairly complex encapsulation structure requests, SSTP offers a secure, well-
(see Figure 6), in which an IP header
contains a TCP header, which in turn
contains an SSTP header, which then
contains a PPP header, which finally
contains the IP packets themselves.
Although the method seems to be
slightly more efficient than Encapsu-
lating Security Payload (ESP), with
an overhead of 8 bytes in the PPP
header, compared with 20 bytes in
IPsec over HTTPS, there is actually no
real need to encapsulate in PPP pack-
PPP IPv4 or IPv6 packet
Encapsulated SSL session
VPNs with SSTP N u ts a n d B o lts
ets. In IPsec, ESP builds directly on
IP. Microsoft is quite obviously seek-
ing to set itself apart by encapsulating
in PPP.
Apart from the fairly complex Win-
dows server configuration, which
mainly involves setting up the cer-
tificate services, and possibly packet
filters for transporting the certificate
performing tunnel technology for the
future. n
Info
[1] Microsoft support for SSTP: [http://​
­support.​­microsoft.​­com/​­kb/​­947032/]
[2] TMG 2010: [http://​­www.​­microsoft.​­com/​
­downloads/​­details.​­aspx?​­familyid=e05a
ecbc‑d0eb‑4e0f‑a5db‑8f236995bccd&​
­displaylang=en]
The Author
Thomas Drilling has been a freelance journalist
and editor for scientific and IT magazines for
more than 10 years. With his editorial office
team, he regularly writes on the subject of open
source, Linux, servers, IT administration, and
Mac OS X. In addition to this, Thomas Drilling is
also a book author and publisher, a consultant
to small and medium-sized companies, and a
regular speaker on Linux, open source, and IT
security.
Admin 01 97
S E RV I C E Contact Info / Authors
WRITE FOR US
Admin: Network and Security is looking
for good, practical articles on system ad-
ministration topics. We love to hear from
IT professionals who have discovered
innovative tools or techniques for solving
real-world problems.
Tell us about your favorite:
• interoperability solutions
• practical tools for cloud environments
• security problems and how you solved
them
• ingenious custom scripts
98 ADMIN 01
Editor in Chief
Joe Casad, jcasad@admin-magazine.com
Managing Editor
Rita L Sooby, rsooby@admin-magazine.com
Contributing Editors
• unheralded open source utilities Oliver Frommel, Uli Bantle, Andreas Bohle,
• Windows networking techniques that Jens-Christoph Brendel, Hans-Georg Eßer,
Markus Feilner, Marcel Hilzinger, Mathias Huber,
aren’t explained (or aren’t explained Anika Kehrer, Kristian Kißling, Jan Kleinert,
well) in the standard documentation. Daniel Kottmair, Thomas Leichtenstern,
Jörg Luther, Nils Magnus
We need concrete, fully developed solu-
Localization & Translation
tions: installation steps, configuration Ian Travis
files, examples – we are looking for a Proofreading & Polishing
complete discussion, not just a “hot tip” Amber Ankerholz
Layout
that leaves the details to the reader. Klaus Rehfeld, Judith Erb
If you have an idea for an article, send Cover
a 1-2 paragraph proposal describing your Illustration based on graphics by James Thew,
123RF
topic to: edit@admin-magazine.com.
Advertising
www.admin-magazine.com/Advertise
United Kingdom and Ireland
Penny Wilby, pwilby@admin-magazine.com
Phone: +44 1787 211 100
North America
Amy Phalen, aphalen@admin-magazine.com
Phone: +1 785 856 3434
All other countries
Hubert Wiest, anzeigen@admin-magazine.com
Phone: +49 89 9934 1123
Corporate Management (Vorstand)
Hermann Plank, hplank@linuxnewmedia.com
Brian Osborn, bosborn@linuxnewmedia.com
Management North America
Brian Osborn, bosborn@linuxnewmedia.com
Associate Publisher
Rikki Kite, rkite@linuxnewmedia.com
Product Management
Hans-Jörg Ehren, hjehren@linuxnewmedia.com
Customer Service / Subscription
For USA and Canada:
Email: subs@admin-magazine.com
AUTHORS Phone: 1-866-247-2802
(Toll Free from the US and Canada)
Falko Benthin 14 Fax: 1-785-274-4305
For all other countries:
Björn Bürstinghaus 25, 60 Email: subs@admin-magazine.com
Phone: +49 89 9934 1167
Thomas Drilling 52, 94 Fax: +49 89 9934 1199
Admin Magazine • c/o Linux New Media •
Florian Effenberger 42 Putzbrunner Str 71 • 81739 Munich • Germany
www.admin-magazine.com
Dan Frost 48 While every care has been taken in the content of the
magazine, the publishers cannot be held responsible for
Thomas Joos 74 the accuracy of the information contained within it or any
consequences arising from the use of it. The use of the
DVD provided with the magazine or any material provided
Daniel Kottmair 64 on it is at your own risk.
Copyright and Trademarks © 2010 Linux New Media Ltd.
Caspar Clemens Mierau 20 No material may be reproduced in any form whatsoever
in whole or in part without the written permission of the
James Mohr 28 publishers. It is assumed that all correspondence sent, for
example, letters, emails, faxes, photographs, articles, drawings,
are supplied for publication or license to third parties on a
Thorsten Scherf 8, 44, 78 non-exclusive worldwide basis by Linux New Media unless
otherwise stated in writing.
Tim Schürmann 68 Printed in Germany
Distributed by COMAG Specialist, Tavistock Road,
Udo Seidel 36 West Drayton, Middlesex, UB7 7QE, United Kingdom
Admin Magazine ISSN 2045-0702
Kurt Seifried 34 Admin Magazine is published by Linux New Media USA, LLC,
719 Massachusetts, Lawrence, KS 66044, USA, and Linux
Sebastian Wolfgarten 86 New Media Ltd, Manchester, England. Company registered in
England.
Harald Zisler 92 Linux is a trademark of Linus Torvalds.
W W W. A D M I N - M AGA Z I N E .CO M
DEDICATED
SERVERS £59
FROM

PER MONT
H
BRAND NEW DELL RANGE WITH WINDOWS OR LINUX

FREE CONTROL PANEL

FREE SAME-DAY SETUP
100 MBPS UNMETERED CONNECTION
NO MINIMUM CONTRACT

£59 £89 £129 £189 £299

DX-250 per month
D3-240 per month
D3-340 per month
D3-520 per month
D3-620 per month

PROCESSOR Intel® Xeon Intel® Xeon Intel® Xeon Intel® Xeon Intel® Xeon
CORES Quad 2.4Ghz Quad 2.66Ghz Quad 2.66Ghz Quad 2.26Ghz 2 x Quad 2.26Ghz
MEMORY 1GB DDR2 ECC 4GB DDR3 ECC 8GB DDR3 ECC 8GB DDR3 ECC 16GB DDR3 ECC
HARD DISKS 1 x 160GB SATA 2 x 250GB SATA 2 x 500GB SATA 2 x 1000GB SATA 4 x 1000GB SATA
RAID None Hardware RAID Hardware RAID Hardware RAID Hardware RAID
CONNECTION 100 Mbps 100 Mbps 100 Mbps 100 Mbps 100 Mbps

OR BUILD YOUR OWN FULLY CUSTOMISED DELL DEDICATED SERVER AT: linux.redstation.com

DELL POWEREDGE SERVERS 100 MBPS UNMETERED SECURE PRIVATE NETWORK

WINDOWS OR LINUX NO MINIMUM CONTRACT 24/7 TELEPHONE SUPPORT
FREE SAME-DAY SETUP REMOTE SERVER CONTROL PRIVATE UK DATA CENTRE

CALL FREE ON: 0800 622 6655 OR VISIT: linux.redstation.com