2.

introduction
this howto attempts to explain how to get pxe execute kickstart in order to
install a lot
of linux machines at the same time fully automatic and unattended. the main idea
behind
this installation infrastructure is simplicity, robustness, and performance.
i assume a certain familiarity of the reader with linux in general and how to
install
and configure software on linux in particular.everything i am describing here is
for red hat
6.2 and newer clients. pick the files for the version you are planning to install
on your
clients. the servers can be red hat 6.2 or newer.
the mechanisms i am describing are designed to work with relatively strict
security: i am
using static ip addresses on all clients and no client has write access on a
server.
things may be simpler for you if your environment has not so tight restrictions.
according to http://www.suse.de/~nashif/autoyast1/html/index.html, the same
installation
infrastructure can be used to install suse linux over the network.

3. pxe
to install a client via pre-boot execution environment (pxe) you don't need a pxe
server!
pxe uses a bootp request to get an ip address and other network information and a
bootloader
program to the client. you can either use a bootp server for doing this or a dhcp
and tftp
server.
in the following sections, i will describe how to set up a dhcp and tftp server.

4. dhcp
install the dhcp server from isc ( http://www.isc.org/) by using red hat's rpm
ftp://ftp.redhat.com
/pub/redhat/redhat-7.2-en/os/i386/redhat/rpms/dhcp-2.0pl5-8.i386.rpm
before you can start the daemon, you need to create an empty leases file: bash#
touch /etc/dhcpd.leases.
secondly, you need a configuration file /etc/dhcpd.conf which looks similar to
this example:

deny unknown-clients;
not authoritative;

option domain-name "slac.stanford.edu";

option domain-name-servers 192.168.0.9, 192.168.0.10;
option subnet-mask 255.255.255.0;

allow bootp;
allow booting;

option ip-forwarding false; # no ip forwarding

option mask-supplier false; # don't respond to icmp mask req

subnet 192.168.0.0 netmask 255.255.255.0 {

option routers 192.168.0.1;
}

group {
next-server tftpsrv; # name of your tftp server
filename "pxelinux.0"; # name of the bootloader program

host node1 {
hardware ethernet 00:02:b3:1d:16:e1;
fixed-address 192.168.0.11;
}
host node2 {
hardware ethernet 00:02:b3:1d:9f:87;
fixed-address 192.168.0.12;
}
}

because pxe does really use bootp, you need to have a tftp server running on the
same machine
where your dhcp server runs.
in the above example, all clients are directed towards "next-server" tftpsrv. you
can use a
simple round-robin method to distribute the load evenly over several tftp server
by putting
the server name into each host section instead of having only one for all hosts in
a group.

group {
filename "pxelinux.0"; # name of the bootloader program

host node1 {
hardware ethernet 00:02:b3:1d:16:e1;
fixed-address 192.168.0.11;
next-server tftpsrv1; # name of first tftp server
}
host node2 {
hardware ethernet 00:02:b3:1d:9f:87;
fixed-address 192.168.0.12;
next-server tftpsrv2; # name of second tftp server
}
}

talk to your networking department about the dhcp protocol! ours had switched off
forwarding
the dhcp broadcasts over the routers. i needed to have an extra ip helper address
configured
in some routers because my dhcp server is not in the same subnet as the clients.
while you are at it: ask them to disable "auto-negotiation" on all your machine
ports. also ask
them to enable "portfast" on all ports where your machines are connected.
this disables the spanning tree protocol network gear uses. if you don't have
both of these
settings you can run into trouble with pxe during the early stages of your
install.
perhaps you need some sort of load balancing for your dhcp server or you are
concerned about
 the stability of your dhcp server and want some backup. in this case you might
try to use two
(or more) dhcp servers sharing the same configuration file over a shared file
system
(nfs or afs for example). the common configuration file makes maintaining more
than one dhcp
server simple and the clients will talk to the one that is available and least
loaded.

5. automatic mac address detection

we, at slac, are using static ip addresses for all our computers. when getting a
whole bunch
of new machines you need to get their mac addresses into the dhcpd.conf file.
if you can use dynamic ip addresses, just designate an interval of ip addresses in
your
dhcpd.conf file to be handed out to your clients.

however, i have written a script to detect mac address for new machines
automatically and fully
unattended. it works by exploiting the topological knowledge about which node is
connected to
which port on the switch. it uses snmp to query the bridging table of the switch
the machines
have to be connected to. it writes a new dhcpd.conf file which now additionally
contains the
newly detected mac addresses. the script currently works for cisco catalyst 6509
and 3750
switches and is available upon request from the author.
here is the usage message of the script:

script to query a switch for mac addresses.

-- version 4.2, author: alf wachsmann <alfw@slac.stanford.edu> --

usage: mac_print.pl
optional flags:
-c(onfig) <file>: configuration file. the default config file is
expected to be in the same directory as this script.
-p(xeprepare): link pxelinux config file in tftpboot directory for
each ip address to do netboot.
-r(reset): remove all links to pxelinux config file in tftpboot
directory for each ip address to do localboot.
-v(erbose): be a bit more chatty.
-t(est): test the configuration file.
-d(ebug): lots of debug output;
especially usefull with -test option.
-h(elp): print this help.

see text at end of this script for a description of how it works.

a configuration file for the script looks like this:

# system type can be 'linux' or 'solaris'.

# the scripts outputs either a dhcpd.conf addition for linux or
# just mac and ip address for solaris
$sys_type = 'linux';
# subnet (in fact: vlan) in which the new machines should be in (see below):
$subnet = 0;

# which switch to query:

$switch = 'swh-farm8';

# which ports on which cards to look for. the order here

# determines which ip address is assigned to which port!
# (there are at most 48 ports per card.)
# card => [from port, to port]
%portdef = (
3 => [1, 48],
4 => [1, 48],
);

# range of ip addresses which should be used for this cluster of

# machines. prefix is always "192.168.".
# column one is the subnet.
# the number of ip addresses in this range must match the number
# of ports to observe! this consistency is checked by the script.
%iprange = (
0 => [11, 106]
);

if you are using this script you want to use another script which checks for
changes in the
dhcpd.conf file and re-starts the dhcp server if necessary (the isc-dhcp server
does not
understand a hup signal).
the script which detects the mac addresses cannot do this because it might run on
a different
host then the dhcp server (this means the dhcpd.conf file has to be in a shared
files system
(afs, nfs, dfs)).

#!/bin/sh
# author: (c) alf wachsmann <alfw@slac.stanford.edu>, july 09, 2001.
# usage: just start the script. ctrl-c out of it.

conffile="/afs/slac/kickstart/dhcp/etc/dhcpd.conf"

# get the md5 checksum of the dhcpd.conf file

md5_res=`md5sum $conffile | awk '{print $1}'`

/etc/rc.d/init.d/dhcpd start

while true
do
md5_new=`md5sum $conffile | awk '{print $1}'`
# if md5 checksum has changed, restart the dhcp server
if [ $md5_res != $md5_new ]; then
/etc/rc.d/init.d/dhcpd restart
echo "restarted dhcp daemon"
md5_res = $md5_new
fi
# no 'sleep' here because this script is time critical
done

6. bootloader
pxe can load a program into the client's memory and start it. i found it the
easiest to manage,
if i first load a bootloader into the clients. the bootloader then loads its
configuration file
via tftp from the next-server (tftpsrv in the dhcpd.conf file example above).
get h. peter anvin's pxelinux part from his syslinux package
( http://syslinux.zytor.com/).
compile it and copy the file pxelinux.0 in your /tftp/ directory on your tftp
server.

the bootloader configuration file determines whether a client boots from its local
hard disk
or over the network.
here are example configuration files for both cases:

red hat 6.2 network boot (filename default.netboot-6.2):

default linux
serial 0,38400n8
label linux
kernel vmlinuz-6.2
append console=console=tty0 ttys0,38400 load_ramdisk=1 \
initrd=initrd-6.2.img network \
ks=nfs:nfssrv:/data/kickstart/ks-6.2.cfg

red hat 7.2 network boot (filename default.netboot-7.2):

default linux
serial 0,38400n8
label linux
kernel vmlinuz-7.2
append ksdevice=eth1 console=tty0 console=ttys0,38400 load_ramdisk=1 \
initrd=initrd-7.2.img network \
ks=nfs:nfssrv:/data/kickstart/ks-72.cfg

watch out for the line breaks in the "append" line. they are here for readability
only -
the kernel cannot handle them!(side note: with the above console definitions
appended to the
linux kernel you are able to follow the kickstart installation on a serial
console.)
you might encounter problems with dhcp if your clients have multiple ethernet
adapters (nics).
the effect is, that pxe itself receives a dhcp offer but the booted kernel
attempting a
kickstart installation does not. a remedy for this problem is to add ip=dhcp to
the list
of kernel parameters in the append line in the default.netboot-xyz file.

boot from local hard disk (filename default):

default linux
label linux
localboot 0

pxelinux.0 tries to read several configuration files. it uses the first one it
finds.
the filenames it looks for are determined by the ip address of the client it is
running on.
it converts the four decimal number parts of an ip address (they are devided by
dots) into
hexadecimal numbers and concatenates them. example: ip address 192.168.0.11 gets
converted into c0 a8 00 0b (without the spaces).

the search for files starts at c0a8000b and proceeds by removing one digit from
the right
(leaving c0a8000) and so forth. when all digits are removed it will try as last
resort the
filename default.

on your tftp server, this algorithm can be used to tell each single machine how to
boot:

/tftp/pxelinux.cfg/
c0a8000b -> default.netboot-7.2 # install red hat 7.2 on
192.168.0.11
c0a8000c -> default.netboot-6.2 # install red hat 6.2 on
192.168.0.12
default.netboot-6.2
default.netboot-7.2
default

this is important if you install a lot of machines at the same time. you can watch

the syslog file on your tftp server and whenever a client got its initial ram disk
transmitted,
you can remove the symlink for that machine from the pxelinux.cfg directory.
this forces the client to load the default configuration which says: "boot from
local disk!"
when it reboots after kickstart is done.

technically, it is not necessary to use a bootloader. pxe can load a linux kernel
directly if
told so by dhcp. i did not try this because of the reasons mentioned in the
previous paragraph
i find it less convenient: you would have to change the dhcpd.conf file to tell a
machine not to
do a network boot any more but a boot from the local hard disk. editing this file
on the fly
is more hassle than changing one symlink.

7. tftp
pxe requires a special tftp server. read pxelinux.doc from the already mentioned
syslinux package for details.
in an earlier version of this document i suggested using tftp-hpa started from
xinetd.
this combination of tools cannot reliably handle much more than 64 clients at a
time! with
more clients not all of them will get an answer from your tftp and you will see
syslog messages like this: tftpd: read: connection refused.
update: per h. peter anvin, the problem with tftp-hpa should be fixed with version
0.17. i haven't tried it myself, though.

to overcome this problem i am now using atftp from

ftp://ftp.mamalinux.com/pub/atftp/. this tftp server can run as stand-alone daemon
which is the mode i am using.

grab the source and compile it. it should compile just fine on red hat linux.

start the daemon on your tftp server with some options:

/usr/sbin/atftpd --daemon --no-multicast --group nobody --tftpd-timeout 0 -m 1000

/tftp

depending on your local group definitions you may not need to change the group id
the daemon is running under. the default group it uses is nogroup.

the tftp server directory /tftp should look like this:

/tftp/initrd-6.2.img
initrd-7.2.img
pxelinux.0
vmlinuz-6.2
vmlinuz-7.2
kickstart_end

/tftp/pxelinux.cfg/c0a8000b -> default.netboot-7.2

c0a8000c -> default.netboot-6.2
default
default.netboot-6-2
default.netboot-7.2

see section linux kernel, initial ram disk for how to get the files vmlinuz-6.2,
vmlinuz-7.2, initrd-6.2.img, and initrd-7.2.img.

if you want to use the script described in section a useful script, you should
have the empty file kickstart_end in your tftp directory:

touch /tftp/kickstart_end

if you are again, as for the dhcp server, concerned about stability of your tftp
server you can, again, put your /tftp/ directory in a shared file system (like nfs
or afs) and have all your tftp servers access this shared space. the method for
preventing endless re-installation loops of your clients describe in section a
useful script will work even in this setting.

--------------------------------------------------------------------------------
9. kickstart
you need a kickstart configuration file to make kickstart work. the file needs to
be in the place you have specified in your bootloader configuration file (see
section bootloader).

content of the file is as usual. see the kickstart howto for further help.

if you want to use the script described in the next section, add the following at
the very end of your kickstart configuration file:

# the last thing happening in kickstart:

# get a file via tftp which indicates we are done.
echo "get kickstart_end" | /usr/bin/tftp tftpsrv

make sure kickstart installs the tftp package on your clients!

if your clients have write access on your server (for instance via nfs) you could
mount this filesystem in a last step, remove the symlink for this client (see
section tftp) and unmount the filesystem again. in this case you don't need the
script described in the next section.

10. a useful script

with your tftp server recording transfered files into syslog you can use the
following script to automatically remove the symlinks in the /tftp/pxelinux.cfg/
directory for each machine. the script watches for a string like tftpd[...]:
serving kickstart_end to 192.168.0.11:32768. this indicates that the machine with
ip address 192.168.0.11 loaded the marker file as the last action before the
reboot after kickstart is done.

note that from version atftp-0.4 to atftp-0.5 the output format changed sightly.
the following script matches the format of atftp-0.5 and later.

start this script on your tftp server(s) with the command

./pxecfg_changer.pl

press ctrl-c to terminate the program.

you are now able to start an installation on a whole bunch of machines without
worrying about missing one link with the consquence that the machine would do its
installation again (and again (and again (...))).

#!/usr/local/bin/perl -w
#
# author: alf wachsmann, <alfw@slac.stanford.edu>, august 15, 2002
# name of script: pxecfg_changer.pl
# version 2.0
# usage:
# $path/pxecfg_changer.pl

use strict;

use vars qw($version $prg $path);

my $debug = 0;
my $verbose = 1;
my $file = '/var/log/messages';

$version = '2.0';
$prg = $0;
$prg =~ s{(.*/)}{};
$path = $1;

my $pxetftppath = "/tftp/pxelinux.cfg";
my $pxedefault = $pxetftppath."/default";
die("directory '$pxetftppath' does not exist!") unless (-d $pxetftppath);
die("file '$pxedefault' does not exist!") unless (-f $pxedefault);

my $hexip_prefix = uc(sprintf("%02x", 192).sprintf("%02x", 168));

my $pid = open(input, "/usr/bin/tail -n 1 -f $file |");

$sig{int} = $sig{term} = sub { system("kill $pid") };

while (<input>) {

# this regexp matches output from atftp-0.6 that looks like this:
# tftpd[9960]: serving kickstart_end to 192.168.0.11:32768
next unless $_ =~ /tftpd\[\d+\]: serving kickstart_end to
192\.168\.(\d+)\.(\d+):\d+$/;
my $hexip = uc(sprintf("%02x", $1).sprintf("%02x", $2));

print "unlink($pxetftppath/$hexip_prefix$hexip)\n" if $debug|$verbose;

unlink($pxetftppath."/".$hexip_prefix.$hexip) unless $debug;
}

8. linux kernel, initial ram disk

two components loaded onto a client are the linux kernel and the initial ram disk.
for red hat 6.2 i got them out of the bootnet.img file.

here is how you get to them:

bash# mkdir /mnt/linux /mnt/linux1

bash# mount /tmp/bootnet.img /mnt/linux -o loop
bash# cp /mnt/linux/initrd.img /tftp/initrd-6.2.img
bash# cp /mnt/linux/vmlinuz /tftp/vmlinuz-6.2

for red hat 7.1 and higher the files are already in a special directory in the
distribution:

bash# cp i386/images/pxeboot/initrd.img /tftp/initrd-7.2.img

bash# cp i386/images/pxeboot/vmlinuz /tftp/vmlinuz-7.2

you can of course use the same method as for red hat 6.2 if you want to.

starting with red hat 9 you have to use the images in the pxeboot directory
because the bootdisk.img does not contain any network drivers!
the installer "anaconda" from red hat linux 6.2 cannot install over "eth1"! if you
have a motherboard nic which cannot do pxe but your machine has a second nic which
can do pxe (which was the case for my clients), you need to modify the "anaconda"
source code.

"anaconda" in red hat 7 and later has a new parameter ksdevice for exactly this
purpose (see the default.netboot-7.2 example in section bootloader).

here is the modification you need to make in file

redhat6.2/i386/misc/src/anaconda/loader/loader.c:

1584,1585c1583
+ /* modification by alf wachsmann <alfw@slac.stanford.edu> */
+ if (kickstartnetwork("eth1", &netdev, "dhcp", flags)) {
---
- if (kickstartnetwork("eth0", &netdev, "dhcp", flags)) {

you get the new loader program into your initial ram disk with the following
commands:

bash# cd i386/misc/src/anaconda/loader
bash# vi loader.c # make the changes from above
bash# make
bash# strip loader-network

bash# mkdir /mnt/linux /mnt/linux1

bash# cp i386/images/bootnet.img /tmp/bootnet.img
bash# mount /tmp/bootnet.img /mnt/linux -o loop
bash# cp /mnt/linux/initrd.img /tmp/initrd.img.gz
bash# gzip -d /tmp/initrd.img.gz
bash# mount /tmp/initrd.img /mnt/linux1 -o loop

bash# cp i386/misc/src/anaconda/loader/loader-network /mnt/linux1/sbin/loader

bash# umount /mnt/linux1

bash# gzip /tmp/initrd.img

bash# cp /tmp/initrd.img.gz /tftp/initrd.img

this new initial ram disk will now perform kickstart over "eth1" (and only over
"eth1").

11. putting it all together

you now should have on one machine

a dhcp server with ip addresses for all your client machines you are going to
install
a tftp server running the atftpd daemon with a /tftp directory populated with
a bootloader program pxelinux.0,
a compressed linux kernel vmlinuz-6.2,
a compressed linux kernel vmlinuz-7.2,
an initial ram disk initrd.img-6.2,
an initial ram disk initrd.img-7.2,
an empty marker file kickstart_end
a subdirectory pxelinux.cfg/ with three actual files:
default.netboot-6.2
default.netboot-7.2 and
default
and one symlink for each client you want to install pointing to the
default.netboot-6.2 or default.netboot-7.2 file
a script running as "root" tail -n 1 -f /var/log/messages | ./pxecfg_changer.pl

depending on your needs, you may need only the 6.2 or the 7.2 versions of the
above files.

you also need

a kickstart configuration file

your linux distribution served over the net via nfs, ftp or http

12. references
here are all the tools and programs you need:

anvin, h. peter, syslinux/pxelinux: a boot loader for linux,

http://syslinux.zytor.com/.
lefebvre, jean-pierre and lefebvre, remi, atftp,
ftp://ftp.mamalinux.com/pub/atftp/.
isc (internet software consortium), isc dynamic host configuration protocol
(dhcp), http://www.isc.org/products/dhcp/.
burgess, mark, cfengine, http://www.iu.hio.no/cfengine/.

some useful literature:

clerc, davic and vuilleumier, marc , linux remote-boot mini-howto: configuring

remote-boot workstations with linux, dos, windows 95/98 and windows nt,
http://cuiwww.unige.ch/info/pc/remote-boot/howto.html.
intel corporation, preboot execution environment (pxe) specification version 2.1,
ftp://download.intel.com/ial/wfm/pxespec.pdf.
intel corporation, pxe software development kit instructions,
ftp://download.intel.com/ial/wfm/pxesdk.pdf.
intel corporation, pxe production development kit instructions,
ftp://download.intel.com/ial/wfm/pxepdkread.pdf.
wachsmann, alf, a general purpose high performance linux installation
infrastructure http://www.slac.stanford.edu/~alfw/publications/slac-pub-9193.pdf
(slac publication).