Working with SystemImager

Make sure to always reimage one machine from each lab and test it before reimaging the
whole lab!

Goals
Going forward we need to:

 create a Golden Image for each lab so re-imaged machine are good to go
 keep a few images archived (and dated) so we can roll back.
 automate the imagining so:
o anyone can do it
o the configuration files become the documentation
 (until automated) document specific requirements and instructions for each lab
o http://collaborate.bu.edu/engit/LinuxSystemOrganization
o http://collaborate.bu.edu/engit/ComputerLabs

Imaging new systems

If you are imaging a new machine with a new IP and DNS name you will need to add it to the
corresponding overrides group on the systemimager server, i.e. eng-imageserver. Run
"si_clusterconfig -e" to update which machines get the overrides, and remember to restart the
rsync service.

Reimaging with an incremental update

You can do this on a system that's up and online, and it will sync only the things that have
changed between this image and the last. Simply log in as root and run:

si_updateclient --server eng-imageserver --image eng-neo64 --override eng-

neo64-vlsi

When reimaging a "live" system, make sure that the kernel (and therefore also the kernel
modules) that the system is currently running is still in the new image that you're syncing to!
If it isn't, weird things may happen until you reboot. If you follow this rule, a reboot will not
usually be necessary, and you can even reimage while someone is logged in or while grid
jobs are running! Firefox, Thunderbird, OpenOffice, or other programs that might get
updated by the reimage may behave strangely, but in general, it's OK.

Reimaging "from scratch"

If the machine is completely busted, new, or you don't mind taking it down, you should
reimage it from scratch. (To completely change hard disk partitioning, reimaging from
scratch is the only option, but this should rarely if ever be necessary.)
If you are in a lab such as CompSim, where SystemImager is set up for net-boot and has the
MAC-->IP translation coded into the image server, simply hit "f12" at BIOS POST (or
whatever BIOS feature allows booting from network).

Reimaging "from scratch" with a USB stick

If not (as is currently the case in Signals, VLSI, and HPCL), burn the USB stick image from
the Netapp onto a USB stick that you have lying around and change the syslinux.cfg file on
the stick to specify the image and network information for the machine you wish to reimage.

This is exactly how to do it:

The stick image is in:

/ad/eng/support/software/linux/bulinux/neo/systemimager/systemimager416-
64M.img

For a 32-bit machine, use a 32-bit stick image:

/ad/eng/support/software/linux/bulinux/neo/systemimager/systemimager416-
i386.img

(Note: these are just the STICK images, not the whole SYSTEM image! The stick images are
very small, minimal linuxes designed to just run Systemimager to get the SYSTEM image
onto the machine over the network.)

You can burn this stick image to a USB stick by sticking the USB stick into any Linux
machine and running "dmesg" to see which device it shows up as. (Probably /dev/sdb.) Copy
the stick image to /tmp, and then become root and run:

dd if=/tmp/systemimager416-64M.img of=/dev/sdb

The light on your stick should flash a bunch of times and when the process says it's complete,
you can pull the stick out and put it in again. Now, change the syslinux.cfg file on the stick to
say what you want. There's an example syslinux.cfg in:

/ad/eng/support/software/linux/bulinux/neo/systemimager/syslinux.cfg

Some machines, such as CompSim machines, work fine with the default kernel and initrd that
comes on the stick, but for most machines, you also need to overwrite the default kernel and
initrd on the stick with the ones in:

/ad/eng/support/software/linux/bulinux/neo/systemimager/kernelsandinitrds/k
ernel-lenovo
/ad/eng/support/software/linux/bulinux/neo/systemimager/kernelsandinitrds/i
nitrd-lenovoallmodules.img

I would recommend you copy that syslinux.cfg over the one on the stick you just burned, and
then change IMAGENAME=eng-neo64
(Note that the example syslinux.cfg is for signals11. You must change the IP information for
the system you are imaging which should have a different static IP/hostname, and if you're
imaging on a different subnet than Signals, you must change all the network values, including
the netmask, to the appropriate values! If you wish for the machine you're imaging to have a
DHCP address, you can remove all the network information from the syslinux.cfg -- but be
sure to keep the "HOSTNAME" field, and make sure it's set to the hostname of a machine
that's the same hardware as the machine you're imaging. Thus, if you're imaging a machine
that's the same hardware as signals11, you should keep the hostname set to signals11, and
once you're done imaging, get rid of the hostname field in /etc/sysconfig/network . (It will not
cause a conflict for two machines to have the same hostname, so you could leave it that way,
but it's ugly.)

If you ever experience issues with bittorrent, you can take out the "bittorrent=y" and it will
image itself the slow way.

Hook the machine up in the correct lab for the IP you've set and boot from this stick, and it
will reimage.

Reimaging "from scratch" with a CD

If the machine will NOT boot from USB (as is the case with B11 and may be the case with
HPCL), you should boot from CD. (You may be able to use the USB stick with HPCL if you
burn the stick on Windows -- for some reason HPCL doesn't seem to like sticks burned on
Linux. We've talked to HP tech support about this and it seems there's just something funny
about those machines being very particular about their USB sticks. It may be safer to just do
HPCL with a CD.)

Burn the ISO at:

/ad/eng/support/software/linux/bulinux/neo/systemimager/systemimager416-
bt.iso

and boot it on the machine you want to reimage (F8 or whatever you need to hit in the BIOS
to boot from CD.) When the CD starts loading, you must manually type in the APPEND line
for the network/image information for your system at the kernel boot: prompt, like so:

boot: kernel initrd=initrd.img root=/dev/ram ramdisk_blocksize=1024

ramdisk_size=80000 DEVICE=eth0 GATEWAYDEV=eth0 IMAGESERVER=128.197.115.78
IMAGENAME=eng-neo64 GATEWAY=128.197.115.1 NETMASK=255.255.255.0
NETWORK=128.197.115.0 BROADCAST=128.197.115.255 IPADDR=128.197.115.126
HOSTNAME=bme-compsim-107

Creating a new GoldenImage

If the system already has SystemImager installed, you can create a new 'GoldenImage' as
follows:

Pre Process
Note: Before running System Imager, make sure the partition scheme of the computer is the
same as the partition table specified in your image (vlsi, signals, etc)

1. Update the system and after upgrade is done, reboot the system.

yum -y upgrade

2. If desired, get the absolute newest nvidia kernel module for it from
http://www.nvidia.com/object/unix.html and chmod 755 this driver and put it in
/mnt/support/software/linux/bulinux/neo/scripts/nvidia/ . Run it and update the nvidia kernel
module on this machine.

sh /mnt/support/software/linux/bulinux/neo/scripts/nvidia/NVIDIA-Linux-
whatever.run

After rebuilding the Nvidia and/or other kernel drivers, reboot the system and check it

telinit 3; telinit 5

3. Update the /etc/rc.local to refer to the new nvidia kernel module instead of the old one.

4. Unmount any Kerberized shares you have and turn off your firewall:

umount -l /ad/eng/* ; umount -l /ad/eng/research/* ; service iptables stop

5. Make sure the SystemImager program files are in your path. They should be in /usr/sbin.
Either CD there or add it to your PATH environment variable.

Overrides

Hardware differences require a few files to differ from lab to lab. Things like the kernel initrd
and the xorg.conf are what you need to be concerned about. Put these in the appropriate
directories for each category of machine in /var/lib/systemimager/overrides. The overrides
are set in the format eng-neo64-labname

Generally, when updating the image, you don't have to worry about the xorg.conf, but you do
need to update the initrd for all hardware platforms. You should log into one machine from
each lab and run "yum -y upgrade" on it to get the new kernel and that will make the new
initrd for that platform. Then copy the initrd file from that machine into the appropriate
overrides directory on the image server. Leave the old initrd files in there too! They're
important in case you ever want to boot an old kernel. Make sure to always image one
machine from each lab and test it before reimaging the whole lab!

In general, we've been using vlsi & signals machines as the golden clients for eng-neo64, but
any lab could be the master image, so put the signals initrds in there as well.

Creating a Golden Image

Log into your golden client as root. Run:

si_prepareclient --server eng-imageserver

Now ssh as root to your image server (eng-imageserver):

run:

si_getimage --golden-client your-golden-client --image your-image-name

Mount your Kerberized shares back

mount -a

To make sure the newest changes are incorporated when you image with the bittorrent
method, update /etc/systemimager/bittorrent.conf on the eng-imageserver to include any new
images in BT_IMAGES and any new overrides in BT_OVERRIDES , and then restart the
"systemimager-server-bittorrent" service. Note that if BT_UPDATE is set to "y", this will
take a long time because it will rebuild the bittorrent tarballs (you need to set it to y after you
have done a new golden client update, but not if you're just updating the overrides.)

Setting up a SystemImager Client

To set up Systemimager initally on a system that is running Linux but doesn't have
Systemimager, run this script (but read below first): 64bit systems

 yum -y install perl-XML-Simple mkisofs

 cd /tmp
 wget http://download.systemimager.org/pub/sis-install/install
 chmod u+x install
 ./install -v --download-only --tag stable systemconfigurator \
 systemimager-client systemimager-common \
 systemimager-x86_64boot-standard systemimager-x86_64initrd_template \
 systemimager-server perl-AppConfig
 rpm -Uvh sis-packages/*.rpm

setup systemimager excludes

#Only exclude if the computer already has the nss_qidb.conf from the right subnet

 echo -e "# BU Linux stuff \n\

 /etc/autocensus.anonkey \n\
 /var/cache/afs/* \n\
 /usr/local/vmware/* \n\
 /etc/krb5.keytab.nss_qidb \n\
 /etc/nss_qidb.conf \n"\ >>
/etc/systemimager/updateclient.local.exclude

Dan's old notes:

 yum -y install perl-XML-Simple mkisofs

 cd /tmp
 wget http://download.systemimager.org/pub/sis-install/install
 chmod u+x install
 ./install -v --download-only --tag stable systemconfigurator \
  systemimager-client systemimager-common \
 systemimager-i386boot-standard systemimager-i386initrd_template \
 systemimager-server perl-AppConfig
 rpm -Uvh sis-packages/*.rpm
 echo -e "# BU Linux stuff \n\
 /etc/autocensus.anonkey \n\
 /var/cache/afs/* \n\
 /usr/local/vmware/* \n\
 # boot can be excluded if kernels are the same \n\
 /boot/* \n\
 /etc/krb5.keytab.nss_qidb \n\
 #Only exclude if the computer already has the nss_qidb.conf from the
right subnet \n\
 /etc/nss_qidb.conf" >> /etc/systemimager/updateclient.local.exclude

Note Well

If you are moving from 32-bit to 64-bit BU Linux, do NOT put the "/boot/*" directory in the
excludes as shown in the script above -- you will need the new boot stuff for the new
architecture to boot. Additionally, make sure to run the following or you will be left with no
Grub configuration:

/sbin/grub-install /dev/sda

If you are updating a kernel as part of the upgrade, as long as you are doing it on identical
hardware, you could take out the /boot/* line also so that the whole thing including the kernel
gets updated, or you could just "yum install kernel" before you run si_updateclient, to make
sure that the newest (and same) kernel is on the new system as on the golden-client.

If you are updating from one architecture to another where the first arch can't run binaries
from the second arch (such as x86 to x86_64), adding in the following two lines to your
command sequence is a useful hack to make sure the machine can save its grub configuration
and reboot at the end, since the new reboot binary will now be the wrong arch and the grub-
install will need to be run manually to be sure the changes take effect:

at the very beginning of the script:

 cp /sbin/reboot /tmp

at the very end of the script:

 /sbin/grub-install /dev/sda
 /tmp/reboot -f

Updating Clients
Chose the appropriate image name for your-image-name:

eng-neo64

Once the image is updated, log into the other clients and run:
si_updateclient --yes --server eng-imageserver --image your-image-name --
override your-override-name --override your-next-override-name

So for 64-bit Neo in the VLSI lab, the command would be:

si_updateclient --yes --server eng-imageserver --image eng-neo64 --override

eng-neo64-vlsi

if you want to run this and walk away add --reboot

Note that, for nested overrides, the overrides will be installed in the order you put them on the
command line -- so you should list the "most important override" last. This is the OPPOSITE
of the way they are listed in si_clusterconfig !

Making a custom Systemimager Install CD

Unlike the USB stick disk image created by si_mkautoinstalldisk, editing files on a CD
iso9660 filesystem created by si_mkautoinstallcd is non-trivial. The best method seems to be
to use the --append " " switch when running si_mkautoinstallcd, as mentioned on
http://wiki.systemimager.org/index.php/SSH :

eng-imageserver:~# si_mkautoinstallcd --out-file=/tmp/systemimager416.iso -

-append "APPEND initrd=initrd.img root=/dev/ram ramdisk_blocksize=1024
ramdisk_size=80000 DEVICE=eth0 GATEWAYDEV=eth0 IMAGESERVER=128.197.115.78
IMAGENAME=eng-neo64"

When booting the CD, if you want to use a static IP address or other options, you could
specify them in the --append switch, but this would require re-burning the CD each time. It is
easier to simply type them at the kernel boot: prompt, like so:

boot: kernel initrd=initrd.img root=/dev/ram ramdisk_blocksize=1024

ramdisk_size=80000 DEVICE=eth0 GATEWAYDEV=eth0 IMAGESERVER=128.197.115.78
IMAGENAME=eng-neo64 GATEWAY=128.197.115.1 NETMASK=255.255.255.0
NETWORK=128.197.115.0 BROADCAST=128.197.115.255 IPADDR=128.197.115.126
HOSTNAME=bme-compsim-107 console=ttyS0,9600

(Only use the "console" statement if you're doing this directly on the head and not through an
IPMI controller.)

It is also possible to edit the iso filesystem after you've created it. To do this, first copy your
iso image to a new iso image to create the template for where you're going to dump the new
data. Now mount the old .iso image as a loop filesystem:

mount -o loop /path/to/yourimage.iso /path/to/mountpoint

and copy the contets from the mountpoint to a temporary directory. cd into that directory and
modify the files you want to modify (syslinux.cfg and/or kernel and initrd).

Now, making sure you're cd'd into the root of the temporary directory where you've got your
newly modified iso filesystem, run:
mkisofs -J -r -T -v -pad -b isolinux/isolinux.bin -c isolinux/boot.catalog
-no-emul-boot -boot-load-size 4 -boot-info-table -o
/path/to/newimagename.iso .

and this will create a new .iso for you to burn.

Troubleshooting System Imager

Step1 : Make sure image is compatible with hardware architecture
Step2 : Do not iclude nss_qidb files in the exclude file list for a fresh
install.
Step3 : If the system shows error with mounting points create point. Ex:
nokrb->mount point does not exist
Step4 : Make sure fstab is pointing to the right boot device similar to
that of old image / new image
Step5 : Also make sure grub config file points to the same boot device as
fstab

Note:

Whatever folders/files are excluded they are done to preserve the hardware
configuration. So when updating make sure that the excluded file/folders
are compatible with that of new ones

To rerun bittorrent seeder if it dies:

/usr/bin/python /usr/bin/launchmany-console /var/lib/systemimager/torrents

--max_upload_rate 0 --rerequest_interval 1 --twisted 0 --
no_start_trackerless_client --no_upnp --bind 128.197.115.78

To use Bittorrent in general:

Run "maketorrent" on the machine that will be your tracker ("yum install bittorrent-gui" if
your machine doesn't already have it) and set the tracker hostname to that machine's
hostname. Make sure to "service iptables stop" so the ports will be open for serving the
torrent. Put the file you want to seed somewhere local on this computer, and select it and
click the button to make the torrent file. When this is done, don't bother clicking "seed" -- we
will use the command-line seeder. There will be a ".torrent" file in the directory where you
ran the maketorrent, and you can seed it from the command line. Put this torrent somewhere
where all the machines you want to receive the torrent on will be able to get at it (such as
/mnt/support/software/VM for the VMs). Now run:

bittorrent-tracker --port 6969 --dfile trackerdfile --logfile tracker.log &

btseed --max_upload_rate 0 /your/torrent/directory

Now that your tracker and seeder are running, on all the nodes you want to receive the torrent
on, run (in the directory where you want to receive the torrent):

bittorrent-console --max_upload_rate 0 /path/to/file.torrent

Here's an example of doing this on a whole bunch of nodes at once:

commandanybg signals 11 19 "cd /tmp; bittorrent-console --max_upload_rate 0
/mnt/support/software/VM/winxpSpr2011.zip.torrent"

Systemimager Excludes
On eng-imageserver:

/etc/systemimager/getimage.exclude -- this is a list of things that

"si_getimage --golden-client ..." will NOT pull up from ANY golden client

/var/lib/systemimager/overrides -- this contains a directory for each lab (as defined in

"si_clusterconfig -e") and can also take directories for individual machine hostnames that we
want to have special exceptions from the pattern even within a given lab. Note that vlsi26 is
the only individual machine in there, and that's because it has a special xorg.conf because it's
connected to a projector

On the individual machine (and in the actual image):

/etc/systemimager/updateclient.local.exclude
and /etc/systemimager/systemconfig.local.exclude

These are lists of things that will not be updated when the system takes an incremental update
("si_updateclient").

I think it makes the most sense to do everything with overrides, as that's the most flexible and
extensible method. Note that with something like the nvidia or fglrx closed-source video
drivers, the only override really needs to be the xorg.conf and the rc.local (which rebuilds the
kernel module when the machine takes a kernel update), as the rest of the stuff that the driver
installs can be in the image or not, and the machine won't care as long as the xorg.conf is not
actually using that driver.

P.S.: The one place where this gets hairy is if you for some reason need to use different
VERSIONS of the SAME driver on different systems that are all getting the same image, as
with a video card that supports an older version of the nvidia driver but not a newer one (like
the bme-compsim workstations, which use the 96-series driver). The master image has the
newer driver, and even though the bme-compsim workstations have the older driver set in
/etc/rc.local via the overrides, after a fresh reimage, they've still got the newer driver data on
their hard disks. I tried to solve this by having the /etc/rc.local actually reinstall the older
driver on top of the new one rather than just using the -K switch to rebuild the kernel module,
but for some reason, it doesn't actually run properly in the automatic rc.local run at boot time,
so you have to actually manually run "/etc/rc.local" after a fresh reimage of a Compsim
workstation. (I suspect the reason for this is weird shell incompatibilities with the nvidia
driver. I'm sure it's solvable, but I didn't want to bang my head against the wall forever trying
to figure it out, since manually rerunning /etc/rc.local for just these machines is only a minor
inconvenience.)