1

Installation of GPFS 3.5.0 on Windows Server 2008 R2

Author: Sean Lee, GPFS SA, Platform Computing Division
Date: 2013/09/17
Version: 1.0
Keyword: 3.5.0, 3.5.0-11, Windows, GPFS, OpenSSH

Introduction

In this document I describe how to install IBM GPFS 3.5.0 on a Windows-only

cluster consisting of 64-bit Windows Server 2008 R2 nodes.

This document is not a replacement for GPFS Concepts, Planning and

Installation Guide and/or public GPFS FAQs. Please review both of those
documents to plan and install GPFS.

The purpose of this document is to better explain how's and why's and to do
it in one simple, easy-to-understand document.

To SSH or ...

There are two approaches to inter-node communication with GPFS on

Windows. In my opinion the GPFS documentation makes this quite difficult to
understand and implement.

I would say the choice basically boils down to this:

1. If you're setting up a mixed-OS cluster (Linux, Windows) or if you
prefer to use SSH, use SSH for inter-node communication.
2. If you're setting up a Windows-only GPFS cluster, you can use either
SSH or the other approach mentioned in GPFS documents mentioned
above.

Since I'm not even mentioning what the other approach is, you're probably
guessing I'm sticking with SSH and if that's the case, you're right. Reasons
in favor of the other approach are the simplicity and speed of installation, but
I am not convinced.

Okay, now that is clear the next question is which SSH implementation to
use? IBM ships an IBM-packaged OpenSSH with GPFS 3.5.0-11, but you can
use any SSH server you want.

The advantage of IBM OpenSSH is that it nicely integrates with Services for
UNIX-based Applications and is predictable in terms of installation, as well as
supported by IBM, so we'll use that. Other approaches include Cygwin (some

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 2

people might like that) and so on, but I have not tried alternatives and don't
know how good or bad they are.

Get Ready

You may want to make sure you have this before you head out on-site:
• Windows updates for Windows Server 2008 R2 SP1
• Windows Server 2008 R2 SP1 QFEs (get all latest for server-related
fixes that were released after SP1 (I won't list them because I they
won't be "latest" just 2-3 weeks from now)). Quick Fix Engineering
(QFE) updates or "patches" are at this moment mandatory.
• Utilities and SDK for Subsystem for UNIX-based Applications
(~250MB in size)

Above are must-have requirements because GPFS cannot be installed and

configured without them.

These updates not only allow you to install GPFS, but also increase the
stability of the OS and therefore the entire GPFS cluster.

Once you're done with that, download and install latest driver (storage,
network, etc.) for your system.

And finally, you may want to prepare these useful tools:

• WinSCP - to upload and download various files over Open SSH protocol
• Putty - let's just say it's friendlier than the Korn shell you find on
Windows
• Filezilla - if you have to upload tons of files to Windows servers

As you can see, you may need to download close to 1GB of stuff, so get
everything that looks like you may need it before you begin.

Install Prerequisites

Install and configure Active Directory Service on a virtual or physical server.

It is not recommended to use GPFS servers to run ADS, although that can
work.

I have the habit of using domain "test.local" and here, because there are no
Linux nodes involved, I will not install any gizmos like the identity-mapping
add-on. Once you're done with this, restart the AD node and configure DNS
for all GPFS nodes.

Have the nodes join AD with fixed IP addresses and reboot them.

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 3

On the AD and GPFS nodes:

1. Go to Server Manager and chose to Add a Feature. Install the
Windows Subsystem for UNIX-based Applications (SUA).
2. Then install Utilities and SDK for Subsystem for UNIX-based
Applications. Here you need to be careful to use Custom Install. Pick
Base Utilities, GNU Utilities and Perl. In the Security Settings Panel
step, check all the three options (by default they are unchecked).
3. Reboot all the nodes

On the AD server:
1. Create a domain user (account) that we'll use to run OpenSSH (as
service). If we'll have Linux nodes join on later or if we're used to
using Linux, it may be best to use "root", so we create a domain user
<DOMAIN>\root (where <DOMAIN> is your domain name), give it a
profile path c:\Users\root\home (note the extra "home" - a
recommendation from the GPFS documentation) and add it to Domain
Admins.
2. In Global Policy Editor (see the official installation guide), allow
<DOMAIN>\root account to run services.

As <DOMAIN>\root, RDP to the GPFS nodes:

1. Disable Windows Firewall if you can. If you cannot, open IN and OUT
for TCP ports 22 and 1191 (see the GPFS documentation) and allow
Windows to alert you when something is being blocked by the
Windows Firewall service, so that you can get promoted and hit Allow
to allow GPFS components to run.
2. Enter all the nodes' IP's and hostnames in the Windows hosts file (just
in case AD fails to resolve hostnames). Do not do this if you're using
the HPC edition of Windows as the GPFS documentation mentions that
this is not allowed in that edition. Personally I prefer to define all IP
addresses (public, private) in the hosts file (e.g. 192.168.0.1 g1 and,
in another row, 10.0.0.1 g1private) so that I can resolve nodes'
names on both private and public GPFS network.
3. If you have anti-virus software, exclude GPFS mount point. If you
must scan GPFS, then enable it on just one server as it will impact the
performance.
4. Disable UAC (User Account Control). You can do that in Control Panel,
just pull the "level" of UAC to the bottom. You need to do this on all
GPFS nodes.

Install GPFS Packages

Login as Admin and install these on all the GPFS nodes, servers and clients.
There are CLI options in case you want to use CLI (msiexec.exe /package
<name>.msi /passive AgreeToLicense=yes).

1. gpfs-3.5-Windows-license.msi
2. gpfs-3.5.0.11-WindowsServer2008.msi

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 4

3. openssh-6.1.1.2.msi

If the second MSI fails to install, your Windows is not up to date (which it will
tell you).
The third is IBM's build of OpenSSH which we will use.

After you're done with all three, reboot.

Then go to Services Management Console on each GPFS node and find the
first GPFS service you see listed in alphabetic order. In the Log On tab, you
will see that the current setting is "Local System Account". We want to use
"This account" and here enter <DOMAIN>\root that we created for this
purpose. Enter this user name and password and restart the service.

Configure Passwordless SSH Login

This is the usual, well-known requirement. On all the nodes we need to do

two things:
1. Use ssh-keygen to create a local private/public key and then
disseminate it around to allow all folks with this key to allow the owner
to passwordless-ly access them using SSH protocol. To generate the
key, start the Korn shell on the node and change directory to
/usr/lpp/mmfs/platform/bin where you can run ssh-keygen.
2. Start SSH daemon with "/etc/init.d/sshd start". You can try to
SSH to the node from your workstation to make sure TCP port 22 is
not being blocked. Use <DOMAIN>\root to log on.

Then gather all the id_rsa.pub files and concatenate them together (use any
utility or text editor, including the vi editor available in the Korn shell on your
GPFS nodes, to combine all the nodes' id_rsa.pub files) in a file called
authorized_keys.

Then copy this authorized_keys file that contains all id_rsa.pub files to each
GPFS node (client and server). The destination directory is /.ssh. Use
WinSCP or CLI (scp authorized_keys root@node:/.ssh/authorized_keys).

Now you need to go to every node and run a test against all other nodes
(with 4 nodes you'd have to do 16 tests). Example commands you'd run on
n1:
ssh n1
ssh n2
ssh n3
ssh n4

You have to do this on all nodes using all nodes' names as they are known to
GPFS.

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 5

Configure GPFS Service

In this section there are following steps:

1. Create a cluster
2. Configure a disk
3. Start GPFS service
4. Create a filesystem
5. Mount a filesystem
6. Wash, rinse & repeat

To create a cluster, you'd use the same command as on Linux, mmcrcluster.

You can start with a single node, like I did for the node gpfswin1:
mmcrcluster -N "gpfswin1:quorum-manager" -p gpfswin1 -r /bin/ssh -R
/bin/scp
I have to accept the license agreement for the new node (for the server role):
mmlicense server --accept -N gpfswin1

Next up, prepare to configure at least one disk. Assuming OS is installed on

disk 1 (C drive), your GPFS disk may be disk 2. You don't need to go to Disk
Manager. Display, online (this is a verb) and initialize this disk with the
mmwindisk command (type "mmwindisk" to get help). Assuming this new disk
is disk number two, you'd create a file like this (excluding the remarks in
red):

%nsd: device=2
ordinal number of disk (from mmwindisk output)

nsd=array1disk2
arbitrary name you choose
servers=gpfswin1
the disk is visible to this/these servers
usage=dataAndMetadata
type of data this disk will store

You can use the exact same "disks stanza file" as above, just make sure that
device number is right for your environment and that your server name is
correct.

Alternatively, here's an example for 2 cluster nodes with 2 disks in a SAN

environment (notice how in a SAN environment multiple servers can see the
same disk if it's presented to both of them):

%nsd: device=2
nsd=array1disk2
servers=gpfswin1,gpfswin2
usage=dataAndMetadata
%nsd: device=3
nsd=array1disk3
servers=gpfswin2,gpfswin1
usage=dataAndMetadata

Now you save this template file to C:\disks.txt and use it to create the
disk(s) inside of GPFS CLI:
mmcrnsd -F /dev/fs/C/disks.txt

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 6

If you didn't do anything wrong, 60 seconds later you can run "mmlsnsd" to
see the result. The new disk should be listed as unused, but available.

We now need to start GPFS on "all" nodes (even we may have just one node)
in order to create a filesystem.
mmstartup -a
If you're starting multiple nodes, wait for 1-2 minutes. Then check the result:
mmgetstate -a
In case of a problem, examine GPFS logs (the locations and log file names
are the same on both Linux and Windows).

To create the filesystem (let's name it fs0) out of the same disks.txt file that
will be by default manually (no automount) mounted at G: (-t g):
mmcrfs fs0 -F /dev/fs/C/disks.txt -A no -t g
There are many options for this command and they may be important, so
consult the documentation if needed.
The new GPFS is ready to be mounted so I'll do it on all the running Windows
servers:
mmmount fs0 all
Since we specified the mount point (G:), that's where the FS will get
mounted to.

Now you can add additional nodes. For example I want to add gpfswin3
which happens to be a client.
mmaddnode -N "gpfs3:client"
Because this node hasn't been started, I'd start it ("mmstartup -N
<hostname>" or simply "mmstartup" on the node itself) and also mount the FS
("mmmount fs0").

Next I may want to use the mmchnode command to change its role, and
mmchcluster to change cluster-wide roles.
You can also add additional disks and create new pools, filesystems, etc.
Refer to GPFS documentation for details.

Optional: Simple Steps for Adding a Linux GPFS Client to

Windows GPFS Cluster

Although that is off-topic, it's short and simple so I'll add it here:
1. Add Linux host's primary IP to DNS as well as all Windows nodes'
hosts file (in the hosts files you can also add names of private
interfaces). Copy those to Linux node's /etc/hosts.
2. Generate SSH keys on Linux node and add its id_rsa.pub to
authorized_keys on all Windows nodes. Copy that file
(authorized_keys) to Linux location /root/.ssh/authorized_keys.
3. On Linux, disable or open the firewall (TCP/22, TCP/1191) if it has to
run.

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 7

4. Run "ssh <hostname>" between the new Linux host and all the
Windows nodes as well as the other way around to test password-less
SSH logon.
5. On Windows cluster manager use the mmaddnode command (e.g.
mmaddnode -N "<hostname>:client") to add the node, then accept the
license for the node (mmchlicense client --accept -N <hostname>).
6. Start service on the client (mmstartup) and mount GPFS (mmmount <fs>
-t <mountpoint>)

Stopping GPFS

To unmount fs0 on Windows:

1. Shutdown all applications and consoles that are using the FS (e.g.
Windows Explorer in g:\test and your PostgreSQL DB)
2. Dismount the FS ("mmumount fs0 all" to do this across the cluster, or
"mmumount fs0 -N <node>" )

To shutdown GPFS service:

mmshutdown -a

Restarting the OS and GPFS Service

If GPFS isn't configured to auto-load, you need to start it up (the mmstartup

command described in text above).
If fs0 isn't configured to auto-mount, you need to mount it using the mmmount
command. Usually it takes a minute for the cluster to get ready to mount
filesystems.

Two common problems causing start-up issues:

1. SSH service isn't running
Start it from Korn shell on Windows:
/etc/init.d/sshd start
2. GPFS Startup Service isn't running
Start it from Services Management Console (or CLI).

Getting Help

Contact Sean Lee (seanlee@tw.ibm.com) or post your error and problem

description on the Wiki forum here:

https://w3-
connections.ibm.com/communities/service/html/communityview?community

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee
 8

Uuid=f33468ab-0e06-4ed6-b866-
5f17b8c21537#fullpageWidgetId=Wfbeac23f99a6_4e70_95d2_8dc289f674e
2&forumsPg=null&topicsPg=null

---

GPFS 3.5.0 on Windows: Installation and Configuration How-To

by Sean Lee