Sap Hana On Rhel On Ibm Xserver Using Gpfs

RED HAT ENTERPRISE LINUX FOR SAP
HANA ON IBM:
INSTALLATION GUIDE
August 2014
Version 2.1
Copyright 2014 Red Hat, Inc. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, and GlusterFS are
trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux is the registered trademark of
Linus Torvalds in the U.S. and other countries. Java is the registered trademark of Oracle America, Inc. in the
United States and other countries. All other trademarks are the property of their respective owners.
www.redhat.com
Disclaimer
This document has been carefully crafted to assist with setup of SAP HANA on Red Hat Enterprise Linux 6 on IBM X-Series
Servers and is subject to change without notification and will not cover the issues encountered in every customer situation. It
should be used only in conjunction with the official product literature from Red Hat and IBM. The information contained in this
document has not been submitted to any formal Red Hat or IBM test and is distributed AS IS.
All statements regarding Red Hat or IBM future direction and intent are subject to change or withdrawal without
notice, and represent goals and objectives only.
Altering the configuration by adding or removing software components will result in a non-certified configuration and void
warranty and support.
This document is for educated service personnel only. If you are not familiar with the described system, we will ask you to
restrain from trying to apply what is described herein you could void the preloaded system installation and void the SAP
certified configuration. This will void the warranty and support of said machine. Please contact the Red Hat at the SAP Linux
Labs or IBM SAP International Competence Center (ISICC) to get enrolled for education prior to installing an IBM Workload
Optimized Solution for SAP HANA appliance.
In case of issues with the SAP HANA appliance, the customer is asked to open a SAP Help Desk request (OSS ticket) first and
foremost. Only by following this path, can we ensure the proper configuration of the IBM Workload Optimized Solution for SAP
HANA. If the customer would open a Red Hat or IBM support ticket for the system, he might be requested to perform system
upgrades to firmware or software to the latest available levels which might not be supported with the SAP HANA appliance. If
identified as a hardware or file system issue, the ticket will be forwarded to the IBM support team and handled appropriately.
Although this may be contrary to standard IBM Support processes, it is the approved and accepted support process for all SAP
Appliances including the SAP HANA appliance.
Red Hat Enterprise Linux for SAP HANA: Installation Guide | August 2014 2
Table of Contents
Setting up Single Node and Cluster Configurations with GPFS and SAP HANA......3
Installing the nodes First steps.................................................................................................................4
Manual Installation of Red Hat Enterprise Linux for SAP HANA...........................................................4
Network configuration............................................................................................................................ 8
GPFS Deployment..................................................................................................................................... 10
Installation of the GPFS packages and configuration files...................................................................10
Preparing the nodes and building the GPFS kernel modules..............................................................13
Configuring GPFS................................................................................................................................ 14
Installation of SAP HANA..........................................................................................................................17
Unattended Installation using Kickstart.....................................................................................................18
Kickstart RHEL for SAP HANA............................................................................................................19
Installing GPFS using the scripts from Kickstart..................................................................................22
Installing SAP HANA using the scripts from Kickstart..........................................................................22
References and Notes..................................................................................................23

Changelog................................................................................................................................................. 24
SETTING UP SINGLE NODE AND CLUSTER

CONFIGURATIONS WITH GPFS AND SAP HANA
This document describes the installation of Red Hat Enterprise Linux (RHEL) for SAP HANA on a
standalone IBM X-Series Server as well as a cluster of IBM X-Series Servers for using SAP HANA in
combination with the cluster file system GPFS. For GPFS, also single-node configurations are regarded as
cluster; they just consist of one node. There are two installation modes described, a manual installation as
well as an automatic installation using Kickstart.
General recommendations for setting up RHEL 6 servers for running SAP HANA can be found in the
following SAP Notes:
SAP Note 2009879
- SAP HANA Guidelines for Red Hat Enterprise Linux (RHEL) Operating System
SAP Note 2013638 - SAP HANA DB: Recommended OS settings for RHEL 6.5
SAP Note 2001528 - Linux: SAP HANA Database SPS 08 revision 80 (or higher) on RHEL 6 or SLES 11
SAP Note 1084263 - 1084263 - Cluster File System: Use of GPFS on Linux
In this guide, a single-node configuration and a three-node cluster configuration are described step by step.
The host name used for the single node setup is hanamachine01; for the three-node setup the
hostnames used are hanamachine0{1,2,3}. In general, all machines used are IBM System X-Series
3850 X6 which are certified according to the IBM Quick Start Guide. [1] (See the References and Notes
section at the end of this document.)
The initial network configuration is as follows:
Host name
hanamachine03
hanamachine02
hanamachine01
MAC-Address (i350)
40:f2:e9:30:11:c0
40:f2:e9:30:11:0c
40:f2:e9:30:08:5c
IP-Address (i350)
10.68.204.178
10.68.206.124
10.68.206.123
Netmask (i350)
255.255.252.0
255.255.252.0
255.255.252.0
Gateway (i350)
10.68.204.1
10.68.204.1
10.68.204.1
Installing the nodes First steps

First, install Red Hat Enterprise Linux for SAP HANA on all machines. The RAID configuration was already
done by the OEM. Please refer to the Implementation Guide [9] for details.
Connect the remote CD/DVD image to the Integrated Memory Module (IMM).
Boot the machine and select F12 for the boot menu.
Choose the boot option IMM1:CD/DVD - IMM Remote Mount for booting from the virtual installation
DVD.
Manual installation of Red Hat Enterprise Linux for SAP HANA

For general information on installing Red Hat Enterprise Linux please refer to the Red Hat Enterprise
Linux 6 Installation Guide [2].
When the black screen pops up choose the entry, Red Hat Enterprise Linux.
When the GRUB boot screen appears press a key before the countdown runs out. You should now
see the following boot selection menu:
Navigate to Install system with basic video driver and press [Enter].
Install the operating system on the first hard drive /dev/sda with about 1 TB in size.
Choose minimal installation and configure the primary network adapter according to the settings
above. Accept the default partitioning scheme (Use all space).
If connection to the Red Hat Customer Portal or a local Red Hat Satellite server is possible please
register the system there as described in the Red Hat Enterprise Linux 6 Installation Guide [2]
and subscribe the system to the RHEL for SAP HANA channel.
If connection to the RH Customer Portal is not possible, check if the Installaton DVD is already
mounted. If this is not the case, mount it, e. g. to /cdrom. After mounting the Installation
DVD, create the file /etc/yum.repos.d/rhel-dvd.repo with the following content:
[RHEL6-DVD]
name=dvd
baseurl=file:///<dvd mount point>
gpgcheck=1
enabled=1
Wipe out all cached yum data with:
# yum clean all
Now install the required packages with:
# yum -y groupinstall base infiniband

# yum -y install gtk2 libicu xulrunner ntp sudo tcsh libssh2 \
rsyslog expect cairo iptraf krb5-workstation krb5-libs.i686 \
nfs-utils lm_sensors libtool-ltdl xauth openssl098e gcc gcc-c++ \
kernel-headers kernel-devel PackageKit-gtk-module libcanberra-gtk2 \
cpufrequtils libunwind unixODBC infiniband-diags perftest qperf \
opensm ibutils lsscsi ipmitool tuned icedtea-web
Install the RHEL for SAP HANA compatibility package. You can find it in the child channel.
Alternatively, you can use the RHEL for SAP HANA ISO image which should have been provided
as part of the installation package. Mount it via loopback using:
# mount -o ro,loop <rhel-for-sap-hana.iso> /mnt
Change to the /mnt/Packages directory and install the compat-sap-c++ package:
# yum localinstall compat-sap-c++[tab]
Disable SELinux. SAP HANA is currently not compatible with SELinux:
# sed -i s/^SELINUX=*/SELINUX=disabled/g \ /etc/sysconfig/selinux
A precise clock is crucial for database applications, especially in a clustered environment.

Therefore NTP should be configured on all HANA nodes:
Modify /etc/ntp.conf to:
server <timehost>.fqdn
Turn NTP off, set the hardware clock and turn it on again:
#
#
#
#
service ntpd stop

ntpdate
service ntpd start
chkconfig ntpd on
The ntpdate service script adjusts the time according to the ntp server every time when the
system comes up. This happens before the regular ntp service is started and ensures an exact
system time even if the time deviation is too big to be compensated for by the ntp service. For
details please refer to note [10].
# echo <timehost>.fqdn >> /etc/ntp/step-tickers

# chkconfig ntpdate on
Disable the following services to avoid overhead:
#
#
#
#
#
#
kdump off
abrtd off
abrt-cpp off
rhsmcertd off
NetworkManager off
opensm off
Make sure that some essential services are enabled:
#
#
#
#
chkconfig
chkconfig
chkconfig
chkconfig
chkconfig
chkconfig
chkconfig
chkconfig
chkconfig
chkconfig
network on
ntpd on
rdma on
tuned on
Create some symbolic links required for running SAP HANA:
#
#
#
#
ln
ln
ln
ln
-s
-s
-s
-s
/usr/lib64/libssl.so.0.9.8e /usr/lib64/libssl.so.0.9.8
/usr/lib64/libssl.so.1.0.1e /usr/lib64/libssl.so.1.0.1
/usr/lib64/libcrypto.so.0.9.8e /usr/lib64/libcrypto.so.0.9.8
/usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1
The usage of Transparent Hugepages (THP) is not supported when running SAP HANA and IBM GPFS.
Therefore THP should be disabled permanently. To disable THP on boot, add the option,
transparent_hugepage=never at the end of all lines stating with kernel in the file
/boot/efi/EFI/redhat/grub.conf.
Transparent Hugepages can be disabled on the running system with the following command:
# echo never > /sys/kernel/mm/transparent_hugepage/enabled
To tune the systems for best performance, activate the tuned with the latency-performance
profile:
# tuned-adm profile latency-performance

# service tuned start
To prevent fork-bombs, RHEL restricts the maximum amount of processes per user. For hardware
with 8 IvyTown sockets and larger, the maximum number of processes for the sapsys group isn't
sufficient. The symptom is that the installation of HANA fails in this case. The solution is to create
the file, /etc/security/limits.d/99-sap.conf
@sapsys soft nproc unlimited
Keep in mind that this can be a potential security issue.

Disable core file creation when processes crash by adding the following to
/etc/security/limits.d/99-sap.conf:
* soft core 0
* hard core 0
Network configuration
For a deep dive in networking using RHEL, please refer to the Red Hat Enterprise Linux 6 Deployment
Guide Part III - Networking [3].
The layout the network configuration is based upon at least one of three network segments:
The primary network connection for things like administrator access
The network for the gpfsnode0n bonded interfaces used for GPFS communication
In case of a single-node configuration the IP address of gpfsnode01 is set to
127.0.1.1/8
The network for the communication used by SAP HANA itself, hananode0n
In case of a single-node configuration the IP address of hananode01 is set to
127.0.2.1/8
First of all, we will have to determine which network interface will be used for the primary network.
In this case, it is an Intel i350 type which is connected to the internal network infrastructure and is
detected as eth4. To make sure you choose the connected one, you can run:
# ethtool -p eth4 30
The interface LED blinks for 30 seconds and can be identified easily.
Configure the primary network interface. Modify /etc/sysconfig/network-scripts/ifcfgeth4 to:
DEVICE=eth4
BOOTPROTO=static
IPADDR=<IP Address>
GATEWAY=<Gateway Address>
PREFIX=<Desired Prefix in decimal>
HWADDR=<MAC Address>
ONBOOT=yes
TYPE=Ethernet
DNS1=<DNS Server 1>
DNS2=<DNS Server 2>
IPV6INIT=no
USERCTL=no
Configure the bonding interfaces. In this case we have Mellanox Infiniband cards in Ethernet mode.
The interfaces to be grouped in this example are eth{0,2} and eth{1,3}. The files
/etc/sysconfig/network-scripts/ifcfg-eth{0..3} look like this:
DEVICE=eth{0..3}
USERCTL=no
ONBOOT=yes
HWADDR=<MAC Address>
SLAVE=yes
MASTER=bond{0,1}
Now we can define the bonded interfaces. These are defined in /etc/sysconfig/networkscripts/ifcfg-bond{0,1}:
DEVICE=bond{0,1}
IPADDR=<IP-Address>
NETWORK=<Network Address>
NETMASK=<Netmask in 4 digits>
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
MTU=9000
We use jumbo frames (MTU=9000) here to maximize performance. Please check with your Network
Management team to make sure jumbo frames are enabled on all switches etc.
Bonding needs a kernel module with the appropriate options. To load it when activating bonding,
create the file /etc/modprobe.d/bonding.conf:
alias bond0 bonding

options bond0 mode=802.3ad xmit_hash_policy=layer3+4 miimon=100
alias bond1 bonding
options bond1 mode=802.3ad xmit_hash_policy=layer3+4 miimon=100
The option mode=802.3ad activates channel aggregation to maximize performance. Please check
with your Network Management team to make sure that it is possible to use this mode, otherwise
use a mode that is supported in the network environment.
Check the file /etc/udev/rules.d/70-persistent-net.rules for proper mapping between
the network devices and MAC-addresses.
Check the file /etc/sysconfig/network. It should contain:
NETWORKING=yes
HOSTNAME=<Short host name>
The HOSTNAME entry must be the short one, not FQDN. Otherwise, SAP HANA refuses to operate.
You can get it by running hostname -s.
Make sure that the DNS domain names and DNS servers are correctly configured in
/etc/resolv.conf:
search <Domain, e.g. example.com>

nameserver <DNS Server 1>
nameserver <DNS Server 2>
A working name resolution is required at all times. Using DNS in a clustered environment can be
dangerous because if the DNS service fails, the cluster would be inoperable. To avoid that the
cluster fails to operate in case the DNS servers aren't reachable, the best practice is to make sure
that /etc/nsswitch.conf contains the following line:
hosts:
files dns
Then list all the cluster members in the file /etc/hosts on each node.
In our three-node example this would be:
192.168.206.123
192.168.123.102
192.168.234.102
192.168.206.124
192.168.123.101
192.168.234.101
192.168.204.178
192.168.123.103
192.168.234.103
hanamachine01 hanamachine01.example.com
gpfsnode01 gpfsnode01
hananode01 hananode01
In a single-node environment, just the first node would be used:
192.168.206.123 hanamachine01 hanamachine01.example.com

127.0.1.1 gpfsnode01 gpfsnode01
127.0.2.1 hananode01 hananode01
To ensure that the traffic inside the cluster of a multi-node configuration is piped through the
internal HANA network in order to gain throughput, a route to each node must be set on each node
[4]. In our three-node example we would set them in /etc/sysconfig/networkscripts/route-bond1:
GATEWAY0=0.0.0.0
NETMASK0=255.255.255.255
ADDRESS0=192.168.206.123
GATEWAY1=0.0.0.0
NETMASK1=255.255.255.255
ADDRESS1=192.168.206.124
GATEWAY2=0.0.0.0
NETMASK2=255.255.255.255
ADDRESS2=192.168.204.178
GPFS deployment
The following steps have been taken from the Implementation Guide for SAP HANA Appliance [5]. The
guide is intended for a different platform and the steps were adapted to RHEL for SAP HANA. Please refer
to this document to obtain further information.
Installation of the GPFS packages and configuration files

The minimum maintenance level of GPFS for running RHEL for SAP HANA on IBM HANA appliances
is 3.5.0-17.
In general, you need to download the packages required for running GPFS directly from IBM or install them
from your local repository or custom channel. This procedure must be repeated on every node in the cluster.
First, install the base package:
# yum -y install gpfs.base
Ensure that the base package is updated to the latest release and install additional packages
required for running GPFS:
# yum -y update gpfs.base

# yum -y install gpfs.docs gpfs.gpl gpfs.msg.en_US
We need a path extension for the GPFS tools and several directories.
Create the file /etc/profile.d/gpfs.csh:
# setenv PATH ${PATH}:/usr/lpp/mmfs/bin
Create the file /etc/profile.d/gpfs.sh:
# export PATH=$PATH:/usr/lpp/mmfs/bin
Now set the permissions appropriately:
# chmod 644 /etc/profile.d/gpfs.csh

# chmod 644 /etc/profile.d/gpfs.sh
The following directories need to be created: the first one for temporary data, the second one for
the GPFS configuration files:
# mkdir -p /tmp/GPFSdump
# mkdir -p /var/mmfs/config
A number of configuration files are required to start GPFS successfully.

The following configuration is an example. You may see more or fewer block devices on your
machine depending on its configuration. Every %nsd-entry requires one dedicated block
device. The first block device, e.g. /dev/sda must not be used because it is normally used to
hold the operating system.
The first configuration file on node n (see /etc/hosts above which host name corresponds
to which node) is /var/mmfs/config/disk.list.data.gpfsnode0n
%nsd: device=/dev/sdb
nsd=data01node0n
servers=gpfsnode0n
usage=dataAndMetadata
failureGroup=100n
pool=system
%nsd: device=/dev/sdc
nsd=data02node0n
servers=gpfsnode0n
usage=dataAndMetadata
failureGroup=100n
pool=system
In this example the two disks dedicated to GPFS are /dev/sd{b,c}.

The second configuration file required is /var/mmfs/config/nodes.list.
In a three-node environment the file should contain the following:
gpfsnode01
gpfsnode02
gpfsnode03
If there is just a single-node configuration the file should contain:
gpfsnode01
The third and last configuration file /var/mmfs/config/nodes.cluster holds the quorum
information.
In a three-node cluster we use:
gpfsnode01:quorum
gpfsnode02:quorum
gpfsnode03:quorum
If there is just a single node configuration, we omit the other nodes here:
gpfsnode01:quorum
Preparing the nodes and building the GPFS kernel modules
For the compilation of the GPFS kernel module, some additional packages must be installed if not
already done:
# yum -y install gcc gcc-c++ kernel-headers kernel-devel
To avoid a reboot of the machine set the PATH variables required manually:
# export PATH=$PATH:/usr/lpp/mmfs/bin
Wipe the signature of the disks. WARNING: make sure to use the correct devices to prevent data
loss:
# dd if=/dev/zero of=/dev/sdb bs=1G count=1

# dd if=/dev/zero of=/dev/sdc bs=1G count=1
GPFS accesses the other nodes using SSH. Therefore SSH keys need to be set up on all nodes.
For a three-node cluster, generate a RSA-key and distribute it to all other nodes and
localhost for SSH-access of GPFS:
# NODES=(hanamachine0{1,2,3} gpfsnode0{1,2,3} hananode0{1,2,3})

# ssh-keygen -t dsa -N "" -f ~/.ssh/id_dsa
# for node in ${NODES[@]}; do
ssh-copy-id -i /root/.ssh/id_dsa.pub $node
done
For a single-node configuration, generate a RSA-key and distribute it just to localhost for
SSH-access of GPFS:
# NODES=(hanamachine01 gpfsnode01 hananode01)

# ssh-keygen -t dsa -N "" -f ~/.ssh/id_dsa
# for node in ${NODES[@]}; do
ssh-copy-id -i /root/.ssh/id_dsa.pub $node
done
Now we can build the kernel module and the portability layer. Keep in mind that after every kernel
update this process must be repeated:
#
#
#
#
cd /usr/lpp/mmfs/src
make Autoconfig
make World
make InstallImages
Configuring GPFS
The majority of the configuration steps are only done on the master node. In this example, it is
hanamachine01 which represents node 2 and therefore corresponds to gpfsnode02.
For a three-node cluster, create the GPFS cluster on the master node with:
# mmcrcluster -n /var/mmfs/config/nodes.cluster -p gpfsnode01 \

-s gpfsnode02 -r /usr/bin/ssh -R /usr/bin/scp -C HANAcluster -A
For a single-node configuration, just include this node with:
# mmcrcluster -n /var/mmfs/config/nodes.cluster -p gpfsnode01 \

-r /usr/bin/ssh -R /usr/bin/scp -C HANAcluster -A
On a three-node configuration assign a GPFS server license on the master node to all nodes in the
cluster:
# mmchlicense server --accept -N gpfsnode01,gpfsnode02,gpfsnode03
If there are more than three servers, use the first three as quorum nodes. The quorum nodes
need server licenses. The remaining nodes must get the FPO license assigned:
On a single-node configuration, just assign the node itself:
# mmchlicense fpo --accept -N gpfsnode0X,gpfsnode0Y,gpfsnode0Z

# mmchlicense server --accept -N gpfsnode01
Create the file /var/mmfs/config/disk.list.data.fs
On the three-node cluster, create a combined disk file on the master node with:
#rm -f /var/mmfs/config/disk.list.data.fs
# for i in {1..3}; do
scp gpfsnode0$i:/var/mmfs/config/disk.list.data.gpfsnode0$i \
/var/mmfs/config
cat /var/mmfs/config/disk.list.data.gpfsnode0$i >> \
/var/mmfs/config/disk.list.data.fs
done
On a single-node configuration, create a disk file for the node itself:
# rm -f /var/mmfs/config/disk.list.data.fs
# cat /var/mmfs/config/disk.list.data.gpfsnode01 >> \
/var/mmfs/config/disk.list.data.fs
On every node, create the GPFS disks with:
# mmcrnsd -F /var/mmfs/config/disk.list.data.gpfsnode0n
Adjust various GPFS settings on the master node:
# mmchconfig dataStructureDump=/tmp/GPFSdump,pagepool=16G,\
maxMBpS=2048,maxFilesToCache=4000,nsdInlineWriteMax=1M,\
readReplicaPolicy=local,enableLinuxReplicatedAIO=yes,\
restripeOnDiskFailure=yes
Important: There must be no tabs or spaces between the comma-separated arguments of

mmchconfig. Otherwise, the command doesn't work and GPFS will come up with a wrong
configuration.
On the master node, start GPFS:
# mmstartup -a
Create the GPFS file system.

On the master node for a multi-node cluster use the following command:
# mmcrfs sapmntdata -F
-B 1M -N 3000000 -v no
--write-affinity-depth
--block-group-factor 1
/var/mmfs/config/disk.list.data.fs -A yes \
-m 2 -M 3 -r 2 -R 3 -j hcluster \
1 -s failureGroupRoundRobin \
-Q yes -T /sapmnt
On a single-node configuration, create the file system with the following:
# mmcrfs sapmntdata -F /var/mmfs/config/disk.list.data.fs -A yes \

-B 1M -N 3000000 -v no -m 1 -M 3 -r 1 -R 3 -Q yes -T /sapmnt
On the master node, create the filesets needed for SAP HANA:
# mmcrfileset sapmntdata hanadata -t "Data Volume for HANA database"

# mmcrfileset sapmntdata hanalog -t "Log Volume for HANA database"
# mmcrfileset sapmntdata hanashared -t \
"Shared Directory for HANA database"
On the master node, mount the GPFS file system with:
# mmmount /sapmnt -a
On the master node, link the filesets into the file system:
# mmlinkfileset sapmntdata hanadata -J /sapmnt/data

# mmlinkfileset sapmntdata hanalog -J /sapmnt/log
# mmlinkfileset sapmntdata hanashared -J /sapmnt/shared
On the master node, check if the filesets have been properly created:
# mmlsfileset sapmntdata
Define quotas. This can be done using a helper script to be obtained from IBM:
# sh ./saphana-quota-calculator.sh -a
GPFS should now be running on all nodes. To check, the following can be used:
# df -hT # -> /sapmnt

# mmdf /dev/sapmntdata
Some permissions need to be set. On the master node, set these to:
# chmod -R 755 /sapmnt/shared

# chmod -R 777 /sapmnt/data
# chmod -R 777 /sapmnt/log
Installation of SAP HANA

The last step is the installation of the database which is done on the master node exclusively. For details on
installing SAP HANA please refer also the SAP HANA Server Installation and Update Guide [6]. The
installation will be done in unattended mode. The following things are required for this step:
A password file called /root/password.xml. It has the following appearance:
<?xml version="1.0" encoding="UTF-8"?>

<Passwords>
<password>5WqX59!,</password>
<sapadm_password>5WqX59!,</sapadm_password>
<system_user_password>5WqX59!,</system_user_password>
<root_password>redhat</root_password>
</Passwords>
Replace the values between the brackets in the XML file with the following values according to
the requirements of the customer:
password
sapadm_password
system_user_password
root_password
The archive of the SAP HANA database software IMDB_SERVER<current version>.SAR

which can be obtained from the SAP Service Marketplace.
Copy the archive to the /tmp directory.
The archive utility SAPCAR_<current version>.exe which is also available from the SAP
Service Marketplace.
Copy the archive utility to the local /root/bin directory.
Extract the archive with:
# cd /tmp
# /<path to>/SAPCAR_<current version>.exe -xvsf IMDB_SERVER<current
version>.SAR
Change the directory to:
# cd /tmp/SAP_HANA_DATABASE
For stability reasons, the memory usage of SAP HANA should be limited. The recommendation is
as follows:
Memory <= 256 MiB: max_mem = 80%
Memory > 256 MiB: max_mem = 90%
# TOT_MEM=$(echo $(cat /proc/meminfo | grep MemTotal | \

# awk '{print $2}')/1024 | bc | cut -d. -f1)
# if [ $TOT_MEM -le 262144 ]; then
MAX_MEM=$(echo $TOT_MEM*0.8 | bc | cut -d. -f1)
else
MAX_MEM=$(echo $TOT_MEM*0.9 | bc | cut -d. -f1)
fi
For the installation of SAP HANA, the customer can choose the SID and Instance number (options
--sid= and number=).
On our three-node cluster example, we can now install the software in unattended mode with:
# LD_PRELOAD=/opt/rh/SAP/lib64/compat-sap-c++.so \
cat /root/password.xml | /tmp/SAP_HANA_DATABASE/hdblcm \
--read_password_from_stdin=xml --sid=ANA --number=00 \
--sapmnt=/sapmnt/shared --datapath=/sapmnt/data/ANA \
--logpath=/sapmnt/log/ANA \
--shell=/bin/bash --hostname=hanamachine01 -b --system_usage=test \
--autostart=1 --components=server,client --timezone=UTC \
--max_mem=$MAX_MEM \
--addhosts=hanamachine02:role=worker,hanamachine03:role=worker \
--internal_network=192.168.234.102/24 \
--home=/usr/sap/ANA/home --userid=1100 --groupid=79 \
--certificates_hostmap=hananode01=hanamachine02.wdf.sap.corp,\
hananode02=hanamachine01.wdf.sap.corp,\
hananode03=hanamachine03.wdf.sap.corp --vm=/usr/bin/java
On the single-node configuration, SAP HANA can be installed as follows:
# LD_PRELOAD=/opt/rh/SAP/lib64/compat-sap-c++.so \
cat /root/password.xml | /tmp/__installer.HDB/hdblcm
--read_password_from_stdin=xml --sid=ANA --number=00 \
--sapmnt=/sapmnt/shared --datapath=/sapmnt/data/ANA \
--logpath=/sapmnt/log/ANA \
--shell=/bin/bash --hostname=hanamachine01 -b --system_usage=test \
--autostart=1 --components=server,client --timezone=UTC \
--max_mem=\$MAX_MEM --vm=/usr/bin/java
After a couple of minutes, SAP HANA should be up and running on all nodes. This can be observed
on each node with:
# su - anaadm
# watch -n1 HDB info
Unattended Installation using Kickstart

To perform an unattended installation, for each node a Kickstart file needs to be prepared. A Kickstart file is
an answer file which is passed over to anaconda, the system installer of RHEL for SAP HANA. It contains all
the answers a user would have made to the installer if he had chosen an interactive installation process. In
addition, the Kickstart file can also contain %pre and %post scripts for automating the configuration
process, e.g. the creation and modification of configuration files or license agreements before the
installation. In this example the Kickstart file also installs scripts for installing and configuring GPFS and SAP
HANA. Please refer also to notes [7] and [8].
Kickstart RHEL for SAP HANA
Begin with the first steps described in the first section of this guide. Start booting with the virtual
DVD.
When the black screen with the countdown pops, up press a key. You are now delegated to the
installation menu.
Navigate to Install system with basic video driver and press a to append the
following to the kernel command line:

hanamachine01 (single-node):
ks=http://10.18.98.198/hanamachine01_single.cfg \
ksdevice=40:f2:e9:30:05:58 ip=10.68.204.181 netmask=255.255.252.0 \
gateway=10.68.204.1
hanamachine01 (multi-node):
ks=http://10.18.98.198/hanamachine01.cfg \
ksdevice=40:f2:e9:30:08:5c ip=10.68.206.123 netmask=255.255.252.0 \
gateway=10.68.204.1
hanamachine02:
ksdevice=40:f2:e9:30:11:0c ip=10.68.206.124 netmask=255.255.252.0 \
gateway=10.68.204.1
hanamachine03:
ksdevice=40:f2:e9:30:11:c0 ip=10.68.204.178 netmask=255.255.252.0 \
gateway=10.68.204.1
Press [Enter] to start the machine. Sometimes, due to network problems it cannot access the
network during the first couple of attempts.
Check for the URL of the Kickstart file and press the tab key. As soon the OK button is highlighted,
press [Space] to confirm. If the problem persists, wait a few seconds and press enter again. Also,
due to these problems the installer cannot see the URL of the RHEL distribution.
Move the cursor using the [Tab] key to URL and choose OK.
As the URL to the distribution, enter ftp://10.18.98.198/pub/rhel6/dvd here.
Now you are asked to confirm the license agreement.

If the Reject button is pressed, the installation process will be stopped and the machine is
rebooted.
All file systems are created and all packages and package groups listed in the Kickstart file in the
%packages section are installed now using Kickstart.
Installing GPFS using the scripts from Kickstart

In general, GPFS must be installed on all three nodes. After reboot of all machines of the cluster right after
the installation, you can see three scripts in the /root directory:
All nodes: /root/install.gpfs.sh

All nodes: /root/configure.gpfs.sh
Master node: /root/install.hana.sh
Install GPFS on all three nodes using:
# /root/install.gpfs.sh
Check the output for errors or warnings.

Create the cluster on the master node gpfsnode02.
On the master node launch:
# /root/configure.gpfs.sh
On all other nodes launch:
# /root/configure.gpfs.sh
Create the file system on the master node with:
# /root/create.gpfs.sh
Now check the installation on each node with:
# df -hT
# mmdf /dev/sapmntdata
Installing SAP HANA using the scripts from Kickstart
The last step is to install SAP HANA on the master node using:
# /root/install.hana.sh
After a couple of minutes, SAP HANA should be up and running on all nodes. This can be observed
on each node with:
# su - anaadm
# watch -n1 HDB info
REFERENCES AND NOTES

Number
Remarks
[1]
http://www-947.ibm.com/support/entry/portal/docdisplay?
lndocid=MIGR-5087035
[2]
https://access.redhat.com/documentation/enUS/Red_Hat_Enterprise_Linux/6/htmlsingle/Installation_Guide/index.html
[3]
https://access.redhat.com/documentation/enUS/Red_Hat_Enterprise_Linux/6/htmlsingle/Deployment_Guide/index.html#part-Networking
[4]
Verbal message from F. Bausch on July 02, 2014
[5]
http://www03.ibm.com/support/techdocs/atsmastr.nsf/5cb5ed706d254a8186256c
71006d2e0a/f1359b7fd3cce99786257ca500214577/$FILE/IBM%20-%20SAP
%20HANA%20Implementation%20Guide%20X6-1.7.73-9.pdf
[6]
http://www.saphana.com/servlet/JiveServlet/download/1031-1314351/SAP_HANA_Server_Installation_Guide_en.pdf
[7]
https://access.redhat.com/documentation/enUS/Red_Hat_Enterprise_Linux/6/htmlsingle/Installation_Guide/index.html#ptAdvanced_installation_options
[8]
http://fedoraproject.org/wiki/Anaconda/Kickstart
[9]
http://www03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD106247
10
https://access.redhat.com/documentation/enUS/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s1Configure_ntpdate_Servers.html
Changelog
Date
Version
Author
Remarks
18.06.2014
1.0
Jrgen Blume
Initial version of the document
20.06.2014
1.1
Jrgen Blume
Correct usage of mmchconfig
25.06.2014
1.2
Jrgen Blume
Some typos fixed
25.06.2014
1.3
Jrgen Blume
Generic host names
30.06.2014
1.4
Jrgen Blume
Added single-node instructions
02.07.2014
1.5
Jrgen Blume
Added reference section and routes
08.07.2014
1.6
Jrgen Blume
Modified the disclaimer
04.07.2014
1.7
Joachim Kunze
Editor changes
07.07.2014
1.8
Frank Danapfel
General updates
09.07.2014
1.9
Jrgen Blume
Minor updates
11.08.2014
2.1
Joachim Kunze
Minor updates

Sap Hana On Rhel On Ibm Xserver Using Gpfs

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Sap Hana On Rhel On Ibm Xserver Using Gpfs

Hochgeladen von

Copyright:

Verfügbare Formate

RED HAT ENTERPRISE LINUX FOR SAP

References and Notes..................................................................................................23

SETTING UP SINGLE NODE AND CLUSTER

Installing the nodes First steps

Manual installation of Red Hat Enterprise Linux for SAP HANA

Wipe out all cached yum data with:

# yum clean all

Now install the required packages with:

# yum -y groupinstall base infiniband

# mount -o ro,loop <rhel-for-sap-hana.iso> /mnt

Change to the /mnt/Packages directory and install the compat-sap-c++ package:

# yum localinstall compat-sap-c++[tab]

Disable SELinux. SAP HANA is currently not compatible with SELinux:

# sed -i s/^SELINUX=*/SELINUX=disabled/g \ /etc/sysconfig/selinux

A precise clock is crucial for database applications, especially in a clustered environment.

service ntpd stop

# echo <timehost>.fqdn >> /etc/ntp/step-tickers

Disable the following services to avoid overhead:

Make sure that some essential services are enabled:

Create some symbolic links required for running SAP HANA:

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

# tuned-adm profile latency-performance

@sapsys soft nproc unlimited

Keep in mind that this can be a potential security issue.

alias bond0 bonding

search <Domain, e.g. example.com>

In our three-node example this would be:

In a single-node environment, just the first node would be used:

192.168.206.123 hanamachine01 hanamachine01.example.com

Installation of the GPFS packages and configuration files

First, install the base package:

# yum -y install gpfs.base

# yum -y update gpfs.base

# setenv PATH ${PATH}:/usr/lpp/mmfs/bin

Create the file /etc/profile.d/gpfs.sh:

Now set the permissions appropriately:

# chmod 644 /etc/profile.d/gpfs.csh

A number of configuration files are required to start GPFS successfully.

In this example the two disks dedicated to GPFS are /dev/sd{b,c}.

If there is just a single-node configuration the file should contain:

Preparing the nodes and building the GPFS kernel modules

# yum -y install gcc gcc-c++ kernel-headers kernel-devel

# dd if=/dev/zero of=/dev/sdb bs=1G count=1

# NODES=(hanamachine0{1,2,3} gpfsnode0{1,2,3} hananode0{1,2,3})

# NODES=(hanamachine01 gpfsnode01 hananode01)

# mmcrcluster -n /var/mmfs/config/nodes.cluster -p gpfsnode01 \

For a single-node configuration, just include this node with:

# mmcrcluster -n /var/mmfs/config/nodes.cluster -p gpfsnode01 \

# mmchlicense server --accept -N gpfsnode01,gpfsnode02,gpfsnode03

On a single-node configuration, just assign the node itself:

# mmchlicense fpo --accept -N gpfsnode0X,gpfsnode0Y,gpfsnode0Z

Create the file /var/mmfs/config/disk.list.data.fs

On a single-node configuration, create a disk file for the node itself:

On every node, create the GPFS disks with:

Adjust various GPFS settings on the master node:

Important: There must be no tabs or spaces between the comma-separated arguments of

Create the GPFS file system.

On a single-node configuration, create the file system with the following:

# mmcrfs sapmntdata -F /var/mmfs/config/disk.list.data.fs -A yes \

# mmcrfileset sapmntdata hanadata -t "Data Volume for HANA database"

On the master node, mount the GPFS file system with:

# mmlinkfileset sapmntdata hanadata -J /sapmnt/data

# df -hT # -> /sapmnt