Logical Hostnames in

Oracle E-Business Suite

Optimize DR and cloning processes.

ANDREJS PROKOPJEVS
Lead Applications Database Consultant
PAVAN MARTI
Applications Database Consultant
About me

Apps DBA from Riga, Latvia.

Speaking SQL since 2001.

In Oracle world since 2004.
“In love” with Oracle EBS since 2006.

@aprokopjevs

Andrejs Prokopjevs prokopjevs@pythian.com

Lead Applications Database Consultant
At Pythian since 2011 https://www.pythian.com/blog/author/prokopjevs/

© 2017 Pythian 2
About me

Apps DBA from Ottawa, Canada.

In Oracle world since 2002.

In EBS world since 2005.

@Pavanmarti

Pavan Marti marti@pythian.com

Applications Database Consultant
At Pythian since 2014 https://www.pythian.com/blog/

© 2017 Pythian 3
ABOUT PYTHIAN

Pythian’s 400+ IT professionals

help companies adopt and
manage disruptive technologies
to better compete

© 2017 Pythian 4
TECHNICAL EXPERTISE Big Data: Harnessing the transformative
power of data on a massive scale

Advanced Analytics: Mining data for Databases: Ensuring databases

insights & business transformation are reliable, secure, available and
using data science continuously optimized

Cloud: Using the disruptive

nature of cloud for accelerated, Infrastructure: Transforming and
cost-effective growth managing the IT infrastructure
that supports the business

DevOps: Providing critical velocity

in software deployment by adopting
DevOps practices

© 2017 Pythian 5
EXPERIENCED GLOBAL EXPERTS

11,800 400 2
Systems currently Pythian experts Millennia of experience
managed by Pythian in 35 countries gathered and shared over
19 years

© 2017 Pythian 6
Agenda

• What is logical hostname.

• Why it should be an important piece of a DR strategy.
• How cloning process can be improved.
• Support for 11i/R12 and R12.2. Why it is very hot now for R12.2.
• Advanced configurations - Linux Containers.

© 2017 Pythian 7
What is logical
hostname

© 2017 Pythian 8
What is a logical hostname?

• Like Apache’s virtual hostname (<VirtualHost>)?

• Logical = Virtual hostname.

• Logical <> Physical hostname.

• Virtual host or address is a mapping of your application logically for an end user.

• That’s it?

• Virtual host or address is also a mapping of your application configuration for a

hardware it runs on as well.

© 2017 Pythian 9
Example #1

• Multiple applications running on single or multiple servers in pool.

server1.domain.com server2.domain.com

app1.domain.com app4.domain.com

app2.domain.com app5.domain.com

app3.domain.com app6.domain.com

© 2017 Pythian 10
Example #2

• Clusters.
Load
Balancer

server1.domain.com server2.domain.com

app1.domain.com app1.domain.com

app2.domain.com app2.domain.com

app3.domain.com app3.domain.com

Cluster 1

© 2017 Pythian 11
Example #3

• Cloud.
LBaaS

ip-10-10-10-1 ip-10-10-10-19 ip-10-10-10-… ip-10-10-17-21

eu-1.cloud.com eu-1.cloud.com eu-1.cloud.com us-1.cloud.com

app1.domain.com app1.domain.com app1.domain.com app1.domain.com

app2.domain.com app2.domain.com app2.domain.com app2.domain.com

app3.domain.com app3.domain.com app3.domain.com app3.domain.com

Auto Scaling group DR

© 2017 Pythian 12
Why it should be an important
piece of a DR strategy

© 2017 Pythian 13
DR – in general

• Option #1: Active-Active

• Option #2: Active-Passive

• Transparency ip-10-10-10-10
eu-1.cloud.com
ip-10-10-17-21
us-1.cloud.com
• Preferred
• Minimum downtime APP 1 APP 1
• Minimum data loss
• SLA
• Maximum allowed switchover
window to be defined.

• Bi – Directional
• And to be tested, at least, on a bi- Production DR
yearly basis.
© 2017 Pythian 14
DR – Oracle E-Business Suite – DB tier

• Physical standby
• $$$ (EE license for standby).
• Lots of manual effort.
• Data loss of, at least, one redo log. ebsdb1.domain.com ebsdb3.domain.com

• Data Guard PROD1 PRODDR

• $$$ (EE license for standby).
• Automatic gap detection and DGB
ebsdb2.domain.com
recovery.
• Flexible and easy switchover.
• Data loss depends on the mode PROD2
running.

Production - RAC DR
• Active Data Guard
• $$$$$$ (additional option)
© 2017 Pythian 15
DR – Oracle E-Business Suite – Apps tier

• Only Active-Passive

• APPS_BASE sync using rsync ebsapp1.domain.com ebsapp3.domain.com

• Most common example. APPS_BASE APPS_BASE

• Switchover CM log/out CM log/out

• Cloning RSYNC
ebsapp2.domain.com
• Switchover back
• Another Cloning
APPS_BASE

Multi-Node production DR

© 2017 Pythian 16
Challenge

• Why cloning?
• EBS configuration is very tight to node name (FND).
• R12.2: EBS configuration is very tight to node name and IP address of that node (+ADOP).
• To enable a new DR node, you need to follow cloning process to get the new node set.

• It takes time
• 30-60 minutes, and even longer.

• Workarounds
• Well scripted and tested automated process, but still adcfgclone.pl takes time.
• Have DR nodes already inside the primary configuration.
▪ Reports of unavailable nodes in the stack.
▪ Still requires post steps (like relocation of the Concurrent Managers).

• Very complex process !!! © 2017 Pythian 17

How it should look like? Hi! I am a logical hostname.

ebsapp1.domain.com ebsapp3.domain.com

app.domain.com app.domain.com

APPS_BASE APPS_BASE

CM log/out RSYNC CM log/out

Production DR

© 2017 Pythian 18
Benefits

• Forget about the cloning.

• Forget about configuration post steps.

• Just restart your applications, as is.

• 5-10 minutes, and we are done?

© 2017 Pythian 19
How cloning process can be
improved

© 2017 Pythian 20
Cloning process
• Cloning using rapid clone
Run adpreclone.pl Run adcfgclone.pl

prod1.domain.com test1.domain.com

Application Tier Application Tier

COPY
prod2.domain.com test2.domain.com

Database Tier Database Tier

Source Target
© 2017 Pythian 21
Cloning process - detailed

• What does adpreclone.pl do

• Creates cloning stage.
• Retrieves database information and creates cloning scripts.

• What does adcfgclone.pl do

• Creates context file.
• Configures the target, registers and relinks the Oracle Homes.
• Optionally restarts the services.

• Post steps

• Complexity depends on the stack size

© 2017 Pythian 22
Challenges
• Why hostname area is important? Why we can’t just copy the files?
• EBS configuration, FND tables.
• Profile options.
• Concurrent Managers validate “uname -n” !!!
• Multi node, Load balancer, OID/SSO integrations.
• … etc. Lots of more.

• How much time typicaly cloning takes?

• Few hours to days depending on architecture and resources.

• Workarounds
• Automation (scripts, Ansible, etc)
• Applications Management Pack (EM)

© 2017 Pythian 23
How it could look like? Hi! I am a logical
hostname.

prod1.domain.com prod2.domain.com test1.domain.com test2.domain.com

app1.domain.com app2.domain.com app1.domain.com app2.domain.com

APPS_BASE APPS_BASE COPY APPS_BASE APPS_BASE

prod3.domain.com test3.domain.com

db1.domain.com db1.domain.com
COPY
EBSDB EBSDB

Source Target

© 2017 Pythian 24
Benefits
• No need of Source and Target steps
• Cloning is just a copy of source to target.
• Any DB tier change (SID, host, port) is just a context update plus AutoConfig.
• Reduces the manual effort. Automation is easy.
• Reduces the work within few hours, or even minutes (cloud).

• But still you may have some extra manual tasks.

• Update of profile options, printer settings.
• Change of the database and applications passwords.
• Change the front-end URL (load balancer).
• Re-registration of OID/OAM if its SSO integrated.
• Customizations, interfaces.

• Can be easily scripted still.

© 2017 Pythian 25
Cloud provisioning example

•
ebsapp.domain.com ebsapp.domain.com
Create VM Image /
APPS_BASE APPS_BASE
Template

• Create and Configure ebsdb.domain.com ebsdb.domain.com

virtual machines from the
DATABASE DATABASE
template
Source New Target
• First-boot bootstrap to
script instance specific post
steps.
VIRTUAL MACHINE TEMPLATES

Virtual Machine Repository

© 2017 Pythian 26
Support for 11i/R12 and
R12.2

© 2017 Pythian 27
Logical hostname support
• Release 11i / R12.0 / R12.1
• Only physical hostname is supported.
• You have to configure logical name as a physical hostname on the server.
• OS commands must return the hostname configured.
▪ uname -n
▪ hostname

• Release R12.2
• Official support and solution are implemented.
• Configuration can be based on logical hostname.
• Requires a specific patch level.

© 2017 Pythian 28
R12.2 requirements
• Patch level
• Patch 25178222:R12.AD.C.Delta.9
• Patch 25180736:R12.TXK.C.Delta.9
• Patch 17075726:R12.FND.C

• Alias in /etc/hosts

• Web entry points to be pointed to logical end point

• s_hostname, s_%host
• s_webentryurlprotocol, s_webentryhost, s_webentrydomain, s_active_webport
• s_login_page
• s_external_url

© 2017 Pythian 29
R12.2 – what is changed?
• Context name, s_hostname, s_%host context variables can be pointed to logical
hostname value.
<host oa_var="s_hostname">ebsapp1</host>

• s_physical_hostname context variable is introduced.

<physical_hostname oa_var="s_physical_hostname">prod1</physical_hostname>

• $APPL_TOP/APPSVIS122_vis122app01.env
currentHost=`hostname|sed 's/\..*//g'|tr "[A-Z]" "[a-z]"`
host=`echo ”ebsapp1"|tr "[A-Z]" "[a-z]"`
physicalHost=`echo ”prod1"|tr "[A-Z]" "[a-z]"`

if [ $host != $currentHost ] && [ $physicalHost != $currentHost ]

then
echo "ERROR: This env should be sourced from $host !”
exit_status=1
© 2017 Pythian 30
R12.2 – what is changed in CM startup?
• One-off Patch 17075726
• Delivers changes to cpadmin.sh, reviver.sh, startmgr.sh and FNDLIBR code.

• Changes in CM initialization
• "uname -n” use is gone.
• Startup is using the configured host value and passes it to the runtime as a parameter.
• Examples:
$ ps -fu applmgr

...

FNDCRM apps/# long hash # FND FNDCRM N 10 c LOCK Y EBSAPP1 2728881

...

/u01/vision/fs1/EBSapps/comn/util/jdk64/bin/java ...
-DEBS_HOSTNAME=ebsapp1 ... oracle.apps.fnd.cp.gsf.GSMServiceController
© 2017 Pythian 31
What to do with a release below R12.2 TXK delta 9?
• Challenge

• Starting Concurrent Managers is the main problem. These validate their target node name with
server hostname.

• R12.2 has a FMW_Home 11g stack that depends on hostname references.

• Any alternatives?

© 2017 Pythian 32
Alternatives
• How about to fake the hostname on your Linux server just for Oracle EBS.

• fakehostname.c is created to replace gethostname() default OS call to return you the desired
hostname value from environment variable.

• fakehostname.c is compiled as 32-bit library. With R12.2 you need both 32 and 64 bit libraries
to be created.

• LD_PRELOAD is set pointing to your compiled library code.

• This tweak can be added to your environment source (.bash_profile) before your EBS
environment is set.

© 2017 Pythian 33
Fake hostname
• Example:
[applmgr@prod1 ~]$ ls -l /lib/libfakehostname.so.1
-rwxr-xr-x 1 root root 7240 Jan 6 19:40 /lib/libfakehostname.so.1
[applmgr@prod1 ~]$ ls -l /lib64/libfakehostname.so.1
-rwxr-xr-x 1 root root 8464 Jan 6 19:40 /lib64/libfakehostname.so.1
[applmgr@prod1 ~]$
[applmgr@prod1 ~]$ export MYHOSTNAME=ebsapp01
[applmgr@prod1 ~]$
[applmgr@prod1 ~]$ hostname
prod1.domain.com
[applmgr@prod1 ~]$
[applmgr@prod1 ~]$ export LD_PRELOAD="libfakehostname.so.1”
[applmgr@prod1 ~]$ hostname
ebsapp01
[applmgr@prod1 ~]$

• Reference:
https://blog.pythian.com/oracle-e-business-suite-virtual-host-names-solution-r12-2/
© 2017 Pythian 34
Advanced configurations
Linux Containers

© 2017 Pythian 35
Linux Containers

• LXC virtualization
• Multiple Linux instances on a single host.

• Shared kernel
• Not a classic virtual machine we used to.
• Hardware is not emulated.
• Isolated process tree view of the OS, security, networking, storage.

• Available since RHEL / OEL 6 (2.6.24 kernel).

• Deprecated in RHEL 7, replaced with LXD improved technology.
• But still can be installed from EPEL repository.

• Supports different Linux OS deployments, no lock to physical host.

• RHEL 6 or 7, Debian, etc.
• OEL5 guest support is also there, but only with Oracle’s UEK. Good for 11i users. © 2017 Pythian 36
Architecture with classic virtualization

Oracle E-Business Suite

Binaries and Libraries Network Storage Security

Guest Operating System with emulated hardware

HyperVisor

HOST OS

KERNEL

© 2017 Pythian 37
Architecture with LXC

Oracle E-Business Suite

Binaries and Libraries

Network Storage Security
of emulated guest OS

Container engine

HOST OS

KERNEL

© 2017 Pythian 38
Configuration

• Default LXC_PATH: /container

• Default log path: /var/log/lxc

• Examples:
[root@ebsapp1 ~]# lxc-create -n app -t oracle -- --release=6.latest --privileged
[root@ebsapp1 ~]# lxc-start -n app -d -o /var/log/lxc/app.log

[root@ebsapp1 ~]# lxc-console -n app

root@app:~# hostname
app.domain.com
root@app:~#

© 2017 Pythian 39
Resource control

• CPU, IO, Memory resources can be limited and controlled.

• Examples:
[root@ebsapp1 ~]# lxc-cgroup -n app cpuset.cpus 0,1
[root@ebsapp1 ~]# lxc-cgroup -n app memory.limit_in_bytes 53687091

© 2017 Pythian 40
E-Business Suite use case

• Create a container for Apps tier.

• Configure logical hostname for the container realm.
• Configure EBS to be based on this logical hostname.

• DR
• Rsync between containers.
• Snapshot or keep in sync the container repository.

• Cloning and instance provisioning

• Create and transfer container templates.
• Clone containers on the same host.

• Officially certified only for R12.2

• UEK R3 Quarterly Update 6 (3.8.13-98) or higher or UEK R4 (Doc ID 1330701.1)
© 2017 Pythian 41
E-Business Suite use case

ebsapp1.domain.com ebsapp3.domain.com

app.domain.com app.domain.com

APPS_BASE APPS_BASE

CM log/out RSYNC CM log/out

Containers Containers

Production DR

© 2017 Pythian 42
Summary

• Official logical hostname support is long awaited and finally available, but only for
R12.2.
• It can streamline the instance lifecycle management, rapid process of cloning.
• Easier instance provisioning from images (cloud deployment and automation).
• DR failover window is minimum.
• There are still unofficial options available for pre-R12.2 releases.
• Linux Containers is a good OS level alternative approach.

© 2017 Pythian 43
THANK YOU
Q&A

© 2017 Pythian 44