Beruflich Dokumente
Kultur Dokumente
Imagination at work
Agenda
2
Introduction
Speaker background
Gary Gordhamer
Principal Technologist ERP Infrastructure
Power & Water Technology
GE Power & Water
25 years of IT experience
24 years with Oracle (6.x up to 12c)
Worked in many different industries including healthcare,
manufacturing, utilities, banking, and printing
@ggordham
linkedin.com/in/ggordhamer/
oraontap.blogspot.com
Power EBS journey - 2014 plan
2014 2015 2016
Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
Power Generation 400GB 5TB
Parts 250 2,500
Users Users
FS
Distributed Power
Exadata
Dresser 2.5TB 8TB +
1,000 3,000 Exalogic
Aero Users Users
Jenbacher
Renewables 2.5TB 1,000 Users
Global VCP
Consolidation of our Oracle ERP systems on a single platform, some EBS 12.1, mostly 12.2.
Some new installs, and some platform migrations from Solaris to Linux and Linux to Linux.
Completed In Process / production go live date
9
Oracle Design
All racks are physically interconnected by InfiniBand – recommended by MAA and product teams!
All systems also have redundant 10Gig Ethernet interfaces to our internal GE Network
Each instance will require the integration of components from each of the three
Oracle engineered components.
Each engineered component can host multiple instances with simple tweaks to
Oracle’s design
13
EBS on Exadata + Exalogic MAA design
vServer 1 OEL 5.8 4CPU x 64GB vServer N OEL 5.8 4CPU x 64GB
• DR provided through
OS Backup
14
Networking, networking, networking
Admin Network 192.168.2.x/22
Very difficult
Client Network 10.200.192.x/22 • Exalogic has 8 networks
EoIB external-mgmt
IPoIB OVM Mgmt 10.2.10.0/24 • Exadata has 3 networks
ZFS Backup Exadata
Appliance scamd01 Exalogic OVM control, heartbeat, movement
aiezd01 scand01
aiezd01sn01
scamd01cel03 Need to make sure traffic is
aiezd01sn02
scamd01cel04
scamd01cel05
always routed over the right
scamd01cel06
scamd01cel07
scand01sn01
IPoIB vserver-shared-storage
network
scand01sn02 172.17.0.0/16
vServer access to ZFS
Lots of help and expertise from our central network, and facilities teams
17
Infrastructure Impact of Exa
Lot’s of effort from Corporate IT, Power IT, and Oracle to design and install all this equipment
18
Things we did not do
• Oracle Traffic Director (OTD) – We continue to use external
hardware load balancers
• Oracle VM Templates for Exalogic – we made our own
templates (needed to GE’ize them either way)
• Automated EBS instance cloning (yet)
• Enable app to DB tier traffic over InfiniBand – still no clear
instructions for this (note AutoConfig leanings)
Note: We also had a lot of back and forth on NFS v4 mounting
options, and managing the number of files in CM log & out
directory. (NFS4 rw,bg,hard,nointr,rsize=131072,wsize=131072)
19
What’s not in the box?
What “services” did we have to build out
• Setting up admin users • OS Backup scripts for Exadata
(EMOC, ZFS, local OS) • Exadata GridLink Listener
• Configure EMOC • OEM Cloud Control agent install
• vServer template with “GE” • Monitoring template setup in OEMCC
customizations
• ZFS Backup Storage setup
• Domain Name Services (DNS)
• LDAP / IDM integration (vs. NIS)
• ZFS Replication configuration
• SNMP / Mail relay • QFSDP management process (patching)
• Linux OS RPM updates
• EBS minimum RPM / OS requirements • New DR processes / instructions
• ExaLogic vServer build process • Change management process
• Internal YUM REPO • Incident management process
• Configuration / usage of ExaBR (scripts)
21
Some more on LDAP vs. NIS (is there out of the box security?)
Network Information System (NIS) Lightweight Directory Access Protocol (LDAP)
• Pre-installed by oracle on the ADM node of • You need to provide, not part of delivered system
Exalogic
• Data is stored in a “database”
• Stores users and passwords in the native files on
the ADM node (E.G. /etc/passwd) • Access to individual data is controlled by LDAP
server
• All servers have access to view all data (all users,
passwords, etc..) • Managed through LDAP tools (less likely to mess
up)
• If a file gets messed up (passwd), creates lots of
issues • ZFS needs to be re-configured to use LDAP
• ZFS is pre-configured for this, but vServer • Exadata, and vServer templates need to be
templates still need customization updated to add LDAP support / configuration
• Exadata needs to be updated with this setup
NFS v4 (part of ZFS) requires a central user directory NIS, AD or LDAP in order to work
22
More Security + DR + DB Connectivity + Platinum Support
Getting people used to security model on Exadata
• Shared ASM, GRID user vs. dedicated servers per database
2
3
Working with Oracle
Exa* marketing vs. reality
Lots of Exa* features are not fully supported / working
with e-Business Suite (yet)
• EBS R12.2 IB app to DB connection
• ExaLogic enhancements for WebLogic
• Exadata – must use PFILES + Autconfig, can not use SPFILES
This system is
now the • Exadata Hostname vs. Public network
ultimate power
in the universe! Exa* specific MOS notes for EBS
• Needed to start on 11g version of Exadata (2014)
We are now on 12c / OEL 6
26
Challenges Cont.
• Oracle EBS teams do not seem to be doing testing / development on Exa*
platform (yet)
• Platinum support = Look at patching Exadata + Exalogic at least twice a year
(we patch 4 times a year today with N-1)
• No Single documentation to follow
(lots of papers for each technology / application)
• Need to verify post Oracle build (ACS) – missing settings or setups on Exadata +
Exaloigc
• Password Management – you will need a good post-it note, or SAPM solution
(Shared Password Management Utility)
• EBS 12.2 DR process is longer than EBS 12.1 due to Weblogic layer configuration
• No updated MAA document for EBS 12.2 (yet)
Lots of MOS notes that contradict each other (or at least collide with each other)
27
Partial references list (60 and counting)
• Deploying Oracle Traffic Director as a Load Balancer for Oracle E-Business Suite R12 (White paper)
• Exalogic – Walgreens – Customize EBS vServer Template, Shanna Chan 8/25/2013
• MOS Doc ID 1616827.1, Managing Concurrent Manager Log and Out Directories
• Oracle E-Business Suite and Oracle Maximum Availability Architecture Best Practices Including a Case Study on Oracle Exalogic and Oracle
Exadata Database Machine (Oracle White paper Feb 2014)
• Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine (Oracle Maximum Availability Architecture White Paper April
2012)
• Consolidating Oracle Applications on Exalogic (An Oracle White Paper March, 2012)
• MOS Doc ID 1133355.1 Migrating an Oracle EBusiness Suite Database to Oracle Exadata Database Machine
• E-Business Suite on Exadata (Oracle Maximum Availability Architecture White Paper October 2012)
• MOS Doc ID 1590941.1 Oracle VM Templates for Oracle EBusiness Suite Deployment Guide, Release 12.2.2
• Oracle VM Templates for E-Business Suite: Deployment Guide (December 2009)
• MOS Doc ID 741818.1 Export/import process for 12.0 or 12.1 using 11gR1 or 11gR2
• MOS Doc ID 438086.1 Application Tier Platform Migration with Oracle EBusiness Suite Release 12
• Deploying Oracle E-Business Suite on Oracle Exalogic Elastic Cloud and Oracle Exadata Database Machine (An Oracle White Paper
September 2012 (Updated June 2013))
• MOS Doc ID 1546861.1 [Linux OS] System Hung with Large Numbers of Page Allocation Failures with "order:5" on Exadata Environments
• MOS Scenarios Doc ID 1363188.1 Configuring Exadata I/O Resource Manager for Common
• MOS Doc ID 762374.1 Step by Step - Configure Direct NFS Client (DNFS) on Linux (11g)
• MOS Doc ID 560853.1 HTTP Server Is Either Slow Or Stops Responding When Installed On A NFS Mounted Drive
• Exalogic Backup and Recovery Best Practices (An Oracle White Paper 16 August 2012)
• MOS Doc ID 1516025.1 How To Configure NIS Master, Slave And Client Configuration In Exalogic Virtual Environment
• MOS Doc ID 1274318.1 Oracle Sun Database Machine Setup/Configuration Best Practices
• MOS Doc ID 396009.1 Database Initialization Parameters for Oracle E-Business Suite Release 12
28
Tools, Tools, Tools, Scripting, Scripting, Scripting
• Enterprise Manager Operations Center (EMOC)
• Oracle Virtual Machine Manager (OVMM)
• ZFS Storage appliance BUI / CLI
• Integrated Lights Out Management (ILOM)
• Shell scripting
• ZFS scripting
• Python (ExaTools)
• Oracle Enterprise Manager Cloud Control (OEM CC)
• Exacheck (Both Exadata and Exalogic)
• ExaTools (Exalogic)
• ExaBR (Exalogic)
30
Patching ExaData + ExaLogic together (QFSDP)
Interactions on IB Switch firmware
Exalogic)Patching)task(s) On3line Off3line Exadata)Patching)task(s) On3line Off3line
• QFSDP in sync across both racks Pro$active+Reboot+Infiniband+switches 0.5 Pro$active+Reboot+Infiniband+switches 0.5
Upgrade+Exalogic+Control+Services 0.5
Upgrade+the+Sun+Network+QDR+InfiniBand+
Patches are “Rolling” Gateway+Switches. 0.75
Upgrade+the+Sun+Datacenter+InfiniBand+ 0.75
• EBS does not support rolling Upgrade+the+ZFS+Storage+Appliance+(ILOM+
and+Software) 2.5
• Some infrastructure items done on-line Shut+down+application+services+(EBS)
Upgrade+Compute+Node+#1+and+verify+
0.5
Upgrade+the+DB+node+#1+operating+
0.5
Site B
DB Node 4 Compute Node 7 6. Shutdown application again (same time or in the future)
DB Node 3 Compute Node 6 7. Migrate application back to “Site A” from “Site B”
Compute Node 5 8. Startup application on “Site A”
9. Patch “Site B”
Compute Node 4
Works well if applications are “regional” in scope
DB Node 2 Compute Node 3
Site A
(you can affect one install at a time)
DB Node 1 Compute Node 2
There multiple hardware layouts to support this
Compute Node 1
(this is an example)
You will need excess capacity of some level
32
More networking headaches
New Oracle PCA rack for development EBS Production Exadata / Exalogic
instances
• 36 virtual machines on Exalogic
• We have 2 EBS instances in build now
• 10 databases on Exadata
36
Some initial performance data
Item Exa* Time VMWare servers
Autoconfig < 2 Minutes
DB Restore (500GB) < 10 Minutes
DB Backup (500GB) < 8 Minutes
ADOP FSClone 1 hour 2 ½ - 3 hours
ADOP Prepare 1 hour 2 hours
ADOP Cutover 1 hour 2 ½ - 3 hours
APP Tier backup (APPL_TOP+) Instantaneous 1+ hours
(ZFS Snapshot) (tar + compress)
37
EXA* Benefits for Global GL
• Performance improvements in multiple areas
! Physical deployment
Real savings for
" Delivery time of new environments cut in half
DBA & project teams
" Cloning time reduced from 72 hours to 36 hours
! Application performance
" Throughput of feeder file processing has increased by 40-50%.
" Throughput of GL interface extraction has increased by 30-40%
" Performance of WEB ADI upload has increased by more than 50%
" Outbound file creation time (such as Taxpro) has reduced by more than 50%
" Performance of ledger run and closing (month end) programs has increased by 60-70%
• Some examples:
• Ledger run reduced from 2 hours 32 minutes to 50 minutes Increased end user
• Month end closing process reduced from 36 min to 9 minutes analysis windows
• Opening new period reduced from 2.5 hours to 30 minutes between runs
38
QA Load Test July non Exa vs Oct Exa (% faster shown)
• Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine
http://www.oracle.com/technetwork/database/features/availability/maa-wp-dr-dbm-130065.pdf
• Backup and Recovery Performance and Best Practices using the Sun ZFS Storage Appliance with the Oracle Exadata Database Machine
http://www.oracle.com/technetwork/database/features/availability/maa-wp-dbm-zfs-backup-1593252.pdf
• Architecture Principles and Implementation Practices for Remote Replication Using Oracle ZFS Storage Appliance
http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/zfssa-replication-2014-1-2120969.pdf