How to Use Fast Connection Failover Simon Haslam, Veriton Limited

to Improve Application Resilience

How to Use Simon Haslam / Veriton

Specialised consultant & Oracle Partner,
Fast Connection Failover established 12 years

to Improve Application Demanding web & call-centre applications

Resilience Architecture & development strategy;

health-checks; disaster recovery; tuning

Oracle Application Server & WebLogic

Simon (clusters, J2EE, ADF, SSO, OID, Reports, etc)

Haslam Database & related technologies

(Solaris/Linux, load balancers, firewalls, )

1 (2.2) 2 (2.2)

Agenda  Whats the problem ? !

1. Why, TAF vs FCF, Architecture Lots of connections to database
2. ONS & FCF Configuration Depending on type of failure, and what the
app is doing, it make take a while to realise
3. Application Changes
But in HA environment we expect
4. Live Demonstration application to carry on after node failure

Unresponsive app, users pressing refresh

button, calls to helpdesk, etc

3 (2.2) 4 (2.2)

 How do we solve this?  Recap: Thick & Thin JDBC

Drivers
1) Database-tier needs to manage Thick/OCI driver
TAF
sessions so that app-tier doesnt java layer talks to OCI layer on the client
necessarily notice anything wrong, which then talks SQLnet over the network to
or, the database
2) Database-tier needs to tell app-tier
to sort itself out. FAN Thin driver
everything is in the java class library
ONS & FCF

5 (2.2) 6 (2.2)

1
How to Use Fast Connection Failover Simon Haslam, Veriton Limited
to Improve Application Resilience

 Recap: Thick & Thin JDBC  Acronym City

Drivers
Java app Java app
TAF = Transparent Application Failover
JDBC FAN = Fast Application Notification
thick/OCI ONS = Oracle Notification Server
driver JDBC
thin OPMN = Oracle Process Monitoring and
driver Notification services (App Server)
Oracle (100% java)
client OCI FCF = Fast Connection Failover
Note: client- UCP = Universal Connection Pool (11.1)
side drivers
NB: <> Implicit Connection Cache (10.x)
Database
7 (2.2) 8 (2.2)

 TAF vs FCF in brief  FCF using Local ONS

TAF (8.1.5+) app server 1
thick JDBC & OCI node 1
connection/session-level conn
inst 1 pool app
recreates cursors (optionally)
FCF (10.1+) ONS 1 app
thin or thick/OCI JDBC (& ~OCI) db 1
app
application-level node 2 2
unused JDBC pooled connections closed & recreated inst 2
load balancing based on UP events OC4J
ONS ONS OPMN
this session covers FCF for Java applications

9 (2.2) 10 (2.2)

 FCF using Local ONS  Remote ONS access

app server 1 Client can connect directly to ONS in
node 1 conn
database tier
inst 1 pool app Clients need to know about all database
ONS 21 nodes
app
db 12 Dont need to run ONS daemon on client
app
node 2 2 Available from Oracle JDBC driver 10.2
inst 2 JVM; other
Java
OC4Jserver
ONS ONS OPMN
ONS in
Oracle Client

11 (2.2) 12 (2.2)

2
How to Use Fast Connection Failover Simon Haslam, Veriton Limited
to Improve Application Resilience

 Remote ONS Subscription  OPMN & ONS Ports

(10.2+)
app server 1
ff keys
db
localhost: 6101
node 1 conn
local (new)
inst 1 pool app ONS-aware
21 ONS JDBC
ONS app
db 12 OPMN
request remote
app only SSL
node 2 2 any: 6004
any: 6201
inst 2 Other
Other Other
JVM OPMN/
OPMN OPMN /
ONS ONS
ONS

Note: Example host:port shown in italics

Diagram does not show other OPMN relationships
13 (2.2) 14 (2.2)

 FCF Pre-requisites  ONS Configuration

JDBC driver thick or thin
JDBC driver versions
Applicable to both RAC and Data Guard
(note: DG needs some work to ensure ONS
messages are sent on site failover)
Usually best done at client. Also possible
on server, or,
using AS automatic topology discovery

By the way: remember Metalink 364855.1:

set LOCAL_LISTENER = <VIP> Inter-node ONS connections secured with
on all RAC instances SSL (uses a wallet)

15 (2.2) 16 (2.2)

 ONS Configuration: client  ONS messages

App Server: Publish/subscribe model
ORACLE_HOME/opmn/conf/opmn.xml Database:
Oracle client: Node
ORACLE_HOME/opmn/conf/ons.config
Instance
you need to install ONS from client or db CD Service
App Server: ONSpmQuery,
(optional) wallet used to secure ONS IAS/PM/PROC_READY
comms.

17 (2.2) 18 (2.2)

3
How to Use Fast Connection Failover Simon Haslam, Veriton Limited
to Improve Application Resilience

Starting ONS on mid-tier/client  ONS Configuration: client

App Server:
Start: $ORACLE_HOME/opmn/conf/opmn.xml:
$ORACLE_HOME/opmn/bin/ons d

Check: <notification-server interface="ipv4">
<port local="6101" remote="6201" request="6004"/>
ons a ping (rc: 0 = successful) <ssl enabled=true" wallet-file="$ORACLE_HOME/opmn/
ons a debug conf/ssl.wlt/veriton"/>
<topology>
Stop: <nodes list="westfield1-vip.hq.veriton.co.uk:6201,
westfield2-vip.hq.veriton.co.uk:6201"/>
ons a shutdown </topology>
</notification-server>
Logging:

loglevel=59 shows connectionsmessages NB: and white-space not in file!

19 (2.2) 20 (2.2)

  ONS Configuration: client

$ORACLE_HOME/opmn/conf/ons.config:

localport=6100
remoteport=6200
nodes=westfield1-vip.hq.veriton.co.uk:6201,
westfield2-vip.hq.veriton.co.uk:6201

NB: and white-space not in file!

21 (2.2) 22 (2.2)

 Universal Connection Pool  Universal Connection Pool

Metalink 743726.1: 23rd Oct 2008 Generic pooling mechanism
Oracle Universal Connection Pool (UCP) and Oracle JDBC
Deprecation of JDBC Implicit Connection Cache
Non-Oracle
We encourage our customers to plan to adopt
Non-relational (e.g. CICS!)
UCP for new applications and plan to change
existing applications as indicated in the UCP Separate library (ucp.jar)
Transition Guide. Many more Oracle HA features than ICC

http://www.oracle.com/technology/
software/tech/java/sqlj_jdbc/index.html

23 (2.2) 24 (2.2)

4
How to Use Fast Connection Failover Simon Haslam, Veriton Limited
to Improve Application Resilience

 JDBC ICC Configuration  JDBC Configuration

App Server:
connectionCachingEnabled data-sources.xml:

fastConnectionFailoverEnabled <managed-data-source connection-pool-name="RACDB" jndi-
name="jdbc/RACDS" name="RACDS"/>
<connection-pool name="RACDB"
ONSConfiguration <connection-factory factory-
class="oracle.jdbc.pool.OracleDataSource" >
<property name="connectionCachingEnabled" value="true"/>
<property name="fastConnectionFailoverEnabled"
value="true"/>
</connection-factory>
</connection-pool>
</data-sources>

NB: and white-space not in file!

25 (2.2) 26 (2.2)

 Application Changes  Application Changes

App will still get failures if connected so 10g: use method on
needs to retry SQL OracleConnectionCacheManager class to
Request a new connection determine if the exception was fatal:
boolean
Replay the transaction
isFatalConnectionError(SQLException e)

The application should not try to rollback 11g: boolean isValid(int timeout)
the transaction as it has already been method on UCP data source
rolled back

27 (2.2) 28 (2.2)

 FCF Demonstration  Caveats

westfield23 Java 5 JVM
Application must retry all transations
Thread 1: Thread 2: Thread 3: since last commit
Sleep Sleep SQL
no conn with conn DB sleep Be very careful about versions between
JDBC and ONS! (both ICC and UCP)
ICC
OPMN
Pool conn conn conn
ONS
Note: ons.jar is not delivered with JDBC
w1 w2 downloads need to get from a full client
RACAAPP RACAAPP Database Service
(or database) installation
Inst 1 ONS Inst 2 ONS

RAC A
29 (2.2) 30 (2.2)

5
How to Use Fast Connection Failover Simon Haslam, Veriton Limited
to Improve Application Resilience

FCF Summary References

Synchronous notification is sensible cf polling JDBC Developers Guide
This area should be easy... but isnt!
Fast push notification of failure to app-tier UCP Developers Guide11gR1 (11.1.0.7)
Not automatically configured (yet) on OAS Samples on OTN
No performance penalty MAA Papers: Barb Lundhild, Fermin Castro
But... application still needs to do processing to
recover from failure

If your java app can retry after SQL errors:

RAC: you should be using it

DG: worth considering

31 (2.2) 32 (2.2)

Thank You for

Listening!

SimonH@veriton.co.uk
OFM admin blog: http://SimonHaslam.co.uk

UKOUG App. Server & Middleware SIG

Wednesday 4th March 2009 Slough
(Combined day with Apps DBA for OEBS SIG)
Corporate UKOUG members get one free place

33 (2.2)