Sie sind auf Seite 1von 31

Optimizing Oracle

Deployments in
Distributed Data Centers

BRKAPP-2018

Session_ID
Presentation_ID

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

Cisco CVD and DCAP


Cisco Validated Design Program
Data Center Assurance Program (DCAP)
App Deployment
Guides

Validated Design
Guides

System Assurance
Testing

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

DCAP Topology
1

7
3

6
8

Branch Name
GSS 4492
BRKAPP-2018 Branch Client
Server
2008 Cisco Systems,
Inc. All rights reserved.
14438_04_2008_c2

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application
Tier

DataBase
Cluster

Other Cisco Live Breakout Sessions


that You May Want to Attend
Relevancy

Products we will discuss


in our Packet Walk
GSS

ISR

WAE 512

ACE

WAE 7326

Application Data Base


Tier
Cluster

BRKAPP-1001 Server Load Balancing Design


BRKAPP-2002 Troubleshooting ACE
BRKAPP-3003 Introduction WAAS
BRKAPP-2005 Deploying WAAS
BRKAPP-3006 Troubleshooting WAAS
BRKAPP-1009 Introduction to Web Application Security
BRKAPP-2011 Scaling Applications in a Clustered
Environment
BRKAPP-2013 Best Practices for Application Optimization
illustrated with SAP, Seibel and Exchange
BRKAPP-2014 Deploying AXG
BRKAPP-1016 Running Applications on the Branch Router

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Cisco Solutions for Oracle


Problem

Description

Latency

WAN characteristics hinder


performance of Oracle
Applications
Oracle Apps lacks built-in
functionality of providing LB
capability
Oracle Apps lacks built-in
functionality of providing key HA
features
Oracle Apps lack built in site
selection and replication
capabilities. GSS

Load Balance

Scalability/High Availability

Disaster Recovery Limitations

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

DCAP: Geographical Map

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

DCAP Packet Walk Overview


(L3 is So Yesterday)

IE and Firefox
on Win XP

Named/bind
9.2.4 MS DNS
Server2003
Enterprise

Version
2.0.2

Application

Application

Application

Presentation

Presentation

Presentation

Session

Session

Session

Transport

Transport

Transport

eBusiness
Oracle 11i

Version
A1.6.3
Version
4.0.11b34
Version
12.4(12)

Application
Presentation

Version
4.0.11b34

Oracle 10g R2
RAC

Application

Application

Presentation

Presentation

Session

Session

Session

Session

Session

Transport

Transport

Transport

Transport

Transport

Network

Network

Network

Network

Network

Network

Network

Network

Network

Data Link

Data Link

Data Link

Data Link

Data Link

Data Link

Data Link

Data Link

Data Link

Physical

Physical

Physical

Physical

Physical

Physical

Physical

Physical

Physical

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845

Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

Begin Packet Walk

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Client DNS Query to Branch NS


Trace Data (Client dns query)
User Datagram Protocol, Src Port: 4112 (4112), Dst Port: domain (53)
Domain Name System (query)
Transaction ID: 0x000f
Flags: 0x0100 (Standard query)
Questions: 1
Answer RRs: 0
Authority RRs: 0
Additional RRs: 0
Queries
wwwin-oefin.gslb.dcap.com: type A, class IN
Name: wwwin-oefin.gslb.dcap.com
Type: A (Host address)
Class: IN (0x0001)

Simulate Client Browser


Loadrunner (testing)
Important timers, TTLs, MS Resolver
Browser Caching
IE, Mozilla, Safari, etc

Trace Data (NS dns response)


User Datagram Protocol, Src Port: domain (53), Dst Port: 4112 (4112)
Domain Name System (response)
Transaction ID: 0x000f
Flags: 0x8400 (Standard query response, No error)
Questions: 1
Answer RRs: 1
Authority RRs: 0
Additional RRs: 0
Queries
wwwin-oefin.gslb.dcap.com: type A, class IN
Name: wwwin-oefin.gslb.dcap.com
Type: A (Host address)
Class: IN (0x0001)
Answers
wwwin-oefin.gslb.dcap.com: type A, class IN, addr 101.1.33.50
Name: wwwin-oefin.gslb.dcap.com
Type: A (Host address)
Class: IN (0x0001)
Time to live: 5 seconds
Data length: 4
Addr: 101.1.33.50

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings

Value Name: DnsCacheTimeout


Data Type: REG_DWORD
Radix: Decimal
Value: (time in seconds)

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

10

Branch NS Receives Client A Query

*.gslb.dcap.com is
delegated to all 4 GSSs

dns

Configuration (Branch NS Zone file)

Delegated sub-zone:

gslb.dcap.com.

gslb
gslb
gslb
gslb

600
600
600
600

Delegated sub-zone:
;
wwwin-oefin
wwwin-oefin
wwwin-oefin
wwwin-oefin

NS
NS
NS
NS

DCAP 4.0 DNS (BIND/


MS DNS

dca-gss-1.dcap.com.
dca-gss-2.dcap.com.
dcb-gss-1.dcap.com.
dcb-gss-2.dcap.com.

wwwin-oefin.gslb.dcap.com.
NS
NS
NS
NS

Gotchas:

dca-gss-1.gslb.dcap.com.
dca-gss-2.gslb.dcap.com.
dcb-gss-1.gslb.dcap.com.
dcb-gss-2.gslb.dcap.com.

Be aware of TTL

Log file on Name Server showing dns delegation


145C
145C
A58
A58

PACKET
PACKET
PACKET
PACKET

UDP
UDP
UDP
UDP

Branch Client

BRKAPP-2018
14438_04_2008_c2

Rcv
Snd
Rcv
Snd

10.0.20.3
201.1.33.11
201.1.33.11
10.0.20.3
101000111000
000110101010
101011111010
1010111

0008
Q
1024
Q
1024 R Q
0008 R Q

Branch Name
Server

[0001
D
[0000
[0084 A
[0084 A

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

NOERROR]
NOERROR]
NOERROR]
NOERROR]

(11)wwwin-oefin(4)gslb(4)dcap(3)com(0)
(11)wwwin-oefin(4)gslb(4)dcap(3)com(0)
(11)wwwin-oefin(4)gslb(4)dcap(3)com(0)
(11)wwwin-oefin(4)gslb(4)dcap(3)com(0)

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

11

Cisco Public

GSS Receives A Query


Show screen (GSS)

Show screen (GSS)

dca-gss-1.gslb.dcap.com#show statistics keepalive


101.1.33.12
OFFLINE
101.1.33.22
ONLINE
101.1.33.31
ONLINE
101.1.33.32
ONLINE
101.1.33.34
OFFLINE
101.1.33.35
OFFLINE
101.1.33.36
ONLINE
201.1.33.32
ONLINE
201.1.33.34
OFFLINE
201.1.33.35
OFFLINE
201.1.33.53
OFFLINE
201.1.33.59
ONLINE

dcb-gss-1.gslb.dcap.com#show statistics keepalive


101.1.33.12
OFFLINE
101.1.33.22
ONLINE
101.1.33.31
ONLINE
101.1.33.32
ONLINE
101.1.33.34
OFFLINE
101.1.33.35
OFFLINE
101.1.33.36
ONLINE
101.1.33.50
ONLINE
201.1.33.34
OFFLINE
201.1.33.35
OFFLINE
201.1.33.53
OFFLINE
201.1.33.59
ONLINE

dcb-gss-1.gslb.dcap.com

dca-gss-1.gslb.dcap.com
dca-gss-2.gslb.dcap.com

dcb-gss-2.gslb.dcap.com

ACE VIPS

ACE VIPS

dca-agg-1-ace-1 DCA
dca-agg-2-ace-1

DCB

DCA

DCB

dcb-ss-1-ace-1
dca-ss-2-ace-1

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

12

GSS Decision Process


Show screen (GSS Rule)

Show screen (GSS Rule)

Show screen (GSS keepalives)


dca-gss-1.gslb.dcap.com#show statistics keepalive http-head all
IP: 101.1.33.50
Keepalive => 101.1.33.50
Termination Method: Reset
Status: OFFLINE
Keepalive Type: Fast
Destination Port: 8000
Http Path: "/"
Host Tag: ""
Packets Sent:
Packets Received:
Positive Probe:
Negative Probe:
Transitions:
VIP GID: 151 LID: 32
Keepalive GID:

3 Balance Clauses
101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

1144808
1199599
0
54822
0
836

WAE 7326

Trace taken on the Name Server (Client/NS/GSS)


Time
1 0.000000

Source
10.0.20.3

Destination
10.0.20.2

Time
2 0.000149

Source
10.0.20.2

Destination
201.1.33.11

Time
3 0.018354

Source
201.1.33.11

Destination
10.0.20.2

Time
4 0.018482

Source
10.0.20.2

Destination
10.0.20.3

NS
10.0.20.2

GSS
201.1.33.11

GSS
201.1.33.11

NS
10.0.20.2

Protocol Info
DNS
Standard query response A 101.1.33.50

Frame 3 (101 bytes on wire, 101 bytes captured)


Ethernet II, Src: 00:19:aa:c1:96:d1 (00:19:aa:c1:96:d1), Dst: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc)
Internet Protocol, Src: 201.1.33.11 (201.1.33.11), Dst: 10.0.20.2 (10.0.20.2)
User Datagram Protocol, Src Port: domain (53), Dst Port: 29310 (29310)
Domain Name System (response)
No.

NS
10.0.20.2

Protocol Info
DNS
Standard query A wwwin-oefin.gslb.dcap.com

Frame 2 (85 bytes on wire, 85 bytes captured)


Ethernet II, Src: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc), Dst: 00:19:aa:c1:96:d1 (00:19:aa:c1:96:d1)
Internet Protocol, Src: 10.0.20.2 (10.0.20.2), Dst: 201.1.33.11 (201.1.33.11)
User Datagram Protocol, Src Port: 29310 (29310), Dst Port: domain (53)
Domain Name System (query)
No.

Client
10.0.20.3

Protocol Info
DNS
Standard query A wwwin-oefin.gslb.dcap.com

Frame 1 (85 bytes on wire, 85 bytes captured)


Ethernet II, Src: Intel_53:0e:72 (00:07:e9:53:0e:72), Dst: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc)
Internet Protocol, Src: 10.0.20.3 (10.0.20.3), Dst: 10.0.20.2 (10.0.20.2)
User Datagram Protocol, Src Port: 1073 (1073), Dst Port: domain (53)
Domain Name System (query)
No.

DataBase Cluster

13

DNS DataFlow
No.

Application Tier

Cisco Public

NS
10.0.20.2

Client
10.0.20.3

Protocol Info
DNS
Standard query response A 101.1.33.50

Frame 4 (101 bytes on wire, 101 bytes captured)


Ethernet II, Src: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc), Dst: Intel_53:0e:72 (00:07:e9:53:0e:72)
Internet Protocol, Src: 10.0.20.2 (10.0.20.2), Dst: 10.0.20.3 (10.0.20.3)
User Datagram Protocol, Src Port: domain (53), Dst Port: 1073 (1073)
Domain Name System (response)

GSS syslog
dcb-gss-2 DNS-7-SELREQNAME[15925] Request from 10.0.20.2:32370
for wwwin-oefin.gslb.dcap.com, type is T_A, id is 9472

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

dcb-gss-2 DNS-7-SELREPPASS[15925] Reply is A, 1 addresses 101.1.33.50, NOERROR, TTL 5, AA


for wwwin-oefin.gslb.dcap.com, Request id is 9472

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

14

Queuing the Packet!Timeout to Talk


About If a GSS Were to Fail
### NS Delegation ###.
NS
NS
NS
NS

DCAP topology

dca-gss-1.dcap.com.
dca-gss-2.dcap.com.
dcb-gss-1.dcap.com.
dcb-gss-2.dcap.com.

NS Backoff timers

ap.
com

Can configure NS
Probes on GSS to NS

www
inoef
in.
gsl

No DNS response
is issued from the GSS

BRKAPP-2018
14438_04_2008_c2

TTL issues

b.d
c

Name Server waiting


for a response

2008 Cisco Systems, Inc. All rights reserved.

15

Cisco Public

Branch Router Intercepts TCP Traffic via


WCCP and Forwards to Branch WAE
Configuration (ISR 3845)

Trace Data (ISR 3845)


No.

Time
Source
1 0.000000
10.0.10.2
> 8000 [SYN] Seq=0 Len=0 MSS=1460

ip wccp 61
ip wccp 62
interface GigabitEthernet0/1.10
description to "Clients"
encapsulation dot1Q 10
ip address 10.0.10.1 255.255.255.0
no ip unreachables
ip wccp 61 redirect in
ip pim sparse-mode
!
interface GigabitEthernet0/1.11
description "to Cisco WAE Appliances"
encapsulation dot1Q 11
ip address 10.0.11.1 255.255.255.0
no ip unreachables
ip wccp redirect exclude in
!
interface GigabitEthernet0/1.12
description "To Wide Area Network"
encapsulation dot1Q 12
ip address 10.0.12.1 255.255.255.248
no ip unreachables
ip wccp 62 redirect in
ip pim sparse-mode

Destination
101.1.33.50

Protocol Info
TCP
58372

Frame 1 (62 bytes on wire, 62 bytes captured)


Ethernet II, Src: Inventec_e5:e0:60 (00:a0:d1:e5:e0:60), Dst: Cisco_32:ab:81
(00:19:56:32:ab:81)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)
Transmission Control Protocol, Src Port: 58372 (58372), Dst Port: 8000 (8000),
Seq: 0, Len: 0
Source port: 58372 (58372)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 28 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0xc79f [correct]
Options: (8 bytes)
Maximum segment size: 1460 bytes
NOP
NOP
SACK permitted

TCP Option 0x21 not present

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

16

Branch WAE Receives TCP Packet from


Branch Router via WCCP
Configuration (Branch WAE)
classifier
match
match
match
match
match

HTTP
dst port
dst port
dst port
dst port
dst port

eq
eq
eq
eq
eq

Trace Data (Branch WAE)


No.
Time
Source
Destination
Protocol Info
1 0.000000
10.0.10.2
101.1.33.50
TCP
58372 > 8000 [SYN] Seq=0 Len=0 MSS=1460

80
8080
8000
8001
3128

map basic
name File-System classifier AFS action optimize full
name Instant-Messaging classifier AOL action pass-through
name Remote-Desktop classifier Altiris-CarbonCopy action
pass-through
name Printing classifier AppSocket action optimize full
name File-System classifier Apple-AFP action optimize full
name Remote-Desktop classifier Apple-NetAssistant action
pass-through
name Instant-Messaging classifier Apple-iChat action passthrough
name File-Transfer classifier BFTP action optimize full
name Systems-Management classifier BMC-Patrol action passthrough
name Other classifier Basic-TCP-services action passthrough
name P2P classifier BitTorrent action pass-through
name SQL classifier Borland-Interbase action optimize full

Frame 1 (62 bytes on wire, 62 bytes captured)


Ethernet II, Src: Inventec_e5:e0:60 (00:a0:d1:e5:e0:60), Dst:
Cisco_32:ab:81 (00:19:56:32:ab:81)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50
(101.1.33.50)
Transmission Control Protocol, Src Port: 58372 (58372), Dst Port:
8000 (8000), Seq: 0, Len: 0
Source port: 58372 (58372)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 28 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0xc79f [correct]
Options: (8 bytes)
Maximum segment size: 1460 bytes
NOP
NOP
SACK permitted

TCP Option 0x21 not


present
101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

17

Cisco Public

ISR Receives Packet from Branch WAE and


Forwards Packet to Its Destination (ACE VIP)
Configuration (Branch WAE)

Trace Data (Branch WAE)


No.

Time
Source
173 3.265868
10.0.10.2
[SYN] Seq=0 Len=0 MSS=1432

ip wccp 61
ip wccp 62

Destination
101.1.33.50

Protocol Info
TCP
22994 > 8000

Frame 173 (78 bytes on wire, 78 bytes captured)


Ethernet II, Src: Cisco_19:bf:80 (00:15:c7:19:bf:80), Dst: Cisco_fe:1b:01
(00:0b:fc:fe:1b:01)
802.1Q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 1000 0101 0100 = ID: 2132
Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)
Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0,
Len: 0
Source port: 22994 (22994)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0x8f36 [correct]
Options: (20 bytes)
Maximum segment size: 1432 bytes
NOP
NOP
SACK permitted
Unknown (0x21) (12 bytes)

interface GigabitEthernet0/1.10
description to "Clients"
encapsulation dot1Q 10
ip address 10.0.10.1 255.255.255.0
no ip unreachables
ip wccp 61 redirect in
ip pim sparse-mode
!
interface GigabitEthernet0/1.11
description "to Cisco WAE Appliances"
encapsulation dot1Q 11
ip address 10.0.11.1 255.255.255.0
no ip unreachables
ip wccp redirect exclude in
!
interface GigabitEthernet0/1.12
description "To Wide Area Network"
encapsulation dot1Q 12
ip address 10.0.12.1 255.255.255.248
no ip unreachables
ip wccp 62 redirect in
ip pim sparse-mode

TCP Option 0x21 now present


101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

18

ACE Interfaces Explained


Configuration (ACE)

Features

interface vlan 2132


description CLIENT_VLAN
bridge-group 10
ip options allow
mtu 2000
no normalization
fragment min-mtu 68
no icmp-guard
access-group input BPDU-ALLOW
access-group input anyone
access-group output anyone
service-policy input ORACLE_TCP_TRAFFIC
no shutdown

Interesting config
HA (Failure)
Gotchas

interface vlan 1135


description WAE_VLAN
ip address 101.1.35.9 255.255.255.0
alias 101.1.35.10 255.255.255.0
peer ip address 101.1.35.8 255.255.255.0
mtu 2000
no normalization
fragment min-mtu 68
mac-sticky enable
no icmp-guard
access-group input anyone
service-policy input REMOTE-MGNT
service-policy input OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
service-policy input NAT_POLICY
no shutdown

Allow TCP Option 0x21


Ensure return traffic from
WAEs follows return
traffic flow

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

19

Cisco Public

Congestive Collapse!Timeout to Talk


About Bridged vs. Routed Mode
ACE acts acts as a router

ACE acts as a bump in the wire

Servers default gateway is


the ACE

Servers default gateway is the


upstream router

interface vlan 1105


description CLIENT_VLAN
bridge-group 10
ip options allow
no normalization
access-group input anyone
service-policy input REMOTE-MGNT
no shutdown
interface vlan 1133
description SERVER_VLAN
bridge-group 10
ip options allow
no normalization
access-group input anyone
nat-pool 1 101.1.33.150 101.1.33.150 netmask 255.255.255.0 pat
service-policy input REMOTE-MGNT
service-policy input ORACLE_VIPS
no shutdown

interface vlan 1105


description CLIENT_VLAN
ip address 201.1.5.252 255.255.255.0
ip options allow
alias 201.1.5.254 255.255.255.0
peer ip address 201.1.5.253 255.255.255.0
no normalization
access-group input anyone
service-policy input REMOTE-MGNT
service-policy input ORACLE_VIPS
no shutdown
interface vlan 1133
description SERVER_VLAN
ip address 201.1.33.252 255.255.255.0
alias 201.1.33.254 255.255.255.0
peer ip address 201.1.33.253 255.255.255.0
access-group input anyone
nat-pool 1 201.1.33.150 201.1.33.150 netmask 255.255.255.0 pat
service-policy input REMOTE-MGNT
no shutdown

interface bvi 10
ip address 101.1.33.252 255.255.255.0
alias 101.1.33.254 255.255.255.0
peer ip address 101.1.33.253 255.255.255.0
description CLIENT_SIDE_L3
no shutdown

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

20

10

TCP Time Wait!Timeout to Talk about If


an ACE Fails
Dedicated FT VLAN between ACE Modules
All Redundancy traffic is sent over this dedicated VLAN
TRP Protocol packets, Heartbeats, Configuration Sync packets, and State
replication packets
ft interface vlan 1120
ip address 101.1.20.100 255.255.255.0
peer ip address 101.1.20.101 255.255.255.0
no shutdown

ft interface vlan 1120


ip address 101.1.20.101 255.255.255.0
peer ip address 101.1.20.100 255.255.255.0
no shutdown

ft peer 1
heartbeat interval 600
heartbeat count 10
ft-interface vlan 1120

ft peer 1
heartbeat interval 600
heartbeat count 10
ft-interface vlan 1120

ft group 1
peer 1
priority 200
associate-context c2
inservice

ft group 1
peer 1
peer priority 200
associate-context c2
inservice

FT Group
No. of Contexts
Context Name
Context Id
Configured Status
Maintenance mode
My State
My Config Priority
My Net Priority
My Preempt
Peer State
Peer Config Priority
Peer Net Priority
Peer Preempt
Peer Id
Last State Change time
Running cfg sync enabled
rom active
Startup cfg sync enabled

BRKAPP-2018
14438_04_2008_c2

:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:

1
1
c2
1
in-service
MAINT_MODE_OFF
FSM_FT_STATE_STANDBY_COLD
200
200
Enabled
FSM_FT_STATE_ACTIVE
100
100
Enabled
1
Tue Feb 26 13:32:46 2008
Enabled

: Enabled

2008 Cisco Systems, Inc. All rights reserved.

FT Group
Configured Status
Maintenance mode
My State
My Config Priority
My Net Priority
My Preempt
Peer State
Peer Config Priority
Peer Net Priority
Peer Preempt
Peer Id
Last State Change time

:
:
:
:
:
:
:
:
:
:
:
:
:

1
in-service
MAINT_MODE_OFF
FSM_FT_STATE_ACTIVE
100
100
Enabled
FSM_FT_STATE_STANDBY_COLD
200
200
Enabled
1
Tue Feb 26 18:25:42 2008

Running cfg
Running cfg
Startup cfg
Startup cfg
completed

:
:
:
:

Enabled
Running configuration sync has
Enabled
Startup configuration sync has

sync
sync
sync
sync

enabled
status
enabled
status

21

Cisco Public

Packet Arrives at the ACE (Packet #1)


Trace Data (Client to WAE)

Configuration (ACE)
No.

Time
Source
173 3.265868
10.0.10.2
[SYN] Seq=0 Len=0 MSS=1432

policy-map multi-match ORACLE_TCP_TRAFFIC


class ORACLE_L4
loadbalance vip inservice
loadbalance policy GO_TO_WAE_FARM
loadbalance vip icmp-reply
class-map match-any ORACLE_L4
2 match virtual-address 101.1.33.50 tcp eq 8000
policy-map type loadbalance first-match GO_TO_WAE_FARM
class class-default
serverfarm WAE_FARM backup ORAAPP_ORACLE_FARM
serverfarm host WAE_FARM
description WAAS SERVERFARM TRANSPARENT MODE
transparent
predictor leastconns
Do not use
probe WAE_ICMP
rserver WAE_1
conn-limit max 6500 min 5000
inservice
rserver WAE_2
conn-limit max 6500 min 5000
inservice

NAT

WAE Threshold

Destination
101.1.33.50

Protocol Info
TCP
22994 > 8000

Frame 173 (78 bytes on wire, 78 bytes captured)


Ethernet II, Src: Cisco_19:bf:80 (00:15:c7:19:bf:80), Dst: Cisco_fe:1b:01
(00:0b:fc:fe:1b:01)
802.1Q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 1000 0101 0100 = ID: 2132
Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)
Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0,
Len: 0
Source port: 22994 (22994)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0x8f36 [correct]
Options: (20 bytes)
Maximum segment size: 1432 bytes
NOP
NOP
SACK permitted
Unknown (0x21) (12 bytes)

TCP Options Set


via Branch WAE

rserver host WAE_1


description WAE 1
ip address 101.1.35.4
inservice
rserver host WAE_2
description WAE 2
ip address 101.1.35.5
inservice
101000111000
000110101010
101011111010
1010111

ID
G
B
A
W
rS
C
A
a
p
R
S
a
E
t
p
n
la
3
4
c
5
M
7
iB
8
4
h
1
o
3
a
c
4
9
d
2
s
a

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

22

11

ACE: Internal Mapping of TCP/UDP Flow


from Client in Branch 2 (Show Conns)
TCP and UDP Flows = 2 X Internal Half Flows

Network
Proccessor

640
1346

ACE Connection ID

1
1

SYN_SEEN
ESTAB
CLOSED

Client:
SRC IP : SRC Port

in TCP
out TCP

VIP:Port

2132 10.0.20.2:46457
1135 101.1.33.50:8000

101.1.33.50:8000
10.0.20.2:46457

ESTAB
ESTAB

Non TCP is
displayed as - -

INIT
SYN-ACK
ESTAB
CLOSED

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

23

Cisco Public

NAMWhere Did the Packet Go?


DCAP Relies on the NAM for
verification and troubleshooting
Service Modules
Set to 1518 bytes in order
to capture entire frames

Configuration (CAT6K)

dca-agg-1#show module
Mod Ports Card Type
--- ----- -------------------------------------1
6 Firewall Module
2
1 Application Control Engine Module
3
1 Application Control Engine Module
4
8 Intrusion Detection System
5
8 Network Analysis Module
7
2 Supervisor Engine 720 (Active)
9
8 CEF720 8 port 10GE with DFC
10
8 CEF720 8 port 10GE with DFC
11
8 CEF720 8 port 10GE with DFC
12
48 CEF720 48 port 10/100/1000mb Ethernet
13
8 CEF720 8 port 10GE with DFC

Model
--------------WS-SVC-FWM-1
ACE10-6500-K9
ACE10-6500-K9
WS-SVC-IDSM-2
WS-SVC-NAM-2
WS-SUP720-3B
WS-X6708-10GE
WS-X6708-10GE
WS-X6708-10GE
WS-X6748-GE-TX
WS-X6708-10GE

analysis module 5 management-port access-vlan 200


monitor session 1 destination analysis-module 5 data-port 1

(ip.addr eq 101.1.33.50 and ip.addr eq 10.0.10.2) and (tcp.port eq 8000 and tcp.port eq 22994)

Must look at both


sides of the flow

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

24

12

Delay the ACK!Timeout to Talk WCCP


vs. WAE in DC
DCAP 4.0 using WCCP interception in DCB and in
DCA, using ACE to load balance to WAEs.
Advantages to using WAE in the DC

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

25

Cisco Public

ACE Sending Packet to WAE (Packet #2)


Configuration (ACE)

Trace Data (ACE to WAE)

interface vlan 1135


description WAE_VLAN
ip address 101.1.35.9 255.255.255.0
alias 101.1.35.10 255.255.255.0
peer ip address 101.1.35.8 255.255.255.0
mtu 2000
no normalization
fragment min-mtu 68
mac-sticky enable
no icmp-guard
access-group input anyone
service-policy input REMOTE-MGNT
service-policy input
OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
service-policy input NAT_POLICY
no shutdown
policy-map multi-match
OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
class VIA_WAE_FARM_L4
loadbalance vip inservice
loadbalance policy ORAAPP_ORIGIN_SERVERS
class-map match-any VIA_WAE_FARM_L4
2 match virtual-address 101.1.33.50 tcp any
policy-map type loadbalance first-match
ORAAPP_ORIGIN_SERVERS
class class-default
sticky-serverfarm sticky-ace-cookie
insert-http SRC_IP header-value "%is"

No.

Time
Source
174 3.268853
10.0.10.2
[SYN] Seq=0 Len=0 MSS=1432

Destination
101.1.33.50

Protocol Info
TCP
22994 > 8000

Frame 174 (78 bytes on wire, 78 bytes captured)


Ethernet II, Src: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01), Dst: Ibm_b4:37:2f
(00:14:5e:b4:37:2f)
Destination: Ibm_b4:37:2f (00:14:5e:b4:37:2f)
DST MAC WAE1
Source: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)
Type: 802.1Q Virtual LAN (0x8100)
SRC MAC of ACE
802.1Q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 0100 0110 1111 = ID: 1135
Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)
Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0,
Len: 0
Source port: 22994 (22994)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0x8f36 [correct]
Options: (20 bytes)
Maximum segment size: 1432 bytes
NOP
NOP
SACK permitted
Unknown (0x21) (12 bytes)

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

26

13

Buffer the Packet!Timeout to talk about


what happens when a WAE Fails
ACE will take WAE out of rotation (what happens to
current sessions?)
What happens when the WAE reaches the max conn
limit?

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

27

Cisco Public

Packet Arrives at the DC WAE


Configuration (DC WAE)

Trace Data (DC WAE)


No.

Time
Source
173 3.265868
10.0.10.2
[SYN] Seq=0 Len=0 MSS=1432

policy-map multi-match ORACLE_TCP_TRAFFIC


class ORACLE_L4
loadbalance vip inservice
loadbalance policy GO_TO_WAE_FARM
loadbalance vip icmp-reply
class-map match-any ORACLE_L4
2 match virtual-address 101.1.33.50 tcp eq 8000
policy-map type loadbalance first-match GO_TO_WAE_FARM
class class-default
serverfarm WAE_FARM backup ORAAPP_ORACLE_FARM
serverfarm host WAE_FARM
description WAAS SERVERFARM TRANSPARENT MODE
transparent
predictor leastconns
probe WAE_ICMP
rserver WAE_1
conn-limit max 6500 min 5000
inservice
rserver WAE_2
conn-limit max 6500 min 5000
inservice

Destination
101.1.33.50

Protocol Info
TCP
22994 > 8000

Frame 173 (78 bytes on wire, 78 bytes captured)


Ethernet II, Src: Cisco_19:bf:80 (00:15:c7:19:bf:80), Dst: Cisco_fe:1b:01
(00:0b:fc:fe:1b:01)
802.1Q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 1000 0101 0100 = ID: 2132
Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)
Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0,
Len: 0
Source port: 22994 (22994)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0x8f36 [correct]
Options: (20 bytes)
Maximum segment size: 1432 bytes
NOP
NOP
SACK permitted
Unknown (0x21) (12 bytes)

rserver host WAE_1


description WAE 1
ip address 101.1.35.4
inservice
rserver host WAE_2
description WAE 2
ip address 101.1.35.5
inservice
101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

28

14

Buffer the Packet! Timeout to talk about

WAE Auto Discovery Though ACE (Gotcha)


4 SYNs
4 SYN/ACKs
4 ACKs

1
2
3
4
5
6
7
8
9
10
11
12

1
2
3
4

SYN
SYN
SYN
SYN

(VLAN
(VLAN
(VLAN
(VLAN

1133)
1135)
1135)
1133)

TCP
TCP
TCP
TCP

0x21
0x21
0x21
0x21

SEQ=4150100321
SEQ=4150100321
SEQ=4150100321
SEQ=4150100321

5
6
7
8

SYN/ACK
SYN/ACK
SYN/ACK
SYN/ACK

(VLAN
(VLAN
(VLAN
(VLAN

1133)
1135)
1135)
1133)

NO TCP OPTION
NO TCP OPTION
TCP OPTION 0x21
TCP OPTION 0x21

SEQ=1193771344
SEQ=1193771344
SEQ=1193771344
SEQ=1193771344

9
10
11
12

ACK
ACK
ACK
ACK

(VLAN
(VLAN
(VLAN
(VLAN

1133)
1135)
1135)
1135)

TCP OPTION 0x21


TCP OPTION 0x21
NO TCP OPTION
NO TCP OPTION

SEQ=2002616674
SEQ=2002616674
SEQ=4150100322
SEQ=4150100322

OPTION
OPTION
OPTION
OPTION

Change made to the sequence number during final


acknowledgements of the TCP handshake (Packet #9)
BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

SEQ Space Jump!

Cisco Public

29

Silly window syndrome! Timeout to


talk ACE/Firewall Integration
Stateful inspection of WAAS optimized traffic requires
that the inspecting device understand the sequence
number shift on optimized TCP connections
The following software versions provide 100%
interoperability with WAAS optimized connections:
IOSFW (Zone-based): 12.4(11)T2 or later
ASA/PIX: 7.2.x or later
FWSM: 3.2.1 or later
ACE: (all versions) TCP Norm, IP Options
BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

30

15

WAE Sends Packet Back to ACE


(Packet #3)
Configuration (ACE)

Trace Data (WAE to ACE)

interface vlan 1135


description WAE_VLAN
ip address 101.1.35.9 255.255.255.0
alias 101.1.35.10 255.255.255.0
peer ip address 101.1.35.8 255.255.255.0
mtu 2000
no normalization
fragment min-mtu 68
mac-sticky enable
no icmp-guard
access-group input anyone
service-policy input REMOTE-MGNT
service-policy input
OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
service-policy input NAT_POLICY
no shutdown
policy-map multi-match
OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
class VIA_WAE_FARM_L4
loadbalance vip inservice
loadbalance policy ORAAPP_ORIGIN_SERVERS
class-map match-any VIA_WAE_FARM_L4
2 match virtual-address 101.1.33.50 tcp any
policy-map type loadbalance first-match
ORAAPP_ORIGIN_SERVERS
class class-default
sticky-serverfarm sticky-ace-cookie
insert-http SRC_IP header-value "%is"

No.

Time
Source
175 3.268869
10.0.10.2
[SYN] Seq=0 Len=0 MSS=1432

Destination
101.1.33.50

Protocol Info
TCP
22994 > 8000

Frame 175 (78 bytes on wire, 78 bytes captured)


Ethernet II, Src: Ibm_b4:37:2f (00:14:5e:b4:37:2f), Dst: Cisco_fe:1b:01
(00:0b:fc:fe:1b:01)
Destination: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)
Source: Ibm_b4:37:2f (00:14:5e:b4:37:2f)
Type: 802.1Q Virtual LAN (0x8100)
SRC MAC of WAE 1
802.1Q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 0100 0110 1111 = ID: 1135
Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)
Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0,
Len: 0
Source port: 22994 (22994)
Destination port: 8000 (8000)
Sequence number: 0
(relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
Window size: 65535
Checksum: 0x8f36 [correct]
Options: (20 bytes)
Maximum segment size: 1432 bytes
NOP
NOP
SACK permitted
Unknown (0x21) (12 bytes)

TCP Options Set


via Branch WAE

WAE default route is ACE Alias IP


101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

ISR 3845

2008 Cisco Systems, Inc. All rights reserved.

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

31

Cisco Public

ACE (Probes to DB Cluster via


Application Tier)
Configuration (ACE)
probe http ORACLE_WEB_PAGE_CHECK
port 8000
interval 2
faildetect 1
passdetect interval 2
credentials sysadmin sysadmin
request method get url /oa_servlets/AppsLogin
expect status 200 200

Trace Data (ACE Probe to Server)

Trace Data (ACE Probe to Server)


Hypertext Transfer Protocol
GET /oa_servlets/AppsLogin HTTP/1.1\r\n
Request Method: GET
Request URI: /oa_servlets/AppsLogin
Request Version: HTTP/1.1
Connection: Close\r\n
Authorization: Basic c3lzYWRtaW46c3lzYWRtaW4=\r\n
Credentials: sysadmin:sysadmin
Host: 101.1.33.47\r\n
\r\n
101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

32

16

Delayed Binding!Timeout to Talk about


ACE Probes
show probe detail

dca-agg-1-ace-1/c2# show probe ORACLE_WEB_PAGE_CHECK detail


probe
: ORACLE_WEB_PAGE_CHECK
type
: HTTP
state
: ACTIVE
description :
---------------------------------------------port
: 8000
address
: 0.0.0.0
interval : 2
pass intvl : 2
fail count: 1
recv timeout: 10
http method
: GET
http url
: /oa_servlets/AppsLogin
conn termination : GRACEFUL
expect offset
: 0
, open timeout
expect regex
: send data
: -

addr type : pass count : 3

: 10

dca-agg-1-ace-1/c2(config)# logging message <0-2147483647>


dca-agg-1-ace-1/c2(config)# logging message 251006 level 7

Syslog message ID

dca-agg-1-ace-1: %ACE-3-251010 Health probe failed for server 101.1.33.47 on port 8000, connection refused by server

syslog output
dca-agg-1-ace-1/c2# show logging message
Message logging:
message 100001: default-level 2
message 101002: default-level 4
message 101004: default-level 6
message 101005: default-level 6
message 101006: default-level 6
message 101007: default-level 6

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

all
(enabled)
(enabled)
(enabled)
(enabled)
(enabled)
(enabled)

View the logging


messages

33

Cisco Public

ACE Makes Decision and Forwards


Packet to Application Tier (Packet #4)
Configuration (ACE)

Trace Data (ACE to Server)

policy-map multi-match
OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
class VIA_WAE_FARM_L4
loadbalance vip inservice
loadbalance policy ORAAPP_ORIGIN_SERVERS
class-map match-any VIA_WAE_FARM_L4
2 match virtual-address 101.1.33.50 tcp any
policy-map type loadbalance first-match
ORAAPP_ORIGIN_SERVERS
class class-default
sticky-serverfarm sticky-ace-cookie
insert-http SRC_IP header-value "%is
serverfarm host
ORAAPP_ORACLE_FARM_WAAS_CONTENT
probe ORACLE_WEB_PAGE_CHECK
rserver ORAAPP01
inservice
rserver ORAAPP02
inservice
rserver ORAAPP03
inservice

No.

Time
207 3.286888

Source
10.0.10.2

Destination
101.1.33.5

Protocol Info
HTTP
GET / HTTP/1.1

Frame 207 (468 bytes on wire, 468 bytes captured)


Ethernet II, Src: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01), Dst: HewlettP_3e:5e:c0
(00:19:bb:3e:5e:c0)
Oracle Server
802.1Q Virtual LAN
Chosen
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.5 (101.1.33.5)
Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 1, Ack:
1, Len: 410
Hypertext Transfer Protocol
GET / HTTP/1.1\r\n
Accept: */*\r\n
ACE inserting
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
HTTP Headers
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1;.NET CLR50727)\r\n
Host: wwwin-oefin.gslb.dcap.com:8000\r\n
Connection: Keep-Alive\r\n
Cookie: oracle.uix=0^^GMT-5:00^p; CHOCO=r275366210\r\n
If-Modified-Since: Mon, 22 Oct 2007 16:17:38 GMT\r\n
If-None-Match: "534a8a-a2c-471ccd22"\r\n
\r\n

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

34

17

Oracle Login Page (Oracle EBusiness Suite)

wwwin-oefin.gslb.dcap.com

Insert Trace Client to App server


on port:8000
101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

35

Cisco Public

Packet to Application Tier (Packet #4)


Trace Data (ACE to Server)

Hypertext Transfer Protocol


GET / HTTP/1.1\r\n
Accept: */*\r\n
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2;
SV1;.NET CLR50727)\r\n
Host: wwwin-oefin.gslb.dcap.com:8000\r\n
Connection: Keep-Alive\r\n
Cookie: oracle.uix=0^^GMT-5:00^p; CHOCO=r275366210\r\n
If-Modified-Since: Mon, 22 Oct 2007 16:17:38 GMT\r\n
If-None-Match: "534a8a-a2c-471ccd22"\r\n

Configuration (Oracle)

grew wwwin-oefin OEFIN_dcap-dca-oraapp01.xml


<externURL oa_var="s_external_url">http://wwwin-oefin.gslb.dcap.com:8000</externURL>
<webentryhost oa_var="s_webentryhost">wwwin-oefin</webentryhost>
<login_page oa_var="s_login_page">http://wwwinoefin.gslb.dcap.com:8000/oa_servlets/AppsLogin</login_page>
<externURL oa_var="s_external_url">http://wwwin-oefin.gslb.dcap.com:8000</externURL>
<webentrydomain oa_var="s_webentrydomain">gslb.dcap.com</webentrydomain>

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

36

18

ACE Show Screens


Show screen (Client to WAE in DCA)

Trace Data (ACE to Server)

Interface: vlan 1133 2132


service-policy: ORACLE_TCP_TRAFFIC
class: ORACLE_L4
VIP Address:
Protocol: Port:
101.1.33.50
tcp
eq
8000
loadbalance:
L7 loadbalance policy: GO_TO_WAE_FARM
VIP Route Metric
: 77
VIP Route Advertise : DISABLED
VIP ICMP Reply
: ENABLED
VIP State: INSERVICE
curr conns
: 2
, hit count
: 44
dropped conns
: 0
client pkt count : 430
, client byte count: 26724
server pkt count : 322
, server byte count: 23295
conn-rate-limit
: 0
, drop-count : 0
bandwidth-rate-limit : 0
, drop-count : 0
L7 Loadbalance policy : GO_TO_WAE_FARM
class/match : class-default
LB action :
primary serverfarm: WAE_FARM
state: UP
backup serverfarm : ORAAPP_ORACLE_FARM
state: UP
hit count
: 44
dropped conns
: 0

Interface: vlan 1135


service-policy: OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
class: VIA_WAE_FARM_L4
VIP Address:
Protocol: Port:
101.1.33.50
tcp
any
loadbalance:
L7 loadbalance policy: ORAAPP_ORIGIN_SERVERS
VIP Route Metric
: 77
VIP Route Advertise : DISABLED
VIP ICMP Reply
: ENABLED
VIP State: INSERVICE
curr conns
: 2
, hit count
: 39
dropped conns
: 6
client pkt count : 269
, client byte count: 14135
server pkt count : 106
, server byte count: 9465
conn-rate-limit
: 0
, drop-count : 0
bandwidth-rate-limit : 0
, drop-count : 0
L7 Loadbalance policy : ORAAPP_ORIGIN_SERVERS
class/match : class-default
LB action :
sticky group: sticky-ace-cookie
primary serverfarm: ORAAPP_ORACLE_FARM
state: UP
backup serverfarm : hit count
: 39
dropped conns
: 0

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

37

Cisco Public

Oracle and SSL


SSL on ACE

Configuration (ACE)
policy-map multi-match ORACLE_TCP_TRAFFIC
class ORACLE_L4
loadbalance vip inservice
loadbalance policy GO_TO_WAE_FARM
loadbalance vip icmp-reply
ssl-proxy server PROXY_1
SSL

# show crypto files


Filename

File File
Expor
Key/
Size Type
table
Cert
----------------------------------------------------------------------cert1.pem
1334 PEM
Yes
CERT
key1.pem
887
PEM
Yes
KEY

Proxy Enabled

class-map match-any ORACLE_L4


2 match virtual-address 101.1.33.50 tcp eq 8000
policy-map type loadbalance first-match GO_TO_WAE_FARM
class class-default
serverfarm WAE_FARM backup ORAAPP_ORACLE_FARM
parameter-map type ssl SSL_PARAM_1
cipher RSA_WITH_RC4_128_MD5
session-cache timeout 3600

SSL Parameter Map

ssl-proxy service PROXY_1


key key1.pem
cert cert1.pem
ssl advanced-options SSL_PARAM_1

SSL Proxy Service

# show stats crypto server


SSL Server Statistics:
SSL alert INTERNAL_ERROR sent:
SSL alert USER_CANCELED sent:
SSL alert NO_RENEGOTIATION sent:
SSLv2 client hello received:
SSLv3 client hello received:
TLSv1 client hello received:
SSLv3 negotiated protocol:
TLSv1 negotiated protocol:
SSLv3 full handshakes:
SSLv3 resumed handshakes:
Cipher sslv3_rsa_rc4_128_md5:
Cipher sslv3_rsa_rc4_128_sha:
Cipher sslv3_rsa_des_cbc_sha:
Cipher sslv3_rsa_3des_ede_cbc_sha:
Cipher sslv3_rsa_exp_rc4_40_md5:
Cipher sslv3_rsa_exp_des40_cbc_sha:
Cipher sslv3_rsa_exp1024_rc4_56_md5:
Cipher sslv3_rsa_exp1024_des_cbc_sha:
Cipher sslv3_rsa_exp1024_rc4_56_sha:
Cipher sslv3_rsa_aes_128_cbc_sha:
Cipher sslv3_rsa_aes_256_cbc_sha:
TLSv1 full handshakes:
TLSv1 resumed handshakes:
Cipher tlsv1_rsa_rc4_128_md5:

0
0
0
48
1626
108
1626
108
1456
90
1626
0
0
0
0
0
0
0
0
0
0
21
87
108

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

ISR 3845
Cisco Public

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

38

19

Gotcha #1 (Incorrectly Formatted hrefs)


SSL Connection Attempt

Incorrectly (Insecure)
Formatted Protocol

Leaving the
SSL Domain

</script>
width="100%"><a id="AppsNavLink" href="http://Insert this info for CLEAR TEXT
Brakiong HREFS

101000111000
000110101010
101011111010
1010111

RST sent to
Client

BRKAPP-2018

ACE
Module
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

39

Cisco Public

HREFS Are Now Formatted Correctly (https://)

SSL Connection Attempt

Maintain
SSL Domain

Secure
Formatted
Protocol

101000111000
000110101010
101011111010
1010111

</script>
width="100%"><a id="AppsNavLink" href="https://wwwinoefin.gslb.dcap.com:8000/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE&akRegionApplicationId=
0&navRespId=54060&navRespAppId=272&navSecGrpId=0&transactionid=596166964&oapc=2&
oas=qTiv-azim1E9ksbjtDKCTA.." class="xd">ABM Manager</a></td></tr><tr><td><img
src="/OA_HTML/cabo/images/t.gif" width="4"></td><td valign="top"><img
id="AppsNavLink" href="https://wwwinoefin.gslb.dcap.com:8000/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE&akRegionApplicationId=
0&navRespId=54061&navRespAppId=272&navSecGrpId=0&transactionid=596166964&oapc

BRKAPP-2018

ACE
Module
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

40

20

What Is the Problem with Redirects?


Since the web server is unaware that SSL offloading is
occurring, the web server will send a 302 redirect
back to the client with a port and protocol in the location
field for what it thinks the client is talking on.
The 302 redirect back to the client will reference a
non-secure port such as :8000 since that is what the
Oracle server is actually listening on.
Ultimately, the client will follow this link, attempt to
connect via a non-secure port, in this case :8000 and
leave the SSL domain as we will see in the next slide.

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Gotcha #2 Insecure Redirect from


Insecure
Redirect

Frame 42 (60 bytes on wire, 60 bytes captured)


Ethernet II, Src: Cisco_e8:1b:91 (00:19:aa:e8:1b:91), Dst: Intel_5d:8f:53 (00:07:e9:5d:8f:53)
Internet Protocol, Src: 101.1.33.50 (101.1.33.50), Dst: 10.0.30.3 (10.0.30.3)
Transmission Control Protocol, Src Port: 8000 (8000), Dst Port: 17769 (17769), Seq: 1437232247, Ack:
2713312565, Len: 0
Source port: 8000 (8000)
Destination port: 17769 (17769)
Sequence number: 1437232247
Acknowledgement number: 2713312565
Header length: 20 bytes
Flags: 0x14 (RST, ACK)
Window size: 32232

RST sent to
Client

RST sent to
the Client

101000111000
000110101010
101011111010
1010111

BRKAPP-2018

ACE
Module
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

42

21

Oracle SSL Java

Opening a form

101000111000
000110101010
101011111010
1010111

BRKAPP-2018

ACE
Module
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

43

Cisco Public

Java Console (SSL)

Java
Console
Java
Console

101000111000
000110101010
101011111010
1010111

BRKAPP-2018

ACE
Module
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

44

22

SSL Sessions

Hypertext Transfer Protocol


GET / HTTP/1.1\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;

Hypertext Transfer Protocol


GET /OA_JAVA/oracle/apps/fnd/jar/fndforms.jar HTTP/1.1\r\n
User-Agent: Java1.3.1.21-internal\r\n

101000111000
000110101010
101011111010
1010111

BRKAPP-2018

ACE14438_04_2008_c2
Module

2008 Cisco Systems, Inc. All rights reserved.

45

Cisco Public

Oracle SSL Configuration


<webentryhost oa_var="s_webentryhost">wwwin-oefin</webentryhost>
<webentrydomain oa_var="s_webentrydomain">gslb.dcap.com</webentrydomain
<webentryurlprotocol
oa_var="s_webentryurlprotocol">https</webentryurlprotocol>
<activewebport oa_var="s_active_webport"
oa_type="PORT">8000</activewebport>
<web_ssl_port oa_var="s_webssl_port" oa_type="PORT">443</web_ssl_port>
<login_page oa_var="s_login_page">https://wwwinoefin.gslb.dcap.com:8000/oa_servlets/AppsLogin</login_page>

All references must


be HTTPS://

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

46

23

Fix on ACE for Redirects


policy-map multi-match
OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERS
class FROM_WAE_VLAN
loadbalance vip inservice
loadbalance policy TO_ORIGIN_SERVERS
loadbalance vip icmp-reply
policy-map multi-match ORACLE_VIPS
class ORACLE_VIP
loadbalance vip inservice
loadbalance policy ORACLE_LB_POLICY
loadbalance vip icmp-reply
ssl-proxy server SSL_PROXY_1

policy-map type loadbalance first-match


ORACLE_LB_POLICY
class class-default
serverfarm WAE_FARM
policy-map type loadbalance first-match
TO_ORIGIN_SERVERS
class class-default
serverfarm ORACLE_SERVERS
action URL_REWRITE

dca-agg-1-ace-1/c2#

show stats http

+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 29717
, TCP data msgs sent
:
Inspect parse result msgs : 0
, SSL data msgs sent
:
sent
TCP fin/rst msgs sent
: 360
, Bounced fin/rst msgs sent:
SSL fin/rst msgs sent
: 11
, Unproxy msgs sent
:
Drain msgs sent
: 0
, Particles read
:
Reuse msgs sent
: 0
, HTTP requests
:
Reproxied requests
: 0
, Headers removed
:
Headers inserted
: 0
, HTTP redirects
:
HTTP chunks
: 0
, Pipelined requests
:
HTTP unproxy conns
: 29486
, Pipeline flushes
:
Whitespace appends
: 0
, Second pass parsing
:
Response entries recycled : 0
, Analysis errors
:
Header insert errors
: 0
, Max parselen errors
:
Static parse errors
: 38
, Resource errors
:
Invalid path errors
: 0
, Bad HTTP version errors :
Headers rewritten
: 3
, Header rewrite errors
:

59395
84
9
29648
29787
29733
0
0
0
0
0
0
0
0
0
0

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

47

Cisco Public

Component Close Up Application


TierConfigurations
Oracle Application Tier is configured in
Active/Active mode across dual
datacenters using shared application
top on Network Appliance NAS.

/apps/oefin/appl_top/admin > df /apps/oefin


Filesystem
1K-blocks
Used Available Use%
Mounted on
nas-oefin.gslb.dcap.com:/vol/dca_oraapp_oefin
157286400 114323320 42963080 73%
/apps/oefin

Oracle/ACE configuration
grep wwwin-oefin OEFIN_dcap-dca-oraapp01.xml
<externURL oa_var="s_external_url">http://wwwinoefin.gslb.dcap.com:8000</externURL>
<webentryhost oa_var="s_webentryhost">wwwin-oefin</webentryhost>
<login_page oa_var="s_login_page">http://wwwinoefin.gslb.dcap.com:8000/oa_servlets/AppsLogin</login_page>
<externURL oa_var="s_external_url">http://wwwinoefin.gslb.dcap.com:8000</externURL>
<webentrydomain
oa_var="s_webentrydomain">gslb.dcap.com</webentrydomain>

Branch Client

BRKAPP-2018
14438_04_2008_c2

DNS Server

GSS

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

NME 502
Cisco Public

Branch WAE

ACE Module

DC WAE

Application
Server

DataBase Server

48

24

Component Close Up Application


TierOracle Forms Configuration
Applet running in browser
for brining up forms in
browser

<server_url oa_var="s_forms_servlet_serverurl">/forms/formservlet</server_url>
<servlet_comment oa_var="s_forms_servlet_comment"/>
<formservlet_session_cookie
oa_var="s_form_session_cookie">true</formservlet_session_cookie>
>
Forms Servlet Configuration
Client interface is provided via a java appliet in a web browser for Oracle forms based
applications
Forms Listener servlet allows http/https transport of forms server traffic from client and
supports standards
load balancing methods
Requires fewer ports to be open in firewall using servlet mode.

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845

WAE 512

ACE Module

101000111000
000110101010
101011111010
1010111

WAE 7326

Application Tier

DataBase Cluster

49

Cisco Public

Component Close Up Database Server


DB tier relies on Application tier for communication from Clients as there is no direct communication b/w Client
and DB tier
Oracle Database 10gR2 configured with Real Application Cluster (RAC) providing HA and scalability for the
Database Tier
Shared SAN storage for RAC configured using Oracle Automatic Storage Management (ASM), various storage
vendors (EMC, HP, Network Appliance), and MDS 9000.
Oracle Interconnect is configured using best practices recommended by Oracle corporation.

<jdbc_url
oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=YES)(FAILOVER=Y
ES)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=dcap-racnode1.gslb.dcap.com)(port=1531))(ADDRESS=(PROTOCOL=tcp)(HOST=dcap-racnode2.gslb.dcap.com)(port=1531)))(CONNECT_DATA=(SERVICE_NAME=OEFIN.dcap.com))
</jdbc_url

HA Configuration for APPS

101000111000
000110101010
101011111010
1010111

Branch Client

BRKAPP-2018
14438_04_2008_c2

DNS Server

GSS

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

NME 502
Cisco Public

Branch WAE

ACE Module

DC WAE

Application
Server

DataBase Server

50

25

App Server to DB Flow Through ACE


Packet Trace (App Server to DB Server)

Network
Processor

640
1346

ACE Connection ID

Branch Client

BRKAPP-2018
14438_04_2008_c2

Branch Name
Server

1
1

Client:
SRC IP : SRC Port

in TCP
out TCP

2132 10.0.20.2:46457
1135 101.1.33.50:8000

Non-Load Balanced Traffic

101.1.33.50:8000
10.0.20.2:46457

ESTAB
ESTAB

Non TCP is
displayed as - -

GSS 4492

2008 Cisco Systems, Inc. All rights reserved.

ISR 3845
Cisco Public

101000111000
000110101010
101011111010
1010111

WAE 512

ACE Module

WAE 7326

Application Tier

DataBase Cluster

51

Disaster Recovery

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

52

26

DR: DCA Down

DCA Down

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

53

Cisco Public

Time Line: DCB Oracle Offline

SSLdump With Decryption

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

54

27

Time Line: DCB Oracle Application


Servers Offline

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

55

Time Line: DCB Oracle Online

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

56

28

Time Line: DCB Oracle Offline

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

57

Cisco Public

In Summary: Cisco Solutions for Oracle


Problem

Solutions

Latency

WAAS

Load Balance

ACE,GSS

Scalability

ACE,GSS

Disaster Recovery

MDS,GSS

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

58

29

Q and A

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

Recommended Reading
Continue your Cisco Live
learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store


BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

Cisco Public

60

30

Complete Your Online


Session Evaluation
Give us your feedback and you could win
fabulous prizes. Winners announced daily.
Receive 20 Passport points for each session
evaluation you complete.
Complete your session evaluation online now
(open a browser through our wireless network
to access our portal) or visit one of the Internet
stations throughout the Convention Center.

Dont forget to activate


your Cisco Live virtual
account for access to
all session material
on-demand and return
for our live virtual event
in October 2008.
Go to the Collaboration
Zone in World of
Solutions or visit
www.cisco-live.com.

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

BRKAPP-2018
14438_04_2008_c2

2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

2006, Cisco Systems, Inc. All rights reserved.


Presentation_ID.scr

31