Troubleshooting Firewalls

Troubleshooting Firewalls
Eric Stuhl Senior Network Consultant Chesapeake NetCraftsmen estuhl@netcraftsmen.net
Copyright 2005
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Packet Flow
Copyright 2005
Understanding the Packet Flow

To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/PIX/FWSM, always Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol Determine the interfaces through which the flow passes
Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both
Copyright 2005
Example Flow
Flow
SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP DST IP: 198.133.219.25 DST Port: 80
Interfaces
Source: Inside Destination: Outside
Client: 10.1.1.9
Servers
Packet Flow
Eng
Accounting Outside Server: 198.133.219.25
With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside
Copyright 2005
Understanding the Packet Flow

Once the device and flow have been identified, walk the path of the packet through the device The packet path through the firewall is illustrated in the next several slides For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making process
Copyright 2005
Packet Processing Flow Diagram

The diagram below will be referenced on the following slides; it is shown here enlarged for reference
Copyright 2005
Packet Processing: Ingress Interface
Packet arrives on ingress interface

Input counters incremented Software input queue is an indicator of load No buffers indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)
Copyright 2005
Packet Processing: Locate Connection
Check first for existing connection If connection exists, flow is matched; bypass ACL check If no existing connection
TCP non-SYN packet, drop and log TCP SYN or UDP packet, pass to ACL checks Established Connection:
ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside
Copyright 2005
Packet Processing: ACL Check
First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"
Copyright 2005
Packet Processing: Match Translation
First packet in flow must match a translation rule* A quick route lookup is done only to determine egress interface Translation rule can be to NAT, or not to NAT NAT order of operations dictates what happens with overlapping translation rules Once translation rule is matched, connection is created
Translation Exists:
ASA-5540# show xlate debug NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00
Syslogs When No Translation Rule Found: (305005No NAT; 305006No Global)

ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80 ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80
Copyright 2005
Translation and NAT Order of Operations
1. 2. 3.
nat 0 access-list (nat-exempt) Match existing xlates Match static commands (Cisco ASA/PIX first match; FWSM best match)
Static NAT with and without access-list Static PAT with and without access-list
First Match
4.
Match nat commands

nat <id> access-list (first match) nat <id> <address> <mask> (best match) If the ID is 0, create an identity xlate Use global pool for dynamic NAT Use global pool for dynamic PAT
Copyright 2005
Packet Processing: Inspections/Sec Checks
Inspections are applied to ensure protocol compliance (Optional) Customized AIC inspections NAT embedded IPs in payload Additional security checks are applied to the packet (Optional) Packets passed to Content Security and Control (CSC) Module
Syslog from Packets Denied by Security Check:

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP
Copyright 2005
Packet Processing: NAT IP Header
Translate the IP address in the IP header Translate the port if performing PAT Update checksums (Optional) Following the above, pass packet to IPS (AIP) module
Copyright 2005
Packet Processing: Egress Interface
Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet) Egress interface is determined first by translation rules If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface Example:
Inside
172.16.0.0/16
Outside DMZ
172.16.12.0/24 172.16.12.4
Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0
Copyright 2005
Packet Processing: L3 Route Lookup
Once on egress interface, an interface route lookup is performed Only routes pointing out the egress interface are eligible Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9
Copyright 2005
Packet Processing: L2 Address Lookup
Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed Layer 2 rewrite of MAC header If Layer 2 resolution failsno syslog show arp will not display an entry for the L3 next hop debug arp will indicate if we are not receiving an ARP reply
Copyright 2005
Packet Processing: Transmit Packet
Packet is transmitted on wire Interface counters will increment on interface Output hardware and software queues indicate buffering at driver level, interface is busy
ASA-5540# show interface gb-ethernet0 interface gb-ethernet0 "outside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c IP address 172.18.124.64, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 3529518 packets input, 337798466 bytes, 0 no buffer Received 32277 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 5585431 packets output, 359059032 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/2) software (0/0)
Copyright 2005
Agenda
Copyright 2005
Cisco ASA/PIXUnderstanding the Architecture

Cisco ASA/PIX platforms process all packets in software (via the central CPU)
All packets are processed first inusually also first out
No software limits on the number of ACEs (rules) that can be configured.

Each ACE takes a minimum of 212 bytes of RAM.
Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)
Copyright 2005
Classifier in Multimode
FWSM has a single MAC address for all interfaces Cisco ASA/PIX has single MAC for shared interfaces (physical interfaces have unique MACs)
Cisco ASA/PIX 7.2 introduces an option to change this
When the firewall receives a packet, it must classify it to determine where to send the packet Packets are classified based on the following
Unique ingress interface/VLAN Packets destination IP matches a global IP
Copyright 2005
Example Inbound traffic is classified to context CTX3, based on the global IP in the static
FWSM Inside 10.1.1.2 Inside VLAN 5 10.1.2.2 Inside VLAN 6 10.1.3.2 static (inside,outside) 10.14.3.89 10.1.3.2
Copyright 2005
DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89 Outside MSFC
SRC IP 192.168.5.4
VLAN 4
Inbound Packet
CTX2
.2
CTX3
.3
Shared Interface
If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*
%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80
*Added to FWSM 3.1

Copyright 2005
Agenda
Copyright 2005
Failover Basics
Active/standby vs. primary/secondary Serial vs. LAN failover Stateful failover (optional) A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall Both firewalls swap MAC and IP addresses when a failover occurs Level 1 syslogs will give reason of failover
Internet
Stateful LAN/Serial Secondary (Standby) Primary (Active)
Corp
Copyright 2005
Verifying Failover Configuration

PIX(config)# show failover Failover On Cable status: N/A - LAN-based failover enabled Failover unit Primary Failover LAN Interface: Failover Ethernet5 (up) Unit Poll frequency 1 seconds, holdtime 3 seconds Interface Poll frequency 5 seconds, holdtime 25 seconds Monitored Interfaces 2 of 250 maximum Version: Ours 7.2(3), Mate 7.2(2) Last Failover at: 18:30:43 UTC Apr 12 2007 This host: Primary - Active Active time: 5371 (sec) Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal Stateful Failover Logical Update Statistics Link : Failover Ethernet5 (up) Stateful Obj xmit xerr General 86 0 sys cmd 74 0
Interface Monitoring
rcv 73 73
rerr 0 0
Copyright 2005
What Triggers a Failover?

Power loss/reload (this includes crashes) on the active firewall SSM interface/module failure The standby becoming healthier than the active firewall
Copyright 2005
What Triggers a Failover?

Two consecutive hello messages missed on any monitored interface forces the interface into testing mode Both units first verify the link status on the interface Next, both units execute the following tests
Network activity test ARP test Broadcast ping test
The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed
Copyright 2005
What to Do After a Failover

Always check the syslogs to determine root cause Example: switch port failed on inside interface of active firewall
Syslogs from Primary (Active) Firewall
ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status Down on interface 1 ASA-1-104002: (Primary) Switching to STNDBYinterface check, mate is healthier
Syslogs from Secondary (Standby) Firewall

ASA-1-104001: (Secondary) Switching to ACTIVEmate want me Active
Copyright 2005
What to Do After a Failover

Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover state This information is not saved across reboots
ASA# show failover state This host State Primary Failed Secondary Active Last Failure Reason Ifc Failure Inside: Failed None Date/Time 12:56:00 UTC May 6 2007
Other host -
====Configuration State=== Sync Done ====Communication State=== Mac set
Copyright 2005
Agenda
Copyright 2005
Troubleshooting Tools
Syslogs Debug commands Show commands Packet capture Packet tracer
Copyright 2005
Uses of Syslogs
Primary mechanism to record traffic to and through the firewall The best troubleshooting tool available
Archival Purposes Debugging Purposes
Console
Syslog Server Internet
SSH Client
Trap SNMP Server Syslog . Buffered
Copyright 2005
ASA Syslog Level vs. Number of Messages
Log Level 0 1 2 3 4 5 6 7
Number of Messages (SUM) Description Ver. 6.3 Emergencies Alerts Critical Errors Warnings Notifications Informational Debugging 0 41 (41) 21 (62) 74 (136) 56 (192) 21 (213) 95 (308) 15 (323) Ver. 7.0 0 62 (62) 29 (91) 274 (365) 179 (544) 161 (705) 234 (939) 217 (1156) Ver. 7.2 0 77 (77) 35 (112) 334 (446) 267 (713) 206 (919) 302 (1221) 258 (1479) Ver. 8.0 0 78 (78) 49 (127) 361 (488) 280 (768) 216 (984) 335 (1319) 266 (1585) Ver. 8.1 0 87 (87) 50 (137) 363 (500) 281 (781) 218 (999) 337 (1336) 267 (1603)
Copyright 2005
FWSM Syslog Level vs. Number of Messages
Log Level 0 1 2 3 4 5 6 7
Number of Messages (SUM) Description Ver. 2.3 Emergencies Alerts Critical Errors Warnings Notifications Informational Debugging 0 58 (58) 21 (79) 94 (173) 131 (304) 26 (330) 116 (446) 23 (469) Ver. 3.1 0 67 (67) 29 (96) 305 (401) 194 (595) 167 (762) 245 (1007) 225 (1232) Ver. 3.2 0 67 (67) 29 (96) 306 (402) 196 (598) 169 (767) 248 (1015) 225 (1240) Ver. 4.0 0 67 (67) 29 (96) 318 (414) 199 (613) 178 (791) 255 (1046) 226 (1272)
Copyright 2005
What Are Modifiable Syslog Levels?

[no] logging message <syslog_id> level <level>
Modifiable syslog levels

Allows one to move any syslog message to any level Levels
0Emergency 1Alert 2Critical 3Errors 4Warnings 5Notifications %PIX-7-111009: User johndoe executed cmd: show run 6Informational 7Debugging
Problem
You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)
The problem is we dont want to log all 1602 other syslogs that are generated at debug level
Copyright 2005
How to Create Modifiable Syslog Levels

Solution
[no] logging message <syslog_id> level <level>
Lower syslog message 111009 to level 3 (error)

ASA(config)# logging message 111009 level 3
Or
ASA(config)# logging message 111009 level error
Now our syslog looks as follows

%ASA-3-111009: User johndoe executed cmd: show run
To restore the default syslog level

ASA(config)# no logging message 111009 level error
Or
ASA(config)# logging message 111009 level 7 http://www.cisco.com/en/US/docs/security/asa/asa80/system/m essage/logmsgs.html
Copyright 2005
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem 2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution 3. Debugs are not conditional* 4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0
Copyright 2005
Debug ICMP Trace

Internet
http://www.cisco.com
Valuable tool used to troubleshoot connectivity issues Provides interface and translation information to quickly determine flow Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled
Example debug icmp trace output
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
Copyright 2005
Logging Debugs to Syslog

Problem
Log only debug output to syslog
Solution
Create a logging list with only syslog ID 711001 Enable debug output to syslogs Log on the logging list
ASA(config)# logging list C-MUG message 711001 ASA(config)# logging debug-trace ASA(config)# logging trap C-MUG
Copyright 2005
Show Output Filters

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Use output filters to filter the output of show command to only the information you want to see To use them, at the end of show <Command>, use the pipe character | followed by
begin
match of output
include exclude grep grep v
Start displaying the output beginning at the first the RegEx, and continue to display the remaining Display any line that matches the RegEx Display any line that does not match the RegEx Same as include Same as exclude
Copyright 2005
Example: Show Output Filters

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Examples
Display the interface stats starting with the inside interface
show interface | begin inside
Display the access-list entries that contain address 10.1.1.5

show access-list | grep 10.1.1.5
Display the config, except for the access-lists

show run | exclude access-list
Display only access-list entries that have non-zero hitcounts

show access-list | grep v hitcnt=0
Display a count of the number of connections each host has

show local-host | include host|count/limit Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted
Copyright 2005
Show CPU Usage

Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled The show cpu usage command displays the CPU over time as a running average
pixfirewall# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%
*First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1)
Copyright 2005
Show Traffic
The show traffic command displays the traffic received and transmitted out each interface of the firewall
ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec
Copyright 2005
Show Xlate and Show Xlate Debug

The show xlate command displays information about the translations through the firewall You can limit the output to just the local or global IP
ASA# show xlate 2 in use, 2381 most used Global 172.18.124.68 Local 10.1.1.9 PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101) debug Adds interface names, idle and xlate ASA# show xlate debug timeouts 2 in use, 2381 most used Flags: D - DNS, d - dump, I - identity, i - inside, n - no random, o - outside, r - portmap, s - static NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:02:03 timeout 3:00:00 TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024 flags r idle 0:00:08 timeout 0:00:30
Copyright 2005
Show Conn and Show Conn Detail

real Interface names added in 7.2(4), 8.0(4)
ASA# show conn 2 in use, 64511 most used TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags
Idle Time, Bytes Transferred
Connection Flags
ASA# show conn detail 2 in use, 64511 most used

Flags: A B E G i k O R R s X -
detail Adds uptime and timeout in 7.2(4), 8.0(4)
awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, outside back connection, F - outside FIN, f - inside FIN, group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, incomplete, J - GTP, j - GTP data, K - GTP t3-response Skinny media, M - SMTP data, m - SIP media, n - GUP outbound data, P - inside back connection, q - SQL*Net data, outside acknowledged FIN, UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
Copyright 2005
ExampleConnection Build Up
1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA 2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A 3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag) 4. The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection 5. Finally, the inside device has sent a data packet and the connection is updated to include the O flag
SYN+ACK SYN Data ACK Connection Flags 42
1 5 3
UI UIO s A UaA
Inside Client Outside Server
Copyright 2005
ExampleConnection Teardown
1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface 2. The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR 3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists
3 1
FIN+ACK ACK FIN
Connection Flags 2
UfFRr UfUfFR
Inside Client Outside Server
Copyright 2005
Connection FlagsQuick Reference

Outbound Connection Inbound Connection
Copyright 2005
TCP Connection Termination Reasons

If a TCP connection is built through the firewall, it will always have a teardown reason The TCP teardown syslog is logged at level 6 If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check the teardown reason
ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]
Copyright 2005
TCP Connection Termination Reasons Quick Reference

Reason Conn-Timeout Deny Terminate Failover Primary Closed FIN Timeout Flow Closed by Inspection Flow Terminated by IPS Flow Reset by IPS Flow Terminated by TCP Intercept Invalid SYN Idle Timeout IPS Fail-Close SYN Control Description Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout Flow Was Terminated by Inspection Feature Flow Was Terminated by IPS Flow Was Reset by IPS Flow Was Terminated by TCP Intercept SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer Than the Timeout Value Flow Was Terminated Due to IPS Card Down Back Channel Initiation from Wrong Side
Copyright 2005
TCP Connection Termination Reasons Quick Reference (Cont.)

Reason SYN Timeout TCP Bad Retransmission TCP Fins TCP Invalid SYN TCP Reset-I TCP Reset-O Description Force Termination After Two Minutes Awaiting Three-Way Handshake Completion Connection Terminated Because of Bad TCP Retransmission Normal Close Down Sequence Invalid TCP SYN Packet TCP Reset Was Sent From the Inside Host TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window Size Variation Tunnel Has Been Torn Down Uauth Deny Unknown Xlate Clear Connection Terminated Due to a Variation in the TCP Window Size Flow Terminated Because Tunnel Is Down Connection Denied by URL Filtering Server Catch-All Error User Executed the Clear Xlate Command
Copyright 2005
show local-host
A local-host entry is created for any IP tracked through the firewall It groups the xlates, connections, and AAA information Very useful for seeing the connections terminating on servers
ASA# show local-host 10.1.1.9 detail Interface inside: 1131 active, 2042 maximum active, 0 denied local host: <10.1.1.9>, TCP connection count/limit = 1/unlimited TCP embryonic count = 0 TCP intercept watermark = 50 UDP connection count/limit = 0/unlimited AAA: user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10) absolute timeout: 0:05:00 inactivity timeout: 0:00:00 Xlate(s): Global 172.18.124.69 Local 10.1.1.9 Conn(s): TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO
Copyright 2005
show service-policy
The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them
ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0 Inspect: ftp, packet 43, drop 0, reset-drop 0 Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: http, packet 562, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, drop 0, reset-drop 0 Inspect: skinny, packet 349, drop 0, reset-drop 0 Inspect: esmtp, packet 0, drop 0, reset-drop 0 ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 349
Copyright 2005
show service-policy flow

Use to determine what policies a given flow will match in the Modular Policy Framework (MPF) Eventually all policies will be in MPF
ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside_policy Class-map: inbound_class Match: access-list telnet_inbound Access rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet Action: Output flow: set connection timeout tcp 0:05:00
Copyright 2005
show asp drop

Packets dropped in the Accelerated Security Path (ASP) will increment a counter Frame drop counters are per packet, flow drops are per flow Some counters have corresponding syslogs
ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) Invalid tcp length (invalid-tcp-hdr-length) Invalid udp length (invalid-udp-length) No valid adjacency (no-adjacency) No route to host (no-route) Reverse-path verify failed (rpf-violated) Flow is denied by access rule (acl-drop) First TCP packet not SYN (tcp-not-syn) Bad TCP flags (bad-tcp-flags) TCP option list invalid (tcp-bad-option-list) TCP MSS was too large (tcp-mss-exceeded) Bad TCP Checksum (bad-tcp-cksum) 10897 9382 10 5594 1009 15 25247101 36888 67148 731 10942 893
*Drop counters are documented in the CMD Ref, under show asp drop
Copyright 2005
Packet Capture
capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any} [eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]
Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the debug packet command 7.2(3) and 8.0(3) added a real-time option ASDM 6.0 adds a capture wizard Capture sniffs packets on an interface that match an ACL, or match line Key steps
Create an ACL that will match interesting traffic Define the capture and bind it to an access-list and interface View the capture on the firewall, or copy it off in .pcap format
Copyright 2005
Packet Capture (Cont.)

Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface Capture buffer saved in RAM (default size 512KB) Default is to stop capturing when buffer is full Default packet length is 1518 bytes Copy captures off via TFTP or HTTPS
Capture Out Outside
Capture In Inside
Copyright 2005
Where Packets Are Captured in Packet Flow
Ingress Packets Captured
Egress Packets Captured
Packets are captured at the first and last points they can be in the flow Ingress packets are captured before any packet processing has been done on them Egress packets are captured after all processing (excluding L2 source MAC rewrite)
Copyright 2005
Capture Command: Example

Problem: User on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (197.133.219.25); the user is getting PATed to 192.168.2.2
Capture In Inside 10.1.3.2 Capture Out Outside www.cisco.com
Internet
198.133.219.25
10.1.3.2
192.168.2.2
Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer Program
Copyright 2005
Capture Command: Example

Step 1: Create ACL for both inside and outside interface
! Outside Capture ACL Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2 ! Inside Capture ACL Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2
Step 2: Create captures on both inside and outside interface

capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518
Step 3: Have inside user access www.cisco.com Step 4: Copy the captures off to a TFTP server
! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap
Or copy using https:

https://<FW_IP>/capture/out/pcap
Copyright 2005
Packet Capture: Example

Step 5: Analyze captures with sniffer program
Outside CAP
Inside CAP
Outbound SYN, No SYN+ACK
Copyright 2005
Capturing Packets Dropped by the ASP

Capture all packets dropped by the ASP
ASA# capture drops type asp-drop all
Capture on a specific drop reason

ASA# capture drops type asp-drop invalid-tcp-hdr-length
ASA# capture drop type asp-drop ? acl-drop all bad-crypto bad-ipsec-natt bad-ipsec-prot bad-ipsec-udp bad-tcp-cksum bad-tcp-flags Flow is denied by configured rule All packet drop reasons Bad crypto return in packet Bad IPSEC NATT packet IPSEC not AH or ESP Bad IPSEC UDP packet Bad TCP checksum Bad TCP flags
Copyright 2005
Packet Tracer
Packet tracer is the future of troubleshooting configuration issues (and many other issues) Introduced in version 7.2 and ASDM 5.2 A packet can be traced by:
Defining the packet characteristics via the CLI Capturing the packets using the trace option
Copyright 2005
Packet Tracer: Overview

A packet tagged with the trace option is injected into the interface, and processed in the data-plane Each action taken on the packet is recorded in the packet itself When the packet reaches the egress interface, or is dropped, it is punted to the control-plane The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration
Copyright 2005
Packet Tracer: Creating Packet via CLI

From the CLI, define the input interface along with source and destination IPs and ports
packet-tracer input <intf> <proto> <src_ip> <src_port> <dst_ip> <dst_port>
ExampleTrace the flow from inside host 10.1.1.2 to http://www.cisco.com (198.133.219.25)

ASA# packet-tracer input inside tcp 10.1.1.2 1025 198.133.219.25 80
Copyright 2005
Packet Tracer: Example Output

ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80 Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: Found no matching flow, creating a new flow Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group in in interface inside access-list in extended permit tcp any any eq www Additional Information: Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map match-all inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect http service-policy global_policy global Additional Information:
Copyright 2005
Packet Tracer: Example Output (Cont.)

... Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information: Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255 ... Phase: 15 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.165.201.1 using egress ifc outside adjacency Active next-hop mac address 000a.f331.83c0 hits 0 >>>>Packet successfully forwarded to fast path<<<<
Copyright 2005
Packet Tracer: Tracing Captured Packet

Create a capture using the trace option
ASA# capture inside access-list web interface inside trace .
Find the packet in the capture you want traced

ASA# 68 1: 2: 3: 4: 5: show capture inside packets captured 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: ...
Then select that packet to be traced
S S . P .
ack ack ack Ack
ASA# show capture inside trace packet-number 4
Copyright 2005
Packet Tracer: ASDM

ASDM includes a nice GUI front-end to the packet tracer tool It is located off the Tools menu Input the packets characteristics in the top half Actions taken on the packet are shown in the bottom half, along with associated config and links back to modify that config entry in ASDM
Copyright 2005
Packet Tracer: ASDM (Screen Shot)
Define Packet
Action Matching Config Link Back to Edit Rule
Final Result
Copyright 2005
Agenda
Copyright 2005
Case Study
Intermittent Access to Web Server
Copyright 2005
Case Study: Intermittent Access to Web Server

Problem Most external clients are not able to load companys web page
NATed to 10.1.1.50
HTTP Requests to 192.168.1.50
Internet Web Server

10.1.1.50 ASA-5510
Clients
Copyright 2005
Traffic Spike
Copyright 2005

show perfmon indicates high number of embryonic connections
ASA-5510# show perfmon PERFMON STATS: Xlates Connections TCP Conns UDP Conns URL Access URL Server Req TCP Fixup TCP Intercept Established Conns TCP Intercept Attempts TCP Embryonic Conns Timeout HTTP Fixup FTP Fixup AAA Authen AAA Author AAA Account VALID CONNS RATE in TCP INTERCEPT: ASA-5510# Current 0/s 2059/s 2059/s 0/s 0/s 0/s 0/s 0/s 0/s 1092/s 0/s 0/s 0/s 0/s 0/s Current N/A Average 0/s 299/s 299/s 0/s 0/s 0/s 0/s 0/s 0/s 4/s 0/s 0/s 0/s 0/s 0/s Average 95.00%
Copyright 2005

Issue show conn to see who is creating the connections
Random Sources Embryonic Conns
ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .
Copyright 2005
Traffic Permitted Connection Count Jumps
SYN Flood Detected
Copyright 2005

Apply TCP Intercept to stop the SYN flood attack
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside
Copyright 2005
TCP Intercept applied
Few clients represent 50+ % of traffic
Copyright 2005

Apply per-client-max option to limit the number of connections any single client can establish
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside
Copyright 2005

per-client-max
TCP Intercept
Copyright 2005
Attacks Being Mitigated by ASA
Attacks Still Occurring

Copyright 2005
Case Study
Poor Voice Quality
Copyright 2005
Case Study: Poor Voice Quality

Problem Poor Outbound Voice Quality at SOHO sites
Outbound RTP Stream
100 Mbps ASA-5505
100 Mbps
Cable Modem
2 Mbps
WAN
Copyright 2005

Solution: Traffic Shaping What is Traffic Shaping, and why is it needed here? Why wont Policing work? Why wont Priority Queuing alone work?
Shape to 2 Mbps
Cable Modem 100 Mbps ASA-5505 100 Mbps
WAN
2 Mbps
Copyright 2005

Configuration Example (Traffic Shaping) Solution Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface.
class-map voice-traffic match dscp af13 ef ! policy-map qos_class_policy class voice-traffic priority ! policy-map qos_outside_policy class class-default shape average 2000000 service-policy qos_class_policy ! service-policy qos_outside_policy interface outside
To view statistics on the operation of the shaper, use the command show service-policy shape
Copyright 2005

Things to keep in mind: Shaping can only be applied to the class class-default Shaping only works in the outbound direction on an interface The shaping value is in bits per second, and must be a multiple of 8000 The shaping policy is applied to all sub-interfaces on a physical interface Not supported on the ASA-5580 platform Not supported in Transparent or Multi-context mode
Copyright 2005
Agenda
Copyright 2005
Tools
ASDM Output interpreter Online learning modules
Copyright 2005
ASDM
Run as a standalone application using the ADSM Launcher This allows for one-stop access to multiple firewalls ASDM 6.0 adds Upgrade Wizard to upgrade ASA and ASDM software direct from cisco.com ASDM 6.1 works with both ASA 8.1 and 8.0 releases ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases
Copyright 2005
ASDM Home Page
Device Information
CPU, Memory, Conns/Sec, Interface Traffic
Real-Time Syslogs
Copyright 2005
Using ASDM for Monitoring
Up to Four Different Graphs Can Be Displayed
Great for Monitoring Trends
Copyright 2005
ASDM: Editing Rules from the Log Viewer
Select Log Entry from Viewer
Right-Click on Message to View or Edit Associated Rule
Copyright 2005
ASDM: Syslogs Explained
Copyright 2005
ASDM 6.0
Drag-and-drop and inplace editing for simplified policy editing User interface customization with dockable windows and toolbars New Firewall Dashboard that provides at-a-glance status of firewall services Live ACL hitcount in firewall rule table for easy policy auditing
Copyright 2005
Output Interpreter
Linked Off the Technical Support and Documentation Tools and Resources Section on CCO
Great Tool for Catching Configuration Errors
Paste in the show run Output and Hit Submit
Copyright 2005
Output Interpreter: Example Output
Warning: Unused Statics
Warning: Unapplied Crypto Map
Warning: Invalid Crypto Map
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
Copyright 2005
Online Learning Modules on CCO

Great way to learn about new features in the ASA Located on CCO From http://www.cisco.com select: Products and Solutions
Security
ASA 5500 Series Adaptive Security Appliances
Training resources Online learning modules
Direct link
http://www.cisco.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list.html
Copyright 2005
Agenda
Copyright 2005
Cisco ASA/PIX/FWSM Best Practices

Enable ip verify reverse-path on all interfaces Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-clientmax
Configure logging to syslog server Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you dont want to see Disable telnet access! Use SSH for management access Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback
Copyright 2005
Cisco ASA/PIX/FWSM Best Practices

Restrict DMZ access inbound to your internal networks Baseline CPU load, connection counts, xlate counts, and traffic (per interface) Run the latest maintenance release in your train Upgrade major feature trains only when you need new features, or after train has matured
Copyright 2005
Cisco ASA/PIX Release Process

Interim Images 7.0(2.1) 7.0(2.2) 7.0(2.3)
GD
GD
Future 7.0(9) EoL
7.0(2)
7.0(4) 7.1(2)
7.0(5)
7.0(6)
7.0(7)
7.0(8)
Maintenance Trains
Bug Fixes Waterfall Down
7.2(2)
7.2(3)
7.2(4) 8.0(3)
7.2(5) 8.0(4) 8.1(2)
7.0(1)
7.1(1)
7.2(1)
8.0(2)
8.1(1)
8.2(1)
Major Feature Releases Time

Copyright 2005
Cisco PIX - End of Sale / End of Life Milestones
Cisco has Announced the End-of-Sale and End-of-Life Dates for Cisco PIX Security Appliances
End of Sale: July 28, 2008 Last day of sale for software, accessories, and licenses: January 28, 2009 End of Software Maintenance Releases: July 28, 2009 End of Support / End of Life: July 27, 2013
Copyright 2005
Gotchas in Upgrading to 7.0

Cisco PIX-515s (non-E) and Cisco PIX-535 should be upgraded from monitor mode Upgrading from monitor mode requires you to copy the 7.0 image over twice
Once from monitor mode (to boot 7.0 and format flash) Once after 7.0 is up and flash has been formatted (to save image in flash)
The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI If there were any errors during the config conversion process, view them by issuing
show startup-config errors
Copyright 2005
Cisco PIX Password Recovery

Password recovery can only be performed by uploading the password recovery utility to the Cisco PIX from monitor mode via TFTP Password recovery will also remove any AAA commands A password recovery utility is created for each major Cisco PIX release (6.1, 6.2, 6.3); however, the utilities are backwards compatible
Example: np63.bin for Cisco PIX 6.3 and prior releases
http://www.cisco.com/warp/public/110/34.shtml
Copyright 2005
Example: Cisco PIX Password Recovery

Example
TFTP Inside monitor> interface 1 monitor> address 14.36.1.88 monitor> file np63.bin 172.18.108.26 14.36.1.88 monitor> gateway 14.36.1.1 monitor> server 172.18.108.26 monitor> tftp tftp np63.bin@172.18.108.26 via 14.36.1.1............... Received 92160 bytes Do you wish to erase the passwords? [yn] y The following lines will be removed from the configuration: enable password 8Ry2YjIyt7RRXU24 encrypted passwd 2KFQnbNIdI.2KYOU encrypted aaa authentication serial console LOCAL aaa authentication telnet console LOCAL aaa authentication ssh console LOCAL aaa authentication enable console LOCAL Do you want to remove the commands listed above from the configuration? [yn] y Passwords and aaa commands have been erased. Rebooting..
Copyright 2005
Case Study
Out-of-order packet buffering
Copyright 2005
Case Study: Out-of-Order Packets

Inspections require ordered packets Packets sent to the SSM (AIP and CSC) require ordered packets Cisco ASA/PIX will buffer up to three packets by default Buffering can be increased on ASA by using the queue-limit option under the tcp-map
Copyright 2005
Case Study: Out-of-Order Packets

Problem Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows If the out-of-order packet buffer isnt large enough, traffic is dropped and packets must be retransmitted
192.168.1.30 Client Inside Outside Server Packet 10 Packet 11 Packet 12 Packet 13 Packet 14 Packet 15 10.16.9.2
Dropped on Network Buffer Dropped by Firewall
Copyright 2005
Case Study: Out-of-Order Packet Buffering Example

How to detect?
ASA# show asp drop Frame drop: ... TCP packet SEQ past window TCP packet buffer full ... 46331 90943
How to fix?
access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global
Copyright 2005
Case Study: Out-of-Order Packet Buffering Example

How to verify?
ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default ... Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0
Copyright 2005
Case Study
TCP MSS (Maximum Segment Size)
Copyright 2005
Case Study: TCP MSS

MSS is the Maximum Segment Sizeor the maximum amount of data that can be sent in a single packet The MSS is set in the SYN packets The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS
Copyright 2005
Case Study: TCP MSS

Problem Some servers have broken TCP stacks and ignore the MSS advertised by the Client The firewall will drop packets that exceed the advertised MSS
Inside Client SYN MSS=1380 Outside Server
192.168.1.30
10.16.9.2
SYN+ACK MSS=1400
DATA=1390
Copyright 2005
Case Study: TCP MSS Example

How to detect?
ASA# show asp drop Frame drop: TCP MSS was too large 943
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
How to fix?
access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global
Copyright 2005
Case Study: TCP MSS Example

How to verify?
ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518 ASA# show capture mss-capture 0 packets captured 0 packets shown
How else could you verify?
Copyright 2005
Case Study
Out of Memory
Copyright 2005
Case Study: Out of Memory

Problem Users are unable to access the Internet No new connections are working All old (long lived) connections continue to work
Step 1: Check the Syslogs

%PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error
Step 2: Check the Amount of Free Memory Available

Hardware: PIX-515E, 64 MB RAM pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ---------------------------Total memory: 67108864 bytes
Copyright 2005

Step 3: What Eats Up Memory (RAM) on the Cisco PIX?
Cisco PIX image (run from RAM) Configuration IPSec database Xlates (translations) Connections What Can Eat Up 64 MB on a Cisco PIX515E?
Step 4: Lets Check the Translations
pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
A Small Global Pool Is Used, Overloading to a PAT Address
Varied Source IPs

Copyright 2005

Step 5: Check the Connections
pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA . . .
Q: Why is the connection count so high?

Copyright 2005

Take a Look at the Traffic Load
pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec
Traffic Flow
Vast majority of traffic is coming in the inside interface and going out the outside interface
Copyright 2005
Inside
Outside

Step 6: Review What We Know and Take Action
pixfirewall# show conn count 147456 in use, 147456 most used pixfirewall# show xlate count 251 in use, 258 most used
Conn Count Is Very High, but xlate Count Is Low

Many connections per xlate Probably one, or a few hosts, are generating the vast majority of connections Most likely due to a virus on the host(s)
Copyright 2005

Step 7: Find the Host(s) Generating All the Connections
pixfirewall# show local-host | local host: <10.1.1.131>, TCP connection count/limit UDP connection count/limit local host: <10.1.1.51>, TCP connection count/limit UDP connection count/limit local host: <10.1.1.236>, TCP connection count/limit UDP connection count/limit . . . local host: <10.1.1.99>, TCP connection count/limit UDP connection count/limit include host|count/limit = 0/unlimited = 0/unlimited = 2/unlimited = 0/unlimited = 0/unlimited = 0/unlimited
Only Show Lines That Have the Word host or count/limit in Them
= 146608/unlimited = 0/unlimited
Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections
Copyright 2005

Step 8: Now That We Found the Host, Lets Look at the Connections It Is Generating
pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 Note: All Connections UDP connection count/limit = 0/unlimited Are Embryonic Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saA TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags saA Connections to Random TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags saA Destinations on TCP/135 TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags saA MS Blaster TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA . . . Copyright 2005

Cisco PIX provides two methods to limit the number of connections per host
TCP intercept Max connections
Question: Which One Can Be Used Here?
TCP intercept wont help because the source address is valid Limiting the maximum number of connections each internal host can have is the only option
Copyright 2005

Step 9: Limit Infected Host(s) Impact on Network Configure the MAX TCP connections for NATed hosts to be 50
pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0
Note: The local-host must be cleared before the new connection limits are applied
pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, The Infected Host Is TCP connection count/limit = 50/50 TCP embryonic count = 50 Limited to 50 TCP TCP intercept watermark = unlimited Connections UDP connection count/limit = 0/unlimited . . .
Copyright 2005

Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit
pixfirewall# show conn count 126 in use, 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ---------------------------Total memory: 67108864 bytes
Things look much better now Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?
nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0
Copyright 2005

Troubleshooting Firewalls

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Troubleshooting Firewalls

Hochgeladen von

Copyright:

Verfügbare Formate

Troubleshooting Firewalls

Eric Stuhl Senior Network Consultant Chesapeake NetCraftsmen estuhl@netcraftsmen.net

Understanding the Packet Flow

Accounting Outside Server: 198.133.219.25

Understanding the Packet Flow

Packet Processing Flow Diagram

Packet Processing: Ingress Interface

Packet arrives on ingress interface

Packet Processing: Locate Connection

Syslog Because of No Connection, and Non-SYN Packet:

Packet Processing: ACL Check

Syslog When Packet Is Denied by ACL:

Packet Processing: Match Translation

Syslogs When No Translation Rule Found: (305005No NAT; 305006No Global)

Translation and NAT Order of Operations

Match nat commands

Packet Processing: Inspections/Sec Checks

Syslog from Packets Denied by Security Check:

Packet Processing: NAT IP Header

Packet Processing: Egress Interface

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

Packet Processing: L3 Route Lookup

Packet Processing: L2 Address Lookup

Packet Processing: Transmit Packet

Cisco ASA/PIXUnderstanding the Architecture

No software limits on the number of ACEs (rules) that can be configured.

DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89 Outside MSFC

*Added to FWSM 3.1

Stateful LAN/Serial Secondary (Standby) Primary (Active)

Verifying Failover Configuration

What Triggers a Failover?

What Triggers a Failover?

What to Do After a Failover

Syslogs from Secondary (Standby) Firewall

What to Do After a Failover

====Configuration State=== Sync Done ====Communication State=== Mac set

ASA Syslog Level vs. Number of Messages

FWSM Syslog Level vs. Number of Messages

What Are Modifiable Syslog Levels?

Modifiable syslog levels

How to Create Modifiable Syslog Levels

Lower syslog message 111009 to level 3 (error)

Now our syslog looks as follows

To restore the default syslog level

* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0

Debug ICMP Trace

Logging Debugs to Syslog

Show Output Filters

Example: Show Output Filters

Display the access-list entries that contain address 10.1.1.5

Display the config, except for the access-lists

Display only access-list entries that have non-zero hitcounts

Display a count of the number of connections each host has

Show CPU Usage

Show Xlate and Show Xlate Debug

Show Conn and Show Conn Detail

Idle Time, Bytes Transferred

ASA# show conn detail 2 in use, 64511 most used

detail Adds uptime and timeout in 7.2(4), 8.0(4)

FIN+ACK ACK FIN

Connection FlagsQuick Reference

TCP Connection Termination Reasons

TCP Connection Termination Reasons Quick Reference