Sie sind auf Seite 1von 35


Jack Fenimore
FSE, Central and Southern Ohio
Where am I starting from?
What are we troubleshooting?
 Did this work before?
 Does the traffic go through the F5?
 Is it reproducible?
 Is there a log server?
 Did the timing of the issue coincide with any other changes?
 Before beginning determine what devices are involved
 Obtain or create a network diagram from the client to the F5 to the pool
Network Map
Module Statistics
Statistics -> Performance
What does BIG-IP iHealth do?
• Displays a snapshot of the BIG-IP system configuration
in a user-friendly format
• Evaluates the configuration against a database of
known issues, common errors, and published F5 best
• Provides tailored feedback about configuration issues,
a description of the issue, recommendations for
resolution, and a link to additional information in the
AskF5 Knowledge Base
Displays System Configuration Snapshot

View all uploaded qkview files

iHealth Diagnostics Page

Reports configuration issues and provides a link to additional

information in AskF5
Packet Flow Review
How Does Traffic Enter a BIG-IP?

• Routing to a listener on the BIG-IP

• Listeners are Internet

• Self IPs
• NATs
• Virtual Servers
External VLAN
NAT to
Packet Processing Priority

1. Existing connection in connection table

2. Packet filter rule
3. Virtual server
5. NAT
6. Self-IP
7. Drop
BIG-IP Virtual Server Types Client

HTTP response HTTP request

• Standard DST: DST:

• Forwarding IP BIG-IP LTM

http_vs chooses RED
VLAN Internal VLAN External

HTTP response
DST: HTTP request

The default gateway for the RED BLUE

RED and BLUE servers is http_pool :8080 :8080 on BIG-IP LTM
Standard Virtual Server Packet Flow
iRules Proxy iRules


Load balancing
iRules HTTP HTTP iRules

iRules SSL SSL iRules

iRules Express Express

VS listener

IPv4 IPv6 IPv4 IPv6

Forwarding (IP) VS Packet Flow

Forward Request

VS listener iRules

IPv4 IPv6 IPv4 IPv6

Virtual Server Priority
1. Specific IP address and specific port
2. Specific IP address and all ports*
3. Network IP address and specific port netmask
4. Network IP address and all ports* netmask
5. All networks and specific port netmask
6. All networks and all ports* netmask
Up Through the Layers
Layer 1
tmsh show /net interface 1.1 all-properties field-fmt
Layer 2
tmsh show /net arp dynamic
tmsh show /net fdb
Layer 3
• Ping

• Check routes

• Tracepath utility

• Traceroute from both directions

• Telnet to the remote port

[root@3900-1:Active:In Sync] config # tracepath

1: ( 0.175ms pmtu 1500
1: ( 2.981ms reached
Resume: pmtu 1500 hops 1 back 1
Connections - - any6.any - any6.any
tmsh show /sys conn
2 - -
Type any
tmsh show /ltm persistence persist-records 3
Acceleration none
Type any
Protocol tcp
Time 52
Protocol Idletcp Timeout 300
Idle Time Unit2 ID 1
Idle Timeout 300
Lasthop /Common/external 00:18:19:9e:b4:75
ID config
1 # Path
Virtual tmsh show ltm persistence persist-records client-addr Lasthop /Common/external 00:18:19:9e:b4:75
Sys::Persistent ConnectionsVirtual Path ClientSide ServerSide
source-address 3
Client Addr any6.any
Total records returned: 1 Server AddrClientSide ServerSide any6.any
Client AddrBits10.0.180.140:51711
5.4K 0
Server AddrBits
4.8K 0
Bits In Packets In 52.9K 6 67.2K 0
Bits Out Packets Out 134.9K 5 54.2K 0
Packets In 21 15
Packets Out 36 25
Situations Specific to F5
MAC Masquerade

• Unique MAC assigned to a traffic group

• Minimize ARP communication or dropped packets during failover by
using a consistent MAC address
• Improve reliability and failover speed
• Improve interoperability with switches slow to process gARP’s
• When a BIG-IP becomes active it will send a gARP for all Virtual
IP’s for which it is now active. If link down on failover is set it will
also perform an interface reset, dropping carrier momentarily
• SOL13502 (SOL7214 for v10.x)
Review of Auto Last Hop
• Tracks the source MAC address and VLAN of incoming connections.
• Return traffic from pools is sent to the MAC transmitted the request,
• Even if the routing table points to a different network or interface
• The BIG-IP can send return traffic to clients even if no matching route.

• Auto Last Hop is a desired behavior and so it is enabled by default.

• F5 Networks recommends leaving enabled
• Under rare circumstances you may want to disable Auto Last Hop

• If disabled the routing table is used to forward the packet

• SOL11796: Overview of the Auto Last Hop setting

TCP Reset Cause

• Informs where and why a TCP reset was generated. (SOL13223)

• A diagnostic enhancement
• Use as necessary for troubleshooting
• Added for all profiles which could cause a TCP RST
• Stream
• FastL4
• FastHTTP 3900-1 err tmm3[8641]: 01230140:3: RST sent from to
• etc., [0x173b10d:5961] TCP RST from remote system
Viewing Reset Cause

• Insert into TCP reset (packet captures)

- tmsh mod sys db tm.rstcause.pkt {value "enable"}
- The default is “disabled”

• Send to syslog (/var/log/ltm)

• tmsh mod sys db tm.rstcause.log {value “enable”}
• The default is “disabled”

• Show reset cause stats

• tmsh show net rst-cause
RST Packets Containing Data (RFC1122)
• What do the RFCs have to say about this?
• A TCP SHOULD allow a received RST to include data.
• It has been suggested that a RST segment could contain ASCII text that
encoded and explained the cause of the RST. No standard has yet been
established for such data.

• Some other stacks do the same (e.g., HP-UX and MacOS)

• Has been known to cause issues in the field
Wireshark Plug-In
• Available from
Pool action on service down
How the system should respond when the target pool member
becomes unavailable – pool object property.

• None: Specifies that the system maintains existing connections,

but does not send new traffic to the member (default)

• Reject: Use "Reject" when you want LTM to explicitly close both
sides of the connection when the server goes DOWN

• Drop: Specifies that the system simply cleans up the

connection, no reset will be sent

• Reselect: Specifies that the system manages established client

connections by moving them to an alternative pool member
I did ABC and now when I log in the GUI I see:

“The configuration has not yet loaded. If this message persists, it may
indicate a configuration problem.”

To determine what is wrong:

tmsh load /sys config partitions all

Attack Prevention and Dynamic Reaping
• SYN flood, DDoS, DoS attack prevention 6 PRESENTATION
• SYN Cookies*

• Dynamic Reaping
• Continually monitors existing TCP connections to ensure
the integrity of the connection table 2 DATA LINK
• Removes the oldest idle connections if it needs to clear 1 PHYSICAL
up more memory
• Protects the BIG-IP against SYN attacks from non-
spoofed IP addresses that fully negotiate a connection
• Avoid changing default values without Support
* The article provides an elaborate explanation
of SYN cookies
Tips on General Configuration
• Set DNS and NTP

• Re-activate your license before upgrading (*Will impact traffic)

• Adjust the Number of Records Per Screen

• Set up a floating IP address on each VLAN

• Understand the BIG-IP operates in STP pass-thru mode

• Virtual Address vs Virtual Server, disabling ARP

• Nagles algorithm