15.04.

2019 Understanding FabricPath | Data Center and Network Technobabble

Understanding FabricPath
Fabric Path

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/at_a_glance_c45-605626.pdf
[http://w w w .cisco.com/en/US/prod/collateral/sw itches/ps9441/ps9402/at_a_glance_c45-605626.pdf]

- L2 routing protocol (MAC in MAC Routing).

- Alternative to running STP, built-in loop prevention and mitigation.

- Even with vPC, you still have STP.
- vPC is a physical triangle and logical P2P link.
- can have only 2 vPC peers.
- thus distribution blocks can be only 2 upstream switches.

- FP builds arbitrary topologies.

- full mesh
- partial mesh
- triangle
- square

- Single CP for unknown ucast, ucast, bcast and mcast traffic.

- Enhances mobility and virtualization in FP network.

- MAC mobility : physically move L2 node, but retain the same MAC address and VLAN association for the VM.

- FP retains config across an ISSU.

- FP is not TRILL.

Requirements:

- FP is not Ethernet (hence HW support is limited), it is not Ethernet in Ethernet, is Ethernet in FP tunneling.
- F line card.
- Enhanced L2 license (on every system that enables FP).
- NX7K and NX55K.
- Beginning NX-OS 6.1, FEX with vPC+ on F2 cards is supported.

FP terminology

CE : Classical Ethernet
- regular ethernet with regular flooding, STP

Leaf Switch
- connects CE domain to FP domain (NX5K/7K)
- ports in both CE and FP domains
- edge switch

Spine Switch
- FP backbone switch with all ports in FP domain only
- core switch

FP Core Ports
- links leaf to spine
- links spine to spine
- switchport mode fabricpath links

CE Edge Ports
- links on leaf connecting to CE domain
- any link that’s not switchport mode fabricpath

IS-IS

How does CP work behind the scenes in FP - how does it know how to route traffic since we don’t use STP.
- IS-IS used in FP core for L2 routing
dustydev.blogspot.com/2014/04/understanding-fabricpath.html 1/4
15.04.2019 Understanding FabricPath | Data Center and Network Technobabble
- goal is to compute shortest path tree between FP nodes (SPT)
- in FP, IS-IS is not used for advertising MAC addresses
- IS-IS uses its own L3 transport (connection less network protocol, CLNP)
- IS-IS is natively extensible (type length values, TLV)
- IS-IS natively supports ECMP (means L2 load balancing without STP, vPC or PC), known traffic uses ECMP
- FP uses IS-IS. As soon as FP is enabled, L2 IS-IS becomes operational without any configuration
- IS-IS allows all devices to have same view of all trees built by system
- Entire FP network is a flat IS-IS same area network (Level 1 : intra)

FP Switch ID

- Identifies node in IS-IS SPT

- System automatically assigns random unique switch ID to each device enabled with FP
- fabricpath switch-id, show fabricpath switch-id
- Graceful merge - ensures if a SWID conflict arises, the newly joining switch will choose a different ID, and until
then no traffic is disrupted. For a time period, switch has 2 SWIDs, packets from both SWIDs are accepted, but
outgoing packets only carry primary SWID.
- Dynamic Address Resolution Protocol (DRAP) assigns SWID.

FP Data Plane

- L2 over L2 tunneling in FP network

- Receive regular Ethernet from end machines (CE)
- Frame enters FP network from CE network
- Ingress interface encapsulates frame with FP header
- Trees (paths) in FP network assign forwarding tag (FTag)
[Frame] from CE ——— FP : [Frame + FTag]
- From FP to CE, egress interface decapsulates frame and leaves CE header.
[Frame + FTag] from FP ——— CE : - FTag = [Frame]
- FP has SRC and DEST FP Switch ID’s in header
- Traffic is L2 routed via STP to DST Switch ID
- Since FP is HW based, FP does not require you reduce the MTU of a regular Ethernet payload (1500). No need
for fragmentation, TCP MSS.
- Both endpoints of FP core link must have same MTU.
- Since it is implemented in HW, it will natively account for overhead in FP header.
- When partitioned FP network is merged or new switch joins fabric, connecting interface is not enabled for data
before all conflicts are resolved.

FP Header

FP frame : FP Header | CE frame

FP header : outer MAC (OMAC)

OSA : outer MAC source address [ Switch ID ]

FP switch ID of switch when frame ingressing an FP network

ODA : outer MAC dest address [ Switch ID | Subswitch ID]

FP switch ID of switch when frame egressing from FP network

FTag [ Etype 0x8903 | FTag | TTL ]

EtherType tells the L2 switch what the next protocol in the payload is. FP hierarchical MAC address carries the
reserved EtherType 0x8903.
Frame is encapsulated with TTL 32. Each FP switch reduces TTL by 1. When TTL = 0, frame is discarded. This is
loop prevention mechanism. You can configure TTL value.

FTag specifies which of the multiple paths the packet traverses in the FP network. FTag is unique within a FP
topology and is assigned at edge port as frame ingresses FP network.

FP MAC Learning

Conversational MAC Learning

- Only learn SRC MAC if you already know DST MAC
- Each interface learns only those MAC addresses for interested hosts, rather than all
dustydev.blogspot.com/2014/04/understanding-fabricpath.html 2/4
15.04.2019 Understanding FabricPath | Data Center and Network Technobabble
- Host needs to actively speak to interface for interface to learn MAC (if interface down, no MAC learnt)
- Optimization of CP, not DP (does not control how traffic is flooded, it controls how switches insert MAC into their
forwarding tables)
- Default mode for FP VLANs (can be enabled for CE VLANs). (SVI exception).
When an SVI is enabled for a VLAN (say for HSRP purpose), it has to ARP all MACs and thus learn them. Hence C MAC
learning is disabled on the VLAN that terminates an SVI.
- Requires NX-OS 5.1 and later and F series module
- Conversational or traditional type of learning is configured under VLAN
- C MAC learning reduces MAC add table size
- I don’t know DST MAC, so I will not add MAC to my CAM, but I will forward it

FP and STP interaction

- FP Leaf switches must be STP root for CE domain (CE VLANs)

- FP Leaf switches are demarc for STP. STP does not span over FP.
- if leaf switches are STP root, STP is limited to CE domain, a fault does not trigger STP reconvergence in FP
domain, it does though in CE domain
- configure STP priority 4096 on all leaf switches (0 works too).
- STP process is virtualized when you run FP. Switch says it is the root. You will notice FP MAC : c84c.75fa.6000
- No STP in FP network. BPDUs do not traverse FP network, dropped at FP edge, except TCNs.
- FP network pretends to be 1 switch from STP pov. All FP edge switches (leaf) send BPDUs with same Bridge ID
c84c.75fa.60xx (xx=domain ID in HEX)

FP and vPC+

- Each FP Switch has its own ID (node ID in the IS-IS SPT)

- vPC peers would have 2 switch IDs when running FP and vPC
- vPC+ allows FP and vPC to work together by letting vPC peers share a FP Switch ID
- looks like one node from IS-IS SPT
- simplifies L2 multipathing

vPC+ Requirements

- vPC peer-link needs to run as switchport mode fabricpath

- vPC peer-link must be F1 module
- vPC peers share fabricpath switch-id under vPC domain (fool is-is into thinking both are one device)
- vPC to vPC+ migration is disruptive
- BRKDCT-2202 (Cisco Live FP Migration Use Case)
- FEX go offline and reboot (migration)

FP Configuration

Enable FP
- install feature-set fabricpath
- feature-set fabricpath

FP Topology
- fabricpath topology 1

FP SWID
- fabricpath switch-id ID
- Use a convention XYZ - X = Pod number, Y = 5 or 7 denoting 5K/7K, Z = switch number. eq: 171, 172, 271, 272

FP Timers
- fabricpath timers allocate-delay seconds
- fabricpath timers linkup-delay seconds
- fabricpath timers transition-delay seconds

Force FP links to come up

- fabricpath force link-bringup

FP VLANs
- under VLAN : mode fabricpath

FP Core Ports
dustydev.blogspot.com/2014/04/understanding-fabricpath.html 3/4
15.04.2019 Understanding FabricPath | Data Center and Network Technobabble
- switchport mode fabricpath

Additional Features
- fabricpath domain default

Verification

- show fabricpath isis adjacency

- clear fabricpath isis adjacency *
- show mac address-table dynamic vlan ID
- show fabricpath route
- show fabricpath isis hostname
- show fabricpath isis database
- show fabricpath isis interface brief

Troubleshooting Scenarios

- MTU mismatch
- VLAN not present
- VLAN not in FP mode
- STP priority not set on L2 gateways (leaf)
N5K-p2-1# 2014 Jan 14 04:34:09 N5K-p2-1 %STP-2-L2GW_BACKBONE_BLOCK: L2 Gateway Backb one port inconsistency
b locking port Ethernet1/12 on VLAN0200.
N5K-p2-1(config)# spanning-tree vlan 200 priority 4096
N5K-p2-1(config)# 2014 Jan 14 04:34:57 N5K-p2-1 %STP-2-L2GW_BACKBONE_UNBLOCK: L2 Gateway Backb one port
inconsistency cleared unb locking port Ethernet1/12 on VLAN0200.
- vPC+ emulated SWID is not same on peers

Notes:
- With 2 links between FP switches, no EtherChannel is required. IS-IS uses SPT and ECMP to send flows on both
links. Do a ‘show int ethX/Y | i output rate’ to confirm this.
- FP switches see FP SWID (switch ID) in the cam table for FP end devices.
- FP switches need to know MAC of end devices, but not interim FP switches.

- FP is scalable up to 16 switches

Posted 10th April 2014 by Unknown

Location: Lone Tree, CO 80124, USA
Labels: cisco, fabricpath, nexus, nx-os

dustydev.blogspot.com/2014/04/understanding-fabricpath.html 4/4