Cisco Nexus 7000 H/W Architecture

Cisco Nexus 7000 Hardware Architecture
BRKARC-3470
BRKARC-3470
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Session Goal
To provide you with a thorough understanding of the Cisco Nexus 7000 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions
This session will not examine NX-OS software architecture or other Nexus platform architectures
Related sessions:
BRKARC-3471: Cisco NX-OS Software Architecture
BRKDCT-2951: Deploying Nexus 7000 in Data Center Networks

BRKDCT-2081: Cisco FabricPath Technology and Design BRKDCT-2048: Deploying Virtual Port Channel in NX-OS BRKDCT-2121: VDC Design and Implementation Considerations with Nexus 7000 BRKARC-3472: NX-OS Routing & Layer 3 Switching BRKCRS-3144: Troubleshooting Cisco Nexus 7000 Series Switches TECDCT-4125: Cisco FabricPath (4h Techtorial) LTRDCT-4047: Deploying Nexus 7000/NX-OS Hands-on Lab (Lab) LTRCRT-5205: Configuring Nexus 7000 Virtualization (Lab)
BRKARC-3470
Cisco Public
What Is Nexus 7000?

Data-center class Ethernet switch designed to deliver high-availability, system scale, usability, investment protection
Supervisor Engines I/O Modules
Chassis
Fabrics
Forwarding Engines
BRKARC-3470
Cisco Public
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture

Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding
Classification Policies NetFlow Summary
BRKARC-3470
Cisco Public
Nexus 7010 Chassis

Integrated cable management with cover
System status LEDs
Front-toback airflow
Air exhaust
Optional front door
Fan trays
21RU
Supervisor slots (5-6) I/O module slots (1-4, 7-10)
Two chassis per 7 rack Crossbar fabric modules
Power supplies Air intake with optional filter

BRKARC-3470
Front
N7K-C7010Cisco Public
Rear
Supported in NX-OS release 4.1(2) and later
Nexus 7018 Chassis

Integrated cable management
System status LEDs

Optional front door Side-to-side airflow
Fan trays
Supervisor slots (9-10)
25RU
Crossbar fabric modules
I/O module slots (1-8, 11-18)
Power supply air intake

BRKARC-3470
Power supplies
Front
N7K-C7018 Cisco Public
Rear
Agenda

BRKARC-3470
Cisco Public
Supervisor Engine
Performs control plane and management functions
Dual-core 1.66GHz x86 processor with 8GB DRAM
2MB NVRAM, 2GB internal bootdisk, compact flash slots, USB
Console, aux, and out-of-band management interfaces Interfaces with I/O modules via 1G switched EOBC Houses dedicated central arbiter ASIC that controls VOQ admission/fabric access via dedicated arbitration path to I/O modules
N7K-SUP1
ID LED
Status LEDs
AUX Port Console Port

BRKARC-3470
USB Ports Management Ethernet Compact Flash Slots

Cisco Public
CMP Ethernet
Reset Button
8
Nexus 7000 I/O Module Families M1 and F1

M family L2/L3/L4 with large forwarding tables and rich feature set
N7K-M148GT-11/N7K-M148GT-11L
N7K-M108X2-12L
N7K-M148GS-11/N7K-M148GS-11L
N7K-M132XP-12/ N7K-M132XP-12L
F family Low-cost, high performance, low latency, low power and streamlined feature set
N7K-F132XP-15
BRKARC-3470
Cisco Public
Supported in NX-OS release 5.0(2a) and later
8-Port 10GE M1 I/O Module

N7K-M108X2-12L
8-port 10G with X2 transceivers
80G full-duplex fabric connectivity Two integrated forwarding engines (120Mpps)

Support for XL forwarding tables (licensed feature)
Distributed L3 multicast replication 802.1AE LinkSec

N7K-M108X2-12L
BRKARC-3470
Cisco Public
11
8-Port 10G XL M1 I/O Module Architecture

N7K-M108X2-12L
EOBC To Central Arbiter
To Fabric Modules
Fabric ASIC
LC CPU
VOQs
Forwarding Engine
Replication Engine
Forwarding Engine
Replication Engine
VOQs
Replication Engine
Replication Engine
10G MAC Linksec

1
10G MAC Linksec

2
10G MAC Linksec

3
10G MAC Linksec

4
10G MAC Linksec

5
10G MAC Linksec

6
10G MAC Linksec

7
10G MAC Linksec

8
Front Panel Ports

BRKARC-3470
Cisco Public
12
N7K-M132XP-12 Supported in all releases N7K-M132XP-12L Supported in NX-OS release 5.1(1) and later
32-Port 10GE M1 I/O Modules

N7K-M132XP-12, N7K-M132XP-12L 32-port 10G with SFP+ transceivers
80G full-duplex fabric connectivity Integrated 60Mpps forwarding engine

XL forwarding engine on L version
Oversubscription option for higher density (up to 4:1) Supports Nexus 2000 (FEX) connections Distributed L3 multicast replication 802.1AE LinkSec
N7K-M132XP-12/ N7K-M132XP-12L
BRKARC-3470
2011 Cisco and/or its affiliates. All Public reserved. Cisco rights
Cisco Public
13
Shared vs. Dedicated Mode

To Fabric
rate-mode shared (default) 10G
11
13
15
Shared mode
Four interfaces in port group share 10G bandwidth Port group group of contiguous even or odd ports that share 10G of bandwidth (e.g., ports 1,3,5,7)
To Fabric
rate-mode dedicated
10G
Dedicated mode
9 11 13 15
First interface in port group gets 10G bandwidth Other three interfaces in port group disabled
BRKARC-3470
Cisco Public
14
32-Port 10G M1 I/O Module Architecture

N7K-M132XP-12, N7K-M132XP-12L
EOBC To Central Arbiter
To Fabric Modules
Fabric ASIC
LC CPU
VOQs
Forwarding Engine
VOQs
Replication Engine Replication Engine
10G MAC 4:1 Mux + Linksec

1 3 5 7

9 11 13 15

17 19 21 23

25 27 29 31

2 4 6 8
10 12 14 16 18 20 22 24 26 28 30 32
Front Panel Ports

BRKARC-3470
Cisco Public
15
48-Port 1G M1 I/O Modules
N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L
Four 1G I/O module options:

48 10/100/1000 RJ-45 ports (N7K-M148GT-11) 48 1G SFP ports (N7K-M148GS-11) 48 1G SFP ports with XL forwarding engine (N7K-M148GS-11L) 48 10/100/1000 RJ-45 ports with XL forwarding engine (N7K-M148GT-11L)
N7K-M148GT-11
All releases
N7K-M148GS-11
Release 4.1(2) and later
Integrated 60Mpps forwarding engine 46G full duplex fabric connectivity

Line rate on 48-ports with some local switching
N7K-M148GS-11L
Release 5.0(2a) and later
Distributed L3 multicast replication 802.1AE LinkSec
N7K-M148GT-11L
BRKARC-3470
Cisco Public
Release 5.1(1) and later

16
48-Port 1G M1 I/O Modules Architecture

N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L
EOBC To Fabric Modules
To Central Arbiter
Fabric ASIC
LC CPU
VOQs
Replication Engine
Forwarding Engine
Replication Engine
12 x 1G MAC
12 x 1G MAC
12 x 1G MAC
12 x 1G MAC
Linksec
Linksec
Linksec
Linksec
Linksec
Linksec
1-12
13-24
Front Panel Ports
25-36
37-48
BRKARC-3470
Cisco Public
17
Supported in NX-OS release 5.1(1) and later
32-Port 1G/10GE F1 I/O Module

N7K-F132XP-15 32-port 1G/10G with SFP/SFP+ transceivers
230G full-duplex fabric connectivity (320G local switching) System-on-chip (SoC) forwarding engine design
16 independent SoC ASICs
Layer 2 forwarding with L3/L4 services (ACL/QoS) Multi-protocol Classic Ethernet, FabricPath, DCB, FCoE
N7K-F132XP-15
BRKARC-3470
Cisco Public
sometimes called switch-on-chip
18
32-Port 1G/10G F1 I/O Module Architecture

N7K-F132XP-15
EOBC To Fabric Modules To Central Arbiter
LC CPU Fabric ASIC Fabric ASIC
Arbitration Aggregator
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
1 2 3 4 5 6
2 X 10G SoC
7 8
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
9 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Front Panel Ports
BRKARC-3470
Cisco Public
19
Agenda

BRKARC-3470
Cisco Public
21
M1 Forwarding Engine Hardware

Hardware forwarding engine(s) integrated on every I/O module 60Mpps per forwarding engine Layer 2 bridging with hardware MAC learning 60Mpps per forwarding engine IPv4 and 30Mpps IPv6 unicast IPv4 and IPv6 multicast support (SM, SSM, bidir)
Hardware Table FIB TCAM Classification TCAM (ACL/QoS) MAC Address Table NetFlow Table
BRKARC-3470
RACL/VACL/PACLs
QoS remarking and policing policies

Policy-based routing (PBR)
Unicast RPF check and IP source guard

Ingress and egress NetFlow (full and sampled)
M1-XL Modules without License 128K 64K M1-XL Modules with License 900K 128K
M1 Modules 128K 64K
128K 512K
Cisco Public
128K 512K
128K 512K
22
M1 Forwarding Engine Architecture

FE Daughter Card
Ingress policing
Ingress Pipeline Egress Pipeline
Ingress NetFlow collection FIB TCAM and adjacency table lookups for Layer 3 forwarding ECMP hashing Multicast RPF check
Ingress ACL and QoS classification

Unicast RPF check
Layer 3 Engine
Egress NetFlow collection
Egress policing
Egress ACL and QoS classification

Ingress MAC table lookups IGMP snooping lookups IGMP snooping redirection
Layer 2 Engine
Egress MAC lookups IGMP snooping lookups

Final lookup result to I/O Module Replication Engine
Cisco Public
Packet Headers from I/O Module Replication Engine

BRKARC-3470
23
F1 Forwarding Engine Hardware

SoC forwarding engine services two front-panel 10G ports
16 SoCs per I/O module
VACL/PACLs
QoS remarking policies FabricPath forwarding Priority Flow-Control (PFC) and Enhanced Transmission Selection (ETS)
480Mpps Layer 2 bridging with hardware MAC learning per I/O module
30Mpps per forwarding engine
Hardware Table
Per SoC 16K 1K in/1K out
Per F1 Module 256K 16K in/16K out
MAC Address Table Classification TCAM (ACL/QoS)
BRKARC-3470
Cisco Public
24
F1 Forwarding Engine Architecture

To/From Central Arbiter
To Fabric
Virtual output queues
From Fabric
Egress forwarding decision (MAC lookup, egress ACL) 2 X 10G SoC
Ingress Buffer (VOQ) Ingress forwarding decision (MAC lookup, FP lookup, ingress ACL/QoS
Forwarding Engine (Egress)
Forwarding Engine (Ingress) Pre-Forwarding Ingress Buffer
MAC Table, ACL, QoS
Egress fabric receive buffer Egress Buffer
Skid buffer Accommodates pause reaction time
Forwarding tables
1G and 10G capable interface MAC
1G/10G MAC
1G/10G MAC
Port A 1G/10G
BRKARC-3470
Two front-panel interfaces per ASIC

Cisco Public
Port B 1G/10G
25
Agenda

BRKARC-3470
Cisco Public
26
Crossbar Switch Fabric Module

Each fabric module provides 46Gbps per I/O module slot
Up to 230Gbps per slot with 5 fabric modules
Different I/O modules leverage different amount of fabric bandwidth

80G per slot with 10G M1 modules
230G per slot with 10G F1 modules
Access to fabric controlled using QoS-aware central arbitration with VOQ
N7K-C7010-FAB-1 N7K-C7018-FAB-1
BRKARC-3470
Cisco Public
27
Multistage Crossbar
Nexus 7000 implements 3-stage crossbar switch fabric
Stages 1 and 3 on I/O modules Stage 2 on fabric modules Fabric Modules
2nd stage
2 x 23Gbps per I/O slot per fabric module
Crossbar Fabric ASIC
Up to 230Gbps per I/O module with 5 fabric modules installed
20 x 23Gbps channels per fabric module

Crossbar Fabric ASIC Crossbar Fabric ASIC
1st stage
Ingress Module
Cisco Public
Egress Module
3rd stage
BRKARC-3470
28
M1 I/O Module Capacity

1G modules
Require 1 fabric for full bandwidth Require 2 fabrics for N+1 redundancy
46Gbps/slot 46Gbps/slot
Fabric Modules
Crossbar Fabric ASICs
230Gbps 46Gbps 184Gbps 138Gbps 92Gbps

per slot bandwidth 4th and 5th fabric modules provide additional redundancy and future-proofing
46Gbps/slot
46Gbps/slot
10G modules
Require 2 fabrics for full bandwidth Require 3 fabrics for N+1 redundancy
BRKARC-3470
46Gbps/slot
Cisco Public
30
F1 I/O Module Capacity

F1 SFP+ module
Operates with any number of fabrics Requires 5 fabrics for maximum bandwidth Redundancy model is graceful bandwidth derating
Fabric Modules
46Gbps/slot
230G
46Gbps/slot
230Gbps 46Gbps 184Gbps 138Gbps 92Gbps

per slot bandwidth
46Gbps/slot
230G
46Gbps/slot
46Gbps/slot
BRKARC-3470
Cisco Public
31
Access to Fabric Bandwidth

Access to fabric controlled using central arbitration
Arbiter ASIC on Supervisor Engine provides fabric arbitration via dedicated arbitration path
Virtual Output Queues (VOQs) at ingress to fabric represent bandwidth availability on egress modules
Four levels of priority per VOQ destination Central arbiter controls admission to VOQ based on bandwidth availability and priority
Buffer credits represent bandwidth availability for each VOQ destination at each priority level
Credits requested by ingress I/O modules with traffic to send into fabric Credits granted by central arbiter based on bandwidth availability Credits returned to the pool by egress I/O modules after receiving traffic from fabric
BRKARC-3470
Cisco Public
32
Benefits of Central Arbitration with VOQ

Ensures priority traffic takes precedence over besteffort traffic across fabric
Four levels of priority for each VOQ destination
Ensures fair access to bandwidth for multiple ingress ports transmitting to one egress port
Central arbiter ensures all traffic sources get appropriate access to fabric bandwidth, even with traffic sources on different modules
Prevents congested egress ports from blocking ingress traffic destined to other ports
Mitigates head-of-line blocking by providing independent queues for individual destinations across the fabric
Enables lossless service for some traffic classes across the fabric
Can provide strict priority and backpressure (blocking instead of dropping) for certain traffic classes, such as FCoE traffic
BRKARC-3470
Cisco Public
34
Agenda
Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture
I/O Module Queuing Layer 2 Forwarding

IP Forwarding IP Multicast Forwarding Classification Policies
NetFlow Summary
BRKARC-3470
Cisco Public
35
I/O Module Queuing

All configuration through Modular QoS CLI (MQC)
Queuing parameters applied using class-maps/policymaps/service-policies
Queuing and scheduling behavior based on I/O module capabilities

Default queuing on all modules enables two classes including egress priority queue
Buffering model varies by I/O module family

M1 modules: hybrid model combining egress port buffered with ingress VOQ buffered architecture F1 modules: pure ingress VOQ buffered architecture
BRKARC-3470
Cisco Public
36
Hybrid Ingress/Egress Buffered Model (M1 I/O Modules)

Ingress port buffer manages congestion in ingress forwarding/replication engines only
Ingress VOQ buffer manages congestion toward egress interface over fabric Egress FIFO buffer just enough to catch frames in flight and keep pipe full
Egress port buffer manages congestion at egress interface
Ingress port buffer Ingress VOQ buffer
Ingress Module
Egress FIFO buffer
Egress port buffer
Ingress Module
Crossbar Fabric
Egress Module
Ingress Module
BRKARC-3470
Cisco Public
37
Ingress Buffered Model (F1 I/O Modules)

Ingress skid buffer just enough to absorb packets received after external flow control asserted Ingress VOQ buffer manages congestion toward egress interfaces Egress FIFO buffer just enough to catch frames in flight and keep pipe full
Ingress skid buffer Ingress VOQ buffer
Ingress Module
Egress FIFO buffer
Ingress Module
Crossbar Fabric
Egress Module
Ingress Module
BRKARC-3470
Cisco Public
38
M1 I/O Module Buffering

Hybrid ingress/egress buffered architecture on 8-port 10G M1 I/O module
To fabric
Egress
LC CPU
Fabric ASIC
FIFO
Shared VOQs
VOQs
Forwarding Engine
Replication Engine
Forwarding Engine
Replication Engine
VOQs
1234
VOQ 16MB
10G MAC
Replication Engine
Replication Engine
1p3q1t
Replication Engine
10G MAC
10G MAC
10G MAC
10G MAC
10G MAC
10G MAC
10G MAC
Linksec
Linksec
Linksec
Linksec
Linksec
Linksec
Linksec
Linksec
5
Front Panel Ports
12345678
Egress 80MB
1p7q4t
Per-Port 10G MAC 10G MAC
12345678
Ingress 96MB
Port 1
Port 2
8q2t
BRKARC-3470
Cisco Public
40
F1 I/O Module Buffering

Pure ingress buffered architecture on 32-port 10G F1 I/O module
To fabric
Egress
LC CPU Fabric ASIC Fabric ASIC Arbitration Aggregator
FIFO
1234
VOQ 1.25MB
Per-Port
1p3q1t
1
Skid
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC 2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC 2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
2 X 10G SoC
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Front Panel Ports
FIFO
Port 1
Port 2
BRKARC-3470
Cisco Public
41
Agenda

BRKARC-3470
Cisco Public
43
Layer 2 Forwarding
MAC table size depends on module type:
M1 MAC table is 128K entries F1 MAC table is 256K entries (16K entries per SoC)
Hardware MAC learning

CPU not directly involved in learning
Forwarding engine(s) on each module have copy of MAC table

New learns communicated to other forwarding engines via hardware flood to fabric mechanism
Software process ensures continuous MAC table sync
Spanning tree (PVRST or MST), Virtual Port Channel (VPC), or FabricPath ensures loop-free Layer 2 topology
BRKARC-3470
Cisco Public
44
Hardware Layer 2 Forwarding Process

In Classic Ethernet and FabricPath edge switches, MAC table lookup drives Layer 2 forwarding
Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs
Source MAC lookup drives new learns and refreshes aging timers
Destination MAC lookup dictates outgoing switchport (CE/FabricPath local) or destination Switch ID (FabricPath remote)
In FabricPath core switches, Switch ID (routing) table lookup drives Layer 2 forwarding
Destination SID lookup dictates outgoing FabricPath interface and next hop
BRKARC-3470
Cisco Public
46
HDR
= Packet Headers
DATA
= Packet Data
CTRL
= Internal Signaling
M1 L2 Packet Flow
Credit grant for fabric access
Supervisor Engine
Central Arbiter
Return credit to pool
12
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC

Transmit to fabric
10
Receive from fabric Return buffer credit
VOQ arbitration and queuing
11
8
Fabric ASIC
ACL/QoS/ NetFlow lookups
Layer 3 Engine
Layer 2 Engine
Forwarding Engine
Layer 3 Engine
Layer 2 Engine
Forwarding Engine
Fabric ASIC
VOQs
VOQs
Submit packet headers for 4 lookup Ingress port QoS
Replication Engine
10G MAC
L2 SMAC/ DMAC lookups Return result
Replication Engine
10G MAC
LinkSec decryption
Receive packet from wire
Linksec
Module 1
13 Module 2
Transmit 15 packet on Cisco Public wire
Egress port QoS
Linksec
LinkSec encryption
14
47
e1/1
e2/1
BRKARC-3470
HDR
= Packet Headers
DATA
= Packet Data
CTRL
F1 L2 Packet Flow
Supervisor Engine
Central Arbiter
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Transmit to fabric
VOQ arbitration and queuing
Fabric ASIC
Fabric ASIC
2 SoC
Ingress L2 & ACL/QoS lookups
1
e1/1
Module 1
e1/2
SoC
Transmit packet on wire
Module 2
e2/2
e2/1
BRKARC-3470
Cisco Public
48
Agenda

BRKARC-3470
Cisco Public
49
IP Forwarding
Nexus 7000 decouples control plane and data plane
Forwarding tables built on control plane using routing protocols or static configuration
OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing
Tables downloaded to forwarding engine hardware for data plane forwarding
BRKARC-3470
Cisco Public
50
IP Forwarding Architecture
Routing protocol processes learn routing information from neighbors IPv4 and IPv6 unicast RIBs calculate routing/next-hop information
Unicast Forwarding Distribution Manager (UFDM) interfaces between URIBs on supervisor and IP FIB on I/O modules IP FIB process programs forwarding engine hardware on I/O modules
FIB TCAM contains IP prefixes
Supervisor Engine
BGP
OSPF
ISIS
RIP
EIGRP
URIB/U6RIB UFDM
IP FIB
Hardware
IP FIB
Hardware
IP FIB
Hardware
Adjacency table contains next-hop information

n7010# PID 20944 n7010# 3573 3574 n7010# 3836 sh processes Runtime(ms) 93 sh processes 117 150 sh processes 1272 cpu | egrep ospf|PID Invoked uSecs 1Sec 33386880 0 0 cpu | egrep u.?rib 44722390 0 0 34200830 0 0 cpu | egrep ufdm 743933460 0 0
Process ospf u6rib urib ufdm
I/O Module
I/O Module
I/O Module
Hardware FIB TCAM ADJ Table
module-9# sh processes cpu | egrep fib 1534 80042 330725 242 0.0 module-9#
BRKARC-3470
ipfib
Cisco Public
51
Hardware IP Forwarding Process

FIB TCAM lookup based on destination prefix (longest-match)
FIB hit returns adjacency, adjacency contains rewrite information (next-hop) Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result
BRKARC-3470
Cisco Public
52
IPv4 FIB TCAM Lookup

Generate Lookup Key 10.1.1.10
10.1.1.2 10.1.1.3 10.1.1.4 10.10.0.10
Generate TCAM lookup key (destination IP address)
Compare lookup key
Ingress unicast IPv4 packet header

Flow Data
Forwarding Engine
Next-hop 1 (IF, MAC) Next-hop 2 (IF, MAC)
Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops
Load-Sharing Hash
10.10.0.100 10.10.0.33
10.1.1.xx 10.1.2.xx 10.1.3.xx 10.10.100.xx
HIT!
Offset
Next-hop 3 (IF, MAC)

Return lookup result
Index, # next-hops
Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops
Hit in FIB Index, returns result# next-hops in FIB DRAM Adjacency index identifies ADJ block
Cisco Public
Next-hop 4 (IF, MAC) # nexthops Adj Index Next-hop 5 (IF, MAC) Next-hop 6 (IF, MAC) Next-hop 7 (IF, MAC)
10.1.1.xx 10.100.1.xx 10.10.0.xx 10.100.1.xx

FIB TCAM
BRKARC-3470
Result
FIB DRAM
Hash selects exact next hop entry
Adjacency Table
53
ECMP Load Sharing

Up to 16 hardware load-sharing paths per prefix
10.10.0.0/16
Use maximum-paths command in routing protocols to control number of load-sharing paths Load-sharing is per-IP flow Configure load-sharing hash options with global ip load-sharing command:
Source and Destination IP addresses
Source and Destination IP addresses plus L4 ports (default) Destination IP address and L4 port
Additional randomized number added to hash prevents polarization

Automatically generated or user configurable value
10.10.0.0/16 via Rtr-A via Rtr-B
BRKARC-3470
Cisco Public
57
HDR
= Packet Headers
DATA
= Packet Data
CTRL
M1 L3 Packet Flow
Fabric Module 1
Fabric ASIC
Transmit to fabric VOQ arbitration and queuing
Supervisor Engine 9
Central Arbiter
12
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC

10
8
Fabric ASIC
6
Layer 3 Engine
Layer 2 Engine
Forwarding Engine
L3 FIB/ADJ lookup Ingress and egress ACL/QoS/ NetFlow lookups
11
Layer 3 Engine
Fabric ASIC
VOQs
Submit packet headers for 4 lookup
Replication Engine
10G MAC
L2 ingress and egress SMAC/ DMAC lookups Return result
Layer 2 Engine
Forwarding Engine
VOQs
Replication Engine
10G MAC
Ingress port QoS
3
Linksec
LinkSec decryption
Module 1
13 Module 2
Transmit packet on 15 Cisco Public wire
Egress port QoS
Linksec
LinkSec encryption
14
60
e1/1
e2/1
BRKARC-3470
Layer 3 Forwarding with F1 I/O Modules

F1 modules do not natively provide Layer 3 switching
Cannot inter-VLAN route on their own
However, one or more M1/M1-XL modules can provide proxy Layer 3 services
M1 ports can proxy route for F1 modules Proxy L3 forwarding enabled by default when VDC in mixedmodule mode
Packets destined to router MAC forwarded to M1 modules for Layer 3 via internal Router Port-Channel
Selection of which port on which M1 module based on EtherChannel hash function Traffic requiring L3 from F1 modules traverses the fabric, vectoring toward M1 ports enabled for proxy L3 M1 module receiving such packets programmed to perform full ingress/egress L3 lookups
BRKARC-3470
Cisco Public
61
Proxy L3 Forwarding
VLAN 10 DMAC router_mac
10.1.10.100 vlan 10 e2/1 e2/2

SoC
10.1.20.100 vlan 20 e3/1 e3/2

SoC
Dest Port internal channel
2 3
F1
interface vlan 10 ip address 10.1.10.1/24 ! interface vlan 20 ip address 10.1.20.1/24
Fabric ASIC
F1
Fabric ASIC
Fabric Module
Fabric
M1
Fabric
Routing: DIP Server 2
MAC: VLAN 20 Next Hop 10.1.20.100
VOQs
6
DMAC server_2
Forwarding Engine
Forwarding Engine
VOQs
Dest Port e3/1

7
Replication Engine 5
Replication Engine
BRKARC-3470
e1/1 e1/2 e1/3
Cisco Public
e1/4
e1/5 e1/6
e1/7 e1/8
62
Agenda

BRKARC-3470
Cisco Public
64
IP Multicast Forwarding
Forwarding tables built on control plane using multicast protocols
PIM-SM, PIM-SSM, PIM-Bidir, IGMP, MLD
Tables downloaded to:

Forwarding engine hardware for data plane forwarding Replication engines for data plane packet replication
BRKARC-3470
Cisco Public
65
IP Multicast Forwarding Architecture

Multicast routing processes learn routing information from neighbors/hosts IPv4 and IPv6 multicast RIBs calculate multicast routing/RP/RPF/OIL information
Supervisor Engine
PIM IGMP PIM6 ICMP6 BGP MSDP MRIB/M6RIB MFDM
Multicast Forwarding Distribution Manager (MFDM) interfaces between MRIBs on supervisor and IP FIB on I/O modules IP FIB process programs hardware:
FIB TCAM
IP FIB
IP FIB
IP FIB
Adjacency table
Multicast Expansion Table (MET) n7010# PID 3842 3850 n7010# 3843 3847 n7010# 3846 sh processes Runtime(ms) 109 133 sh processes 177 115 sh processes 2442 cpu | egrep pim|igmp|PID Invoked uSecs 1Sec Process 32911620 0 0 pim 33279940 0 0 igmp cpu | egrep m.?rib 33436550 0 0 mrib 47169180 0 0 m6rib cpu | egrep mfdm 743581240 0 0 mfdm
Hardware
I/O Module
Hardware
I/O Module
Hardware
I/O Module
Hardware
FIB TCAM
MET
ADJ Table
module-9# sh processes cpu | egrep fib 1534 80153 330725 242 0.0 module-9#
BRKARC-3470
ipfib
Cisco Public
66
IPv4 Multicast FIB TCAM Lookup

Generate TCAM lookup key (source and group IP address)
Ingress multicast packet header

Compare lookup key
Generate Lookup Key 10.1.1.10, 239.1.1.1
Forwarding Engine
RPF, ADJ Index RPF, ADJ Index
RPF, ADJ Index RPF, ADJ Index RPF, ADJ Index FIB DRAM
10.1.1.12, 239.1.1.1 10.1.1.10, 232.1.2.3

10.4.7.10, 225.8.8.8
HIT! 10.1.1.10, 239.1.1.1
MET Index MET Index MET Index

MET Index
Adj Index
10.6.6.10, 239.44.2.1
FIB TCAM
Hit in FIB returns result in FIB DRAM
MET Index
Adjacency Table
Result
Identifies multicast adjacency entry
Replication Engine
Replication for each OIF in MET block
OIFs OIFs OIFs OIFs

MET
Cisco Public
Replicate
MET index used to find OIFs for replication
BRKARC-3470
67
Egress Replication
IIF
Distributes multicast replication load among replication engines of all I/O modules with OIFs
Local OIF
Module 1
Replication MET Engine
Input packets get lookup on ingress forwarding engine For OIFs on ingress module, ingress replication engine performs the replication For OIFs on other modules, ingress replication engine replicates a single copy of packet over fabric to those egress modules
Each egress forwarding engine performs lookup to drive replication
2 Fabric Module
Fabric ASIC
Fabric Copy
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Replication engine on egress module performs replication for local OIFs

Local OIFs
BRKARC-3470
Local OIFs
Local OIFs
71
Cisco Public
HDR
= Packet Headers
DATA
= Packet Data
L3 Multicast Packet Flow

Fabric Module 1
Fabric ASIC
Transmit to fabric
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Dequeue multicast distribution copy from fabric
11
VOQ queuing
10
Fabric ASIC
Transmit multicast fabric distribution packet 9 Submit packet headers for 4 lookup Ingress port QoS
6
Layer 3 Engine Layer 2 Engine
Forwarding Engine
L3 multicast FIB lookup Ingress ACL/QoS/ NetFlow lookups Egress ACL/QoS/ NetFlow lookups L2 ingress snooping lookup
12
Layer 3 Engine
Fabric ASIC
Replicate for local OIF delivery
VOQs
Replication Engine 10G MAC
15
Layer 2 Engine
VOQs
13
Return MET result
Linksec
Replicate for fabric delivery
Forwarding Engine
Egress port QoS
Replication Engine
10G MAC
Submit packet headers for egress lookups
14
LinkSec encryption
Module 1
LinkSec decryption
16
L2 egress snooping lookup
17 Module 2
Transmit packet on wire
Linksec
18
e1/1
2
19
e2/1
BRKARC-3470
Cisco Public
72
Agenda

BRKARC-3470
Cisco Public
73
Classification Policies
Enforce security and QoS policies based on Layer 2, Layer 3, and Layer 4 information Classification TCAM (CL TCAM) provides ACL lookups in forwarding engine
64K(non-XL) or 128K (XL) hardware entries in M1 modules 1K ingress/1K egress hardware entries per SoC in F1 modules (16K/16K entries per module)
Security policies
Router ACL (RACL) for IPv4, IPv6, ARP VLAN ACLs (VACLs) for IPv4, MAC Port ACLs (PACLs) for IPv4, MAC Secure Group Tag ACLs (SGACLs) for Cisco TrustSec
QoS classification policies for IPv4 and IPv6

Ingress (Per-L2 switchport, per-VLAN, per-interface routed/subinterface) Egress (Per-VLAN, per-interface routed/subinterface)
BRKARC-3470
Cisco Public
74
CL TCAM Lookup (ACL)

Packet header: SIP: 10.1.1.1 DIP: 10.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80
Compare lookup key
Generate TCAM lookup key (source/dest IPs, protocol, L4 ports, etc.)
Security ACL ip access-list example permit ip any host 10.1.2.100

deny ip any host 10.1.68.44 deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23
deny udp any any eq 514
Generate Lookup Key
SIP | DIP | Protocol | SPORT | DPORT
permit tcp any any eq 80 permit udp any any eq 161
10.1.1.1 | 10.2.2.2 | 06 | 84C8 | 0050
Forwarding Engine
Permit Deny
xxxxxxx | 10.2.2.2 xx | xxx | xxx xxxxxxx | 10.1.2.100| |xx | xxx | xxx
xxxxxxx | 10.1.68.44 | xx | xxx | xxx xxxxxxx | 10.33.2.25 | xx | xxx | xxx
Deny
Permit Deny Deny Permit Result
X=Mask
xxxxxxx | xxxxxxx | |06 | |xxx | 0050 xxxxxxx xxxxxxx 06 xxx | 0016

xxxxxxx | xxxxxxx | 06 | xxx | 0017 xxxxxxx | xxxxxxx | 11 | xxx | 0202
HIT! xxxxxxx | xxxxxxx | 06 | xxx | 0050

xxxxxxx | xxxxxxx | 11 | xxx | 00A1 CL TCAM
Hit in CL TCAM returns result in CL SRAM
Cisco Public
Permit
CL SRAM
Result affects final packet handling
BRKARC-3470
76
Agenda

BRKARC-3470
Cisco Public
78
NetFlow
NetFlow support provided by M1 I/O modules
No support for F1F1 or F1M1 proxy-routed flows
NetFlow table is 512K entries, shared between ingress/egress NetFlow Hardware NetFlow entry creation
CPU not involved in NetFlow entry creation/update
All modules have independent NetFlow table Full and sampled NetFlow supported by hardware
BRKARC-3470
Cisco Public
79
Full vs. Sampled NetFlow

NetFlow configured per-direction and per-interface
Ingress and/or egress on per-interface basis
Each interface can collect full or sampled flow data Full NetFlow: Accounts for every packet of every flow on interface, up to capacity of NetFlow table Sampled NetFlow: Accounts for M in N packets on interface, up to capacity of NetFlow table
BRKARC-3470
Cisco Public
83
Sampled NetFlow
Random packet-based sampling
M:N sampling: Out of N consecutive packets, select M consecutive packets and account only for those flows in the hardware NetFlow table
Sampled flows aged and exported from NetFlow table normally Advantages
Reduces NetFlow table utilization Reduces CPU load on switch and collector
Disadvantages
Accuracy may be sacrificedcollector or user must extrapolate total traffic load based on configured sampling rate
BRKARC-3470
Cisco Public
84
NetFlow Data Export

To NetFlow Collector
Generate NetFlow v5 or v9 export packets
I/O Module
Fabric ASIC via Inband VOQs
Supervisor Engine
LC CPU
NetFlow Table
Aged Flows
Forwarding Engine I/O Module

LC CPU
NetFlow Table
Hardware Flow Creation
Main CPU
Switched EOBC
Aged Flows
Forwarding Engine I/O Module

LC CPU
NetFlow Table
via mgmt0
Mgmt Enet
To NetFlow Collector
Aged Flows
Forwarding Engine
BRKARC-3470
Cisco Public
87
Agenda

BRKARC-3470
Cisco Public
89
Nexus 7000 Architecture Summary

Variety of front-panel interface and transceiver types with LinkSec, VOQ, and other advanced hardware features
Future-proofed chassis designs with density and airflow options
Control plane protocols, system and network management
Supervisor Engines
I/O Modules
Chassis
Fabrics
Forwarding Engines
BRKARC-3470
Hardware services, including unicast/multicast, bridging/routing, ACL/QoS classification, and NetFlow statistics
Cisco Public
Lossless-capable fabric with 230G/slot bandwidth to interconnect I/O modules and provide investment protection
90
Conclusion
You should now have a thorough understanding of the Nexus 7000 switching architecture, I/O module design, packet flows, and key forwarding engine functions
Any questions?
BRKARC-3470
Cisco Public
91
Complete Your Online Session Evaluation

Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Dont forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and ondemand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
BRKARC-3470
Cisco Public
93
Visit the Cisco Store for Related Titles http://theciscostores.com
BRKARC-3470
Cisco Public
94
BRKARC-3470
Cisco Public
95
Thank you.
BRKARC-3470
Cisco Public
96

Cisco Nexus 7000 H/W Architecture

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Cisco Nexus 7000 H/W Architecture

Hochgeladen von

Copyright:

Verfügbare Formate

Cisco Nexus 7000 Hardware Architecture

2011 Cisco and/or its affiliates. All rights reserved.

BRKDCT-2951: Deploying Nexus 7000 in Data Center Networks

What Is Nexus 7000?

Supervisor Engine and I/O Module Architecture

Classification Policies NetFlow Summary

2011 Cisco and/or its affiliates. All rights reserved.

Nexus 7010 Chassis

System status LEDs

Optional front door

Supervisor slots (5-6) I/O module slots (1-4, 7-10)

Two chassis per 7 rack Crossbar fabric modules

Power supplies Air intake with optional filter

Supported in NX-OS release 4.1(2) and later

Nexus 7018 Chassis

System status LEDs

Supervisor slots (9-10)

Crossbar fabric modules

I/O module slots (1-8, 11-18)

Power supply air intake

N7K-C7018 Cisco Public

Supervisor Engine and I/O Module Architecture

Classification Policies NetFlow Summary

2011 Cisco and/or its affiliates. All rights reserved.

AUX Port Console Port

USB Ports Management Ethernet Compact Flash Slots

2011 Cisco and/or its affiliates. All rights reserved.

Nexus 7000 I/O Module Families M1 and F1

Supported in NX-OS release 5.0(2a) and later

8-Port 10GE M1 I/O Module

8-port 10G with X2 transceivers

80G full-duplex fabric connectivity Two integrated forwarding engines (120Mpps)

Distributed L3 multicast replication 802.1AE LinkSec

2011 Cisco and/or its affiliates. All rights reserved.

8-Port 10G XL M1 I/O Module Architecture

10G MAC Linksec

10G MAC Linksec

10G MAC Linksec

10G MAC Linksec

10G MAC Linksec

10G MAC Linksec

10G MAC Linksec

10G MAC Linksec

Front Panel Ports

32-Port 10GE M1 I/O Modules

80G full-duplex fabric connectivity Integrated 60Mpps forwarding engine

Shared vs. Dedicated Mode

32-Port 10G M1 I/O Module Architecture

Replication Engine Replication Engine

Replication Engine Replication Engine

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

10G MAC 4:1 Mux + Linksec

Front Panel Ports

48-Port 1G M1 I/O Modules

N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L

Four 1G I/O module options:

Integrated 60Mpps forwarding engine 46G full duplex fabric connectivity

Distributed L3 multicast replication 802.1AE LinkSec