Brkarc 3470 Cisconexus7000 7700switcharchitecture2016lasvegas 2hours 160930084524

Cisco Nexus 7000 / 7700 Switch
Architecture
Nemanja Kamenica (nkamenic@cisco.com)
Engineer, Technical Marketing
BRKARC-3470
Session Abstract
This session presents an in-depth study of the architecture of the latest generation
of Nexus 7000 and Nexus 7700 data center switches. Topics include supervisors,
fabrics, I/O modules, forwarding engines, and physical design elements, as well
as a discussion of key hardware-enabled features that combine to implement
high-performance data center network services.
BRKARC-3470
2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Goal
To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching

architecture, chassis, supervisor, fabric, and I/O module design, key forwarding
engine functions, and complete packet flows
This session will not examine NX-OS software architecture or other Nexus
platform architectures
BRKARC-3470
Agenda
Introduction to Nexus 7000 / Nexus 7700
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
Introduction to Nexus 7000 / Nexus 7700 Platform

Data-center class Ethernet switches designed to deliver high performance, high
availability, system scale, and investment protection
Designed for wide range of Data Center deployments, focused on feature-rich
10G/40G/100G density and performance
Supervisor Engines
Chassis
I/O Modules
Fabrics
BRKARC-3470
Nexus 7000 / Nexus 7700 Common Foundation

Nexus 7000
Nexus 7700
General purpose DC switching w/10/40/100G
Targeted at Dense 40G/100G deployments
Common Foundation
Same release vehicles, versioning, feature-sets

Common configuration model
Common operational model
Common fabric ASICs (Fab2) and architecture

Same central arbitration model
Same VOQ/QOS model
Identical forwarding ASICs (F2E, F3, M3)
Consistent hardware feature sets
Consistent hardware scale
BRKARC-3470
Agenda
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
NX-OS 4.1(2) and later
Nexus 7000 Chassis Family

Back
Nexus 7010
Nexus 7018
25RU
Side
21RU
Side
Front
Front
N7K-C7010
Rear
Front
N7K-C7018
Rear

Nexus 7004
Nexus 7009
Back
14RU
Side
7RU
Side
Side
Front
Front
N7K-C7009
N7K-C7004
Rear
Rear
BRKARC-3470
Nexus 7718
Nexus 7700 Chassis Family

Back
Nexus 7710
Back
26RU
14RU
Front
Front
Front
Front
Rear
Rear
N77-C7718
N77-C7710
Nexus 7706
NX-OS 7.2(0)D1(1) and later

Back
Nexus 7702
Back
3RU
9RU
Front
Front
Front
Front
Rear
N77-C7702
Rear
N77-C7706
BRKARC-3470
10
Agenda
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
Supervisor Engine 2 / 2E
Provides all control plane and management functions

Supervisor Engine 2 (Nexus 7000)
Supervisor Engine 2E (Nexus 7000 / Nexus 7700)
Base performance
High performance
One quad-core 2.1GHz CPU with 12GB DRAM
Two quad-core 2.1GHz CPU with 32GB DRAM
Connects to fabric via 1G inband interface
Interfaces with I/O modules via 1G switched EOBC
Onboard central arbiter ASIC
N77-SUP2E
Controls access to fabric bandwidth via dedicated arbitration path to I/O modules
N7K-SUP2/N7K-SUP2E
ID and Status
LEDs
ID and Status
LEDs
Management
Console Port
Ethernet
USB Host
Ports
USB Log
Flash
USB Expansion
Flash
BRKARC-3470
Console Port
USB Expansion
Flash
Management
Ethernet
12
Supervisor Engine 2 / 2E Architecture

To Module CPUs
Switched
1GE EOBC
To Module VOQs
To Fabric Modules
Dedicated
Arbitration
Path
Fabric ASIC
Switched
EOBC
VOQs
Dedicated
Arbitration
Path
Central
Arbiter
1GE Inband
I/O Controller
Bootflash
(eUSB)
NVRAM
2GB
Main CPU
32MB
Main CPU
Sup2E
Only
DRAM
Console
Mgmt0
USB expansion
USB logflash
USB device port
2.1GHz
Quad-Core
BRKARC-3470
12GB/32GB
2.1GHz
Quad-Core
13
Reference: Component Functions Supervisor

Engines
Main CPU(s) Runs all system-level NX-OS processes and handles all control
plane and management functions
Switched EOBC Provides switch 1G connections to each module CPU for

internal system management and communication
I/O Controller Provides all I/O functions for supervisor components
VOQs Interface to central arbiter and local crossbar fabric, implements Virtual
Output Queuing
Fabric ASIC Local fabric that provides first/third stage of three-stage crossbar
Central Arbiter Dedicated ASIC that controls access to fabric based on

destination interface and priority of requests
BRKARC-3470
14
M1
M2
1G / 10G
M3 delivers best of Mand F-series capabilities
10G / 40G / 100G
M-Series Modules
10G / 40G /
M3
L2/L3/L4 with large forwarding tables and rich feature set
F1
F2/F2E
10G
F3
10G
F-Series Modules
F3 closes the
F/M feature gap!
10G / 40G / 100G
High performance, low latency with streamlined feature set
F2E
10G / 40G / 100G
10G
BRKARC-3470
F3
15
Nexus 7000 M2 I/O Modules
10G / 40G / 100G M2 I/O modules
Share common hardware architecture multi-chipset
Two integrated forwarding engines (120Mpps)
Layer 2/Layer 3 forwarding with L3/L4 services

(ACL/QOS) and advanced features
(MPLS/OTV/GRE etc.)
Large forwarding tables (900K FIB/128K ACL)
802.1AE LinkSec on all ports
N7K-M224XP-23L
N7K-M206FQ-23L
N7K-M202CF-22L
Module
Port Density
Optics
Bandwidth
M2 10G
24 x 10G (plus Nexus 2000 FEX support)
SFP+
240G
M2 40G
6 x 40G (or up to 24 x 10G via breakout)
QSFP+
240G
M2 100G
2 x 100G
CFP
200G
BRKARC-3470
16
Nexus 7000 M2 I/O Module Architecture

N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
EOBC
To Fabric Modules
To Central Arbiters
LC
CPU
Fabric ASIC
Arbitration
Aggregator
Forwarding
Engine
VOQs
VOQs
VOQs
Replication
Engine
VOQs
Forwarding
Engine
Replication
Engine
Replication
Engine
Replication
Engine
LinkSec +
12 X 10G MAC -or3 X 40G MAC -or1 X 100G MAC
LinkSec +
12 X 10G MAC -or3 X 40G MAC -or1 X 100G MAC
Front Panel Ports
BRKARC-3470
17
Reference: ASIC Functions M2 Modules
LinkSec + MAC Provides port ASIC functions, including buffering/queuing, and performs 802.1ae
encryption/decryption for front-panel ports
Replication Engine Bridge between front panel port, forwarding engine, and fabric; performs multicast
and SPAN replication
Forwarding Engine Performs all Layer 2, Layer 3, and Layer 4 forwarding decisions and policy
enforcement
VOQs Interface to central arbiter and local crossbar fabric, implements Virtual Output Queuing
Arbitration Aggregator Muxes arbitration requests from VOQs before sending to central arbiter on
Supervisor Engine
(LC CPU Linecard CPU, runs module-specific NX-OS processes and interfaces with Supervisor
Engine over EOBC)
BRKARC-3470
18
M2 Module 40G and 100G Flow Limits
M2 modules use 10G Virtual

Queuing Index (VQI)
Ingress Modules
Each VQI sustains 10G traffic flow

Spines
Spines
Spines
Spines
Fabrics
All packets in given 5-tuple flow

hash to single VQI using portchannel load-balancing algorithm
Single-flow limit is 10G
Destination
VQIs
10G 10G
40G
40G
100G
1 VQI 1 VQI
4 VQIs
4 VQIs
10 VQIs
Egress Interfaces
BRKARC-3470
19
Nexus 7000 F3 I/O Modules
10G / 40G / 100G F3 I/O modules

Share common hardware architecture
SOC-based forwarding engine design
N7K-F348XP-25
N7K-F312FQ-25
6 independent SOC ASICs per module
Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS)

and advanced features (FP/MPLS/OTV/GRE/VXLAN etc.)
Onboard Fabric Services Accelerator (FSA)
Require Supervisor Engine 2 / 2E
Module
Port Density
Optics
Bandwidth
F3 10G
48 x 1G/10G
SFP+
480G
F3 40G
12 x 40G (or up to 48 x 10G via breakout)
QSFP+
480G
F3 100G
6 x 100G
CPAK
550G
N7K-F306CK-25
BRKARC-3470
20
Nexus 7700 F3 I/O Modules
10G / 40G / 100G F3 I/O modules

N77-F348XP-23
6 independent SOC ASICs per 10G module

12 independent SOC ASICs per 40G/100G module
Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS)

and advanced features (FP/MPLS/OTV/GRE/VXLAN etc.)
N77-F324FQ-25
Module
Port Density
Optics
Bandwidth
F3 10G
48 x 1G/10G
SFP+
480G
F3 40G
24 x 40G (or up to 76 x 10G + 5 x 40G via

breakout)
QSFP+
960G
F3 100G
12 x 100G
CPAK
1.2T
N77-F312CK-26
BRKARC-3470
21
Nexus 7000 F3 48-Port 1G/10G Module Architecture

N7K-F348XP-25
To Fabric Modules
EOBC
To Central Arbiters
FSA
CPU
Arbitration
Aggregator
x6
1G switch
Fabric ASIC
x6
LC Inband
to FSA
CPU
to ARB
8 X 10G
SOC 1
3
2
8 X 10G
SOC 2
5
4
7
6
9
8
11
10
12
8 X 10G
SOC 3
13
14
15
16
17
18
19
21
20
22
8 X 10G
SOC 4
23
24
25
26
27
28
29
8 X 10G
SOC 5
31
30
32
33
34
35
36
8 X 10G
SOC 6
37
38
39
40
41
42
43
45
44
47
46
48
Front Panel Ports (SFP/SFP+)

BRKARC-3470
22
Nexus 7000 F3 12-Port 40G Module Architecture

N7K-F312FQ-25
To Fabric Modules
EOBC
To Central Arbiters
Arbitration
Aggregator
FSA
CPU
x6
1G switch
Fabric ASIC
x6
LC Inband
to FSA
CPU
to ARB
2 X 40G
SOC 1
2 X 40G
SOC 2
2 X 40G
SOC 3
2 X 40G
SOC 4
2 X 40G
SOC 5
10
2 X 40G
SOC 6
11
12
Front Panel Ports (QSFP+)

BRKARC-3470
23

N7K-F306CK-25
To Fabric Modules
EOBC
To Central Arbiters
Arbitration
Aggregator
FSA
CPU
x6
1G switch
Fabric ASIC
x6
LC Inband
to FSA
CPU
to ARB
1 X 100G
SOC 1
1 X 100G
SOC 2
1 X 100G
SOC 3
1 X 100G
SOC 4
1 X 100G
SOC 5
1 X 100G
SOC 6
Front Panel Ports (CPAK)
BRKARC-3470
24
Nexus 7700 F3 48-Port 1G/10G Module Architecture

N77-F348XP-23
To Fabric Modules
EOBC
To Central Arbiters
FSA
CPU
Arbitration
Aggregator
x6
1G switch
Fabric ASIC
Fabric ASIC
x6
LC Inband
to FSA
CPU
to ARB
8 X 10G
SOC 1
3
2
8 X 10G
SOC 2
5
4
7
6
9
8
11
10
12
8 X 10G
SOC 3
13
14
15
16
17
18
19
21
20
22
8 X 10G
SOC 4
23
24
25
26
27
28
29
8 X 10G
SOC 5
31
30
32
33
34
35
36
8 X 10G
SOC 6
37
38
39
40
41
42
43
45
44
47
46
48

BRKARC-3470
25

N77-F324FQ-25
To Fabric Modules
EOBC
To Central Arbiters
FSA
CPU
Arbitration
Aggregator
x 12
1G switch
Fabric ASIC
Fabric ASIC
x 12
LC Inband
to FSA
CPU
to ARB
2 X 40G
SOC 1
2 X 40G
SOC 2
2 X 40G
SOC 3
2 X 40G
SOC 4
2 X 40G
SOC 5
2 X 40G
SOC 6
10
11
12
2 X 40G
SOC 7
13
14
2 X 40G
SOC 8
15
16
2 X 40G
SOC 9
17
18
2 X 40G
SOC 10
19
20
2 X 40G
SOC 11
21
22
2 X 40G
SOC 12
23
24

BRKARC-3470
26

N77-F312CK-26
To Fabric Modules
EOBC
To Central Arbiters
FSA
CPU
Arbitration
Aggregator
x 12
1G switch
Fabric ASIC
Fabric ASIC
x 12
LC Inband
to FSA
CPU
to ARB
1 X 100G
SOC 1
1 X 100G
SOC 2
1 X 100G
SOC 3
1 X 100G
SOC 4
1 X 100G
SOC 5
1 X 100G
SOC 6
1 X 100G
SOC 7
1 X 100G
SOC 8
1 X 100G
SOC 9
1 X 100G
SOC 10
1 X 100G
SOC 11
1 X 100G
SOC 12
10
11
12
Front Panel Ports (CPAK)

BRKARC-3470
27
Reference: ASIC Functions F3 Modules
8 X 10G / 2 X 40G / 1 X 100G SOC 10/40/100G capable system-on-chip;

provides Port ASIC, Replication Engine, Forwarding Engine, and VOQ functions
Arbitration Aggregator Muxes arbitration requests from SOCs before sending

to central arbiter on Supervisor Engine
(FSA CPU Fabric Services Accelerator, a linecard CPU with built-in application
acceleration for higher performance BFD, sampled Netflow, and other functions;
runs module-specific NX-OS processes and interfaces with Supervisor Engine
over EOBC)
BRKARC-3470
28
Nexus 7700 M3 I/O Modules
10G / 40G / 100G M3 I/O modules

Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS) and advanced

features (FP*/MPLS/OTV/GRE/VXLAN etc.)
Large forwarding tables (2M FIB*/128K ACL)
256-bit 802.1AE LinkSec on all ports, all speeds
Module
Port Density
Optics
Bandwidth
M3 10G
48 x 1G/10G
SFP+
480G
M3 40G
24 x 40G
QSFP+
960G
N77-M348XP-23L
N77-M324FQ-25L
* Hardware capability
BRKARC-3470
29
Nexus 7700 M3 48-Port 1G/10G Module Architecture

To Fabric Modules
EOBC
To Central Arbiters
10G FSA
CPU
Arbitration
Aggregator
Fabric ASIC
to SOC 2
TCAM
Fabric ASIC
to SOC 1
TCAM
Buffer
Buffer
to FSA
CPU
to ARB
24 X 10G
SOC 1
3
2
5
4
7
6
9
8
11
10
12
13
14
24 X 10G
SOC 2
15
16
17
18
19
20
21
22
23
24
25
26
27
29
28
31
30
32
33
34
35
36
37
38
39
40
41
42
43
45
44
47
46
48
BRKARC-3470
30
Nexus 7700 M3 24-Port 40G Module Architecture

To Fabric Modules
EOBC
To Central Arbiters
10G FSA
CPU
Arbitration
Aggregator
Fabric ASIC
to SOC 4
to SOC 3
TCAM
Buffer
to ARB
TCAM
6 X 40G
SOC 1
Fabric ASIC
TCAM
Buffer
6 X 40G
SOC 2
10
to SOC 1
TCAM
to FSA
Buffer CPU
Buffer
6 X 40G
SOC 3
11
12
13
14
15
16
x4
to FSA
CPU
6 X 40G
SOC 4
17
18
19
20
21
22
23
24
BRKARC-3470
31
Reference: ASIC Functions M3 Modules
24 X 10G / 6 X 40G / 2 X 100G SOC 10/40/100G capable system-on-chip;

provides Port ASIC, Replication Engine, Forwarding Engine, and VOQ functions
TCAM External FIB/ACL TCAM for storing routing and policy entries
Buffer External packet buffer memory
Arbitration Aggregator Muxes arbitration requests from SOCs before sending

to central arbiter on Supervisor Engine
(FSA CPU Fabric Services Accelerator, a linecard CPU with built-in application
acceleration for higher performance BFD, sampled Netflow, and other functions;
runs module-specific NX-OS processes and interfaces with Supervisor Engine
over EOBC)
BRKARC-3470
32
Fabric Services Accelerator (FSA) for F3 / M3
High-bandwidth inband connectivity from SOCs to

FSA (6Gbps for F3, 20Gbps for M3)
Multi-Mpps packet processing
2 X 2GB dedicated DRAM
Performance/scale boost for distributed fabric

services
High-rate sampled Netflow 50Kpps sampled

copies per module
EOBC
High-performance module CPU with on-board

acceleration engines
Distributed BFD 15 msec hello timer, 45

msec dead time for 250 sessions per module
FSA CPU
Dual-Core
LC CPU
2GB
DRAM
Acceleration
Engines
I/O
2GB
DRAM
6 x 1Gbps (F3) or 2 x 10Gbps (M3)

Module Inband
BRKARC-3470
33
F3 / M3 Module 40G and 100G Flows
Virtual Queuing Index (VQI)

sustains 10G, 40G, or 100G traffic
flow based on destination interface
type
Ingress Modules
Spines
Spines
Spines
Spines
Fabrics
No single-flow limit full 40G/100G

flow support
Destination
VQIs
10G 10G
40G
40G
100G
1 VQI 1 VQI
1 VQI
1 VQI
1 VQI
Egress Interfaces
BRKARC-3470
34
I/O Module Interoperability
General module interoperability rule is: +/-1 generation in same Virtual Device
Context (VDC)
F3 interoperability: coexists with either M2, or with F2/F2E, in same VDC
M3 interoperability: coexists with F3 in same VDC
Interoperability model for current modules is Ingress Forwarding with Lowest

Common Denominator
No proxy routing with F3 / M3
BRKARC-3470
35
Ingress Forwarding with Lowest Common

Denominator Model
Module receiving packet from wire makes all forwarding decisions for that packet
Supported feature set and scale based on Lowest Common Denominator
Feature available if all modules support the feature

Table sizes based on lowest capacity
Module Types
in VDC
Layer 2
Layer 3
VPC
MPLS
OTV
Fabric
Path
VXLAN
Table Sizes
M3
M3 size
M3 + F3
F3 size
F3
F3 size
F3 + M2
F3 size
F3 + F2E
F2E size
M2 + F2E + F3
Not supported
BRKARC-3470
36
Interoperability Forwarding Model Matrix

Module Combination
Interoperability Model
M1 + M2
Lowest Common Denominator
M + F1
Proxy Forwarding
M + F2E
Proxy Forwarding
F2 + F2E
F2 + F2E + F3
M2 + F3
M3 + F3
M2 + F2/F2E + F3
NOT SUPPORTED
BRKARC-3470
37
Agenda
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
Crossbar Switch Fabric Modules

N77-C7718-FAB-2
N77-C7710-FAB-2
N77-C7706-FAB-2
Provide interconnection of I/O modules
Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC
Each installed fabric increases available per-payload slot bandwidth

Fabric Module
Supported Chassis
Per-fabric module
bandwidth
Max fabric
modules
Total bandwidth per

slot
Nexus 7000 Fabric 2
7009 / 7010 / 7018
110Gbps per slot
550Gbps per slot
Nexus 7700 Fabric 2
7706 / 7710 / 7718
220Gbps per slot
1.32Tbps per slot
Different I/O modules leverage different amount of available fabric bandwidth
Access to fabric bandwidth controlled using QOS-aware central arbitration with

VOQ
N7K-C7018-FAB-2
N7K-C7010-FAB-2
N7K-C7009-FAB-2
BRKARC-3470
39
Multistage Crossbar
Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric
Stages 1 and 3 on I/O modules
Stage 2 on fabric modules
2nd stage
Fabric Modules
Fabric Modules
1
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
550G
1.32T
220G
(4 x 55G)
110G
(2 x 55G)
1st stage
Fabric
ASIC
Fabric
ASIC
Fabric ASIC
Fabric ASIC
Ingress
Module
Egress
Module
3rd stage
1st stage
Fabric ASIC
Fabric ASIC
Ingress Module
Nexus 7000
Fabric ASIC
Egress Module
Nexus 7700
BRKARC-3470
Fabric ASIC
3rd stage
40
I/O Module Capacity Nexus 7000

One fabric:
550Gbps
110Gbps
440Gbps
220Gbps
330Gbps
Fabric 2 Modules
per slot bandwidth
Fabric
ASIC
Any port can pass traffic to any

other port in VDC
Local Fabric
(240G)
Fabric
ASIC
Three fabrics:
240G M2 module has maximum

bandwidth
Local Fabric
(480G)
Fabric
ASIC
Five fabrics:
Fabric
ASIC
480G F2E/F3 module has maximum

bandwidth
Fabric
ASIC
BRKARC-3470
41
What About Nexus 7004?
Nexus 7004 has no fabric modules
Each I/O module has local fabric with 10 available fabric channels
I/O modules connect back-to-back via 8 fabric channels
Two fabric channels borrowed to connect supervisor engines
Sup Slot 1
Fabric
ASIC
Sup Slot 2
Fabric
ASIC
2 * 55G
fabric channels
M2/F3
Module 3
Fabric
ASIC
Fabric
ASIC
M2/F3
Module 4
8 * 55G local fabric channels

interconnect I/O modules (440G)
BRKARC-3470
42
I/O Module Capacity Nexus 7700

Fabric 2 Modules
1320Gbps
1100Gbps
880Gbps
660Gbps
440Gbps
220Gbps
One fabric:
per slot bandwidth
Local Fabric
#1 (480G)
Fabric
ASICs
Local Fabric
#1 (960G)
Fabric
#2
480G F2E/F3 10G module has maximum

bandwidth
960G F3 40G module has maximum

bandwidth
Fabric
ASICs
Local Fabric
#1 (1.2T)
Fabric
#2
5
Fabric
ASICs
Six fabrics:
3
Fabric
ASICs
Five fabrics:
Fabric
ASICs
Any port can pass traffic to any other port

in VDC
Three fabrics:
1
Fabric
#2
1.2T F3 100G module has maximum

bandwidth
6
Fabric
ASICs
BRKARC-3470
43
What About Nexus 7702?
Nexus 7702 has no fabric modules
Single I/O module all traffic locally switched
Two fabric channels connect to supervisor engine
F3/M3
Module
Fabric
ASIC
Fabric
ASIC
1* 55G
fabric channel
Supervisor
Fabric ASIC
BRKARC-3470
44
Agenda
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
Hardware Forwarding Process
Ingress queuing and scheduling
Perform forwarding lookups
Forward through fabric
Egress queuing and scheduling
BRKARC-3470
46
Ingress / Egress Queuing and Scheduling
Nexus 7000 / Nexus 7700 use queuing policies and network-QOS policies to
define queuing and scheduling behavior
Default policies always in effect in absence of any user configuration
Queuing model varies by I/O module architecture
M2 modules: hybrid model combining egress-buffered architecture with ingressbuffered architecture

F-series modules / M3 modules: pure ingress-buffered architecture
BRKARC-3470
47
M2 10G module used as example
M2 Hybrid Ingress/Egress Buffered

Ingress port buffer Manages congestion of ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Egress VOQ buffer Receives frames from fabric
Egress port buffer Manages congestion at egress interface
Flow control
boundary
Ingress
port buffer
Ingress
VOQ buffer
Ingress Module
Egress
VOQ buffer
Crossbar
Fabric
Egress
port buffer
Egress Module
BRKARC-3470
48
N7700 10G F3 module used as example
F3/M3 Ingress Buffered

Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Egress VOQ buffer Receives frames from fabric
For more information about queuing and scheduling on Nexus 7000:
BRKDCT-3346 - End-to-End QoS Implementation and Operation with
Cisco
Nexus Switches
Ingress
Egress
VOQ buffer
VOQ
buffera.m.
Thursday, Jul 14, 8:00
Ingress Module
Crossbar
Fabric
Egress Module
BRKARC-3470
49
F3 Buffer Sharing
10G F3 used as example
Default Dedicated per Port Buffer

e1/1
All buffer
dedicated
per port
e1/2
e1/3
e1/4
e1/5
e1/6
e1/7
Optional Shared Buffer + Dedicated per Port Buffer

e1/8
q1
q1
q1
q1
q1
q1
q1
q1
q2
q2
q2
q2
q2
q2
q2
q2
q3
q3
q3
q3
q3
q3
q3
q3
q4
q4
q4
q4
q4
q4
q4
q4
Dedicated
portion
e1/1
e1/2
e1/3
e1/4
e1/5
e1/6
e1/7
e1/8
q1
q2
q3
q4
q1
q2
q3
q4
q1
q2
q3
q4
q1
q2
q3
q4
q1
q2
q3
q4
q1
q2
q3
q4
q1
q2
q3
q4
q1
q2
q3
q4
q1
Shared
portion
q2
q3
q4
Ingress
Buffer
Ingress
Buffer
F3 SOC
All ingress buffer carved equally among ports
One port cannot use another ports unused buffer
F3 SOC
Total buffer split into dedicated per-port portion and

shared portion
Any port can consume buffer in shared portion
Increases burst absorption, assuming not all ports

burst simultaneously
BRKARC-3470
50
Hardware Forwarding Lookups
Layer 2 and Layer 3 packet flow virtually identical in hardware
Forwarding engine / decision engine pipeline provides consistent L2 and L3

lookup performance
Pipelined architecture also performs ingress and egress ACL, QOS, and Netflow
lookups, affecting final forwarding result
BRKARC-3470
51
M2 Forwarding Engine Hardware
Two hardware forwarding engines

integrated on every M2 I/O module
OTV / GRE
RACL/VACL/PACL
Layer 2 switching (with hardware MAC

learning)
QOS remarking and policing policies
Layer 3 IPv4/IPv6 unicast and multicast
Ingress and egress Netflow (full and

sampled)
MPLS/VPLS/EoMPLS
Hardware Table
M-Series Modules
without Scale License
M-Series Modules with

Scale License
MAC Address Table
128K
128K
FIB TCAM
128K IPv4 / 64K IPv6
900K IPv4 / 350K IPv6
Classification TCAM (ACL/QOS)
64K IPv4 / 32K IPv6
Netflow Table
1M
1M
BRKARC-3470
52
M1/2-Series Forwarding Engine Architecture

FE Daughter Card
L3 Engine
Layer 3 FIB
FIB TCAM
Egress Netflow
collection
Netflow
Table
Netflow
FIB TCAM and adjacency table

lookups for Layer 3 forwarding
ECMP hashing
Ingress Netflow
collection
Egress ACL/QOS
classification
Policing
CL TCAM
Classification
(ACL/QOS)
Ingress policing
Egress lookup
pipeline
Egress policing
Ingress lookup
pipeline
Ingress ACL/QOS
classification
L2 Engine
MAC
Table
L2 Lookup (post-L3)
L2 Lookup (pre-L3)
Egress MAC lookups
Ingress MAC table

lookups
Port-channel hash result
Ingress Parser
HDR
From I/O Module
Replication Engines
Final Results
Receive packet header

for lookup from
Replication Engine
To I/O Module
Replication
Engines
BRKARC-3470
2016
Return final result

(destination + priority)
to Replication Engine
Cisco and/or its affiliates. All rights reserved. Cisco Public
53
F3 Forwarding Engine Hardware
8 x 10G, 2 x 40G, or 1 x 100G SOC with

decision engine
RACL/VACL/PACL

learning)
Ingress/egress sampled Netflow
Layer 3 IPv4/ IPv6 unicast and multicast
MPLS/VPLS/EoMPLS
FabricPath forwarding
OTV / GRE tunnels
LISP
VXLAN
Hardware Table
F3 Capacity
MAC Address Table
64K
FIB TCAM
64K IPv4/32K IPv6
16K IPv4 / 8K IPv6
BRKARC-3470
54
F3 Decision Engine
Return final result

(destination + priority)
to Ingress Buffer
To Ingress
Buffer
Final Results
Egress MAC lookups
L2 Lookup (post-L3)
FIB TCAM and adjacency table

lookups for Layer 3 forwarding
ECMP hashing
FIB
TCAM
Egress ACL/QOS
classification
Layer 3 FIB
Policing
Egress policing
Ingress policing
MAC
Table
CL
TCAM
Classification
(ACL/QOS/SNF)
Egress lookup
pipeline
Ingress lookup
pipeline
Ingress ACL/QOS/SNF
classification
Ingress MAC table

lookups
Port-channel hash result
L2 Lookup (pre-L3)
Ingress Parser
Receive packet from Port Logic block

Send payload to Ingress Buffer
Send header to Decision Engine
Decision Engine
PKT
HDR
From Ingress
Port Logic
F3 SOC
BRKARC-3470
55
M3 Forwarding Engine Hardware
24 x 10G, 6 x 40G, or 2 x 100G SOC with

decision engine
RACL/VACL/PACL
Ingress/egress sampled Netflow
MPLS/VPLS/EoMPLS
Large-scale external TCAM for FIB/ACL
External packet buffer memory

learning)
OTV / GRE tunnels
Layer 3 IPv4/ IPv6 unicast and multicast
LISP*
FabricPath forwarding*
VXLAN
Hardware Table
F3 Capacity
MAC Address Table
128K (384K*)
FIB TCAM
1M IPv4 / 512K IPv6 (2M/1M*)
BRKARC-3470
56
M3 Decision Engine
To Ingress
Buffer
Final Results
L2 Lookup (post-L3)
Large FIB TCAM

and CL TCAM
external to SOC
FIB
TCAM
Layer 3 FIB
CL
TCAM
Classification
(ACL/QOS/SNF)
MAC
Table
Policing
Egress lookup
pipeline
Ingress lookup
pipeline
L2 Lookup (pre-L3)
Ingress Parser
PKT
HDR
From Ingress
Port Logic
BRKARC-3470
Decision Engine
M3 SOC
57
Layer 2 Hardware Forwarding
Layer 2 forwarding driven by MAC table lookups
Source and destination MAC lookups performed for each frame, based on
{VLAN,MAC} pairs
Source MAC lookup drives new learns and refreshes aging timers
Destination MAC lookup returns destination port
BRKARC-3470
58
MAC Table Lookup
F3 module used as example
From Ingress Parser

Extract VLAN and
MAC address
Ingress frame
header
Decision Engine
HDR
(VLAN,MAC) pair fed

into two separate hash
functions
MAC Table
V100,MAC D
(2*16*2048 = 64K entries)
MAC Table
Hash #2
Bank 1
Bank 2
16 pages
1
2
3
4
5
Hash result #1
selects row on each
page of Bank 1
V10,MAC B
V10,MAC B
V10,MAC B
V10,MAC B
V10,MAC B
6
V10,MAC B
7
V10,MAC B
8
V10,MAC B
9
V10,MACV10,MAC
B
A
10
V10,MAC B
11
V10,MAC
B
V68,MAC
B
12
V10,MAC
B
13
V10,MAC B
14
V10,MAC B
15
V10,MAC B
16
V100,MAC D
17
V10,MAC B
18
V10,MAC B
19
V10,MAC B
20
V10,MAC B
21
V10,MAC B
22
V10,MAC B
23
V10,MAC B
24
V10,MAC B
25
V10,MAC B
26
V10,MAC B
27
V10,MAC B
28
V10,MAC B
29
V10,MAC B
V30,MAC
30
V10,MAC
B C
31
V10,MAC B
32
V100,MAC D
V100,MAC
MATCH!
V22,MAC E
Hash result #2
selects row on each
page of Bank 2
V621,MAC F
BRKARC-3470
Compare (VLAN,MAC)
pair to selected row on
each page of each bank
Matching entry returns

destination index
2048 rows
MAC Table
Hash #1
L2 Lookup
To Ingress
Lookup Pipeline
59
MAC Table Details
MAC table is a hash table
Effective capacity of any hash table dependent on quality of input to hash
Hash collisions can occur and 100% utilization may not be possible
Hash collisions more likely when many MACs are similar, e.g.:
Performance testers (all ports sending incrementing MAC addresses)
Virtualized environments (new VMs deployed en masse with incrementing MACs)
Homogenous server environments (all hosts using the same NICs from the same vendor
and all purchased at roughly the same time)
IP multicast deployments (many L2 group MACs which start with 01005E and likely to
have sequential destination IP group addresses)
BRKARC-3470
60
Layer 3 Hardware Forwarding
Layer 3 forwarding driven by FIB table lookups
Forwarding tables built by control plane on supervisor engine
OSPF, EIGRP, IS-IS, BGP, statics, etc.
Tables downloaded to forwarding engine hardware for data-plane forwarding
FIB TCAM lookup based on longest-match destination IP prefix lookup
FIB match returns rewrite (next-hop) information in adjacency table
BRKARC-3470
61
IP FIB TCAM Lookup

Generate lookup key based on
destination IP and compare to
FIB TCAM entries
From Ingress
HDR
Lookup Pipeline
10.1.1.10
Ingress unicast IP
packet header
Mask out dont

care bits while
comparing key
HIT!
Hit in FIB
returns result in
FIB DRAM
Flow data from packet

header fed into loadsharing hash function
Adj Index, # next-hops
10.1.1.3
10.1.1.4
10.10.0.10
10.10.0.100
10.10.0.33
10.1.1.xx
10.1.2.xx
10.1.3.xx
10.10.100.xx
10.1.1.xx
10.100.1.xx
10.10.0.xx
10.100.1.xx
FIB DRAM
Forwarding
Engine
Modulo of hash result

and # next-hops selects
exact ADJ entry
SIP + DIP +
SPort + DPort
10.1.1.2
FIB TCAM
IP FIB Lookup
Next-hop 1 (IF, MAC)
Load-Sharing
Hash
Hash
Result
Offset
Return lookup
result
mod
# nexthops

Adj Index
Adjacency index
identifies ADJ
block to use
BRKARC-3470
Result
To Ingress
Lookup
Pipeline
Adjacency Table
62
Classification Lookups
Matching packets
Used to decide whether to apply a particular policy to a packet
Layer 2, Layer 3, and/or Layer 4 information

Enforce security, QOS, or other policies
Some examples:
Match TCP/UDP source/destination port numbers to enforce security policy

Match source IP addresses to apply policy-based routing (PBR)
Match 5-tuple to apply QOS marking policy
Match protocol-type to apply Control Plane Policing (CoPP)
etc.
BRKARC-3470
63
CL TCAM Lookup ACL

Generate lookup key
based on packet fields
and compare to CL
TCAM entries
From Ingress/Egress
Lookup Pipeline
Packet header
HDR
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx | 10.1.68.44 | xx | xxx | xxx
HIT!
Decision Engine
Mask out dont

care bits while
comparing key
Classification
Permit
xxxxxxx
xxxxxxx
| 10.1.2.100
| 10.2.2.2 ||xx
xx||xxx
xxx||xx
xxx
ACEs from
security ACL
(x = dont care)
Security ACL
Hit in CL TCAM
returns result in
CL SRAM
ip access-list example
permit ip any host 10.1.2.100
deny
ip any host 10.1.68.44
deny
deny
deny
permit tcp any any eq 22
deny
tcp any any eq 23
deny
udp any any eq 514
permit tcp any any eq 80
permit udp any any eq 161
Deny
Deny
xxxxxxx | 10.24.77.7| xx | xxx | xxx
Deny
Deny
xxxxxxx
xxxxxxx || xxxxxxx
xxxxxxx || tcp
tcp || xxx
xxx || 80
22
Permit
xxxxxxx | xxxxxxx | tcp | xxx | 23
Deny
xxxxxxx | xxxxxxx | udp | xxx | 514
Deny
xxxxxxx | xxxxxxx | tcp | xxx | 80
Permit
xxxxxxx | xxxxxxx | udp | xxx | 161
Permit
CL TCAM
CL SRAM
Actions from
security ACL
Return lookup result,

affecting final packet
handling (forward or drop)
To Ingress/
Egress Lookup
Pipeline
Fields to match:
src IP | dst IP | protocol | src port | dst port
BRKARC-3470
64
CL TCAM Lookup QOS

Generate lookup key
based on packet fields
and compare to CL
TCAM entries
From Ingress/Egress
Lookup Pipeline
Packet header
HDR
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
Mask out dont

care bits while
comparing key
xxxxxxx
xxxxxxx||10.3.3.xx
10.2.2.xx||xx
xx||xxx
xxx||xxx
xx
xxxxxxx | 10.4.12.xx | xx | xxx | xxx
ACEs from QOS
policy ACLs
(x = dont care)
HIT!
QOS Policy ACLs
Decision Engine
Classification
Policer ID 1
Hit in CL TCAM
returns result in
CL SRAM
Policer ID 1
Policer ID 1
Policer ID 1
10.0.1.xx
10.1.1.xx||xxxxxxx
xxxxxxx||udp
tcp | xxx | xxx
Remark DSCP 32
10.1.1.xx | xxxxxxx | udp | xxx | xxx
Remark DSCP 32
10.0.1.xx | xxxxxxx | tcp | xxx | xxx
Remark DSCP 40
10.1.1.xx | xxxxxxx | tcp | xxx | xxx
Remark DSCP 40
xxxxxxx | 10.2.3.xx | tcp | xxx | 23
Remark IP Prec 3
xxxxxxx | 10.5.5.xx | tcp | xxx | 23
Remark IP Prec 3
CL TCAM
CL SRAM
ip access-list police
permit ip any 10.3.3.0/24
ip access-list remark-dscp-32
permit udp 10.0.1.0/24 any
permit udp 10.1.1.0/24 any
ip access-list remark-dscp-40
permit tcp 10.0.1.0/24 any
permit tcp 10.1.1.0/24 any
ip access-list remark-prec-3
permit tcp any 10.2.3.0/24 eq 23
permit tcp any 10.5.5.0/24 eq 23
Actions from
QOS policy
To Ingress/
Egress Lookup
Pipeline
Return lookup result,
affecting final packet
handling (police or remark)
Fields to match:
src IP | dst IP | protocol | src port | dst port
BRKARC-3470
65
Full and Sampled Netflow

Netflow collects flow data for export to collector(s)
Full Netflow: Accounts for every packet of every flow on interface, up to capacity
of hardware Netflow table
Available on M2 modules only
Sampled Netflow: Accounts for M in N packets on interface using random packetbased sampling
M2: Accounts sampled flows, up to capacity of hardware Netflow table
F3/M3: Accounts hardware sampled flows in software using FSA
BRKARC-3470
66
M2 Netflow Table
Netflow Table actually consists of three hardware components in M2 forwarding
engine:
Netflow Hash Table: Contains Netflow Entry Keys and corresponding indexes
to Netflow Entry Table (speeds lookups and minimizes hash collisions)
Netflow Entry Table: Contains actual Netflow flow data
Netflow Statistics Table: Contains statistics for corresponding flow entries
BRKARC-3470
67
M2 Netflow Lookup
Packet header
From Ingress/Egress
Lookup Pipeline
Generate Flow Key

from packet flow data
HDR
Forwarding Engine
Compare Flow Key
to indexed entry in
Netflow Entry Table
Flow Key fed into two

separate hash functions
to generate Lookup Keys
and Entry Keys
Netflow Lookup
SIP / DIP / L4 / etc.

Netflow Table
Hash #1
Lookup
Key 1
1
2
Stats
Stats
Entry Key 2
Entry Key Index
Entry Key Index
Stats
MATCH!
Stats
Entry Key Index
Stats
Stats
Stats
Netflow Entry Table

(512K)
Netflow
Statistics Table
Entry Key Index

Entry Key
Entry Key 1
1
Entry Key Index

Lookup Keys select
row in Netflow Hash
Table banks
2 pages
Entry Key Index
Entry Key
Entry
EntryKey
Key2 Index
Lookup
Entry
Key 2 MATCH!
Bank 1
512K rows
Entry Key 1
Netflow Table
Hash #2
Continue
ingress/egress
pipeline processing
To Ingress/
Egress Lookup
Pipeline
Key Index
Bank 2
Netflow Hash Table
Compare Entry
Keys to selected
row on each page
On match, use Index value to

access Netflow Entry Table
(On miss, create new entry)
BRKARC-3470
On match, update statistics

in corresponding Netflow
Statistics Table entry
68
Sup CPU transmits NDE

packets either via
mgmt0 or via Inband to
collector(s)
NDE on M2 Modules
LC CPU builds NDE
packets (IP+UDP+NDE)
and sends them to Sup
via EOBC
via mgmt0
Supervisor Engine
Fabric Modules
Sup
CPU
via Supervisor Inband
Fabric
EOBC
LC CPU
periodically ages
out Netflow table
entries
LC
CPU
Fabric
Forwarding
Engine
VOQs
Data-plane traffic
traverses
forwarding engines
on each module
NF
Table
Hardware Flow
Creation
L3 Engine
Flow entries
created/updated
in Netflow table
(full or sampled)
L2 Engine
M2 Module
Replication
Engine
Port ASIC
LC
CPU
Fabric
LC
CPU
Fabric
Forwarding
Engine
VOQs
Forwarding
Engine
VOQs
NF
Table
NF
Table
Hardware Flow
Creation
L3 Engine
L2 Engine
M2 Module
Replication
Engine
Port ASIC
BRKARC-3470
Hardware Flow
Creation
L3 Engine
L2 Engine
M2 Module
Replication
Engine
Port ASIC
69
F3/M3 Sampled Netflow
Hardware-based sampling with software-based Netflow cache in FSA
Classification lookup selects Netflow sampler-table entry
Sampler table defines which sampler to use (defines M:N)
Copy of randomly sampled packets sent to FSA via module inband
Sampled copies sliced to reduce bandwidth consumption
Sampled copies rate-limited to 50K packets per second to avoid over-running

FSA
BRKARC-3470
70
Sup CPU transmits NDE

packets either via
mgmt0 or via Inband to
collector(s)
NDE on F3/M3 Modules
via mgmt0
Supervisor Engine
Fabric Modules
LC CPU builds NDE

packets (IP+UDP+NDE)
and sends them to Sup
via EOBC
Sup
CPU
via Supervisor Inband
Fabric
EOBC
LC CPU builds
software Netflow
cache based on
samples and
periodically ages
out entries
DRAM
NF
Table
Fabric
Decision Engine
Rate Limiting
Samples subjected
to HW rate limiter
Sampler Table
Sampler marks M:N
random packets to
sample
NF
Table
Ingress
Buffer
Rate Limiting
Sampler Table
Classification
(ACL/QOS/SNF)
SOC
M3 Module
Fabric
NF
Table
FSA
CPU
Module
Inband
Decision Engine
Hardware
Forwarding
DRAM
FSA
CPU
Data-plane traffic
traverses decision
engine on each SOC
Module
Inband
Packets marked for

sampling copied to
LC inband
Classification block
in Decision Engine
selects sampler
DRAM
FSA
CPU
Fabric
Module
Inband
Ingress
Buffer
Decision Engine
Rate Limiting
Sampler Table
Hardware
Forwarding
Classification
(ACL/QOS/SNF)
Ingress
Buffer
Hardware
Forwarding
Classification
(ACL/QOS/SNF)
SOC
SOC
M3 Module
F3 Module
BRKARC-3470
71
Agenda
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
M2 System Architecture
10G M2 module used as example
Supervisor Engine
Central Arbiter
Fabric Module 1
Fabric Module 2
Fabric Module 3
Fabric Module 4
Fabric Module 5
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Forwarding Engine 1
Virtual
Queuing
e1/1
SP
DWRR
e2/24
Egress Buffer
Ingress Buffer
VOQ 1
VOQ 4
RE 1
RE 4
L2 Engine
RE 1 RE 2
DWRR
Port
ASIC 1
RE 3
RE 4
RE 3
Port ASIC 1
Port ASIC 2
Module 1
Local
Ports
RE 2
Local
Ports
MAC
Table
VOQ 3
L3 Engine
VOQ 2
e2/24
VOQ 1
q1
q2
q3
q4
VOQ 4
NF
Table
VOQ 3
CL
TCAM
VOQ 2
FIB
TCAM
FE 2
SP
Port
ASIC 2
DWRR
Module 2
e1/1
BRKARC-3470
e2/24
73
M2 Packet Flow
Buffer credit
granted
Supervisor Engine
Credit
Ingress/
egress L3
lookups,
ACL/QOS,
Netflow
Ingress/egress
L2 lookups
and portchannel hash
result
Forwarding Engine 1
Fabric Module 5
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Buffer on egress
based on destination
port + priority
Packet queued in
ingress buffer based on
source port + priority
Ingress Buffer
Packet headers
sent to FE
REHDR
1
PKT
Schedule and
transmit toward
destination port
RE 3
CRC, storm control, VLAN
translation, Linksec
decryption, etc.
Local
Ports
Final lookup result:

destination port + priority
Receive packet
from wire
DWRR
Port
ASIC 1
DWRR
e2/24
Egress Buffer
VOQ 4
RE 1 RE 2
RE 2
SP
RE 4
L2 Engine
Ingress port
QOS based on
COS / DSCP
VOQ 3
e1/1
VOQ 2
Virtual
Queuing
VOQ 1
q1
Dst+Pri
q2
q3
q4
Credit
Receive
from fabric
FE 2
VOQ 4
Queue packet
descriptor in VOQ
(destination port +
priority)
VOQ 1
Module 1
Fabric Module 4
Req
e2/24
MAC
Table
Fabric Module 3
Fabric ASIC
NF
Table
L3 Engine
Fabric Module 2
VOQ 3
CL
TCAM
Central Arbiter
Fabric Module 1
VOQ 2
FIB
TCAM
Transmit
to fabric
10G M2 module used as example
RE 3
RE 4
Linksec encryption,
VLAN translation,
etc.
Port ASIC 1
Port ASIC 2
Local
Ports
Request buffer
credit for
destination port
+ priority
Return buffer credit

(destination port +
priority)
SP
Egress port
QOS based on
COS
Port
ASIC 2
DWRR
Module 2
PKT HDR
e1/1
BRKARC-3470
Transmit
packet on wire
e2/24
74
F3/M3 System Architecture
Supervisor Engine
Central Arbiter
Fabric Module 1
Fabric ASIC
Fabric ASIC
Fabric Module 2
Fabric Module 3
Fabric ASIC
Fabric ASIC
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric Module 6
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC Fabric ASIC
Decision Engine
e2/9
Virtual
Queuing
q1
q2
q3
q4
e1/25
Layer 2 Lookups
Layer 3 Lookups
Classification for
ACL / QOS / SNF
MAC
Table
FIB
TCAM
SP
CL
TCAM
Ingress
Buffer
DWRR
e2/9
Ingress Parser
Module 2
e1/25
e2/9
BRKARC-3470
75
F3 SOC 6
F3 SOC 5
F3 SOC 4
Port Logic
F3 SOC 3
Module 1
F3 SOC 2
F3 SOC 1
F3 SOC 6
F3 SOC 5
F3 SOC 4
F3 SOC 3
F3 SOC 2
F3 SOC 1
Port Logic
Egress Buffer
F3/M3 Packet Flow
Buffer credit
granted
Supervisor Engine
Credit
Fabric Module 1
Transmit
to fabric
Request buffer
credit for
destination port
+ priority
Fabric ASIC
Fabric ASIC
Return buffer credit

(destination port +
priority)
Fabric Module 2
Central Arbiter
Fabric Module 3
Fabric ASIC
Fabric ASIC
Fabric Module 4
Fabric Module 5
Fabric ASIC
Fabric Module 6
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC Fabric ASIC

Final lookup result:
destination port + priority
Req
e2/9
Queue packet
descriptor in VOQ
(destination port +
priority)
Virtual
Queuing
Ingress/egress L2 and L3
lookups, ACL/QOS
lookups, Netflow sampling
FIB
TCAM
SP
CL
TCAM
Ingress
Buffer
BRKARC-3470
Module 2
e2/9
76
F3 SOC 6
e1/25
Transmit on
wire
F3 SOC 5
PKT HDR
Port Logic
F3 SOC 4
Ingress packet
parsing
VLAN
translation,
etc.
Egress Buffer
F3 SOC 3
Schedule for
transmission
e2/9
F3 SOC 2
Receive packet
from wire
Port Logic
Packet headers
sent to DE
DWRR
Buffer on egress
based on destination
port + priority
F3 SOC 1
CRC, storm
control, VLAN
translation, etc.
Ingress
Parser
PKT HDR
F3 SOC 6
Module 1
e1/25
Layer 2 Lookups
Layer 3 Lookups
Classification for
ACL / QOS / SNF
MAC
Table
F3 SOC 5
F3 SOC 4
F3 SOC 3
F3 SOC 2
F3 SOC 1
Payload queued in
ingress buffer based
on COS / DSCP
q1
q2
Dst+Pri
q3 HDR
PKT
q4
Credit
Receive
from fabric
Decision Engine
Agenda
Fabric Architecture
Hardware Forwarding
Packet Walks
Conclusion
Conclusion
You should now have a thorough understanding of the

Nexus 7000 / Nexus 7700 switching architecture including
chassis, supervisors, I/O modules, and fabrics, as well as
forwarding lookups and complete system-level packet flows
Any questions?
BRKARC-3470
78
Reference: Acronym Decoder
ACLAccess Control List
NDENetflow Data Export
ADJAdjacency
OTVOverlay Transport Virtualization
ASICApplication Specific Integrated Circuit
PACLPort ACL
CFPC Formfactor Pluggable
PBRPolicy-Based Routing
CoPPControl Plane Policing
QOSQuality of Service
COSClass of Service
QSFP+40G Quad Small-Formfactor Pluggable
DEDecision Engine
RACLRouter ACL
DSCPDifferentiated Services Code Point
REReplication Engine
DWRRDeficit Weighted Round Robin
RPFReverse Path Forwarding
ECMPEqual Cost Multi Path
RURack Unit
EOBCEthernet Out-of-Band Channel
SFP+10G Small-Formfactor Pluggable
FCoEFiber Channel over Ethernet
SNFSampled Netflow
FEForwarding Engine
SOCSystem-on-chip/switch-on-chip
FEXFabric Extender (Nexus 2000 family)
SPStrict priority (queue)
FIBForwarding Information Base
TCAMTernary CAM
FSAFabric Services Accelerator
VACLVLAN ACL
GREGeneric Route Encapsulation
VDCVirtual Device Context
LISPLocator/Identifier Separation Protocol
VOQVirtual Output Queuing
MPLSMultiprotocol Label Switching
VQIVirtual Queuing Index
VXLANVirtual Extensible LAN
BRKARC-3470
79
Complete Your Online Session Evaluation
Give us your feedback to be

entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys

through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
BRKARC-3470
80
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Table Topics
Meet the Engineer 1:1 meetings
Related sessions
BRKARC-3470
81
Thank you

Brkarc 3470 Cisconexus7000 7700switcharchitecture2016lasvegas 2hours 160930084524

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Brkarc 3470 Cisconexus7000 7700switcharchitecture2016lasvegas 2hours 160930084524

Hochgeladen von

Copyright:

Verfügbare Formate

Cisco Nexus 7000 / 7700 Switch

To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching

Introduction to Nexus 7000 / Nexus 7700

Supervisor Engine and I/O Module Architecture

Introduction to Nexus 7000 / Nexus 7700 Platform

Nexus 7000 / Nexus 7700 Common Foundation

General purpose DC switching w/10/40/100G

Targeted at Dense 40G/100G deployments

Same release vehicles, versioning, feature-sets

Common fabric ASICs (Fab2) and architecture

Introduction to Nexus 7000 / Nexus 7700

Supervisor Engine and I/O Module Architecture

NX-OS 4.1(2) and later

Nexus 7000 Chassis Family

NX-OS 5.2(1) and later

NX-OS 6.2(2) and later

Nexus 7700 Chassis Family

NX-OS 7.2(0)D1(1) and later

Introduction to Nexus 7000 / Nexus 7700

Supervisor Engine and I/O Module Architecture

Provides all control plane and management functions

Supervisor Engine 2E (Nexus 7000 / Nexus 7700)

One quad-core 2.1GHz CPU with 12GB DRAM

Two quad-core 2.1GHz CPU with 32GB DRAM

Connects to fabric via 1G inband interface

Interfaces with I/O modules via 1G switched EOBC

Onboard central arbiter ASIC

Supervisor Engine 2 / 2E Architecture

USB device port

Reference: Component Functions Supervisor

Switched EOBC Provides switch 1G connections to each module CPU for

I/O Controller Provides all I/O functions for supervisor components

Central Arbiter Dedicated ASIC that controls access to fabric based on

M3 delivers best of Mand F-series capabilities

10G / 40G / 100G

L2/L3/L4 with large forwarding tables and rich feature set

10G / 40G / 100G

High performance, low latency with streamlined feature set

10G / 40G / 100G

Nexus 7000 M2 I/O Modules

10G / 40G / 100G M2 I/O modules

Share common hardware architecture multi-chipset

Two integrated forwarding engines (120Mpps)

Layer 2/Layer 3 forwarding with L3/L4 services

Large forwarding tables (900K FIB/128K ACL)

802.1AE LinkSec on all ports

24 x 10G (plus Nexus 2000 FEX support)

6 x 40G (or up to 24 x 10G via breakout)

Nexus 7000 M2 I/O Module Architecture

Reference: ASIC Functions M2 Modules

M2 Module 40G and 100G Flow Limits

M2 modules use 10G Virtual

Each VQI sustains 10G traffic flow

All packets in given 5-tuple flow

Nexus 7000 F3 I/O Modules

10G / 40G / 100G F3 I/O modules

6 independent SOC ASICs per module

Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS)

Onboard Fabric Services Accelerator (FSA)

Require Supervisor Engine 2 / 2E

12 x 40G (or up to 48 x 10G via breakout)

Nexus 7700 F3 I/O Modules

10G / 40G / 100G F3 I/O modules

6 independent SOC ASICs per 10G module