Nexus 5000 Architecture: BRKDCT-2825

Nexus 5000 Architecture
BRKDCT-2825
BRKDCT-2825
14651_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2
© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
Session Objectives
At the End of the Session, the Participants Should
Be Able to:
Understand the rationale behind I/O consolidation
Understand the Nexus 5000 architecture
Describe the data path inside a Nexus 5000
BRKDCT-2825
Agenda
I/O Consolidation, the Reasons

I/O Consolidation, the Solution
Nexus 5000, System Hardware Overview
Nexus 5000, Internal Architecture
Nexus 5000, Fabric Data Path
Nexus 5000, Forwarding and Policy Enforcement
BRKDCT-2825

Presentation_ID.scr
Before I/O Consolidation
Parallel LAN/SAN
Infrastructure
LAN SAN A SAN B Inefficient use of Network
Infrastructure
5+ connections per server—
higher adapter and
cabling costs
Adds downstream port costs;
cap-ex and op-ex
Each connection adds additional
points of failure in the fabric
Longer lead time for server
provisioning
Multiple fault domains—
complex diagnostics
Management complexity
Ethernet FC
BRKDCT-2825
I/O Consolidation
Reduction of server adapters
Simplification of access layer
LAN SAN A SAN B
and cabling
Gateway free implementation—
fits in installed base of existing
LAN and SAN
Nexus
L2 Multipathing Access—
Nexus 5000
5000
Distribution
Lower total cost of ownership
Fewer cables
Investment protection (LANs
and SANs)
Consistent operational model
Enhanced Ethernet and FCoE Ethernet FC

BRKDCT-2825

Presentation_ID.scr
Recipe for a Consolidated Access Layer
Converged Network
Unified Fabric
Adapter (CNA)
A 10 Gigabit Ethernet Switch An Enhanced 10GE Adapter
with Native Fibre Channel over Capable of Fibre Channel over
Ethernet Support Ethernet Encapsulation
An Enhanced 10GE Adapter

and a Software Layer for FCoE
Encapsulation
BRKDCT-2825
Evolution of Ethernet Physical Media

Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s
10Mb 100Mb 1Gb 10Gb
UTP Cat 3 UTP Cat 5 UTP Cat 5 X2

SFP Fiber SFP+ Cu (BER better than 10 -18 )
SFP+ Fiber
Cat 6/7
Power Transceiver
Technology Cable Distance (each side) Latency
SFP+ CU
Twinax 10m ~0.1W ~0.25μs
Copper
SFP+ USR MM OM2 10m

1W ~0.1μs
ultra short reach MM OM3 100m
SFP+ SR MM OM1 33m

1W ~0.1μs
short reach MM OM3 300m
Cat6 55m ~8W 2.5μs

10GBASE-T Cat6a/7 100m ~8W 2.5μs
Cat6a/7 30m ~4W 1.5μs
BRKDCT-2825

Presentation_ID.scr
SFP+ Ethernet Interconnect
Smallest 10GE form factor
Hot swappable
Optical SFP+ interoperates
with other 10GE modules
XFP
XENPAC SFP+ Optical Module
X2
Nexus 5000 support following
SFP+ Copper “direct connect”
1m, 3m, 5m (10m future)
TwinAx cable (thin)
Cables are pre-terminated
(lower cost)
Optical Fiber
SR optics
SFP+ Copper
LR (future)
BRKDCT-2825
“Enhanced” Ethernet?
Feature/
Standard
Benefit
Enable multiple traffic types to share a
Lossless Ethernet
common Ethernet link without interfering
IEEE 802.1Qbb
with each other
Class of Service
Enable consistent management of Quality
Based Bandwidth
of Service at the network level by providing
Management
consistent scheduling
IEEE 802.1Qaz
Data Center Bridging Management protocol for Enhanced

Exchange, DCBX Ethernet capabilities
BRKDCT-2825

Presentation_ID.scr
Class-Based Fabric Services
Priority Based
Bandwidth Management Priority Based Flow Control
Transmit Queues Receive Buffers

Offered Load Realized Load Ethernet Link
3G/s HPC Traffic 2G/s Zero Zero

3G/s 3G/s 2G/s 20%
3G/s One One
30% 3G/s Storage Traffic 3G/s Two Two

3G/s 3G/s 3G/s 3G/s Three STOP PAUSE Three
50%
Four Four
3G/s 4G/s 6G/s 3G/s LAN Traffic 5G/s Five Five
4G/s
Six Six
t1 t2 t3 t1 t2 t3 Seven Seven
Enables intelligent sharing of Enables lossless behavior

bandwidth between traffic classes for each class of service
control of bandwidth PAUSE sent per priority
802.1Qaz Enhanced Transmission when buffers limit exceeded
BRKDCT-2825
Nexus 5000: System

Hardware Overview
BRKDCT-2825

Presentation_ID.scr
Nexus 5000, Product Portfolio
Industry’s First I/O Consolidation Virtualization Fabric for
Enterprise Data Center
Nexus 5020
Switch
56-Port L2 Switch
40 Ports 10GE/FCoE, fixed
2 Expansion Modules
Expansion
Modules
Fibre Channel FC + Ethernet Ethernet
8 Ports 1/2/4G FC 4 Ports 10GbE/FCoE 6 Ports 10GE/FCoE
4 Ports 1/2/4G FC
BRKDCT-2825
Front Panel
NX5020
Dual Redundant
N+1 Redundant Fans
Power Supplies
Replaceable Components on the Front for Easy Access

BRKDCT-2825

Presentation_ID.scr
Rear Panels
All 10GE Ports Are FCoE Capable!

Nx5020
10/100/1000 Expansion Power

Console Base 10GE
Out of Band Management Modules Entry
Cables Connect in the Rear for Ease of Server Wiring

BRKDCT-2825
Internal Architecture
BRKDCT-2825

Presentation_ID.scr
Hardware Architecture
10 GE Interfaces 10 GE 1/2/4 Gbps Fibre Channel
Interfaces to Storage Network
SFP+ SFP+ SFP+ SFP+ SFP+ SFP SFP

xcvr xcvr xcvr xcvr xcvr xcvr xcvr 10 GE
10 GE
SFP+ SFP+ SFP+ SFP+ SFP+ SFP SFP & FC
xcvr xcvr xcvr xcvr xcvr xcvr xcvr
Memory
Intel 3100 PCI Controller

Unified Unified Unified Unified
Port Port Port Port
Controller Controller Controller Controller
FLASH
Intel
LV Xeon
(1.66 GHz)
Unified NVRAM
Crossbar
Fabric
Serial RS-232 Console
Unified Unified Unified PCIe

Port Port Port
Controller Controller Controller
XAUI XAUI XAUI 1GE

SFP+ SFP+ SFP+ SFP+
SFP+
Dual NIC Dual NIC
xcvr xcvr xcvr xcvr
xcvr
SFP+ SFP+ SFP+ SFP+
xcvr xcvr xcvr xcvr SFP+
xcvr
XFI XFI XFI
10/100/1000 Management
10 GE Interfaces
BRKDCT-2825
Unified Crossbar Fabric

58 port crossbar and scheduler
Three unicast and one multicast
crosspoints
Central tightly coupled scheduler
Request, propose, accept, grant,
acknowledge semantics
Total SRAM 24.6 Mbits
Packet enhanced iSLIP scheduler Gates 12.4 Million
Distinct unicast and multicast Transistors ~200 Million
schedulers Metal Layers
Signal Pins
7
1286
Eight classes of service SerDes 232 @ 3.75Gbps
Egress buffer credits

DWRR class of service
DWRR ingress interface
BRKDCT-2825

Presentation_ID.scr
Unified Port Controller
Media access controllers
1/10G Ethernet and 1/2/4G
Fibre Channel
Packet buffering and queuing
Total of 1.875 MBytes used in four slices
Forwarding controller
Total SRAM 35 Mbits
Ethernet and Fibre Channel Total TCAM 1 Mbit
Layered policy engine Logic Gates 18 Million
Four data path slices

Transistors ~300 Million
Metal Layers 7
One 1/10G Ethernet or two 1/2/4G Fibre Total Pins 900
Channel ports SerDes 32 @ 3.75Gbps
Connects to one UCF port

All switching done in UCF crossbar
BRKDCT-2825
480 KBytes of buffering
Switch ASIC Architecture 10GE LAN Uplink

Fibre Channel
SAN Uplinks
1/10GE Attached Server
SAN B
Transceiver Transceiver
XAUI – 10 Gbps 1/2/4G Fibre Channel XAUI – 10 Gbps

4 @ 3.125G 1 @ 1.0625/2.125/4.25G 4 @ 3.125G
FC FC
1/10G MAC 1/10G MAC
MAC MAC
Forwarding Forwarding Forwarding

Parsing & Parsing & Parsing &
Editing ? Editing Editing ?
Virtual Virtual Virtual

Queues Queues Queues
Packet Egress Packet Egress Packet Egress
Buffer Queues Buffer Queues Buffer Queues
Slice 1 Slice 2 Slice 3 Slice 4 Slice 1 Slice 2 Slice 3 Slice 4
Unified Port Controller Unified Port Controller

4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps
Fabric Buffer Fabric Buffer Fabric Buffer

Unicast and
Multicast
Schedulers
58 source busses in total

BRKDCT-2825

Presentation_ID.scr
Fibre Channel
SAN Uplinks
1. Decode, align, SAN B

synchronize bytes
XAUI – 10 Gbps Decrypt, verify, 1/2/4G Fibre Channel XAUI – 10 Gbps
4 @ 3.125G authenticate frames 1 @ 1.0625/2.125/4.25G 4 @ 3.125G
FC FC
1/10G MAC 1/10G MAC
MAC MAC

Virtual 2. Extract frame fields Virtual Virtual

Queues Add/remove headers and Queues Queues
Packet Egress edit frame contents Packet Egress Packet Egress


Unicast and
Multicast
Schedulers

BRKDCT-2825

Fibre Channel
SAN Uplinks
SAN B

4 @ 3.125G 1 @ 1.0625/2.125/4.25G 4 @ 3.125G
3. Evaluate frame
FC FC
1/10G MAC fields for forwarding, MAC MAC
1/10G MAC
filtering, and editing

4. Store frame content

when waiting
Unified Port Controller

5. Queue frames and Unified Port Controller
manage crossbar
service
4 @ 3.75G requests
– 12Gbps 4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps

Unicast and
Multicast
Schedulers

BRKDCT-2825

Presentation_ID.scr
Fibre Channel
SAN Uplinks
SAN B

4 @ 3.125G 1 @ 1.0625/2.125/4.25G 4 @ 3.125G
FC FC
1/10G MAC 1/10G MAC
MAC MAC 8. Extract frame fields
Add/remove headers and
edit frame contents

7. Landing place for

frames in flight
Unified Port Controller Unified Port Controller 6. Match requests,

available outputs,
4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps and fairness criteria

Unicast and
Multicast
Schedulers

BRKDCT-2825

Fibre Channel
SAN Uplinks
SAN B

4 @ 3.125G 1 @ 1.0625/2.125/4.25G 4 @ 3.125G
10. Encrypt frames
and encode bytes
FC FC
1/10G MAC 1/10G MAC
MAC MAC

Virtual Virtual Virtual 9. Evaluate frame

Queues Queues Queues fields for, filtering,
Packet Egress Packet Egress Packet Egress and editing


Unicast and
Multicast
Schedulers

BRKDCT-2825

Presentation_ID.scr
Switch Fabric
Data Path
BRKDCT-2825
Media Access Controllers

Each Unified Port Controller (UPC) slice has…
One 1 Gigabit Ethernet MAC
Cisco MDS and Catalyst 4000 lineage
One 10 Gigabit Ethernet MAC
Purchased from “More-than-IP”
Validated by University of New Hampshire testing
Two 1/2/4 Gigabit Fibre Channel MACs
Cisco MDS lineage
Two of the slices in each UPC have an 802.1AE LinkSec
encryption engine
Integrated Flow Control handling
Ethernet—802.3X “PAUSE” and Cisco Priority Flow Control
Fibre Channel—BB_credits
BRKDCT-2825

Presentation_ID.scr
Crossbar Overview
Tightly coupled scheduler and crosspoint
20% link speedup (12 Gbps)
Unicast scheduler
Virtual output queuing
3x fabric speed up
3 crosspoints
Multiple frames transferred per scheduling event
“Superframing”
Multicast scheduler
System class queuing
Separate crosspoint
Fanout splitting, grant coalescing, and retry
BRKDCT-2825
Unicast Virtual Output Queuing

Eliminates head-of-line blocking
Frames for idle outputs bypass
Port 2
congested outputs
Packet
Buffer
Effective use of crossbar resources

Scheduler “maximally matches”
desired connectivity Port 1
Packet
Ingress stores frame is packet buffer

Buffer
Switch
Fabric Packet
Buffer
Keeps list of packets to each egress Port 3
Port and system class

448 queues for each ingress port
Packet
Buffer
Port 4
Scheduler notified about desired

connections
Scheduler maximizes throughput
Egress scheduling
Fairness among ingress ports
BRKDCT-2825 Crossbar usage

Presentation_ID.scr
Day in the Life of a Unicast Frame
1. Frame pointer posted to virtual
output queue
Port 2
Packet
Buffer
Frame data in packet buffer
2. VOQ posts request to scheduler
Port 1 2
Scheduler
3. Scheduler arbitrates and Packet
Buffer
Switch
3 Fabric
grants access 1
4
7
Packet
Buffer
Port 3
Allocates crossbar 5
4. Frame sent to fabric buffer
Packet
Buffer
6
Port 4
5. Fabric buffer sends to egress
Notifies dcheduler
6. Egress sends frame on wire
7. Egress indicates freed buffer resources
BRKDCT-2825
Scheduler Overview
Accept
VOQs IF1
Egress Ingress
Scheduler Scheduler
VOQs IF2
Egress Ingress
Scheduler Scheduler
VOQs IF3 Egress Ingress

Scheduler Scheduler
Request Grant
Proposal/mandate
BRKDCT-2825

Presentation_ID.scr
Unicast Scheduler Algorithm
Egress Scheduler
A priority is selected
Fixed priority, or
DWRR
An ingress is selected within that priority
Highest priority “current preferred” ingress is given a “mandate”
iSLIP maximally matches remaining requesters
Ingress Scheduler
Egress Schedulers make a proposal
Ingress Scheduler selects an egress
Fixed Round Robin selection
The selected Egress Scheduler updates its own “current preferred”
In multi-pass scheduling, this step happens only for first-pass
BRKDCT-2825
selections
Multicast, Fabric Replication
Ingress Fabric Egress

Mcast
U-VOQ
U-VOQ
A
BB
BM U-VOQ
U-VOQ
Mcast Mcast
Ucast
U-VOQ
U-VOQ
Mcast A A
B CC
C M-VOQ
M-VOQ AA
Mcast
A
Use Cases
Ethernet multicast
BRKDCT-2825

Presentation_ID.scr
Multicast MAC Lookups
MAC table
32K entries total (unicast, multicast, Fibre Channel)
1K entries (software setting) for multicast
Populating multicast MAC table

IGMP snooping
Static
Multicast MAC lookup miss

Source only multicast (for L3 multicast)
Forward frame to interfaces linked to multicast routers
Learned via PIM snooping
Flooding (for L2 multicast)
BRKDCT-2825
Multicast Class Queuing

Separates class contention and
flow control
Port 2
MC Ingress
Queues
Priority
Queues
Egress
Priority
Ingress stores frame in packet buffer

and keeps list of packets on
each class Port 1
MC Ingress
Priority
Port 3
MC Ingress
Priority
Queues Queues
Scheduler notified about connection Egress

Priority
Egress
Priority
set at head of queue

Queues Queues
Scheduler maximizes throughput

MC Ingress
Queues
Queues
Priority
Priority
Port 4
Egress
Fairness among ingress ports

Crossbar usage
Allows multiple transfers for non-overlapping
connection sets
Eight queues per ingress port

BRKDCT-2825

Presentation_ID.scr
Multicast Scheduling Algorithm
Ingress Scheduler selects a class for service
DWRR priority selection
Ingress 0
Sends request to required Egress Schedulers class queues
Exact egresses selected by required packet fan-out

Ingress 0
Egress Scheduler evaluates all Ingress requests Priority
scheduler
Scheduler
Global Multicast Round Robin pointer sets priority

Request vector Proposal vector
All Egress select same Ingress with same weight ==
Multicast Round Robin moved on each grant
Egress Scheduler checks path availability
Round Robin
Egress 0
Multicast
Egress 1
Output buffer credit in UPC Scheduler
Scheduler
Egress 57
Multicast Fabric Buffer empty Scheduler
Generate proposal to selected Ingress

Request
Scheduler
Proposal
Ingress Scheduler collects proposals Multicast Round Robin
If proposals matches all requested, generate full grant

If fan-out split is enabled
Generate grant for partially matching subset
BRKDCT-2825 Start timer to collect the rest of the required proposals
Forwarding
BRKDCT-2825

Presentation_ID.scr
Forwarding Pipeline
Parsed Packet
Wire rate “fixed” latency
Vlan Translation
Parsed frame fields,

Collect Interface Table (4K)
Virtual Interface
Table (512) Configuration and
State Vlan State Table
(1K) configuration, and control
Fibre Channel
Switch Table (4K)
plane state are evaluated to
Determine
Destination determine destination(s)
(ingress only) Multicast Vector
Station Table Table (4K)
(16K)
Policy engine filters based
Ethernet
Learning
on configuration, bindings,
RBACL Label Table
(2K)
and layered ACLs
ACL Search Engine
(2K) Policy Enforcement
Binding Table
(2K) Layered equal cost multi
Zoning Table
(2K)
path expansion
Fibre Channel
Multipath
PortChannel Table
Fibre Channel
Multipath Table (1K) Expansion (16)
(ingress only)
EtherChannel/PortChannel
Editing Instructions &

BRKDCT-2825
Virtual Output Queue List
Parsing Ethernet IP Packets

Destination address
Source address
Ethertype = .1Q CoS d VLAN
Ethertype = 2
Ver IHL TOS Total len
Identification Flg Frgm offset che
ck
TTL Proto Header cksum che sum Parsed Packet
Source address
ck Vlan
Translation
Virtual
Table (4K)
Destination address Interface Table
(512)
Interface State
Vlan State
Table (1K)
IP options Fibre Channel

Switch Table
Forwarding (4K)
(ingress only) Multicast
Vector Table
Src port Dst port Station Table
(16K) (4K)
Seq number Ethernet

Learning
Ack number RBACL Label
Table
Hdr len Flags Win size ACL TCAM Policy
(2K)
Binding Table
(2K) Enforcement (2K)
Cksum Urgent ptr Zoning Table
(2K)
Fibre Channel Multipath PortChannel

TCP options and data FC Multipath Table Expansion Table
Sc (1K) (ingress only) (16)
hec
FCS k Virtual Output
Queue List
BRKDCT-2825

Presentation_ID.scr
Parsing FCoE Packets Fibre Channel frames are
FCoE encapsulated prior to
forwarding
Destination address
Source address
Ethertype = FCoE Ver
Reserved
SOF Parsed Packet
Vlan
r_ctl d_id Virtual
Translation
Table (4K)
Interface Table Interface State
cs_ctl s_id (512) Vlan State
Table (1K)
type f_ctl Fibre Channel
Switch Table
seq_id df_ctl seq_cnt Forwarding (4K)
(ingress only) Multicast
Vector Table
ox_id rx_id Station Table
(16K) (4K)
Parameters Ethernet
Learning
RBACL Label
Table
(2K)
Policy
Payload ACL TCAM
(2K) Enforcement
Binding Table
(2K)
CR
Cc Zoning Table
(2K)
hec
CRC k Fibre Channel Multipath PortChannel
FC Multipath Table Expansion Table
EOF Reserved
Sc (1K) (ingress only) (16)
hec
FCS k Virtual Output
Queue List
BRKDCT-2825
Acquiring Interface State

Physical Interface Table
Default priority
Expected encapsulations (802.1Q, FCoE)
FCoE encapsulations for Fibre Channel physical ports
Virtual Interface Table
Default VLAN
Interface Security and QoS ACL labels
Binding check configuration
Ethernet learning
Secure Group Tag assignment
VLAN State Table
VLAN membership list
Virtual Interface granularity
VLAN flood vectors
Unknown unicast, multicast, and broadcast
VLAN Security and QoS ACL labels
BRKDCT-2825

Presentation_ID.scr
Ethernet Forwarding
16K Entry dLeft hash table (StationTable)
Searched by {VLAN, destination address}
Selects
Local port
Multicast index
Unknown addresses forwarded by VLAN multicast vectors

Unknown unicast
Unregistered multicast
Broadcast
IP Multicast forwarded by MAC address

IP multicast groups registered by IGMP v1, v2, v3 snooping
Multicast vectors allocated dynamically based on destination membership
Same mechanism forwards Fibre Channel in the local domain and

N_port Virtualizer
BRKDCT-2825
Modern Hardware Hash Searches

Items keyed on hash
hash Associated Data

Many switches use single
search
key
function “bucket hash”
== == == == == == == == Little or no overflow support
“traditional” hash search
one hash function and no overflow
“dLeft” hash search uses two
hash functions to increase
Items keyed on hash A occupancy
hash A Data correlated with one hash is
uncorrelated in other
== == == == Associated Data Random occupancy ~90%
Items keyed on hash B
search
Nexus 5000 adds TCAM for
key hash B
bucket overflow
Priority
1–3% of capacity
== == == ==
TCAM for overflow

Hash is “right sized” CRC
division
“dLeft” hash search
two uncorrelated hash functions and overflow Four polynomials for each search
Two are selected
BRKDCT-2825

Presentation_ID.scr
Ethernet Address Learning
Ingress and Egress learning searches

Line rate on for all frames
Facilitates distributed table population
Ingress notifies supervisor to develop database

Supervisor pushes new addresses to all Unified Port
Controllers
Adds entries if missed
Re-enforces existing entries
Supervisor queries tables to check for consistency

Maintains aging state
CPU removes entries that are obsolete

BRKDCT-2825
Fibre Channel Forwarding
4K Entry dual index search table

Searched by {VSAN, domain_id}
Misses are Fibre Channel exceptions
Selects
Local port or PortChannel
Remote Fibre Channel switch
Locally attached hosts and N_port Virtualizer forwarded

same as Ethernet
BRKDCT-2825

Presentation_ID.scr
Policy Engine
VLAN Membership
check
Frames evaluated by multi- pass fail
stage engine Interface, VLAN, and
Switch Port Analyzer (SPAN) and Diagnostic Sampling

MAC Binding
pass fail
Combination of arrays, hash MAC and L3 Binding

(IP & Fibre Channel)
Control Plane Redirect/Snooping

tables, and Ternary CAMs pass fail
Fibre Channel Zone
Searches occur in parallel

membership check
pass fail
Results evaluated in pipeline Port ACLs
permit deny
Diagnostics and control plane VLAN ACLs (ingress)
“tap” pipeline at any point permit deny
Role Based ACLs

(egress)
permit deny
QoS ACLs (ingress)

policer
permit
drop
Permit
to SPAN to Sup
BRKDCT-2825
session
Access Control List Search Engine

Priority
Priority
Priority
TCAM
2Kx432
Control Plane ACLs (128)
SPAN and Diagnostic ACLs (64)
Port ACLs Vlan ACLs QoS ACLs

Search Key (768) (1024) (64 ingress)
Role Based ACLs
(egress)
2048 Ternary match Access Control Entries (ACEs)
Each entry available to all functions
Labels allow sharing of Access Control Entries
Access control lists have a label
Policy definition points select a label
Interfaces, VLANs, roles
Labels and frame fields form search keys
Flexible region assignment
Tune access control list resource allocation to network policies
Access Control Lists scope
VLAN and control plane are global scope—same on all Unified Port Controllers
Port, QoS, role based, and SPAN are local scope—specific to each Unified Port Controllers
BRKDCT-2825

Presentation_ID.scr
Multipath Expansion
Two stage expansion process
Each can lead to the next
Same mechanism for all expansions
Fibre Channel
switching?
Configuration of expansion unique to each expansion
FC Multipath
Fibre Channel Switching
Expansion
Selects a path to a target Fibre Channel switch
Fibre Channel Shortest Path First (FSPF)
EtherChannel/
PortChannel 1K entries each selects up to sixteen Fibre Channel
Expansion ports, Ethernet ports, or Ethernet/SAN PortChannels
EtherChannel and SAN PortChannel

List of Virtual
Output Queues
Selects a path to a physically adjacent device
Sixteen multipath entries each selects up to sixteen
Ethernet ports or sixteen FC ports
BRKDCT-2825
Expansion Algorithm
Polynomial select Field select Number of equal paths
Ethernet DA
CRC-8 A
Ethernet SA x8 + x5 + x4 + 1
256
IP DA or FC D_ID possibilities
Galois Field 2 ÷ XOR Modulo
Selected
Path
IP SA or FC S_ID
TCP DP CRC-8 B
x8 + x5 + x3 + x2 + x1 + 1
TCP SP or FC OX_ID
Relevant frame fields

Ethernet Source Address and Destination Address always available
IP frames allows inclusion of IP v4/v6 Source and Destination Address
TCP/UDP frames can include source and destination ports
Fibre Channel frames can include D_ID and S_ID
OX_ID can also be included per VSAN
Each field is divided by one of two CRC-8 polynomials

Result of field CRC division is combined via bitwise XOR
Result selected using modulo division by number of equal cost paths
256 possibilities are reduced to avoid bias
BRKDCT-2825

Presentation_ID.scr
Editing FC Packets Internal Destination address
for VOQ
Internal Source address
r_ctl d_id Ethertype = DTAG FTAG TTL
cs_ctl s_id
type f_ctl Destination address
seq_id df_ctl seq_cnt
ox_id rx_id
Parameters Source address
Payload
Ethertype = FCoE Ver

CRC
Reserved
Parsed Packet
SOF
Vlan
Translation
Virtual
Interface Table Interface State
Table (4K) r_ctl d_id
(512) Vlan State
Table (1K) cs_ctl s_id
Fibre Channel
Switch Table type f_ctl
Forwarding (4K)
(ingress only) Multicast seq_id df_ctl seq_cnt
Station Table Vector Table
(16K) (4K)
ox_id rx_id
Ethernet
Learning
Parameters
RBACL Label
Table
(2K)
ACL TCAM Policy Binding Table
(2K) Enforcement (2K) Payload
Zoning Table
(2K)
Fibre Channel Multipath PortChannel

Multipath Table Expansion Table
(1K) (ingress only) (16) CRC
EOF Reserved
Virtual Output FCS
Queue List
BRKDCT-2825
Session Objectives Summary

In This Session, You:
Understood the rationale behind I/O consolidation
Understood the Nexus 5000 architecture
Saw the data path inside a Nexus 5000
BRKDCT-2825

Presentation_ID.scr
Key Takeaways
The Key Takeaways of This Presentation Are:
Nexus 5000 hardware overview
Nexus 5000 internal architecture
Nexus 5000 data path
BRKDCT-2825
Q and A
BRKDCT-2825

Presentation_ID.scr
Recommended Reading
Continue your Cisco Live

learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books
Available Onsite at the Cisco Company Store

BRKDCT-2825
Complete Your Online

Session Evaluation
Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.
BRKDCT-2825

Presentation_ID.scr
BRKDCT-2825

Presentation_ID.scr

Nexus 5000 Architecture: BRKDCT-2825

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Nexus 5000 Architecture: BRKDCT-2825

Hochgeladen von

Copyright:

Verfügbare Formate

Nexus 5000 Architecture

© 2006, Cisco Systems, Inc. All rights reserved. 1

 I/O Consolidation, the Reasons

© 2006, Cisco Systems, Inc. All rights reserved. 2

Enhanced Ethernet and FCoE Ethernet FC

© 2006, Cisco Systems, Inc. All rights reserved. 3

An Enhanced 10GE Adapter

Evolution of Ethernet Physical Media

10Mb 100Mb 1Gb 10Gb

UTP Cat 3 UTP Cat 5 UTP Cat 5 X2

SFP+ USR MM OM2 10m

SFP+ SR MM OM1 33m

Cat6 55m ~8W 2.5μs

© 2006, Cisco Systems, Inc. All rights reserved. 4

Data Center Bridging Management protocol for Enhanced

© 2006, Cisco Systems, Inc. All rights reserved. 5

Transmit Queues Receive Buffers

3G/s HPC Traffic 2G/s Zero Zero

30% 3G/s Storage Traffic 3G/s Two Two

 Enables intelligent sharing of  Enables lossless behavior

Nexus 5000: System

© 2006, Cisco Systems, Inc. All rights reserved. 6

Replaceable Components on the Front for Easy Access

© 2006, Cisco Systems, Inc. All rights reserved. 7

All 10GE Ports Are FCoE Capable!

10/100/1000 Expansion Power

Cables Connect in the Rear for Ease of Server Wiring

© 2006, Cisco Systems, Inc. All rights reserved. 8

SFP+ SFP+ SFP+ SFP+ SFP+ SFP SFP

Intel 3100 PCI Controller

Unified Unified Unified PCIe

XAUI XAUI XAUI 1GE

Unified Crossbar Fabric

 Distinct unicast and multicast Transistors ~200 Million

schedulers Metal Layers

 Eight classes of service SerDes 232 @ 3.75Gbps

Egress buffer credits

© 2006, Cisco Systems, Inc. All rights reserved. 9

Layered policy engine Logic Gates 18 Million

 Four data path slices

One 1/10G Ethernet or two 1/2/4G Fibre Total Pins 900

Channel ports SerDes 32 @ 3.75Gbps

Connects to one UCF port

Switch ASIC Architecture 10GE LAN Uplink

XAUI – 10 Gbps 1/2/4G Fibre Channel XAUI – 10 Gbps

Forwarding Forwarding Forwarding

Virtual Virtual Virtual

Slice 1 Slice 2 Slice 3 Slice 4 Slice 1 Slice 2 Slice 3 Slice 4

Unified Port Controller Unified Port Controller

Fabric Buffer Fabric Buffer Fabric Buffer

58 source busses in total

Unified Crossbar Fabric

© 2006, Cisco Systems, Inc. All rights reserved. 10

1. Decode, align, SAN B

Forwarding Forwarding Forwarding

Virtual 2. Extract frame fields Virtual Virtual

Slice 1 Slice 2 Slice 3 Slice 4 Slice 1 Slice 2 Slice 3 Slice 4

Unified Port Controller Unified Port Controller

Fabric Buffer Fabric Buffer Fabric Buffer

58 source busses in total

Unified Crossbar Fabric

Switch ASIC Architecture 10GE LAN Uplink

I/O Consolidation, the Reasons

Enables intelligent sharing of Enables lossless behavior

Distinct unicast and multicast Transistors ~200 Million

Eight classes of service SerDes 232 @ 3.75Gbps

Four data path slices

Effective use of crossbar resources

Ingress stores frame is packet buffer

Scheduler notified about desired

Populating multicast MAC table

Multicast MAC lookup miss

Ingress stores frame in packet buffer

Scheduler notified about connection Egress

Scheduler maximizes throughput

Eight queues per ingress port

Generate proposal to selected Ingress

Parsed frame fields,

Unknown addresses forwarded by VLAN multicast vectors

IP Multicast forwarded by MAC address

Same mechanism forwards Fibre Channel in the local domain and