Brkarc 3454

In-depth and personal with the Cisco Nexus
2000 Fabric Extender Architectures, Features,

and Topologies
Pavan S Kaushik
Technical Marketing Engineer
BRKARC-3454
Never heard of FEX before ??
No Problem !!
FEX Gurus in the house ??

No Problem !!
BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Goals
Brief Nexus 5000/6000/7000/9000 Parent platform introduction.

Discuss Nexus 2000 platform architectures, supported parent switches,
actual designs and best practices.
Nexus 2000 topologies, features and implementation.
vPC/FCoE/FC/FP/QoS/ACI Deep dive and troubleshooting will not be
covered in this session.
House Rules
We have Nexus 2300 switches in the Room !! Do take a look.
Keep the session interactive.
Highly detail-oriented questions will be taken offline(ex. Optics/Model

compatibility etc).
Agenda
Introduction
Parent switches overview
Platform overview
Features deep-dive and packet-walk
Design and topologies
Configuration and Provisioning
Connectivity options and Optics
Summary
Introduction
Nexus 2000 / FEX
Cisco Nexus 2000 Fabric EXtenders
Modular Switch Parent Switch*

In-chassis Linecards With Remote Linecards (FEX)

* - Parent switch can be Fixed or Modular
FEX: Virtualized Access Switching
Extending the boundaries of Data Center Fabric
Flexible and simplified top-of-rack (ToR) designs

Simplified Management, Centralized configuration
Multiple Parent Switch support N5K, N6K, N7K, N9K
Unified Ports provide flexible LAN and SAN deployment
...
Virtualized Switch
Nexus 2000 Solution
Low TCO
Low OPEX
Low CAPEX
Simplified
Lower Devices to Manage
Operation
Central Point of Management
Nexus 2000 Architecture with Single Point
of Management
Traditional Architecture Architecture
Agnostic(ACI, FA) Flexibility
Enhanced Buffers
Low over subscription
Enhanced
High Port Density Performance
Parent switches overview :
Nexus 5000/6000/7000/9000
Nexus 2000 - Parents Switches Family
Fixed Switching Platforms Modular Switching Platforms
Nexus 5600
Nexus 2200/ Nexus 6000 Nexus 7000 Nexus 7700
Nexus 2300
Nexus 5000/5500
Application Centric Infrastructure (ACI)

Nexus 9000 Standalone
Nexus 9000
NEXUS 5600 SERIES
Nexus 5672UP Nexus 56128P Nexus 5624Q Nexus 5648Q Nexus 5696Q
7.0(1)N1(1) 7.0(2)N1(1) 7.1(0)N1(1a) 7.1(1)N1(1) 7.0(4)N1(1)
Switch Fabric
1.44 Tbps 2.56 Tbps 1.92 Tbps 3.84 Tbps 7.68 Tbps
Throughput
PERFORMANCE
Port-to-Port Latency ~1.0 us
Layer 3 Capability Integrated Line-Rate
Switch Footprint 1RU 2RU 1RU 2RU 4RU

10 Gigabit Ethernet
72 128 96 192 384
Port
DENSITY
40 Gigabit Ethernet
6 8 24 48 96
Port
Unified Ports 16 48 N/A N/A 48
1/10 Gigabit Ethernet

1152 1152 1152 1152 2304
With FEX
SCALE
Packet Buffer 25MB per 3x40G (or 12x10G)
FEX Support Yes* Yes* Yes* Yes* Yes*
* Verify model-specific compatibility through DataSheets / ReleaseNotes / Hardware Installation Guides
Nexus 7000 Product Family
Shipping up to 384x40G ports and 192x100G ports
Nexus 7000 Family
F3 Modules
768 x 10-Gbps Ports or
768 x 10-Gbps Ports* or 192 x 100-Gbps Ports
192 x 40-Gbps Ports
384 x 10-Gbps Ports* or 384 x 10-Gbps Ports or

96 x 40-Gbps Ports 192 x 40-Gbps Ports or
96 x 100-Gbps Ports
336 x 10-Gbps Ports *
84 x 40-Gbps Ports
96 x 10-Gbps Ports * 48 x 100-Gbps Ports
24 x 40-Gbps Ports 48x 10-Gbps Ports or
24x 40-Gbps Ports or
Nexus 7010 Nexus 7018

Nexus 7000 / 7700 Line Card Modules
M1 1G/10G M2 10G / 40G / 100G M3 1G / 10G / 40G
F2E 10G F3 40G

F1 10G F2 10G
F2E 10G
F3 10G / 40G / 100G
F3 closes the F/M feature gap!
Nexus 7000/7700 Supervisor FEX support
Nexus 7000 SUP2 Nexus 7000 SUP2E Nexus 7700 SUP2E
CPU Quad-Core Xeon Dual Quad-Core Xeon Dual Quad-Core Xeon
Speed (GHz) 2.13 2.13 2.13
Memory (GB) 12 32 32
Max VDCs 4 (user) +1 (admin) 8 (user) +1 (admin) 8 (user) +1 (admin)
Max VLANs per FEX 2000 2000 2000
Max VLANs per FEX HIF 75 75 75
Max. no. of FEX/Ports 32 FEX /1536 Ports 64 FEX /3072 Ports 64 FEX /3072 Ports
NEXUS 7000 Line Card FEX support
M2-24XP M2-06FQ F2-48XP F2E-48XP F3-12FQ F2E-48XP F3-48XP F3-24FQ
Chassis
Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7700 Nexus 7700 Nexus 7700
Supported
Ports and
24x10GE 6x40GE 48x1/10GE 48x1/10GE 12x40GE 48X1/10GE 48x1/10GE 24x40GE
Speed
Interfaces SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+
Max Fabric
3 3 5 5 3 3 3 5
Cards and BW
240 240 480 480 480 480 480 960
(GBps)
Performance
120 720 720 720 720 720 1440
(Mpps) 120
vPC (Support) Yes Yes Yes Yes Yes Yes Yes Yes
FCoE, FP
No Yes Yes Yes Yes Yes Yes Yes
Support
Max. Power (W) 795 795 450 450 500 450 450 650
FEX Support Yes* Yes* Yes* Yes* Yes* Yes* Yes* Yes*
* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides
Cisco Nexus 9500 Platform Switches
Density in DC Optimized Footprint
Cisco Nexus 9500
16-Slot
Fabric Bandwidth:
3.2 Tbps/ slot
8-Slot
4-Slot
Cloud Deployment Options:
21 RU
13 RU
7 RU
ACI Multi-Site, Telemetry,
Cloud Scale
Nexus 9516 Nexus 9508 Nexus 9504 Multi-Generation

Payload Slots 16 8 4 Investment Protection:
Capacity 57.6 Tbps 28.8 Tbps 14.4 Tbps No Mid-plane, Power Supply Headroom
100G Density 512 256 128 for 100/400G and integrated optics/
50G Density 1024 512 256 encryption
40G Density 576 288 144
10/25G Density 2304/2048 1152/1024 576/512
Nexus 9500 Supervisor FEX support
Supervisor A Supervisor B
CPU 4 Core, 1.8GHz 6 Core, 2.2GHz
Memory (GB) 16 GB 24GB
Storage (SSD) 64 GB 256GB
Max. Power (W) 80 90
FEX Support 32 FEX /1536 Ports 32 FEX / 1536 Ports
VLANs per FEX 2000 2000
VLANs per FEX HIF 75 75
NEXUS 9500 Line Card FEX support
X9636PQ X9536PQ X9564PX X9432PQ X9464PX
Chassis Supported Nexus 9504/9508 Nexus 9500 Nexus 9500 Nexus 9500 Nexus 9500
48x1/10GB, 48x1/10GB,
Ports and Speed 36x40GB 36x40GB 32x40GB
4x40GB 4x40GB
SFP/SFP+ SFP/SFP+
Interfaces QSFP+ QSFP+ QSFP+
QSFP QSFP+
Performance
120 720 720 720 720
(Mpps)
vPC (Support) Yes Yes Yes Yes Yes
Buffer 36 MB 104 MB 104 MB 24 MB 12 MB
Max. Power (W) 795 450 450 310 500
FEX Support Yes* Yes* Yes* Yes* Yes*

Nexus 9300 Fixed Series
ACI Leaf and NX-OS Switches
Analytics
Quad-core CPU
64 GB SSD
Integrated Uplink | CY14-15 24 GB Memory
9332 QSFP+ (32x40G)

32p QSFP Switch
Modular Uplink | 2HCY13 28p 40G + 4p 100G Q4CY16
24p 40G + 6p 100G
9396 SFP+(48x10+12x40G) 9372(E) SFP+ (48x10+6x40G)
93180YC-EX (48x25G+6x100G)
9372 (E) Copper (48x10+6x40G)

9396 Copper (48x10+12x40G) + NEW
93108TC-EX (48x10GT+6x100G) Q2CY16
93120 Copper (96x10+6x40G)

93128 Copper (96x10+8x40G)
2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Nexus 9300 Switches FEX support
Nexus 9332PQ Nexus 9372PX Nexus 9396PX
Form Factor (RU) 1 1 2
Throughput(Tbps) 2.56 1.44 1.44
1/10GB Ports X 48 48
40 GB Ports 32 6 12
Latency (us) <2 <2 <2
Buffer 25 MB 25 MB 50 MB
Max. Power (W) 650 650 650
Applications ACI Leaf ACI Leaf ACI Leaf
FEX Support Yes Yes Yes

FEX Scalability
Nexus Nexus Nexus Nexus Nexus Nexus Nexus
5500 5600 5696 7000 7700 9300 9500
L3 16 24 32 64 64 16 32
L2 24 24 48 64 64 16 32
Cisco Parent Switch
Cisco Nexus 2000 FEX
16 - 64
FEX
+
Supports 3000+ ports per single point of management

2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Platform Overview:
Nexus 2000 / FEX
Nexus 2000 / FEX Naming Convention
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
N2K-C2232PP-10GE
13: Cisco Nexus 2000 Series Identifier
4: Dash
5: Chassis
6: Series Number
7: Series Generation
89: Number of Ports
10: Host Physical Interface Type (P = Requires SFP+, T = Copper, UP = Unified Port, F = Includes FET)
11: Uplink Physical Interface Type (P = Requires SFP+, Q = QSFP, T = Copper, F = Includes FET, M = Module)
12: Dash
1316: Host Connection Speed, Enhanced model or Bundle identifier (-BUN)
History of Cisco Nexus 2000 Series
Simplified Operations with Revolutionary Scale
Base-T FEX SFP/SFP+ FEX
N2K-C2348TQ / TQ-E
(40G Uplink, Enhanced Buffers)
N2K-C2348UPQ-1/10GE
(40GE Uplink, Enhanced Buffers)
N2K-C2332TQ-1/10/40GE
(40G Uplink, Enhanced Buffers)
N2K-C2248PQ 10GE
(QSFP Uplink)
B22 IBM
Blade FEX
N2K-C2232TM-E-10GE 1/10G FEX for
IBM Blade servers
N2K-C2232PP-10GE
B22 FTS FEX B22 Dell FEX
N2K-C2248TP-E-1GE 1/10G FEX for 1/10G FEX for
(Enhanced Buffers) B22 HP FEX Fujitsu Blade servers Dell Blade servers
1/10G FEX for
HP Blade servers
Nexus 2000 100M/1GBASE-T
Nexus 2224TP Nexus 2248TP Nexus 2248TP-E
Network Interfaces 2 (SFP+) 4 (SFP+) 4 (SFP+)
Host Interfaces 24 48 48
Performance 88Gbps 176Gbps 176Gbps
FCoE Support No No No
Buffer 8 MB 8 MB 32 MB
Max. Power (W) 80 W 95 W 95 W
Oversubscription 1.2:1 1.2:1 1.2:1
Nexus Parent Switches Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000
Nexus 2000 1/10GBASE-T
Nexus 2232TM Nexus 2232TM-E Nexus 2332TQ Nexus 2348TQ/E*
Form Factor (RU) 1 1 1 1
Network
8 (SFP+) 8 (SFP+) 4 (QSFP+) 6 (QSFP+)
Interfaces
Host Interfaces 32 32 32 48
Performance 560Gbps 560Gbps 1.2Tbps 1.2Tbps
FCoE Support No No No Yes (FCoE)
Buffer 8 MB 8 MB 32 MB 32 MB
Max. Power (W) 350 W 240 W 350 W 350 W
Oversubscription 4:1 4:1 2:1 2:1
Nexus Parent Nexus 5000, 6000, 7000, Nexus 5000, 6000, 7000, Nexus 5000, 6000, Nexus 5000, 6000, 7000,
Switches 9000 9000 7000, 9000 9000
* Some N2348TQ-E items are on the roadmap
Nexus 2000 10G (SFP+)
Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ
Network Interfaces 8 (SFP+) 4 (QSFP+) 6 (QSFP+)
Host Interfaces 32 48 48
Performance 560Gbps 960Gbps 1.2Tbps
FCoE Support Yes Yes Yes
Buffer 8 MB 32 MB N2H, 8 MB H2N 32 MB
Max. Power (W) 210 W 175 W 350 W
Oversubscription 4:1 3:1 2:1
Nexus 5000, 6000, 7000, 9000

Nexus Parent Switches Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000
UCS FI
Nexus 2348UPQ Overview
PORT SIDE OF FEX
48 Fixed 1/10G SFP+ Unified Capable Ports. 6x40G QSFP+ Ports

48x1/10G SFP+ Ports
Traditional Ethernet or FCoE
48 Unified Ports provide up to 16GFC
32M of Buffer with Lower Oversubscription ratio of 2:1
Parent Switch N5k, N6k, N7k, N9k.
Choice of Port Side

Intake/Exhaust Airflow
FAN SIDE OF FEX
Redundant 1+1 Redundant N+1 Fan

Modules
AC/DC Power Supplies Unit
Nexus 2348UPQ Buffer Allocation
Total 32MB shared by 6 NIF(Network
Interface) and 48 HIF(Host Interface)
Dedicated buffer pool and shared buffer NIF 1-6x40GE
pool Each of the NIF or HIF can manage

all 32MB buffer dedicated
32MB
Queue-limits can be used to control shared
buffer-usage by bursty flows.
HIF 1-48
Nexus 2000 Internal Architecture
Nexus 2248PQ vs. Nexus 2348UPQ
4x 40Gbe QSFP+ 6x 40Gbe QSFP+

(4x10G-Mode only) (40G or 4x10G-Mode)
QSFP QSFP QSFP QSFP QSFP QSFP QSFP QSFP QSFP QSFP
Uplink/Buffer Uplink/Buffer
3rd Generation ASIC
ASIC A ASIC B
Port 1 - 48
Port 1 - 24 Port 25 - 48
(Unified Ports: Ethernet/FCoE/FC)
(Ethernet/FCoE) (Ethernet/FCoE)
* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle
Cisco Nexus 2348UPQ
Compared with other Nexus 2000 1/10GE Fabric Extender
Model Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ
48x SFP/SFP+ (1/10G)

Host Ports 32x SFP/SFP+ (1/10G) 48 x SFP/SFP+ (1/10G)
(true 40G or 4x 10G-Mode)
4x 40Gbe QSFP+ 6x 40Gbe QSFP+*

Uplink Ports 8x 10GbE SFP+
(4x 10G-Mode only) (true 40G or 4x 10G-Mode)
FCoE
157.5KB per port
Buffer 157.5KB per port 32MB Shared Buffer
2x 16M Shared Buffer (Uplink)
* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle
Fabric Extender 100M/1G/10G Copper
Nexus 2348TQ-E
48 port 3rd Generation Fabric Extender
Nexus 23XX FEX Uplink support

Cost optimized version of N2K-C2348TQ.
Max. 6 x 40GE Uplinks
Identical to N2K-C2348TQ in features.
or
48 x 100M/1G/10G host/downlink interfaces
Max. 8 x 10GE Uplinks
Large 32MB Buffer
(throught breakout cables)
Supports FCoE for LAN/SAN convergence.
48 100/1000 RJ45 Downlinks

Fabric Extender 100M/1GE
Nexus 2248TP-E-GE Overview
48 ports 100M/1G-BaseT
Design scenario:
Enhanced Fabric Extender
High density 100M/1G access
48x 100/1000M host/downlink Cost effective 100M/1G solution
interfaces.
Buffer optimized for specific Data Center
4x 10GE on network/uplink interfaces workloads such as Big Data, Hadoop and
Upstream N5K supports various FEX Distributed Storage
(mix-and-match)
Key differences from N2248TP:
6 user-defined QoS queues (8 QoS Optimized Buffers
queues) Enhanced Debug counters
48 100/1000 RJ45 Downlinks

Nexus 2248TP-E Buffer Allocation
Total 32MB shared by 4 NIF(Network
Interface) and 48 HIF(Host Interface)
Dedicated buffer pool and shared buffer
pool. Size of each pool is configurable by
software NIF 1-4x10GE
Current 128KB/port dedicated at NIF. dedicated

Rest (around 26MB) is shared 32MB
shared
Each of the NIF or HIF can manage all
32MB buffer
HIF 1-48
Bigdata and Hadoop
NIF 1-4x10GE
dedicated
32MB Buffer
shared
N2248TP-E HIF 1-48

Cloudera
certified !!
Cisco Unified Fabric
N2248-TP-E
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white_paper_c11-690561.html
Nexus 2232PP-10GE Fabric Extender
8 SFP+ Uplinks
32 1G/10GE SFP+ Downlinks
32 ports 1/10G Fabric Extender Design Scenario
32x 1/10GE SFP+ host interfaces 1/10G Server Connectivity (SFP+ ToR)
8x SFP network interfaces
Connectivity Flexibility
Predictable low latency
Nexus 2232TM-E
What is same as 2232TM?
Form Factor, ports, power supplies, etc.
Cabling options and distance supported
Port to port latency
What is different?
A newer and better 10GBase-T PHY (40nm)
Better BER characteristics expected qualification underway
Reduction in power consumption
8 x 10GE SFP+ Uplinks
32 1G/10GT Downlinks
Nexus 2232TM-E 10GBASE-T
IEEE 802.3.an standard
Structured Copper Cabling with RJ45 connectors
Supports 10Gbps throughput for up to 100 meters or 330 feet
Leverages Category 6, Category 6A, and Category 7 type cabling
Technology Cable Distance PHY Power Transceiver
(each side) Latency
10GBASE-T Cat6a/7 100m ~6W ~3ms
65nm Cat6a/7 30m ~4W ~3ms
10GBASE-T Cat6a/7 100m ~5W ~2.5ms

40nm Cat6a/7 30m ~3W ~2.5ms
Nexus 2248PQ-10GE Fabric Extender
4 QSFP+ Uplinks
48 1G/10GE SFP+ Downlinks
48 ports 10/1G Fabric Extender Design Scenario
4x QSFP (16x10GE SFP+) on network interfaces High Density 10GE SFP+ ToR
Feature parity with Nexus 2232PP Connectivity Flexibility
Additional uplink buffers (2x16MB) Predictable low latency
Nexus 2000 Overview
Redundant 1+1 AC/DC Power Supplies
Redundant N+1 Fan Modules
Color coding for both power supplies and Fan for port side exhaust
and port side intake airflow
Port Side Exhaust
2348UPQ/2348TQ PPT
Port Side Intake
Nexus 2000 Airflow
Port Side Intake

(Cold Aisle-Network Rack Airflow Alignment)
Port Side Exhaust

(Hot Aisle-Server Rack Airflow Alignment)
Features deep-dive
and Packet Walk
Nexus 5000/6000 and 2000
Packet Forwarding Overview
Nexus 2000 Nexus 5000/6000 Nexus 2000
1 Ingress Unified Egress

UPC Crossbar UPC
FEX ASIC
Fabric
FEX ASIC
6
2 3 4 5
3. Nexus 5000 UPC 4. If required
performs ingress 6. VNTag
egress queuing stripped and
1. Frame received on forwarding and and flow control
N2K HIF port queuing frame forwarded
out on N2K HIF
port
2. Nexus 2000
appends VNTag and 5. Nexus 5000 UPC
forwards frame to appends destination
fabric uplink VNTag and forwards
frame on fabric link
Nexus 2000 Fabric Extender (FEX) VNTag
Nexus 2000 Fabric Extender operates as a remote line card and Logical Interface
does not support local switching (LIF) on the
ingress UPC is
All forwarding is performed on the Nexus 5500/5600/6000 UPC used to forward
VNTag is a Network Interface Virtualization (NIV) technology that the packet
extends (Logical Interface = LIF) to the Nexus 2000 VIF referred LIF
Packet is forwarded
to as a Host Interface (HIF) over fabric link using
a specific VNTag
VNTag is added/removed to the packet between Fabric
Extender and Nexus 5500/5600/6000/7000/9000
VNTAG is a 6Bytes Virtual NIC Tag used for forwarding DA[6]
decisions HIF SA[6]
VNTAG[6]
802.1Q[4]
VNTAG Ethertype d p destination virtual interface
l source virtual interface Frame

Payload
N2K ASIC maps
specific VNTag to
CRC[4]
HIF interface
Nexus 2000 Port Channels
Nexus 2348/2248 Port Channels
Nexus 2000 series FEX support local port channels 1. Packet is
received and
A local port channel on the N2K is still seen as a single lookup forwards
extended port out a LIF (N2K)
interface
2. Packet is
Extended ports are each mapped to a specific VNTag forwarded over
fabric link using a
HW hashing occurs on the N2K ASIC specific VNTag for
the destination 3. N2K ASIC
Number of local port channels on each N2K is based on N2K LIF (port hashes locally and
the local ASIC channel) transmits packet
on one HIF
interface
22xx Support up to 24 local port channels of up to 8
interfaces each as well as vPC (total of 2 x 8 = 16 ports)
23xx Support up to 24 local port channels of up to 8

interfaces each as well as vPC (total of 2 x 8 = 16 ports)
VM VMK
Pinning a HIF to a Port Channel
All server ports are pinned To the

Port Channel
Port Channel
Automatic load balancing
in case of a link failure
Definite traffic flow path

visibility and dedicated
bandwidth

FCoE and Unified Ports
Nexus 2000 flexible definition of port function
Fibre Channel over Ethernet Ethernet

(FCoE) allows encapsulation and Fibre
transport of Fibre Channel traffic Channel
Traffic
over a shared Ethernet network
SAN A / B isolation is maintained
across the network Nexus 5500 Ethernet and
Fibre Channel Switches
SAN SAN
Unified Ports may be configured to A B
support either native Fibre Channel F
C
or Ethernet Fibre Channel

Disk Array
FCoE Links
Servers with Converged

Network Adapters (CNAs)
Nexus 2000 SPAN
Receiver Host FEX Host ports CAN be SPAN source
ports
(Except with F2 Line card in Nexus 7000
parent switch)
FEX Fabric ports CAN be SPAN source

ports with all parent switches
FEX Host ports CANNOT be SPAN

destination port with any parent switch
Sender Host Receiver Host
Nexus 2000 QoS
Tuning the Port Buffers
Each Fabric Extender (FEX) has local port buffers (FEX leverages
a shared memory model) 10G Source (NFS)
10G NFS
You can control the queue limit for a specified Fabric Extender for
egress direction (from the network to the host) Gen 2 UPC
You can use a lower queue limit value on the Fabric Extender to
prevent one blocked receiver from affecting traffic that is sent to Unified Crossbar Fabric
other non-congested receivers ("head-of-line blocking)
Gen 2 UPC
# Disabling the per port tail drop threshold

N5672UP-1(config)# system qos
N5672UP-1(config-sys-qos)# no fex queue-limit
40G Fabric
N5672UP-1(config-sys-qos)#
# Tuning of the queue limit per FEX HIF port Nexus 2000
N5672UP-1(config)# fex 100 FEX ASIC
N5672UP-1(config-fex)# hardware N2248T queue-limit 356000
N5672UP-1(config-fex)# hardware N2248T queue-limit rx/tx ?

<CR> 1G Sink
<2560-652800> Queue limit in bytes
Nexus 2348UPQ
32MB Shared Buffer 10G Attached Source (NAS Array)
NAS
Speed mismatch between 10G NAS and 1G server requires iSCSI
QoS tuning
32MB shared buffer to handle larger traffic bursts
Hadoop, AVID are examples of bursty applications
NFS
10G
You can control the queue limit for a specified Fabric
Extender for egress direction (from the network to the host)
VM VM VM
#2 #3 #4
1G Attached Server
N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 rx
N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 tx
Tune 2348UPQ to support a extremely large
N5672UP-1(config)#interface e110/1/1 burst (Hadoop, AVID, )
N5672UP-1(config-if)# hardware N2348UPQ queue-limit 4096000
Nexus 2000 Offload
CDP
Efficient utilization of Nexus parent switch CPU
TCAM entries are installed on corresponding FEX
ASICs
LLDP
QoS offload is disabled by default
LACP
DCBX
QoS
N5672UP-1(config)# fex 100
N5672UP-1(config-fex)# hardware n2348uPQ qos-policy-offload
Nexus 2000 QoS Offload Supported
N2224TP Fabric Extender 24x1G 2x10G SFP+ Module
N2232PP Fabric Extender 32x10G SFP+ 8x10G SFP+ Module
N2232TM Fabric Extender 32x10GBase-T 8x10G SFP+ Module
N2248T Fabric Extender 48x1G 4x10G SFP+ Module
N2248TP E Fabric Extender 48x1G 4x10G SFP+ Module
N2248PQ Fabric Extender 48x10G SFP+ 16x10G SFP+ Module
N2232TM-E Fabric Extender 32x10GBase-T 8x10G SFP+ Module
N23xx Fabric Extender 48x10G SFP+ 6x40G QSFP Module
Nexus 2000 STP Design Considerations
N2K HIF ports have BPDU Guard enabled by default (it is not
possible to disable currently)
If a BPDU is received port will transition to err-disable
state
Global BPDU Filter compliments BPDU Guard
On link up port will send 11 BPDUs and then stop (in order to
reduce CPU load)
E E E
1. X-Connected
N5672UP-1#show spanning-tree int eth100/1/1 detail patch cable
Port 1025 (Ethernet100/1/1, vPC) of VLAN0190 is designated forwarding 2. BPDU Sent on

Port path cost 1, Port priority 128, Port Identifier 128.1025 Link-Up
Snip->>> 3. BPDU Guard
The port type is edge by port type edge trunk configuration err-disables
Link type is point-to-point by default edge port and
Bpdu guard is enabled
Bpdu filter is enabled by default prevents loop
4. BPDU are not
BPDU: sent 11, received 0
sent once link is
up and active
Nexus 2000 : Routing Protocol Neighborships
Routing Protocol neighborship across FEX requires Priority
queueing for Control Plane traffic at FEX HIF, among others.
Currently this feature is only Supported on Nexus 7k starting
6.2(2), except for N2248PQ.
Note:
Nexus 5000/6000/9000 Series switches do NOT support this. Routing-Protocol
It is not recommended to connect high performance / critical Neighborship
servers to a FEX, rather use the Parent Port. HIF
On the N7k-FEX, this option may be used for non-critical low

bandwidth devices that need routing neighborships.
Router
FCoE Deployment
FEX as Unified TOR Ethernet
FC The fabric extenders (FEX) can
FCoE be used at the ToR
Converged
ToR FEX connect to the end or
LAN row switches which manages
the FEX
FEX work like external line cards

Nexus 5K FC/FCoE SAN to the end or row switches
The EoR switches splits the

Ethernet and FC traffic
Nexus 2K
B22 Dell FEX

B22F FEX 1/10G FEX for
1/10G FEX for Dell Blade servers
B22 HP FEX FTS Blade servers
1/10G FEX for B22 IBM FEX
HP Blade servers 1/10G FEX for
IBM Blade servers
Nexus 7K - FEX FCoE Features Update
NXOS 7.3(0)N1(1) introduces

support for:
FCoE over FEX with F3 (N2K-

C2348UPQ-10GE, B22HP, N2K-
C2232PP-10GE)
F3 FCoE support for FEX with
physical port vPC and vPC+
FC Ports on Nexus 2348UPQ UP FEX
48x1/10G SFP+ Ports 48 Fixed 1/10G SFP+ Unified Capable Ports supporting Ethernet or
6x40G QSFP+ Ports
or 16G FC FCoE or FC
4x10G SFP+ Ports Flexibility
Unified Ports provide up to 24x 16GFC ports or 48x 8GFC ports
32M of Buffer with Lower Oversubscription ratio of 2:1
Parent Switch for Ethernet/FCoE: Nexus 5K, 6K, 7K and 9K
Parent Switch for FC: Nexus 5600
The FC ports on Nexus 2348UPQ are supported only on the Cisco Nexus 5600 switches.
Shipped with Ethernet/FCoE support in July 2014 Only F port is supported on FC-FEX
with 7.0(3)N1(1) NP Port type is not supported so NPV device cannot
Enabled FC support in March 2016 with 7.3(0)N1(1) connect to the FC FEX directly
Fibre Channel FEX supported number (ID) range is F-port trunking and F-port channel are not supported
from 100 to 130 on FC-FEX ports
Support native Fibre Channel speed 2/4/8/16G The default BB credits on each FC interface are 32;
Fibre Channel ports must be configured in groups of 4 and can be set to max 128
ports and starts from beginning Scale per parent switch: 24 FEX, MAX 96 FC ports
First FEX Solution Designed for All Storage Connectivity 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Configure 16G FC Ports on UP FEX
To support 16G FC, the fabric mode on the parent Nexus 5600 should be
in 40G mode.
Only odd-numbered HIF ports can be configured as the 16G FC.
When the odd-numbered HIF port is configured as 16G FC, the adjacent
even-numbered ports must be changed to the out-of-service state. Those
adjacent even-numbered ports next to it cannot be used.
When the speed is set to Auto, the 16G host port comes up at 8G speed.
For the ports to come up at 16G speed, use the switchport speed 16000
command to explicitly set the speed to 16G.
stg-5672-16G-1(config-vsan-db)# interface fc121/1/11
stg-5672-16G-1(config-if)# switchport speed 16000
fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service
stg-5672-16G-1(config-if)#
Extending ACI Policy Based Forwarding into Existing Data
Center Networks
Extend Policy Based Forwarding
Extend Visibility, Fault and Audit
Automated Device Management for extended Fabric nodes
ACI Enabled
Remote N9K
N5K N2K N6K

vSwitch AVS
vSwitch AVS HyperV OVS
Extended ACI Fabric
Design and Topologies
Single Homed Nexus, Single Homed FEX Attached Host
Nexus 7000/5000 Port Channel
Simple and Easy to Configure
End to end
Nexus 7000
Traffic is automatically re-distributed
across remaining NIFs upon failure
Nexus 5000
Simple L2/L3 Flow visibility HIF/NIF to
next hop nexus switch
Nexus 2000
Single link to host: no redundancy at
Server or FEX level.
UCS / Hosts
Dual Homed Nexus, Single Homed FEX Attached Host
Nexus 7000/5000 Virtual Port Channel
Easy to Configure FEX
Nexus 7000 Network Resiliency and High availability

between N7k & N5k with vPC / double-sided
vPC.
Nexus 5000
Simple L2/L3 visibility from HIF-NIF to N5k
Nexus 2000 Single link to host: no redundancy at Server

or FEX level.
UCS / Hosts
Dual Homed Nexus/FEX,
Single Homed FEX Attached Host
Nexus 7000/5000/2000 Virtual Port Channel
High Availability and network

resiliency between N7k & N5k& FEX
Nexus 7000
No Blocking ports in STP when using

vPC.
Nexus 5000
L2/L3 Flows can be based on primary
or secondary vPC switch.
Nexus 2000
UCS / Hosts
Dual Homed Nexus/FEX, Dual Homed FEX AA-Host
Nexus 7000/5600/2000 Virtual Port Channel

Network Resiliency and High
availability across network and
compute
Nexus 7000
Increased Network Bandwidth
Nexus 5600 Active NIC teaming can also be

used for further options.
Nexus 2000
UCS / Hosts
Active
Dual Homed Nexus/FEX, Dual Homed FEX AS-Host
Nexus 7000/5000/2000 Virtual Port Channel Network Resiliency and High
Availability across network and
compute
Nexus 7000
Increased Network Bandwidth
Nexus 5000 Active/Standby NIC teaming can

be dependent on CNA capabilities
Nexus 2000
UCS / Hosts
Active
Standby BRKARC-3454 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Nexus 2000 (FEX) Straight-Through Deployment with VPC
Port-channel connectivity from the server

Two Nexus switches bundled into a vPC S1 S2
pair Fabric Links
Suited for servers with Dual NIC and
capable of running Port-Channel
HIF Fex 100 HIF Fex 101
VPC
Nexus 2000 (FEX)Active-Active Deployment with VPC
S1 S2
Fabric Extender connected to two Nexus
5X00 / 6000 or Nexus 7000/7700
Suited for servers with Single NIC or
Dual NIC not having port-channel
capability. Fabric Links
Scale implications of less FEX per
system and less VPC
Note : Fex 100

Fex 101
This design is currently not supported on Nexus 9X00 HIF HIF
Nexus 2000 (FEX) Active-Active Scale &
Limitations (N7X00)
N7X00 can support up to 64 FEXs
N7X00 supports max. 15 Active-Active FEX in 7.2(0)D1(1)
N7X00 supports max. 32 Active-Active FEX in 7.2(0)D1(1)
Straight-Through FEX and Active-Active FEX cannot exist on the
same ASIC instance
Layer 3 HIF ports are not supported with Active-Active FEX
Active-Active FEX is not supported with vPC+
Nexus 2000 (FEX) - Enhanced VPC
Port-channel connectivity to dual-homed

S1 S2
FEXs
From the server perspective a single access
switch with port-channel support each line
card supported by redundant supervisors
Ideal design for a combination of single Fabric Links
NIC and Dual NIC servers with port-
channel capability
Scale implications of less FEX per system
Fex 100 Fex 101
and less VPC
HIF HIF
Note :
This design is currently not supported on N7000 / N7700 and N9X00
Physical Port vPC
vPC domain vPC domain
FEX101 FEX102 FEX101 FEX102

e101/1/1 VPC1 VPC1 e102/1/1 e101/1/1 VPC1 VPC1 e102/1/1
Po1 Po1
interface e101/1/1
Port-channel vPC switchport Physical port vPC
vpc 1
lacp mode active
vPC configuration on a physical Layer 2 port as opposed to a port-channel

Front panel ports and FEX ports connected to F2/F2e/F3 only
Improves scaling as separate PC interface not created for single-link VPC leg
Key benefit: more than 1000 host facing VPCs with FEX
Nexus 2000 (FEX) Active-Active (Unsupported)
Reference : Supported and Unsupported FEX Topologies: N5000/6000/7000/9000

http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations

VXLAN VXLAN VXLAN VXLAN
vtep vtep vtep vtep vtep vtep
Hypervisor Hypervisor Hypervisor Hypervisor
*Switch Local Port-Channel (non VPC) is supported

- TOR to Hypervisor
- TOR to FEX
- FEX to Hypervisor
- TOR to FEX to Hypervisor

vtep vtep vtep vtep vtep vtep vtep

vtep vtep vtep vtep vtep vtep
vtep vtep vtep vtep

*Switch Local Port-Channel (non VPC) is supported

- TOR to Hypervisor
- TOR to FEX
- FEX to Hypervisor
- TOR to FEX to Hypervisor

vtep vtep vtep vtep

Unupported Combinations

vtep vtep vtep vtep vtep vtep vtep
vtep vtep vtep vtep

*FEX connectivity with combination of Hypervisor

(Host originated VXLAN) and Bare-Metal Servers
is not supported
Configuration and Provisioning
Nexus 2000
Configuration
N5672UP-1(config)# feature fex
N5672UP-1(config-if)# int port-channel100
N5672UP-1(config-if)# switchport mode fex-fabric
N5672UP-1(config-if)# fex associate 100
N5672UP-1(config-if)# vpc 100
N5672UP-1(config-if)# no shut
Configuration
N5672UP-1(config)#show fex 100
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]
Extender Serial: FOC1818R2M9
Extender Model: N2K-C2348UPQ, Part No: 73-15489-03
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth2/5
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po100 - Interface Up. State: Active
Eth2/5 - Interface Up. State: Active
Eth2/6 - Interface Up. State: Active
Verification
Pre-Provisioning Configuration
N5672UP-1(config)# slot 101
N5672UP-1(config-slot)# provision model N2K-C2348UPQ
Provision
N5672-1(config-slot)# show run int eth101/1/1
!Command: show running-config interface Ethernet103/1/1

!Time: Mon Mar 23 10:58:01 2015
version 7.1(0)N1(1a)
interface Ethernet101/1/1
N5672-1(config-if)# int eth101/1/1

N5672-1(config-if)# switchport access vlan 190
N5672-1(config-if)# no shut
Configuration & Verification
Nexus 2248TP-E Enhanced Counters
N5672UP-1(config-if)# sh queuing interface eth100/1/1
Ethernet110/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+-----------
19200 24320 48640 Ingress queue limit(Configurable)
Queueing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+---- Egress queues:
2 0 0 1 2 3 4 5 6 WRR 100 1600
CoS to queue mapping
Queue limit: 66560 bytes Bandwidth allocation
MTU
Queue Statistics:
queue rx tx Egress queue limit(Configurable)
------+---------------+---------------
2 24434 3066978
Port Statistics: Drop counters

rx drop rx mcast drop rx error tx drop mux ovflow
---------------+---------------+---------------+---------------+--------------
0 0 0 0 InActive
Nexus 2348UPQ Enhanced Counters
N5672UP-1(config-if)# show queuing interface ethernet 100/1/1
if_slot 32, ifidx 0x1f630000
Ethernet100/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+----------- Ingress queue limit
5120 15360 71680
Queueing: Egress queues:

queue qos-group cos priority bandwidth mtu CoS to queue mapping
--------+------------+--------------+---------+---------+----
2 0 0 1 2 3 4 5 6 WRR 100 1600 Bandwidth allocation
MTU
Queue limit: 2088960 bytes
Queue Statistics: Egress queue limit

queue rx tx
------+---------------+---------------
2 126 49
Drop counters
Port Statistics:
rx drop rx mcast drop rx error tx drop mux ovflow
---------------+---------------+---------------+---------------+--------------
0 0 0 0 InActive
Priority-flow-control enabled: no
Ethernet Flow Control (802.3x) Configuration
N5672UP-1(config-if)# show interface ethernet 100/1/1 flowcontrol
--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
---------------------------------------------------------------------
Eth100/1/1 on off off off 0 0
N5672UP-1(config-if)# interface ethernet 100/1/1

N5672UP-1(config-if)# flowcontrol send on/off
N5672UP-1(config-if)# flowcontrol receive on/off
Why do I care to honor flow control?
FCoE configuration
N5672UP-1(config)# feature fcoe (If FCoE not enabled)
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Enabled FCoE QoS policies successfully
N5672UP-1(config)# vlan 100

N5672UP-1(config)# fcoe vsan 100
N5672UP-1(config)# vsan database
N5672UP-1(config)# vsan 100
N5672UP-1(config)# fex 100
N5672UP-1(config)# fcoe
N5672UP-1(config)# interface vfc 1011
N5672UP-1(config)# bind interface ethernet 100/1/1
N5672UP-1(config)# switchport trunk allowed vsan 100
N5672UP-1(config)# no shut
Is it not Fairly easy to configure FCoE?
Configuration Verification
N5k-1#show fex 100 detail
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]
FEX Interim version: 7.1(0)N1(1a)
Switch Interim version: 7.1(0)N1(1a)
Extender Serial: FOC1818R2M9
Extender Model: N2K-C2348UPQ, Part No: 73-15489-03
Snip->>>>>>
Logs:
03/18/2015 09:52:21.289192: Module register received
03/23/2015 09:18:18.516214: Module Offline
03/23/2015 09:18:18.561004: Deleting route to FEX
03/23/2015 09:18:18.568532: Module disconnected
03/23/2015 09:21:53.581415: Module register received
03/23/2015 09:21:53.583664: Registration response sent
03/23/2015 09:21:54.634249: create module inserted event.
03/23/2015 09:21:54.635044: Module Online Sequence
03/23/2015 09:22:03.006536: Module Online
Configuring FEX with ACI
* - Verify model specific FEX support with ACI





Connectivity Options
Nexus 2000
Nexus 2300 Connectivity Options
10G SFP+ Connectivity 40G QSFP+ Connectivity
Nexus Parent Switch Nexus Parent Switch
Nexus 2300 Connectivity Options
Network Interface (NIF)
The Network Interface Fabric Ports can operate 10G/40G mode
FET-40G, QSFP-Bidi, QSFP+, Active Twinax, including AoC cables can be used to connect to a parent switch
while operating in 40G mode
Host Interface (HIF)

SFP+ Host Interface will support SR, LR, GLC-T, GLC-SX-MM, AoC
Supported Twinax: 1, 3, 5M (passive) and 7, 10M (active)
*Check the parent switch documentation for the latest supported cable options Twinax Passive cable is
Not supported on NIF
OM3-MM Fiber Cable
For QSFP-40G-SR4 and

QSFP-40G-CSR4
12-Fiber MPO Connector

MPO/MTP connector with 12
pins
Use 4 fiber pairs
4 TX and 4 RX allows for 40 GE
4 unused fibers in the center
QSFP-40G-SR4
40G-SR4 40G-SR4
MPO-12 MPO-12
Support 100M with OM3 MMF

and 150M with OM4 MMF
10G-SR
10G-SR
40G-SR4 10G-SR
10G-SR
MPO-12 LC Breakout
QSFP 40G Bidirectional Transceiver 40G
deployment with 10G cabling
Ciscos innovation has created a new QSFP pluggable 40Gbit/s
Bidirectional (BiDi) Transceiver that works with existing infrastructure:
Multi-mode fiber
Using the existing 10Gbit/s Multimode infrastructure
QSFP-40G-SR-BD has dual LC Connector
40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and
received simultaneously on two wavelengths
QSFP-40G-CSR4
Supports 300M with OM3 MMF
and 400M with OM4 MMF
Compatible with Cisco QSFP-
40G-CSR4 40G-SR4/CSR4
40G-SR4
MPO-12 MPO-12
Compatible with 10G SFP-SR
10G-SR
10G-SR
40G-CSR4 10G-SR
10G-SR
MPO-12 LC Breakout
QSFP-40G-SR-BD Supports 100m with OM3 with
MMF and 150m with OM4
(OM4+) MMF
Multimode
LC Duplex LC duplex connectors
40G-BD 40G-BD
Wavelength: ~850nm and
~900nm
Pull-Tab Color: grey
Multimode 10G-SR
10G-SR
40G-BD LC Duplex 10G-SR
10G-SR
Physical Connection Choices
40G QSFP+ 40G QSFP+
QSFP-LR4 QSFP-LR4
40G QSFP+ 40G QSFP+

QSFP-SR-BD QSFP-SR-BD
40G QSFP+ 40G QSFP+

QSFP-SR4/CSR4 QSFP-SR4/CSR4
1M, 3M, 5M Passive 40G QSFP+

40G QSFP+ 7M, 10M Active
40G QSFP+ 10G SFP+

QSFP-SR4/CSR4
40G QSFP+ QSFP-4x10G 10G SFP+
FET-40G (aka QSFP+ FET)
Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004
Supported on Cisco Nexus 5500/N5600/6000/N7000/N9000 and Nexus 2248PQ-10G/Nexus 2300
Interoperable with FET-10G
Support for 100m distance with OM3
Cisco Nexus 5600/6000 Cisco Nexus 5600/6000
FET-40G
FET-40G
FET-10G
FET-40G Cisco Nexus 2232PP and 2232TM-E 2232TM
Cisco Nexus 2248TP-E 2248TP
Cisco Nexus 2248PQ-10G/N2300
Convert 40GE Interface to Four 10 GE Interfaces
Convert the QSFP Interfaces to Four 10 GE Interfaces When Connecting to a SFP+ Port or to the QSFP Interfaces That
Operate in Four 10 GE Mode, such as the QSFP Uplink of N2248PQ-10G or the Cisco Nexus 5500 QSFP GEM.
Apply global CLI to change interface types N6004(config)# interface breakout slot
1 1 port 1-6 map 10g-4x
to 10 GE
Every three contiguous QSFP interfaces resides
on one Port ASIC.
The port range specified in the CLI has to
include all ports on the ASIC N6004(config)# poweroff module ?
<1-8> Please enter module number
2 Power off the affected modules N6004(config)# poweroff module 1
Every group of 12 QSFP interfaces is managed
as one module, even for the fixed interfaces
3 Power on the affected modules N6004(config)# no poweroff module 1

N6004(config)#
QSA-QSFP+ to SFP Adapter
Convert one QSFP port to one SFP+ port.
Provides 1G/10G transceiver support
Long reach connection between for QSFP+ devices and SFP+ devices
Following 1G/10G transceivers are qualified with QSA in release
10G-LR
1G GLC-T, 1G SX, LX
Requires a mode change on QSFP+ port to 4x10G

Only first 10G interface can be used. Remaining three 10G shown as Link Not Connected
1G SFP or 10G SFP+

QSFP+ Adapter Transceiver
Summary
Nexus 2000
Key Takeaways
Understand FEX Architectures & Capabilities
Realize the benefits offered by a FEX design
Evaluate FEX for YOUR network requirements.
Simplify your Flexible and Scalable ToR design provides Large number of Ports,
Network with low number of management points.
Lower CAPEX Easy to Provision, manage, troubleshoot.

OPEX Lowest Cost option for connectivity
Unified
Network Flexible Ethernet, FCoE and FC solutions with FEX
Acronyms and Abbreviations
HIF Host Interface SoC System on Chip / Switch on Chip
NIF Network Interface FC Fiber Channel
CIF CPU Interface FCoE Fiber Channel over Ethernet
VNTag Virtual NIC Tag UP Unified Port
H2N Host to Network UCF Unified Crossbar Fabric
N2H Network to Host UPC Unified Port Controller
ASIC Application-Specific Integrated Circuit IGMP Internet Group Management Protocol
vPC Virtual Port-Channel ALE Application Leaf Engine
FEX Fabric Extender NFE Network Forwarding Engine
VXLAN Virtual Extensible LAN TCAM Ternary Content Addressable Memory
ACI Application Centric Infrastructure FIB Forwarding Information Base
APIC Application Policy Infrastructure Controller HPC High-Performance Computing
OTV Overlay Transport Virtualization Mroute Multicast Route
STP Spanning-Tree LC Line Card
LPM Longest Prefix Match EOBC Ethernet Out-of-Band Channel
CPU Central Processing Unit
Complete Your Online Session Evaluation
Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Dont forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk

Insights on market trends and forecasts
Preview of key technologies and capabilities
Innovative demonstrations of the latest and greatest products
Better understanding of how Cisco can help you succeed
Register to attend the session live now or

watch the broadcast on cisco.com
Thank you
Appendix
9 Reference Slides
10G MAC Tx
Host to VNTag Rewr
8
802.1q Rewr
Network 7
6 Read Mux Tx Arb
Output Queue Time Stamp
5
37 read 4
Shared Packet Buffer 32MB Input port
Accounting
37 write Flow Cntr
3
VNTag 802.1q VNTag
Forwarding Rewrite Rewrite Svif Check
HIF and HIF
2 ACL Data
10G MAC Rx 10G MAC Tx
1
1 Reference Slides
Network to 10G MAC Rx 10G MAC Tx
Host 2 Unicast FW
Multicast FW
VNTag 802.1q VNTag
NIF Forwarding Rewrite Rewrite
3
37 read 4
Shared Packet Buffer 32MB Input port
Accounting
37 write Flow Cntr
5
Output Queue
6
Read Mux Tx Arb
7
VNTag Rewr 802.1q Rewr
HIF
8 10G MAC Tx
9
Reference Slides
Typical UP FEX Topologies
Straight-Through FEX Active-Active FEX
Nexus Nexus
5600 5600
VP VP
C FPC 40G FCoE C
FPC
40G FCoE 40GE
Nexus Nexus
2348UPQ 2348UPQ
VSAN A 10GE VSAN B VSAN A 10GE VSAN B
8/16G FC 8/16G FC
10GE NIC + FC HBA 10GE NIC + FC HBA

FC-HIF ports configuration must be the same in both the The Active-Active FEX is supported, but only one FC FEX
primary and the secondary VPC switches. can be associated with a parent switch in the pair.
For the Active-Active FEX, the port-conversion CLI should switch1(config)# fex 101
be configured on both the vPC peer switches. switch1(config-fex)# fcoe
10G FCoE CNA

Reference Slides
Configuration of 16G FC Ports on UP FEX

To support 16G FC, the fabric mode on the parent Nexus 5600 should be
in 40G mode.
Only odd-numbered HIF ports can be configured as the 16G FC.
When the odd-numbered HIF port is configured as 16G FC, the adjacent
even-numbered ports must be changed to the out-of-service state. Those
adjacent even-numbered ports next to it cannot be used.
When the speed is set to Auto, the 16G host port comes up at 8G speed.
For the ports to come up at 16G speed, use the switchport speed 16000
command to explicitly set the speed to 16G.
stg-5672-16G-1(config-vsan-db)# interface fc121/1/11
stg-5672-16G-1(config-if)# switchport speed 16000
fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service
stg-5672-16G-1(config-if)#
VFC Creation on 4-Port (HIF) VPC Reference Slides
Supported from Cisco NX-OS release 7.2(0)N1(1)
Only supported on N5600/N6000 platforms (Bigsur ASIC)

N5600-1 N5600-2 Only supported on Nexus 2300 FEX (Tiburon ASIC)
Not supported for NPV mode
Straight-Through FEX N5600-1 N5600-2

vfc1 vfc2 vfc3 vfc4
FEX-1 FEX-2
vfc1 vfc2 vfc3 vfc4 vPC
CNA
vPC
Host
CNA
Host
Switch(config)# int port-channel 500
Switch(config-if)# fcoe multi-vfc enable the feature

Brkarc 3454

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Brkarc 3454

Hochgeladen von

Copyright:

Verfügbare Formate

In-depth and personal with the Cisco Nexus

2000 Fabric Extender Architectures, Features,

FEX Gurus in the house ??

Brief Nexus 5000/6000/7000/9000 Parent platform introduction.

We have Nexus 2300 switches in the Room !! Do take a look.

Keep the session interactive.

Highly detail-oriented questions will be taken offline(ex. Optics/Model

Modular Switch Parent Switch*

Flexible and simplified top-of-rack (ToR) designs

Application Centric Infrastructure (ACI)

Port-to-Port Latency ~1.0 us

Layer 3 Capability Integrated Line-Rate

Switch Footprint 1RU 2RU 1RU 2RU 4RU

1/10 Gigabit Ethernet

Packet Buffer 25MB per 3x40G (or 12x10G)

FEX Support Yes* Yes* Yes* Yes* Yes*

* Verify model-specific compatibility through DataSheets / ReleaseNotes / Hardware Installation Guides

384 x 10-Gbps Ports* or 384 x 10-Gbps Ports or

Nexus 7010 Nexus 7018

M1 1G/10G M2 10G / 40G / 100G M3 1G / 10G / 40G

F2E 10G F3 40G

F3 closes the F/M feature gap!

CPU Quad-Core Xeon Dual Quad-Core Xeon Dual Quad-Core Xeon

Speed (GHz) 2.13 2.13 2.13

Max VDCs 4 (user) +1 (admin) 8 (user) +1 (admin) 8 (user) +1 (admin)

Max VLANs per FEX 2000 2000 2000

Max VLANs per FEX HIF 75 75 75

Interfaces SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+

Nexus 9516 Nexus 9508 Nexus 9504 Multi-Generation

10/25G Density 2304/2048 1152/1024 576/512

CPU 4 Core, 1.8GHz 6 Core, 2.2GHz

Memory (GB) 16 GB 24GB

Storage (SSD) 64 GB 256GB

Max. Power (W) 80 90

FEX Support 32 FEX /1536 Ports 32 FEX / 1536 Ports

VLANs per FEX 2000 2000

VLANs per FEX HIF 75 75

vPC (Support) Yes Yes Yes Yes Yes

Buffer 36 MB 104 MB 104 MB 24 MB 12 MB

Max. Power (W) 795 450 450 310 500

FEX Support Yes* Yes* Yes* Yes* Yes*

9332 QSFP+ (32x40G)

9396 SFP+(48x10+12x40G) 9372(E) SFP+ (48x10+6x40G)

9372 (E) Copper (48x10+6x40G)

93120 Copper (96x10+6x40G)

Form Factor (RU) 1 1 2

Throughput(Tbps) 2.56 1.44 1.44

Latency (us) <2 <2 <2

Max. Power (W) 650 650 650

Applications ACI Leaf ACI Leaf ACI Leaf

FEX Support Yes Yes Yes

Supports 3000+ ports per single point of management

Form Factor (RU) 1 1 1

Network Interfaces 2 (SFP+) 4 (SFP+) 4 (SFP+)

Performance 88Gbps 176Gbps 176Gbps

Max. Power (W) 80 W 95 W 95 W

Oversubscription 1.2:1 1.2:1 1.2:1

Form Factor (RU) 1 1 1 1

Performance 560Gbps 560Gbps 1.2Tbps 1.2Tbps

FCoE Support No No No Yes (FCoE)

Max. Power (W) 350 W 240 W 350 W 350 W

Oversubscription 4:1 4:1 2:1 2:1

Form Factor (RU) 1 1 1