Cisco Live 2016

7/11/2016

Cisco Live 2016

7/11/2016

Cisco Live 2016

7/11/2016

Cisco Live 2016

7/11/2016

Cisco Live 2016

7/11/2016

Storage area networks are designed to transport block level protocols between servers and storage devices such
as disk arrays and tape libraries.
Allows for boot data and mass shared storage independent of compute resources
Provides flexibility and scaling for the changing and growing needs for data storage

Cisco Live 2016

7/11/2016

Block protocols (FC, FCoE, iSCSI) each have their own specific networking requirements that we are going to cover
in detail
File Storage Protocols (NFS, CIFS) are fairly independent of the network configuration outside of possible
dedicated vlans and QoS settings with standard ethernet/ip networking and will not be covered
Viewing the lab exam blue print you can see Nexus 7700, 5600, 2300, UCS B and C Series Servers, and Dual
attached jbod. This tells us we could expect multi-hop fc and fcoe environment with edge devices attached at any
and all levels with a mix of feature and traffic shaping requirements. As such we should know how to configure
FC and FCoE on each platform

Cisco Live 2016

7/11/2016

Cisco Live 2016

7/11/2016

SCSI channels are individually half duplex, which each side informing the other when it has finished its operation
and handing transmit control over. However with multiple channels data can be physically flowing in both
directions at the same time for different exchanges.

Cisco Live 2016

7/11/2016

There are multiple other scsi operations including queries, unit testing, format, and erase.

Cisco Live 2016

7/11/2016

10

Cisco Live 2016

7/11/2016

Configuration manipulation for QoS for lossless iscsi can be found here:
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5600/sw/san_switching/7x/b_5600_SAN_Swit
ching_Config_7x/configuring_iscsi_tlv.html

11

Cisco Live 2016

7/11/2016

Fame is 2148 Bytes at Max (2112 data and 36 framing)

SRC FCID: FCID of the originating device
DST FCID: FCID of the destination device
Type: Protocol type in the frame
Seq_ID: Sequence this frame belongs to
SEQ_CNT: Sequential count of frames
OXID: Originator Exchange ID

12

Cisco Live 2016

7/11/2016

13

Cisco Live 2016

7/11/2016

14

Cisco Live 2016

7/11/2016

15

Cisco Live 2016

7/11/2016

Initiator sends flogi to the switch

Switch responds with an accept and assigns and FCID to the initiator
Initiator plogis to the switches name server
Switch returns a list of devices the server can talk to
Initiator sends a plogi to its zoned targets
Target accepts the plogi and the devices begin to exchange parameters and permissions.

16

Cisco Live 2016

7/11/2016

VSAN is a Cisco only feature. When connected to a 3rd party switch you will need to have 1 vsan in the interop
mode for that vendor to communicate
Trunking for E ports is always available
Trunking for F ports is available with features fport-channel-trunk
VSAN names are optional but useful for identifying what the vsan is fore

17

Cisco Live 2016

7/11/2016

State: Active this means that the vsan is currently available to be used by end devices.
Interoperability mode:default. Interop mode is used to connect to various vendor devices
Interop 1 is the industry standard interop and would have to be set on all switch vendor switches
Interop 2 and 3 are to allow cisco switches to interop with Brocade switches without requiring config changes on
the brocade
Interop 4 is for interop with legacy mcdata switches
Load balancing is covered in an later section on fc/fcoe load balaning
A vsan must be suspended to change the interop mode using the following:
5648-A#conf t
5648-A(config)# vsan database
5648-A(config-vsan-db)# vsan 105 suspend
Load balancing determines how egress traffic is hashed on equal cost paths
An operational state of down indicates that there are currently no ports on this switch that have this vsan up and
active

18

Cisco Live 2016

7/11/2016

State: Active this means that the vsan is currently available to be used by end devices.
Interoperability mode:default. Interop mode is used to connect to various vendor devices
Interop 1 is the industry standard interop and would have to be set on all switch vendor switches
Interop 2 and 3 are to allow cisco switches to interop with Brocade switches without requiring config changes on
the brocade
Interop 4 is for interop with legacy mcdata switches
Load balancing is covered in an later section on fc/fcoe load balancing
A vsan must be suspended to change the interop mode using the following:
5648-A#conf t
5648-A(config)# vsan database
5648-A(config-vsan-db)# vsan 105 suspend
Load balancing determines how egress traffic is hashed on equal cost paths
An operational state of down indicates that there are currently no ports on this switch that have this vsan up and
active

19

Cisco Live 2016

7/11/2016

Domains have to be unique to a vsan but do not have to be unique in the fabric
The Domains do not have to be the same fore each vsan on a switch

20

Cisco Live 2016

7/11/2016

The FC domain will only change on a disruptive restart

The disruptive option is a hidden command and will force a logout and relogin of all locally connected devices in
that vsan
The [local] option shows us which switch in the list that this commands was ran from
The [Principal] switch is uniquely assigned to each vsan and is in charge of handing out domain ids to other
switches in the vsan that do not have static configurations as well as other domain level fc control plane
operations

21

Cisco Live 2016

7/11/2016

7706-A(config)# int fc 1/24

7706-A(config-if)# fspf cost 50 vsan 100
Switch calculates route cost to adjacent switches based on port speed
Route cost to remote switches is calculated by adding the cost of the adjacent route to the cost of routes learned
for that switch

22

Cisco Live 2016

7/11/2016

23

Cisco Live 2016

7/11/2016

24

Cisco Live 2016

7/11/2016

Enhanced zoning also provides a more efficient use of the zoning database space resulting in a small foot print for
the same sized zoneset

25

Cisco Live 2016

7/11/2016

26

Cisco Live 2016

7/11/2016

The * lets us know that this device is logged into the fabric
The FCID tells us which domain the device is logged into, the 1st octet (40) indicates that this device is logged into
the switch with the fcdomain ID of 40 (hex 28) for vsan 105

27

Cisco Live 2016

7/11/2016

This command gives us the status for the zone settings and zoneset .
Distribute: active means that only the active zoneset is distributed and that a zoneset distribute vsan 105 would
be need to push the full zoneset to the switches in the rest of the fabric
Mode basic lets us know that on this vsan is configured for basic only which would allow for multiple concurrent
edits
Name: CiscoLive shows that the current active zoneset is the one named CiscoLive
Status: This shows that the last activation completed. If there is a failure of an activation the notice will be here.

28

Cisco Live 2016

7/11/2016

MTU should be 2240 to account for a full sized FC frame being encapsulated

29

Cisco Live 2016

7/11/2016

30

Cisco Live 2016

7/11/2016

31

Cisco Live 2016

7/11/2016

32

Cisco Live 2016

7/11/2016

33

Cisco Live 2016

7/11/2016

34

Cisco Live 2016

7/11/2016

35

Cisco Live 2016

7/11/2016

Configuration for port connected to FCoE edge device

1) Configure L2 interface
2) Set Spanning-tree port type edge trunk
3) Must trunk at least the native vlan + fcoe vlan
3b) Can trunk other Ethernet vlans
4) Create vfc
5) Bind the vfc to the L2 interface
6) The VFC should then be added to the vsan database

36

Cisco Live 2016

7/11/2016

Enode is the fcoe edge node

FCF (fibre channel forwarder) is the virtual fibre channel switch
Vlan discovery occurs over the native vlan and is used to determine the FCoE vlan for the following
communications
FCF Discovery is used to determine the FCF that will accept a login.
The FCF advertisement includes the mac for the FCF as well as tuning options (FIP time out, fcf priority, etc)
Once we know the FCF mac for a vlan a flogi is sent and accepted
At this point we move over to FCoE for the plogi and all subsequent FC commands
There will be periodic fip messaging between the FCF and eNode to ensure that the node it still online

37

Cisco Live 2016

7/11/2016

Evpc provides for a virtual port channel to a host connected to a pair of fex that are in turn connected to a VPC
pair
This requires a vpc from the 5k Pair to each FEX and then an additional vpc to the 2 host ports
This is a common L2 ethernet feature on the nexus line and should be known. Below if a config for 1 side of a
VPC:
Feature enablement
feature vpc
Config a domain and the IP address of its peer
vpc domain 21
peer-keepalive destination 14.39.10.122
Configure L2 link between vpc peers
interface port-channel1
switchport mode trunk
spanning-tree port type network
speed 40000
vpc peer-link

Build VPC to fex

38

Cisco Live 2016

7/11/2016

interface port-channel101
switchport mode fex-fabric
speed 40000
vpc 101

38

Cisco Live 2016

7/11/2016

This configuration assume that EVPC has already been configured and that fcoe

39

Cisco Live 2016

7/11/2016

40

Cisco Live 2016

7/11/2016

41

Cisco Live 2016

7/11/2016

42

Cisco Live 2016

7/11/2016

43

Cisco Live 2016

7/11/2016

Logins are load balanced across the available uplinks for a specific vlan.
A link failure will cause the devices logged in over that interface to relogin in over the remaining uplink
Port channels allow for all logins to be recorded to the logical port channel interface instead of a physical link
The biggest benefit of this is that a single link failure will not cause a host to log out
Ports can be statically pinned to uplinks

44

Cisco Live 2016

7/11/2016

45

Cisco Live 2016

7/11/2016

46

Cisco Live 2016

7/11/2016

47

Cisco Live 2016

7/11/2016

48

Cisco Live 2016

7/11/2016

The number of no drop classes must be consistent between switches or DCBX will fail

49

Cisco Live 2016

7/11/2016

50

Cisco Live 2016

7/11/2016

With flow based, a host/target pair will always use the same link in a port channel
With exchange based each operation (read, write, query, etc) will be hashed individually to a link

51

Cisco Live 2016

7/11/2016

52

Cisco Live 2016

7/11/2016

53

Cisco Live 2016

7/11/2016

54

Cisco Live 2016

7/11/2016

55

Cisco Live 2016

7/11/2016

56

Cisco Live 2016

7/11/2016

57

Cisco Live 2016

7/11/2016

58

Cisco Live 2016

7/11/2016

59

Cisco Live 2016

7/11/2016

60

Cisco Live 2016

7/11/2016

61

Cisco Live 2016

7/11/2016

62

Cisco Live 2016

7/11/2016

63

Cisco Live 2016

7/11/2016

64

Cisco Live 2016

7/11/2016

65

Cisco Live 2016

7/11/2016

66

Cisco Live 2016

7/11/2016

67

Cisco Live 2016

7/11/2016

Feature type NPV indicates that this is a npv switch

Feature type scsi-fcp:init fc-qs indicates that this device is an initiator

68

Cisco Live 2016

7/11/2016

69

Cisco Live 2016

7/11/2016

The interface shows the physical (fc1/10) or logical address the device is logged in on
VSAN indicates the vsan the device is logged in on
Port name is the pwwn that is logged in. If there is an alias configured, it will be shown below the pwwn

70

Cisco Live 2016

7/11/2016

The admin port mode lets you know the configured mode for the port (F, N, E)
The port vsan is which vsan in the vsan database that the port is assigned
Trunk vsans:
Allowed and active: These vsans are fully active and passing traffic
Up: These are vsans that are up on the trunk but not active
Isolated: the are vsans that existing on both side of the trunk but have been isolated due to a check failure
Commonly this failure will be either a zone merge failure or vsan interop mismatch
Initializing: This is the state of a configured vsan that is attempting to come online on the link.

71

Cisco Live 2016

7/11/2016

VFC is down for L2 down. This could mean you need to check the bound interface
Bound interface is the L2 port (ethernet, vethernet, or port-channel) that is bound to this vfc

72

Cisco Live 2016

7/11/2016

Similar to show cdp neighbor for ethernet

Shown on a per vsan basis

73

Cisco Live 2016

7/11/2016

Zone merge fail will result in a vsan suspended on the trunk

Merge failure messages are fairly accurate; Zoning Modes do not match means that 1 side is basic and 1 side is
enhanced
Failure can also occur if a different zoneset is active on both switches, or if the same zoneset is active but it has
different zone/member configuration

74

Cisco Live 2016

7/11/2016

75

Cisco Live 2016

7/11/2016

76

Cisco Live 2016

7/11/2016

77

Cisco Live 2016

7/11/2016

78

Cisco Live 2016

7/11/2016

79

Cisco Live 2016

7/11/2016

80

Cisco Live 2016

7/11/2016

81