JohnHufferd Fibre Channel Over Ethernet-FCoE

Fibre Channel over Ethernet (FCoE)
John L Hufferd, Consultant

Hufferd Enterprises
SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA.
Member companies and individual members may use this material in
presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without
modification
The SNIA must be acknowledged as the source of any material used in the
body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.

Neither the author nor the presenter is an attorney and nothing in this
presentation is intended to be, or should be construed as legal advice or an
opinion of counsel. If you need legal advice or a legal opinion please
contact your attorney.
The information presented herein represents the author's personal opinion
and current understanding of the relevant issues involved. The author, the
presenter, and the SNIA do not assume any responsibility or liability for
damages arising out of any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
2010 Storage Networking Industry Association. All Rights Reserved.
Abstract
The Fibre Channel (T11.3) standards committee
developed a Standard called Fibre Channel over
Ethernet (FCoE)
The FCoE standard specifies the encapsulation of
Fibre Channel frames into Ethernet Frames and the
amalgamation of these technologies into a network
fabric that can support Fibre Channel protocols and
other protocols such as TCP/IP, UDP/IP etc.
A Direct End-to-End FCoE variant has been
accepted for the next version of the Standard
The tutorial will show the Fundamentals of these
FCoE concepts and describe how they might be
exploited in a Data Center environment
Agenda
Introduction
FCoE Fabrics & Convergence
Architecture
Discovery & Link Instantiation
With FCFs
Direct End-to-End (w/o FCFs)
Topologies with FCFs

Scenarios with FCFs
Summary
Introduction
This presentation provides an overview of Fibre Channel over
Ethernet (FCoE)
One should think about FCoE as placing the FC protocol on a new
physical link
New Lossless Ethernet links instead of physical FC links
But it is still Fibre Channel
The protocol has been defined in the INCITS Fibre Channel (T11.3)
technical committee
The needed Lossless Ethernet links has been defined in the IEEE
802.1
A newly defined Direct End-to-End (VN2VN) protocol will also
be explained
FCoE Fabrics
&
Convergence
FCoE Fabrics
(part 1)
(This is NOT Traditional Ethernet)
FCoE requires specific Ethernet extensions to be implemented
Lossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE) configurations are required
Jumbo frame support is strongly recommended (not a standard, but widely available)
Deployments of FCoE should utilize the advances in Ethernet recently

defined in IEEE 802.1, specifically:
Priority-based Flow Control (PFC) 802.1Qbb
Enhanced Transmission Selection (ETS) 802.1Qaz
DCB (capability) eXchange (DCBX) Protocol 802.1Qaz
CN -- Congestion Notification (802.1Qau) New
These 802.1 advances are important for Converged Flows (Messaging,

Clustering and Storage)
This set of functions is called CEE Converged Enhanced Ethernet (intended for a
Data Center Environment) or (in the IEEE ) DCB -- Data Center Bridging
FCoE Fabrics must be built with FCoE CEE/DCB Switches that:
Are called FC Forwarder FCF

Are part of a lossless Ethernet Fabric and have CEE/DCB Lossless Ethernet ports
Also provide functions of traditional FC switches (capabilities and services)
Converged Enhanced Ethernet (CEE) &

FCoE Switch (FCF) with FC connections
Implementations are combining the features and capabilities of
a CEE/DCB Switch with the features and capabilities of a FC
switch which will:
Support Ethernet and IP standards for switching, pathing
and routing
Support FC standards for switching, pathing and routing
Support current and CEE/DCB Ethernet
Standards
FC
FC
CEE/DCB Ethernet Ports

(with IP & FCoE VF_Port &
FC
VE_Port capabilities)
Adapt between FCoE and FC
The FCoE Ports have F_Port or E_Port functions

Called VF_Ports and VE_Ports
(Because many logical (virtual) ports can share one physical port)
Connections to an Integrated
CEE/DCB - FCoE Switch
Fibre Channel is carried over lossless Ethernet as a L3 protocol
Customer Applications
SCSI
Network Applications iSCSI
UDP
TCP
IP
Fibre Channel
FCoE
Lossless Ethernet MAC (CEE/DCB)

IP address 123.45.67.89
(FCoE VN_Port)
Ethernet port with IP & FCoE VF_Port
capabilities
Integrated Lossless Ethernet (CEE/DCB) Switch

with FCoE Switch (FCF) capabilities
FCoE Fabrics
(part 2)
(This is NOT Traditional Ethernet)
CEE/DCB only Ethernet switches may also exist in an FCoE capable Fabric but
one or more switches with FCoE capabilities must also exist
FCoE fabrics must inter-operate seamlessly with real FC Fabrics
FC services must operate identically on FCoE fabrics and Fibre Channel fabrics
FCoE must support all Fibre Channel advanced features (e.g. virtual fabrics, IFR,
security, etc.) transparently
FCoE will not require changes to FC software (Apps, Drivers, etc.)
However, vendors will enhance Drivers & Mgnt to exploit new capabilities
FCoE is NOT a replacement for FCIP

FCIP is for inter-switch links beyond the Data Center
FCIP uses TCP/IP

10
The Compelling Value of Convergence

is at the (Server Edge) Interface
OS1
Web Server
OS3
DB Server
OS2
App Server
Hyper Visor
OS2
App Server
FC
HBA
OS1
Web Server
Messaging
FC SAN
Ethernet
TCP/IP
E-HBA
(CEE)
MPI
RDMA
IB/Ethernet
Cluster
Today
Data Center
CEE/DCB
Network
OS2
App Server
OS3
DB Server
E-HBA
(CEE)
OS1
Web Server
OS3
DB Server
NIC
TCP acceleration.
MPI, RDMA
over Ethernet
FCoE
With
CEE/DCB
E-HBA
(CEE)
Data Center
CEE/DCB
Network
Converged Enhanced Ethernet (CEE/DCB)

(A Lossless Ethernet with Priority based Flow Control and Scheduling)
Dramatic Interface reduction in adapters, switch ports, cabling, power, & cooling
4-6 cables can be reduced to 2 Interfaces/cables per server
Seamless connection to the installed base of existing SANs and LANs

Effective sharing of high bandwidth links
11
But Even a Total Data Center (CEE/DCB) Fabric

Requires Phase in (starting at the Server Edge Interface)
Mainframe
Remote
Offices
FICON
iSCSI
Storage
CEE/DCB-FCoE
Ethernet SW
FC
FC & CEE/DCB
Ethernet
SW Blades
FC
FCoE
Outfacing
IP Network
Data Center
Fabric
(LAN/WAN)
Including iSCSI
Gateways
Messaging
NAS
FICON
FC
FICON Storage
Controller
CEE/DCB Ethernet
SW
File Storage
Arrays (NAS)
Business
Campus
with iSCSI
connections
Note: with multiple Data Centers

there may also be interconnects
with DWDM, FCIP/iFCP, etc.
FCoE permits intermixing of multiple Connection types/protocols

Clustering messaging, General Messaging, and Storage
The Data Center Fabric will Trunk to the Outfacing Network
But many Customers may want keep a mixed environment on-going
FC Link
FICON
Link
Ethernet
Link
12
Architecture

13
FC Encapsulation Into Ethernet Frames

(2 FCoE Related Packet types)
FC-4
FC-4
FC-3
FC-3
FC-2
Unchanged
FC Levels
FC-2V
FC-2V
FC-2M
FC-2P
FCoE Mapping
FC-1
MAC
FC-0
PHY
(Located in the
FC Entity*)
(Located in the
FCoE Entity*)
IEEE 802.3
Layers
* Discussed in later slides
Fibre Channel over Ethernet (FCoE) Packets

Ethernet
Header
Ethertype
FCoE
(8906h)
FCoE
Header
FC Header
SCSI Commands/Data
FCS
FC Imbedded Frames: Same as in Physical FC

Protocol control information: Version, SOF, EOF, etc.
Frame Check Sequence
(CRC)
Ethernet Header provides things needed for the

physical network, including Ethertype
FCoE Initialization Protocol (FIP) Packets

Ethernet
Header
Ethertype
FIP
(8914h)
FIP
Header
Descriptors
FCS
Discovery, Link establishment, maintenance & disconnect (Login/Logout, etc.) Parameters

Protocol control information: Version, Op-codes, etc.
14
FIP Protocol and FCoE Protocol

Discovery Phase
FCFs Discover each other, & VLANs (if any) then form a Fabric
ENodes Discover VLANs (if any) & then ENodes & FCFs Discover
Potential VN_Port VF_Port pairing
Capabilities of Potential pairing
Login Phase
ENodes chose among discovered FCFs Ports

Creates association between ENode Ports and FCF Ports
VN_Port VF_Port Logical FC Link
Allowed methods for the ENode MAC Addressing
FCoE
Initialization
Protocol
(FIP)
Fabric Provided MAC Addresses (FPMA) Assigned by the Fabric

Server Provided MAC Addresses (SPMA) Assigned by the Server
Uses: FLOGI, FLOGI ACC, LOGO, ELS,
End-to-End path control & Data Transfer Phase

PLOGI/PRLI
All other FC protocol frames (FC4 ULPs. etc.)

FCoE
Protocol
15
FCs Encapsulation in Ethernet (FCoE)

Word
31-24
23-16
15-8
7-0
Destination MAC Address (6 Bytes)
1
Optional IEEE 802.1q
4 Byte
Tag goes here
2
3
Source MAC Address (6 Bytes)
Reserved
Reserved
6
This field varies
In size
n
Ethernet frame
Size is 64 Bytes to
2220Bytes
Ver (4b)
ET=FCoE (16 bits)
n+1
n+2
Reserved
Reserved
(12 bits)
SOF (8 bits)
Encapsulated FC Frame
FC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)
(including FC-CRC)
EOF (8 bits)
Reserved
Ethernet FCS

16
FIP Operation Format

Word
0
31-24
23-16
15-8
7-0
Destination MAC Address (6 Bytes)
1
Optional IEEE 802.1q
4 Byte
Tag goes here
2
3
4
See Appendix
Below for
Descriptor list items
Descriptor list
varies
In size
Source MAC Address (6 Bytes)

Ver (4b)
ET=FIP (16 bits)
Reserved
FIP Operation Code

Reserved
FIP Operation
Code
FIP SubCode
Reserved
FIPLength
subcode
Descriptor List
Flags
Descriptor List
PAD to minimum length or mini-Jumbo length
FP SP
(12 bits)
Descriptor
List Length
Flags
Ethernet frame
n+1
Ethernet FCS
size
Is 64Bytes to 2220Bytes Capability Bits (SPMA or FPMA)
Available bit
ASF
F S S F
P P
Solicited bit
FCF bit
17
FCF Model
OR
Link
End
Point
(LEP)

OR
18
HBA Model of the ENODE with

Multiple Logical FC interfaces
Multiple FC NPIV
instances on a single
logical FC Host interface
FC-3 /FC-4s
FC-3 /FC-4s
VN_Port
FC
Entity
VN_Port
FC
Entity
FCoE_LEP
FCoE
Entity
FCoE_LEP
FCoE
Entity
For each logical N_Port

(VN_Port) there is one FLOGI
and perhaps 100s of FDISC
Each VN_Port is seen by
the Host as a separate
(logical) FC Login/FDISK
connection
...
FCoE
Controller
The number of (logical) FC

connections is
implementation dependent
Lossless Ethernet MAC
Ethernet_Port
Burnt-in
MAC Address
In this
model this
is where
FC-2V
functions
live
In this model
this is where
the
Encapsulation
/DeEncapsulation
functions live
MAC Address of
FCoE_LEP
(VN_Port)
With FPMA it is
different than that
of the FCoE
controller
Only one MAC Address is required for the FCoE Controller

FCF will supply MAC addresses for each VN_Port
May also have Multiple implementations of this if HBA has Multiple Physical Ports
19
Multiple Logical FC connections

via a single Ethernet MAC
VF_Port
VN_Port
FCoE_LEP
Examples of
single MACs
with
connections
to two
different
FCFs
FCoE_LEPs
FCF-MAC(A)
MAC(H1)
FCF A
VF_Port
H1
FCoE_LEPs
FC
Fabric
VN_Port
FCoE_LEP
Lossless
Ethernet
Switch
VN_Port
FCoE_LEP
H2
FCF-MAC(B)
VF_Port
FCoE_LEPs
FCF B
VF_Port
FCoE_LEPs
MAC(H2)
VF_Port,
VF_Port_Name
VN_Port
The Logical FC Link is defined by a MAC Address pair

VN_Ports,
A VN_Port MAC Address
VN_Port_Names
A VF_Port MAC Address
For a logical FC link the FCoE Frames are always sent to and received from a specific FCFs
MAC Address
Therefore, pathing to and from the FC driver is always defined by the MAC Address of the
partner FCFs VF_Port
FCoE_LEP
Traditional
Ethernet
LAN
Note: VF ports get created off of the FLOGI from the VN port and multiple VF ports can sit behind a single
FCF physical interface with a single MAC address
20
Functions of an FCoE Converged Network

Adapter (CNA) in the Initiator
A
B C
Host PCIe interface
FCoE
ASIC
External
Port
FCoE
function
NIC Function
Lossless
Ethernet
MAC
Has a Normal NIC interface

(A) to the Host
Has one or more Normal
FC interfaces (B,C) to the
Host
FC
Function
FCoE functions not seen by

the Host
FCoE functions perform the
Encapsulation and Deencapsulation
The FCoE function
Instantiates Logical FC
N_Ports, called VN_Ports

21
CNA with Multiple Logical Interfaces
The FCoE controller will perform the FIP functions and will instantiate new
VN_Ports as FCoE Link End Points (LEP)
With a new MAC address specified by the FCF (FPMA)
A
FCoE Chip
MAC
Address of
Burnt-in
MAC
FC
Entity
FCoE
Controller
FC
Entity
FC-3 /FC-4s
VN_Port
FCoE
Entity
Lossless Ethernet
MAC Ethernet_Port
FCoE_LEP
Etype =
FCoE
or
FIP?
FC-3 /FC-4s
VN_Port
FCoE
FCoE_LEP
Entity
NIC
MAC
Address of
Burnt-in
MAC
FCoE
Function
MAY have one burnt-in

MAC address for both IP
and FCoE/FIP packets
Or
MAY have different Burntin MACs for IP and

FCoE/FIP packets
FC Function
Used to separate HW
based FCoE from other
Ethernet Traffic
Most NICs come with
several Burnt-in MAC
Addresses

22
Discovery and Link Instantiation

(FIP -- FCoE Initiation Protocol)

23
Initial Login Flow Ladder (2 FIP Phases)

(within any specific VLAN)
End-Node
FCF
FCoE Initialization
Protocol (FIP)
Discovery
Discovery
Phase
Unicast FCF MAC Address

et.al. in Jumbo Frame
FLOGI
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
FC Command
(Using the FCF
selected MAC
Address as the SA)
Login Phase
Normal FC
Processing
FC Command responses
FCoE Protocol
See Appendix for more details on Discovery Consideration and Actions, including discovering VLANs
25
New Direct VN_Port to VN-Port

(VN2VN)
DCB/CEE Ethernet
Switch (A)
CNA &
VN_Port (X)
(L1)
(L2)
(L3)
DCB/CEE Ethernet
Switch (B)
(L11)
(L5)
(L4)
(2)
(L8)
(L6)
Storage
Controllers
Host
Systems
(1)
(L7)
CNA &
VN_Port (Y)
(3)
(a)
(4)
(b)
(c)
(d)
A CNA to CNA FCoE path between these Switch ports is now also possible even
without an FCF using Direct VN2VN mode
DCB/CEE Ethernet
Switch (A)
CNA &
VN_Port (X)
CNA &
VN_Port (Y)
(L4)
(L1)
(L2)
(L3)
Storage
Controllers
Host
Systems
(1)
(2)
(a)
(b)
Additionally an internal CNA to CNA FCoE path within a CEE/DCB Switch is also
possible even without an FCF using Direct VN2VN mode
26
New Initial Login Flow Ladder

(Direct End-to-End VN2VN)
End-Node
End-Node
Probe & Claim
FCoE Initialization
Protocol (FIP)
(after Randomly
computing ID)
(repeat
until no
conflicts)
Unicast Probe & Claim responses
Discovery
Phase
FLOGI
Login Phase
FLOGI ACC to accepted

VN_Port
FC Command
Normal FC
Processing
FC Command responses
FCoE Protocol
27
New Discovery and Link Instantiation

(for VN2VN)
Create
VN_Port
MAC Addr
Host
Systems
Create
VN_Port
MAC Addr
Establish FC-ID & MAC

Address & Show
Capabilities
Storage
Controllers
Randomly chooses an identity (FC-ID & MAC Address)

Then uses a Multicast FIP processes called Probe, & Claim to insure ID uniqueness
This will announce the VN_Ports identities and the VN_Ports capabilities to other VN_Ports
A VN_Port will save the announcements from other VN_Ports for choosing a peer VN_Port
Note: FC-ID & MAC Address should be saved, if possible, for next Reboot
Send/Receive
FLOGI/FLOGI ACC &
then FCoE Frames
between VN_Ports
Send/Receive
FLOGI/FLOGI ACC &
then FCoE Frames
between VN_Ports
Host
Systems
FLOGI/FLOGI ACC between

compatible VN_Ports
Then send FCoE frames
(PLOGI, PRLI, Storage
Commands & Data, etc.)
Storage
Controllers
After IDs and Potential Partners (VN_Ports) are identified within the Level 2 Ethernet:
FLOGI FIP frames, & FCoE frames will be exchanged directly between the VN_Ports
After Link Instantiation the IDs will be Periodically Beaconed (Multicast)
Beaconing permits detection of link loss (via time-outs) & incorrect LAN joins
28
Topologies with FCFs

29
A Simple CEE/FCoE (FCF) Fabric
CEE/DCB FCoE (FCF)

Switch
FC
An CEE/DCB - FCoE (Integrated) Switch may connect to a Traditional FC

switch/fabric via the FC E-Port
Ethernet
FC
30
Multiple Topologies using FCoE Switches

(FCFs)
FCoE Switch
FC
Fabric
CEE/DCB
Network with all
CEE/DCB - FCoE
Switches
A Lossless Ethernet Network can be made up of all CEE/DCB - FCoE (Integrated)

Switches
CEE/DCB
Network
FC
Fabric
Lossless Ethernet (CEE/DCB) switches configured into a Lossless

Ethernet (CEE/DCB) Network can Front the FCoE Switch
CEE/DCB
Network
3
CEE/DCB FCoE
Switch
CEE/DCB FCoE
Switch
CEE/DCB
Network
FCoE
Switch
CEE/DCB - FCoE
Switch
FCoE
Switch
CEE/DCB
Network
2
FC
Fabric
CEE/DCB - FCoE (Integrated) Switches deployed at the edges of the

Lossless Ethernet CEE/DCB Network
CEE/DCB - FCoE Switches connected via VE_Ports and a Lossless
Ethernet CEE/DCB Network
CEE/DCB
Network
1
FC
Fabric
FCoE
Switch
CEE/DCB
Network
4
A VE_Port in an FCF may connect to another VE_Port in another FCF

And an FCF FC E_Port may connect to an FC switch E_Port in the FC
Fabric

31
Ethernet
FC
Scenarios with FCFs

32
Scenario 1: FCoE & IP Flows
Internet
Classical
Ethernet
Network
CEE/DCB
Network
FCoE
Switches
FC
Fabric
CEE/DCB
Network
FCoE
Switch
FCoE Flows
IP Flows
33
Scenario 2: FCoE Right & Wrong
Internet
Classical
Ethernet
Network
CEE/DCB
Network
FCoE
Switches
FC Fabric
FCoE
Switch
CEE/DCB
Network
FCoE Flows
Invalid FCoE Flows
34
Summary

35
FCoE Summary
T11.3s FC-BB-5 Ad-Hoc Working Group completed the specification in June 2009
FCoE is a simple, efficient mechanism for encapsulating Fibre Channel in Ethernet frames
on a New Ethernet type Network
Not a traditional Ethernet Interface or fabric
A New Network the Converged Enhanced Ethernet (CEE) Network

CEE (also called DCB) defined in the IEEE 802.1 standards working group
FC protocol frames will just be inserted into these Ethernet frames
An evolutionary deployment model was designed into FCoE
Specification permits the installation to evolve from FC to FCoE
Any Fabric mix of FC and FCoE is possible

May only need FCoE at the Server Edge with a Converged Interface
But a total FCoE SAN is also possible (using FCFs)
Value in reduced Server Edge Cables, Adapters, Power, and Cooling
All FCoE devices should interoperate with Real FC devices
FCoE is made for a Data Center Fabric Not applicable for the Outfacing Network
T11.3s FC-BB-6 Ad-Hoc Working Group accepted VN2VN for the next specification
Look for VN2VN to provide viable FCoE SANs for the Entry/Low-End IT Environment
Will be seen to be competitive with iSCSI, especially with the SW version of FCoE
Expect CEE/DCB to be included in 1Gb Ethernet Switches

36
Q&A / Feedback
Please send any questions or comments on this
presentation to SNIA: tracknetworking@snia.org
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti
Robert Snively
Joe Pelissier
Howard Goldstein
Suresh Vobbilisetty
John Hufferd
Walter Dey
Silvano Gai
For additional information refer to

http://www.t11.org/fcoe
37
Thank You!
Appendix
Full ENode Model
(with VN2VN)
Additional Info on FCoE

Fabrics
FCoE Relation to ISO
Layers
Flows
Additional Topologies
FSPF and STP
FIP Considerations and
Actions
FIP Descriptors
Pause vs BB_Credit
39
Full ENode Model

(with VN2VN & multiple VLANs depicted)
FC-3 /FC-4s
VN_Port FC-2V
FCoE_LEP
FCoE
Entity
VLAN-1
(Maybe the
default
VLAN)
VLAN-4
VLAN-3
VLAN-2
Each instantiation's N_Port_ID & MAC Address is independent of the others

There can be duplicates (if they are in different VLANs)
40
FCoE Fabrics (part 3)

FCoE is a direct mapping of Fibre Channel over an Ethernet
network
FCoE is layered on top of a Lossless Ethernet
FSPF used to route FCoE packets
Ethernet Spanning Tree type protocol, RSTP, MSTP, etc, is at a
layer below
FCoE allows an evolutionary approach towards consolidation of

fabrics
The Fibre Channel N_Port, F_Port and E-Port constructs must be
retained
With FCoE, ports may be connected with Logical Ethernet Links
May pass through Lossless Ethernet NICs & Switches
Identified by pairs of end point MAC addresses
Physical Ethernet Links can replace physical FC Links

Physical Ethernet Links can carry all Ethernet traffic, including FCoE,
but combined traffic needs the CEE/DCB capabilities
41
FCoE Fabrics
(part 4)
Integrated FCoE Switches are being built that

support traditional Ethernet traffic, FCoE traffic , & FC
traffic
The FCoE solutions appear as a Fibre Channel to a
Fibre Channel experienced customer
FCoE keeps the Fibre Channel operations independent
from Ethernet forwarding
Keeps Management /Troubleshooting simple
Common physical structures, different logical structures
Based on Ethertype (Ethertype = FCoE, or FIP)
Storage Management should be unchanged

42
FCoE Relation to ISO Layers

43
Flows

44
Logical Fabric Topology

An FCoE Switch receives FCoE frames addressed to its FC-MAC address and
forwards them based on the D_ID of the encapsulated FC frame
Logical
Transaction Path
CEE/DCB
Network
FCoE
Switch
Ethernet
FC
H2
FCoE
Switch
#2
#1
CEE/DCB
Network
#3
CEE/DCB
Network
#4
FCoE
Switch
H4
H3
H1
FC
S2
CEE/DCB
Network
S1
An FCoE Switch rewrites the SA and DA of an FCoE frame

Ethernet
Destination
& Source
Encapsulated
FC Frame
D_ID
S_ID
Path #1
FCoE-A MAC
FCoE-H2 MAC
FC_ID for S1
FC_ID for H2
Path #2
FCoE-B MAC
FCoE-A MAC
FC_ID for S1
FC_ID for H2
Path #3
FCoE-C MAC
FCoE-B MAC
FC_ID for S1
FC_ID for H2

Path #4
FCoE-S1 MAC
FCoE-C MAC
FC_ID for S1
FC_ID for H2
45
Single Ethernet Fabric

with FCoE Switches
H3
H2
Ethernet
Switch
Ethernet
Switch
#1
FCoE A
Switch
Ethernet
Switch
Lossless Ethernet
(CEE/DCB) Network
Path #1
FCoE-A MAC
FCoE-H2 MAC
Path #2
FCoE-C MAC
FCoE-A MAC
FC Fabric
#2
Ethernet
Switch
H1
Ethernet
Destination
& Source
FCoE
Switch
#3
Ethernet
Switch
S2
S1
Path #3
FCoE-S1 MAC
FCoE-C MAC
Encapsulated
FC Frame
D_ID
FC_ID for S1
FC_ID for S1
FC_ID for S1
S_ID
FC_ID for H2
FC_ID for H2
FC_ID for H2

Logical
Transaction Path
Ethernet
FC
46
FC Host to FCoE Storage

H3
H5
S2
H2
Ethernet
Switch
#3
FC Fabric
#2
Lossless Ethernet
(CEE/DCB) Network
S3
Ethernet
Switch
S1
H1
Ethernet
Destination
& Source
#1
FCoE
Switch
C
Ethernet
Switch
FCoE
A
Switch
Ethernet
Switch
Ethernet
Switch
Path #3
Path #2
FCoE-S2 MAC
FCoE-A MAC
FCoE-A MAC
FCoE-C MAC
Path #1
Encapsulated
FC Frame
D_ID
FC_ID for S2
FC_ID for S2
FC_ID for S2
S_ID
FC_ID for H5
FC_ID for H5
FC_ID for H5

Logical
Transaction Path
Ethernet
FC
47

48
Lossless
Ethernet
(CEE/DCB)
Switch (X)
FCoE
Switch (A)
(1)
FCoE
Switch (E)
FC
FCoE
Switch (B)
Lossless
Ethernet
(CEE/DCB)
Switch (Y)
FCoE
Switch (F)
Example of Topologies with FCoE Edge Switches

49
Lossless
Ethernet
Switch
(CEE/DCB)
FCoE
Switch (A)
(2)
FCoE
Switch (E)
FC
Lossless
Ethernet
Switch
(CEE/DCB)
FCoE
Switch (B)
FCoE
Switch (F)
Example of Topologies with CEE/DCB Edge Switches

50
FSPF and STP

51
FSPF & STP Concepts with FCoE (basic)
FCoE Switch
FCoE Switch

52
FSPF & STP Concepts with FCoE (Interconnected)
FCoE Switch
FCoE Switch

53
Equivalent FC Topology

54
FIP Considerations and Actions

55
VLAN Determination
There may need to be a dynamic determination of what VLANs
can be used for FCoE operations
But this is optional -- the FIP VLAN discovery protocol is not needed if
these VLANs are already known or if VLANs are not used
Expect this to be administratively determined
But if dynamic VLAN information is needed:

FCFs will issue a VLAN information Request to other FCFs
Sent via a Multicast to All_FCF_MACs
Receiving FCF VE_Port capable MACs will respond with a Unicast
Notification which contains a list of VLANs that can be used
The receiving FCF VE_Port may use one or more of these VLANs
ENodes will issue a VLAN information Request

Sent via a Multicast to All_FCF_MACs
Receiving FCF VF_Port capable MACs will respond with a Unicast
Notification which contains a list of VLANs that can be used
The ENode may use one or more of these VLANs
56
FCoE Discovery Considerations

After VLANs, if any, are determined, the discovery process uses two types of messages,
Solicitations and Advertisements
This part of the FIP Discovery Phase helps define the FCF Ports that are available for the
Link instantiation Phase
The ENodes discover the FCF ports that can become VF_Ports and FCFs discover other
FCF ports that can become VE_Ports
ENodes Solicit (via Multicast of All-FCF-MACs) Advertisements from FCFs while specifying
their capabilities
In response FCFs Advertise (via Jumbo Unicast) their VF_Port capable MACs availability and
capabilities back to the solicitating ENode
FCF Multicast (to All-FCF-MACs) their VE_Port capable MACs existence to other FCFs
Sometimes a New FCF will come on line and Multicast its availability to ALL_
ENode_MACs and All_FCF_MACs
The FIP Discovery phase exchanges solicitation and/or Advertisements between (HBAs
and/or FCFs) FCoE Controllers
57
FCF Discovery Actions

An FCF supporting VE_Ports:
Discovers other VE_Port capable FCF-MACs, connected to the
same Lossless Ethernet segment, by:
Transmitting a multicast Solicitation to All-FCF-MACs
(with the FCF bit set to one)
Receiving back Jumbo Unicast Advertisements from VE capable

MACs
Which also verifies the support of Ethernet Jumbo frames in the path
Sending Jumbo Unicast Advertisements from its own VE

capable MACs
In response to receipt of a Multicast to All-FCF-MACs from another FCF
VE_Port
Instantiates VE_Port to VE_Port connections and Exchanges FC

ELP (Extended Link Protocol) and Fabric configuration (using
Ethertype=FIP) with the other FCFs VE_Port capable MACs
58
ENode Actions
When an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACs
connected to the same Lossless Ethernet segment by:
Transmitting a multicast Solicitation to All-FCF-MACs (with the FCF bit
set to zero)
Receiving back Jumbo Unicast Advertisements from compatible VF_Port
capable FCF-MACs
(May store the discovered FCF-MACs in an FCF port list)
When an ENode receives an Advertisement that a new FCF

port is available, it may send a Unicast Solicitation to it and
receive a Jumbo Unicast Advertisement in reply
To verify the support of Ethernet Jumbo frames in the path
May then perform FLOGIs (with Ethertype=FIP) to a vendor

specific subset of the FCF-MACs in the FCF port list
59
FIP Keep Alive

FCoE connections may not be directly attached to the FCF
May flow through intermediate Lossless Ethernet switches,
Needs to be a method to detect that something was wrong in the path
Between VN_Ports and VF_Ports as well as between VE_Ports
Timers connected to the ENodes & FCFs can determined that a port is
sending messages
Need to determine if a port is just dormant, but still alive
Therefore, a periodic message needs to be sent.
The periodic unsolicited Advertisements from the FCF can be used for
Keep Alive in the following directions
FCF ENode
FCF FCF
But something else is needed for the ENode FCF direction

Because there is no periodic unsolicited Advertisement from a ENode
So -- a special FIP Keep Alive Message was created to just inform the
FCF that the ENode and its VN_Ports are still alive
60
FIP KAs (Keep Alives) & the FCoE Controllers

KA Timers for the Link
(Optional Periodic KA on behalf of
the VN_Ports to the VF_Ports:
default= 90 sec.)
KA Timers for the Link

(default every 8 Sec.)
FC Switching Element
FC-3 /FC-4s FC-3 /FC-4s FC-3 /FC-4s
ENode-MAC
VN_Port
VN_Port
VN_Port
FCoE_LEP
FCoE_LEP
FCoE_LEP
FCoE
Controller
FCF-MAC
FC
FCoE
FCoE
Controller
VF_Port
FCoE_LEP
FCoE_LEP
FCoE_LEP
Ethernet
Ethernet_Port
Ethernet_Port
ENode-MAC
FCF-MAC
VN_Port1
MAC
VN_Port2
MAC
VN_Port3
MAC
Lossless
Ethernet
Switch
FCF-MAC
FCF-MAC
The Periodic FCF Advertisements

Optional Periodic KA on behalf of
keeps forwarding state refreshed
the VN_Ports to the VF_Ports keeps
forwarding state refreshed
FCF-MAC
61
FIP Clear Virtual Link

When a port connection is determined by a side to be not functional
There is a requirement for the side to attempt to clear all state,
(Especially state which may have been established in any intermediate switches)
The VN_Port can issue a FIP Logout (LOGO)

But if the ENode is dead or its link busted, the FCF needs to clear things on
its own
But there is no FC capability for a FC switch to Logout a N_Port
So -- a special FIP function was created called Clear Virtual Link
Can be issued by the FCF
When intermediate switches see this frame they should cleanup their
ACLs etc.
62
FIP Descriptors

63
FIP Operation Codes

Operation Code
Subcode
0001h
01h
Discovery, Solicitation
02h
Discovery, Advertisement
01h
FLOGI/FDISC/LOGO/ELP, Request
02h
FLOGI/FDISC/LOGO/ELP, Reply
01h
FIP Keep Alive
02h
01h
FIP VLAN Request
02h
FIP VLAN Notification
FFF8h .. FFFEh
00h .. FFh
Vendor Specific
All others
All others
Reserved
0002h
0003h
0004h
Operation

64
Type and Values of FIP Descriptors

Type
Value
Type
Value
00
Reserved
09
Fabric Log Out
01
Priority
10 (0Ah)
ELP
02
MAC Address
11 (0Bh)
VN_Port ID
03
FC-MAP
12 (0Ch)
FKA_ADV_Period
04
Node_Name /
Switch_Name
13 (0Dh)
Vendor ID
05
Fabric
14 (0Eh)
VLAN
06
Max FCoE Size
15 (0Fh) 240 (F0h)
Reserved
07
FLOGI
241 (F1h) 254 (FEh)
Vendor Specific
08
NPIV FDISK
255 (FFh)
Reserved
(128-240 are Non Critical)

65
FIP Descriptors (1)
Type = 5
Reserved
Len = 4
VF_ID
FC_MAP
Fabric_Name

66
FIP Descriptors (2)
Len = 11/10/9

67
FIP Descriptors (3)
Type = 13
Len = 3
Reserved
Vendor_ID
Type = 14
Len = 1
Rsrvd
FCoE VID

68
VLAN Request & Notification
VLAN Request
ENode/FCF FCF
FIP Operation Code = 0004h
Descriptor List Length = 2-4
Type = 2
Reserved
VLAN Notification
FCF FCF/ENode
SubCode = 01h
FLAGS
Length = 2
Descriptor List Length = 2-n

Type = 2
Length = 3
SubCode = 02h
FLAGS
Length = 2
ENode/FCF-MAC Address
Type = 4
Reserved
FCF-MAC Address
Reserved
ENode/Switch Name (Optional)
Type = 14
Len = 1
Rsrvd
FCoE VID -1
Type = 14
Len = 1
Rsrvd
FCoE VID-n
F = 1b to indicate that the request is sent by a VE_Port capable

FCF-MAC

69
Discovery Solicitation & Advertisement

Solicitation ENode FCF
= 12
Type = 4
Solicitation FCF FCF

Reserved
Descriptor List Length = 8

Type = 2
SubCode = 01h
FLAGS
Type = 3
Length = 2
Reserved
Type = 4
FC-MAP
Length = 3
Reserved
VF_ID
FC_MAP
Fabric_Name
Type = 12
Reserved
Len = 4
Reserved
Length = 2
FCF-MAC Address
Reserved
Switch_Name
Type = 5
Len = 3
Len = 2
Reserved
FKA_ADV_Period
Padding to Max Receive Size of Soliciting Entity, if
Solicited (i.e. if S=1b, otherwise no padding
Switch Name
Type = 6
Length = 1
Max Receive Size
F = 1b to indicate that the Solicitation is sent by a VE_Port capable

FCF-MAC (i.e., it is soliciting VE_Port capable FCF-MACs)

70
FIP FLOGI
Request (ENODE FCF )
Reject (FCF ENODE)

Accept (FCF ENODE)

71
FIP NPIV FDISC
Reject (FCF ENODE)

Accept (FCF ENODE)

72
FIP Fabric LOGO
Reject (FCF ENODE)

Accept (FCF ENODE)

73
FIP ELP
Request (FCF FCF )
Reject (FCF FCF )

Accept (FCF FCF )

74
FIP Keep Alive
Padding to Ethernet minimum length
ENode FCF
(The Keep Alive from the FCF ENode/FCF is a normal unsolicited Advertisement)
75
FFFFFDh
( Fabric Controller
)
FFFFFDh
(Fabric Controller)
FCF FCF
FCF ENODE
(Note: the ENode uses FIP Fabric LOGO to Clear the Virtual Link from its side)
76
Vendor Specific FIP Message
Vendor Specific Operation Code =FFF8 -- FFFE with any SubCode

Vendor Specific Type (in Descriptor List) = F1h FEh
Reserved Types = 0Fh F0h, & FFh
77
Pause vs. BB_Credit

Both mechanisms are used to avoid dropping frames
With different trade-offs
The Pause mechanism requires at least the (2 x RTT x

bandwidth) product on a link as buffer space
But allows Buffer handling in an arbitrary way
Well suited for networks with limited (bandwidth x delay) product
(e.g. within the data center)
The Pause frame is handled by the MAC layer

Similar to the R_RDY handling by the FC-1 level
The BB_Credit mechanism prevents losing frames

over any link
But links go under-utilized if link credits (& buffers) are < that needed
for (RTT x BW)
Requires buffer handling in maximum frame size units
78
The Origins of a Data Center Fabric

2009 CEE/DCB
(Converged Enhanced
Ethernet/Data Center Bridging)
-- a Lossless Ethernet)
Increasing Performance
10Gbps
Integration of Carrier-grade features

1Gbps
Incremental Protocol Enhancements
100Mbps
Logical Partitioning
10Mbps
Evolution from shared media to dedicated media
Introduction of Ethernet
1973
Increasing Scalability, Feature, Function
The technology has evolved continuously, showing a great ability to adapt to new
technologies and increasing business requirements
79

JohnHufferd Fibre Channel Over Ethernet-FCoE

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

JohnHufferd Fibre Channel Over Ethernet-FCoE

Hochgeladen von

Copyright:

Verfügbare Formate

Fibre Channel over Ethernet (FCoE)

John L Hufferd, Consultant

SNIA Legal Notice

This presentation is a project of the SNIA Education Committee.

Topologies with FCFs

FCoE requires specific Ethernet extensions to be implemented

Deployments of FCoE should utilize the advances in Ethernet recently

These 802.1 advances are important for Converged Flows (Messaging,

FCoE Fabrics must be built with FCoE CEE/DCB Switches that:

Are called FC Forwarder FCF

Converged Enhanced Ethernet (CEE) &

CEE/DCB Ethernet Ports

Adapt between FCoE and FC

The FCoE Ports have F_Port or E_Port functions

Lossless Ethernet MAC (CEE/DCB)

Integrated Lossless Ethernet (CEE/DCB) Switch

FCoE is NOT a replacement for FCIP

Fibre Channel over Ethernet (FCoE)

The Compelling Value of Convergence

Converged Enhanced Ethernet (CEE/DCB)

Seamless connection to the installed base of existing SANs and LANs

But Even a Total Data Center (CEE/DCB) Fabric

Note: with multiple Data Centers

FCoE permits intermixing of multiple Connection types/protocols

Fibre Channel over Ethernet (FCoE)

FC Encapsulation Into Ethernet Frames

Fibre Channel over Ethernet (FCoE) Packets

FC Imbedded Frames: Same as in Physical FC

Ethernet Header provides things needed for the

FCoE Initialization Protocol (FIP) Packets

Discovery, Link establishment, maintenance & disconnect (Login/Logout, etc.) Parameters

FIP Protocol and FCoE Protocol

ENodes chose among discovered FCFs Ports

Allowed methods for the ENode MAC Addressing

Fabric Provided MAC Addresses (FPMA) Assigned by the Fabric

Uses: FLOGI, FLOGI ACC, LOGO, ELS,

End-to-End path control & Data Transfer Phase

Fibre Channel over Ethernet (FCoE)

FCs Encapsulation in Ethernet (FCoE)

Destination MAC Address (6 Bytes)

Source MAC Address (6 Bytes)

ET=FCoE (16 bits)

Fibre Channel over Ethernet (FCoE)

FIP Operation Format

Destination MAC Address (6 Bytes)

Source MAC Address (6 Bytes)

ET=FIP (16 bits)

FIP Operation Code

PAD to minimum length or mini-Jumbo length

Fibre Channel over Ethernet (FCoE)

HBA Model of the ENODE with

For each logical N_Port

The number of (logical) FC

Lossless Ethernet MAC

Only one MAC Address is required for the FCoE Controller

Multiple Logical FC connections

The Logical FC Link is defined by a MAC Address pair

Functions of an FCoE Converged Network

Has a Normal NIC interface

FCoE functions not seen by

Fibre Channel over Ethernet (FCoE)

CNA with Multiple Logical Interfaces

MAY have one burnt-in

MAY have different Burntin MACs for IP and