FCoE - Fibre Channel Over Ethernet

Fibre Channel over Ethernet (FCoE)
John L Hufferd, Brocade
SNIA Legal Notice

The material contained in this tutorial is
copyrighted by the SNIA.
Member companies and individuals may use this
material in presentations and literature under the
following conditions:
Any slide or slides used must be reproduced without
modification
The SNIA must be acknowledged as source of any
material used in the body of any document containing
material from these presentations.
This presentation is a project of the SNIA

Education Committee.
2008 Storage Networking Industry Association. All Rights Reserved.
Abstract
A new concept is currently moving through the
Fibre Channel (T11) standards committee called
Fire Channel over Ethernet (FCoE).
The FCoE standard will specify the
encapsulation of Fibre Channel frames into
Ethernet Frames and the amalgamation of these
technologies into a network fabric that can
support Fibre Channel protocols and other
protocols such as TCP/IP, UDP/IP etc.
The tutorial will show the Fundamentals of the
FCoE concept and describe how it might be
exploited in a Data Center environment.
Agenda
Introduction
Goals & Requirements
Consolidation
Architecture
Discovery & Link Instantiation
Topologies
Scenarios
Summary
Introduction
This presentation provides an overview of Fibre
Channel over Ethernet (FCoE)
One should think about FCoE as placing the FC
protocol on a new physical link
Ethernet links instead of physical FC links
But it is still Fibre Channel
The protocol is being defined in the INCITS Fibre

Channel (T11) technical committee
Many details of the protocol still need to be defined
but the significant major issues have been resolved
Target T11 Standards completion is the 2H08
The Origins of Data Center Ethernet

2008-2009 CEE
(Converged Enhanced Ethernet)
Increasing Performance
10Gbps
Integration of Carrier-grade features
1Gbps
Incremental Protocol Enhancements
100Mbps
Logical Partitioning
10Mbps
Evolution from shared media to dedicated media
Introduction of Ethernet
1973
Increasing Scalability, Feature, Function
The technology has evolved continuously, showing a great ability to

adapt to new technologies and increasing business requirements
Goals/Requirements

FCoE Goals/Requirements
(part 1)
FCoE Fabrics must be built with FCoE Switches

Switches with Ethernet ports that provide FCoE capabilities and
services Called, in the standard, an FC Forwarder FCF
Switches that include the functions of traditional FC switches
Standard Ethernet switches may also exist in the fabric but switches
with FCoE capabilities are required
FCoE fabrics must operate seamlessly with real FC Fabrics

FC services must operate identically on FCoE fabrics and
Fibre Channel fabrics
FCoE must support all Fibre Channel advanced features
(e.g. virtual fabrics, IFR, security, etc.) transparently
FCoE is NOT a replacement for FCIP or iFCP
FCIP & iFCP use TCP/IP
FCIP/iFCP is for inter-switch links beyond the Data Center
Integrated Ethernet & FCoE switch (FCF)

with FC connections
Some implementations may combine the features and capabilities of an Ethernet
Switch with the features and capabilities of a FC switch
Support Ethernet and IP standards for switching, pathing and routing
Support FC standards for switching, pathing and routing
Support current and enhanced Ethernet Standards
Adapt between FCoE and FC
FC
FC
FC
CEE Ethernet Ports

(with IP & FCoE VF_Port &
VE_Port capabilities)
An FCoE Port (N_Port, F_Port or E_Port)

Has the same function as in FC
But is layered on top of Ethernet
Called VN_Port, VF_Port and VE_Port
Because many logical (virtual)
Ports can share one physical port
Note:
FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol
Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as
STP (Spanning Tree Protocol) and MSTP (Multiple STP)
See additional FSPF and STP explanation in Appendix
Connections to a Combo FCoE Switch

Fibre Channel is carried over lossless Ethernet as a L3 protocol
Applications
UDP
SCSI
SCSI
iSCSI
Fibre Channel
TCP
IP
FCoE
Lossless Ethernet MAC (CEE)

IP address 123.45.67.89
(FCoE VN_Port)
Ethernet port with IP & FCoE VF_Port

capabilities
Combo Lossless Ethernet (CEE) Switch

with FCoE Switch (FCF) capabilities
10
(part 2)
FCoE requires specific Ethernet extensions to be implemented

Lossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE)
configurations are required
Jumboframe support is required (not a standard, but widely available)
Deployments of FCoE should utilize the advances in Ethernet

currently being discussed in IEEE 802.1, specifically:
Priority-based Flow Control (PFC)
Enhanced Transmission Selection (ETS)
DCB Capability eXchange Protocol (DCB CXP)
These 802.1 advances are important for Consolidated

Flows (Messaging, Clustering and Storage)
This set of functions has been called Data Center Ethernet, or CEE
Converged Enhanced Ethernet (intended for a Data Center
Environment)
FCoE should require no changes to FC software

11
Consolidation

12
High End 10GE Server & NIC/HBA

Consolidation
OS1
Web Server
With CEE
OS3
DB Server
OS2
App Server
OS2
App Server
FC
HBA
OS1
Web Server
Messaging
Ethernet
TCP/IP
FC SAN
E-HBA
(CEE)
MPI
RDMA
Today
IB/Ethernet
Cluster
OS2
App Server
OS3
DB Server
E-HBA
(CEE)
Hyper Visor
(vmWare, Xen, etc)
OS1
Web Server
NIC
TCP acceleration.
MPI, RDMA
over Ethernet
FCoE
Data Center
CEE Network
OS3
DB Server
E-HBA
(CEE)
Data Center
CEE Network
Dramatic reduction in adapter, switch ports and cabling

4-6 cables to 2 cables per server
Seamless connection to the installed based of existing SANs

and LANs
Requires high performance lightweight frame mappers vs.
heavy weight gateways
FCoE has no need to terminate and re-initiate a SCSI connection
(e.g. iSCSI to FC)
Effective sharing of high bandwidth links

13
Architecture

14
FC Encapsulation Into Network Packets

(2 FCoE Related Packet types)
FC-4
FC-4
FC-3
FC-3
FC-2
FC-2
FC-1
FCoE Mapping
MAC
PHY
FC-0
FC Levels
(Unchanged)
IEEE 802.3
Layers
Fibre Channel over Ethernet (FCoE) Packets

Ethernet
Header
Ethertype
FCoE
FCoE
Header
FC Header
SCSI Commands/Data
FCS
FC Imbedded Frames: Same as in Physical FC

Protocol control information: Version, SOF, EOF, etc.
Frame Check Sequence
(CRC)
Ethernet Header provides things needed for the

physical network, including Ethertype
FCoE Initialization Protocol (FIP) Packets

Ethernet
Header
Ethertype
FIP
FIP
Header
Descriptors
FCS
Discovery and Login/Logout Parameters

Protocol control information: Version, Op-codes, etc.
15
FIP Protocol and FCoE Protocol

Discovery Phase
FCFs Discovery each other and form a Fabric
ENodes and FCFs Discover
Potential VN_Port VF_Port pairing
Capabilities of Potential pairing
Login Phase
ENodes chose among discovered FCFs Ports
Creates association between ENode Ports and FCF Ports
VN_Port VF_Port Logical FC Link
FCoE
Initialization
Protocol (FIP)
Two allowed alternatives for the ENode MAC Addressing

Fabric Provided MAC Addresses (FPMA)
Server Provided MAC Addresses (SPMA)
Chosen by FCF (FPMA & SPMA Described latter)
Uses: FLOGI, FLOGI ACC, LOGO,
Data Transfer Phase

PLOGI/PRLI
All other FC protocol frames (ELS, FC4 ULPs. etc.)
FCoE Protocol
16
FIP Operation Format

Word
0
31-24
23-16
15-8
7-0
Destination MAC Address (6 Bytes)
1
Optional IEEE 802.1q
4 Byte
Tag goes here
See Appendix for
Descriptor list items
Descriptor list
varies
In size
2
3
4
Source MAC Address (6 Bytes)

Ver (4b)
ET=FIP (16 bits)
Reserved
FIP Operation Code

Reserved
FIP Operation
Code
Descriptor List
Reserved
FIPLength
subcode
Flags
(12 bits)
FIP SubCode
FP SP
Flags List Length
Descriptor
S F
F S S F
P P
Descriptor List
Ethernet frame
n+1
size
Is 64Bytes to 2220Bytes
PAD to minimum length or mini-Jumbo length
Ethernet FCS
Capability Bits (SPMA or FPMA)
Solicited bit
FCF bit
17
FCs Encapsulation in Ethernet (FCoE)

Word
31-24
23-16
15-8
7-0
Destination MAC Address (6 Bytes)
0
1
Optional IEEE 802.1q
4 Byte
Tag goes here
2
3
Source MAC Address (6 Bytes)
Reserved
Reserved
In size
n
n+1
Ethernet frame
n+2
size
Is 64Bytes to 2220Bytes
Reserved
Reserved
6
This field varies
Ver (4b)
ET=FCoE (16 bits)
(12 bits)
SOF (8 bits)
Encapsulated FC Frame
FC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)
(including FC-CRC)
EOF (8 bits)
Reserved
Ethernet FCS

18
FCF Model
Link
End
Point
(LEP)

19
ENode (HBA) Model
Link
End
Point
(LEP)
Each ENode (HBA) may have multiple Physical Ethernet Ports

Each Physical Port may have multiple Logical VN_Ports
20
Model of the ENODE with Multiple

Logical FC interfaces
Multiple FC NPIV
instances on a single
logical FC Host interface
For each logical N_Port

(VN_Port) there is one FLOGI
and perhaps 100s of FDISC
Each VN_Port is seen by
the Host as a separate
(logical) FC connection
The number of (logical) FC
connections is
implementation dependent
FC-3 /FC-4s
FC-3 /FC-4s
VN_Port
FC
Entity
VN_Port
FC
Entity
FCoE_LEP
FCoE
Entity
FCoE_LEP
FCoE
Entity
...
FCoE
Controller
Lossless Ethernet MAC
Ethernet_Port
MAC Address
of Burnt-in
MAC
In this
model this
is where
FC-2
functions
live
In this model this
is where the
Encapsulation
/De-Encapsulation
functions live
MAC Address of
FCoE_LEP
(VN_Port)
May or may not be
the same as the
FCoE controller
Only one MAC Address is required for the FCoE Controller and the VN_Ports on a
single physical MAC (aka Server Provided MAC Address SPMA)
FCF may chose to specify new MAC addresses for each VN_Port (aka Fabric
Provided MAC Address FPMA)
21
Multiple Logical FC connections

via a single Ethernet MAC
Examples of
single MACs
with
connections
to two
different
FCFs
Switch
The Logical FC Link is defined by a MAC Address pair

A VN_Port MAC Address
A VF_Port MAC Address
For a logical FC link the FCoE Frames are always sent to and received from a specific FCFs
MAC Address
Therefore, pathing to and from the FC driver is always defined by the MAC Address of the
partner FCFs VF_Port
22
Functions of an FCoE Initiator ASIC

A
B C
Host PCIe interface
FCoE
ASIC
External
Port
FCoE
function
NIC Function
Lossless
Ethernet
MAC
Has a Normal NIC interface

(A) to the Host
Has one or more Normal
FC interfaces (B,C) to the
Host
FC
Function
FCoE functions not seen by

the Host
FCoE functions perform the
Encapsulation and Deencapsulation
The FCoE function
Instantiates a Logical FC
N_Port, called a VN_Port

23
HBA with Multiple Logical FC Interfaces
The FCoE controller will perform the FIP functions and will instantiate new
VN_Ports as FCoE Link End Point (LEP)
With the same MAC address as the FCoE Controller (SPMA)
Or with a new MAC address specified by the FCF (FPMA)
A
FCoE Chip
MAC
Address of
Burnt-in
MAC
FCoE Function
FC-3 /FC-4s
FC
Entity
FCoE
Controller
VN_Port
FCoE
Entity
FC
Entity
FC-3 /FC-4s
FCoE_LEP
Etype=
FCoE
or
FIP?
VN_Port
FCoE
FCoE_LEP
Entity
NIC
MAY have one burnt-in

MAC address for both IP
and FCoE/FIP packets
Or
MAY have different Burntin MACs for IP and

FCoE/FIP packets
Ethernet_Port
Lossless Ethernet MAC
MAC
Address of
Burnt-in
MAC

Used to separate HW
based FCoE from other
Ethernet Traffic
Most NICs come with
several Burnt-in MAC
Addresses
24
FIP (FCoE Initiation Protocol)

Discovery and Link Instantiation

25
ENode to FCF Logical Links

An ENode must Locate FCFs with a Discovery protocol and then establish a Logical FC
Link with an FCF (VN_Port VF_Port) before an Normal FC frame flow
VF_Port
VN_Port
FCoE_LEPs
FCoE_LEP
MAC(H1)
FCF-MAC(A)
FCF A
VF_Port
H1
FCoE_LEPs
FC
fabric
VN_Port
FCF-MAC(B)
FCoE_LEP
VN_Port
FCoE_LEP
MAC(H2)
Lossless
Ethernet
Switch
VF_Port
FCoE_LEPs
FCF B
Note: The FCFs must

discover each other and
create a Fabric before the
ENode Discover FCFs
VF_Port
H2
FCoE_LEPs
VN_Port
VF_Port,
VF_Port_Name
FCoE_LEP
VN_Ports,
VN_Port_Names
Classical
Ethernet
LAN
FCF-MAC(x): A MAC address of a Lossless Ethernet port of FCF x

MAC(y): A MAC address of a Lossless Ethernet port of ENode y
26
Initial Login Flow ladder (2 Phases)

FCF
End-Node
Discovery
Multicas
to ALL
FCoE Initialization
Protocol
FCFs
Discovery
Phase
Unicast FCF MAC Address

et.al. in Jumbo Frame
FLOGI
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
FC Command
(Using the FCF
selected MAC
Address as the SA)
Login Phase
Normal FC
Processing
FC Command responses
(using the commands SA as a DA)
See Appendix for more details in Discovery Consideration and Actions
FCOE Protocol

27
Subsequent Login Flow ladder (1 Phase)

End-Node
FCF
FCoE Initialization
Protocol
FLOGI
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
Login Phase
FC Command
(Using the FCF
selected MAC
Address as the SA)
Normal FC
Processing
FC Command responses
(using the commands SA as a DA)
See Appendix for more details in Discovery Consideration and Actions
FCOE Protocol

28
Topologies

29
FCoE Switch & FC Fabric

Note: FCoE servers
and storage will
probably use an FCoE
HBA (or chip set)
FCoE Switch
An FCoE Switch may connect to a normal FC switch

Via the FC E-Port
FC
FC
Ethernet
FC
30
Multiple FCoE Switch (FCF) Topologies
FCoE Switch
Lossless
Ethernet
Fabric
FC
FC
A Lossless Ethernet Fabric can be made up of Combo FCoE Switches
Lossless Ethernet
Fabric
FC
FC
Lossless Ethernet switches configured into a Lossless Ethernet

Fabric can Front the FCoE Switch
Lossless
Ethernet
3
FCoE
Switch
FCoE
Switch
FCoE
Switch
FCoE
Switch
Lossless Ethernet
FC
FC
Lossless
Ethernet
1
Lossless
Ethernet
2
FCoE
Switch
FC
FC
FCoE
Switch
Lossless
Ethernet
4
FCoE Switches deployed at the edges of the Lossless Ethernet Fabric

FCoE Switches connected via VE_Ports and Lossless Ethernet
A VE_Port in an FCF may connect to other VE_Port in another FCF

And an FCF FC E_Port may connect to an FC switch E_Port

Ethernet
FC
31
Status of Current Data Center

Networking
Data Center
Server & Storage Network
Management Group
Outfacing (IP) Network

Management Group
There can be 3 different

networks
System Area Network (SyAN)

Clustering Fabric
InfiniBand
Myrinet
Ethernet
CTC
System Area Network (SyAN)

Used for Clustering/Low Latency
Storage area Network (SAN)

Used to Access to Storage
Remote Offices
LAN/WAN
Data Center
Messaging
NAS
Storage Area Network
LAN/WAN External Networks

Used for General messaging
Used for Client-Server Messaging
Used for NAS
Often divided into at least 3

management domains
Data Center Server (clustering)
Network
Data Center Storage Network
- Fibre Channel
LAN/WANs
Note: with multiple Data Centers there may also
be interconnects with DWDM, FCIP/iFCP, etc.
Messaging
NAS

32
Current Fabrics
Remote
Offices
Clustering Network
iSCSI
Storage
Mainframe
Focus: Low
Latency & High
Bandwidth
Ficon
FC
Outfacing
IP Network
FC
Network
FC
FC
(LAN/WAN)
Messaging
NAS
Ficon
Local & Remote

Business Campus
Ficon Storage
Controller
FC Link
Clustering Network
File Storage
Arrays (NAS)
Data Center Server & Storage Network Management Group
Focus: Protection,
Bandwidth/Congestion
Management
Ficon
Link
Ethernet
Link
IP Network Management Group
There is a FC Storage Network, a Clustering Network & an Outfacing Network

Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.

33
View of the Future Data Center

Networks
Data Center Server &
Storage Network
Management Group

Management Group
The Server/Storage Networks

will become a Consolidated
Fabric
Remote
Offices
Storage & System

Area Network
Logically Single Fabric
(with FCoE)
Other IP Nets
(LAN/WAN)
Messaging
NAS
Managed by the Data Center

System and Storage
Management Group
Includes Storage and
clustering provisioning
SAN and SyAN managed
as a single fabric
Focus: High Bandwidth and
Low Latency
The Outfacing (IP) Management

Group remains the same
Note: with multiple Data Centers there may also
be interconnects with DWDM, FCIP/iFCP, etc.
Focus: Protection, Bandwidth

and Congestion Management
34
Phase in: FC & High Performance Lossless

Ethernet Unify into a Data Center Fabric
Mainframe
Remote
Offices
Ficon
iSCSI
Storage
Ethernet
SW
FC
FC & Ethernet
SW Blades
FC
Outfacing
IP Network
DataCenter
Fabric
(LAN/WAN)
Including iSCSI
Gateways
FCoE
Messaging
NAS
Ficon
FC
Ficon Storage
Controller
Ethernet
SW
File Storage
Arrays (NAS)
Business
Campus
with iSCSI
connections
Note: with multiple Data Centers

there may also be interconnects
with DWDM, FCIP/iFCP, etc.
FCoE permits intermixing of multiple Connection types/protocols

Clustering messaging, General Messaging, and Storage
The DataCenter Fabric will Trunk to the Outfacing Network (including iSCSI sys)
Some Customers may want keep a mixed environment on-going
FC Link
Ficon
Link
Ethernet
Link
35
Scenarios

36
Scenario 1: FCoE & IP Flows
Internet
Classical
Ethernet
Lossless
Ethernet
FCoE
Switches
FC
FC
Lossless
Ethernet
FCoE
Switch
FCoE Flows
IP Flows
37
Scenario 2: FCoE Right & Wrong
Internet
Classical
Ethernet
Lossless
Ethernet
FCoE
Switches
FC
FC
Lossless
Ethernet
FCoE
Switch
FCoE Flows
Invalid FCoE Flows
38
Summary

39
Summary
FCoE is a simple, efficient mechanism for
encapsulating Fibre Channel in Ethernet frames
FCoE is being standardized in INCITS Fibre
Channel (T11) technical committee
Target completion is 2H08
Maximum benefit of Fibre Channel is achieved:

Evolutionary model of FC Switches and FC SANs
Emphasis placed on capitalizing on the benefits of
Converged Enhanced Ethernet (CEE)
Being discussed in the IEEE 802.1 standards working group
40
Thank You!
Q&A / Feedback
Please send any questions or comments on this
presentation to SNIA: tracknetworking@snia.org
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti
Robert Snively
Joe Pelissier
Howard Goldstein
Suresh Vobbilisetty
John Hufferd
Walter Dey
Silvano Gai
For additional information refer to

http://www.t11.org/fcoe
42
Appendix
Additional
Requirements
FCoE Relation to ISO
Layers
Flows
Additional Topologies
FSPF and STP
FIP Considerations
and Actions
FIP Descriptors
Pause vs. BB_Credit
43
FCoE Goals/Requirements (part 3)

FCoE must be a direct mapping of Fibre Channel over
an Ethernet network
FCoE must be layered on top of Ethernet
FSPF used to route FCoE packets
Ethernet Spanning Tree (STP), MSTP, etc, is at a layer below
FCoE to allow an evolutionary approach towards

consolidation of fabrics
The Fibre Channel N_Port, F_Port and E-Port constructs must
be retained
With FCoE, ports may be connected with Logical Ethernet Links
May pass through Ethernet switches
Identified by pairs of end point MAC addresses
Physical Ethernet Links can replace physical FC Links

Physical Ethernet Links can carry all Ethernet traffic, including
FCoE, but combined traffic needs the CEE capabilities
44
(part 4)
Combo FCoE Switches may be built that support

normal Ethernet traffic, FCoE traffic , & FC traffic
The FCoE solutions should appear as a Fibre Channel to
a Fibre Channel experienced customer
FCoE should keep the Fibre Channel operations
independent from Ethernet forwarding
Keeps management /Troubleshooting simple
Common physical structures, different logical structures
Based on Ethertype (Ethertype = FCoE)
Storage Management should be unchanged

45
FCoE Relation to ISO Layers

46
Flows

47
Logical Fabric Topology

An FCoE Switch receives FCoE frames addressed to its FC-MAC address and
forwards them based on the D_ID of the encapsulated FC frame
Logical
Transaction Path
Lossless
Ethernet
FCoE
Switch
Ethernet
FC
H2
FCoE
Switch
#2
#1
Lossless
Ethernet
Lossless
Ethernet
#3
#4
FCoE
Switch
H4
H3
H1
FC
FC
S2
Lossless
Ethernet
S1
An FCoE Switch rewrites the SA and DA of an FCoE frame

Ethernet
Destination
& Source
Encapsulated
FC Frame
D_ID
S_ID
Path #1
FCoE-A MAC
FCoE-H2 MAC
FC_ID for S1
FC_ID for H2
Path #2
FCoE-B MAC
FCoE-A MAC
FC_ID for S1
FC_ID for H2
Path #3
FCoE-C MAC
FCoE-B MAC
FC_ID for S1
FC_ID for H2

Path #4
FCoE-S1 MAC
FCoE-C MAC
FC_ID for S1
FC_ID for H2
48
Single Ethernet Fabric with FCoE Switches

H3
H2
Ethernet
Switch
Ethernet
Switch
#1
FCoE A
Switch
FCoE
Switch
Ethernet
Switch
Ethernet
Switch
#2
Lossless Ethernet
C
#3
Ethernet
Switch
S2
S1
H1
Ethernet
Destination
& Source
FC
FC
Path #1
Path #2
FCoE-A MAC
FCoE-H2 MAC
FCoE-C MAC
FCoE-A MAC
Path #3
FCoE-S1 MAC
FCoE-C MAC
Encapsulated
FC Frame
D_ID
FC_ID for S1
FC_ID for S1
FC_ID for S1
S_ID
FC_ID for H2
FC_ID for H2
FC_ID for H2

Logical
Transaction Path
Ethernet
FC
49
FC Host to FCoE Storage

H3
H5
S2
H2
Ethernet
Switch
Ethernet
Switch
#3
FCoE
Switch
Ethernet
Switch
FCoE
A
Switch
FC
FC
#2
Ethernet
Switch
Lossless Ethernet
S3
Ethernet
Switch
S1
H1
Ethernet
Destination
& Source
#1
Path #3
Path #2
FCoE-S2 MAC
FCoE-A MAC
FCoE-A MAC
FCoE-C MAC
Path #1
Encapsulated
FC Frame
D_ID
FC_ID for S2
FC_ID for S2
FC_ID for S2
S_ID
FC_ID for H5
FC_ID for H5
FC_ID for H5

Logical
Transaction Path
Ethernet
FC
50

51
FCoE
Switch (A)
(1)
Lossless
Ethernet
Switch
FCoE
Switch (E)
FC
FC
FCoE
Switch (B)
Lossless
Ethernet
Switch
FCoE
Switch (F)
Example of Topologies with Rack Mount servers

52
Lossless
Ethernet
Switch
(2)
FCoE
Switch (A)
FCoE
Switch (E)
FC
FC
Lossless
Ethernet
Switch
FCoE
Switch (B)
FCoE
Switch (F)
Example of Topologies with Blade Servers

Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) Support
53
FSPF and STP

54
FSPF & STP Concepts with FCoE (basic)

55
FSPF & STP Concepts with FCoE (interconnected)

56
Equivalent FC topology

57
FIP Considerations and Actions

58
FCoE Discovery Considerations

The Discovery phase of the FCoE Initialization Protocol (FIP)
uses two types of messages, Solicitations and Advertisements
The FIP Discovery Phase helps define the FCF Ports that are
available for the Link instantiation Phase
The ENodes discover the FCF ports that can become VF_Ports
and FCFs discover other FCF ports that can become VE_Ports
ENodes Solicit (via Multicast) Advertisements from FCFs while
specifying their capabilities
In response FCFs Advertise their availability and capabilities back
to the ENodes
FCFs Multicast their existent to other FCF
The FIP Discovery phase exchanges solicitation and/or

Advertisements between (HBA and/or FCF) FCoE Controllers

59
FCF Discovery Actions

FCF supporting VE_Ports:
Discovers other VE_Port capable FCF-MACs, connected
to the same Lossless Ethernet segment, by:
Transmitting a multicast Solicitation to All-FCFs
(with the FCF bit set to one)
Receives back Jumbo Unicast Advertisements from VE capable

MACs
To verify the support of Ethernet Jumbo frames in the path
In response to receipt of a Multicast to All-FCFs from another

FCF
Has its own VE capable MACs sends their own Jumbo Unicast
Advertisements
Instantiates VE_Port to VE_Port connections and

Exchanges FC ELP (Extended Link Protocol) and Fabric
configuration (using Ethertype=FCoE) with the other
FCFs VE_Port capable MACs
60
ENode Actions
When an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACs
connected to the same Lossless Ethernet segment
Transmits a multicast Solicitation to All-FCFs (with the FCF bit set
to zero)
Receives Jumbo unicast Advertisements from select compatible
VF_Port capable FCF-MACs
Stores the discovered FCF-MACs in an FCF list
When an ENode receives an Advertisement that a

new FCF is available, it may send a unicast
Solicitation to it and receive a Jumbo unicast
Advertisement in reply
To verify the support of Ethernet Jumbo frames in the path
May then perform FLOGIs (with Ethertype-FIP) to a

vendor specific subset of the FCF-MACs in the FCF
list
61
FIP Descriptors

62
FIP Descriptors (1)

63
FIP Descriptors (2)

64
Pause vs. BB_Credit

Both mechanisms are used to avoid dropping frames
With different trade-offs
The Pause mechanism requires at least the (2 x RTT x

bandwidth) product on a link as buffer space
But allows Buffer handling in an arbitrary way
Well suited for networks with limited (bandwidth x delay) product
(e.g. within the data center)
The Pause frame is handled by the MAC layer

Similar to the R_RDY handling by the FC-1 level
The BB_Credit mechanism prevents loosing frames

over any link
But links go under-utilized if link credits (& buffers) are < that
needed for (RTT x BW)
Requires buffer handling in maximum frame size units
65

FCoE - Fibre Channel Over Ethernet

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

FCoE - Fibre Channel Over Ethernet

Hochgeladen von

Copyright:

Verfügbare Formate

Fibre Channel over Ethernet (FCoE)

John L Hufferd, Brocade

SNIA Legal Notice

This presentation is a project of the SNIA

The protocol is being defined in the INCITS Fibre

The Origins of Data Center Ethernet

Increasing Scalability, Feature, Function

The technology has evolved continuously, showing a great ability to

Fibre Channel over Ethernet (FCoE)

FCoE Fabrics must be built with FCoE Switches

FCoE fabrics must operate seamlessly with real FC Fabrics

Integrated Ethernet & FCoE switch (FCF)

CEE Ethernet Ports

An FCoE Port (N_Port, F_Port or E_Port)

Connections to a Combo FCoE Switch

Lossless Ethernet MAC (CEE)

Ethernet port with IP & FCoE VF_Port

Combo Lossless Ethernet (CEE) Switch

FCoE requires specific Ethernet extensions to be implemented

Deployments of FCoE should utilize the advances in Ethernet

These 802.1 advances are important for Consolidated

FCoE should require no changes to FC software

Fibre Channel over Ethernet (FCoE)

High End 10GE Server & NIC/HBA

Dramatic reduction in adapter, switch ports and cabling

Seamless connection to the installed based of existing SANs

Effective sharing of high bandwidth links

Fibre Channel over Ethernet (FCoE)

FC Encapsulation Into Network Packets

Fibre Channel over Ethernet (FCoE) Packets

FC Imbedded Frames: Same as in Physical FC

Ethernet Header provides things needed for the

FCoE Initialization Protocol (FIP) Packets

Discovery and Login/Logout Parameters

FIP Protocol and FCoE Protocol

Two allowed alternatives for the ENode MAC Addressing

Uses: FLOGI, FLOGI ACC, LOGO,

Data Transfer Phase

FIP Operation Format

Destination MAC Address (6 Bytes)

Source MAC Address (6 Bytes)

ET=FIP (16 bits)

FIP Operation Code

PAD to minimum length or mini-Jumbo length

FCs Encapsulation in Ethernet (FCoE)

Destination MAC Address (6 Bytes)

Source MAC Address (6 Bytes)

ET=FCoE (16 bits)

Fibre Channel over Ethernet (FCoE)

Fibre Channel over Ethernet (FCoE)

ENode (HBA) Model

Each ENode (HBA) may have multiple Physical Ethernet Ports

Model of the ENODE with Multiple

For each logical N_Port

Lossless Ethernet MAC

Multiple Logical FC connections

The Logical FC Link is defined by a MAC Address pair

Functions of an FCoE Initiator ASIC

Has a Normal NIC interface

FCoE functions not seen by

Fibre Channel over Ethernet (FCoE)

HBA with Multiple Logical FC Interfaces

MAY have one burnt-in

MAY have different Burntin MACs for IP and