Fibre Channel

Or, why iSCSI is a BAD IDEA

Tom Keiser
tkeiser@cede.psu.edu

Penn State Linux Users Group

Fibre Channel p.1/60

Overview
Historical Background What is Fibre Channel? FC Hardware Topologies and Addressing The FC Protocol How fast is it? Why is it better than iSCSI?

Fibre Channel p.2/60

History of Storage Networks

Storage networks evolved for several reasons To abstract the complexity of storage technologies from the host system To allow multiple systems to share a disk cabinet or array They were a necessary step in the development of cluster lesystems for VMS, OS/390, etc. More recently, they have permitted data centers to place storage outside of metropolitan areas in specialized storage centers

Fibre Channel p.3/60

History of Storage Networks

DEC developed a precursor to the SAN for its VMS clusters using the DSSI and CI interconnects IBM developed SSA for its RS/6000 systems The SSA protocol stack is much simpler than FC The lowest layers of the SSA stack are a streamlined derivative of FC SSA is application-specic; FC is generic shared SCSI is common in small installations

Fibre Channel p.4/60

History of Fibre Channel

Architecture developed by Sun Microsystems as a part of Project StoreX Protocol developed by IBM IBM decided FC was too complex for many applications SSA was IBMs simplied solution Fibre originally came in several speeds ranging from 125Mbps to 1062.5Mbps Early bre installations were private loops

Fibre Channel p.5/60

Fibre Channel

The Basics

Fibre Channel p.6/60

What is Fibre Channel?

A high-performance serial storage network Current implementations run at 1 and 2 Gbps 4 and 8 Gbps implementations are planned Common Physical Cabling: Copper cabling with HSSDC, HSSDC2, or DB9 connectors Single-mode or multi-mode ber with SC or LC connectors

Fibre Channel p.7/60

What is Fibre Channel?

FC, like most major data link layer protocols, encapsulates common transport protocols SCSI IP VI (virtual interface protocol used in clusters) HIPPI ESCON

Fibre Channel p.8/60

FC Hardware Overview
Host Bus Adapters (HBAs) Interface PCI, SBUS, XIO64, etc. with FC Integrate SRAM frame buffers and RISC processors to ofoad framing, segmentation, etc. Hubs Make a logical ring topology physically appear as a star topology Automatically detect failed or unplugged devices and route the ring around them

Fibre Channel p.9/60

FC Hardware Overview
Gigabit Interface Connectors (GBICs) Abstract the technical details related to cable type and laser type from expensive I/O processors Hot-swappable Available in two form factors Legacy (GBIC) Small Form-factor Pluggable (SFP) Available in a variety of connector types Make transition to new cable technologies cheaper

Fibre Channel p.10/60

FC Hardware Overview
Switches Incorporate large buffers to ensure long-distance links for metropolitan-area SANs are not starved Handle load balancing and route determination across inter-switch links Provide fabric name services to aid in service location Include zoning capabilites to enforce security policies

Fibre Channel p.11/60

Hardware Manufacturers
HBAs Emulex, Qlogic, JNI, LSI Logic, Sun Microsystems Switches Brocade, McData, Ancor/Qlogic, Cisco GBICs IBM, Finisar, AMP, Stratos Lightwave, Methode Routers and Gateways Pathlight, ADIC, Cisco, Qlogic

Fibre Channel p.12/60

Fibre Channel

Network Addressing and Topology

Fibre Channel p.13/60

Fibre Network Logon Types

Private Simplistic protocol that uses Arbitrated Loop Physical Addressing (AL_PA) Runs on an arbitrated loop topology Public (Fabric Aware) Flexible protocol that runs on switched fabrics Uses fabric name services for target location Switching done on Hardware addresses (WWNs)

Fibre Channel p.14/60

Public Addressing
Every port has a 48-bit hardware address, called a World-Wide Port Name (WWPN) Each node in the system has a World-Wide Node Name (WWNN) Frames are routed based upon source and destination WWPN values in frame headers WWNNs help facilitate multipathing technologies by revealing an N:1 mapping

Fibre Channel p.15/60

Arbitrated Loops
Serial Ring Topology Loop Initialization Primitive (LIP) is used to establish and maintain loop An initiator must acquire the loop, and thus form a bidirectional symmetric logical link with another node on the ring Only one pair of nodes per loop can communicate at a time Stability can be disrupted by LIP storms Loop can be constructed using TX simplex cabling, or using hubs RX

Fibre Channel p.16/60

Switched Fabric
One or more switches interconnected by Inter-Switch Links (ISLs) Each port is congured by the administrator for a specic mode of operation Switches provide zones (similar to IEEE 802.1Q VLANs) to allow coexistence of multiple security domains Service discovery is aided by the fabric name services that run on every switch WWNs are used to address targets, so much larger networks are possible

Fibre Channel p.17/60

Common Port Types

L_Port Node Port Arbitrated Loop Mode N_Port Node Port Point-to-Point Mode FL_Port Switch Port Arbitrated Loop Mode

Fibre Channel p.18/60

Common Port Types

F_Port Switch Port Point-to-Point Mode Many older I/O processors do not support this (e.g. Qlogic ISP2100, LSIFC909) T_Port Qlogic-proprietary ISL Port E_Port ISL Port

Fibre Channel p.19/60

Common Port Types

TL_Port Qlogic Proprietary Private Switch Port Performs AL_PA to WWPN Address Translation SL_Port Qlogic Proprietary Private Switch Port Segments a Private Loop Allows one simultaneous I/O per segment, versus one per loop

Fibre Channel p.20/60

Port to Switch Topologies

Public Loop Topology Public Point-to-Point Topology

Node

Switch

Switch

NL_Port

FL_Port

F_Port

Node

Node

Node

NL_Port

NL_Port

N_Port

Fibre Channel p.21/60

Common Fabric Topologies

Degenerate Case Single-switch fabric Cascade Consists of a line or ring of switches Every switch has one or two adjacent switches Latency scales with , which is very bad Number of ISLs scales with , which is optimal Cascade is well suited to small fabrics

Fibre Channel p.22/60

Common Fabric Topologies

Cascade Fabric Topology

Switch 1

Switch 2

Switch 3

Fibre Channel p.23/60

Common Fabric Topologies

Mesh Switches and ISLs form a complete graph Graph diameter is constant, which makes latency constant (assuming all equipment operational, and no congestion) Number of ISLs scales with Worst-case ISL to usable port ratio Prohibitively expensive for large fabrics Highest performance Good for medium-sized fabrics

Fibre Channel p.24/60

Common Fabric Topologies

Mesh Fabric Topology

Switch 2

Switch 1

Switch 3

Switch 4

Fibre Channel p.25/60

Common Fabric Topologies

Multi-Stage Switches belong to one of two disjoint classes Edge switches contain links to nodes and core switches Core switches contain links to edge switches Topology forms a complete bipartite graph Every edge switch has a link to every core switch, and vice versa ISL count scales with

Fibre Channel p.26/60

Common Fabric Topologies

Mult-Stage Usable port count scales with , Where is the number of edge switches is the number of core switches is the port count of edge switch is the number of trunks between edge switch and core switch Latency is constant, but greater than the latency for a Mesh This topology is well suited to large fabrics
   

 

 

Fibre Channel p.27/60

Common Fabric Topologies

Multistage Fabric Topology
Core Switches

Switch 1

Switch 2

Edge Switches

Switch 3

Switch 4

Switch 5

Fibre Channel p.28/60

Fibre Channel

The Protocol

Fibre Channel p.29/60

FC Protocol Stack
FC0 Physical Layer Handled by GBIC electronics FC1 8b/10b encoding layer developed by IBM Necessary for receiver clock recovery Encoding tries to keep the ratio of 0s to 1s near 1 Limits same bit run length to 5 Detects single-bit errors

Fibre Channel p.30/60

FC Protocol Stack
FC1 268 of 1024 values are valid 256 data values 12 control values Start of Frame (SOF), End of Frame (EOF), Idle, Receiver Ready (R_RDY), etc. FC2 Framing, Segmentation, Reassembly Flow Control Layer

Fibre Channel p.31/60

FC Protocol Stack
FC3 FC Services Layer Facilitates broadcasting and multicasting Provides hunt groups, e.g. routing to a free port Provides striping across multiple ports Attempts to provide a topology-agnostic view of the network to anything above it in the stack FC4 Mapping to Upper Layer Protocol (ULP)

Fibre Channel p.32/60

The Messaging Hierarchy

FC is a frame-based protocol Frames are either 512, 1024, or 2048 bytes Frame size is negotiated during port login Practically everything uses 2048 byte frames An atomic message unit is called a Sequence Sequences consist of one or more frames A transaction is called an Exchange Exchanges consist of one or more sequences

Fibre Channel p.33/60

The Messaging Hierarchy

Fibre Channel p.34/60

Frame Types
Link Control Frame Provides for ow control Examples include ACK for class 1 or 2 frames Contains no payload Data Frame Provides for data transport services Examples include encapsulated protocol data Contains a payload of 02112 bytes

Fibre Channel p.35/60

Frame Contents
Start of Frame Sequence (4 bytes) Frame Header (24 bytes) Payload (02112 bytes) Cyclic Redundancy Checksum (4 bytes) End of Frame Sequence (4 bytes)

Fibre Channel p.36/60

FC Frame Header Layout

Offset 0 4 8 12 16 20 Routing Ctrl Reserved ULP Type Sequence ID Word Contents Destination WWN Source WWN Frame Control Data Field Ctrl Sequence Count Responder Exchange ID

Originator Exchange ID

Parameter

Fibre Channel p.37/60

Fibre Channel

Classes of Service and Flow Control

Fibre Channel p.38/60

Service Class 1
Stateful connection between two N_Ports Frames are received in-order ACK link control frames are used for ow control Full channel bandwidth is guaranteed End-to-end ow control is used If HBA supports Intermix mode, unrelated Class 2 and 3 frames can be interleaved with the class 1 stream

Fibre Channel p.39/60

Service Class 2
Connectionless Unordered delivery is possible Multi-mode ow control Buffer-to-buffer by fabric-generated R_RDY frames End-to-end by N_Ports sending ACKs Fabric may multiplex/demultiplex frames in transit Fabric attempts to notify originator of non-delivery

Fibre Channel p.40/60

Service Class 3
Connectionless Unordered delivery is possible Delivery is not guaranteed Buffer-to-buffer ow control via fabric-generated R_RDY frames No notication to originator of dropped frames Fabric may multiplex/demultiplex frames in transit

Fibre Channel p.41/60

Service Class 4
Not dened in ANSI FC-PH; classes 46 are discussed vaguely on the internet Stateful virtual circuit Frames are delivered in-order Delivery is guaranteed Provides a fractional-bandwidth virtual circuit Meant for isochronous data services No know implementations

Fibre Channel p.42/60

Flow Control
Each port has a pool of frame buffers Qlogic switches statically assign buffers to each port Thus, they have the lowest latency in the industry Long-distance ports can steal buffers from adjacent ports, at the expense of deactivating them Brocade switches have a global frame buffer pool

Fibre Channel p.43/60

Flow Control
Each frame buffer is referred to as a credit A port may send as many frames as it has available credits before receiving an ACK Thus, link distances are effectively limited by the number of available buffer credits at each end of the link

Fibre Channel p.44/60

Flow Control
End-to-End Flow Control Class 1 and Class 2 frames use this Implemented as an ACK frame A single ACK may acknowledge, one, multiple sequential, or all frames in a sequence Buffer-to-Buffer Flow Control Class 2 and Class 3 frames use this Implemented as an R_RDY frame One R_RDY is required per received frame

Fibre Channel p.45/60

Fibre Channel

The Practical Side

Fibre Channel p.46/60

How fast is it?

1Gbps is not very accurate Actual 1Gbps link speed is 1062.5 Mbaud 8b/10b encoding reduces this to 850 Mbps Assuming optimal 2012/2048 data ratio, we get:
       

(1)

Hence the advertised 100 MB/s for 1Gbps Fibre Channel

Fibre Channel p.47/60

Why is iSCSI bad technology?

Implementing SCSI over TCP is simply overkill The SCSI protocol does not lend itself to stateful transports, hence SCSI is an FC class 3 service SCSI has its own built-in error detection methodology, so a reliable transport layer is redundant There are simpler ways to get ow control than by implementing Reno or Vegas in Silicon

Fibre Channel p.48/60

Why is iSCSI bad technology?

A key issue with SANs is that they are physically separate from public networks, and thus are inherently more secure LANs are generally architected in a much more haphazard manner than SANs, thus leading to stability and robustness concerns Mixing bulk I/O trafc and time-sensitive IP-based trafc will lead to disputes over QoS settings

Fibre Channel p.49/60

Why is iSCSI bad technology?

Average MTU in IP networks is much lower Dense Wave Divison Multiplexing (DWDM) has made wide-areas SANs common Eliminates FC IP routers Allows for high-bandwidth remote backup and mirroring Doesnt produce large block-IO related strains on expensive infrastructure, such as IP routers Reduces latency by eliminating protocol translation and encapsulation

Fibre Channel p.50/60

Why is iSCSI bad technology?

The iSCSI protocol family is still very immature Because iSCSI HBAs will be attached to corporate ethernet backbones, there will be an extraordinarily greater chance of malicious trafc hitting the HBAs. The complexity of implementing a TCP stack in silicon coupled with the small amount of prior-work makes the probability of error high

Fibre Channel p.51/60

Customary Design Choices

FC fabric designs follow normal HA theory Multiple HBAs on separate buses are placed in each node Two identical, redundant fabrics are built so that a complete fabric failure will not affect operations FC disks are dual-port so they can be attached to both fabrics

Fibre Channel p.52/60

Examples
Bypass Card Bypass Card Bypass Card Loop A GBIC

Disk 2

Disk 1

Disk 0

Fibre JBOD

Bypass Card

Bypass Card

Bypass Card

Loop B GBIC

Fibre Channel p.53/60

Examples
Typical Redundant Fibre RAID System
Fabric A Fabric B

Host Loop A

Host Loop B

Controller A

High-Bandwidth, Low-Latency Cache Coherency Interconnect

Controller B

Disk Loop A

Disk Loop B

JBOD

JBOD

JBOD

JBOD

Fibre Channel p.54/60

Examples
Simple Fibre Channel Fabric
JBOD

Public Loop Using Simplex Cabling

Fibre Hub

Private Loop Using a Hub

Loop A

JBOD

Fabric Switch
Public RAID Controller(s)

Loop B Public Initiator Private JBOD Public JBOD JBOD

Fibre Channel p.55/60

Examples
Simple Redundant Fibre Channel Fabric
Public Initiator Public Initiator Private Initiator JBOD

Fabric Switch

Loop A

Public JBOD

JBOD

Fabric Switch
Private JBOD

Public RAID Controller(s)

Loop B JBOD

Fibre Channel p.56/60

Fibre Channel

Conclusions

Fibre Channel p.57/60

Conclusions
The FC protocol is nally reaching maturity Fibre Channel has become the standard SAN interconnect technology No replacement technologies are going to be available in the next few years iSCSI is only a replacement for FC in lower-end installations where performance and stability are less critical

Fibre Channel p.58/60

References
ANSI FC-PH Specication, Revision 4.3 LSI Logic LSIFC909 Architecture Manual Qlogic SAN Management Users Guide Qlogic Sanbox-16HA Users Guide
http://hsi.web.cern.ch/HSI/fcs/spec/ overview.html http://www.nersc.gov/jed/ Presentations/net-tutorial

Fibre Channel p.59/60

Fibre Channel

Because iSCSI is a BAD IDEA

(And thanks for coming to this talk)

A created using xemacs, visio, and LTEXwith the prosper class

Fibre Channel p.60/60