Designing A Routed Fibre Channel Storage Area Network

STORAGE AREA NETWORK
Architectural Brief:
Designing a Routed Fibre Channel
Storage Area Network
This paper explains a new feature called Integrated Routing enabled on
select 8 Gbit/sec platforms with Brocade Fabric OS 6.1.0. It provides
reference architectures and guidance for designing Storage Area
Network (SAN) solutions using Brocade Fibre Channel Routing (FCR)
technology.
Architectural Brief
CONTENTS
Introduction.........................................................................................................................................................................................................................................3
FC Routing Product Overview.......................................................................................................................................................................................................4
Brocade DCX and FR4-18i with Integrated Routing..................................................................................................... 4
Brocade 5300 and 5100 Switches with Integrated Routing ...................................................................................... 5
Brocade 48000 Director with FR4-18i Blade............................................................................................................... 5
Brocade 7500 Extension Switch................................................................................................................................... 5
SAN Routing Overview....................................................................................................................................................................................................................6
Edge Fabrics................................................................................................................................................................... 6
EX_Ports and FIDs ......................................................................................................................................................... 7
Exporting and LSANs ..................................................................................................................................................... 8
Backbone Fabrics ........................................................................................................................................................10
FC-NAT and Logical Domains ......................................................................................................................................11
Terminology Review .....................................................................................................................................................13
SAN Routing Use Cases ..............................................................................................................................................................................................................14
Small-Scale Use Cases ................................................................................................................................................14
Large-Scale use Cases ................................................................................................................................................16
Edge-to-Edge and Peer-to-Peer Design ...............................................................................................................17
Centralized Resources .........................................................................................................................................18
Multi-Site DR Solution ..........................................................................................................................................19
General Design Considerations................................................................................................................................................................................................20
HA Design .....................................................................................................................................................................20
No fabric redundancy with one FCR platform.....................................................................................................21
No fabric redundancy with two or more FCR platforms .....................................................................................21
Dual-redundant fabrics each with one FCR platform .........................................................................................21
Dual-redundant fabrics each with two or more FCR platforms..........................................................................21
Interoperability .............................................................................................................................................................22
Scalability .....................................................................................................................................................................22
General Nature of Scalability Limits....................................................................................................................22
Fault Containment................................................................................................................................................23
Fabric Services .....................................................................................................................................................23
Allowable Number of Edge Fabrics......................................................................................................................24
Per-Chassis Scalability .........................................................................................................................................24
Complex Topologies.....................................................................................................................................................25
FICON............................................................................................................................................................................25
Summary .........................................................................................................................................................................................................................................25
Designing a Routed Fibre Channel Storage Area Network
2 of 25
Architectural Brief
INTRODUCTION
Storage Area Networks are networks primarily intended to provide connectivity between hosts and storage
devices such as disks, RAID (Redundant Array of Inexpensive Disks) arrays, and tape libraries. SANs, unlike
traditional Local Area Networks (LANs), simply cannot lose data in transport. Applications and operating
systems are generally tolerant of dropped packets and variable performance on the LAN, because software
that relies on the LAN was built with the understanding that LANs sometime become congested and drop
data. Storage was originally directly attached and for these same software components the assumptions
were storage would be fast and reliable due to the use of a point to point connection and not a network.
Because of the more stringent performance and reliability requirements inherent in SANs, the industry
as a whole decided more than a decade ago to standardize on Fibre Channel (FC) as a SAN transport. As a
practical matter, it therefore would be more accurate to define a SAN as a high-performance, low-latency,
high-reliability, and lossless network, designed primarily to enable block-level host-to-storage (initiator-totarget) connectivity, most often consisting of FC devices (switch, director, backbone), Host Bus Adaptors
(HBAs), and storage devices.
As organizations continued to expand they required more comprehensive solutions than traditional FC
fabrics were providing. To address these needs, Brocade developed routing services, which increase SAN
functionality, scalability, and versatility. When a set of traditional FC switches are interconnected, they form
one network segment called a fabric, with a single name server space, one zoning realm, one Fabric
Shortest Path First (FSPF) database, one PID (Port ID) format, one routing table, one set of timeout values,
and so on.
An FC router enables a two-tier hierarchy of fabrics and solves many of the problems associated with nonrouted Layer 2 networks. An FC router is to an FC fabric as an IP (Internet Protocol) router is to an Ethernet
segment. Brocade Fibre Channel Routing (FCR) enables a hierarchical network design for Fibre Channel
that allows SANs to scale, isolate, and make fabrics more manageable.
NOTE: This paper discusses topics that are treated in greater detail in the Fabric OS Administrators Guide
for version 6.1. It is recommended that you consult that product manual when you are ready to deploy a
routing solution.
Where have all the SAN islands gone?

SAN islands were omnipresent during the early years of Fibre Channel. It was simply that FC networks were
just starting out and they tended to be small in size. An enterprise working to deploy a new application
would purchase the equipment for that application, including the array and switches, forming a new SAN
island. This occurred over and over until the enterprise was faced with the issues of having far too many
assets to manage and terrible inefficiencies. This resulted in efforts to consolidate into larger arrays and
directors. Consolidation continues to this day and now includes newer server consolidation technologies.
Brocade FC routing products meet deep, real-world customer needs in ways that are optimized for the
demands of mission-critical storage traffic.
This paper is for IT professionals who are considering a routed SAN solution or who want to broaden their
knowledge of SAN routing technologies. It covers the following topics:
Overview of Brocade FCR products
Discussion of the architecture of the routing service
Specific examples of routed SAN solutions
Design considerations
3 of 25
Architectural Brief
FC ROUTING PRODUCT OVERVIEW

Brocade offers FCR as a base feature on the Brocade 7500 Extension Switch and the Brocade FR4-18i
Extension Blade, with no additional licensing required. New Brocade ASICs, which enable 8 Gbit/sec port
speeds, support a new capability, Integrated Routing (IR), on all 8 Gibt/sec blades (FC8-16,
FC8-32, and FC8-48) and certain 8 Gbit/sec switches with the release of Fabric OS (FOS) 6.1. Integrated
Routing offers the same capabilities as FCR, except that it is user-configurable per port on FC8 blades in the
Brocade DCX Backbone and on Brocade 5300 and 5100 Switches. (The Brocade 300 and embedded
8 Gbit/sec switch modules do not offer IR.)
Brocade DCX and FR4-18i with Integrated Routing

The Brocade DCX Backbone, shown in Figure 1, is a revolutionary new product offering massive
performance, scalability, and the flexibility to enable mission-critical production deployments to grow and
evolve both physically and virtually. The internal switching fabric supports 4 Tbit/sec (4096 Gbit/sec)
aggregate bandwidth across the backplane and supports full-performance local switching. The Brocade DCX
running FOS 6.1 supports up to eight FC8 hot-swappable blades, which all optionally support IR.
The Brocade FR4-18i has 16 FC ports that support U_Ports and EX_Ports, plus 2 Gigabit Ethernet (GbE)
ports, each of which supports up to 8 FCIP (FC over IP) tunnels. The FC ports can be used for attachment of
end devices, that is, F_Ports to N_Ports, for the connection of Inter-Switch Links (ISLs) via two E_Ports, or
FC-to-FC routing for Inter-Fabric Links (IFLs) via one EX_Ports and one E_Port.
Integrated Routing is enabled with the release of Fabric OS 6.1. The FC routing protocol on the Brocade
7500/FR4-18i (EX_Ports) and IR ports (EX_Ports) is the same on each, so they are fully interoperable when
connected to the same fabrics. As of FOS 6.1, IR can be activated on up to 128 ports per Brocade DCX
chassis. When there are two Brocade DCX chassis connected via Inter-Chassis Links (ICLs), a total of 256
IR ports can be available. If a Brocade FR4-18i is installed in a Brocade DCX and an IR EX_Port has been
enabled, EX_Ports on the Brocade FR4-18i cannot be enabled at the same time and vice-versa. If FCR is
implemented without FCIP, IR is preferred on the Brocade FR4-18i, due to the greater flexibility and port
speed (8 Gbit/sec vs. 4 Gbit/sec) of IR.
NOTE: FCIP using the Brocade FR4-18i is not precluded with IR, and VEX_Ports and VE_Ports continue to be
fully supported.
As of Fabric OS 6.1, up to 4 Brocade FR4-18i FC Extension Blades can be installed into the Brocade DCX.
This blade turns the chassis into a fully-integrated FC switching and multiprotocol routing platform. The DCX
can be fitted with other blades and supports up to 352 ports with one Brocade FR4-18i.
Figure 1. Port side of a populated Brocade DCX Backbone with Integrated Routing
4 of 25
Architectural Brief
Brocade 5300 and 5100 Switches with Integrated Routing

Starting with Brocade FOS 6.1.0, the Brocade 5300 and 5100 Switches also support IR. No additional
hardware is required; but a software license key is required to activate this feature. All of the ports on the
Brocade 5300 (up to 80 ports) and Brocade 5100 (up to 40 ports) can be IR enabled via user configuration.
Figure 2. Brocade 5300 (top) and Brocade 5100 (bottom), both 8 Gbit/sec with IR capabilities
Brocade 48000 Director with FR4-18i Blade

The Brocade 48000 Director, shown in Figure 3, is a widely adopted, high-performance, high-reliability, and
very power-efficient director. As of FOS 6.1, up to two Brocade FR4-18i blades can be integrated into the
Brocade 48000. The Brocade 48000 can be fitted with other blades and supports up to 352 ports when
one FR4-18i has been installed. Brocade 48000 supports 1 Tbit/sec (1,024 Gbit/sec) aggregate bandwidth
across the backplane.
Figure 3. Brocade 48000 Director and FR4-18i Extension Blade and FCIP gateway
Brocade 7500 Extension Switch

The Brocade 7500, shown in Figure 4, is a fixed-configuration, 16-port 4 Gbit/sec FC switch with 2 GbE
ports for FCIP connectivity. All the ports have the same capabilities as the FR4-18i. In addition to being
qualified to perform typical FC switching functions, the Brocade 7500 can route FC across all 16 ports and
across the GbE ports over FCIP tunnels. The 16 FC ports auto-negotiate 1, 2, or 4 Gbit/sec speeds. The
internal architecture is non-blocking and supports up to 128 Gbit/sec of aggregate internal bandwidth.
Extended fabrics using FCR can also benefit from FC FastWrite (FC-FW), which is an acceleration technique
for SCSI writes. Contact your Brocade Systems Engineer (SE) for additional FC-FW details.
5 of 25
Architectural Brief
Figure 4. Brocade 7500 Extension Switch and FCIP gateway
SAN ROUTING OVERVIEW

Brocade FCR provides connectivity between two or more fabrics without merging them. In other words, the
fabrics maintain separate fabric-wide services and the switches in each fabric do not communicate with
switches in other fabrics. The router allows for the creation of Logical Storage Area Network (LSAN) zones,
which provide node connectivity that spans these unmerged fabrics. Because a router can connect nodes in
autonomous fabrics without merging the fabrics, it offers significant advantages for change management,
network management, scalability, interoperability, reliability, availability, and serviceability.
An FC router is like a switching firewall or a Layer 3 switch with hardware-enforced access control. FC
routers provide selective data connectivity via LSAN zones and prevent fabric services from propagating
between fabrics. Thus, disruptions to a fabric are isolated and other fabrics are not adversely impacted.
Another benefit of isolating services is that the FC router feature allows each fabric to maintain independent
administration. This prevents administrator errors from propagating between fabrics; it also isolates
hardware and software failures.
NOTE: In the paper, the terms FC router and router are both used to describe a Brocade platform with
routing enabled.
Router advantages, none of which would be true for a single large fabric, include:
The scaling of a routed fabric has no affect on the other routed fabrics.
If routed SANs are constructed with traditional switches, the overall network size can vastly exceed
that of a conventional, non-routed fabric.
Fabric reconfigurations do not propagate between edge fabrics.
Faults in fabric services are contained.
Problems caused by an errant HBA are contained.
Interoperability between different SAN operating systems can be achieved.
Zoning errors do not propagate.
Edge Fabrics
A fabric is formed by a standalone FC switch or by FC switches that are connected directly to each other via
ISLs. When fabrics are connected through the intermediary of a router, they are called edge fabrics. SAN is
the term used to refer to one or more fabrics, routed or not routed, that connect a common set of nodes.
Most enterprises implement a SAN with two (referred to as dual) redundant disk fabrics, shown in Figure
5 as Fabric 1 and Fabric 2. Each of the redundant networks could include multiple edge fabrics. Figure 5
illustrates two generic edge fabrics comprising two redundant FC routers. (In the real world, the number of
edge fabrics is usually greater than two.)
6 of 25
Architectural Brief
Figure 5. Generic routed fabric architecture
EX_Ports and FIDs

Switches in a fabric interconnect using E_Ports and the resulting connection is an Inter-Switch Link.
Similarly, FC routers connect to edge fabrics using EX_Ports. An EX_Port is a demarcation point that marks
the end of an autonomous fabric. Such a fabric can be attached to one or more routers. An E_Port (fabric
side)-to-EX_Port (router side) connection is an Inter-Fabric Link.
EX_Ports are like normal E_Ports from the perspective of the edge fabric and the protocols that run across
them are typical of all Brocade E_Ports. The distinction is that EX_Ports limit what each edge fabric sees in
terms of devices and services from other edge fabrics. This is accomplished by the EX_Port appearing
to the edge fabric like a regular FC switch (called a Front Domain) and proxy devices that use
Fibre Channel Network Address Translation (FC-NAT). Services in the edge fabric cannot extend
beyond the logical domains created for the purpose of representing proxy devices.
To differentiate between edge fabrics, each EX_Port has a user-configured Fabric Identifier (FID), which
specifies a unique label for the fabric to which it is attached. An edge fabric can be thought of as being the
FID x fabric in much the same way that an IP subnet can be though of in terms of its subnet address.
EX_Ports must have the same FID if they are attached to the same edge fabric or the IFLs will partition.
In the example above, all EX_Ports on both routers connected to Fabric 1 have FID=1 set. There are four
FID=1 EX_Ports across the two routers and four FID=2 EX_Ports.
7 of 25
Architectural Brief
Exporting and LSANs

The act of projecting a node into another edge fabric via a proxy is called exporting. When a host is
exported from Fabric 1 to Fabric 2, it must also be exported from Fabric 2 to Fabric 1, because FC
communications are always bidirectional. When an LSAN zone is created, exporting from both edge fabrics
occurs. Figure 6 shows a pair of devices that have been mutually exported between the two edge fabrics.
Figure 6. Exporting devices

When a set of devices on different edge fabrics are allowed to communicate through an FC router, the
resulting connectivity group is an LSAN zone. The LSAN zone entity associated with the export example in
Figure 6 is illustrated in Figure 7. Many different LSAN zones can exist between edge fabrics in the same
way that many zones can exist in a single fabric. Devices can be members of multiple LSAN zones and LSAN
zones can overlap with traditional zoning within local fabrics as well.
Figure 7. LSAN zones
8 of 25
Architectural Brief
Day-to-day administration of LSAN zones is performed using familiar zoning tools in each edge fabric. This
allows existing tools from Brocade or third parties to work as usual, minimizing the need to retrain SAN
administrators. Special LSAN zones in edge fabrics are created to indicate to the router which devices
should be exported. For an initiator to form a PLOGI with a target, an LSAN zone is created on each edge
fabric containing those particular devices. For nodes in two edge fabrics to communicate, there must be two
LSAN zonesone on each fabric. This method of routing nodes can be extended to three, four, or more
fabrics just by creating more LSAN zones in the appropriate edge fabrics.
Figure 8. LSAN zones

To an edge fabric, LSAN zones are indistinguishable from traditional zones, which is why they are
compatible with previous Brocade Fabric OS and Brocade M-Enterprise OS (M-EOS) versions. There are only
two distinguishing features of
an LSAN zone:
First, they must begin with the prefix LSAN_ so that routers and administrators can recognize them.
Second, they must contain only port World Wide Names (pWWNs) or device aliases that map to a
pWWN (since pWWNs are globally unique). As a best practice, LSAN zones should contain only devices
intended for inter-fabric sharing.
9 of 25
Architectural Brief
Backbone Fabrics
When multiple edge fabrics are connected or if remote fabrics are linked over long distances using FCIP,
routers can incorporate a centralized backbone fabric, as shown in Figure 9 (not to be confused with the
Brocade DCX Backbone, shown in Figure 1).
Figure 9. FCR backbone fabric connection to a number of edge fabrics

Backbone fabrics are used to expand scalability of connections to edge fabrics and to distribute that
connectivity across a campus or further across distant geographical locations. In Figure 9, multiple edge
fabrics are on the top tier, routers in the middle, and a standard fabric interconnects the routers. The
backbone allows a host in Fabric 1 to be exported to Fabric 14, even when those edge fabrics are not
attached to any common FC routers. Notice that routers do not use EX_Ports for router-to-backbone
connections, they use standard E_Ports. Also, when a host or storage device is connected via F_Port or
FL_Port interfaces directly to a router or director with an FCR blade installed, that device is in the backbone
fabric.
The ability to route into and out of the backbone fabric is an important feature, particularly when using
IR and the FR4-18i blade. With those products, all of the non-router ports in the chassis are part of the
backbone fabric, as described in the examples below:
First, a Brocade DCX containing six FC8-32 blades and two FR4-18i blades has 32 ports, which could
connect to various edge fabrics as EX_Ports or to nodes as F_Ports, four GbE ports supporting up to 32
FCIP tunnels, four FCIP ports, and 192 traditional 8 Gbit/sec FC ports.
The 192 8 Gbit/sec FC ports plus all the FR4-18i ports not configured as EX_Ports would be in the
backbone.
Second, if a Brocade DCX used IR and a certain number of ports were configured as EX_Ports, the
remaining ports would be in the backbone. Using backbone-to-edge routing allows nodes to be
connected to backbone ports and routed to edge fabrics. LSAN zones can be configured between
these devices and edge fabrics, in the same way as if the backbone were another edge fabric.
It is also possible to build redundant parallel backbone fabrics for High Availability (HA) applications. Each
router can have only one backbone attachment; however, multiple routers can each have a separate
backbone. If one router were connected to different backbone switches, the two backbone fabrics would
merge into a single fabric through the router.
10 of 25
Architectural Brief
For HA deployments, it is recommended that you deploy independent dual-redundant backbone fabrics. For
customers who implement dual-redundant fabrics today, there is usually little or no benefit to connecting
the fabrics together, even with a router. There is an availability and reliability benefit from not doing so.
Dual-redundant fabrics should remain physically isolated from each other to prevent a situation in which an
unforeseen error can bring down the entire network. Enterprises with dual-redundant fabrics should
implement at least two backbone fabrics, one for each of the redundant fabrics. If each fabric had
resiliency, that is, two routers per fabric, there would be a total of four backbone fabrics, two for one of
the redundant fabrics and two for the other.
From the point of view of devices on the backbone fabric, only one backbone fabric can be traversed. A
backbone fabric can never be connected through a router to another backbone fabric to reach a final
destination. This architecture, called Multi-Hop Routing, is not supported.
FC-NAT and Logical Domains

FC-NAT automatically determines the proxy PID presented to the foreign edge fabric for the device that has
been exported. This is important because it affects addressing and how an administrator will identify these
devices, for example, when looking at the edge fabrics name server.
Fibre Channel fabrics traditionally have a physical topology, in which physical ports of physical devices
connect via physical cables to physical ports of other physical devices. FSPF discovers the logical topology
of the switches by correlating links between those switches and storing that information in a database. All
switches discover an identical version of the database. The logical topology is run through the Dijkstra
algorithm to calculate the routing table from the perspective of that specific switch, producing a unique
routing table. Prior to SAN routers, physical and logical topologies were identical.
The SAN router introduces a new model, in which logical domains are presented. These logical domains
contain proxy devices. Logical domains are virtual independent switches created in software within the
router. From the perspective of the edge fabrics the logical domains are just another switch in the fabric.
They do not correlate directly to any physical entities; nonetheless, logical domains are presented to the
edge fabric via protocols such as FSPF LSA (Link State Advertisements), NS (Name Server), RSCNs
(Registered State Change Notifications), Management Server, etc
A device that has been exported from one edge fabric and into another is called a proxy device. Proxy
devices are brought online by routers when LSAN zones dictate their creation and the devices specified by
the zones are online. There is no need to bring a proxy device online if the real devices are not online. End
devices are notified when proxy devices are brought online or offline through normal fabric mechanisms
such as RSCNs and name server registrations.
Figure 10 extends the example from Figure 6 to show the way Fabric 2 would look when viewed from Fabric
1s point of view. Each EX_Port connected to Fabric 1 projects a Front Domain (FD) and each FD projects a
route into the edge fabric to the Translation Domains (TD). Each TD represents an entire remote edge
fabric. The EX_Ports for an FID will send FSPF LSA for all the TDs, which will result in calculated routes to
the TDs in all the switches in the FID. In some cases, TDs can be referred to as xlate domains. Even if a
remote edge fabric contains 30+ FC switches there will be just one TD representing the entire fabric.
All EX_Ports connected to the same FID from the same switch will receive the same domain ID for the FD, in
this case router 1s FD gets domain 1 and router 2s FD gets domain 2. These FDs are assigned by the
principal switch located within the edge fabric using traditional means of domain ID assignment. Assigning
a single domain ID to each FD with multiple EX_Ports is referred to as FD Consolidation.
The routers coordinate with each other using FCRP (FC Routing Protocol) to decide how to consistently
present devices from remote edge fabrics that they have access to. In this case, since they both have
access to Fabric 2, one of the routers will present that domain to the principal switch for assignment
(referred to as Request Domain ID or RDI) and receives domain 5; now all of Fabric 2 is represented by
domain 5. FCRP communicates between routers 1 and 2 to ensure that the domain for edge fabric 2 is
11 of 25
Architectural Brief
consistent and that the PIDs assigned to proxy devices are consistent on both attached routers. Obviously, if
the same device was represented by two different proxy devices, one in each router, determining which one
to post in the name server and which one to use for zoning would be problematic. Moreover, if the router
with the active entries were to go offline, communications would fail. It is essential that the domains and
devices remain in synchronization.
Figure 10. Internal topology

All exported devices must be translated via FC-NAT into the respective TD. The TD domain ID becomes the
first byte (domain byte) of the PID. To maximize FC-NAT address space translated devices are given FC PIDs
using both the area and port bytes. The combined field is incremented by one for each proxy added to the
domain. Proxy addresses look like public loop devices, even though they are really N_Ports in their source
fabrics, as shown below.
Byte 1: Domain ID 0 239
Byte 2: Area ID 0 255
Byte 3: Port ID 00 or 80 (allows addressing over

256 ports per platform)
Whatever PID is assigned to a given exported device is persistent. If all power was lost in the data center
causing every host, storage array, and network device (including the routers) to be simultaneously rebooted
it would not cause any TD PIDs to change. Furthermore, the translation table can be saved and reloaded
using the configUpload and configDownload commands. Even in the event of catastrophic failure of all SAN
routers, such that they have to be replaced, the mappings can be restored. This may be important for
operating systems that depend on the proxy PID remaining constant. The routers also provide commands to
manually set TD PIDs, if that is required.
NOTE: A devices real PID is nearly always different from the proxy PID. A devices nWWN (node World Wide
Name) and pWWN (port World Wide Name) are always the same and are never translated, therefore, a
pWWN is a unique identifier in a SAN and a PID is not.
In Figure 10, the disk in Edge Fabric 1 might have a PID of 13-01-00 if it were attached to domain 13 and
port 1. The host in Edge Fabric 2 could also potentially have a PID of 13-01-00 since the same domain and
port can exist in both fabrics. When the host is projected into Edge Fabric 1 it is translated to domain 5,
12 of 25
Architectural Brief
example PID could be 05-00-01. A Fibre Channel analyzer on Edge Fabric 1 would view the local
conversation as being between 13-01-00 and 05-00-01, regardless of the real PID the host had in Edge
Fabric 2. To determine which devices are involved would require looking at the pWWNs using element
management on the routers or SAN management software.
Backbone fabrics present a slight variation on this model since they do not use an FD. An edge fabric
attaches to one or more FD via an external EX_Port(s) and stays a separate edge fabric demarcated by
EX_Port(s). Backbone fabrics communicate to the routing service through embedded ports within the router.
The backbone fabric is a single entity no matter how many routers and switches are connected. Router
ports connecting to the backbone are all E_Ports, meaning that all switches including the routers
themselves will merge into a single fabric. Note that the edge fabrics on the EX_Port side will not merge.
Because the internal FC switch portion of the routers actually become part of the backbone fabric it is not
necessary to have FDs to demarcate the backbone fabric. Through the embedded router ports the
backbone fabric will see a TD for every edge fabric that a mutual LSAN zone has been configured.
Terminology Review
Term
Definition
Backbone
Routers allow a backbone fabric to interconnect other routers for a more scalable and flexible
routed SAN. Each router can have many edge fabric connections but only one backbone fabric.
Routers connect to the backbone fabric via E_Ports and all N_Port and NL_ Port connections to a
router are part of the backbone fabric. With IR on the Brocade DCX or a FR4-18i blade installed in
a Brocade DCX or 48000, a large number of hosts and storage devices can be connected to the
backbone fabric.
NOTE: Do not confuse the use of the word backbone in the term backbone fabric with the
Brocade DCX Backbone platform, shown in Figure 1.
EX_Port
FC routers use EX_Ports instead of E_Ports as routed interfaces and to demarcate the edge of a
fabric. To connect a router to a switch in an edge fabric, connect an EX_Port to an E_Port in the
edge fabric. EX_Ports limit fabric services from propagating.
FCR
Fibre Channel Routing is a method of establishing host and storage communications between
fabrics that remain autonomous.
FID
A unique identifier for each edge fabric. Two routers connected to the same edge fabric must
designate the same FID for that edge fabric.
FD
A Front Domain (FD) is a logical domain created at EX_Ports. FD consolidation uses a single
domain ID for all EX_Ports on the same router connected to the same FID. From the perspective
of FSPF, edge fabrics connect to FD, which in turn connects to TD.
IR
Integrated Routing is the same as FCR, except that the routing function has been integrated into
the switching 8 Gbit/sec ASICs and is supported on the Brocade DCX Backbone and Brocade
5300 and 5100 Switches.
ISL
The connection between two E_Ports is called an Inter-Switch Link.
IFL
The connection between an EX_Port and an E_Port is called an Inter-Fabric Link
Edge fabric
A traditional FC fabric connected to a router via an EX_Port (IFL). Edge fabrics are, for the most
part, where hosts and storage are attached.
LSAN
A Logical SAN is a group of correlated devices that are permitted to communicate between edge
fabrics. Communications traverse only one EX_Port or VEX_Port when one of the fabrics is the
backbone and exactly two EX_Ports if the end devices are both in edge fabrics.
13 of 25
Architectural Brief
Term
Definition
LSAN zone
LSAN zones are used to define the devices that participate in an LSAN and to configure
connectivity across routers.
PID
Port ID is the FC ID assigned to every port. The PID has 3 bytes: Domain | Area | Port.
Traditionally, Brocade has set Port equal to the ALPA to facilitate FCAL. When connecting
an N_Port, the Domain ID is the switch identifier, Area = port number and ALPA = 0.
When assigned by a TD, the Domain = the TD domain ID and the Area and ALPA bits
combine to allow addressing of more than 65,000 devices per TD.
TD
Translate Domain (TD) is a logical domain created behind FDs. Every edge fabric and the
backbone fabric have a TD if there is a device in the edge fabric that has an LSAN zone associated
with it. TDs house proxies of devices in a remote edge fabric created when exported via LSAN
zones from the local edge fabric.
SAN ROUTING USE CASES

This section provides specific examples of Brocade FC routing solutions.
Small-Scale Use Cases

The simplest deployment case for FCR involves connecting two fabrics into a single backbone. Two fabrics
are deployed for redundancy, as shown in Figure 11. Routing edge fabrics is usually not intended to solve
scalability problems; it is most often used for:
Interoperability issues
Vendor support agreements
Production status requiring isolation
SAN administration issues
Resource optimization
Tape connectivity
Figure 11. Redundant routed fabrics
14 of 25
Architectural Brief
In this example, a customer needed to separate the production and pre-production environments to prevent
problems in the pre-production fabric (for example, caused by testing new switch firmware versions) from
affecting the production systems. Nonetheless, they still needed connectivity between the environments so
that they can efficiently load data from the production SAN onto their test systems, which were used to
prototype enhancements to applications before being released into production.
To ensure HA, the customer implemented redundant fabrics keeping their HA model intact. Each of two
routers was connected to one of the two pairs of fabrics. Since the customer only needed 40 IFLs or less
per backbone, they elected to use a pair of Brocade 5100 switches with IR licenses. This design provided
them with all the needed connectivity in addition to significant room for growth.
An alternative to adding more switches is to enable Integrated Routing on existing Brocade DCX, 5300, or
5100 platforms in Fabrics A1 and B2. Using Fabrics A1 and B2 puts the demarcation for each edge fabric
on opposite sides to reduce the chance of a SAN-wide outage in the event of an unforeseen issue on one
side or the other. This is another example of backbone/edge routing, because when IR is configured in this
example, one side becomes the backbone fabric and the other side becomes the edge fabric. Each side has
one of each to mitigate an overall outage. Fabric A1 is the backbone fabric and presents an EX_Port to
Fabric B1, while Fabric B2 is the backbone fabric and presents an EX_Port to Fabric A2.
One common variation on this configuration involves long-distance connectivity for Disaster Recovery (DR).
Many customers need to connect a pair of fabrics at the primary data center to a pair of fabrics at a remote
site to perform Remote Data Replication (RDR). Figure 12 shows an example of a customer use case with
Brocade 7500 routers, and Figure 13 shows a similar design using the FR4-18i blade at the primary site.
Figure 12. Disaster recovery: variation 1

The example in Figure 12 is straightforward. The fabrics at each site are edge fabrics. The backbone fabrics,
router-to-router connections, are through the Wide Area Network (WAN). This explains why the WAN links are
labeled as ISLs instead of IFLs. The WAN cloud in this example could comprise various transmission
technologies, including: native metro FC links using dark fiber, Dense Wavelength Division Multiplexing
(DWDM), Synchronous Optimal Network (SONET), or a long-distance FCIP solution. FCIP links are merely
FC ISLs encapsulated in IP, and from a FC topology standpoint, should be considered as such. Unlike
common connections to a fabric, connecting to a LAN or WAN does not cause FC fabrics to merge.
15 of 25
Architectural Brief
Figure 13. Disaster recovery: variation 2

Figure 13 re-introduces backbone-to-edge routing. This customer has two Brocade DCX chassis at the
primary site and two Brocade 300 Switches at the DR facility, located 10 km away. A DWDM solution is in
place between the two sites. Previously, the IT Manager had been using a service to collect backup tapes
every day from the primary site and ship them to the DR site. She wanted to implement SAN-level
connectivity between the sites to eliminate the physical transportation solution. There was also a
requirement to provide isolation of fabric services and maintain administrative separation.
The most efficient solution is to enable IR on the Brocade DCX and connect the Brocade 300 Switches
across the DWDM, as shown in Figure 13. In this configuration, each fabric at the production site becomes
a backbone fabric with the DR site consisting of one edge fabric for each Brocade DCX Backbone at the
primary site.
Large-Scale use Cases

One of the driving forces behind developing a SAN router was to enhance scalability. Metcalfes Law broadly
states that the value of a network grows exponentially as its size increases. The larger a SAN becomes, the
greater the value it provides to an organization. On the other hand, it is axiomatic that larger SANs are more
difficult and expensive to build and require more resources to manage. This section shows example
deployments using FCR technology and demonstrates how routers enhance scalability.
16 of 25
Architectural Brief
Edge-to-Edge and Peer-to-Peer Design

Figure 14 shows how a large SAN built around redundant routers would look.
Figure 14. Large-scale SAN

This SAN was designed to provide connectivity between 10 pre-existing fabrics in an island consolidation
solution. One of the fabrics (FID=5) was quite large, while the others were relatively small or medium in size.
The sum total of the domains and ports in the 10 fabrics exceeded any single fabric supported scalability
limits and was operationally impractical to manage; nevertheless, the enterprise needed node connectivity
between all fabrics.
Previous methods for handling connectivity between fabrics were to physically connect hosts or storage
devices only during the times when inter-fabric connectivity was required. It was not practical to merge
fabrics, even on a temporary basis, because of the significant risk of an outage. Installing FC routers
permitted a wire once approach to SAN implementation. When data needed to be accessed across
different fabrics, it was no longer necessary to rewire the SAN, just add LSAN zones. To disconnect interfabric communications requires only removing those LSAN zones.
This example shows how a very high degree of scalability can be achieved with even a relatively small
number of routers. It also illustrates the scalability impact of routing on each fabric involved. Fabric 5, for
instance, was a large and complex fabric to begin with. If a host from Fabric 9 were to be exported into
17 of 25
Architectural Brief
Fabric 5, that would be a simple task compared to physical connectivity, which has all the complications
and instabilities associated with expanding the fabric even more. Even if every other edge fabric were as
large and complex as Fabric 5, only a very small percent of devices would be seen by Fabric 5.
Fabric 5 does not need to know about events happening in other fabrics, such as zoning changes, FSPF
topology changes, and Name Server database updates. The router insulates edge fabric services from each
other. If this network had been created as a single large fabric, every switch would require the complete
zoning database, all FSPF Link State Advertisements, and all Name Service (NS) updates. Increased
services can create timing and resource issues that prevent the fabric from being stable or even from
forming at all.
Centralized Resources
The previous example showed a way of creating selective, peer-to-peer connectivity between edge fabrics.
Many enterprises with large scale requirements need such connectivity in order to connect many devices
into one centralized SAN. Figure 15 shows such a case.
Figure 15. Large-scale tape consolidation

This design is similar to the Figure 13 variation in that it uses backbone/edge routing. The centralized fabric
uses IR to attach all of the organizations tape devices, while the four edge fabrics contain the hosts and
storage arrays sorted by division or application group. The backbone becomes the tape fabric. Notice the
EX_Port connection methodology. On the Brocade DCX deployed for tape, 16 EX_Ports on blades can be
configured and connected to 16 E_Ports on blades on each of the applications chassis. Each edge fabric is
connected to the backbone with 4 IFLs, 2 to each of two 8 Gbit/sec port blades. Those connections are
made to different blades on both sides to ensure that a single blade failure cannot eliminate all
connectivity.
18 of 25
Architectural Brief
Notice that the IFLs between each pair of blades use adjacent ports. If four IFLs are used, then it is wise to
have two groups of two. This follows Brocade ISL attachment best practices, which are:
Spread connections between at least two blades for high availability. Two groups are used to provide
redundancy; in addition, DPS (Dynamic Path Selection) is performed across the two groups.
Keep any additional ISLs co-located within the 8-port group to allow the use of optional ISL Trunking.
Brocade ISL Trunking creates a single logical link from 2 to 8 physical connections. If more IFLs were
needed in the future, it would be beneficial to reserve ports for them in advance such that they can be kept
within the same port groups. This aggregates more bandwidth for every IFL connected. In the past
depending on the number of FR4-18i blades, there were a limited number of EX_Ports (16 per blade) and
port groups (2 per blade). This made it difficult to spread the IFLs across many port groups to provide
available ports in the same group for future growth. IR enables a 1 port group for every 8 ports in a Brocade
DCX FC8 blade to be used for this type of application.
If the existing directors are Brocade 48000 Directors, Brocade FR4-18i blades will still have to be used. In
this case, the 16 IFLs connect using the same methodology as described above, except that the IFLs are
confined to the EX_Ports on the Brocade FR4-18i.
Of course, while this design is intended to allow access to the central tape fabric, it does not preclude
connectivity from edge fabric to edge fabric should that be needed in the future.
Multi-Site DR Solution
This example is similar to the case shown in Figure 13, except over a Metropolitan Area Network (MAN) to a
remote DR site.
Figure 16. Multi-site DR solution with FCR blades
19 of 25
Architectural Brief
The customer in this example had six offices in a metro area that needed SAN-level connectivity to a DR site
50 km away. The diagram represents one fabric; the other fabric is identical. At their headquarters, they
had over 750 host and storage ports connected to a pair of Brocade DCX Backbones connected by ICLs.
The manufacturing facility had a single 384-port Brocade DCX Backbone. Each of four sales offices had
individual Brocade 5100 Switches. The design they selected was to install a Brocade DCX chassis at the DR
site. A Brocade Extended Fabrics license is used at each site to ensure full-speed connectivity to the DR site
across the MAN. This design creates 6 edge fabrics (FID 1 through 6) and one backbone fabric, which is the
DR site fabric itself.
This is similar to the example in Figure 14, in that it connects many edge fabrics to a set of centralized
resources, specifically those at the DR site. It is possible to configure an LSAN from a sales office to
headquarters, but the traffic would need to flow through the DR site. If during the planning phase it
appeared likely that on a regular basis traffic patterns would flow from the satellite offices to headquarters
or vice versa, it would be more efficient to locate the backbone at headquarters and eliminate the
intermediate hop through the DR site. This assumes that the MAN connections could be made directly
from headquarters to the sales offices.
The Brocade DCX Backbone at the DR site uses IR EX_Ports to form the connections to the remote edge
fabrics. The same best practices described in the previous example can also be used here; however,
additional consideration must be made for Brocade ISL Trunking in this scenario. Brocade ISL Trunking
requires that the distances of all the IFLs in the group be essentially the same. When DWDM is using the
same fiber to transmit multiple IFLs, each on its own lambda, this is not a problem because inherently all
the IFLs are the same length. Keep in mind, however, that the Brocade trunk groups are not used for
redundancy as much as the deployment of multiple Brocade trunk groups. Think of each Brocade trunk
group as a single logical ISL or IFL. When two Brocade trunk groups are used, each trunk group should use
a different fiber run or operate over a different span of the DWDM ring.
GENERAL DESIGN CONSIDERATIONS

It is useful to look at specific case studies as in the previous section; however, in most practical situations
the design problem is at least somewhat different. Therefore, it is necessary for a designer to be aware of
the reasoning behind each design, so that they can make intelligent decisions and changes when
necessary. Many aspects of routed SAN design techniques and best practices are intuitive to SAN designers
and engineers already. Some aspects may not be intuitive and, unless these subjects are explicitly
discussed, results may be sub-optimal or even dysfunctional. Many organizations cannot tolerate
unavailable systems for any reason, especially due to a preventable storage outage. This section goes into
some of the less-intuitive aspects of routed SAN design in an effort to help designers avoid those problem
areas.
HA Design
High Availability network designs are always recommended. SAN downtime can bring mission-critical
systems offline resulting in huge loses for an enterprise. A rule of thumb is that if an organization served by
the SAN is adversely affected when attached devices suddenly stop working, beyond the point of the
expense to deploy and manage a second redundant fabric, then that SAN should be made redundant. The
greater the degree of impact of such an event, the greater the degree of redundancy that should be used.
This leads to an important SAN design principle: Always match the redundancy model used to the criticality
of the systems it protects. In other words, the more important the system, the more robust the redundancy.
20 of 25
Architectural Brief
There are four levels of redundancy in SAN design:

1.
Single non-resilient fabric
2.
Single resilient fabric
3.
Redundant but non-resilient fabrics
4.
Redundant and resilient fabrics
In the context of FCR solutions, these would be characterized as described in the following sections.
No fabric redundancy with one FCR platform

The platform might have multiple IFLs, but the platform itself is still a Single Point of Failure (SPoF). If the
router itself failed then inter-fabric connectivity is lost. Administrative activities also subject the network to
disruptive downtime when hardware upgrades or certain software operations are performed.
No fabric redundancy with two or more FCR platforms

The failure of an FCR platform would not cause an outage, nor would potentially disruptive operations,
such as hardware upgrades. However, the entire non-redundant network is subject to failure and is not
considered an HA design. For example, if each host has only one HBA connection and an edge fabric is
down, then all hosts connected to that fabric will also be down.
Dual-redundant fabrics each with one FCR platform

The hosts have multipathing software installed to handle failover and load balancing between the dualredundant fabrics. This is an HA solution and is probably the most popular deployment model for FC routers.
The risk here is that communications between edge fabrics for one of the two redundant fabrics may cease
if an FCR platform fails. During periods of failure, the remaining fabric is left non-redundant. For some
enterprises this may incur too much risk, depending on the applications crossing the router.
Dual-redundant fabrics each with two or more FCR platforms

This is the best approach from an HA viewpoint. If an FCR platform fails, the dual-redundant fabrics can
continue to function, preventing the excessive risk of operating on a single non-redundant fabric. While this
design has the greatest availability, it also costs the most to implement.
HA SAN design starts with dual-redundant fabrics and is an industry-wide best-practice. Two physically
separate and independent fabrics are used. Hosts and storage devices have connections to each of these
fabrics, which will continue to operate even in the event of a catastrophic failure affecting one entire fabric.
HA design also provides protection against an outage when certain kinds of upgrades or other maintenance
activities are performed. Brocade Fabric OS features non-disruptive firmware upgrades and downgrades,
though changing some fabric configuration parameters requires disabling the platform. Scheduled
downtime is as detrimental to application availability as unplanned downtime; the application is still not
available and in a 24x7 data center, that is a problem. The difference is in the timingusers expect planned
downtime and can plan for it. They do not expect unplanned downtime. Nevertheless production stops and
Service Level Agreements (SLAs) are in jeopardy. Upgrades on the Brocade 7500/FR4-18i interrupt FCIP
service for a short period of time, but Fibre Channel traffic, including routed traffic, is not disrupted. (FCIP
links are ISLs extended over IP, extending edge fabrics or backbones.)
The Brocade 7500, 5300 and 5100 are all standalone platforms, not fully hardware-redundant platforms,
implying that a single failure may disrupt the entire unit, for example, due to an issue with the main PCB.
For these devices, redundancy is provided by having multiple units in place. The bottom line is this: there
will always be some administrative operations that result in service interruption, even on an enterprise-class
platform. The best way to ensure availability is to use redundant platforms. If maintaining the connection
21 of 25
Architectural Brief
between fabrics is considered important to operations, each site should have at least two distinct FCR
platforms connected to each dual-redundant fabric.
In general, the redundancy and resiliency designed into each level is in accordance with budget constraints.
Just because redundant power supplies in switches could be eliminated to save money does not imply that
they should be eliminated. The same applies to deploying redundant FCR platforms. The decision about
how much redundancy to build into each tier of the network comes down to site-specific tradeoffs among
price, performance, and availability.
Interoperability
All EX_Ports, either IR or Brocade 7500/FR4-18i, support interoperability to Brocade FOS products
in native mode, as well as classic Brocade M-EOS products and classic McDATA EOS products in both
McDATA native fabric mode and open fabric mode. The scalability limits of Brocade M-EOS edge fabrics are
the same as if the M-EOS fabric were a standalone fabric: 31 domains and 2,048 devices. This is true for
both the M-EOS fabric modes.
M-EOS fabrics cannot act as the backbone or be part of the backbone fabric. To route nodes between an MEOS fabric and FOS fabrics requires the use of an EX_Port, which means that the M-EOS fabric would have
to be an edge fabric.
NOTE: Interoperability with Cisco MDS FC platforms is not supported.
Scalability
Scalability can mean quite a few different things in the context of storage networking. For example, it can be
a measure of how much data a particular RAID enclosure accommodates. In the context of SAN design, it
usually refers to how many ports a network model can support without needing to be fundamentally
restructured. Pertinent to FCR scalability are these considerations:
How many network devices, such as switches and routers, can be deployed in a fabric?
How many separate edge fabrics can communicate across the routers?
How many devices can be exported at any given time?
How large can a backbone fabric be?
All router platforms, including the 8 Gbit/sec port blades for the Brocade DCX, the Brocade 7500/FR4-18i,
and Brocade 5300 and 5100 Switches process FCR in hardware and support a large number of proxy
devices. Inherent in the nature of the technology is scalability support.
This section discusses how some of those scalability limitations apply to routed SAN design. There are
definite scalability limits specific to various aspects of FCR, which may change as Fabric OS versions
mature. (See Fabric OS Release Notes for up-to-date scalability metrics.)
General Nature of Scalability Limits

All networks have scalability limitations. Even the IPv4 protocol, which forms the basis for the Internet, has
reached its limit and is starting to be replaced by the IPv6 protocol, which also has limits albeit much higher
ones. The limitations of a network, generally, can be classified into categories such as manageability, fault
containment, vendor supported, network services, and the protocol addressing structure.
The protocol limits of FC are much higher than any network that can be built. Limiting factors of a FC
network are not centered around the FC protocol as much as around the fabric services that run on the
network. Databases supporting zoning, FSPF, and name services that are synchronized across all switches
in a fabric are limiting, as well as the computing power to process those services on each switch. Larger FC
networks take longer to converge, and as the number of switches increases the opportunity for instability
22 of 25
Architectural Brief
increases; therefore, there is a reason to keep the size of FC networks to a minimum. Each OEM vendor has
established its own rules of scalability (vendor support matrices are beyond the scope of this document).
Fault Containment
The networking industry as a whole realized years ago that Ethernet segments had a practical scalability
limit related to broadcast domains and fault isolation. Even if it were possible to create a single Ethernet
segment with thousands of devices, it would not be a good idea to do so. Inevitably, no matter how diligent
the network engineering and management teams were, eventually something unanticipated would go wrong
taking down a segment. A malfunctioning Network Interface Card (NIC) resulting in a broadcast storm was
beyond their control, and the best defense was to mitigate the number of affected users by way of
containment. Each broadcast that arrives at a host interrupts its CPU and thousands would incapacitate a
host. The more hosts on a segment, the bigger the disruption and the likely chance that more hosts will
participate in the broadcast storm.
To address this issue, network engineers began limiting the size of segments in order to limit the scope of
problems on a segment. The industry eventually settled on a preferred Ethernet segment size of one class C
IP subnet, about 250 devices, and treated that as a scalability limit for Ethernet. Of course, this implies that
multiple segments have to be connected together somehow, because more than 250 devices exist in an
organization. To strike a balance between providing connectivity and limiting the scope of faults, engineers
created a hierarchical networking model based on routers. This approach is now available for storage
networks as well, by connecting fabrics together via FC routers.
An analogy is frequently made between broadcast storms and fabric reconfigurations. With large flat
fabrics, certain kinds of reconfigurations can create stability issues in much the same way that certain kinds
of broadcast storms can destabilize an Ethernet segment. Larger fabrics are proportionally more likely to
experience convergence timing issues during reconfigurations, just as larger Ethernet segments are more
likely to experience broadcast stormsand in both cases this creates a scalability constraint. In a Fibre
Channel SAN, a fabric reconfiguration in one edge fabric is not propagated to another through a router,
which is analogous to the way that an IP router contains broadcast storms.
FC fabrics have advanced software running on their switches, permitting multi-thousand port Layer 2 fabrics
to exist without fault. However, for customers pushing the limits of fabric scalability without routing, fabric
convergence time must be considered and can be an issue. This issue simply does not exist between
smaller routed edge fabrics.
Fabric Services
Considering that the edge fabric terminates at the logical domains in the router, the zoning database, name
services, and fabric topology data is not propagated to other edge fabrics. Name Server (NS) entries cross
edge fabric boundaries for devices that have been explicitly exported via LSAN zones in the form of a proxy.
NS databases can grow only so large due to limited resources in the switch. Exported devices from a native
edge fabric form a proxy device in a foreign edge fabric and consume an entry in the foreign fabrics NS, the
same as any other attached host or array port in that fabric. Each virtualized device discovered by N_Port ID
Virtualization (NPIV) also consumes an entry in the NS. This indicates that adding proxy devices to an
already large edge fabric can push the NS to its limits.
For example, if one fabric has a scalability limit of 3,000 NS entries and currently has 2,500 devices and so
does another fabric in the same environment, an administrator could not merge the fabrics without going
beyond each fabrics NS capabilities. On the other hand, if two fabrics were connected via SAN routers, then
up to 500 devices could be shared between each edge fabric without hitting the NS scalability limits.
Best practice is to only create LSAN zones for devices that actually need to communicate across the
routers, rather than adding all devices to LSAN zones arbitrarily.
23 of 25
Architectural Brief
Allowable Number of Edge Fabrics

The processor and memory in routers are limited, and thus the number of edge fabrics a router can support
is finite. To maximize scalability, a balance between individual edge fabric size and the number of edge
fabrics must be established. For example, if a single data center is consolidating 5,000 ports into a single
routed SAN, a designer could chose to target two 2,500-port edge fabrics, ten 500-port edge fabrics, or one
hundred 50-port edge fabrics. All other things being equal, using ten 500-port fabrics would probably be the
best approach, considering that each individual edge fabric would have a manageable size and the number
of edge fabrics would also be manageable. This is more of a guideline than it is a rule or best practice, since
there are often competing considerations with greater weightings.
Another approach for large-scale environments is to divide the solution into multiple backbones. The first
step is to ensure that Fabrics A and B are not inter-connected in any way, that is, to the same set of routers.
This is the correct approach from an HA standpoint and this alone doubles the overall scalability. Next,
consider whether or not all the fabrics require connectivity into the backbone. If the design involves multiport arrays that can accommodate attachment to each and every fabric, it might not be necessary to route
between fabrics, just connect them all to the array. Every fabric then has access to the same set of storage
volumes. Not all storage arrays have enough ports to meet this alternative approach.
In any case, this level of scalability analysis is rarely warranted. Unless the environment is going to involve
more than a dozen fabrics per backbone, this will simply not come up. It actually has more applicability to
manageability than scalability: It is easier to manage fabrics with hundreds of ports than managing fabrics
with thousands of ports.
Per-Chassis Scalability
Brocade DCX Backbone. The Brocade DCX supports up to 4 FR4-18i blades with no caveats; up to 8
FR4-18i blades can be used in a Brocade DCX as long as only the FCIP and FC-FW are being used and no
routing functions. All 128 FC ports can be used as EX_Ports. For FCR-only implementations, IR is
recommended over FR4-18i blades, due to the flexibility and higher port speed. IR on the Brocade DCX
supports 128 EX_Ports per chassis, enabling connectivity to a large number of edge fabrics each with an
adequate number of IFLs for aggregated bandwidth and redundancy. If two chassis are linked via ICL as a
backbone, each can support 128 EX_Ports for a total of 256 EX_Ports. The main purpose of Brocade FR418i blades in the Brocade DCX is to provide FCIP capabilities.
IR EX_Ports and Brocade 7500/FR4-18i EX_Ports use identical protocols and therefore are completely
interoperable when connected to the same edge and backbone fabrics. While this may be true, Brocade
FR4-18i EX_Ports cannot coexist in the same chassis with enabled IR EX_Ports and vice-versa. It has to be
one or the other. VEX_Ports used with FCIP are not restricted. In a Brocade DCX configured to use IR, if an
FR4-18i is also configured to use VEX_Ports, that is fully supported. (VEX_Ports are simply EX_Ports
encapsulated in IP.)
Brocade 5300 and 5100 Switches. The Brocade 5300 supports 80 IR EX_Ports per switch (out of a total of
80 ports). The Brocade 5100 supports 40 IR EX_Ports per switch (out of a total of 40 ports).
Brocade 48000 Director. The Brocade 48000 supports 8 Gbit/sec blades on FOS 6.1; however, the IR
feature is not supported due to the 4 Gbit/sec CP4 ASICs that switch backbone traffic. The Brocade 48000
can support two FR4-18i blades; however, more than two blades fan can be used for FCIP and FC-FW if no
routing functions are configured. If you need additional routing ports, the alternatives are to: deploy more
FR4-18i blades across different chassis, use the Brocade 7500, or deploy the Brocade DCX, 5300, or 5100.
In fact, there are a number of advantages to the latter approach, including increased scalability, higher
availability (if multiple backbones are used), and often lower cost.
24 of 25
Architectural Brief
Complex Topologies
At the highest level, only Core/Edge (CE) designs and their variations are supported by FCR. That is, there
are two tiers and no more or less than two tiers, made up of the edge fabrics and one or more parallel
backbone fabrics. It is not currently possible to have traffic cross more than one backbone in series from
end to end or in a circular fashion, known as multi-hop routing. This design decision is related to hop-count
restrictions recommended for inter-data-center, block-level communications. Many applications cannot
withstand even a moderate amount of latency and still continue to perform as expected by users. Data can
traverse only a limited number of ISLs and/or IFLs and still maintain acceptable performance.
It is possible to accommodate nearly all requirements using the topology offered by FCR and IR, while
continuing to maintain high performance standards. The reality is that after reaching the proxy located in
the router, the data must pass through either a backbone or another edge fabric or both before reaching
the final destination. This can triple the number of real hops to a destination.
There can be quite a bit of variation in each edge and backbone fabric. An edge fabric or backbone does
not need to be a Core/Edge design and often will not or should not be. The backbone(s) between routers in
most of the examples in this paper are cascaded or meshed in some way, which is good practice. Many of
the examples use single-domain fabrics and that is also best practice in environments that are small
enough. Even if the existing fabrics are full meshes or arbitrarily complex partial meshes, the router can
connect to them without issues. However, just because something can be done does not imply that it
should be done. It is always best to use as simple a topology as possible. Remember Occam's Razor: if you
can make a network smaller in radius and simpler in design, it is best to do so. (William of Occam was the
medieval European philosopher who said, The simplest solution is usually the best.)
FICON
FCR does not support FICON. The use of proxies in FCR makes it impractical for users to configure control
unit addresses in the mainframe. In addition, there are other important communications that must occur
under various situations that are not accommodated by FCR. For the time being, cascaded FICON fabrics
have to be merged, including those that span MAN/WAN connections.
SUMMARY
This paper reviewed enabling technologies for routed SAN solutions. By adding hierarchical networking to
Fibre Channel, Brocade has combined the flexibility and scalability of data networks with the performance
and reliability of FC fabrics. FCR brings unprecedented reliability, manageability, security, flexibility, and
scalability to SANs, and makes deployments practical that were not even theoretically possible before its
introduction. A number of architectures were discussed, including a backbone fabric with multiple edge
fabrics and routing among those fabrics through the backbone fabric. Also included were designs that route
from the backbone fabric to an edge fabric.
2008 Brocade Communications Systems, Inc. All Rights Reserved. 05/08 GA-AB-060-00
Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing
symbol, DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or
in other countries. All other brands, products, or service names are or may be trademarks or service marks of, and
are used to identify, products or services of their respective owners.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied,
concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the
right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This
informational document describes features that may not be currently available. Contact a Brocade sales office for
information on feature and product availability. Export of technical data contained in this document may require an
export license from the United States government.
25 of 25

Designing A Routed Fibre Channel Storage Area Network

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Designing A Routed Fibre Channel Storage Area Network

Hochgeladen von

Copyright:

Verfügbare Formate

STORAGE AREA NETWORK

STORAGE AREA NETWORK