DCX Icl

Brocade Virtual Classroom Training
DCX Data Center Backbone:

8 Gbit/sec Port Behavior
© 2008 Brocade Communications Systems, Incorporated. 1

0108
© 2008 Brocade Communications Systems, Incorporated.

Revision 0108
Page 1
Objectives
This module will describe the following:
• The features of the Condor2 ASIC.
• The frame flow through the DCX switch
• The ICL (Inter Chassis Link) feature unique to the DCX product

0108

Revision 0108
Page 2
The Condor2 ASIC
Extended functionally: Support for current features

• Port speeds – 1, 2, 4, or 8 • ISL Trunking - Up to 8 ISLs per
Gbit/sec; E/F/FL1/M_Port2 Trunk
• Virtual Channels – Up to 16 VCs • Routing – Static, FSPF, DPS

per port
• Advanced Performance Monitoring
• Increased Buffer to Buffer Credits3
• Zoning – Hard and Session
enforced
• Other features: Port swap, mirror

port2, NPIV

0108
Footnote 1: No FL support on the 48 port blade.
Footnote 2: Mirror ports is not supported on the Condor2 ASIC with Fabric OS
v6.0.0 release.
Footnote 3: Each Condor2 ASIC has 1420 user BB credits: Each front-end port is
allocated 8 credits (8 credits x 16 ports = 128 credits) which leaves 1292 credits
available per Condor2 ASIC. Note: The 48 port blade has 24 front-end ports per
ASIC (8 credits x 24 ports = 192 credits) which leaves 1228 credits available per
Condor2 ASIC) Use the portbuffershow command to see available credits.

Revision 0108
Page 3
FC Port Blade Frame Flow
If the SID and DID are on the same Condor2 ASIC, local switching is
used – just as with the Condor ASIC
• The frame does not have to go through the CR8 (core) blade
Frames going from one Condor2 to another Condor2 on the same

Port blade must go through the CR8 (core) blade
• No direct connections between the Condor2 ASICs on the same
port blade - same as on the Brocade 48000
Condor2 ASIC latency is 700ns per ASIC (2.1µs through the switch)1

0108
Footnote 1: Through the switch means the frame comes in on one blade goes
through core blade and out another blade on the switch. The Condor ASIC latency
is about 700ns per ASIC and 2.1µs through the switch.

Revision 0108
Page 4
DCX Architecture: 16 Port Blade
CR8 Blade Each line is a FC8-16 Blade
2 x 8 Gbit/sec
Condor2 Trunk
ICL1
FC front end
Condor2 ports 0-15
Condor2
Condor2
ICL0
Condor2
Each line is a
FC8-16 Blade
8 x 8 Gbit/sec CR8 Blade
Trunk
Condor2
ICL1 FC front end
Condor2 ports 0-15
Condor2
Condor2
ICL0
Condor2

0108
Each Condor2 ASIC has 40 ports; on the 16 port blade only 32 are used: 16
external and 16 Internal.
If the initiator and target are on the same ASIC the frame would not go through the
core and local switching would be used.
If the frame coming out of the core blade is going out the ICL cable each condor2
ASIC on the core has 8 x 8 Gbit/sec Trunk going to the other DCX.
If the frame coming out of the core blade is going to a 16 port blade then each
condor2 ASIC on each core blade would have 2 x 8 Gbit/sec Trunk going the
Condor2 ASIC on the 16 port blade.
If the frame coming out of the core blade is going to a 32 or 48 port blade then each
condor2 ASIC on each core blade would have 2 x 8 Gbit/sec Trunk going the both
of the Condor2 ASICs on the 32 or 48 port blade.

Revision 0108
Page 5
2 x 8 Gbit/sec
FC front end
Condor2 Trunk
ports 8-15, 24-31
ICL1 Condor2
Condor2
FC front end
Condor2 ports 0-7, 16-23
ICL0 Condor2
Condor2
Each line is a
FC8-32 Blade
Trunk FC front end
Condor2
ICL1
Condor2
FC front end
Condor2
ICL0
Condor2

0108
Each Condor2 ASIC has 40 ports; on the 32 port blade only 32 are used: 16
external and 16 Internal.

Revision 0108
Page 6
2 x 8 Gbit/sec
FC front end
Condor2 Trunk
ports 8-23, 40-47
ICL1 Condor2
Condor2
FC front end
ICL0 Condor2
Condor2
Each line is a
FC8-48 Blade
Trunk FC front end
Condor2
ICL1
Condor2
FC front end
Condor2
ICL0
Condor2

0108
Each Condor2 ASIC has 40 ports; on the 48 port blade all 40 ports are used: 24
external and 16 Internal. This is why the 48 port blade is 24 to 16 oversubscribed.

Revision 0108
Page 7
Discussion Questions – Part 1
The number of buffer to buffer credits has been increased to which of the following values:
1420
1292
Which port speeds are supported:

1 Gbit
2 Gbit
4 Gbit
8 Gbit
All of the above
What is the maximum number of virtual channels per port?

8
16
32
Which command would you use to determine how many buffer to buffer credits are available?
trunkshow
portbuffershow
licenseshow

0108
Before continuing with this module, please answer these questions.

Revision 0108
Page 8
ICLs Double The Core
Problem: One DCX switch as the core is

not big enough to meet the needs of the
fabric
Solution: ICLs - Inter Chassis Links

These are ISL connections between two
DCX switches on the CR8 (core) blades
Creates a dual DCX core fabric:

Doubles the port count at the core of the
fabric without the use of any user ports

0108
ICL kit comes with the following:

Four ICL cables
Two ICL licenses (one per DCX)

Revision 0108
Page 9
What Are ICLs?
ICLs are 8 Gbit/sec ISL connections

between two DCX switches on the
CR8 (core) blades
• Speed locked at 8 Gbit/sec
• Copper-based proprietary
connector
• No SFPs
• Each cable provides 16 x 8
Gbit/sec bandwidth
• Licensed feature

0108
ICL cable connector:

Revision 0108
Page 10
DCX ICL Connections
CR8 (core) blades must be in slots 5 & 8
ICL used to connect ONLY two DCX chassis

together1
No user ports are required for ICL

connections
DPS (Dynamic Path Section) is used on the

ICLs
Each ICL cable provides 2 x 64 Gbit/sec

Trunk2

0108
Footnote 1: This is a requirement with Fabric OS v6.0.0.

Footnote 2: Does not require a Trunking license on the switch to Trunk the ICL
ports.

Revision 0108
Page 11
ICL Cabling
ICL0 (bottom port) must be connected

to ICL1 (top port) on the other DCX
ICL1 (top port) must be connected to
ICL0 (bottom port) on the other DCX1
Cables can be cross connected to the

other slot. Example: Cables connected
to slot 5 on one DCX can be connected
to slot 8 on the other DCX
ICL Cables are 2 Meters in length

0108
Footnote 1: Cables and connectors are color coded to avoid misconnection.
ICL1
ICL0

Revision 0108
Page 12
Other ICL Cabling Supported Configurations
All cables crossed Inter cables crossed Top/bottom cables crossed

0108
Footnote 1: All ICL cables are 2 Meters in length.
These work just like other ISL connections if a cable is pulled would result in the
failover to other paths (ICL connections or external ISL connections between the
two switches).
The only difference between ICLs and ISL are ICLs do not use user ports and the
port speed is hard set and Trunking is automatically enabled.

Revision 0108
Page 13
ICL Color Codes
ICL cable and port connectors will be colored coded

to make connections easy

0108

Revision 0108
Page 14
ICL: Inter Chassis Link
512 Gbit/sec total bandwidth (1 Tbit/sec bidirectional)

• 2 Core blades x 32 ports per blade x 8 Gbit/sec per port = 512
Gbit/sec total bandwidth between two DCX chassis
• Two connector ports per blade; Each connector port aggregates 16
ICL ports (128 Gbit/sec bandwidth per cable)
ICL connections can only be used to connect two DCX chassis

together
ICL License is required on each DCX

0108

Revision 0108
Page 15
DCX Core Fabric
With ICLs Installed
Maximum user port count doubles
• 768 ports user ports
512 Gbit/sec total bandwidth between

two DCX chassis without the use of
any user ports
ICLs are oversubscribed 4:11
Two Domains

0108
Footnote 1: For any-to-any topologies where any port connected to one chassis
may be connected to any port on the other chassis, ICLs are oversubscribed 4:1 at
8 Gbit/sec between the chassis (Core Blades in Chassis 1 to Core Blades in
Chassis 2).
If more inter-chassis bandwidth is required, nothing precludes the use of front ports
as ISLs in complement to ICLs. When each chassis is fully configured to 384 ports
and any port may connect to any other port, then the two chassis are 6:1
oversubscribed (1.5x oversubscribed at each chassis multiplied by 4x
oversubscribed across ICLs) at 8Gbit/sec.

Revision 0108
Page 16
ICL Port 16 ports per
Numbers ICL connector
Index Port Index Port
415 31 447 31
408 24 440 24
Slot 5 Slot 8
407 23 439 23
400 16 ICL port 1 432 16
Index Port ICL port 0 Index Port

399 15 431 15
392 8 424 8
Slot 5 Slot 8
391 7 423 7
384 0 416 0
Slot 5 Slot 8
0108
The FC ports on the port blades take index numbers 0 – 383. The ICL port take the
next 64 index numbers 384-447.
16 ports per ICL connector

64 total ICL ports
2 X 8 port Trunks per cable

8 total Trunks

Revision 0108
Page 17
ICL Ports
Appear as regular FC ports with certain restrictions

Most properties are static. No “portcfg” allowed on these ports
• Speed is locked at 8 Gbit/sec
• Trunking enabled
• Credit sharing is on
• Support for QoS
• No long distance support

0108
Here is what happens when you try to set the port speed on an ICL port:
DCX10:admin> portcfgspeed 5/1 4
Operation failed - Configuration change is not allowed on ICL
ports.

Revision 0108
Page 18
ICL Ports - licenseshow
ICLs are enabled by installing an Inter Chassis Link license

DCX10:admin> licenseshow
<truncated output>
DMKtfZrrBaDZZYDXC4JBER7mTGBmQD9FBJWrG:
Inter Chassis Link license
Licenses are now greater then 16 characters

0108
In the licenseshow command output above, we see that a Inter Chassis Link
(ICL) is installed on the DCX.

Revision 0108
Page 19
ICL Ports - switchshow1
In the switchshow command output, the ICL ports appear as standard E_Ports,
locked at 8 Gbit/sec, and grouped in eight-port Trunks
DCX10:admin> switchshow
<truncated output>
400 5 16 ------ cu 8G Online E-Port 10:00:00:05:1e:43:06:00 "sw025234"
(Trunk master)
401 5 17 ------ cu 8G Online E-Port (Trunk port, master is Slot 5 Port 16 )
409 5 25 ------ cu 8G Online E-Port 10:00:00:05:1e:43:06:00 "sw025234"
(Trunk master)
<truncated output>
0108
Footnote 1: This is part of the switchshow output for one ICL cabled connected
between two DCX boxes. In the example above, the ICL cable is connected to the
bottom connector on core blade in slot 5. Some key points:
• 8G – All ports are hard set to 8 Gbit/sec
• There are two 8 x 8 Gbit/sec Trunks per cable or 2 x 64 Gbit/sec Trunks or 128
Gbit/sec total bandwidth per cable.
ICL ports may be enabled / disabled just like any other port on the switch. The only
difference is they are E_Ports only, hard set to 8 Gbit/sec, with QoS and Trunking
enabled.

Revision 0108
Page 20
ICL Ports - trunkshow
The 16 ICL ports appear as two 8-port Trunks in the trunkshow

command1.
DCX:admin> trunkshow
1: 400->384 10:00:00:05:1e:43:06:00 2342deskew 15 MASTER
401->385 10:00:00:05:1e:43:06:00 234 deskew 15
402->386 10:00:00:05:1e:43:06:00 234 deskew 15
403->387 10:00:00:05:1e:43:06:00 234 deskew 15
404->388 10:00:00:05:1e:43:06:00 234 deskew 15
405->389 10:00:00:05:1e:43:06:00 234 deskew 15
406->390 10:00:00:05:1e:43:06:00 234 deskew 15
407->391 10:00:00:05:1e:43:06:00 234 deskew 15
2: 409->392 10:00:00:05:1e:43:06:00 234 deskew 16 MASTER

408->393 10:00:00:05:1e:43:06:00 234 deskew 16
410->394 10:00:00:05:1e:43:06:00 234 deskew 15
411->395 10:00:00:05:1e:43:06:00 234 deskew 16
412->396 10:00:00:05:1e:43:06:00 234 deskew 15
413->397 10:00:00:05:1e:43:06:00 234 deskew 15
414->398 10:00:00:05:1e:43:06:00 234 deskew 16
415->399 10:00:00:05:1e:43:06:00 234 deskew 15

0108
Footnote 1: The trunkshow command output shown above is for the ports seen
earlier in the switchshow command output.
Footnote 2: 234 is the domain ID of the switch (WWN 06:00) that this Trunk is
connected to.

Revision 0108
Page 21
ICL Ports - islshow
The two ICL Trunk masters appear as in the islshow command1

• Note that the speed is shown as 8 Gbit/sec, and the overall
bandwidth is 64 Gbit/sec
DCX:admin> islshow
1:400->384 10:00:00:05:1e:43:06:00 2342 sw025234 sp:
8.000G bw: 64.000G TRUNK QOS
2:409->392 10:00:00:05:1e:43:06:00 234 sw025234 sp:
8.000G bw: 64.000G TRUNK QOS

0108
Footnote 1: The islshow command output shown above is for the ports seen
earlier in the switchshow command output.
Footnote 2: 234 is the domain ID of the switch (WWN 06:00) that this Trunk is
connected to.

Revision 0108
Page 22
ICL Ports - portcfgshow
The ICL ports support a limited set of port hardware features in the
portcfgshow command
DCX:admin> portcfgshow
Ports of Slot 5 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
-----------------+--+--+--+--+----+--+--+--+----+--+--+--+----+--+--+--
Speed1 AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN
AL_PA Offset 13 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Trunk Port ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
Long Distance .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
VC Link Init .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked L_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked G_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Disabled E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
ISL R_RDY Mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
RSCN Suppressed .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Persistent Disable.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
QOS E_Port2 ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
Mirror Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..

0108
Footnote 1: In Fabric OS v6.0 the portcfgshow output does not display the speed
correctly this should show up as hard set to 8G.
Footnote 2: QoS will show enabled once the license has been installed the link
bounced. The link must bounce to create the 16 VCs.
Trunks can not be disabled:

dcx10:admin> portcfgtrunkport 5/16 0
Operation failed - Configuration change is not allowed on ICL
ports.

Revision 0108
Page 23
ICL LEDs
One set of LEDs for both ICL0 and ICL1 connectors
Green Amber Condition

(LINK) (ATTN)
Off Off (Cable is not present) OR (Local end is not ready)
OR (Far end is not ready)
Off On N/A
On Off ICL is fully functional
On On (Cable is present) AND (Local end is ready) AND

(Blinking) (Far end is ready) AND (Attention is required1)
Footnote 1: Means something is wrong such as a port failure

0108

Revision 0108
Page 24
Discussion Questions – Part 2
True or False: ICLs are 8 Gbit/sec ISL connections between two DCX
switches on the CR8 (core) blades?
True or False: The use of ICL creates a dual DCX core fabric that doubles
the port count at the core of the fabric without the use of any user ports?
Which one of the following statements is correct?

An ICL license is required on each DCX
No license is required for an ICL
An ICL license is included in the purchase of a DCX
True or False: You can change the port speed on an ICL port from 8 Gb to a
slower port speed?

0108

Revision 0108
Page 25
Discussion Questions – Part 2 (cont.)
When an ICL is installed, the maximum user port count doubles to

which value?
384
768
1152
When you purchase an ICL kit, what does the kit contain?
Four ICl cables
Two ICL licenses
Two ICL cables
One ICL license
0108

Revision 0108
Page 26
Summary
8 Gbit/sec Condor2 ASIC port behavior is the same as the 4 Gbit/sec

port behavior on the Condor ASIC, only with twice the speed.
ICLs are a licensed feature
ICLs allow 512 Gbit/sec bandwidth between two DCX switches
ICLs allow the number of ports in a DCX core fabric to double

(without using any user ports) up to a maximum of 768 ports

0108

Revision 0108
Page 27
Brocade Virtual Classroom Training
End of
DCX Data Center Backbone:
8 Gbit/sec Port Behavior
0108

Revision 0108
Page 28

DCX Icl

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

DCX Icl

Hochgeladen von

Copyright:

Verfügbare Formate

Brocade Virtual Classroom Training

DCX Data Center Backbone:

© 2008 Brocade Communications Systems, Incorporated. 1

© 2008 Brocade Communications Systems, Incorporated.

This module will describe the following:

• The features of the Condor2 ASIC.

• The frame flow through the DCX switch

© 2008 Brocade Communications Systems, Incorporated. 2

© 2008 Brocade Communications Systems, Incorporated.

Extended functionally: Support for current features

• Virtual Channels – Up to 16 VCs • Routing – Static, FSPF, DPS

• Other features: Port swap, mirror

© 2008 Brocade Communications Systems, Incorporated. 3

Footnote 1: No FL support on the 48 port blade.

© 2008 Brocade Communications Systems, Incorporated.

Frames going from one Condor2 to another Condor2 on the same

© 2008 Brocade Communications Systems, Incorporated. 4

© 2008 Brocade Communications Systems, Incorporated.

© 2008 Brocade Communications Systems, Incorporated. 5

© 2008 Brocade Communications Systems, Incorporated.

© 2008 Brocade Communications Systems, Incorporated. 6

© 2008 Brocade Communications Systems, Incorporated.

© 2008 Brocade Communications Systems, Incorporated. 7

© 2008 Brocade Communications Systems, Incorporated.

Which port speeds are supported:

What is the maximum number of virtual channels per port?

© 2008 Brocade Communications Systems, Incorporated. 8

Before continuing with this module, please answer these questions.

© 2008 Brocade Communications Systems, Incorporated.

Problem: One DCX switch as the core is

Solution: ICLs - Inter Chassis Links

Creates a dual DCX core fabric:

© 2008 Brocade Communications Systems, Incorporated. 9

ICL kit comes with the following:

© 2008 Brocade Communications Systems, Incorporated.

ICLs are 8 Gbit/sec ISL connections

© 2008 Brocade Communications Systems, Incorporated. 10

ICL cable connector:

© 2008 Brocade Communications Systems, Incorporated.

CR8 (core) blades must be in slots 5 & 8

ICL used to connect ONLY two DCX chassis

No user ports are required for ICL

DPS (Dynamic Path Section) is used on the

Each ICL cable provides 2 x 64 Gbit/sec

© 2008 Brocade Communications Systems, Incorporated. 11

Footnote 1: This is a requirement with Fabric OS v6.0.0.

© 2008 Brocade Communications Systems, Incorporated.

ICL0 (bottom port) must be connected

Cables can be cross connected to the

ICL Cables are 2 Meters in length

© 2008 Brocade Communications Systems, Incorporated. 12

Footnote 1: Cables and connectors are color coded to avoid misconnection.

© 2008 Brocade Communications Systems, Incorporated.

© 2008 Brocade Communications Systems, Incorporated. 13

Footnote 1: All ICL cables are 2 Meters in length.

© 2008 Brocade Communications Systems, Incorporated.

ICL cable and port connectors will be colored coded

© 2008 Brocade Communications Systems, Incorporated. 14

© 2008 Brocade Communications Systems, Incorporated.

512 Gbit/sec total bandwidth (1 Tbit/sec bidirectional)

ICL connections can only be used to connect two DCX chassis

ICL License is required on each DCX

© 2008 Brocade Communications Systems, Incorporated. 15