Beruflich Dokumente
Kultur Dokumente
Brocade SilkWorm 4100 Design Guide Contents Introduction......................................................................................................................................3 Architecture of the SilkWorm 4100.................................................................................................3 4 Gbit/sec Is the Strategic Industry Direction..................................................................................4 SilkWorm 4100 SAN Design...........................................................................................................4 SilkWorm 4100 SAN Deployment ..................................................................................................5 SilkWorm 4100 SAN Management .................................................................................................6 Fabric OS 4.4 Overview ..................................................................................................................6 Summary ..........................................................................................................................................7 Appendix A: 4 Gbit/sec Fibre Channel Line Rate ...........................................................................7 Appendix B: Link Balancing and Aggregation .............................................................................10 Appendix C: Distance Extension ...................................................................................................17
Introduction
Brocade has recently introduced its fourth-generation switch, the SilkWorm 4100. The SilkWorm 4100 is the most feature-rich and advanced switch introduced by any vendor to date. This paper describes the technical features and benefits of the SilkWorm 4100 in detail. From time to time, this paper will refer to the Brocade Design Deployment and Management guide (DDM), which can be obtained from the Brocade Partner Web site. In addition, this paper will cover the specific enhancements of the SilkWorm 4100 but will refer to the DDM for standard SAN guidelines.
Brocade SilkWorm 4100 Design Guide 1. If the SAN is extended over dark fiber, xWDM, or SONET, the SilkWorm 4100 has extended distance capabilities described in Appendix C: Distance Extension. These capabilities can enhance performance greatly over longer distances, because the SilkWorm 4100 supports up to 255 buffer credits on a single port. This allows full-speed 1 Gbit/sec operation up to approximately 500 km, 2 Gbit/sec operation at approximately 250 km, or 4 Gbit/sec operation at approximately 100 km. 2. If the SilkWorm 4100 is part of an exclusively 4 Gbit/sec network or a certain part has 4 Gbit/sec ISLs or trunks, organizations should consider reducing the total amount of ISLs by half, assuming that all the nodes are still 2 Gbit/sec. One side note is that a minimum of two ISLs between any edge switch and the core is required to provide high availability. If the fabric is mixed speeds (has both 2 Gbit/sec and 4 Gbit/sec nodes), the ISL count will likely be something in between, depending on traffic patterns. Note that when connecting SilkWorm 4100s, the platform supports 8-way trunking and DPS between trunks. These features alone can improve performance as much as 4 Gbit/sec interfaces in many cases, and DPS in particular can operate even when connecting the SilkWorm 4100 to existing 2 Gbit/sec switches. Further information on this can be found in Appendix B: Link Balancing and Aggregation. Deploying the SilkWorm 4100 will help ensure investment protection and reduce deployment costs: new SAN designs will require fewer links to achieve the same performance, which increases ROI and reduces management overhead.
Brocade SilkWorm 4100 Design Guide auto-negotiate, 1 being 1 Gbit/sec, 2 being 2 Gbit/sec, and so on. It is also advised to log a call with support providers or directly with Brocade in the case of a speed negotiation issue. Depending on which Brocade Partner provides the SilkWorm 4100, it will be delivered with one of two different rail kits. One option is fixed rails, which simply fixes the switch into the rack, and a second option is sliding rails that allow the SilkWorm 4100 to slide in and out of the cabinet.
Brocade SilkWorm 4100 Design Guide o Non-disruptive SecModeEnable o Secure Fabric OS over gateway support o Support for up to 15 different user accounts o Secure Fabric OS scalability up to 2560-port fabrics FDMI enhancements such as FDMI hostname support (all products) SNMP enhancements such as link up/down support and RAS errorlog (all products)
Summary
With a cross-sectional internal bandwidth of a quarter of a terabit, the SilkWorm 4100 is the highestperforming 32-port SAN switch in the industry by a wide margin. It is the optimal choice for organizations that want the benefits of a SAN switch at a low entry price point with the option to scale on a pay-as-you-grow basis at a low incremental cost. The SilkWorm 4100 can meet mission-critical test criteria and offers significant advantages as organizations develop and grow. The proven intelligence of the industry-leading SilkWorm product family gives organizations the investment protection of Brocade storage networking solutions and the industrys largest and most developed partner ecosystem. With advanced flexibility, performance, and high-availability features, the SilkWorm 4100 can improve resource utilization and administrator productivity to lower the overall storage total cost of ownership.
Brocade SilkWorm 4100 Design Guide For these and many other reasons, the industry acknowledged that 2 Gbit/sec speeds were no longer sufficient for storage networks. The choice was whether to use 4 or 10 Gbit/sec as the primary strategy. It turned out that 4 Gbit/sec had substantial technical advantages related to deployment, and provided at least the same performance benefits as 10 Gbit/sec. Hosts and storage devices that were exceeding the 2 Gbit/sec interface capacity were not doing so by a large amount. Some tape drives were designed to stream at between 3 and 4 Gbit/sec, and some hosts could match these speeds. However, only a small number of the highest-end systems in the world could exceed 4 Gbit/sec, and even these could not generally sustain 10 Gbit/sec streams. Perhaps the biggest barrier to widespread deployment of 10 Gbit/sec was its innate incompatibility with existing infrastructure. It required different optical cables, used different media, and was not backward compatible with 1 or 2 Gbit/sec. Needing to rip and replace all HBAs and storage controllers at once, not to mention an entire data center cable plant, would not only be prohibitively expensive but operationally impossible in the always-on data centers that power todays global businesses. It became clear because of these factors that the optimal speed for nodes would be 4 Gbit/sec. However, there was still a case to be made for ISLs at 10 Gbit/sec. Replacing the optical infrastructure would be less of a technical issue with backbone connections, because there are always fewer of them than there are node connections. Additionally, some high-end installations require that switch-to-switch connections run faster than 4 Gbit/sec. Indeed, some networks require backbones to run at far higher than 10 Gbit/sec speeds. No matter how fast an individual interface can be made, there always seems to be an application that needs more bandwidth. Brocade decided to solve this with trunking for 4 Gbit/sec interfaces, giving 4 Gbit/sec networks performance superiority over 10 Gbit/sec while still lowering costs and simplifying deployments. (Brocade can create a single 256 Gbit/sec balanced path using 4 Gbit/sec trunking plus DPS.) Another technical factor that had to be considered was network redundancy. Most organizations configure links in pairs, so that there will be no outage if one link should fail. With a single 10 Gbit/sec link, any component failure will result in an outage, which means that the minimum realistic configuration between two switches is 20 Gbit/sec (two 10 Gbit/sec links). Relatively few applications require so much bandwidth between each pair of switches. Given the cost of 10 Gbit/sec interfaces, redundancy would be harder to cost-justify, and from a network engineering point of view, any factor that motivates organizations towards non-redundant deployments is a negative market force. To fully appreciate this, consider the performance parity case. If three 4 Gbit/sec links are configured, and one fails, the channel is 33 percent degraded. For a network with the exact same performance requirement, a single 10 Gbit/sec link is needed, which is more expensive than the three 4 Gbit/sec interfaces combined, and requires more expensive single-mode optical infrastructure. If that link fails, the network has an outage, because 100 percent of bandwidth is lost. This requires a second 10 Gbit/sec link to be provisioned, with all the associated cost and deployment complexity, even though the additional performance is not required. If a 10 Gbit/sec proponent were to argue that two times the performance were really needed, the 4 Gbit/sec proponent could configure six 4 Gbit/sec links, which would still cost far less, have higher availability, and perform identically. In fact, the larger the configuration and the higher the performance requirements, the greater the advantages of 4 Gbit/sec from both a performance and availability point of view. All of this adds up to substantial technical advantages for 4 Gbit/sec above 10 Gbit/sec for the vast majority of deployment cases. Until mainstream nodes can saturate 4 Gbit/sec channels, 4 Gbit/sec Fibre Channel is likely to remain the mainstream interface speed for storage networks. 8
Brocade intends to offer 4 Gbit/sec blades that can coexist with SilkWorm 24000 2 Gbit/sec blades in the same chassis, but at least two other vendors require forklift chassis upgrades. Be sure to ask if a 2 Gbit/sec chassis purchased today will support 4 and 10 Gbit/sec blades in the future, and if these can coexist with existing blades.
Brocade SilkWorm 4100 Design Guide The time lag between edge switches and directors is not considered to be a problem: the industry does not believe that 2 Gbit/sec is by any means obsolete. Most organizations do not immediately require 4 Gbit/sec interfaces, and many will be able to use their 2 Gbit/sec switches for years to come. Some time after the first 4 Gbit/sec switches ship, node vendors will start to come out with 4 Gbit/sec interfaces. Most organizations will not have an immediate need for 4 Gbit/sec HBAs, for example, so it is likely that only net-new installations will use this speed. (This is one reason backward compatibility with 1 and 2 Gbit/sec was so important.) By the end of 2005, it is expected that all major vendors will ship 4 Gbit/sec interfaces by default on products in every segment, and that the vast majority of new deployments will use this speed almost exclusively.
Brocade SilkWorm 4100 Design Guide DLS does a best-effort job of distributing I/O by balancing source port routes. However, some ports might still carry more traffic than others, and DLS cannot predict which routes will be hot when it sets up routes, since they must be allocated before I/O commences. Also, traffic patterns tend to change over time, so no matter how routes were distributed initially, it would still be possible to have hot spots appear later, and changing the route allocation randomly at runtime would cause out-of-order delivery.2 Balancing the number of routes allocated to a given path is not the same as balancing I/O, and so DLS does not do a perfectly even job of balancing traffic. The DLS feature is useful, and since it is free and works automatically, some form of it is used in virtually all Brocade multi-switch fabrics. However, DLS does not solve all performance problems, so there is a need for more evenly balanced methods. Such methods require hardware support, since path selection needs to be done on a frame-by-frame basis, and doing this in software would have a disastrous performance impact. The trunking methods described below fill this need.
One Fibre Channel switch vendor attempted to use a similar method to change routes while I/O was in flight. As it quickly discovered, this caused massive degrees of out-of-order delivery: something that violates Fibre Channel standards and, as a practical concern, breaks many applications. 3 There are limitations to the amount of skew than an ASIC can tolerate. These are high enough limits that they do not generally apply. The real-world applicability of the limitation is that it is not possible to configure one link in a trunk to go clockwise around a large dark fiber ring while another link goes counterclockwise. As long as the differences in cable length are measured in a few tens of meters or less, there will not be an issue. However if the differences are larger, a trunk group cannot form. Instead, the switch would create two separate ISLs, and use either DLS or DPS to balance them.
11
Figure 1. Frame-Level Trunking Concept Frame-Level Trunking Advantages The main advantage of frame-level trunking is that it provides optimal performance: a trunk group using this method truly aggregates the bandwidth of its members. The feature also increases availability by allowing non-disruptive addition of members to a trunk group and minimizing the impact of failures. Frame-Level Trunking Limitations On Brocade 2 Gbit/sec Fibre Channel switches,4 multiple groups of two to four 2 Gbit/sec links each can be combined into balanced pipes of 4 to 8 Gbit/sec each. Figure 1 shows a simple case of this between two SilkWorm 3850 switches. Figure 2 shows another trunked configuration, this time between two SilkWorm 3850 edge switches and two SilkWorm 24000 directors using two 2-port trunk groups each.
Such as the SilkWorm 3200, 3250, 3600, 3800, 3850, 3900, 12000, and current 24000 blades.
12
Figure 2. Frame Trunking plus DLS On newer switches such as the SilkWorm 4100 and forthcoming SilkWorm 24000 blades, it is possible to configure multiple groups of up to eight 4 Gbit/sec links each. The effect is just like Figure 1, except the performance per-trunk is quadrupled: instead of forming multiple 8 Gbit/sec pipes, the SilkWorm 4100 can create balanced 32 Gbit/sec pipes (64 Gbit/sec full-duplex). In connecting a SilkWorm 4100 or other future 4 Gbit/sec switches to a 2 Gbit/sec switch, a lowest common denominator approach is used, meaning that the trunk groups will be limited to four by 2 Gbit/sec instead of eight by 4 Gbit/sec. As mentioned previously, frame-level trunking requires that all ports in a given trunk must reside within an ASIC port group on each end of the link. While a frame-level trunk group will outperform either DLS or DPS solutions every time, using links only within port groups limits configuration options. The solution is to combine frame-level trunking with one of the other methods, as shown in Figure 2. This shows frame-level trunking operating within port groups and DLS operating between trunks. Even though the ISLs are all within a trunk group on the edge switches, the four links could not form a trunk because they go to different port groups on different cores. Changing that would affect availability. On 2 Gbit/sec switches, port groups are built on contiguous 4-port groups called quads. On a SilkWorm 3250, for example, there are two quads: ports 03 and ports 47. On 4 Gbit/sec switches like the SilkWorm 4100, trunking port groups are built on contiguous 8-port groups called octets. In that product, there are four octets: ports 07, 815, 1623, and 2431. It is also possible to configure multiple trunks within a port group. For example, on the SilkWorm 3850, it is possible to configure one trunk on ports 1213 and a separate trunk on ports 14-15. These trunks could go to different core switches in a core-to-edge network or to different blades on a director. Figure 2 illustrates this case as well. It is also important to understand how frame-level trunking interacts with Brocade Extended Fabrics software.
13
Brocade SilkWorm 4100 Design Guide Each ASIC has a certain number of buffer-to-buffer credits. On 2 Gbit/sec switches, these are sufficient to operate all four ports in a quad in any mode over short distances.5 Over longer distances, it is necessary to limit the functions of some of the ports. For example, all 2 Gbit/sec switches can support a 4port trunk at 10 km, but at 50 km only a 2-port trunk is supported with the other two ports limited to node attachment.6 At 100 km, only one long-distance port can be configured, which precludes trunking. It is possible to configure multiple 100 km links on the SilkWorm 3016, 3250, 3850, and 24000: up to one per quad. However, since these cannot be in the same port group, it is necessary to use DLS to balance them, not trunking. 4 Gbit/sec switches such as the SilkWorm 4100 have more flexible support for trunking over distance. In the 4 Gbit/sec switches, buffers are shared across the entire chip, not limited by quads or octets. It is possible, for example, to configure up to 8-port by 4 Gbit/sec trunks at 50 km (32 Gbit/sec trunk group) or 4-port by 4 Gbit/sec trunks at 100 km (16 Gbit/sec trunk group). In some cases, it might be desirable to configure trunks using 2 Gbit/sec links. For example, the trunk group might cross a DWDM that does not have 4 Gbit/sec support. In this case, an 8-port 2 Gbit/sec trunk can span up to 100 km.
Short can be about 25 km without performance degradation and longer if full 2 Gbit/sec throughput per port is not required. For long-distance trunking, certain minimum Fabric OS firmware versions might be required. 7 Effectively is an important word. This does happen almost all the time, but there could be some Fibre Channel devices that do not map SCSI operations to exchange IDs, and other protocols like FICON and IP/FC do not necessarily behave this way either.
6
14
Brocade SilkWorm 4100 Design Guide Exchange-Level Trunking Advantages While performance might be slightly lower in some cases, there are also several advantages to the DPS method. For one thing, exchange-level trunking does not need to occur within ASIC port groups the way frame-level trunking must be configured. This allows load balancing across different core switches in a core-to-edge network or different blades in a director, rather than mere DLS route balancing. Figure 3 shows an example of this.
Figure 3. Frame Trunking plus DLS In addition, DPS is not exclusive of frame-level trunking. It is possible to balance several groups of ports using the frame-level method, and then balance between the resulting trunk groups using the exchange-level method. This provides the optimal balance of performance (frame-level is faster) and availability (exchange-level provides the flexibility to balance high-availability network topologies.) This is also illustrated in Figure 3. Next, DPS can balance I/O sent from an enabled platform to any other platform even if the destination does not support the feature. Path selection is made by the transmitting switch, and the receiver does not need to do anything special to ensure in-order delivery. This allows full backward compatibility with existing switches and provides some performance benefit even if not all switches in a fabric are using the latest technology. This is shown in Figure 4.
15
Figure 4. DPS in a Mixed Fabric Finally, DPS can balance I/O across long-distance configurations not supported by frame-level trunking. For example, if there are two links configured between two sites that take substantially different paths, there might be too much skew to form a frame-level trunk. DPS would still be able to balance these links, since it does not rely on de-skew timers for in-order delivery. Figure 5 shows an example of this.
Brocade SilkWorm 4100 Design Guide performance right away, a flexible design will facilitate a high-performance implementation when needed in the future.
The SilkWorm 4100 has a total of 1024 buffer credits, which are shared among the 32 ports. Twentyfour of these credits are used for the embedded port, and the rest are available for user consumption. With the SilkWorm 4100, F_ and FL_Ports receive eight buffer credits by default, and local E_Ports (L0 mode) receive 26 buffer credits. (This is the same amount of credit for both 2 and 4 Gbit/sec ports.) A minimum of eight buffer credits is reserved for each port so no ports are ever starved of credits. The rest is available in a buffer pool, which can be configured for use by any of the 32 ports. With the SilkWorm 4100, buffers are no longer automatically assigned out of the pool as in previous products, because line-speed can easily be achieved with eight buffer credits for a local device. Assigning more credits to a node can be accomplished with the portCfgLongDistance command as long as the Extended Fabrics license is installed. Long-distance links are no longer automatically assigned additional credits unless long-distance mode is configured for E_Ports. A single port can be assigned up to 255 buffer credits, which equates to approximately500 km at 1 Gbit/sec, approximately 250 km at 2 Gbit/sec, and approximately 125 km at 4 Gbit/sec. The table below illustrates the different configurations over distance supported by the SilkWorm 4100, including frametrunked links over distance. To achieve the longer distances, it is necessary to have a SilkWorm 4100 (or future 4 Gbit/sec blades for the SilkWorm 24000) on both ends of the long-distance link. Table 1. Extended Trunking Distances and Data Transfer Speeds 4 Gbit/sec Ports Trunked 8 4 2 1 Throughput 32 Gbit/sec 16 Gbit/sec 8 Gbit/sec 4 Gbit/sec Distance 30 km 60 km 125 km 125 km 2 Gbit/sec Throughput 16 Gbit/sec 8 Gbit/sec 4 Gbit/sec 2 Gbit/sec Distance 60 km 125 km 175 km 250 km 1 Gbit/sec Throughput NA NA NA 1 Gbit/sec Distance NA NA NA 500 km
2005 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the Brocade B weave logo, Fabric OS, Secure Fabric OS OS, and SilkWorm are registered trademarks of Brocade Communications Systems, Inc., in the United States and other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. Important Notice: Use of this paper constitutes consent to the following conditions. This document is supplied AS IS for informational purposes only, without warranty of any kind, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this book may require an export license from the United States government.
17