Ip Fabric Evd PDF

EXTREME VALIDATED DESIGN
Extreme IP Fabric Architecture
53-1004890-04
April 2018
© 2018, Extreme Networks, Inc. All Rights Reserved.
Extreme Networks and the Extreme Networks logo are trademarks or registered trademarks of Extreme Networks, Inc. in the United States and/or other
countries. All other names are the property of their respective owners. For additional information on Extreme Networks Trademarks please see
www.extremenetworks.com/company/legal/trademarks. Specifications and product availability are subject to change without notice.
© 2017, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other
countries. Other brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at www.brocade.com/en/legal/
brocade-Legal-intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,
equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without
notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade
sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the
United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this
document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license
agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
2 Extreme IP Fabric Architecture

Contents
List of Figures ................................................................................................................................................................................ 5
Preface ............................................................................................................................................................................................ 7
Extreme Validated Designs....................................................................................................................................................... 7
Purpose of This Document ....................................................................................................................................................... 7
Target Audience ....................................................................................................................................................................... 7
Authors ..................................................................................................................................................................................... 8
Document History ..................................................................................................................................................................... 8
About Extreme Networks .......................................................................................................................................................... 8
Introduction .................................................................................................................................................................................... 9
Terminology .............................................................................................................................................................................. 9
IP Fabric Overview ....................................................................................................................................................................... 11

Evolution of Data CenterFabrics .............................................................................................................................................. 12
Layer 2 Aggregation ........................................................................................................................................................ 12
Layer 2 Fabric Architectures ............................................................................................................................................ 13
IP Fabric (AKA Routing to ToR) ....................................................................................................................................... 14
Leaf-Spine Architecture .......................................................................................................................................................... 15
Leaf-Spine Layer 3 Clos Topology (Two-Tier) ................................................................................................................. 15
Optimized 5-Stage Layer 3 Clos Topology (Three-Tier) .................................................................................................. 16
IP Fabric Control Plane ........................................................................................................................................................... 17
Pervasive eBGP .............................................................................................................................................................. 18
iBGP for Routing Inside a PoD ........................................................................................................................................ 19
IP Fabric Validated Designs........................................................................................................................................................... 21

Pervasive eBGP ..................................................................................................................................................................... 21
Hardware Matrix ..................................................................................................................................................................... 22
IP Fabric Configuration ........................................................................................................................................................... 22
Common Configuration on All Nodes in the Fabric .......................................................................................................... 23
Node ID Configuration for VDX Platforms................................................................................................................. 23
Fabric Infrastructure Links ........................................................................................................................................ 24
Loopback Interfaces and Router ID .......................................................................................................................... 25
Server-Facing Links and Networks on ToRs ................................................................................................................... 26
VDX ToRs ................................................................................................................................................................. 26
SLX 9140 ToRs ........................................................................................................................................................ 29
MCT on SLX 9540 EdgeLeafs ....................................................................................................................................... 32
eBGP Control-Plane Configuration .................................................................................................................................. 35
Deployment Model-1: eBGP Configuration for Optimized 5-Stage Clos ................................................................... 35
Deployment Model-2: eBGP Configuration for 3-Stage Clos Fabric ......................................................................... 42
Illustration Examples ............................................................................................................................................................... 47
Network Reachability Between Racks andPoDs .............................................................................................................. 47
Verification ................................................................................................................................................................ 49
Design Considerations .................................................................................................................................................................. 54

Scale................................................................................................................................................................................ 54
Extreme IP Fabric Architecture 3

RecommendationsforISLPorts in aVDX vLAGPair Leaf .................................................................................................... 54
Recommendations for ICL Ports in an SLX MCT Pair ..................................................................................................... 54
Generalized TTL Security Mechanism for BGP (GTSM) ................................................................................................. 55
Appendix—Configuration of the Nodes ......................................................................................................................................... 57

SLX 9140 MCT Pair ToR/Leaf .............................................................................................................................................. 57
Peer1 ............................................................................................................................................................................... 57
Peer2 ............................................................................................................................................................................... 64
SLX 9240 Spine ...................................................................................................................................................................... 70
SLX 9850 Super-Spine ......................................................................................................................................................... 74
SLX 9540 EdgeLeaf .............................................................................................................................................................. 78
VDX vLAG Pair Leaf ............................................................................................................................................................... 81
References .................................................................................................................................................................................... 95

List of Figures
Figure 1 on page 12—L2 Aggregation
Figure 2 on page 13—L2 Fabric
Figure 3 on page 14—IP Fabric (Routing to ToR)
Figure 4 on page 15—Leaf-Spine L3 Clos Topology
Figure 5 on page 16—Optimized 5-Stage L3 Clos Topology
Figure 6 on page 18—IP Fabric with eBGP as the Control Protocol

Figure 7 on page 19—IP Fabric with iBGP as the Control Protocol Inside a PoD
Figure 8 on page 21—Pervasive eBGP in an Optimized 5-Stage IP Fabric
Figure 9 on page 48—Connectivity Between the Racks and PoDs

Preface
Preface
• Extreme Validated Designs. .................................................................................................................................. 7
• Purpose of This Document. ................................................................................................................................... 7
• Target Audience. .................................................................................................................................................... 7
• About the Authors. ................................................................................................................................................. 7
• Document History. ................................................................................................................................................. 8
• About Extreme Networks. ..................................................................................................................................... 8
Extreme Validated Designs

Helping customers consider, select, and deploy network solutions for current and planned needs is our mission. Extreme Validated
Designs offer a fast track to success by accelerating that process.
Validated designs are repeatable reference network architectures that have been engineered and tested to address specific use cases and
deployment scenarios. They document systematic steps and best practices that help administrators, architects, and engineers plan,
design, and deploy physical and virtual network technologies. Leveraging these validated network architectures accelerates deployment
speed, increases reliability and predictability, and reduces risk.
Extreme Validated Designs incorporate network and security principles and technologies across the ecosystem of service provider, data
center, campus, and wireless networks. Each Extreme Validated Design provides a standardized network architecture for a specific use
case, incorporating technologies and feature sets across Extreme products and partner offerings.
All Extreme Validated Designs follow best-practice recommendations and allow for customer-specific network architecture variations that
deliver additional benefits. The variations are documented and supported to provide ongoing value, and all Extreme Validated Designs
are continuously maintained to ensure that every design remains supported as new products and software versions are introduced.
By accelerating time-to-value, reducing risk, and offering the freedom to incorporate creative, supported variations, these validated
network architectures provide a tremendous value-add for building and growing a flexible network infrastructure.
Purpose of This Document

This Extreme Validated Design provides guidance for designing and implementing an IP fabric in a data center network using Extreme
hardware and software. It details the reference architecture for deploying an IP fabric using the SLX platform as the spine and super-
spine in 3-stage and 5-stage Clos topologies.
The design practices documented here follow best-practice recommendations, but variations to the design are supported as well.
Target Audience
This document is written for Extreme systems engineers, partners, and customers who design, implement, and support data center
networks. This document is intended for experienced data center architects and engineers. It assumes that the reader has a good
understanding of data center switching and routing features.

Preface
Authors
• Krish Padmanabhan
Sr Principal Engineer, System and Solution Engineering
• Eldho Jacob
Principal Engineer, System and Solution Engineering
The authors would like to acknowledge the following for their technical guidance in developing this validated design:
• Abdul Khader
Director, System and Solution Engineering
• Vivek Baveja
Director, Product Management
Document History
Date Part Number Description
December 8, 2016 53-1004890-01 Initial release.
April 21, 2017 53-1004890-02 Included new SLX platforms. Refer to the Hardware Matrix on page 20 for details
on the platforms and their PINs.
January 2018 53-1004890-03 Updated document to reflect Extreme's acquisition of Brocade's data center
networking business.
April 2018 53-1004890-04 Format change
About Extreme Networks

Extreme Networks® (NASDAQ: EXTR) networking solutions help the world’s leading organizations transition smoothly to a world where
applications and information reside anywhere. This vision is designed to deliver key business benefits such as unmatched simplicity,
non-stop networking, application optimization, and investment protection.
Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity
and cost while enabling virtualization and cloud computing to increase business agility.
To help ensure a complete solution, Extreme Networks partners with world-class IT companies and provides comprehensive education,
support, and professional services offerings. (www.ExtremeNetworks.com)

Introduction
Introduction
The IP fabric architecture is targeted for large enterprises and data centers planning to migrate from traditional Layer 2 fabrics to a
Layer 3 fabric. This document is about building IP fabrics with Extreme VDX and SLX switching platforms. The SLX series of switches
and routers are Extreme’s next-generation platforms addressing the scale requirements of MSDC customers.
The configurations and design practices documented here are fully validated and conform to the IP fabric reference
architectures. The intention of this Extreme Validated Design document is to provide reference configurations and document best
practices for building cloud-scale data-center networks using VDX and SLX switches and IP fabric architectures.
It should be noted that this document does not cover the network virtualization or Layer 2/Layer 3 multitenancy aspects of IP fabric.
Network virtualization in IP fabric is covered in the Network Virtualization in IP Fabric with BGP EVPN1 Extreme Validated Design
document.
Terminology
Term Description
ARP Address Resolution Protocol
AS Autonomous System
ASN Autonomous System Number
BGP Border Gateway Protocol
eBGP External Border Gateway Protocol
ECMP Equal Cost Multi-Path
iBGP Internal Border Gateway Protocol
IP Internet Protocol
MCT Multi-Chassis Trunk
ND Neighbor Discovery
NLRI Network Layer Reachability Information
ORF Outbound Route Filtering
PoD Point of Delivery
ToR Top of Rack switch
URIB Unicast Route Information Base
vLAG Virtual Link Aggregation Group
VLAN Virtual Local Area Network
VM Virtual Machine

Introduction

IP Fabric Overview
IP Fabric Overview
• Evolution of Data Center Fabrics. .......................................................................................................................11
• Leaf-Spine Architecture. ......................................................................................................................................15
• IP Fabric Control Plane. ........................................................................................................................................ 17
Extreme IP fabric provides a Layer 3 Clos deployment architecture for data center sites. It is a paradigm shift from traditional Layer 2
switching fabrics. With Extreme IP fabric, all links in the Clos topology are Layer 3 links.
In comparison with the traditional 2-tier access/aggregation topologies and L2 fabrics where the L2/L3 demarcation happens on a
device typically more than a hop away, the L2/L3 boundary in IP fabric is pushed to the ToR or the leaf node itself (AKA routing to the
ToR). Leafs advertise the server subnets attached to them directly into the routing control-plane protocol. Modern data centers zeroed in
on BGP as the preferred control-plane protocol. Because the infrastructure is built on IP, advantages like the following are leveraged:
loop-free communication using industry-standard routing protocols, ECMP, a very high solution scale, and standards-based
interoperability.

IP Fabric Overview
Evolution of Data Center Fabrics

Layer 2 Aggregation
Figure 1 depicts the aggregation of several L2 switches into a pair of aggregation device in MLAG mode. This aggregation pair acts as
the L2/L3 boundary; in other words, it aggregates all server VLANs and acts as a first-hop router for those VLANs. It also provides
connectivity between the data center and the external networks. (The L2/L3 boundary is pushed to a third layer in certain scale-up
topologies.)
FIGURE 1 L2 Aggregation
This model provides L2 extension naturally between racks. But there are trade-offs. Most of the network functions are concentrated on
the aggregation pair. The number of ToRs, VLANs, MAC entries, IP subnets, ARP/ND entries, route scale, and so on is determined by
these two devices for the entire fabric. The scale of the network and diameter (or number of racks) is limited by these two devices for the
entire fabric. The VLANs or broadcast domains must be pruned properly according to the membership or interest in various racks to
avoid large broadcast domains.

IP Fabric Overview
Layer 2 Fabric Architectures

These fabrics aim to provide Layer 2 services by borrowing the routing concepts of Layer 3 networks, such as ECMP. For example:
VCS fabric. Host reachability is computed based on reachability to the switch to which the host is connected. (Much like a path to a
network is behind the router to which the subnet is connected.) The fabric can be modeled as a leaf/spine Clos fabric as shown in
Figure 2.
Typically the spines would be the first-hop routers for the server VLANs. Compared to the model shown in Figure 1, this allows scaling
out the number of spines to increase the bandwidth available to the ToRs, thereby reducing the oversubscription.
FIGURE 2 L2 Fabric

IP Fabric Overview
IP Fabric (AKA Routing to ToR)

For very large enterprise networks or cloud service providers (SaaS or PaaS providers), the L2 fabrics or L2 scale-out architectures are
not scalable enough to meet the infrastructure requirements. IP fabric, as the name suggests, is built on IP; i.e., the fabric nodes—the
ToRs and spines—are interconnected using L3 links instead of switch ports or TRILL-capable ports, and a Layer 3 routing control
protocol runs between these nodes for reachability between the ToRs. This type of fabric takes advantage of all the best practices of L3
routing and data plane. A smaller scale topology might benefit from a link-state protocol such as OSPF. Large-scale topologies,
however, typically use BGP. Extreme validated design recommends BGP as the protocol for underlay network reachability.
In IP fabric, the L2/L3 boundary is pushed to the ToR. The server VLANs are terminated in the ToR of each rack. This also gives the
flexibility of reusing the same VLAN numbers on different racks, but still keeping them as distinct broadcast domains. This is a much
more scalable architecture since the ToR handles only a rack of compute/storage and appliances compared to the spines/aggregation
devices handling several racks of resources. The route scale is still important, but one may choose to use the techniques of route
aggregation, default routing to spines, route filtering, etc.
IP fabric is targeted at customers who have simple L3 requirements of aggregating several L3 subnets and those who would like to build
an IP underlay infrastructure to meet the requirements of overlay networking, either host-based or network-based. The overlay-based
network virtualization is not within the scope of this document.
FIGURE 3 IP Fabric (Routing to ToR)

IP Fabric Overview
Leaf-Spine Architecture
Leaf-Spine Layer 3 Clos Topology (Two-Tier)
The leaf-spine topology has become the de facto standard for networking topologies when building medium- to large-scale data center
infrastructures. The leaf-spine topology is adapted from Clos telecommunications networks. The Extreme IP fabric within a PoD
resembles a two-tier or 3-stage folded Clos fabric. The two-tier leaf-spine topology is shown in Figure 4. The bottom layer of the IP
fabric has leaf devices (top-of-rack switches), and the top layer has spines. The role of the leaf is to provide connectivity to the endpoints
in the data center network. These endpoints include compute servers and storage devices, as well as other networking devices like
routers, switches, load balancers, firewalls, and any other physical or virtual networking endpoints. Because all endpoints connect only to
the leaf, policy enforcement, including security, traffic-path selection, QoS marking, traffic policing, and shaping, is implemented at the
leaf.
The leafs act as the L2/L3 boundary for the server segments in an IP fabric.
The role of the spine is to provide connectivity between leafs. The major role of the spine is to participate in the control-plane and data-
plane operations for traffic forwarding between leafs.
FIGURE 4 Leaf-Spine L3 Clos Topology
As a design principle, the following requirements apply to the leaf-spine topology:

• Each leaf connects to all spines in the network through 40-Gbps Ethernet links.
• Spines are not interconnected with each other.
• Leafs are not interconnected with each other for data-plane purposes. (Two leafs may be interconnected for control-plane
operations such as forming a server-facing vLAG. This is referred to as vLAG pair leaf.)
• The network endpoints do not connect to the spines.
This type of topology has the predictable latency and also provides the ECMP forwarding in the underlay network. The number of hops
between two leaf devices is always two within the fabric. This topology also enables easier scale-out in the horizontal direction as the data
center expands and is limited by the port density and bandwidth supported by the spine devices.
This validated design recommends the same hardware in the spine layer. Mixing different hardware is not recommended.

IP Fabric Overview
Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)

Multiple PoDs based on leaf-spine topologies can be connected for higher scale in an optimized 5-stage folded Clos (three-tier)
topology. This topology adds a new tier to the network, known as a super-spine. Super-spines function similar to spines: BGP control-
plane and data-plane forwarding between the PODs and from the PODs to destinations outside the fabric via the border leafs. No
endpoints are connected to the super-spines. Figure 5 shows four super-spine switches connecting the spine switches across multiple
data center PoDs.
The connection between the spines and the super-spines follows the Clos principles:
• Each spine connects to all super-spines in the network.
• Neither spines nor super-spines are interconnected with each other.
FIGURE 5 Optimized 5-Stage L3 Clos Topology

IP Fabric Overview
IP Fabric Control Plane

From the control-plane perspective, there are two deployment options:
• Pervasive eBGP
• iBGP within a POD
In this validated design, we recommend eBGP as the control plane in the fabric. IP fabric is also referred to as routing to ToR in
comparison with the traditional multitier access and aggregation networks. In those traditional networks, the L2/L3 boundary is on the
aggregation devices. In IP fabric, this boundary is moved to the edge or leaf. The leaf nodes directly advertise their server subnets into
the BGP control plane. Reachability between the server subnets on various racks is established using control-plane learning.
Moving the intelligence to the leaf helps with scaling out the number of spines depending on the bandwidth or oversubscription issues in
the network. For instance, the ratio between the bandwidth of the server ports and the uplinks ports (from leaf to spine layer). If the
oversubscription is too high, additional spines may be added.

IP Fabric Overview
Pervasive eBGP
This deployment model refers to the usage of eBGP peering between the leaf and the spine in the fabric. This design using eBGP as a
routing protocol within the data center is based on the IETF draft Use of BGP for Routing in Large-Scale Data Centers2.
In this model, each leaf node is assigned its own autonomous system (AS) number. The other nodes are grouped based on their role in
the fabric, and each of these groups is assigned a separate AS number, as shown in Figure 6. Using eBGP in an IP fabric is simple and
also provides the ability to apply BGP policies for traffic engineering on a per-leaf or per-rack basis since each leaf or rack in a PoD is
assigned a unique AS number. Private AS numbers are used in the fabric. One design consideration for the AS number assignment is that
a 2-byte AS number provides a maximum of 1023 private AS numbers (ASN 64512 to ASN 65534); if the IP fabric is larger than 1023
devices, we recommend using 4-byte private AS numbers (ASN 4,200,000,000 to 4,294,967,294).
• Each leaf in a PoD is assigned its own AS number.

• Leafs advertise the server subnets directly into BGP.
• All spines inside a PoD belong to one AS.
• All super-spines are configured in one AS.
• Edge or border leafs belong to a separate AS.
• Each leaf peers with all spines using eBGP.
• Each spine peers with all super-spines using eBGP.
• Each border leaf peers with all super-spines using eBGP.
• No BGP peering occurs between nodes in the same layer.
FIGURE 6 IP Fabric with eBGP as the Control Protocol

IP Fabric Overview
iBGP for Routing Inside a PoD

This model is given here for informational purposes and is not included in the validated design section of this document.
In this deployment model, each PoD and edge services PoD is configured with a unique AS number, as shown in Figure 7. The spines
and leafs in a PoD are configured with the same AS number. The iBGP design is different than the eBGP design because iBGP must be
fully meshed with all BGP-enabled devices in an IP fabric. In order to avoid the complexities of a full mesh, spines must act as route
reflectors toward the leaf nodes inside the PoD.
eBGP is used to peer between spines and super-spines. The super-spine layer is configured with a unique AS number; all super-spines
use the same AS number.
FIGURE 7 IP Fabric with iBGP as the Control Protocol Inside a PoD

IP Fabric Overview

IP Fabric Validated Designs

• Pervasive eBGP. .................................................................................................................................................21
• Hardware Matrix. .................................................................................................................................................22
• IP Fabric Configuration ........................................................................................................................................22
• Illustration Examples. ..........................................................................................................................................47
This section provides the details of key deployment models with the validated configuration templates. Extreme validated design
recommends two models for IP fabric deployment; these deployment models are categorized based on how the underlay is designed for
interconnecting leaf, spine, super-spine, and border-leaf nodes. Only the pervasive eBGP model is included in this document for 3- and
5-stage fabrics.
Pervasive eBGP
Figure 8 shows the design for a 5-stage IP fabric using eBGP as the control protocol. Note that the border leafs are connected to the
super-spines in this design. For small topologies, a 3-stage fabric may be sufficient. As shown in Figure 8, each POD is a 3-stage fabric.
For 3-stage fabrics, the border leafs are directly connected to the spines.
FIGURE 8 Pervasive eBGP in an Optimized 5-Stage IP Fabric

Hardware Matrix
TABLE 1 Extreme Switch Platforms Supported in IP Fabric
Places in the Network Platform Minimum Software Version
Leaf Nodes SLX 9140 SLX-OS 17s.1.02
VDX 6740 Network OS 7.2.0a
VDX 6940-144S Network OS 7.2.0a

Spine Nodes SLX 9240 SLX-OS 17s.1.02
SLX 9850 SLX-OS 17r.1.01b
Super-Spine Nodes SLX 9850 SLX-OS 17r.1.01b
Edge or Border Leaf SLX 9140 SLX-OS 17s.1.02
SLX 9540 SLX-OS 17r.1.01b
IP Fabric Configuration
This section covers the provisioning and validation of the IP fabric network topology. The configuration is given in four parts:
1. All common configuration required on all nodes in the fabric.
2. The configuration required on the server-facing side of the ToRs or leafs.
3. The MCT configuration required on edge leafs.
4. The BGP control plane, which is split into two models: 3-stage and 5-stage fabrics.

Common Configuration on All Nodes in the Fabric

Node ID Configuration for VDX Platforms
This configuration is applicable only for the VDX platforms used as a dual ToR or redundant leaf in the IP fabric. SLX platforms do not
require this configuration.
All VDX platforms are set to a VCS ID of 1 by default. Two nodes having a common VCS ID will form a VCS fabric between them. Since
these nodes will be independent in the IP fabric, we need to ensure that they do not form a VCS fabric between them. This is achieved by
configuring a unique VCS ID on each node.
In the validated design, each VDX node is configured with a unique VCS ID. The RBridge ID may be re-used. We recommend using
RBridge ID 1 for individual leafs and RBridge IDs 1 and 2 for the vLAG pair.
!Note: this is done from the exec prompt

POD1-Leaf5# vcs vcsid 409 set-rbridge-id 1
The vLAG pair leaf is assigned its own unique VCS ID, and each node in the vLAG pair has a separate RBridge ID. For example, in the
validated design, Leaf1 is a 2-node vLAG pair.
vLAG Peer 1
POD1-Leaf1-1# vcs vcsid 405 set-rbridge-id 1
vLAG Peer 2
POD1-Leaf1-2# vcs vcsid 405 set-rbridge-id 2
Verify the Configuration
In the following output, RBridge 2 is the principal switch. All configuration for both nodes in the vLAG leaf can be done from this principal
switch.
POD1-Leaf1-1# show vcs

Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 405
VCS GUID : a40a9358-9c72-4bc6-a1e9-86c1c7aff2bd
Total Number of Nodes : 2
Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
1 10:00:00:27:F8:F0:76:E0* 10.20.34.45 Online Online POD1-Leaf1-1
2 >10:00:00:27:F8:F0:5B:B0 10.20.34.46 Online Online POD1-Leaf1-2

Fabric Infrastructure Links
All nodes in the IP fabric—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. These links are referred to as fabric
infrastructure links. In the validated design,
• 40Gbps or 100 Gbps links are used between the nodes. All fabric links between two layers – say Leaf layer and Spine layer –
must be of identical bandwidth. Mixing links of different speeds must be avoided.
• All these links are configured as Layer 3 interfaces with a /31 IPv4 address and a /127 IPv6 address.
• The MTU for these links is set to Jumbo MTU.
• Disable the fabric ISL and trunk features on VDX platforms.
VDX Platforms
interface FortyGigabitEthernet 1/0/49

mtu 9216
no fabric isl enable
no fabric trunk enable
ip mtu 9100
ip proxy-arp
ip address 10.0.1.1/31
ipv6 address fdf8:10:0:1::1/127
ipv6 mtu 9100
no shutdown
!
SLX Platforms
interface Ethernet 3/2
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 address fdf8:10:2:1::/127
ipv6 mtu 9100
no shutdown
!

Loopback Interfaces and Router ID

Each device in the fabric needs one loopback interface with a unique IPv4 address for the purpose of the router ID.
VDX vLAG Pair Leaf/ToR
Note that the configuration for the vLAG ToR is done from the principal switch. Use the show vcs command.
rbridge-id 1
interface Loopback 2
no shutdown
ipv6 address fdf8:10:122:1::2/128
ip address 10.122.1.2/32
!
ip router-id 10.122.1.2
!
rbridge-id 2
no shutdown
ipv6 address fdf8:10:122:1::3/128
ip address 10.122.1.3/32
!
!
Individual Nonredundant VDX Leaf

For the nonredundant VDX leaf, there is just one node.
rbridge-id 1
no shutdown
ipv6 address fdf8:10:122:5::2/128
ip address 10.122.5.2/32
!
!
SLX Platforms
SLX platforms do not need an RBridge configuration as do the VDX platforms.
ipv6 address fdf8:10:125:1::2/128
ip address 10.125.1.2/32
no shutdown
!

Server-Facing Links and Networks on ToRs

VDX ToRs
Server VLANs
interface Vlan 101
interface Vlan 102
...
VE Interfaces for the Server VLANs

VE interfaces are configured to provide the first-hop routing functionality for the hosts in the VLAN. The following configuration
parameters are required:
• The IPv4 address and prefix length for the VLAN subnet
• Jumbo MTU for both IPv4 and IPv6 packet forwarding
rbridge-id 1
interface Ve 101
ipv6 mtu 9000
ip mtu 9000
ip address 10.5.101.1/24
no shutdown
!
!
FHRP for the vLAG Pair ToRs

For the vLAG pair leaf, VRRPe is required for first-hop router functionality in the server VLANs. With VRRPe extension, both switches in
the pair act as the active router in the VLANs. One VRRP group is configured for IPv4 and one is configured for IPv6. Hosts use the
virtual IP as the gateway address.
rbridge-id 1 rbridge-id 2
interface Ve 101 interface Ve 101
ipv6 address fdf8:10:1:101::1/64 ipv6 address fdf8:10:1:101::2/64
ipv6 mtu 9000 ipv6 mtu 9000
ipv6 vrrp-extended-group 100 ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:101::254 virtual-ip fdf8:10:1:101::254
enable enable
preempt-mode preempt-mode
priority 150 priority 140
advertisement-interval 1 advertisement-interval 1
short-path-forwarding short-path-forwarding
! !
ip mtu 9000 ip mtu 9000
ip proxy-arp ip proxy-arp
ip address 10.1.101.1/24 ip address 10.1.101.2/24
vrrp-extended-group 10 vrrp-extended-group 10
virtual-ip 10.1.101.254 virtual-ip 10.1.101.254
advertisement-interval 1 advertisement-interval 1
enable enable
preempt-mode preempt-mode
priority 150 priority 140
! !
no shutdown no shutdown
! !
! !

Server Access Links on the Individual Leaf/ToR
The server-facing or access links are on the leaf nodes. In the validated design,
• 10-Gbps links are used for server-facing VLANs.
• These links are configured as Layer 2 trunk or access ports with VLANs associated.
• Disable fabric ISL and trunk features.
• Spanning tree is disabled.1
interface TenGigabitEthernet 1/0/4 Enabled as a trunk port
switchport Add the required VLANs to the trunk port
switchport mode trunk
switchport trunk allowed vlan add 101-200
switchport trunk tag native-vlan
spanning-tree shutdown
no shutdown
Server Access Links on the vLAG Pair/ToR

vLAG configuration involves the following:
• Node ID configuration on the pair of devices
• Inter-switch links or ISL configuration on both devices
• Configuration of server-facing port channels and the required VLANs on them
Node ID Configuration on the vLAG Pair

Refer to Node ID Configuration for VDX Platforms on page 20 for assigning the node ID to the vLAG pair.
• Pod1-Leaf1-1, rbridge-id 1
• Pod1-Leaf1-2, rbridge-id 2
POD1-Leaf1-1# show vcs
Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 405
VCS GUID : c98c32fb-1472-4cc5-a7fa-aeb8570627f9
Total Number of Nodes : 2
Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
1 >10:00:00:27:F8:F0:76:E0* 10.20.34.45 Online Online POD1-Leaf1-1
2620:100:0:fa48:34::45
2 10:00:00:27:F8:F0:5B:B0 10.20.34.46 Online Online POD1-Leaf1-2
2620:100:0:fa48:34::46
1
If there are L2 switches or bridges between the leaf and servers, spanning tree must be enabled. If there is a possibility of enabling bridges inadvertently
under the leaf nodes, we recommend enabling spanning tree and configuring the server ports as edge ports.
POD1-Leaf3(conf-if-te-1/0/4)# spanning-tree autoedge

ISL Configuration
As shown in the illustration below, the vLAG pair is interconnected by two 10-Gbps Ethernet ports for ISL.
Server Port-Channel Configuration

In the configuration shown below, port channel 1 is configured as a vLAG.
interface TenGigabitEthernet 1/0/1 interface TenGigabitEthernet 2/0/1

fabric isl enable fabric isl enable
fabric trunk enable fabric trunk enable
interface TenGigabitEthernet 1/0/2 interface TenGigabitEthernet 2/0/2

fabric isl enable fabric isl enable
fabric trunk enable fabric trunk enable
Rbridge 1 Rbridge 2
Te 1/0/9 ISL Links Te 2/0/9
vLAG
interface Port-channel 1
vlag ignore-split
mtu 9022
switchport
no shutdown
!
interface TenGigabitEthernet 1/0/9
channel-group 1 mode active type standard
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!

SLX 9140 ToRs

Server VLANs
On SLX platforms, the router-interface for a VLAN is specified under the VLAN as shown below.
vlan 76
router-interface Ve 76
!
vlan 77
!
VE Interfaces for the Server VLANs

VE interfaces are configured to provide the first-hop routing functionality for the hosts in the VLAN. The following configuration
parameters are required:
• Jumbo MTU for both IPv4 and IPv6 packet forwarding
interface Ve 76
ipv6 address fdf8:10:0:4c::1/64
ipv6 mtu 9000
ip mtu 9000
ip address 10.0.76.1/24
no shutdown
!
FHRP for the MCT Pair ToR (Redundant Dual ToR)

The SLX platforms use MCT for ToR redundancy. VRRPe is required for first-hop router functionality in the server VLANs. With VRRPe
extension, both switches in the pair act as the active router in the VLANs. One VRRP group is configured for IPv4 and one for IPv6.
Hosts use the virtual IP as the gateway address.
ipv6 address fdf8:10:0:50::2/96 ipv6 address fdf8:10:0:50::2/96
ipv6 vrrp-extended-group 2 ipv6 vrrp-extended-group 2
virtual-ip fdf8:10:0:50::254 virtual-ip fdf8:10:0:50::254
enable enable
no preempt-mode no preempt-mode
! !
vrrp-extended-group 1 vrrp-extended-group 1
virtual-ip 10.0.80.254 virtual-ip 10.0.80.254
enable enable
no preempt-mode no preempt-mode
! !
! !
Server Access Links on the Individual Leaf/ToR

interface Ethernet 0/20 Enabled as a trunk port
mtu 9022 Add the required VLANs to the trunk port
switchport
no shutdown
!

Server Access Links on the MCT Pair/ToR
On SLX leafs, server redundancy is achieved by the multi-chassis trunking with the LAG. This is similar to the vLAG functionality
available on VDX platforms.
ICL Configuration
As shown in the illustration below, the MCT pair is interconnected by two 40-Gbps Ethernet ports for ICL.
Cluster Configuration
The cluster configuration requires the following:

• Peer-interface port-channel or peering interface (port-channel 10).
• Control VLAN and VE interface (VLAN 4090 and VE 4090).
• Peer IP address (the IP address configured on the control VE of the peer).
• VLANs configured on the cluster.

Server or Client Port-Channel Configuration
In the configuration shown below, port channel 101 is configured as a multi-chassis LAG port channel. In the configuration shown—
“client Server1 1”—identifier 1 on both peers designates the client port channel as a dual-homed LAG. This identifier must match on
both peers.
interface Ethernet 0/49 interface Ethernet 0/49
speed 40000 speed 40000
channel-group 10 mode active type standard channel-group 10 mode active type standard
! !
! !
interface Port-channel 10 interface Port-channel 10
mtu 9216 mtu 9216
! E 0/49 !
MCT Peer1 MCT Peer2
E 0/50
E 0/1
E 0/1
mLAG
vlan 4090 vlan 4090

router-interface Ve 4090 router-interface Ve 4090
! !
! !
cluster pod1-cluster 1 cluster pod1-cluster 1
peer-interface Port-channel 10 peer-interface Port-channel 10
peer 10.0.1.5 peer 10.0.1.4
df-load-balance df-load-balance
deploy deploy
client Server1 1 client Server1 1
client-interface Port-channel 101 client-interface Port-channel 101
deploy deploy
! !
evpn default evpn default
route-target both auto ignore-as route-target both auto ignore-as
rd auto rd auto
vlan add 76-200 vlan add 76-200
! !
mtu 9022 mtu 9022
switchport switchport
switchport mode trunk-no-default-native switchport mode trunk-no-default-native
switchport trunk allowed vlan add 101-110 switchport trunk allowed vlan add 101-110
! !
! !
router bgp router bgp
local-as 4259840001 local-as 4259840001
capability as4-enable capability as4-enable
neighbor 10.0.1.5 remote-as 4259840001 neighbor 10.0.1.4 remote-as 4259840001
neighbor 10.0.1.5 bfd neighbor 10.0.1.4 bfd
! !
address-family evpn address-family evpn
neighbor 10.0.1.5 activate neighbor 10.0.1.4 activate
! !
! !

MCT on SLX 9540 Edge Leafs

The edge leafs are configured in an MCT cluster for dual-homed firewall or appliance connectivity. The edge leafs are typically configured
with one VRF for DC routes and with another VRF (or default VRF) for Internet connectivity. (Deployment designs may vary. A sample
validated design with configuration is shown in this section.)
In the configuration example shown here, the internal fabric routes are downloaded into the DC-VRF on the edge leafs. The Internet
connectivity is provided in the default VRF. The appliance is connected between these two VRFs using VE interfaces over an MCT client
port channel or LAG for redundancy or dual-homing to the two edge leafs.
Steps involved in configuring the edge leafs:

• Configure an MCT cluster with two edge leafs.
• Create a VRF: DC-VRF.
• Configure fabric links connecting to the spines in the DC-VRF.
• Configure Internet-facing links in the default VRF.
• Configure two VE interfaces: one in the DC-VRF and another in the default VRF. (In the figure above, VE-2 is in the DC-VRF
and VE-3 is in the default VRF.)
• Configure VRRP groups for these two VE interfaces. These VRRP VIPs act as gateways for the firewall or service appliance.
• In the DC-VRF, configure a static default route toward the firewall. Originate the default route into BGP toward the fabric.
• The global or default VRF may have a default route toward the Internet edge.
(The configuration templates shown below are applicable to both edge leafs in the MCT cluster. Where there are differences, the
configuration blocks are shown side-by-side.)

VRF Configuration
vrf DC-VRF Default routes pointing to the service

address-family ipv4 unicast appliance.
ip route 0.0.0.0/0 10.0.2.3
!
address-family ipv6 unicast
ipv6 route ::/0 fdf8:10:0:2::3
!
!
VLAN/VE/VRRP Configuration
BGP Configuration
router bgp Originate default route into the fabric.
address-family ipv4 unicast vrf DC-VRF
graceful-restart
default-information-originate
!
graceful-restart
!

MCT Cluster Configuration
There are a few minor differences in the MCT configuration for the SLX 9540 platform compared to the SLX 9140 platforms used as
leafs.

! !
! !
mtu 9216 mtu 9216
switchport mode trunk Rbridge 2 mode trunk
switchport
switchport trunk allowed vlan add 4 switchport trunk allowed vlan add 4
! !
E 0/11
9240 MCT peer1 9240 MCT Peer2
E 0/12
E 0/9 E 0/9
vlan 4 vlan 4
router-interface Ve 4 router-interface Ve 4
! !
! !
interface Loopback 1 interface Loopback 1
! !
ip router-id 10.55.55.1 ip router-id 10.56.56.1
ip route 10.56.56.1/32 10.55.56.1 ip route 10.55.55.1/32 10.55.56.0
! !
cluster Edge-cluster 1 cluster Edge-cluster 1
member vlan add 2-3 member vlan add 2-3
peer-interface Ve 4 peer-interface Ve 4
peer 10.56.56.1 peer 10.55.55.1
client-isolation-strict client-isolation-strict
deploy deploy
client 6720 1 client 6720 1
client-interface Port-channel 2 client-interface Port-channel 2
esi 0:0:0:0:0:0:0:1:1 esi 0:0:0:0:0:0:0:1:1
deploy deploy
! !
mtu 9216 mtu 9216
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan add 2-3 switchport trunk allowed vlan add 2-3
! !
! !
router bgp router bgp
local-as 4200007000 local-as 4200007000
capability as4-enable capability as4-enable
fast-external-fallover fast-external-fallover
neighbor 10.56.56.1 remote-as 4200007000 neighbor 10.55.55.1 remote-as 4200007000
neighbor 10.56.56.1 update-source loopback 1 neighbor 10.55.55.1 update-source loopback 1
neighbor 10.56.56.1 bfd neighbor 10.55.55.1 bfd
address-family l2vpn evpn address-family l2vpn evpn

neighbor 10.56.56.1 activate neighbor 10.55.55.1 activate
neighbor 10.56.56.1 encapsulation mpls neighbor 10.55.55.1 encapsulation mpls
! !

eBGP Control-Plane Configuration

Deployment Model-1: eBGP Configuration for Optimized 5-Stage Clos
Consider the following key points as a design principle for eBGP peering between the fabric nodes. Refer to Figure 8 on page 19 for
topology information.
• In the validated design, a 4-byte private AS range is used.
• Each leaf is in a private AS.
• The vLAG pair or MCT pair leaf (dual or redundant ToR) is considered as one leaf even though the control plane is independent.
Both devices in the pair are in the same private AS.
• All spines within a PoD are in one private AS.
• All super-spines are in one private AS.
• All border leafs are in one private AS.
• BGP peering between two nodes is established over the fabric link between the nodes. Fabric link IPv4 and IPv6 addresses are
used as the source and destination for the BGP sessions between the two nodes.
• eBGP IPv4 and IPv6 peering is established with MD5 authentication between the layers of nodes. Multiple AFI exchange over
one peering session22 (either IPv4 or IPv6) is planned in an upcoming software release. Until then, we must have separate
peering—IPv4 peering for IPv4 NLRI exchange and IPv6 peering for IPv6 NLRI exchange.
• Use peer groups for common configuration related to the BGP peers.
• Enable MD5 authentication on all BGP neighbors in the peer groups.
• Enable BFD on all BGP neighbors in the peer groups.
• Enable both IPv4 and IPv6 address families. Advertise the IPv4 and IPv6 addresses of the loopback interface into the
respective address families. This is useful for debugging and troubleshooting reachability between the nodes.
• Activate the IPv6 peer groups under the IPv6 address family.
Spine Configuration
All spines within a PoD have a similar configuration. Peer groups are used to simplify the configurations and also for efficiency in BGP
update processing. In a 5-stage fabric, spines are connected to leafs inside their PoD and to super-spines.
• Configure the directly connected leaf IP addresses in one peer group called leaf-group.
• Configure the directly connected leaf IPv6 addresses in a peer group called leaf-group-ipv6.
• Configure the directly connected super-spine IP addresses in a peer group called superspine-group.
• Configure the directly connected super-spine IPv6 addresses into another peer group called superspine-group-ipv6.
2 Exchanging both IPv4 and IPv6 AFI over a single BGP peering session is not supported. Support is planned in the upcoming releases of NOS
and SLX-OS.

!POD1-Spine1 4 Byte AS number of this device

router bgp
local-as 4259905537
capability as4-enable
fast-external-fallover
! Configure a peer-group for ipv4
neighbor leaf-group1 peer-group exchange with Leafs.
neighbor leaf-group1 description To Leaf AS-6500X.1 from 65001.1 Enable MD5 authentication and BFD
neighbor leaf-group1 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor leaf-group1 bfd
! Add the directly connected leafs’ IPv4
neighbor 10.0.1.0 remote-as 4259840001 addresses into leaf-group.
neighbor 10.0.1.0 peer-group leaf-group1 Each leaf is in a different AS, so it must
neighbor 10.0.2.0 remote-as 4259840001 be specified seperately and not with
neighbor 10.0.2.0 peer-group leaf-group1
peer-group
neighbor 10.0.3.0 remote-as 4259840002
neighbor 10.0.3.0 peer-group leaf-group1 In case of a dual-ToR, both nodes are in
neighbor 10.0.4.0 remote-as 4259840003 one AS.
!
neighbor leaf-group1-ipv6 peer-group Configure a peer-group for IPv6
neighbor leaf-group1-ipv6 description To Leaf AS-6500X.1 exchange with Leafs.
neighbor leaf-group1-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor leaf-group1-ipv6 bfd Enable MD5 authentication.
!
neighbor fdf8:10:0:1:1::1 remote-as 4259840001 Add IPv6 addresses of the directly
neighbor fdf8:10:0:1:1::1 peer-group leaf-group1-ipv6 connected Leafs to this peer-group.
neighbor fdf8:10:0:2:1::1 remote-as 4259840001
neighbor fdf8:10:0:2:1::1 peer-group leaf-group1-ipv6
neighbor fdf8:10:0:4:1::1 peer-group leaf-group1-ipv6 Configure two peer-groups for Super-
! spines – one for Ipv4 and one for IPv6
neighbor super-spine-group1 peer-group
neighbor super-spine-group1 remote-as 4200000010
neighbor super-spine-group1 description To super-spine AS-4200000010
neighbor super-spine-group1 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor super-spine-group1 bfd
!
neighbor super-spine-group1-ipv6 peer-group
neighbor super-spine-group1-ipv6 remote-as 4200000010
neighbor super-spine-group1-ipv6 description To super-spine
neighbor super-spine-group1-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor super-spine-group1-ipv6 bfd
!
neighbor 10.0.51.0 peer-group super-spine-group1
!
neighbor fdf8:10:0:51:1::1 peer-group super-spine-group1-ipv6
! Enable ipv4 Address-Family
address-family ipv4 unicast Enable graceful restart
neighbor leaf-group1 enable-peer-as-check
maximum-paths 8
graceful-restart
! Enable IPv6 Address-Family.
address-family ipv6 unicast Explicitly activate the peer-groups
neighbor super-spine-group1-ipv6 activate created for IPv6 exchange
neighbor super-spine-group1-ipv6 enable-peer-as-check
neighbor leaf-group1-ipv6 activate
neighbor leaf-group1-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
!
Each spine should establish IPv4 and IPv6 peering with all leafs inside the PoD and super-spines. (Note that the leaf nodes in a vLAG
pair share one common AS number between them, and super-spines belong to one AS number.)

pod1-spine1# show ip bgp summary

BGP4 Summary
Router ID: 10.101.1.1 Local AS Number: 4259905537
Confederation Identifier: not configured
Confederation Peers:
Maximum Number of IP ECMP Paths Supported for Load Sharing: 64
Number of Neighbors Configured: 34, UP: 9
Number of Routes Installed: 810, Uses 77760 bytes
Number of Routes Advertising to All Neighbors: 15279 (926 entries), Uses 55560 bytes
Number of Attribute Entries Installed: 13, Uses 1352 bytes
Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend
10.0.1.0 4259840001 ESTAB 0h53m36s 127 0 461 0
10.0.2.0 4259840001 ESTAB 2d 1h 5m 132 0 331 0
10.0.3.0 4259840002 ESTAB 10d 1h 0m 105 0 358 0
10.0.4.0 4259840003 ESTAB 10d 1h 0m 112 0 353 0
10.0.5.0 4259840003 ESTAB 10d 1h 0m 112 0 461 0
10.0.6.0 4259840004 ESTAB 10d 1h 0m 109 0 462 0
10.0.7.0 4259840004 ESTAB 10d 1h 0m 109 0 355 0
10.0.51.0 4200000010 ESTAB 2d10h57m 3 0 461 0
10.0.51.2 4200000010 ESTAB 9d22h38m 1 0 462 0
...
...
pod1-spine1# show ipv6 bgp summary

BGP4 Summary
fdf8:10:0:1:1::1 4259840001 ESTAB 0h55m41s 132 0 452 0
fdf8:10:0:2:1::1 4259840001 ESTAB 2d 1h 7m 130 0 328 0
fdf8:10:0:3:1::1 4259840002 ESTAB 10d 1h 2m 104 0 354 0
fdf8:10:0:4:1::1 4259840003 ESTAB 10d 1h 2m 109 0 351 0
fdf8:10:0:5:1::1 4259840003 ESTAB 10d 1h 2m 109 0 456 0
fdf8:10:0:6:1::1 4259840004 ESTAB 10d 1h 2m 108 0 352 0
fdf8:10:0:7:1::1 4259840004 ESTAB 10d 1h 2m 108 0 456 0
fdf8:10:0:51:1::1 4200000010 ESTAB 2d11h 0m 1 0 458 0
fdf8:10:0:51:2::1 4200000010 ESTAB 9d22h40m 1 0 457 0
...
...
Leaf Configuration
All leafs within a PoD have a similar configuration. Peer groups are used to simplify the configuration and also for efficiency in BGP
update processing. Leafs are connected to spines only.
• Configure the directly connected IP addresses of the spines into a peer group spine-group.
• Configure the directly connected IPv6 addresses of the spines into a peer group spine-group-ipv6.
• Advertise both IPv4 and IPv6 server subnets. Using a route map, filter the subnets of fabric links from being advertised into
BGP and allow only the server subnets and loopback IP address. Advertising loopback IP and IPv6 address helps debug
routing and node reachability issues. A sample route map is given below. This can be modified according to the deployment
requirements.
IPv4 Route-Map Configuration
ip prefix-list fabric_links_ip seq 10 permit 0.0.0.0/0 ge 31 le 31
route-map ToR-map deny 10
match ip address prefix-list fabric_links_ip
!
route-map ToR-map permit 20
!

ipv6 prefix-list fabric_links_ipv6 seq 10 permit ::/0 ge 127 le 127
route-map ToR-map-ipv6 deny 10
match ipv6 address prefix-list fabric_links_ipv6
!
route-map ToR-map-ipv6 permit 20
!

!POD1-leaf1-1
router bgp
local-as 4259840001
!
neighbor spine-group peer-group
neighbor spine-group remote-as 4259905537 Peer-group config grouping spines’ IPv4
neighbor spine-group description To spine addresses.
neighbor spine-group password $9$MCgKGaNt6OASX68/7TC6Lw== Enable MD5 authentication and BFD
neighbor spine-group bfd
!
neighbor 10.0.1.1 peer-group spine-group
!
neighbor spine-group-ipv6 peer-group
neighbor spine-group-ipv6 remote-as 4259905537 Peer-group config grouping spines’ IPv6
neighbor spine-group-ipv6 description To spine AS-65001.1 addresses.
neighbor spine-group-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw== Enable MD5 authentication and BFD
neighbor spine-group-ipv6 bfd
!
neighbor fdf8:10:0:1:1::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:1:4::2 peer-group spine-group-ipv6 Enable IPv4 Address-Family
!
Redistribute connected to advertise the
redistribute connected route-map ToR-map-ip VLAN subnets and /32 loopback
neighbor spine-group enable-peer-as-check addresses.
neighbor 10.0.1.11 route-map out BGP-med Enable graceful-restart
maximum-paths 8
graceful-restart Enable IPv6 Address-Family.
! Activate the peer-groups configured for
address-family ipv6 unicast IPv6 route exchange.
redistribute connected route-map ToR-map-ipv6 Advertise /128 loopback IPv6 address and
neighbor spine-group-ipv6 activate Host VLAN IPv6 subnets
neighbor spine-group-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!

Check the BGP neighbors. The leaf must be peering with all spines within the PoD for IPv4 address-family route exchange. As shown
below, for a dual or vLAG ToR, check the neighbors on both nodes.
pod1-leaf1# show ip bgp summary
BGP4 Summary
10.0.1.1 4259905537 ESTAB 1h46m30s 329 0 127 0
10.0.1.3 4259905537 ESTAB 1d23h58m 329 0 456 0
10.0.1.5 4259905537 ESTAB 1d23h58m 329 0 456 0
10.0.1.7 4259905537 ESTAB 1d23h58m 329 0 456 0
pod1-leaf1# show ip bgp summary

BGP4 Summary
fdf8:10:0:1:1::2 4259905537 ESTAB 1h47m36s 322 0 454 0
fdf8:10:0:1:2::2 4259905537 ESTAB 1d23h59m 322 0 454 0
fdf8:10:0:1:3::2 4259905537 ESTAB 1d23h59m 322 0 132 0
fdf8:10:0:1:4::2 4259905537 ESTAB 1d23h59m 322 0 454 0
Super-Spine Configuration
This is applicable to all super-spines. Peer groups are used to simplify the configuration. Super-spines peer with the spines in each PoD
and with the border leafs. In the validated design, the super-spines connect to two PoDs. The configuration may be replicated for multiple
PoDs.
• Super-spines connect to the spines in each PoD and to the border leafs using fabric links.
• Create a peer group for each PoD for IPv4 peering, and exchange IPv4 routes:
– pod1_spine-group—Add the directly connected neighbor IP addresses of all spines in PoD1 to this group.
– pod2_spine-group—Add the directly connected neighbor IP addresses of all spines in PoD2 to this group.
• Create a peer group for each PoD for IPv6 peering, and exchange IPv6 routes:
– pod1_spine-group-ipv6—Add the directly connected neighbor IPv6 addresses of all spines in PoD1 to this group.
– pod2_spine-group-ipv6—Add the directly connected neighbor IPv6 addresses of all spines in PoD2 to this group.
• Create a separate peer group for the pair of edge leafs: edge-group and edge-group-ipv6. Add the directly connected neighbor
IPv4 and IPv6 addresses of the edge leafs to these groups.
• Enable MD5 authentication to all peer groups.
• Enable BFD on all peer groups.

router bgp
local-as 4200000010
!
neighbor edge-group peer-group Peer-group config pointing to
neighbor edge-group remote-as 4200007000 edge leafs
neighbor edge-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor edge-group bfd
!
neighbor 10.2.1.17 peer-group edge-group
!
neighbor edge-group-ipv6 peer-group
neighbor edge-group-ipv6 remote-as 4200007000
neighbor edge-group-ipv6 password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor edge-group-ipv6 bfd
!
neighbor fdf8:10:2:1::17 peer-group edge-group-ipv6 Peer-group config pointing to spines in
neighbor fdf8:10:2:1::19 peer-group edge-group-ipv6 PoD1
!
neighbor pod1-spine-group peer-group
neighbor pod1-spine-group remote-as 4259905537
neighbor pod1-spine-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod1-spine-group bfd
!
neighbor 10.0.51.1 peer-group pod1-spine-group
!
neighbor pod1-spine-group-ipv6 peer-group
neighbor pod1-spine-group-ipv6 remote-as 4259905537
neighbor pod1-spine-group-ipv6 password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod1-spine-group-ipv6 bfd
!
neighbor fdf8:10:0:51:1::2 peer-group pod1-spine-group-ipv6
! Peer-group config pointing to Spines in
neighbor pod2-spine-group peer-group PoD2
!
!
!
neighbor fdf8:10:2:1::9 peer-group pod2-spine-group-ipv6
neighbor fdf8:10:2:1::11 peer-group pod2-spine-group-ipv6 Enable ipv4 Address-Family
neighbor fdf8:10:2:1::13 peer-group pod2-spine-group-ipv6 Advertise loopback interface’s IPv4
neighbor fdf8:10:2:1::15 peer-group pod2-spine-group-ipv6 address.
! Enable graceful-restart.
network 10.125.1.1/32
neighbor edge-group enable-peer-as-check
neighbor pod2-spine-group enable-peer-as-check
neighbor pod1-spine-group enable-peer-as-check Enable ipv6 Address-Family
maximum-paths 8
Advertise IPv6 address of the loopback
graceful-restart
interface.
!
address-family ipv6 unicast Activate the peer-groups configured for
network fdf8::10:124:1:1/128 IPv6 routes exchange.
neighbor edge-group-ipv6 activate Enable graceful-restart
neighbor edge-group-ipv6 enable-peer-as-check
neighbor pod2-spine-group-ipv6 activate
neighbor pod2-spine-group-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
40 ! Extreme IP Fabric Architecture
Border/Edge Leaf Configuration

Border leafs are connected to super-spines in a 5-stage fabric and to WAN edge devices. Note the following key points when
configuring border leafs in a 5-stage fabric:
• Border leafs peer with each super-spine using fabric links. They exchange both IPv4 and IPv6 routes with the super-spines.
• Border leafs are also connected to the WAN edge devices. In this validated design, we have eBGP peering between the border
leafs and WAN edges. Some deployments may require iBGP or simple IGP peering with the WAN edge depending on the
scale. (The WAN edge may simply advertise a default route.)
• Configure two peer groups, super-spine and super-spine-ipv6, for IPv4 and IPv6 peers. Add the directly connected neighbor
addresses of the super-spines into these groups.
• For WAN edge connectivity, configure two peer groups, wan-group and wan-group-ipv6, for IPv4 and IPv6 peering. Add the
directly connected neighbor addresses of the WAN edge devices into these groups.
• Enable MD5 authentication and BFD on the IPv4 and IPv6 super-spine peer groups.
Note that the super-spine peer groups are in the global VRF in the configuration below. They may be configured under a separate VRF
as discussed in MCT on SLX 9540 Edge Leafs on page 30.

!edge-leaf1
router bgp
local-as 4200007000
!
neighbor super-spine peer-group
neighbor super-spine remote-as 4200000010
neighbor super-spine description IPv4 peering to super-spines Peer-groups config pointing to
neighbor super-spine password 2 $MlVzZCFAbg== super-spines
neighbor super-spine bfd
!
neighbor 10.2.1.16 peer-group super-spine
!
neighbor super-spine-ipv6 peer-group
neighbor super-spine-ipv6 remote-as 4200000010
neighbor super-spine-ipv6 description IPv6 peering to super-spines
neighbor super-spine-ipv6 password 2 $MlVzZCFAbg==
neighbor super-spine-ipv6 bfd
!
neighbor fdf8:10:2:1::16 peer-group super-spine-ipv6
! Peer-groups config pointing to
neighbor wan-group peer-group WAN-edges
neighbor wan-group remote-as 4200007001
neighbor wan-group description IPv4 peering to WAN Edge
!
neighbor 192.168.1.2 peer-group wan-group
!
neighbor wan-group-ipv6 peer-group
neighbor wan-group-ipv6 remote-as 420007001
neighbor wan-group-ipv6 description IPv6 peering to WAN Edge
!
neighbor fdf8:192:168:1::2 peer-group wan-group-ipv6
! Enable both IPv4 and IPv6
address-family ipv4 unicast Address-Families
network 10.123.3.1/32
neighbor wan-group enable-peer-as-check Activate wan-group and super-
neighbor super-spine enable-peer-as-check spine groups for IPv6 route
maximum-paths 8 exchange
graceful-restart
!
neighbor wan-group-ipv6 activate
neighbor wan-group-ipv6 enable-peer-as-check
neighbor super-spine-ipv6 activate
neighbor super-spine-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
Deployment Model-2: eBGP Configuration for 3-Stage Clos Fabric

Refer to Figure 8 on page 19 for topology information. The control-plane routing configuration for a 3-stage fabric is very similar to that
of the 5-stage fabric with the exception of peering to super-spines by spines and border leafs. Border leafs are directly connected to
spines.
Note the following key points:
• In the validated design, a 4-byte private AS range is used.
• Each leaf is in a private AS.
• The vLAG or MCT pair leaf (dual or redundant ToR) is considered as one leaf even though the control plane is independent.
Both devices in the pair are in the same private AS.
• All spines within a PoD are in one private AS.
• All super-spines are in one private AS.
• All border leafs are in one private AS.

• BGP peering between two nodes is established over the fabric link between the nodes. Fabric link IPv4 and IPv6 addresses are
used as the source and destination for the BGP sessions between the two nodes.
• eBGP IPv4 and IPv6 peerings with MD5 authentication are established between the layers of nodes. Multiple AFI exchange
over one peering session33 (either IPv4 or IPv6) is planned in an upcoming software release. Until then, we need to have
separate peering—IPv4 peering to exchange IPv4 NLRI and IPv6 peering to exchange IPv6 NLRI.
• Use peer groups for common configuration related to the BGP peers.
• Enable MD5 authentication to all BGP neighbors.
• Enable both IPv4 and IPv6 address families. Advertise the IPv4 and IPv6 addresses of the loopback interface into respective
address families. This is useful for debugging and troubleshooting reachability between the nodes.
• Activate IPv6 peer groups under the IPv6 address family.
• Enable MD5 authentication to all peer groups.
• Enable BFD on all peer groups.
Spine Configuration
All spines within a PoD have a similar configuration for IPv4 underlay. In a 3-stage fabric, spines peer with leafs and border leafs. Peer
groups are used to simplify the configurations and also for efficiency in BGP update processing.
• Configure the directly connected leafs' IPv4 addresses into one peer group: leaf-group.
• Configure the directly connected leafs' IPv6 addresses into one peer group: leaf-group-ipv6.
• Configure the edge leafs IPv4 fabric link addresses into one peer group: edge-group.
• Configure the edge leafs IPv4 fabric link addresses into one peer group: edge-group-ipv6.
3
Exchanging both IPv4 and IPv6 AFI over a single BGP peering session is not supported. It is planned in the upcoming releases of NOS and
SLX-OS.

!POD1-Spine1 4 Byte AS number of this device

router bgp
local-as 4259905537
! Configure a peer-group for ipv4
neighbor leaf-group1 peer-group exchange with Leafs.
neighbor leaf-group1 description To Leaf Enable MD5 authentication and BFD
! Add the directly connected leafs’ IPv4
neighbor 10.0.1.1 remote-as 4259840001 addresses into leaf-group.
neighbor 10.0.1.1 peer-group leaf-group1 Each leaf is in a different AS, so it must
neighbor 10.0.1.3 remote-as 4259840001 be specified seperately and not with
peer-group
neighbor 10.0.1.5 peer-group leaf-group1 In case of a dual-ToR, both nodes are in
neighbor 10.0.1.7 remote-as 4259840003 one AS.
!
neighbor leaf-group1-ipv6 peer-group Configure a peer-group for IPv6
neighbor leaf-group1-ipv6 description To Leaf AS-6500X.1 exchange with Leafs.
neighbor leaf-group1-ipv6 bfd Enable MD5 authentication.
!
neighbor fdf8:10:0:1::1 remote-as 4259840001 Add IPv6 addresses of the directly
neighbor fdf8:10:0:1::1 peer-group leaf-group1-ipv6 connected Leafs to this peer-group.
neighbor fdf8:10:0:1::3 remote-as 4259840001
neighbor fdf8:10:0:1::3 peer-group leaf-group1-ipv6
neighbor fdf8:10:0:1::5 peer-group leaf-group1-ipv6
neighbor fdf8:10:0:1::7 peer-group leaf-group1-ipv6 Configure two peer-groups for edge-leafs
! – one for Ipv4 and one for IPv6
neighbor edge-group peer-group
neighbor edge-group remote-as 4200000021
neighbor edge-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor edge-group bfd
!
!
neighbor edge-group-ipv6 peer-group
neighbor edge-group-ipv6 remote-as 4200007000
neighbor edge-group-ipv6 password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor edge-group-ipv6 bfd
!
neighbor fdf8:10:2:1::17 peer-group edge-group-ipv6
neighbor fdf8:10:2:1::19 peer-group edge-group-ipv6
!
neighbor leaf-group1 enable-peer-as-check Enable ipv4 Address-Family
neighbor edge-group enable-peer-as-check Enable graceful restart
maximum-paths 8
graceful-restart
!
address-family ipv6 unicast Enable IPv6 Address-Family.
neighbor leaf-group1-ipv6 activate Explicitly activate the peer-groups
neighbor leaf-group1-ipv6 enable-peer-as-check created for IPv6 exchange
neighbor edge-group-ipv6 activate
neighbor edge-group-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
!

Leaf Configuration
All leafs within a PoD have a similar configuration as that shown in the 5-stage model.
• Configure the directly connected spines' fabric link IPv4 addresses into one peer group: spine-group.
• Configure the directly connected spines' fabric link IPv4 addresses into one peer group: spine-group-ipv6.
• For the vLAG or MCT pair leaf, the BGP configuration is applied independently on each node.
• Advertise the IPv4 and IPv6 addresses of the loopback interface (for debugging purposes) and the IPv4 and IPv6 server-facing
subnets. Use a route map to exclude the addresses of the fabric links. A sample route map is given below. This configuration
can be modified according to the deployment requirements.
!
!

!
!
!POD1-leaf1-1
router bgp
local-as 4200001001
!
neighbor spine-group remote-as 4200001000
neighbor spine-group description connected to 4 spines Peer-group config grouping spines’ IPv4
neighbor spine-group password 2 $PVNHITJVPWQ= addresses.
! Enable MD5 authentication
!
neighbor spine-group-ipv6 remote-as 4200001000
neighbor spine-group-ipv6 description connected to 4 spines
neighbor spine-group-ipv6 password 2 $MlVzZCFAbg== Peer-group config grouping spines’ IPv6
! addresses.
neighbor fdf8:10:0:1:: peer-group spine-group-ipv6 Enable MD5 authentication
neighbor fdf8:10:0:2:: peer-group spine-group-ipv6
!
redistribute connected route-map ToR-map Enable IPv4 Address-Family
neighbor spine-group enable-peer-as-check
Redistribute connected to advertise the
maximum-paths 8
graceful-restart VLAN subnets
! Enable graceful-restart
redistribute connected route-map ToR-map-ipv6 Repeat the same for IPv6 Address-Family
neighbor spine-group-ipv6 activate
maximum-paths 8
graceful-restart
!
!

Border/Edge Leaf Configuration

Note the following key points when configuring a border leaf in a 5-stage fabric:
• Border leafs peer with each of the super-spines using the fabric links.
• Border leafs are also connected to the WAN edge devices. In this validated design, we have eBGP peering between the border
leafs and the WAN edges. Some deployments may require iBGP or simple IGP peering with the WAN edge, depending on the
scale. (The WAN edge may simply advertise a default route for external reachability.)
• Configure two peer groups, spine-group and superspine-group-ipv6, for the IPv4 and IPv6 peers. Add the directly connected
neighbor addresses of the super-spines into these groups.
• For WAN edge connectivity, configure two peer groups, wan-group and wan-group-ipv6, for IPv4 and IPv6 peering. Add the
directly connected neighbor addresses of the WAN edge devices into these groups.
• Enable MD5 authentication and BFD on the spine peer groups.
• Enable both IPv4 and IPv6 address families. Advertise the IPv4 and IPv6 addresses of the loopback interface into the
respective address families.
Note that the spine peer groups are in the global VRF in the configuration below. They may be configured under a separate VRF as
discussed in MCT on SLX 9540 Edge Leafs on page 30.

router bgp
local-as 4200007000
!
neighbor super-spine peer-group
neighbor super-spine remote-as 4200000010
neighbor super-spine description IPv4 peering to super-spines
neighbor super-spine password 2 $MlVzZCFAbg== Peer-groups config pointing to
neighbor super-spine bfd super-spines
!
!
neighbor super-spine-ipv6 peer-group
neighbor super-spine-ipv6 remote-as 4200000010
neighbor super-spine-ipv6 description IPv6 peering to super-spines
neighbor super-spine-ipv6 password 2 $MlVzZCFAbg==
neighbor super-spine-ipv6 bfd
!
!
neighbor wan-group peer-group Peer-groups config pointing to
neighbor wan-group remote-as 4200007001 WAN-edges
neighbor wan-group description IPv4 peering to WAN Edge
!
!
neighbor wan-group-ipv6 peer-group
neighbor wan-group-ipv6 remote-as 420007001
neighbor wan-group-ipv6 description IPv6 peering to WAN Edge
!
!
address-family ipv4 unicast Enable both IPv4 and IPv6
neighbor super-spine enable-peer-as-check Address-Families
neighbor wan-group enable-peer-as-check
maximum-paths 8 Activate wan-group and super-
graceful-restart spine groups for IPv6 route
! exchange
neighbor wan-group-ipv6 activate
neighbor wan-group-ipv6 enable-peer-as-check
neighbor super-spine-ipv6 activate
neighbor super-spine-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
Illustration Examples
In this section, we illustrate the use cases using sections of the validated design network topology as appropriate. This will help the reader
to further understand the deployment scenarios.
Network Reachability Between Racks and PoDs

Figure 9 shows a section of the topology to illustrate the following with configuration and verification: for this example, two racks are
shown in PoD1 and one rack is shown in PoD2.
• In PoD1, Rack1 has two redundant vLAG pair ToRs, leaf1-1 and leaf1-2.
• In PoD1, Rack2 has two redundant vLAG pair ToRs, leaf1-1 and leaf1-2.
• In PoD2, Rack6 has an individual or nonredundant ToR, Leaf6.
• Under each leaf, one server VLAN is shown with the respective VE interface IP address and FHRP/VIP address in the case of a

vLAG pair.
This example shows the various CLI output required to verify the server subnet reachability between the leafs. Note that there is no
manipulation of BGP paths such as route policies and route aggregation. For additional information, refer to Design Considerations on
page 51.
FIGURE 9 Connectivity Between the Racks and PoDs

Verification
Let's look at the step-by-step verification of a server subnet advertised from Leaf1 into the fabric
and the propagation of this route to other leafs.
Verify Network Advertisement from PoD1 Leaf1
Server VLAN
pod1-leaf1-1# show vlan 101
VLAN Name State Ports Classification
(F)-FCoE (u)-Untagged
(R)-RSPAN (c)-Converged
(T)-TRANSPARENT (t)-Tagged
================ =============== ========================== =============== ====================
101 VLAN0101 ACTIVE Po 1(t)
VE Interface for the VLAN
pod1-leaf1-1# show ip int ve 101

Ve 101 is up protocol is up
Primary Internet Address is 10.1.101.1/24 broadcast is 10.1.101.255
IP MTU is 9018
Proxy Arp is Enabled
IP fast switching is enabled
Vrf : default-vrf
Verify the URIB Entry

pod1-leaf1-1# show ip route 10.1.101.0/24 detail
IP Routing Table for VRF "default-vrf"
'*' denotes best ucast next-hop
'[x/y]' denotes [preference/metric]
10.1.101.0/24, attached
*via DIRECT, Ve 101, [0/0], 22d0h, direct, tag 0
Verify BGP Peering with All Four Spines

pod1-leaf1-1# show ip bgp summary
BGP4 Summary
*Dynamically created based on a listen range command
Dynamically created neighbors: 0/100(max)
10.0.1.0 4200001000 ESTAB 14d23h24m 3769 0 101 0
10.0.2.0 4200001000 ESTAB 14d23h24m 3769 0 101 0
10.0.3.0 4200001000 ESTAB 14d23h24m 3769 0 101 0
10.0.4.0 4200001000 ESTAB 14d23h24m 3769 0 101 0

BGP and BRIB Entry
The BGP tables show whether the route is advertised to all BGP neighbors (spines).
Next hop 0.0.0.0 indicates the route being locally originated.
pod1-leaf1-1# show ip bgp 10.1.101.0/24

Number of BGP Routes matching display condition : 1
Status codes: s suppressed, d damped, h history, * valid, > best, i internal
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop MED LocPrf Weight Path
*> 10.1.101.0/24 0.0.0.0 0 100 32768 ?
Route is advertised to 4 peers:
10.0.4.0(4200001000) 10.0.3.0(4200001000) 10.0.2.0(4200001000)
10.0.1.0(4200001000)
pod1-leaf1-1# show ip bgp route 10.1.101.0/24

Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
S:SUPPRESSED F:FILTERED s:STALE
Prefix Next Hop MED LocPrf Weight Status
1 10.1.101.0/24 0.0.0.0 0 100 32768 BL
AS_PATH:
Route is advertised to 4 peers:
10.0.4.0(4200001000) 10.0.3.0(4200001000) 10.0.2.0(4200001000)
10.0.1.0(4200001000)
Verify the Route Entry on the Spine

As shown below, the spine should receive two entries, as each node in the dual ToR advertises the same server subnet. BRIB should
have these two entries as multipath eBGP entries (ME).
pod1-spine1# show ip bgp 10.1.101.0/24

*> 10.1.101.0/24 10.0.1.1 0 100 0 4200001001 ?
* 10.1.101.0/24 10.0.1.3 0 100 0 4200001001 ?
...
pod1-spine1# show ip bgp route 10.1.101.0/24
1 10.1.101.0/24 10.0.1.1 0 100 0 BME
AS_PATH: 4200001001
2 10.1.101.0/24 10.0.1.3 0 100 0 ME
AS_PATH: 4200001001
Last update to IP routing table: 15d1h19m46s
...
The URIB should have two ECMP paths to this subnet.

pod1-spine1# show ip route 10.1.101.0/24 detail
10.1.101.0/24,
*via 10.0.1.1, Fo 1/0/1, [20/0], 15d1h, eBgp, tag 0
*via 10.0.1.3, Fo 1/0/3, [20/0], 15d1h, eBgp, tag 0

Verify the Route Entry on the Super-Spine

The super-spine BGP table should show four entries, since four spines from PoD1 are advertising this path. All four paths must be
selected as a multipath entry and installed as an ECMP route in the URIB.
ss1-mm1# show ip bgp 10.1.101.0/24

*> 10.1.101.0/24 10.2.1.3 100 0 4200001000 4200001001 ?
* 10.1.101.0/24 10.2.1.1 100 0 4200001000 4200001001 ?
* 10.1.101.0/24 10.2.1.5 100 0 4200001000 4200001001 ?
* 10.1.101.0/24 10.2.1.7 100 0 4200001000 4200001001 ?
...
ss1-mm1# show ip bgp route 10.1.101.0/24

1 10.1.101.0/24 10.2.1.3 0 100 0 BME
AS_PATH: 4200001000 4200001001
2 10.1.101.0/24 10.2.1.1 0 100 0 ME
AS_PATH: 4200001000 4200001001
3 10.1.101.0/24 10.2.1.5 0 100 0 ME
AS_PATH: 4200001000 4200001001
4 10.1.101.0/24 10.2.1.7 0 100 0 ME
AS_PATH: 4200001000 4200001001
...
ss1-mm1# show ip route 10.1.101.0/24

Total number of IP routes: 3895
10.1.101.0/24
*via 10.2.1.1, Eth 3/2, [20/0], 1d19h, eBgp, tag 0
Verify the Route Entry on the Remote Leaf in PoD2

On the remote leaf in PoD2, we should see the route entry with four ECMP paths in URIB since there are four spines in that PoD also.
pod2-leaf6# show ip bgp 10.1.101.0/24
*> 10.1.101.0/24 10.1.2.14 100 0 4200002000 4200000010 4200001000 4200001001 ?
* 10.1.101.0/24 10.1.1.14 100 0 4200002000 4200000010 4200001000 4200001001 ?
* 10.1.101.0/24 10.1.4.14 100 0 4200002000 4200000010 4200001000 4200001001 ?
* 10.1.101.0/24 10.1.3.14 100 0 4200002000 4200000010 4200001000 4200001001 ?
pod2-leaf6# show ip bgp route 10.1.101.0/24

1 10.1.101.0/24 10.1.2.14 0 100 0 BME
AS_PATH: 4200002000 4200000010 4200001000 4200001001
2 10.1.101.0/24 10.1.1.14 0 100 0 ME
AS_PATH: 4200002000 4200000010 4200001000 4200001001
3 10.1.101.0/24 10.1.4.14 0 100 0 ME
AS_PATH: 4200002000 4200000010 4200001000 4200001001
4 10.1.101.0/24 10.1.3.14 0 100 0 ME
AS_PATH: 4200002000 4200000010 4200001000 4200001001
...
pod2-leaf6# show ip route 10.1.101.0/24

Total number of IP routes: 5694
Type Codes - B:BGP D:Connected O:OSPF S:Static U:Unnumbered +:Leaked route; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
Destination Gateway Port Cost Type Uptime
* 10.1.101.0/24 10.1.1.14 Fo 1/0/97 20/0 Be 2h34m
10.1.2.14 Fo 1/0/98 20/0 Be 2h34m
10.1.3.14 Fo 1/0/103 20/0 Be 2h34m
10.1.4.14 Fo 1/0/104 20/0 Be 2h34m

Design Considerations
• Scale. ...................................................................................................................................................................52
• Recommendations for ISL Ports in a VDX vLAG Pair Leaf.................................................................................... 52
• Recommendations for ICL Ports in an SLX MCT Pair. .......................................................................................... 52
• Generalized TTL Security Mechanism for BGP (GTSM). ...................................................................................... 52
Scale
The following table gives various scale parameters and platforms used in this validated test topology. Note that this is not a measure of
the maximum scale that can be supported with Extreme switches in an IP fabric.
Parameter Value
Number of server racks/leafs (dual ToRs) 100
Number of spines 4
Number of server VLAN segments per rack 50
Number of hosts in each compute rack 2000
Number of super-spines 2
RecommendationsforISLPorts in aVDX vLAGPair Leaf

• It is recommended to pick ISL ports from the same port group on the switch. Port-group information about the leaf platforms is
given in the Extreme VDX hardware installation guides3.
• For redundancy, it is recommended to have a minimum of two ISL ports between the switches in the vLAG pair.
• The ports must be of the same speed.
Recommendations for ICL Ports in an SLX MCT Pair

• For redundancy, a minimum of two ports are recommended between the switches in the MCT pair.
• The ports must be of the same speed.
Generalized TTL Security Mechanism for BGP (GTSM)

This configuration is applicable for eBGP peering only. It can be applied to a specific neighbor or to a peer group.
router bgp
local-as 4200001001
neighbor spine-group ebgp-btsh
Route Aggregation, Filtering, Policies, and Default Routing in the Fabric

BGP provides a robust policy infrastructure for IP fabrics to accommodate the small-scale forwarding tables on the leafs (and border
leafs too). A few design considerations with configuration examples are given below.
• Aggregate networks in all PoDs at the super-spine, and advertise only the summary route to the spines and border leafs. Doing
so, each PoD does not learn the routes from other PoDs. In the example below, a super-spine aggregates all subnets in the
PoDs into two networks and advertises only the summary to its neighbors (spines in each PoD and the border leaf).
router bgp
aggregate-address 10.0.0.0/8 summary-only
• Aggregate all networks in a PoD at the spines, and advertise only the summary route to the leafs inside that PoD. Doing so, the
leaf does not learn every subnet attached to the other leafs in the PoD. Traffic forwarding to those subnets occurs using the

aggregate route pointing to the spines. Also, the same aggregate is sent to the border leaf as well.
• Propagate only the default route from the border leaf to the spines (in 3-stage) or the super-spines (in 5-stage) for reachability
to external destinations. As a protection, these devices can be configured not to accept the default route from other nodes.
In the example shown below, the super-spine is configured to deny the default route (if any) advertised by the spines in the
PoDs.
ip prefix-list pfx-list-default seq 10 permit 0.0.0.0/0

!
route-map deny-default deny 10
match ip address prefix-list pfx-list-default
!
route-map deny-default permit 20
!
router bgp
neighbor pod1-spine-group route-map in deny-default
neighbor pod1-spine-group route-map in deny-default
• ORF prefix-list capability is a powerful tool that can be used by a BGP peer to tell its neighbor the list of prefixes (or routes) that
it is interested in receiving from the neighbor. This capability is a way to indicate the destinations of interest based on the
applications or services running behind the leaf. This is different from receiving all routes and then filtering unwanted prefixes.
The sender sends only the routes in the range of the prefix list.
ORF prefix-list capability can also be leveraged for receiving only the default route by, say, the super-spine from the border leaf.
In the example shown below, the super-spine indicates to the border leaf that it would like to receive only the default route and
not any others.

!
router bgp
neighbor border-leaf capability orf prefixlist
neighbor border-leaf prefix-list pfx-list-default in

Appendix—Configuration of the Nodes

• SLX 9140 MCT Pair ToR/Leaf. ...........................................................................................................................54
• SLX 9240 Spine. ..................................................................................................................................................67
• SLX 9850 Super-Spine. ........................................................................................................................ 71
• SLX 9540 Edge Leaf. ..........................................................................................................................................75
• VDX vLAG Pair Leaf. ............................................................................................................................ 78
This appendix includes the relevant configurations of a few nodes in a 5-stage IP fabric.
SLX 9140 MCT Pair ToR/Leaf

Peer1
clock timezone America/Los_Angeles
!
ntp server 10.31.2.80 use-vrf mgmt-vrf
switch-attributes chassis-name SLX9140
switch-attributes host-name pod1-SLX-1
!
vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
ipv6 protocol vrrp-extended
!
mtu 9022
ipv6 mtu 9000
ip mtu 9000
!
vlan 76
description non-MCT vlan on SLX-2
!
vlan 100
!
vlan 101
description Server vlan on MCT Po-101
!
!
vlan 200
!
vlan 4090
description MCT control vlan
!
!
route-map BGP-med permit 10
set metric add 100
!
!
!
!

!
protocol vrrp-extended
!
evpn default
route-target both auto ignore-as
rd auto
vlan add 76-200
!
router bgp
local-as 4259840001
neighbor spine-group description To spine AS-65001.1 from AS-65000.1
neighbor spine-group password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor spine-group-ipv6 description To spine AS-65001.1
neighbor spine-group-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor fdf8:10:0:1::1 peer-group spine-group-ipv6
neighbor 10.0.1.5 bfd
redistribute connected route-map ToR-map
neighbor 10.0.1.11 route-map out BGP-med
maximum-paths 8
!
redistribute connected route-map ToR-map-ipv6
maximum-paths 8
!
address-family evpn
neighbor 10.0.1.5 activate
!
!
!
no shutdown
!
interface Ve 76
ip mtu 9000
ip address 10.0.76.1/24
ipv6 mtu 9000
ipv6 vrrp-extended-group 2
virtual-ip fdf8:10:0:4c::254
enable
no preempt-mode
short-path-forwarding
!
vrrp-extended-group 1
virtual-ip 10.0.76.254
enable
no preempt-mode
!
no shutdown
!
!
interface Ve 101
ip mtu 9000
ip address 10.0.101.1/24

ipv6 mtu 9000
virtual-ip fdf8:10:0:65::254
enable
no preempt-mode
!
virtual-ip 10.0.101.254
enable
no preempt-mode
!
no shutdown
!
!
interface Ve 200
ip mtu 9000
ip address 10.0.200.1/24
ipv6 address fdf8:10:0:c8::1/96
ipv6 mtu 9000
virtual-ip fdf8:10:0:c8::254
enable
no preempt-mode
!
virtual-ip 10.0.200.254
enable
no preempt-mode
!
no shutdown
!
interface Ve 4090
ip mtu 9100
no shutdown
!
interface Management 0
no tcp burstrate
no shutdown
vrf forwarding mgmt-vrf
no ip address dhcp
ip address 10.20.227.246/24
!
speed 10000
description MCT lag member to Server Racks
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!

speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 40000
description MCT peer-link intf
lacp timeout long
no shutdown
!
speed 40000
lacp timeout long
no shutdown
!
speed 40000
mtu 9216
description Link to Spines
ip mtu 9100
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ipv6 mtu 9100
no shutdown
!

speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
switchport mode trunk-no-default-native
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport

no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
cluster pod1-cluster 1
peer-interface Port-channel 10
peer 10.0.1.5
df-load-balance
deploy
client SLX-2 1
client-interface Port-channel 101
deploy
!
client SLX-2 2
deploy
!
client SLX-2 3
deploy
!
client SLX-2 4
deploy
!
client SLX-2 5
deploy
!
client SLX-2 6
deploy
!
client SLX-2 7
deploy
!
client SLX-2 8
deploy

!
client SLX-2 9
deploy
!
client SLX-2 10
deploy
!
!

Peer2
!
switch-attributes host-name pod1-SLX-2
!
vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
!
mtu 9022
ipv6 mtu 9000
ip mtu 9000
!
vlan 76
!
!
vlan 100
!
vlan 101
!
!
vlan 200
!
vlan 4090
description MCT control vlan
!
!
route-map BGP-med permit 10
set metric add 100
!
!
!
!
!
!
evpn default
route-target both auto ignore-as
rd auto
vlan add 76-200
!
router bgp
local-as 4259840001
neighbor spine-group description To spine AS-65001.1 from AS-65000.1
neighbor spine-group password $9$MCgKGaNt6OASX68/7TC6Lw==


neighbor spine-group-ipv6 description To spine AS-65001.1
neighbor spine-group-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor fdf8:10:0:1:10::1 next-hop-self
neighbor fdf8:10:0:1:10::1 bfd
maximum-paths 8
!
maximum-paths 8
!
address-family evpn
!
!
!
no shutdown
!
interface Ve 76
ip mtu 9000
ip address 10.0.76.2/24
ipv6 mtu 9000
virtual-ip fdf8:10:0:4c::254
enable
no preempt-mode
!
enable
no preempt-mode
!
no shutdown
!
interface Ve 101
ip mtu 9000
ip address 10.0.101.2/24
ipv6 mtu 9000
enable
no preempt-mode
!
virtual-ip 10.0.101.254
enable
no preempt-mode

!
no shutdown
!
interface Ve 200
ip mtu 9000
ip address 10.0.200.2/24
ipv6 address fdf8:10:0:c8::2/96
ipv6 mtu 9000
virtual-ip fdf8:10:0:c8::254
enable
no preempt-mode
!
virtual-ip 10.0.200.254
enable
no preempt-mode
!
no shutdown
!
interface Ve 4090
ip mtu 9100
no shutdown
!
no tcp burstrate
no shutdown
ip address dhcp
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000


lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
lacp timeout long
no shutdown
!
speed 10000
mtu 9216
description non-MCT link to Server
switchport
no shutdown
!
speed 40000
lacp timeout long
no shutdown
!
speed 40000
lacp timeout long
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100

ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
description MCT peer-link LAG
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216

description MCT LAG

switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
speed 10000
mtu 9216
description MCT LAG
switchport
no shutdown
!
cluster pod1-cluster 1
peer-interface Port-channel 10
peer 10.0.1.4
df-load-balance
deploy
client SLX-1 1
deploy
!
client SLX-1 2
deploy
!
client SLX-1 3
deploy
!
client SLX-1 4
deploy
!

client SLX-1 5
deploy
!
client SLX-1 6
deploy
!
client SLX-1 7
deploy
!
client SLX-1 8
deploy
!
client SLX-1 9
deploy
!
client SLX-1 10
deploy
!
!
SLX 9240 Spine

!
switch-attributes host-name pod1-spine1
!
vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
mtu 9216
ipv6 mtu 9100
ip mtu 9100
!
router bgp
local-as 4259905537
neighbor leaf-group1 peer-group
neighbor leaf-group1 description To Leaf AS-6500X.1 from 65001.1
neighbor leaf-group1-ipv6 peer-group
neighbor leaf-group1-ipv6 description To Leaf AS-6500X.1 from 65001.1
neighbor leaf-group1-ipv6 bfd
neighbor super-spine-group1 peer-group
neighbor super-spine-group1 remote-as 4200000010
neighbor super-spine-group1 description To super-spine AS-4200000010 from 65001.1
neighbor super-spine-group1 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor super-spine-group1 bfd
neighbor super-spine-group1-ipv6 peer-group
neighbor super-spine-group1-ipv6 remote-as 4200000010
neighbor super-spine-group1-ipv6 description To super-spine AS-4200000010 from 65001.1
neighbor super-spine-group1-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor super-spine-group1-ipv6 bfd
neighbor fdf8:10:0:1:: remote-as 4259840001
neighbor fdf8:10:0:1:: peer-group leaf-group1-ipv6


neighbor fdf8:10:101:1::2 bfd
neighbor fdf8:10:101:a::2 remote-as 4259840014
neighbor fdf8:10:101:a::2 bfd
neighbor fdf8:10:101:b::2 remote-as 4259840015
neighbor fdf8:10:101:b::2 bfd
neighbor fdf8:10:101:c::2 remote-as 4259840016
neighbor fdf8:10:101:c::2 bfd
neighbor fdf8:10:101:d::2 remote-as 4259840017
neighbor fdf8:10:101:d::2 bfd
neighbor fdf8:10:101:e::2 remote-as 4259840018
neighbor fdf8:10:101:e::2 bfd
neighbor fdf8:10:101:f::2 remote-as 4259840019
neighbor fdf8:10:101:f::2 bfd


neighbor leaf-group1 enable-peer-as-check
neighbor super-spine-group1 enable-peer-as-check
maximum-paths 8
graceful-restart
!
neighbor leaf-group1-ipv6 enable-peer-as-check
neighbor super-spine-group1-ipv6 enable-peer-as-check
neighbor fdf8:10:0:3:: activate
neighbor fdf8:10:101:19::2 activate


neighbor fdf8:10:101:f::2 activate
neighbor fdf8:10:101:e::2 activate
neighbor fdf8:10:101:d::2 activate
neighbor fdf8:10:101:c::2 activate
neighbor fdf8:10:101:b::2 activate
neighbor fdf8:10:101:a::2 activate
neighbor fdf8:10:0:51:2::1 activate
maximum-paths 64
graceful-restart
!
address-family evpn
!
!
ip address 10.1.11.1/32
no shutdown
!
!
no tcp burstrate
no shutdown
ip address dhcp
!
speed 40000
mtu 9216
description L3 link to SLX-9140 leaf
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100

no shutdown
!
speed 40000
mtu 9216
description L3 link to VDX leaf
ip mtu 9100
ip proxy-arp
ipv6 address fdf8:10:0:6:1::2/96
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
description L3 link to N9K Leaf
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
description L3 link to N9K Leaf
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
description L3 link to VDX leaf
ip mtu 9100
ip proxy-arp
ipv6 mtu 9100
no shutdown
!
speed 100000
mtu 9216
description L3 link to SLX-9850 Super-Spine
ip mtu 9100
ip proxy-arp
ip address 10.0.51.1/31
ipv6 address fdf8:10:0:51:1::2/96
ipv6 mtu 9100
no shutdown
!
mtu 9216
description L3 link to SLX-9850 Super-Spine
ip mtu 9100
ip proxy-arp
ip address 10.0.51.3/31
ipv6 address fdf8:10:0:51:2::2/96
ipv6 mtu 9100
no shutdown
!
SLX 9850 Super-Spine


!
hardware
port-group 3/1
mode 40g
!
port-group 3/2
mode 40g
!
port-group 3/3
mode 40g
!
port-group 3/4
mode 100g
!
port-group 3/5
mode 100g
!
port-group 3/6
mode 40g
!
!
switch-attributes chassis-name SLX-9850
switch-attributes host-name Super-Spine1
!
vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
ipv6 prefix-list denydefault-fromspine seq 10 permit ::/0
ipv6 prefix-list receivedgwonly-fromborderleaf seq 10 permit ::/0
!
ip prefix-list allowdefaultonly seq 10 permit 0.0.0.0/0
ip prefix-list denydefault-fromspine seq 10 permit 0.0.0.0/0
ip prefix-list pfx-list-denydefault-from-spine seq 10 permit 0.0.0.0/0
!
route-map denydefault-fromspine permit 20
!
route-map denydefault-fromspine deny 10
match ip address prefix-list pfx-list-denydefault-from-spine
!
route-map denydefault-fromspine-ipv6 permit 10
match ipv6 address prefix-list denydefault-fromspine
!
route-map denydefault-fromspine-ipv6 permit 20
!
router bgp
local-as 4200000010
neighbor edge-leaf peer-group
neighbor edge-leaf remote-as 4200007000
neighbor edge-leaf password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor edge-leaf bfd
neighbor edge-leaf-ipv6 peer-group
neighbor edge-leaf-ipv6 remote-as 4200007000
neighbor edge-leaf-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor edge-leaf-ipv6 bfd
neighbor pod1-spine-group password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor pod1-spine-group-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==


neighbor fdf8:10:0:55:1::1 peer-group edge-leaf-ipv6
neighbor fdf8:10:0:56:1::1 peer-group edge-leaf-ipv6
neighbor 10.0.55.11 peer-group edge-leaf
neighbor 10.0.56.11 peer-group edge-leaf
network 125.1.1.1/32
neighbor pod2-spine-group route-map in denydefault-fromspine
neighbor pod1-spine-group route-map in denydefault-fromspine
neighbor edge-leaf capability orf prefixlist
neighbor edge-leaf prefix-list pfx-list-default in
neighbor edge-leaf enable-peer-as-check
maximum-paths 8
graceful-restart
!
aggregate-address fdf8:10::/32 summary-only
neighbor pod2-spine-group-ipv6 route-map in denydefault-fromspine-ipv6
neighbor pod1-spine-group-ipv6 route-map in denydefault-fromspine-ipv6
neighbor edge-leaf-ipv6 capability orf prefixlist
neighbor edge-leaf-ipv6 activate
neighbor edge-leaf-ipv6 enable-peer-as-check
neighbor edge-leaf-ipv6 prefix-list receivedgwonly-fromborderleaf in
maximum-paths 8
graceful-restart
!
address-family l2vpn evpn
!
ip address 10.125.1.1/32
ipv6 address fdf8:10:125:1::1/128
no shutdown
!
no tcp burstrate
no shutdown
ip address dhcp
!
mtu 9216

description L3 link to VDX-6940 Pod2-Spine3

ip mtu 9100
ip proxy-arp
ip address 10.22.1.12/31
ipv6 mtu 9100
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ip address 10.22.1.14/31
ipv6 mtu 9100
no shutdown
!
mtu 9216
description L3 link to SLX-9540 Edge-Leaf1
ip mtu 9100
ip proxy-arp
ip address 10.0.55.10/31
ipv6 address fdf8:10:0:55:1::2/96
ipv6 mtu 9100
no shutdown
!
mtu 9216
description L3 link to SLX-9540 Edge-Leaf2
ip mtu 9100
ip proxy-arp
ip address 10.0.56.10/31
ipv6 address fdf8:10:0:56:1::2/96
ipv6 mtu 9100
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ip address 10.22.1.8/31
ipv6 mtu 9100
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ip address 10.22.1.10/31
ipv6 mtu 9100
no shutdown
!
mtu 9216
description L3 link to SLX-9240 Pod1-Spine1
ip mtu 9100
ip proxy-arp
ip address 10.0.51.0/31
ipv6 address fdf8:10:0:51:1::1/96
ipv6 mtu 9100
no shutdown
!
mtu 9216
ip mtu 9100

ip proxy-arp
ip address 10.0.53.0/31
ipv6 address fdf8:10:0:53:1::1/96
ipv6 mtu 9100
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ip address 10.0.54.0/31
ipv6 address fdf8:10:0:54:1::1/96
ipv6 mtu 9100
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ip address 10.0.52.0/31
ipv6 address fdf8:10:0:52:1::1/96
ipv6 mtu 9100
no shutdown
!
SLX 9540 Edge Leaf

!
switch-attributes host-name edge-leaf1
!
vrf DC-VRF
ip route 0.0.0.0/0 10.0.2.3
!
ipv6 route ::/0 fdf8:10:0:2::3
!
!
vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
!
vlan 2
description VLAN carrying traffic between Datacenter & firewall
!
vlan 3
description VLAN carrying traffic between Internet/WAN & Firewall
!
vlan 4
description MCT peering vlan
!
router bgp
local-as 4200007000
neighbor Super-Spine peer-group
neighbor Super-Spine remote-as 4200000010
neighbor Super-Spine password $9$MCgKGaNt6OASX68/7TC6Lw==

neighbor Super-Spine bfd

neighbor Super-Spine-ipv6 peer-group
neighbor Super-Spine-ipv6 remote-as 4200000010
neighbor Super-Spine-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor Super-Spine-ipv6 bfd
neighbor edge-router peer-group
neighbor edge-router remote-as 101
neighbor edge-router password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor edge-router bfd
neighbor 55::2 remote-as 101
neighbor 55::2 bfd
neighbor 10.56.56.1 update-source loopback 1
neighbor 55.0.0.2 peer-group edge-router
redistribute connected
graceful-restart
!
graceful-restart
neighbor 10.0.55.2 capability orf prefixlist receive
neighbor 10.0.55.2 peer-group Super-Spine
neighbor 10.0.55.10 capability orf prefixlist receive
neighbor 10.0.55.10 peer-group Super-Spine
!
neighbor 55::2 activate
graceful-restart
!
graceful-restart
neighbor fdf8:10:0:55:1::2 capability orf prefixlist receive
neighbor fdf8:10:0:55:1::2 peer-group Super-Spine-ipv6
neighbor fdf8:10:0:55:2::2 capability orf prefixlist receive
neighbor fdf8:10:0:55:2::2 peer-group Super-Spine-ipv6
!
address-family l2vpn evpn
neighbor 10.56.56.1 encapsulation mpls
!
!
ip address 10.55.55.1/32
no shutdown
!
vrf forwarding DC-VRF
ip address 10.55.55.2/32
no shutdown
!
interface Ve 2
ip mtu 9100
ipv6 mtu 9100
enable
no preempt-mode
!
enable

no preempt-mode
!
no shutdown
!
interface Ve 3
ip mtu 9100
ip address 192.168.3.1/24
ipv6 address fdf8:192:168:3::1/96
ipv6 mtu 9100
enable
no preempt-mode
!
virtual-ip 192.168.3.254
enable
no preempt-mode
!
no shutdown
!
interface Ve 4
ip mtu 9100
ip proxy-arp
ip address 10.55.56.0/31
ipv6 mtu 9100
no shutdown
!
no tcp burstrate
no shutdown
ip address dhcp
!
!
description MCT lag member to FireWall
lacp timeout long
no shutdown
!
description MCT peer-link LAG members
lacp timeout long
no shutdown
!
description MCT peer-link LAG members
lacp timeout long
no shutdown
!
speed 40000
mtu 9216
description L3 link to Super-Spine1
ip mtu 9100
ip proxy-arp
ip address 10.0.55.11/31
ipv6 address fdf8:10:0:55:1::1/96
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
description L3 link to Super-Spine2


ip mtu 9100
ip proxy-arp
ip address 10.0.55.3/31
ipv6 address fdf8:10:0:55:2::1/96
ipv6 mtu 9100
no shutdown
!
speed 40000
mtu 9216
description L3 link to WAN Edge-Router
ip mtu 9100
ip proxy-arp
ipv6 address 55::1/96
ipv6 mtu 9100
no shutdown
!
!
mtu 9216
description MCT peer-link LAG
switchport
switchport trunk allowed vlan add 4
no shutdown
!
mtu 9216
description MCT LAG to firewall
switchport
no shutdown
!
cluster BL-cluster 1
member vlan add 2-3
peer-interface Ve 4
peer 10.56.56.1
client-isolation-strict
deploy
client 6720 1
esi 0:0:0:0:0:0:0:1:1
deploy
!
!
VDX vLAG Pair Leaf

switch-attributes 1
chassis-name VDX6740
host-name pod1-leaf1-1
!
switch-attributes 2
chassis-name VDX6740
host-name pod1-leaf1-2
!
vcs virtual-fabric enable
interface Vlan 1
!
interface Vlan 101
!
interface Vlan 102
!
interface Vlan 103
!
interface Vlan 104

!
interface Vlan 105
!
interface Vlan 106
!
interface Vlan 107
!
interface Vlan 108
!
interface Vlan 109
!
interface Vlan 110
!
protocol lldp
advertise dcbx-fcoe-app-tlv
advertise dcbx-fcoe-logical-link-tlv
advertise dcbx-tlv
advertise optional-tlv system-name
system-description Brocade-VDX-VCS 1
!
vlan dot1q tag native
port-profile UpgradedVlanProfile
vlan-profile
switchport
switchport trunk allowed vlan all
!
!
port-profile default
vlan-profile
switchport
switchport trunk native-vlan 1
!
!
port-profile-domain default
port-profile UpgradedVlanProfile
!
class-map cee
!
class-map default
!
rbridge-id 1
switch-attributes chassis-name VDX6740
switch-attributes host-name pod1-leaf1-1
vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
!
!
!
!
!
!
router bgp
local-as 4200001001
neighbor spine-group description connected to 4 spine-groups
neighbor spine-group password 2 $MlVzZCFAbg==

neighbor spine-group capability as4 enable

neighbor spine-group-ipv6 description connected to 4 spine-groups
neighbor spine-group-ipv6 password 2 $MlVzZCFAbg==

maximum-paths 8
graceful-restart
!

maximum-paths 8
graceful-restart
!
!
shutdown
ipv6 address fdf8:10:122:1::1/128
ip address 10.122.1.1/32
!
no shutdown
ipv6 address fdf8:10:122:1::2/128
ip address 10.122.1.2/32
!
interface Ve 101
ipv6 mtu 9000
enable
preempt-mode
priority 150
advertisement-interval 1
!
ip mtu 9000
ip proxy-arp
ip address 10.1.101.1/24
virtual-ip 10.1.101.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 102
ipv6 mtu 9000
enable
preempt-mode

priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.102.1/24
virtual-ip 10.1.102.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 103
ipv6 mtu 9000
enable
preempt-mode
priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.103.1/24
virtual-ip 10.1.103.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 104
ipv6 mtu 9000
enable
preempt-mode
priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.104.1/24
virtual-ip 10.1.104.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 105
ipv6 mtu 9000
enable
preempt-mode

priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.105.1/24
virtual-ip 10.1.105.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 106
ipv6 mtu 9000
enable
preempt-mode
priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.106.1/24
virtual-ip 10.1.106.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 107
ipv6 mtu 9000
enable
preempt-mode
priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.107.1/24
virtual-ip 10.1.107.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 108
ipv6 mtu 9000
enable
preempt-mode

priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.108.1/24
virtual-ip 10.1.108.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 109
ipv6 mtu 9000
enable
preempt-mode
priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.109.1/24
virtual-ip 10.1.109.254
enable
preempt-mode
priority 150
!
no shutdown
!
interface Ve 110
ipv6 mtu 9000
enable
preempt-mode
priority 150
!
ip mtu 9000
ip proxy-arp
ip address 10.1.110.1/24
virtual-ip 10.1.110.254
enable
preempt-mode
priority 150
!
no shutdown
!
!=========================================
rbridge-id 2
switch-attributes chassis-name VDX6740
switch-attributes host-name pod1-leaf1-2

vrf mgmt-vrf
ip route 0.0.0.0/0 10.20.32.1
!
!
!
!
!
!
!
router bgp
local-as 4200001001
neighbor spine-group description connected to 4 spine-groups
neighbor spine-group password 2 $MlVzZCFAbg==
neighbor spine-group capability as4 enable
neighbor spine-group-ipv6 description connected to 4 spine-groups
neighbor spine-group-ipv6 password 2 $MlVzZCFAbg==
neighbor spine-group-ipv6 capability as4 enable

maximum-paths 8
graceful-restart
!

maximum-paths 8
graceful-restart
!
!
shutdown
ipv6 address fdf8:10:122:1::2/128
ip address 10.122.1.1/32
!
no shutdown
ipv6 address fdf8:10:122:1::3/128
ip address 10.122.1.3/32
!
interface Ve 101
ipv6 mtu 9000
enable
preempt-mode

priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.101.2/24
virtual-ip 10.1.101.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 102
ipv6 mtu 9000
enable
preempt-mode
priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.102.2/24
virtual-ip 10.1.102.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 103
ipv6 mtu 9000
enable
preempt-mode
priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.103.2/24
virtual-ip 10.1.103.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 104
ipv6 mtu 9000
enable
preempt-mode

priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.104.2/24
virtual-ip 10.1.104.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 105
ipv6 mtu 9000
enable
preempt-mode
priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.105.2/24
virtual-ip 10.1.105.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 106
ipv6 mtu 9000
enable
preempt-mode
priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.106.2/24
virtual-ip 10.1.106.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 107
ipv6 mtu 9000
enable
preempt-mode

priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.107.2/24
virtual-ip 10.1.107.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 108
ipv6 mtu 9000
enable
preempt-mode
priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.108.2/24
virtual-ip 10.1.108.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 109
ipv6 mtu 9000
enable
preempt-mode
priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.109.2/24
virtual-ip 10.1.109.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Ve 110
ipv6 mtu 9000
enable
preempt-mode

priority 140
!
ip mtu 9000
ip proxy-arp
ip address 10.1.110.2/24
virtual-ip 10.1.110.254
enable
preempt-mode
priority 140
!
no shutdown
!
interface Management 1/0
no tcp burstrate
ip icmp echo-reply
no ip address dhcp
ip address 10.20.34.57/22
ipv6 icmpv6 echo-reply
ipv6 address 2620:100:0:fa48:34::57/64
no ipv6 address autoconfig
no ipv6 address dhcp
no shutdown
!
interface Management 2/0
no tcp burstrate
ip icmp echo-reply
no ip address dhcp
ip address 10.20.34.58/22
ipv6 icmpv6 echo-reply
ipv6 address 2620:100:0:fa48:34::58/64
no ipv6 address autoconfig
no ipv6 address dhcp
no shutdown
!
description ISL link to the peer
fabric isl enable
fabric trunk enable
no shutdown
!
fabric isl enable
fabric trunk enable
shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long

no shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!
fabric isl enable
fabric trunk enable
no shutdown
!
fabric isl enable
fabric trunk enable
no shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!
lacp timeout long
no shutdown
!
mtu 9216
description Link to Spine1
ip mtu 9100
ip proxy-arp

ipv6 mtu 9000

no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9000
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9000
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
ipv6 mtu 9000
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
no shutdown
!
mtu 9216
ip mtu 9100
ip proxy-arp
no shutdown
!


mtu 9216
ip mtu 9100
ip proxy-arp
no shutdown
!
vlag ignore-split
mtu 9022
switchport
no shutdown
!
vlag ignore-split
mtu 9022
switchport
no shutdown
!
vlag ignore-split
mtu 9022
switchport
no shutdown
!
vlag ignore-split
mtu 9022
switchport
no shutdown
!
vlag ignore-split
mtu 9022
switchport
no shutdown
!

References
References
1. Network Virtualization in IP Fabric with BGP EVPN - Extreme Validated Design
https://cloud.kapostcontent.net/pub/e043a3a3-dded-4e23-960e-e2df94f8afe0/network-virtualization-
evd?kui=K5TiUQIwFu0s1wTdY6YFRw
2. Use of BGP for Routing in Large-Scale Data Centers
https://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/
3. Extreme VDX hardware installation guides
https://documentation.extremenetworks.com/networkos/HW/vdx6940-installguide.pdf
https://documentation.extremenetworks.com/networkos/HW/vdx6740-installguide.pdf
4. Extreme SLX 9850 Router
https://www.extremenetworks.com/product/slx-9850-router/
5. Extreme SLX 9850 hardware installation guides
https://documentation.extremenetworks.com/slxos/HW/53-1004400-07_98508SLX_IG_Dec2017.pdf
6. Extreme SLX 9540 Switch Hardware Installation Guide
https://documentation.extremenetworks.com/slxos/HW/53-1004984-03_9140SLX_IG_Nov2017.pdf
https://documentation.extremenetworks.com/slxos/HW/53-1004985-03_9240SLX_IG_Nov2017.pdf

Ip Fabric Evd PDF

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Ip Fabric Evd PDF

Hochgeladen von

Copyright:

Verfügbare Formate

EXTREME VALIDATED DESIGN

Extreme IP Fabric Architecture

2 Extreme IP Fabric Architecture

List of Figures ................................................................................................................................................................................ 5

IP Fabric Overview ....................................................................................................................................................................... 11

IP Fabric Validated Designs........................................................................................................................................................... 21

Design Considerations .................................................................................................................................................................. 54

Extreme IP Fabric Architecture 3

Appendix—Configuration of the Nodes ......................................................................................................................................... 57

4 Extreme IP Fabric Architecture

Figure 2 on page 13—L2 Fabric

Figure 3 on page 14—IP Fabric (Routing to ToR)

Figure 4 on page 15—Leaf-Spine L3 Clos Topology

Figure 5 on page 16—Optimized 5-Stage L3 Clos Topology

Figure 6 on page 18—IP Fabric with eBGP as the Control Protocol

Figure 8 on page 21—Pervasive eBGP in an Optimized 5-Stage IP Fabric

Figure 9 on page 48—Connectivity Between the Racks and PoDs

Extreme IP Fabric Architecture 5

Extreme Validated Designs

Purpose of This Document

Extreme IP Fabric Architecture 7

About Extreme Networks

8 Extreme IP Fabric Architecture

Extreme IP Fabric Architecture 9

10 Extreme IP Fabric Architecture

Extreme IP Fabric Architecture 11

Evolution of Data Center Fabrics

12 Extreme IP Fabric Architecture

Layer 2 Fabric Architectures

Extreme IP Fabric Architecture 13

IP Fabric (AKA Routing to ToR)

FIGURE 3 IP Fabric (Routing to ToR)

14 Extreme IP Fabric Architecture

FIGURE 4 Leaf-Spine L3 Clos Topology

As a design principle, the following requirements apply to the leaf-spine topology:

Extreme IP Fabric Architecture 15

Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)

FIGURE 5 Optimized 5-Stage L3 Clos Topology

16 Extreme IP Fabric Architecture

IP Fabric Control Plane

Extreme IP Fabric Architecture 17

• Each leaf in a PoD is assigned its own AS number.

FIGURE 6 IP Fabric with eBGP as the Control Protocol

18 Extreme IP Fabric Architecture

iBGP for Routing Inside a PoD

FIGURE 7 IP Fabric with iBGP as the Control Protocol Inside a PoD

Extreme IP Fabric Architecture 19

20 Extreme IP Fabric Architecture

IP Fabric Validated Designs

FIGURE 8 Pervasive eBGP in an Optimized 5-Stage IP Fabric

Extreme IP Fabric Architecture 21

VDX 6940-144S Network OS 7.2.0a

1. All common configuration required on all nodes in the fabric.

2. The configuration required on the server-facing side of the ToRs or leafs.

3. The MCT configuration required on edge leafs.

22 Extreme IP Fabric Architecture

Common Configuration on All Nodes in the Fabric

!Note: this is done from the exec prompt

POD1-Leaf1-1# vcs vcsid 405 set-rbridge-id 1

Verify the Configuration

POD1-Leaf1-1# show vcs

Extreme IP Fabric Architecture 23

interface FortyGigabitEthernet 1/0/49