Data Center Interconnect Evd

EXTREME VALIDATED DESIGN
BGP-EVPN-Based Data Center Interconnect
53-1004313-04
January 2018
© 2018, Extreme Networks, Inc. All Rights Reserved.
Extreme Networks and the Extreme Networks logo are trademarks or registered trademarks of Extreme Networks, Inc. in the United States and/or other
countries. All other names are the property of their respective owners. For additional information on Extreme Networks Trademarks please see
www.extremenetworks.com/company/legal/trademarks. Specifications and product availability are subject to change without notice.
© 2017, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other
countries. Other brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at www.brocade.com/en/legal/
brocade-Legal-intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,
equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without
notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade
sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the
United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this
document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license
agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

2 53-1004313-04
Contents
Preface...................................................................................................................................................................................................................................5
Extreme Validated Designs.................................................................................................................................................................................................................... 5
Purpose of the Document......................................................................................................................................................................................................................5
Target Audience.......................................................................................................................................................................................................................................... 5
Document History......................................................................................................................................................................................................................................5
Terminology..........................................................................................................................................................................................................................7
Introduction.......................................................................................................................................................................................................................... 9
Extreme EVPN-Based DCI Deployment Model—Overview................................................................................................................................... 11
EVPN DCI Deployment Model 1—BGP-EVPN-Based L2 and L3 Extension ...................................................................................................13

IP Fabric DC Component Review.................................................................................................................................................................................................... 13
BGP-EVPN-Based L2 and L3 Extension—Extending the BGP Control Plane.......................................................................................................... 15
EVPN DCI Deployment Model 2—BGP-EVPN-Based L2 Extension..................................................................................................................21

BGP-EVPN-Based L2 Extension—DCI Tier to DCI Tier.......................................................................................................................................................22
Validated Design—EVPN DCI with BGP-EVPN-Based L2 and L3 Extension...................................................................................................25

Topology Description............................................................................................................................................................................................................................ 26
Configuration Steps................................................................................................................................................................................................................................27
Configuration—Border Leaf to Spine Layer 3.............................................................................................................................................................................27
Configuration—Border Leaf to WAN Edge Layer 3................................................................................................................................................................. 30
Configuration—Border Leaf eBGP Multihop...............................................................................................................................................................................34
Example 1—DCI L2 Extension......................................................................................................................................................................................................... 35
Example 2—DCI L2 Extension and L2 Multitenancy..............................................................................................................................................................46
Multitenancy Across DCS Using Different TORs.............................................................................................................................................................46
Multitenancy Across DCS Using the Same TOR.............................................................................................................................................................54
Example 3—DCI VLAN Routing...................................................................................................................................................................................................... 64
Asymmetric Routing ....................................................................................................................................................................................................................64
Symmetric Routing....................................................................................................................................................................................................................... 70
Example 4—Adding Services to Border-Leaf Nodes..............................................................................................................................................................78
Example 5—Extending a Tenant VRF to the WAN Edge.......................................................................................................................................................87
Example 6—Providing Internet Route Reachability for Tenant VRFs at TORs Through Public VRF at Border-Leaf............................... 100
Validated Design—EVPN DCI with BGP-EVPN-Based L2 Extension.............................................................................................................. 113

Topology Description......................................................................................................................................................................................................................... 113
Data Center Interconnect Tier......................................................................................................................................................................................................... 114
Configuration Steps.............................................................................................................................................................................................................................116
Configuration: DCI Tier to WAN Edge.........................................................................................................................................................................................123
Example 1—Layer 2 Extension......................................................................................................................................................................................................124
Example 1a: L2 Extension Between DCI Tier – DCI Tier..........................................................................................................................................124
Example 1b: L2 Extension Between DCI Tier - Leaf Node..................................................................................................................................... 140
Example 2—VLAN Routing.............................................................................................................................................................................................................154
Example 2a: Symmetric VLAN Routing Between Two Flexible Type Data Centers .................................................................................... 154
Example 2b: Symmetric VLAN Routing between flexible type and IP Fabric Data Centers......................................................................165
Example 3—Providing Internet Route Reachability for Tenant VRFs at DCI Tier Through Public VRF .........................................................176
Example 4a—DCI L2 Extension....................................................................................................................................................................................................187
Example 4b—VLAN Asymmetric Routing................................................................................................................................................................................ 210

53-1004313-04 3
Validated Design: EVPN DCI with BGP-EVPN-Based L2 and L3 Extension through Spines.................................................................... 225
Topology Description......................................................................................................................................................................................................................... 225
Hardware/Software Matrix ...............................................................................................................................................................................................................226
Configuration Steps.............................................................................................................................................................................................................................226
Configuration: Spine to Spine Layer 3........................................................................................................................................................................................ 226
Example 1—DCI L2 Extension...................................................................................................................................................................................................... 228
Example 2—DCI VLAN Routing .................................................................................................................................................................................................. 241
Inter VLAN traffic........................................................................................................................................................................................................................ 241
Symmetric Routing ...................................................................................................................................................................................................................249
Design Considerations..................................................................................................................................................................................................261
Tunnel Scale........................................................................................................................................................................................................................................... 261
Tunnels * VLANs.................................................................................................................................................................................................................................. 261
BGP-EVPN-Based L2 and L3 Extension Validated Scale............................................................................................................................................... 261
BGP-EVPN-Based L2 Extension Validated Scale................................................................................................................................................................262
References.......................................................................................................................................................................................................................263

4 53-1004313-04
Preface
• Extreme Validated Designs...............................................................................................................................................................................5
• Purpose of the Document.................................................................................................................................................................................5
• Target Audience.....................................................................................................................................................................................................5
• Document History................................................................................................................................................................................................ 5
Extreme Validated Designs

Extreme validated designs are reference architectures that are created and validated by Extreme engineers to address various customer
deployment scenarios and use cases. These validated designs provide a well-defined and standardized architecture for each deployment
scenario, and they incorporate a broad set of technologies and feature sets across Extreme's product range that address customer-
unique requirements. These designs are comprehensively validated end-to-end so that the design solutions and configurations can be
deployed more quickly, more reliably, and more predictably. Extreme validated designs are continuously validated using a test automation
framework to ensure that once a design has been validated, it remains validated on new software releases and products.
Purpose of the Document

This Extreme validated design provides guidance for implementing EVPN-based Data Center Interconnect (DCI) using Extreme
hardware and software. It details the Extreme reference architecture for two unique deployment models:
• BGP-EVPN-based L2 extension
• BGP-EVPN-based L2 and L3 extension
It should be noted that not all features, such as automation practices, zero-touch provisioning, and monitoring of the Extreme IP fabric,
are included in this document. Future versions of this document are planned to include these aspects of the Extreme IP fabric solution.
The design practices documented here follow the best-practice recommendations, but there are variations to the design that are
supported as well.
Target Audience
This document is written for Extreme system engineers and network architects who design, implement, and support data center
networks. This document is intended for experienced data center architects and network administrators/engineers. The reader must have
a good understanding of data center switching and routing features and Multi-Protocol BGP/MPLS VPN for understanding multitenancy
in VXLAN EVPN networks.
Document History
Date Part Number Description
April 2016 53-1004313-01 Initial version.

September 13, 2016 53-1004313-02 Multitenancy across DCS using the same TORs.
Providing Internet route reachability for tenant VRFs at TORs through public VRF at border leaf.

53-1004313-04 5
Document History
Date Part Number Description
Layer 2 extension in EVPN DCI with BGP-EVPN-based L2 extension.
VLAN routing in EVPN DCI with BGP-EVPN-based L2 extension.
Providing Internet route reachability for tenant VRFs at DCI tier through public VRF.
Design considerations.
December 2016 53-1004313-03 EVPN DCI with BGP-EVPN-based L2 and L3 extension through Spines.
January 2018 53-1004313-04 Updated document to reflect Extreme's acquisition of Brocade's data center networking business.

6 53-1004313-04
Terminology
Terms Description
Active-Active vLAG Active-Active Virtual Link Aggregation Group

AF Address Family
ARP Address Resolution Protocol
ASN Autonomous System Number
BGP Border Gateway Protocol
BL Border Leaf
BUM Broadcast, Unicast, and Multicast
CLI Command-Line Interface
DC Data Center
DCI Data Center Interconnect
DCS Data Center Site
eBGP Exterior Border Gateway Protocol
ECMP Equal Cost Multi-Path
EVPN Ethernet Virtual Private Network
iBGP Interior Border Gateway Protocol
IMR Inclusive Multicast Route
IP Internet Protocol
IRB Integrated Routing and Bridging
LAG Link Aggregation Group
LDP Label Distribution Protocol
LSP Label Switched Path
MAC Media Access Control
MH Multihop
BGP Border Gateway Protocol
MPLS Multi-Protocol Label Switching
ND Neighbor Discovery
Overlay GW Overlay Gateway
PoD Point of Delivery
RD Route Distinguisher
RT Route Target
ToR Top of Rack
UDP User Datagram Protocol
VCS Virtual Cluster Switching
vLAG Virtual Link Aggregation Group
vLAG pair Virtual Link Aggregation Group pair
VLAN Virtual Local Area Network
VM Virtual Machine
VNI Virtual Network Identifier

53-1004313-04 7
Terms Description
VPN Virtual Private Network
VRF Virtual Routing and Forwarding
VTEP VXLAN Tunnel End Point
VXLAN Virtual Extensible LAN
WAN Wide Area Network

8 53-1004313-04
Introduction
Based on the principles of the New IP, Extreme is building on the proven success of the VDX platform by expanding our cloud-optimized
network and network virtualization architectures to meet customer demand for higher levels of scale, agility, and operational efficiency.
This document describes network designs for interconnecting data center sites leveraging BGP EVPN. The intention of this Extreme
validated design document is to provide reference configurations and document the best practices for interconnecting data centers using
VDX switches with BGP EVPN.
This document describes the following architectures:


53-1004313-04 9
10 53-1004313-04
Extreme EVPN-Based DCI Deployment
Model—Overview
Many data center deployments are required to span multiple geographically separated sites for availability and performance. Availability
in this context comes from site/tenant-level backup and redundancy to safeguard against infrastructure failures and provide increased
application and service reliability. The requirement of the data center network to span multiple sites may include extending the Layer 3
(and, in many cases, the Layer 2) reachability between sites.
There are two EVPN-based DCI deployment models detailed in this document:
Both of these models leverage VXLAN for efficient tunneling of traffic across a core network between data centers; they are differentiated
by how each data center "hands off" traffic to the core network, i.e., either at Layer 2 or at Layer 3.
The BGP-EVPN-based L2 and L3 model is targeted at interconnecting EVPN-based IP fabric data centers; whereas the EVPN-based
L2 model provides a more generic DCI solution with L2 VLAN extension from any type of data center deployment, e.g., VCS or a BGP
EVPN IP fabric. There are multiple design considerations for each; a brief summary is given in the following table, and details are
discussed further in the upcoming sections.
TABLE 1 BGP-EVPN-Based DCI Model Comparison

DCI Requirement BGP-EVPN-Based L2 & L3 Extension (DCI for BGP-EVPN-Based L2 Extension (DCI for
EVPN-Based IP Fabric DCs) Flexible DC Type)
Layer 2 extension Yes Yes
(L2 EVPN control-plane learning between DCs) (Data-plane learning between border leaf and
DCI tier)
Inter-VLAN routing Yes Yes
(Asymmetric or symmetric routing with L3 VNI) (Asymmetric or symmetric routing at DCI tier)
VLAN re-use Yes Limited
(VLAN re-use between tenants and leafs) (VLAN-to-VNI mapping at DC leaf only) (VLANs converge at the DCI tier and DC edge,
e.g. border leaf of EVPN-based IP fabric)
Control-plane segmentation Not segmented Segmented
(Demarcation between DCs and the DCI) The control plane is extended via the WAN and The control plane is extended via the WAN
is shared between data centers. between DCI tiers, but is not shared between
data centers. Segmentation can be avoided with
control-plane extension from the DCI tier to the
leaf node.
VXLAN tunnel scale Tunnels span between leafs of EVPN-based IP DCI tier to DCI tier tunnel scale (dependency
fabric DCs (tunnel scale: many to many) on number of remote sites)
Scale consideration at DC leaf (VXLAN tunnel Tunnels contained inside EVPN-based IP

from a given leaf to each remote leaf sharing a fabric DC (tunnel scale: many to one, i.e. N
common VNI). leafs to border leaf)
Scale consideration at border leaf (BL will have a

tunnel to each leaf requiring VLAN extension).
VLANs extended over VXLAN Scale consideration at leaf (many to many) Scale consideration at DCI tier and BL nodes
for IP fabric EVPN DC (many to one)
High VLAN and VNI scale between data centers
is possible with distributed scale across N leafs.

53-1004313-04 11
12 53-1004313-04
EVPN DCI Deployment Model 1—BGP-
EVPN-Based L2 and L3 Extension
• IP Fabric DC Component Review...............................................................................................................................................................13
• BGP-EVPN-Based L2 and L3 Extension—Extending the BGP Control Plane.....................................................................15
The BGP-EVPN-based L2 and L3 extension DCI deployment model is designed for interconnecting BGP-EVPN-based (IP fabric) data
centers by extending the control plane between sites. With a common control plane, the interconnected sites behave as a single logical
data center, enabling efficient traffic patterns across an IP interconnect network. This deployment model provides the following key
benefits:
• Layer 2 extension and Layer 3 VRF host routing
• Dynamic VXLAN tunnel discovery and establishment
• BUM reduction with MAC address reachability exchange and ARP/ND suppression
• Conversational ARP/ND
• VM mobility support
• VXLAN head-end replication and single-pass efficient VXLAN routing
• Open standards and interoperability
IP Fabric DC Component Review

Before jumping into interconnecting EVPN-based IP fabric data centers, let's review the basic IP fabric design and its key elements. The
design is based on a leaf-spine multistage (e.g., 3 or 5) folded Clos topology that leverages Layer 3 ECMP between the leaf and spine
nodes. An example topology is shown in the following figure:

53-1004313-04 13
IP Fabric DC Component Review
FIGURE 1 A 3-Stage Folded Clos Topology with Border Leaf
The basic IP fabric topology consists of the following elements:

• Spine layer
• Leaf layer
• Border leaf
Spine Layer
The role of the spine is to provide interconnectivity between the leafs. Network endpoints do not connect to the spines. Since most policy
is implemented at the leafs, the major role of the spine is to participate in the control-plane and data-plane operations for traffic
forwarding between leaf switches. Some differentiating characteristics of spine nodes include:
• Individual nodes have Layer 3 connectivity to each physical leaf switch.
• Spine nodes are not physically or logically connected to each other.
Leaf Layer
The role of the leaf switch is to provide connectivity to the endpoints in the network. These endpoints include compute servers and
storage devices, as well as other networking devices like routers and switches, load balancers, firewalls, and any other networking
endpoint—physical or virtual. For network efficiency, policy enforcement, including security, traffic path selection, Quality of Service (QoS)
marking, traffic policing, shaping, and traffic redirection, is implemented on leaf switches. Some differentiating characteristics at the leaf
layer are:
• Server VLANs terminate at the leaf switches (Layer 2 from devices to leaf).
• Leaf switches can be deployed individually as a top-of-rack device or as a pair providing switch-level redundancy with active-
active vLAG connections to servers.

14 53-1004313-04
BGP-EVPN-Based L2 and L3 Extension—Extending the BGP Control Plane
• L3 connectivity exists between the spine and leaf switches using L3 physical ports.
• Routing underlay: BGP is used to propagate IPv4/IPv6 routes with BGP neighbors formed from each leaf switch to each spine.
• Load balancing is achieved with L3 ECMP.
• Leaf-to-spine inter-switch point-to-point L3 links configured as “IP Unnumbered” or /31 subnets to conserve IP addresses
and optimize hardware resources (best practice).
Border Leaf
The role of the border leaf switches in the network is to provide external connectivity to the data center site and access to associated
access services like firewalls, load balancers, and edge VPN routers. The border leaf switches together with the edge racks housing these
common services form the edge services PoD. Since all North-South traffic will pass through the border leaf switches, it is important to
account for the bandwidth requirements for both:
• Internet traffic (external access to/from the data center)
• Data Center Interconnect (DCI) traffic (traffic passing between interconnected data centers, e.g., backup)
The ratio of the aggregate bandwidth of the uplinks connecting to the spines (two-tier case) or super-spines (three-tier case) to the
aggregate bandwidth of the uplink connecting to the WAN edge routers determines the over-subscription ratio for traffic exiting the data
center site.
The figure above shows the positioning and connectivity of a border leaf switch pair in a two-tier topology: that is, border leaf switches are
connected to all spines in the DC PoD (same as standard leaf switches) and also have external-facing connections to the WAN edge. In
the case of a three-tier fabric topology, border leaf switches would be connected to the super-spines (third tier), providing external
connectivity for N data center PoDs. The border-leaf to spine/super-spine connections are strictly Layer 3 with a BGP EVPN underlay;
whereas the border-leaf to WAN connections can be either Layer 2 or Layer 3 or a combination of both depending on the requirements
and the DCI deployment model. The upcoming sections will focus on the DCI deployment model details.
BGP-EVPN-Based L2 and L3 Extension—Extending

the BGP Control Plane
In the case of the BGP-EVPN-based L2 and L3 extension deployment model, WAN edge routers and IP/MPLS network are providing
only IP reachability and transport between data centers. Border leaf nodes in each data center learn how to reach each other from their
respective WAN edge routers. This is achieved by border leaf nodes advertising their peering address (e.g., local loopback) to the local
WAN edge router, which will, in turn, share the routing information with the remote WAN edge routers and remote border leaf nodes.
Once border leaf nodes have IP reachability to each other, an eBGP (multihop) session can be established. The following figure shows an
example of multihop eBGP peering between border leaf nodes in DC 1 and border leaf nodes in DC 2.

53-1004313-04 15
FIGURE 2 DC1-DC2 eBGP Multihop Peering Between Border Leaf Nodes
To extend the EVPN control plane between sites, the EVPN address family is enabled for the eBGP multihop peering between border
leaf nodes. Continuing the example above and enabling the EVPN address family, the border leaf nodes will send EVPN routes from
their respective data centers to the remote data center; e.g., the border leaf from DC1 sends EVPN routes from DC1 to DC2 and vice
versa. The border leaf nodes then propagate the routes into their local data center. Depicted in the figure below, both data centers now
dynamically share routing information (i.e., IPv4 for VTEP reachability and EVPN) by extending the BGP control plane between sites.
While the control plane is extended over a separate network (e.g. third-party service provider), the internal EVPN routes are not
exchanged with the network providing the extension. That is, by establishing the BGP peering directly between border leaf nodes, BGP
update messages are exchanged directly between border leaf nodes only and not with the WAN edge routers. The WAN edge routers will
route the BGP control traffic only across the transport network. The route information exchanged between the border leaf and the WAN
edge is limited to the following:
• Border leaf router ID: For establishing eBGP MH neighborship.
• Leaf switch VTEP IPs: Forwarding across the IP core network is based on the destination VTEP IP.

16 53-1004313-04
FIGURE 3 DC1-DC2 DCI with Extended Control Plane (EVPN)
Behavior/Core Functions
Multiple data center sites sharing a common BGP-EVPN control plane will behave as a single logical IP fabric data center, enabling L2
VLAN extension and routing between VLANs between leaf switches at different sites.
Layer 2 Extension
Through the exchange of EVPN routes that contain VXLAN tunnel endpoint (VTEP) IP addresses between sites, leaf switches discover
remote leaf switch VTEP IP addresses (automatic VTEP discovery via EVPN Type 3 IMR). Leaf switches that share common VNIs will
dynamically create VXLAN tunnels between them using the discovered VTEP IP addresses.
The figure below shows an example of tunnel formation from a leaf switch in DC1 to a leaf switch in DC2, providing Layer 2 VLAN
extension. Layer 2 traffic is "tunneled" by encapsulating it into an IP User Datagram Protocol packet with an additional VXLAN header.
The outer IP source and destination for tunneled traffic are the source and destination VXLAN tunnel endpoint (VTEP) IP addresses in
this case, the leaf switches in DC1 and DC2 respectively. All transit routers forward the encapsulated Layer 2 traffic based on the outer IP
header, and only the router configured with the destination VTEP de-capsulates the packet to expose the inner Layer 2 frame . With the
Layer 3 handoff deployment model, the border leaf nodes provide both control-plane extension through the exchange of BGP EVPN
routes and data-plane forwarding for IP traffic (including tunneled VXLAN traffic) between sites.
The figure below shows an example of tunnel formation between leaf switches in DC1 and DC2 over an IP/MPLS network. After VXLAN
tunnel formation between leaf switches, Layer 2 traffic will be tunneled between sites. A 5-step example for L2 forwarding is shown:

53-1004313-04 17
FIGURE 4 DC1-to-DC2 VXLAN Tunnel Formation
1. A host in data center 1 forwards Ethernet traffic to its directly attached leaf switch (e.g. known unicast or BUM traffic).
2. Leaf switch in data center 1 receives the L2 traffic, learns or refreshes the source MAC address (data-plane learning), looks up
the destination MAC address, and encapsulates the received Ethernet frame into an IP User Datagram Protocol packet in which
the IP source/destination will be equal to the VTEP source/destination IP addresses plus a VXLAN header using automatic
(1:1) or user-defined VNI mapping and forwards the traffic to the spine layer.
NOTE
The source MAC address learned by the leaf switch is shared within the data center using BGP EVPN update
messages. The border leaf exchanges the BGP update messages with remote DC2 via its border leaf nodes (control-
plane learning). BGP updates are shared directly between border leaf nodes via eBGP multihop peering; i.e., updates
are not shared or leaked from the border leaf to the WAN edge.
3. The following nodes in this example all perform forwarding based on the destination VTEP IP address of the encapsulated
VXLAN packet (from Step 2):
• DC 1 spine
• DC 1 border leaf
• WAN edge and IP/MPLS core
• DC2 border leaf
• DC2 spine
4. The DC2 destination leaf switch receives traffic with a destination IP address matching the local VTEP address, performs
decapsulation revealing the inner Ethernet frame, and forwards traffic in the destination VLAN over the L2 interface toward the
target host.
5. The destination host in DC2 receives L2 traffic from its directly attached leaf switch.
Inter-VLAN Routing

18 53-1004313-04
The Layer 3 deployment model supports both asymmetric and symmetric routing for inter-VLAN traffic. Symmetric routing is the
recommended approach for the L3 DCI deployment model to simplify the configuration requirements and efficiently use the resources
at the leaf layer.
• Asymmetric routing—Both source and destination VLANs and associated gateways are configured on ingress and egress leaf
switches. Traffic is routed between the source and destination VLAN by the ingress leaf and is then tunneled to the remote leaf
using the VNI that is mapped to the destination VLAN. The inner L2 frame is then decapsulated at the remote egress leaf and
forwarded in the destination VLAN.
• Symmetric routing—The destination VLAN and gateway are not configured on the ingress leaf switch, and a common VNI is
used for extension between racks. Remote prefixes are advertised within the BGP EVPN address family as reachable with a
next hop equal to the remote leaf VTEP IP address and a VNI shared between to be used for tunneling traffic between local and
remote racks. When the same VLAN extension is not configured between two leaf nodes, leaf switches will not exchange
inclusive multicast routes (Type 3 routes). In the symmetric case, the leaf switches exchange L3 prefixes (Type 5 routes used for
automatic VTEP discovery), which will form a VXLAN tunnel between the leaf switches using a common VNI. A simplified
example is given in the following figure to illustrate the high-level steps for symmetric routing.
– The ingress leaf in DC 1 receives traffic from the VLAN 204 subnet and performs L3 lookup for the destination subnet
VLAN 201, and it resolves the NH to a remote VTEP in DC 2 with a VNI 2001 to be used for transport (associated with
the source and destination leaf switches).
– VXLAN-encapsulated traffic is routed between DC1 and DC2, and the destination IP address is the DC2 leaf VTEP.
– The egress leaf in DC2 de-capsulates the VXLAN traffic, performs L3 lookup for the destination subnet, and via the
destination VLAN GW, resolves the destination ARP and forwards traffic accordingly at L2 to the target host in VLAN 201.
FIGURE 5 VLAN Reuse Between Tenants in DC1 to DC2
The control-plane capability of the border leaf is unique within the IP fabric since it will not filter BGP-EVPN routes based on route
targets; i.e., it passes on advertisement of all routes to its neighbors similar to a spine node and also has the capability of initiating and
terminating tunnels as standard leaf switches. The specific configuration requirements are detailed in the validated design sections that
follow.
Shared Control Plane

53-1004313-04 19
One of the requirements for the BGP-EVPN-based L2 and L3 extension model is that the control plane is shared between sites. This
model is best suited for deployments where the operational/administrative control is centralized between sites to allow for effective
control and configuration, e.g., ensuring consistent VLAN-to-VNI mapping in local and remote data centers.

20 53-1004313-04
EVPN DCI Deployment Model 2—BGP-
EVPN-Based L2 Extension
• BGP-EVPN-Based L2 Extension—DCI Tier to DCI Tier................................................................................................................. 22
The BGP-EVPN-based L2 extension Data Center Interconnect (DCI) deployment model is designed to provide interconnection between
data centers at Layer 2 regardless of the data center type, e.g. VDX VCS, IP fabric. This deployment model introduces a new layer
referred to as the DCI tier, which connects to WAN edge routers at Layer 3 and the data center at Layer 2. The Layer 2 connection to the
data center refers to untagged or tagged Ethernet (802.3/802.1Q), and for redundancy, the recommended topology is to use two DCI
tier nodes connected as a vLAG pair. The following figure illustrates the DCI tier placement and connectivity to the WAN edge.
FIGURE 6 DCI Tier Network Placement
The DCI tier leverages the same underlying concepts described for the border leaf nodes in the Layer 3 handoff model; that is, DCI tier
nodes share a common extended control plane between sites. The differentiator is that ingress traffic to the border leaf is strictly Layer 2,
and the DCI tier nodes perform VTEP functions for inter-site traffic. The use of a shared EVPN control plane between DCI tiers enables
efficient forwarding across an IP interconnect network in addition to the following:
• Layer 2 extension and Layer 3 VRF host routing
• Dynamic VXLAN tunnel discovery and establishment (between DCI tier nodes)
• BUM reduction with MAC address reachability exchange and ARP/ND suppression
• Conversational ARP/ND
• VXLAN head-end replication and single-pass efficient VXLAN routing
• Open standards and interoperability

53-1004313-04 21
BGP-EVPN-Based L2 Extension—DCI Tier to DCI Tier
BGP-EVPN-Based L2 Extension—DCI Tier to DCI

Tier
As with the BGP-EVPN-based L2 and L3 extension deployment model, the L2 extension model uses eBGP multihop for peering with
EVPN for extending the control plane between sites and the WAN edge routers and an IP/MPLS network only to provide IP reachability.
The difference with the BGP-EVPN-based L2 model is that the peering between sites is between DCI tier nodes, and the interface to the
local data center is Layer 2. DCI tier nodes in each data center learn how to reach each other from their respective WAN edge routers.
This is achieved by DCI tier nodes advertising their peering address (e.g. local loopback) to the local WAN edge router, which, in turn, will
share the routing information with the remote WAN edge routers and remote DCI tier nodes. Once DCI tier nodes have IP reachability to
each other, an eBGP (multihop) session can be established. The following figure shows an example of multihop eBGP peering between
DCI tier nodes in DC 1 and border leaf nodes in DC 2.
FIGURE 7 DC1-DC2 DCI Tier eBGP Multihop Peering
Layer 2 Extension
Through the exchange of EVPN routes between DCI tier nodes, automatic VTEP discovery occurs (updates contain VTEP IP addresses).
DCI tier nodes sharing common VNIs will dynamically create VXLAN tunnels between them using the discovered VTEP IP addresses.
The following figure shows an example of tunnel formation between DCI tier nodes over an IP/MPLS network. After VXLAN tunnel
formation between DCI tier nodes, Layer 2 traffic will be tunneled between sites. A 5-step example for L2 forwarding is shown:

22 53-1004313-04
FIGURE 8 Packet Path Between Two Data Center Sites
1. Data center 1 forwards an Ethernet frame to its local DCI tier node (e.g. known unicast or BUM traffic).
2. The DCI tier at data center 1 receives the L2 traffic and learns or refreshes the source MAC address (data-plane learning), looks
up the destination MAC address, and encapsulates the received Ethernet frame into an IP UDP packet in which the IP source/
destination will be equal to the VTEP source/destination IP addresses plus a VXLAN header using automatic (1:1) or user-
defined VNI mapping, and it forwards the traffic to the spine layer and forwards the traffic to the WAN edge.
NOTE
The source MAC address learned by the DCI tier is shared using MP BGP-EVPN routes with remote DCI tier nodes
(control-plane learning), and BGP updates are shared directly between DCI tier nodes via eBGP multihop peering (i.e.,
updates are not shared or leaked to the WAN edge).
3. The WAN edge receives encapsulated traffic and performs forwarding based on the outer IP header (e.g., simple L3 forwarding
or MPLS depending on the core network).
4. Traffic received at the remote DCI tier with a destination IP address matching the local VTEP address is decapsulated, revealing
the inner Ethernet frame, and is forwarded in the destination VLAN over the L2 interface connected to data center 2.
5. Data center 2 receives the Ethernet traffic from the DCI tier as L2 traffic and adds or refreshes the source MAC address in its
table (data-plane learning).
In short, DCI tier nodes perform data-plane learning over their local L2 interfaces and control-plane learning over their L3 interfaces for
remote MAC addresses, ARP, etc. The result is efficient forwarding by DCI tier nodes because remote MAC addresses and ARPs are
shared with remote DCI tier nodes, reducing the amount of BUM traffic over the core network.
Inter-VLAN Routing

53-1004313-04 23
The BGP-EVPN-based L2 extension deployment model is targeted at extending Layer 2 VLANs across a shared core network. For
cases where routing between VLANs is required, there are two ways to achieve it- Asymmetric and Symmetric routing. In Asymmetric
routing the packet is routed first inside the DC, then switched to destination. Symmetric routing achieves routing at the gateway level
using common L3 VNI extension. When the individual data center control planes are separated by an L2 boundary (i.e., DC to DCI tier),
inter-VLAN traffic will be routed asymmetrically. The DCI tier nodes then receive and transport traffic in a single VLAN to the remote site.
When data center control planes are extended across without a boundary, Symmetric routing is efficient.
VLAN Scoping/Multitenancy
Traffic between sites is tunneled using VXLAN encapsulation as described in the example above, and the VLAN to VXLAN VNI mapping
is configured at the DCI tier nodes. For traffic between sites, the separation is based on the VNI. That is, inter-site forwarding with this
deployment model will only occur for cases where the VNI is common between local and remote DCI tier nodes. Therefore, different
tenants at different sites can use overlapping VLANs provided they use unique VNIs for transport across the core network.
Separated Control Plane/Administrative Control

For the BGP-EVPN-based L2 extension deployment model, the BGP EVPN control plane between DCI tier nodes is shared between
sites; whereas, the control plane between data centers is separated and relies on data-plane learning between the DC and DCI tier. The
separation between remote data centers may be well suited to operational scenarios where administrative control for data centers at
different sites is split between different teams in an enterprise, allowing each to manage and configure their data centers independently
and only hand off a VLAN to the DCI tier for extension; versus the BGP-EVPN-based L2 and L3 extension deployment model, where
administrative control over interconnected data centers would be better suited to a single team since the control plane between DCs is
shared.

24 53-1004313-04
Validated Design—EVPN DCI with BGP-
• Topology Description.......................................................................................................................................................................................26
• Configuration Steps.......................................................................................................................................................................................... 27
• Configuration—Border Leaf to Spine Layer 3....................................................................................................................................... 27
• Configuration—Border Leaf to WAN Edge Layer 3............................................................................................................................ 30
• Configuration—Border Leaf eBGP Multihop..........................................................................................................................................34
• Example 1—DCI L2 Extension....................................................................................................................................................................35
• Example 2—DCI L2 Extension and L2 Multitenancy........................................................................................................................ 46
• Example 3—DCI VLAN Routing................................................................................................................................................................. 64
• Example 4—Adding Services to Border-Leaf Nodes.........................................................................................................................78
• Example 5—Extending a Tenant VRF to the WAN Edge................................................................................................................. 87
• Example 6—Providing Internet Route Reachability for Tenant VRFs at TORs Through Public VRF at
Border-Leaf.......................................................................................................................................................................................................100
This section provides step-by-step configuration examples for the BGP-EVPN-based L2 and L3 extension deployment model based
on a test topology, and it walks through common use cases with selected show commands to demonstrate intended functions.
FIGURE 9 Topology

53-1004313-04 25
Topology Description
• In Data Center Site1, all leaf nodes are connected to four spine nodes (with IPV4 addresses configured on interfaces in /31
subnet) using IPv4 eBGP adjacency with all four spine nodes in the same AS 64610. Leaf 1 and Leaf 2 are single, and Leaf3-
Leaf4, Leaf5-Leaf6, Border-Leaf1-Border-Leaf2 are a vLAG-pair. Leaf 1 is in AS 64630, Leaf 2 is in AS 64650, Leaf 3-Leaf
4 are in AS 64640, Leaf5-Leaf 6 are in AS 64670, and Border-Leaf1-Border-Leaf2 are in AS 64680. ECMP is achieved
using multipath eBGP.
• In Data Center Site2, all leaf nodes are connected to four spine nodes (with IPV4 addresses configured on interfaces in /31
subnet) using IPv4 iBGP adjacency with spine nodes being route-reflectors. All nodes are in AS 64620. Peer group is
configured to establish the BGP adjacency. ECMP is achieved using BGP add-path capability. Border-Leaf3-Border-Leaf4 are
a vLAG pair and all other leaf nodes Leaf 7, Leaf 8, Leaf 9, and Leaf 10 are single nodes.
• Leaf-spine adjacencies are activated under L2VPN EVPN address-family on all leaf and spine switches. Leaf-spine adjacencies
are configured with next-hop-unchanged to advertise routes from EVPN peers to other EVPN peers without changing the next
hop.
• In spine switches, retain route-target all is configured under EVPN address-family. This is to prevent stripping of RTs when
passing routes from one hop to another hop. Leaf switches compare RTs before installing routes with import RT under local
EVPN instance, RT advertised by each leaf node should be maintained before reflecting to other leaf nodes.
• VTEP addresses (Loopbacks) are advertised using the network command. Next-hop-recursion is used for next-hop-
reachability on Data Center Site2 since it is iBGP and redistribute connected is used on all spine nodes to provide next-hop
reachability.
• Border-Leaf1 and Border-Leaf2 are connected to WAN edge1 and WAN edge2 respectively using 4-10G port ECMP and
LAG. Border-Leaf3 and Border-Leaf4 are connected to WAN edge3 and WAN edge4 respectively using 4-10G port ECMP
and LAG. Border-Leaf node pairs are connected to respective WAN edge node pairs (with IPv4 address configured on LAG
interfaces in /31 subnet) using IPv4 eBGP adjacency with all WAN edge nodes in same AS 30614.
• L3 MPLS VPN adjacency is established between Site1 and Site2 WAN edge nodes.
• eBGP multihop session is established between Border-Leaf pair on Data Center Site1 and Border-Leaf pair on Data Center
Site2. Multihop BGP adjacency between Border-Leaf pairs on DCS1 and DCS2 are activated under EVPN address-family.
• Leaf to Host interfaces are configured as an active-active vLAG (aggregation of multiple physical links across multiple switches
from a single fabric forming single logical interface). The interfaces can be in access or trunk VLANs with IPV4, IPV6 any cast
address configured to allow VM mobility within or across DCS.
• Overlay gateway is configured in global context on all leaf nodes (applies to both nodes in case of two node vLAG pair) with
type of overlay to be used, respective VLAN VNI mapping, VTEP membership, switches membership, and VXLAN monitoring
like VLAN stats and SFLOW.
• EVPN instance is configured under rbridge mode for each leaf with RD, RT, VNIs to be extended.
• The retain route-target all command is configured on border-leaf nodes in order to advertise EVPN routes between data center
sites without stripping RT to form tunnel between leaf nodes from Site 1 and Site 2. In this approach, overlay gateway and
EVPN instance configurations can be avoided on border-leaf nodes. In case of symmetric routing, VRF configuration is not
needed on border-leaf nodes. Hence, border-leaf nodes will not form tunnels to other leaf nodes.
• If services have to be added on border-leaf nodes, they have to have tunnels. For this, needed VLAN-VNI mapping should be
added under overlay-gateway configuration with EVPN instance on border leaf nodes.
Hardware/Software Matrix
Role of Node Chassis Name (Possible Chassis Types) Minimum Software Version Required
Leaf BR-VDX6940-36Q Network OS 7.0 and later
BR-VDX6940-144S

26 53-1004313-04
Configuration—Border Leaf to Spine Layer 3
BR-VDX6740T
BR-VDX6740
Border leaf BR-VDX6940-36Q Network OS 7.0 and later
BR-VDX6940-144S
Spine BR-VDX8770-4/8 Network OS 7.0 and later
BR-VDX6940-36Q
BR-VDX6940-144S
DCI tier BR-VDX6940-36Q Network OS 7.0 and later
BR-VDX6940-144S
BR-VDX6740
WAN edge MLXe-4/8/16/32 NetIron 5.9.00
Configuration Steps
The BGP-EVPN-based L2 and L3 extension deployment model is characterized by the following:
• Use of Layer 3 interfaces between the border leaf nodes and the WAN edge routers
• Layer 3 reachability between border leaf nodes in different data centers via the WAN edge routers (IP transport)
• BGP neighborship between border leaf nodes in different data centers (eBGP multihop) with EVPN AF enabled

Interface configuration on Border-Leaf1 to Spine A (similar configuration is needed on interfaces to other spines from Border-Leaf1 and
on interfaces from Border-Leaf2 to spines).
BGP Configuration on Border-Leaf1 to spines (similar configuration is on Border-Leaf2 to spines with respective IP addresses).

53-1004313-04 27
Verify that eBGP neighborship is established on Border Leaf 1 to Spine A.
Verify that EVPN neighborship is established on Border Leaf 1 to Spine A.

28 53-1004313-04
Interface Configuration on Border-Leaf3 to Spine I (similar configuration is needed on interfaces to other spines from Border-Leaf3 and
on interfaces from Border-Leaf4 to Spines).
BGP Configuration on Border-Leaf3 to Spines (similar configuration is on Border-Leaf2 to Spines with respective IP addresses).
Verification can be done similar to verification of Border-Leaf1 to Spine.

53-1004313-04 29
Configuration—Border Leaf to WAN Edge Layer 3

• It is recommended to use two-node vLAG pair on border leaf to avoid a single point of failure.
• It is recommended to have full-mesh eBGP adjacency between each border-leaf node to WAN edge nodes.
• It is recommended to use LAG between the border-leaf node and the WAN edge.
• The network command is used to advertise the loopback interface to border-leaf nodes.
• eBGP adjacency between the border leaf and the WAN edge is not activated under EVPN address-family.
• Tracking the links between WAN edge and border leaf can help isolate a potential traffic black hole when all the links from one
BL to WAN Edge fail. This can be achieved by configuring link-tracking feature on the border leaf nodes.
TABLE 2 Border Leaf to WAN Edge Interface Connections

Connection Between Nodes Type of Connection
Border-Leaf1 to WAN Edge 1 4 - path ECMP (10G each)

Border-Leaf2 to WAN Edge 2 4 - 10G port LAG
Border-Leaf3 to WAN Edge 3 4 - path ECMP (10G each)
Border-Leaf4 to WAN Edge 4 4 - 10G port LAG
Interface Configuration on Border-Leaf1 to WAN Edge1 (a similar configuration is needed on other ECMP ports and on ECMP ports
used in Border-Leaf3).
Interface Configuration on Border Leaf 2 to WAN Edge2 (a similar configuration is needed on Border-Leaf4).

30 53-1004313-04
BGP configuration on Border-Leaf1 to WAN Edge1 ( a similar configuration is needed on other border-leaf nodes too).
Verification of eBGP neighborship from Border Leaf 1 to WAN Edge 1 can be done using the show ip bgp summary command as in
Border-Leaf1 to Spine verification.
WAN Edge 1 to Border Leaf 1 Interface Configuration (a similar configuration is needed on other interfaces connected to border-leaf
nodes and on other WAN edges too).

53-1004313-04 31
WAN Edge 1 to MPLS Core Interface Configuration (a similar configuration is needed on other WAN edges too. This interface will be
added into MPLS configuration).
BGP configuration on WAN Edge 1 (a similar configuration is needed on other WAN edges).

32 53-1004313-04
Verify that eBGP neighborship is established from Border Leaf 1 to WAN edge 1.

53-1004313-04 33
Configuration—Border Leaf eBGP Multihop
Configuration—Border Leaf eBGP Multihop

• Full-mesh eBGP multihop adjacency is established between each border-leaf node from a DCS to two border-leaf nodes in
another DCS and are activated under EVPN address-family. (No peering is established between local border-leaf nodes that are
a vLAG pair.)
• The retain route-target all command is configured under EVPN address-family of border-leaf nodes to advertise EVPN routes
to peers without stripping the route target. In this approach, the user doesn’t have to configure an overlay-gateway with VLAN-
to-VNI mapping, EVPN instance, or VRF configurations for symmetric routing on border-leaf nodes for exchanging EVPN
routes with other DCS.
• Border-leaf nodes will not form VXLAN tunnels to other leaf nodes.
Full-mesh eBGP multihop configuration to Border-Leaf1 (site 1) to Border-Leaf3(site 2) and Border-Leaf4 (site2) (a similar
configuration is needed on other border-leaf nodes).
Verification can be done similar to Border-Leaf1 BGP and BGP EVPN verification.

34 53-1004313-04
Example 1—DCI L2 Extension

In the following example for the BGP-EVPN-based L2 and L3 extension deployment model:
• VLAN 203 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) with VNI 20003.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes.
• Configuration examples of servers, interfaces, BGP, overlay-gateway, and EVPN instance on leaf nodes are discussed in the
section that follows.
• Refer to sections "Configuration: Border Leaf to Spine Layer3", "Border Leaf to WAN Edge Layer3", and "Border Leaf eBGP
Multihop for Border-Leaf and DCI Configurations".
FIGURE 10 DCI L2 Extension
Server Configurations
Server 1 Bond interface configuration for CentOS server attached to Leaf 5 and Leaf 6 of Data Center Site1.
Server 2 interface configuration for CentOS VM attached to Leaf 8 of Data Center Site2.

53-1004313-04 35
Leaf Node Configurations on DC1
Port-channel interface configuration to Server 1 on Leaf 5 and 6.
VLAN interface configuration on Leaf 5 and 6.
VE interface configuration to Server 1 on Leaf 5 and 6.

36 53-1004313-04
Loopback interface configuration (VTEP address) on Leaf 5 and 6.
BGP Configurations on Leaf 5.

53-1004313-04 37
BGP Configuration on Leaf 6.

38 53-1004313-04
Overlay gateway configuration on Leaf 5 and Leaf 6.
NOTE
VLAN-to-VNI mapping can be done manually or automatically. If automatic mapping is enabled, the VNI-to-VLAN mapping is
1:1, i.e. VLAN 201 maps to VNI 201.
EVPN instance configuration on Leaf 5 and Leaf 6 (per rbridge).

53-1004313-04 39
Port-channel verification on Leaf 5 and Leaf 6.
BGP and EVPN verification on Leaf 5 can be done similar to Border-Leaf1 to spine (Leaf 6 can be verified using the same command).
vLAG-pair verification on Leaf 5 (Leaf 6 can be verified using the same command).
Anycast gateway verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
Leaf Node Configurations on DCS2
Interface configuration to Server 2 on Leaf 8.

40 53-1004313-04
VLAN interface configuration on Leaf 8.
VE interface configuration to Server 1 on Leaf 8.
Loopback interface configuration (VTEP address) on Leaf 8.

53-1004313-04 41
Overlay gateway configuration on Leaf 8 (under config mode).
EVPN instance configuration on Leaf 8 (under rbridge mode)
Inclusive multicast route verification on Leaf 5 for VNI associated with VLAN 203 (the same command can be used to verify on other
nodes).

42 53-1004313-04
Tunnel status verification on Leaf 5 (the same command can be used to verify on other nodes).
Individual tunnel verification on Leaf 5 (the same command can be used to verify on other nodes).

53-1004313-04 43
VLAN verification on Leaf 5 and Leaf 6 for 203 (the same command can be used to verify on other nodes).
Server 1 attached to Leaf 5 and 6 issuing ARP.

44 53-1004313-04
ARP verification on Leaf 5 (locally learnt ARP entries can be verified using this command).
ARP suppression verification on Leaf 8 (remote ARP learnt via BGP EVPN can be verified using show ip arp suppression-cache).
Server 2 attached to Leaf 8 issuing ARP.
Local and remote MAC verification on Leaf 5 for VLAN 203 ( the same command can be used to verify on other leaf nodes).

53-1004313-04 45
Example 2—DCI L2 Extension and L2 Multitenancy
Server 1 to Server 2 traceroute traffic
DC1 Border-Leaf1 to DC2 Leaf 8 traceroute traffic

Multitenancy Across DCS Using Different TORs
• VLAN 203 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) using VNI 20003.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (using VNI 20003).
• Traffic between Leaf 3 and 4 and Leaf 7 is verified using traceroute from servers attached to the leaf nodes (using VNI 30003).
• Configuration examples of servers, interfaces, BGP, overlay-gateway, and EVPN instance on leaf nodes (Leaf 3 and 4, and Leaf
7) are discussed in the following section.
• Refer to Example 1 for the configurations and verifications of Servers, VLAN 203, corresponding port-channel in that VLAN,
Overlay-Gateway, EVPN instance, BGP and tunnel for Leaf 5 and 6 and Leaf 8.
• Refer to sections "Configuration: Border Leaf to Spine Layer 3", "Border Leaf to WAN edge Layer 3", and "Border Leaf eBGP

46 53-1004313-04
FIGURE 11 DCI L2 Extension and L2 Multitenancy
Server 5 interface configuration for Windows VM attached to Leaf 3 and Leaf 4 of Data Center Site1.

53-1004313-04 47
Server 6 interface configuration for Windows VM attached to Leaf 7 of Data Center Site2.

48 53-1004313-04
Loopback interface configuration (VTEP address) on Leaf 3 and 4.
Overlay gateway configuration on Leaf 3 and 4 (under config mode).

53-1004313-04 49
EVPN instance configuration on Leaf 3 and 4 (under rbridge mode).

50 53-1004313-04
EVPN instance configuration on Leaf 7 (under rbridge mode).
Inclusive multicast route verification on Leaf 3 for VNI associated with VLAN 203 (the same command can be used to verify on Leaf 4
and Leaf 7).

53-1004313-04 51
and Leaf 8).

52 53-1004313-04
VLAN verification on Leaf 3 for 203 (the same command can be used to verify on other nodes).
Tunnel status verification on Leaf 3 (the same command can be used to verify on other nodes).
ARP verification on Leaf 3 (Locally learnt ARP entries can be verified using this command).

53-1004313-04 53
Server 5 to Server 6 traceroute traffic (connected to Leaf 3 and 4 and Leaf 7 with extended VNI 30003).
Server 1 to Server 2 traceroute traffic (connected to Leaf 5 and 6 and Leaf 8 with extended VNI 20003).
Multitenancy Across DCS Using the Same TOR

• VLAN 203 is extended between Data Center Site1 (Leaf 5 & 6) and Data Center Site2 (Leaf 8) using Virtual Fabric VLANs
7000 & 7001 and VNIs 7000 & 7001 respectively.
• VE interface 7000 & 7001 is configured under a VRF vrf3 & vrf4 on Leaf 5 & 6 (DCS1) and Leaf 8 (DCS2) respectively.
• VE interfaces 7000 and 7001 are configured with overlapping subnets (172.17.18.0)
• Traffic between Leaf 5 & 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (using VNI 7000 and
VNI 7001).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on leaf nodes (Leaf 5 & 6, and Leaf 8) are
discussed in the below section.
• Refer Example 1 for the configurations and verifications of BGP and tunnel for Leaf 5 & 6 and Leaf 8.

54 53-1004313-04
• Refer to sections "Configuration: Border Leaf to Spine Layer 3", "Border Leaf to WAN edge Layer 3", and "Border Leaf eBGP
Server 1 Bond interface configuration for CentOS server attached to Leaf 5 and Leaf 6 of Data Center Site1
Server 2 interface configuration for CentOS VM attached to Leaf 8 of Data Center Site2

53-1004313-04 55
Server 4 interface configuration for Windows VM attached to Leaf 8 of Data Center Site2

56 53-1004313-04
Interface configuration to Server 1 on Leaf 5 & 6
Interface configuration to Server 3 on Leaf 5 & 6
VLAN interface configuration on Leaf 5 & 6

53-1004313-04 57
VE interface configuration to Server 1 on Leaf 5 & 6
VE interface configuration to Server 3 on Leaf 5 & 6
Overlay Gateway configuration on Leaf 5 & 6 (under config mode)

58 53-1004313-04
EVPN instance configuration on Leaf 5 & 6 (under rbridge mode)
Interface configuration to Server 2 on Leaf 8
Interface configuration to Server 4 on Leaf 8
VLAN interface configuration on Leaf 8

53-1004313-04 59
VE interface configuration to Server 2 on Leaf 8
VE interface configuration to Server 6 on Leaf 8
Overlay Gateway configuration on Leaf 8 (under config mode)
EVPN instance configuration on Leaf 8 (under rbridge mode)

60 53-1004313-04
Inclusive-multicast route verification on Leaf 5 for VNI associated with VLAN 7000 (same command can be used to verify on Leaf 6
and Leaf 8)

53-1004313-04 61
VLAN verification on Leaf 5 for 7000 (same command can be used to verify on other nodes and for VLAN 7001)
ARP verification on Leaf 5 in VRF vrf4 (Locally learnt ARP entries can be verified using this command)
ARP suppression verification on Leaf 5 for VLAN 7001 (Remote ARP learnt via BGP EVPN can be verified using show ip arp
suppression-cache)

62 53-1004313-04
ARP verification on Leaf 5 in VRF vrf3 (Locally learnt ARP entries can be verified using this command)
ARP suppression verification on Leaf 5 for VLAN 7000 (Remote ARP learnt via BGP EVPN can be verified using show ip arp
suppression-cache)
MAC verification on Leaf 5 for VLAN 7001
Server 1 to Server 2 traceroute traffic (Connected to Leaf 5 & 6 and Leaf 8 with extended VNI 7001)
Server 3 to Server 4 traceroute traffic (Connected to Leaf 5 & 6 and Leaf 8 with extended VNI 7000)

53-1004313-04 63
Example 3—DCI VLAN Routing

Asymmetric Routing
• VLAN 203 and 204 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) with VNIs 20003
and 20004 respectively.
• Traffic between Leaf 5 and 6 (Site1) and Leaf 8 (Site2) is verified using traceroute from servers attached to the leaf nodes
(between VLAN 203 and 204).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on leaf nodes are discussed in the below
section.
• Refer to Example 1 for the configurations and verifications of VLAN 203, corresponding port-channel in that VLAN, Overlay-
gateway, EVPN instance, BGP and tunnel.
• Refer to sections Configuration: "Border Leaf to Spine Layer3", "Border Leaf to WAN Edge Layer3", and Border Leaf eBGP
Multihop for Border-leaf and DCI Configurations".
FIGURE 12 DCI VLAN Routing—Asymmetric Routing

64 53-1004313-04
Server 1 bond interface configuration for CentOS server attached to Leaf 5 and Leaf 6 of Data Center Site1.

53-1004313-04 65
Overlay gateway configuration on Leaf 5 and Leaf 6 under config mode).
EVPN instance configuration on Leaf 5 and Leaf 6 (under rbridge mode).
Port-channel verification on Leaf 5 and Leaf 6.

66 53-1004313-04

53-1004313-04 67
and Leaf 8).
VLAN verification on Leaf 5 and Leaf 6 for 204 (the same command can be used to verify on Leaf 8).
Server 1 Attached to Leaf 5 and 6 issuing ARP.

68 53-1004313-04
Local and remote MAC verification on Leaf 5 for VLAN 204 (the same command can be used to verify on other leaf nodes).

53-1004313-04 69
Server 1 to Server 3 traceroute traffic.
Conversational ARP verification on Leaf 5 by sending continuous traffic between Server 1 and Server 3.
Symmetric Routing
• VLAN 203 is configured on Data center Site1 (Leaf 5 and 6) and VLAN 204 is configured on Data Center Site2 (Leaf 8).
• VRF vpn1 is configured on Leaf 5, Leaf 6 and Leaf8 with respective import, export route-targets and with common L3 VNI
2005.
• This VNI 2005 is not needed to add under EVPN instance. But VLAN-to-VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces of 203, 204, and VNI VLAN VE will be configured under VRF vpn1.
• VRF address-family must be enabled under BGP configuration to advertise EVPN type 5 routes.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (between VLAN
203 and 204).
• Configuration examples of servers, interfaces, VRF, overlay-gateway, and EVPN instance on leaf nodes are discussed in the
following section.
• Refer to Example 1 for tunnel, port-channel, and VLAN verifications.
• Refer to sections Configuration: "Border Leaf to Spine Layer3", "Border Leaf to WAN Edge Layer 3", and "Border Leaf eBGP

70 53-1004313-04
FIGURE 13 DCI VLAN Routing—Symmetric Routing

53-1004313-04 71
VRF configuration on Leaf 5 and 6.
VRF VNI VLAN and VE interface configuration on Leaf 5 and 6.

72 53-1004313-04
VE interface configuration to server on Leaf 5 and 6.
Overlay gateway configuration on Leaf 5 and Leaf 6 (under config mode).
BGP configuration on Leaf 5 (similar configuration is needed on Leaf 6) .

53-1004313-04 73
VRF configuration on Leaf 8.
VRF VNI VLAN and VE interface configuration on Leaf 8.

74 53-1004313-04
VE interface configuration toward server on Leaf 8.
BGP configuration on Leaf 8.

53-1004313-04 75
VRF VNI verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
L3 prefixes (type 5 routes) verification on Leaf 5 (the same command can be used to verify on other leaf nodes).

76 53-1004313-04
VRF route verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
Server 1 attached to Leaf 5 and 6 issuing ARP.

53-1004313-04 77
Example 4—Adding Services to Border-Leaf Nodes

In some situations, border-leaf nodes can hold some services and need to extend some VLANs with other leaf nodes or to perform
routing in VRF. To achieve this, overlay-gateway and EVPN instance or VRF with common L3 VNI configuration will be added on
border-leaf nodes for some VLANs. Retain route-target all will also be configured under EVPN address-family on border-leaf nodes to
pass the EVPN routes between DC sites for the VNIs that are not configured on border-leaf nodes.
• VLAN 203 and 204 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) with VNIs 20003
and 20004 respectively.
• In addition to retain route-target all under BGP on border-leaf nodes in DCS1 and DCS2, VTEP address, overlay-gateway and
EPVN-instance with VNI mapping 20003 and 20004 will be configured. Border-leaf nodes will form tunnels with leaf nodes
that extended the VNIs 20003 and 20004. Also, they will forward the EVPN routes for VNI 30003.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (for VNI 20003,
20004).
• Traffic between Leaf 3 and 4 and Leaf 7 is verified using traceroute from servers attached to the leaf nodes (for VNI 30003).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on border-leaf nodes are discussed in the
following section.
• Refer to Use Case 1, Use Case 2, and Use Case 3 for the configurations and verifications of VLAN 203, VLAN 204,
corresponding port-channel in that VLAN, Overlay-Gateway, EVPN instance, BGP and tunnel.
• Refer to sections "Configuration—Border Leaf to Spine Layer 3", "Border Leaf to WAN Edge Layer 3", and "Border Leaf eBGP

78 53-1004313-04
FIGURE 14 Adding Services to Border Leaf Nodes

53-1004313-04 79
Server 5 interface configuration for Windows VM attached to Leaf 3 and Leaf 4 of Data Center Site1.
Server 6 interface configuration for Windows VM attached to Leaf 7 of Data Center Site2.
Border-leaf Node Configurations on DCS1
VLAN interface configuration on Border-Leaf1 and Border-Leaf2 (similar configuration is needed on Border-Leaf3 and Border-Leaf4).

80 53-1004313-04
VE interface configuration on Border-Leaf1 and Border-Leaf2 (similar configuration is needed on Border-Leaf3 and Border-Leaf4).
Loopback interface configuration (VTEP address) on Border-Leaf1 and Border-Leaf2.
Unique loopback interface on Border-Leaf1 and Border-Leaf2 to establish eBGP multihop session with Border-Leaf3 and Border-
Leaf4.

53-1004313-04 81
Loopback interface configuration (VTEP address) on Border-Leaf3 and Border-Leaf4.
Unique loopback interface on Border-Leaf3 and Border-Leaf4 to establish eBGP multihop session with Border-Leaf1 and Border-
Leaf2.
BGP configuration on Border-Leaf1.

82 53-1004313-04
Overlay gateway configuration on Border-Leaf1 and Border-Leaf2 (under config mode, similar configuration is needed on Border-Leaf3
and Border-Leaf4).
EVPN instance configuration on Leaf 3 and 4 (under rbridge mode, similar configuration is needed on Border-Leaf3 and Border-Leaf4).
Inclusive multicast route verification on Border-Leaf1 for VNI 20003 (the same command can be used to verify on other border-leaf
nodes).

53-1004313-04 83
Inclusive multicast route verification on Border-Leaf1 for VNI 30003 (the same command can be used to verify on other border-leaf
nodes).

84 53-1004313-04
Tunnel status verification on Border-Leaf1 (the same command can be used to verify on other border-leaf nodes).

53-1004313-04 85
ARP verification on Leaf 5 after issuing ARP ping from all servers (locally learnt ARP entries can be verified using this command).

86 53-1004313-04
Example 5—Extending a Tenant VRF to the WAN Edge
Example 5—Extending a Tenant VRF to the WAN

Edge
• VLAN 204 is configured in Data center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8).
• VRF vpn1 is configured on Leaf 5, Leaf 6, Border-Leaf1, Border-Leaf2, and Leaf8 with respective import, export route-targets
and with common L3 VNI 2005.
• This VNI 2005 is not needed to add under EVPN instance on all nodes. But VLAN-to-VNI mapping is needed under overlay-
gateway configuration.
• VE interfaces of 204 and VNI VLAN VE (2005) will be configured under VRF.
• VRF address-family must be enabled under BGP configuration to advertise Type 5 routes.
• Since VE 204 on both DCS1 and DCS2 is in same subnet, VNI mapping for VLAN 204 can be added under overlay-gateway
and respective VNI can be added under EVPN instance on Site1 Leaf 5 and 6 and Site2 Leaf 8. This makes the inclusive-
multicast route exchange between leaf nodes (Leaf 5 and 6 and Leaf 8).
• To extend the VRF to WAN edge, one of the connections between Border-Leaf1 and WAN Edge1 is configured in two different
VLANs (800, 802) with VE 800 in default VRF (to support DCI interconnect and to have multihop eBGP adjacency to border-
leaf nodes on DCS2) and VE 802 in VRF vpn1. Similar changes made on WAN edge 1.
• A link from Border-leaf2 to WAN edge1 is enabled in VLAN 801 with VE 801 in VRF vpn1 on both sides.
• eBGP adjacency between border-leaf nodes (BL1 and BL2) and WAN edge1 is established under VRF vpn1 address-family
with redistribute connected.

53-1004313-04 87
• This makes route exchange between Leaf 5 and 6, Leaf 8 in VLAN 204, and WAN-edge1 in VLAN 801 and 802. (Type 5
route-exchange)
FIGURE 15 Extending Tenant VRF to WAN Edge

88 53-1004313-04
VRF configuration on Leaf 5 and 6.

53-1004313-04 89
VRF VNI VLAN and VE interface configuration on Leaf 5 and 6.
VE interface configuration to server 4 on Leaf 5 and 6.

90 53-1004313-04
Overlay gateway configuration on Leaf 5 and Leaf 6 (under config mode).
EVPN instance configuration on Leaf 5 and Leaf 6 (under rbridge mode).
BGP has to be configured similar to Example 5 on Leaf 5 and 6.

53-1004313-04 91
VRF configuration on Leaf 8.
VRF VNI VLAN and VE interface configuration on Leaf 8.
VE interface configuration toward server on Leaf 8.

92 53-1004313-04
Leaf Node Configurations on Border-Leaf1 and Border-Leaf2.
Interface configuration on Border-Leaf1 to WAN edge1.
Interface configuration on Border-Leaf2 to WAN edge1.
VLAN interface configuration on Border-Leaf1 and Border-Leaf2.
VRF configuration on Border-Leaf1 and Border-Leaf2.

53-1004313-04 93
VRF VNI VLAN and VE interface configuration on Border-Leaf1 and Border-Leaf2.
VE interface configuration to WAN Edge1 on Border-Leaf1 (Similar configuration is needed on WAN edg1).

94 53-1004313-04
VE interface configuration to WAN Edge1 on Border-Leaf2 (Similar configuration is needed on WAN edg1).
Overlay gateway configuration on Border-Leaf1 and Border-Leaf2 (under config mode).
EVPN instance configuration on Border-Leaf1 and Border-Leaf2 (under rbridge mode).

53-1004313-04 95
BGP configuration on WAN Edge 1 (similar configuration is needed on other WAN edges).

96 53-1004313-04
Inclusive multicast route verification on Border-Leaf1 for VNI 20004 (since VNI for VLAN 204 defined in VRF vpn1 is extended, both
Leaf 5 and 6, and Leaf 8 advertises the IMR route).

53-1004313-04 97
Local and remote L3 prefixes (Type 5 routes) in Border-Leaf1,

98 53-1004313-04
VRF route verification on Border-Leaf1.
VRF route verification on Leaf 5 (the same command can be used to verify on Leaf 6 and Leaf 8).

53-1004313-04 99
Example 6—Providing Internet Route Reachability for Tenant VRFs at TORs Through Public VRF at Border-Leaf
VRF route verification on WAN Edge1.
Server 3 to WAN edge1 ping traffic.
Example 6—Providing Internet Route Reachability for

Tenant VRFs at TORs Through Public VRF at Border-
Leaf
• Virtual Fabric VLAN 7000 is configured with customer-tag 203 in Data center Site1 (Leaf5 & 6) and VLAN 530 is configured
Data Center Site2 (Leaf 8).
• VRF tenant-vrf is configured on Leaf 5 & 6 (DCS1) and on Leaf 8 (DCS2) with a common L3 VNI 5060 with respective import
and export route-targets.
• There is no need to add the L3 VNI 5060 under EVPN instance but VLAN VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces 7000 on Leaf 5&6 (DCS1) and 203 on Leaf 8 (DCS2) will be configured under VRF tenant-vrf.
• VE interface corresponding to L3 VNI 5060 must be enabled under tenant-vrf on Leaf 5 & 6 (DCS1) and on Leaf 8 (DCS2).

100 53-1004313-04
• VRF tenant-vrf address-family must be enabled under BGP configuration to advertise Type 5 routes.
• ISP is connected to BL1 & BL2 on DCS1 and BL3 & BL4 on DCS2.
• To extend the tenant-vrf from Leaf 5&6 and Leaf 8 to ISP one of the connections between Border-Leaf1 and WAN Edge1 is
configured in two different VLANs (800, 802) with VE 800 in default VRF (to support DCI interconnect and to have multi-hop
eBGP adjacency to border-leaf nodes on DCS2) and VE 802 in VRF public-vrf. Similar changes made on WAN edge 1.
• Similar configurations are needed between Border-Leaf2 and WAN Edge2 in VLANs 30 (default-vrf) and 31 (public-vrf),
Border-Leaf3 and WAN Edge3 in VLANs 850 (Default VRF) and 851 (public-vrf), and Border-Leaf4 and WAN Edge4 in
VLANs 40 (Default VRF) and 41 (public-vrf).
• EBGP adjacency between Border-Leaf nodes (BL1 & BL2, BL3 & BL4) with respective WAN Edges (WE1 & WE2, WE3 &
WE4) is established using VE interfaces mentioned above in respective VRFs.
• WAN edge is configured to advertise only default routes to respective Border-Leaf nodes in public-vrf.
• Route leak is configured between BL and individual leaf nodes in respective DCS with import and export route-targets under
VNIs.
• VLAN to VNI mapping for the VNI added under EVPN instance for route leak must be added under overlay-gateway on Leaf 5
& 6 and BL1 & BL2. Similar configuration is needed on Leaf 8 and BL3 and BL4.
• VE interface corresponding to the VNI for route-leak must be enabled on both Leaf 5 & 6 and on Border-Leaf1 and Border-
Leaf2. Similar configuration is needed on Leaf 8 and BL3 and BL4.
• This makes route exchange between tenant-vrfs of Leaf 5 & 6 with public-vrfs of BL1 & BL2, tenant-vrf of Leaf 8 with public-
vrfs of BL3 & BL4.
• Traffic to internet route is verified from a server attached to Leaf 5 & 6 with ping.
NOTE
In this example, private IPV4 addresses are used from TOR to ISP. This can be modified to public IPV4 addresses with NAT
placed either at WAN edge or at ISP.

53-1004313-04 101
Interface configuration to Server 3 on Leaf 5 & 6 (Similar configuration is needed on Leaf 8)

102 53-1004313-04
VLAN interface configuration on Leaf 5 & 6
VRF configuration on Leaf 5 & 6 (Similar configuration is needed on Leaf 8)
VE interface configuration to Server 3 on Leaf 5 & 6 (Similar configuration is needed on Leaf 8)

53-1004313-04 103
EVPN instance configuration on Leaf 5 and Leaf 6 (under rbridge mode) (Similar configuration is needed on Leaf 8. Leaf 8 will import
default route to reach internet routes from BL3 & BL4.)
VLAN interface configuration on Leaf 5 & 6 for L3 VNI VLANs (Similar configuration is needed on Leaf 8

104 53-1004313-04
VE interface configuration on Leaf 5 & 6 for L3 VNI VLANs (Similar configuration is needed on Leaf 8)
BGP configuration on Leaf 5 (Similar configuration is needed on Leaf 6 and Leaf 8)
Interface configuration on Border-Leaf1 to WAN edge1
Interface configuration on Border-Leaf2 to WAN edge2

53-1004313-04 105
VLAN interface configuration on Border-Leaf1 & Border-Leaf2
VRF configuration on Border-Leaf1 and Border-Leaf2 (similar configuration needed on BL3 & BL4)
VRF VNI VLAN and VE interface configuration on Border-Leaf1 and Border-Leaf2

106 53-1004313-04
VE interface configuration to WAN Edge1 on Border-Leaf1 (Similar configuration is needed on WAN edg1)
VE interface configuration to WAN Edge2 on Border-Leaf2 (Similar configuration is needed on WAN edg2)
Overlay Gateway configuration on Border-Leaf1 and Border-Leaf2 (under config mode)

53-1004313-04 107
EVPN instance configuration on Border-Leaf1 and Border-Leaf2 (under rbridge mode
BGP configuration on Border-Leaf1 (Similar configuration is needed on BL2, BL3 & BL4)
BGP configuration on WAN Edge1 (Similar configuration is needed on WE2, WE3 & WE4)

108 53-1004313-04
Local and remote L3 prefixes (Type5 routes) on Leaf 5
Local and remote L3 prefixes (Type5 routes) on Border-Leaf1

53-1004313-04 109
Route verification on Border-Leaf1

110 53-1004313-04
Route verification on Leaf 5
Server 3 to ISP connected to WE1 and WE2 ping traffic
Route verification on WAN edge1

53-1004313-04 111

112 53-1004313-04
Validated Design—EVPN DCI with BGP-
EVPN-Based L2 Extension
• Topology Description....................................................................................................................................................................................113
• Data Center Interconnect Tier....................................................................................................................................................................114
• Configuration Steps....................................................................................................................................................................................... 116
• Configuration: DCI Tier to WAN Edge................................................................................................................................................... 123
• Example 1—Layer 2 Extension................................................................................................................................................................ 124
• Example 2—VLAN Routing........................................................................................................................................................................154
• Example 3—Providing Internet Route Reachability for Tenant VRFs at DCI Tier Through Public VRF ................... 176
• Example 4a—DCI L2 Extension.............................................................................................................................................................. 187
• Example 4b—VLAN Asymmetric Routing...........................................................................................................................................210
This section provides a detailed configuration for the BGP-EVPN-based L2 extension deployment model based on a test topology, and
it walks through common use cases with selected show commands to demonstrate intended functions. This section is further divided
into:
• Topology description
• DCI tier and configurations
• Examples
FIGURE 16 Three Data Center Sites Interconnected Using BGP EVPN

53-1004313-04 113
Data Center Interconnect Tier

As described in earlier sections, Data Center Interconnect tier devices are directly attached to WAN edge devices at Layer 3 and the data
center at Layer 2. When using a pair of devices at the DCI tier, an active-active vLAG configuration is strongly recommended for
redundancy and fault tolerance. With this deployment model, inter-site reachability is limited to the set of VLANs configured on the
interface to the DCI tier.
The following matrix shows the data center tier devices and their types used for validation of this deployment model. This matrix is an
extension of matrices shown in the "BGP-EVPN-Based L2 and L3 Extension" section. In the BGP-EVPN-based L2 extension case,
traffic flow is from server to server as follows:
server - leaf - spine - border leaf - DCI tier - WAN edge - DCI tier - border leaf - leaf - server
Whereas in L3 handoff, the border leaf hands off the traffic to the WAN edge directly.
While interconnecting the IP fabric data center and the flexible (for example, VCS) data center type, the DCI tier layer can be eliminated at
the IP fabric type DC site, and EVPN control-plane extension can be configured between leaf nodes in IP fabric DC to DCI tier nodes in
the flexible DC site. In such a case, traffic flow is from:
server - leaf - spine - border-leaf - WAN edge cloud - DCI tier - VCS leaf - server
Role of Node Chassis Name (Possible Chassis Types) Version Required
Leaf BR-VDX6940-36Q Network OS 7.0 and later
BR-VDX6940-144S
BR-VDX6740T
BR-VDX6740
Border leaf BR-VDX6940-36Q Network OS 7.0 and later
BR-VDX6940-144S
Spine BR-VDX8770-4/8 Network OS 7.0 and later
BR-VDX6940-36Q
BR-VDX6940-144S
DCI tier BR-VDX6940-36Q Network OS 7.0 and later
BR-VDX6940-144S
BR-VDX6740
WAN edge MLXe-4/8/16/32 NetIron 5.9.00
Control-plane extension between sites is enabled by establishing eBGP (multihop) peering from local DCI tier nodes to remote DCI tier
nodes with the EVPN address family enabled. The VLAN-to-VNI mapping for the VLANs to be extended is configured on the DCI tier
nodes. This model allows for VLAN extension between different DC types, e.g. VCS to BGP-EVPN-based IP fabric.
In the case of extending a BGP-EVPN-based (IP fabric) data center, the leaf nodes can encapsulate the server traffic and send it over to
the border leaf, the border leaf switches the traffic to the DCI tier, and DCI tier nodes encapsulate the L2 traffic to remote DCI tier nodes.
The following figure shows the high-level packet path.

114 53-1004313-04
FIGURE 17 Packet Path from Data Center Site 1 to Site 2
A. The server in data center site 1 sends an Ethernet frame to the destination in site 2. The packet is forwarded to the leaf node, which is
configured as the default GW for the server.
B. The leaf node learns the source MAC address and shares it with the EVPN peers, i.e., border leaf (control plane) using IMR route. The
leaf node encapsulates the received Ethernet frame into an IP User Datagram Protocol packet and sends it over the VXLAN tunnel where
the VLAN is extended (data plane).
C. The border leaf removes the encapsulation and floods the frame to all interfaces that are configured with the same VLAN, in this
example, the port channel toward the DCI tier. The DCI tier learns the MAC from the source MAC address and shares it with remote
peers in other sites via a BGP-EVPN update.
NOTE
VTEP IP addresses are carried as the BGP next-hop attribute in every EVPN route. This allows BGP to discover remote
VTEPs. The Inclusive Multicast Ethernet Tag Route allows the receiving BGP router to discover which VLANs are common
between the two routers and extend them over the VXLAN tunnel.
D. The encapsulated traffic packet is received at the remote data center site. The VXLAN header encapsulation is removed, and the
original frame is forwarded using L2 forwarding.
E. The destination receives the packet on the active-active vLAG from the DCI tier.
The upcoming sections walk through the configuration and verification steps for this deployment model.

53-1004313-04 115
Configuration Steps
Configuration Steps
The traffic handoff to the border leaf can be done in two ways:
1. Over L2—Having all leaf nodes part of active-active vLAG pair.
2. Over L3—Using VXLAN tunnels from leaf to border leaf.
FIGURE 18 BGP-EVPN-Based (IP Fabric) Data Center with Leaf, Spine, and Border Leaf
High-level overview of setting up the DCI for Layer 2 handoff case:
1. Configure the port channel between the border leaf and DCI tier nodes.
2. Configure BGP and activate EVPN adjacency between DCI tier nodes.
3. Configure an overlay gateway instance and activate it on all DCI tier nodes.
4. Configure an EVPN instance on all DCI tier nodes.
Creating VLANs to be extended (on leaf, border leaf, and DCI tier):

116 53-1004313-04
Configuration Steps
Port-channel configuration on the border leaf toward the DCI tier:
Interface configuration for interfaces participating in the port channel (on border leaf and DCI tier):
NOTE
The above interface configurations are to be applied on all interfaces participating in the port
channel.
Port-channel configuration on the DCI tier node:
Loopback interface configuration on the DCI tier:

53-1004313-04 117
Configuration Steps
NOTE
The router ID is used for the BGP neighborship formation. If the router ID is not explicitly defined, the first loopback address is
automatically chosen as the router ID.
EVPN instance configuration on the DCI tier:
BGP configuration on the DCI tier:

118 53-1004313-04
Configuration Steps
BGP EVPN address family:
Port-channel verification to see all that inks are UP:

53-1004313-04 119
Configuration Steps
Show BGP neighborship:
Show BGP EVPN neighborship:

120 53-1004313-04
Configuration Steps
Show tunnel brief:
Show tunnel status to particular site, site 2 for example:

53-1004313-04 121
Configuration Steps
Verify tunnel association for the VLAN that is extended to remote sites:
Traceroute verification from one DCI tier to another:
Route verification:
Inclusive multicast route verification on DCI tier 11 for the VNI associated with VLAN 203:

122 53-1004313-04
Configuration: DCI Tier to WAN Edge
Configuration: DCI Tier to WAN Edge

• It is recommended to use two node vLAG pair on DCI Tier to avoid single point of failure
• It is recommended to have full mesh eBGP adjacency between each DCI Tier to WAN edge nodes.
• It is recommended to use 4 – 10G port LAG between DCI Tier node and WAN edge.
• Network command is used to advertise the loopback interface to WAN Edge nodes.
• EBGP adjacency between DCI Tier and WAN edge is not activated under EVPN address-family.
• Tracking the links between WAN edge and DCI Tier can help isolate a potential traffic black hole when all the links from one BL
to DCI Tier node fail. This can be achieved by configuring link-tracking feature on the DCI Tier nodes.

53-1004313-04 123
Example 1—Layer 2 Extension

Example 1a: L2 Extension Between DCI Tier – DCI Tier
FIGURE 19 Two Flexible Type Data Center Sites Interconnected Using EVPN at DCI Tiers
In the following example for the BGP-EVPN-based L2 handoff model:

• VLAN 1998 is extended between Data Center Site1 and Data Center Site 2 with VNI set to auto. (In auto VNI configuration, the
VNI value is the same as VLAN ID.)
• Both VCS Leaf nodes and DCI Tier nodes are configured with the VLANs.
• Anycast gateway is configured on DCI tier devices.
• Servers are connected to the VCS Leaf nodes, and traffic is verified with the traceroute command between them.
• Configuration example of servers, interfaces, BGP, overlay-gateway, and EVPN instance on DCI tiers are discussed in the
Server 1 bond interface configuration - Data Center Site 1

124 53-1004313-04
Server 2 interface configuration - Data Center Site 2
Port-channel interface configuration on Leaf M & N to Server 1
Port-channel interface configuration to DCI Tier 11 and 12

53-1004313-04 125
Port-channel interface configuration to DCI Tier 11 and 12

126 53-1004313-04
VLAN interface configuration on Leaf O & P in Data Center Site 1, Leaf M & N in Data Center Site 2
VLAN interface configuration on DCI Tier 11 and 12 in Data Center Site 1:
ARP and ND suppression are to be configured when Anycast Gateway configuration is present.
VE interface configuration to Server 1 on DCI Tier 11 and 12

53-1004313-04 127
Loopback interface configuration (VTEP address) on DCI Tier 11 and 12
BGP configurations on DCI Tier 11

128 53-1004313-04
BGP configuration on DCI Tier 12

53-1004313-04 129
Overlay Gateway configuration on DCI Tier 11 and 12
EVPN instance configuration on DCI Tier 11 and 12 (per rbridge)

130 53-1004313-04
Port-channel Verification on Leaf O and P Toward Server
Port-channel Verification on Leaf O and P Toward DCI Tier 11 and 12
EVPN verification on DCI Tier 11
vLAG-pair verification on DCI Tier

53-1004313-04 131
Anycast gateway verification on DCI Tier 11 (the same command can be used to verify on other DCI tier nodes)
Leaf node configurations on Data Center Site2
Port-channel configuration to Server 2 on Leaf M & N
VLAN interface configuration on Leaf M and N
VLAN interface configuration on DCI Tier 21 and 22

132 53-1004313-04
VE interface configuration on DCI Tier 21 and 22
BGP configuration on DCI tier 21

53-1004313-04 133

134 53-1004313-04
Overlay Gateway configuration on DCI Tier 21 and 22 (under global config)
EVPN instance configuration on DCI tier 21 and 22 (under rbridge mode)

53-1004313-04 135
Inclusive-multicast route verification on DCI Tier 11 for VNI associated with VLAN 1998 (the same command can be used to verify on
other nodes)
Tunnel status verification on DCI Tier 11 (the same command can be used to verify on other nodes)

136 53-1004313-04
Individual tunnel verification on DCI Tier 11 (the same command can be used to verify on other nodes)
VLAN verification on DCI Tier 11 for 1998 (the same command can be used to verify on other nodes)
Server 1 attached to Leaf O and P issuing ARP

53-1004313-04 137
ARP verification on DCI Tier 11 (the locally learned ARP entries can be verified using this command)
ARP suppression verification on DCI Tier 11 (the remote ARP learned via BGP EVPN can be verified using show ip arp suppression-
cache)
Server 2 attached to Leaf 8 issuing ARP
ARP suppression verification on DCI Tier 21 (the remote ARP learned via BGP EVPN can be verified using show ip arp suppression-
cache)

138 53-1004313-04
Local and remote MAC verification on DCI Tier 21 for VLAN 203 (the same command can be used to verify on other leaf nodes)
DC1 Border-Leaf1 to DC2 Leaf 8 traceroute traffic

53-1004313-04 139
Example 1b: L2 Extension Between DCI Tier - Leaf Node

FIGURE 20 IP Fabric Data Center Site Interconnected with Flexible Type DC Using EVPN at DCI Tiers
Server 1 interface configuration for Windows VM attached to Leaf-I of Data Center Site1
Server 2 Bond interface configuration for CentOS server attached to Leaf O and Leaf P of Data Center Site2

140 53-1004313-04
Port-channel interface configuration to Server 1 on Leaf I
VLAN interface configuration on Leaf I
VE interface configuration to Server 1 on Leaf I

53-1004313-04 141
Loopback interface configuration (VTEP address) on Leaf 5 & 6
BGP configurations on Leaf I

142 53-1004313-04
Overlay Gateway configuration on Leaf I
EVPN instance configuration on Leaf I
Port-channel Verification on Leaf I

53-1004313-04 143
BGP and EVPN verification on Leaf I can be done similar to Border-Leaf1 to spine.
Anycast gateway verification on Leaf I
Port-channel interface configuration to Server 1 on Leaf O & Leaf P

144 53-1004313-04
VLAN interface configuration on Leaf O & Leaf P
VE interface configuration to Server 1 on DCE11 & DCE12
Loopback interface configuration (VTEP address) on DCE11 & DCE1

53-1004313-04 145
BGP configuration on DCE11

146 53-1004313-04

53-1004313-04 147
Overlay Gateway configuration on DCE 11 and 12 (under config mode)
EVPN instance configuration on DCE 11 and 12 (under rbridge mode)

148 53-1004313-04
Inclusive-multicast route verification on Leaf I for VNI associated with VLAN 1999

53-1004313-04 149
Tunnel status verification on Leaf I
Individual tunnel verification on Leaf I

150 53-1004313-04
VLAN verification on Leaf I for VLAN 1999
Server 1 ARP table on server attached to Leaf I
ARP verification on Leaf I

53-1004313-04 151
ARP suppression verification on Leaf I (the remote ARP learned via BGP EVPN can be verified using show ip arp suppression-cache)
Server 2 attached to Leaf O & P issuing ARP
ARP verification on DCE 11 and 12 (locally learned ARP entries can be verified using this command)
ARP suppression verification on Leaf I (the remote ARP learned via BGP EVPN can be verified using show ip arp suppression-cache)

152 53-1004313-04
Local and Remote MAC verification on Leaf I for VLAN 1999
DC1 DCE 11 to DC2 Leaf I traceroute traffic

53-1004313-04 153
Example 2—VLAN Routing

Example 2a: Symmetric VLAN Routing Between Two Flexible Type Data
Centers
• VLAN 2003 is configured on Data center Site1 (DCI Tier 11-12) and VLAN 2004 is configured on Data Center Site2 (DCI
Tier 21-22).
• VRF "vpn1" is configured on DCI tiers 11-12 and 21-22 with respective import, export route-targets and with common L3
VNI 4041.
• This VNI 4041 is not needed to be added under EVPN instance, but VLAN to VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces of 2003, 2004, and VNI VLAN VE (4041) will be configured under VRF vpn1.
• VRF address-family should be enabled under BGP configuration to advertise Type 5 routes.
• Traffic between DCI Tier 11-12 and DCI Tier 21-22 is verified using traceroute from servers attached to the leaf nodes (Leaf
M-N and, O-P) between VLAN 2003 and 2004.
below section.
FIGURE 21 Symmetric VLAN Routing Between Two Flexible Type Data Center Sites

154 53-1004313-04
Server 1 bond interface configuration for server attached to Leaf O-P vLAG pair of Data Center
Site1
Server 2 interface configuration for CentOS VM attached to Leaf M-N vLAG pair of Data Center
Site2
Port-channel interface configuration to Server 1 on Leaf O-P

53-1004313-04 155
Port-channel interface configuration toward DCI Tier 11-12 on Leaf O-P
VLAN interface configuration on Leaf O-P

156 53-1004313-04
VLAN configuration on DCI Tier 11-12
VRF configuration on DCI Tier 11-12
VRF VNI VLAN and VE interface configuration on DCI Tier 11-12

53-1004313-04 157
VE 2003 Interface configuration on DCI Tier 11-12
Overlay Gateway configuration on DCI Tier 11-12 (under config mode)
BGP configuration on DCI Tier 11 (similar configuration is needed on DCI Tier 12)

158 53-1004313-04
Port-channel configuration to Server 2 on Leaf M-N
VLAN interface configuration on Leaf M-N

53-1004313-04 159
Interface configuration on DCI Tier21-22
VE interface configuration toward server on DCI Tier 21-22

160 53-1004313-04

53-1004313-04 161
VRF VNI verification on Leaf 5 (the same command can be used to verify on other leaf nodes)

162 53-1004313-04
L3 prefixes (Type 5 routes) verification on DCI Tier 21 (the same command can be used to verify on other tier nodes)
VRF route verification on DCI Tier 21 (the same command can be used to verify on other DCI tier nodes)

53-1004313-04 163
Server 1 attached to Leaf M-N issuing ARP
Server 1 to Server2 traceroute traffic

164 53-1004313-04
Example 2b: Symmetric VLAN Routing between flexible type and IP

Fabric Data Centers
• VLAN 4004 is configured on flexible type Data center Site1 (DCI Tier 11-12) and VLAN 4003 is configured on IP Fabric Data
Center Site2 (Leaf I).
• VRF "SYM_RNG" is configured on DCI tiers 11-12 and Leaf I with respective import, export route-targets and with common
L3 VNI 4040.
• This VNI 4040 is not needed to be added under EVPN instance, but VLAN to VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces of 4003, 4004, and VNI VLAN VE (4040) will be configured under VRF SYM_RNG.
• Traffic between DCI Tier 11-12 and Leaf I is verified using traceroute from servers attached to the leaf nodes Leaf M-N in Data
Center Site 1 and Leaf I in Data center Site 2 between VLAN 4004 and 4003.
below section.

53-1004313-04 165
FIGURE 22 Symmetric VLAN Routing Between Flexible Type and IP Fabric Data Center Sites
Server 1 bond interface configuration for server attached to Leaf O-P vLAG pair of Data Center
Site1
Server 2 interface configuration for CentOS VM attached to Leaf M-N vLAG pair of Data Center
Site2

166 53-1004313-04
Port-channel interface configuration to Server 1 on Leaf O-P
Port-channel interface configuration toward DCI Tier 11-12 on Leaf O-P

53-1004313-04 167
VLAN interface configuration on Leaf O-P
VLAN configuration on DCI Tier 11-12

168 53-1004313-04
VE 4004 interface configuration on DCI Tier 11-12

53-1004313-04 169
BGP configuration on DCI Tier 11 (a similar configuration is needed on DCI Tier 12)

170 53-1004313-04
Port-channel configuration to Server 2 on Leaf I
VLAN interface configuration on Leaf I

53-1004313-04 171
VRF configuration on Leaf I
VRF VNI VLAN and VE interface configuration on Leaf I
VE interface configuration toward server on Leaf I
Overlay Gateway configuration on Leaf I (under config mode)

172 53-1004313-04
BGP configuration on Leaf I
VRF VNI verification on Leaf I

53-1004313-04 173
L3 prefixes (Type 5 routes) verification on Leaf I

174 53-1004313-04
VRF route verification on Leaf I
Server 2 attached to Leaf I issuing ARP

53-1004313-04 175
Example 3—Providing Internet Route Reachability for Tenant VRFs at DCI Tier Through Public VRF
Server 1 to Server2 traceroute traffic
Example 3—Providing Internet Route Reachability for

Tenant VRFs at DCI Tier Through Public VRF
• VLAN 3009 is configured in flexible type Data center - Site1 (DCI Tier 11-12) and VLAN 530 is configured Data Center Site2
(Leaf 8).
• Tenant VRF vpn10 is configured on DCI Tier 11-12(DCS1) and on Leaf 8 (DCS2), tenant vrf 'tenant-vrf' with a common L3
VNI 5060 with respective import and export route-targets.
• There is no need to add the L3 VNI 5060 under EVPN instance but VLAN VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces 3009 on DCI Tier 11-12 (DCS1) and 3009 on Leaf 8 (DCS2) will be configured under VRF tenant-vrf.
• VE interface corresponding to L3 VNI 5060 must be enabled under tenant-vrf on DCI Tier 11-12 (DCS1) and on Leaf 8
(DCS2).
• Tenant VRF address-family must be enabled under BGP configuration to advertise Type 5 routes.
• ISP is connected to DCI Tier 11-12 on DCS1 and BL3 and BL4 on DCS2.
• To extend the tenant vrf 'vpn10' from DCI tier 11-12 to ISP, one of the connections between DCI Tier 11 and WAN Edge1 is
configured in two different VLANs 4001 and 4002 with VE 4001 in default-vrf and VE 4002 in public vrf.
• Similar configurations are needed between DCI Tier 12 and WAN.
• EBGP adjacency between DCI Tier and Border-Leaf nodes (BL3 and BL4) with respective WAN Edges (WE1 and WE2, WE3
and WE4) is established using VE interfaces mentioned above in respective VRFs.
• WAN edge is configured to advertise only default routes to respective Border-Leaf nodes in public-vrf.
• In flexible type Data Center site, DCS1, route leak is configured under public vrf by using static route pointing to each tenant vrf
as next-hop.
• In IP Fabric Data Center site, DCS2, route leak is configured between BL and individual leaf nodes in respective DCS with
import and export route-targets under VNIs.

176 53-1004313-04
• VLAN to VNI mapping for the VNI added under EVPN instance for route leak must be added under overlay-gateway on DCI
Tier 11-12. Similar configuration is needed on Leaf 8 and BL3 and BL4.
• VE interface corresponding to the VNI for route-leak must be enabled on DCI Tier 11-12. Similar configuration is needed on
Leaf 8 and BL3 and BL4.
• This makes route exchange between tenant-vrfs with public-vrfs of DCI Tier 11-12, tenant-vrf of Leaf 8 with public-vrfs of
BL3 and BL4.
• Traffic to internet route is verified from a server attached to Leaf O-P with ping.
NOTE
In this example, private IPV4 addresses are used from DCI Tier to ISP. This can be modified to public IPV4 addresses with
NAT placed either at WAN edge or at ISP.
FIGURE 23 Extending a Tenant VRF at Flexible Type and IP Fabric Data Center with Common L3VNI
Server 1 bond interface configuration for CentOS server attached to Leaf O and Leaf P of Data Center
Site1

53-1004313-04 177
Interface configuration to Server 1 on Leaf O and P (a similar configuration is needed on Leaf 8)
VLAN interface configuration on DCI Tier 11-12
VRF configuration on DCI Tier 11-12 (a similar configuration is needed on Leaf 8)

178 53-1004313-04
VE interface configuration to Server 1 on DCI Tier 11-12 (a similar configuration is needed on Leaf 8)
EVPN instance configuration on DCI Tier 11-12 (under rbridge mode) (a similar configuration is needed on Leaf 8. Leaf 8 will import
default route to reach internet routes from BL3 and BL4.)
VLAN interface configuration on DCI Tier 11-12 for L3 VNI VLANs (a similar configuration is needed on Leaf 8)
VE interface configuration on DCI Tier 11 for L3 VNI VLANs (a similar configuration is needed on Leaf 8)

53-1004313-04 179
BGP configuration on DCI Tier 11 (a similar configuration is needed on DCI Tier 12 and Leaf 8)
Interface configuration on DCI Tier 11 to WAN edge1
Interface configuration on DCI Tier 12 to WAN edge2

180 53-1004313-04
VRF configuration on DCI Tier 11 (a similar configuration is needed on BL3 and BL4)

53-1004313-04 181
VE interface configuration to WAN Edge1 on DCI Tier 11 (a similar configuration is needed on WAN edge1)
VE interface configuration to WAN Edge2 on DCI Tier 12 (a similar configuration is needed on WAN edge2)

182 53-1004313-04
BGP configuration on DCI Tier 11 (a similar configuration is needed on DCI Tier 12, BL3, and BL4)
BGP configuration on WAN Edge1 (a similar configuration is needed on WE2, WE3, and WE4)
Local and remote L3 prefixes (Type5 routes) on DCI Tier 11

53-1004313-04 183
Route verification on DCI Tier 11
Route verification on DCI Tier 11

184 53-1004313-04
Server 1 to ISP connected to WE1 and WE2 ping traffic
Route verification on WAN edge1

53-1004313-04 185

186 53-1004313-04
Example 4a—DCI L2 Extension
With DCI L2 extension example

• Extending vlan 203 across two data center sites, a BGP EVPN based (IP Fabric) data center - site 1and a classic VCS fabric
data center - site 4.
• VLAN 203 is extended within data center site 1, between Leaf 5&6 and border-leaf using VNI 20003.
• VLAN 203 is extended between Data center site 1 (DCI tier 11 &12) and Data center site 4 (DCI tier 41 & 42) using VNI 203.
Configuration aspects:
• vLAG connectivity to servers
• Establishing eBGP multihop neighborship between leaf and border-leaf
• Establishing eBGP multihop neighborship between data center edge nodes
• Configuring VLAN 203 on leaf, border-leaf and DCI tier nodes
• Verifying ping between servers 1 and 4
Server Configuration
DC1: Configuration on server bond interface in data center site 1

53-1004313-04 187
DC4: Configuration on server bond interface in data center site 4
Switch Configuration
DC1: Configuring VLAN interface on Leaf 5 & 6, border leaf (nodes that are part of IP Fabric data center)
DC1 & DC4: DCI tier 11 & 12 in data center site 1, leaf nodes, DCI tier 41 & 42 in data center site 4
DC1: VE configuration for default
DC4: FVG configuration on leaf

188 53-1004313-04
DC1: Port-channel configuration on Leaf 5 & 6 pair in data center site 1 (IP Fabric data center)
DC4: Port-channel configuration on leaf nodes in datacenter site 4 toward server
DC1: Port-channel configuration on border-leaf and DCI tier nodes in site 1
DC4: Port-channel configuration between DCI tier 41-42 and leaf nodes in site 4

53-1004313-04 189
DC1 & DC4: Configuring interfaces with channel-group
DC1: Loopback interface configurations on Leaf 5&6
DC1: Loopback interface configurations on border-leaf

190 53-1004313-04
DC1: Loopback interface configurations on DCE11 & 12
DC4: Loopback interface configurations on DCE41 & 42

53-1004313-04 191
DC1: Overlay Gateway configurations on Leaf 5 & 6, border-leaf under global config mode
DC1 & DC4: Overlay Gateway configurations on DCI tier 11, 12, 41, & 42
DC1: EVPN instance configuration on Leaf 5 & 6, border-leaf (under rbridge mode)

192 53-1004313-04
DC1 & DC4: EVPN instance configuration on DCI tier 11, 12, 41, & 42
BGP Configuration on Leaf 5 & 6 in Data Center Site 1
As discussed in the "Validated Design: EVPN DCI with L3 handoff" section, the data center site is a BGP EVPN based IP fabric data
center with leaf-spine topology.
• All leaf and border leaf nodes peer with Spine nodes.
• Leaf-Spine adjacencies are activated under L2VPN EVPN address-family. Adjacencies are configured with next-hop-
unchanged to advertise routes from EVPN peers to other EVPN peers without changing the next-hop.
• In Spine switches, retain route-target all is configured under EVPN address-family. This is to prevent stripping of RTs when
EVPN instance, RT advertised by each Leaf node should be maintained before reflecting to other Leaf nodes.

53-1004313-04 193
DC1: BGP configuration on border-leaf nodes
DC1: BGP configuration on spine nodes
DC1: BGP configuration on Leaf 5&6

194 53-1004313-04
Show Command Verification
Port-channel Verification
DC1: Leaf 5 & 6
DC1: Border-leaf toward DCI tier

53-1004313-04 195
DC4: DCE edge 41 & 42
DC4: Leaf nodes in site 4
BGP Neighborship Verification
DC1: Leaf 5&6

196 53-1004313-04
DC1: Border leaf
DC1: DCI tier 11 & 12

53-1004313-04 197

198 53-1004313-04
Tunnel Verification
DC1: Tunnel from Leaf5&6 to Border-Leaf

53-1004313-04 199
DC1 & DC4: Tunnel between DCI tier 11-12 & 41-42

200 53-1004313-04
DC1 & DC4: Tunnel between DCI tier 11-12 & 41-42 detail output
VLAN Extension Verification
DC1: Leaf 5 & 6
DC1: Border Leaf

53-1004313-04 201
DC1: DCI tier 11-12
DC4: DCI tier 41-42
IMR Route Verification
DC1: Leaf 5&6

202 53-1004313-04
DC1: Border leaf

53-1004313-04 203

204 53-1004313-04

53-1004313-04 205

206 53-1004313-04

53-1004313-04 207
MAC Verification
DC1: Leaf 5 & 6
DC1: Border Leaf

208 53-1004313-04
On DCI tier 11 & 12
DC4: Leaf node in data center site 4
Ping from Server in Site 1 to Site 4
DC1: Server
DC4: Server

53-1004313-04 209
Example 4b—VLAN Asymmetric Routing

For the BGP-EVPN-based L2 extension deployment model, asymmetric routing can be performed through configuration of the source
VLAN and destination VLANs at both sites to allow for routing at the ingress leaf, followed by switching/tunneling to the destination leaf.
In this example, traffic is routed between a host in DC1 (IP Fabric) that is a member of VLAN 203 and a host in VLAN 204 that resides
in DC4 (VCS). The IP Fabric data center site (DC1) is configured with a Static Anycast gateway for both source and destination VLANs.
As noted DC4 is a VCS fabric i.e. non-EVPN domain is configured with Fabric Virtual Gateway (FVG) for routing between VLANs.
A summary of the traffic flow is given below followed by a detailed example.

• Ingress traffic from a host in DC1 is routed between the source and destination VLAN by its directly connected leaf switch
• VXLAN encapsulation is added to the traffic and tunneled to the local border leaf switches in DC1
• Border leaf switches remove the VXLAN encapsulation and switches the traffic at L2 to the local DCI tier in site 1
• DCI tier nodes in site 1 encapsulate the packet in VXLAN header and send it over to remote EVPN peer DCI tier at site 4
• The DCI tier in site 4 removes the VXLAN encapsulation and the inner L2 frame is then forwarded to the destination using L2
forwarding via leaf nodes
Example
VLAN 203 & 204 are extended between Data Center 1 (IP Fabric) and Data Center 4 (VDX VCS):
• DC 1 (IP Fabric): Leaf 5 & 6, Border-leaf, DCI Tier 11-12
– DC 1 VLAN 203 and 204 Static Anycast GW is configured on leaf and border leaf switches
• Data Center Site 4 (VDX VCS): Leaf and DCI tier 41-42
– DC4 VLAN 203 and 204 Fabric Virtual Gateway is configured for the VCS
The following example below builds on the configuration steps from the Example 1 above. For example, VLAN, port-channel, Overlay-
Gateway, EVPN instance and BGP.
Topology

210 53-1004313-04
Server Configuration
DC1: Configuration on server bond interface in data center site 1 (VLAN 203 subnet)
DC4: Configuration on server bond interface in data center site 4 (VLAN 204 subnet)
Switch Configuration
DC1: Configuring VLAN interface on Leaf 5 & 6 and border leaf nodes

53-1004313-04 211
DC1 & DC4: Configuring VLAN interface on DCI tier 11 & 12 in data center site 1, DCI tier 41 & 42 in data center site 4
DC1: In addition to the VE 203 configuration, VE 204 interface is configured on Leaf 5 &6, border leaf in data center site 1
DC4: On site 4 leaf nodes, Fabric Virtual Gateway is configured in global configuration mode
DC4: IP MTU on VE

212 53-1004313-04
DC1: Port-channel configuration on Leaf 5&6
DC4: Leaf to server port-channel configuration on leaf node in site 4
DC1: Port-channel configuration on border leaf and DCI tier nodes in site 1
DC4: Port-channel configuration on DCI tier nodes and leaf nodes in site 4

53-1004313-04 213
DC1: Overlay Gateway configuration on Leaf 5 & 6, border leaf under configuration mode
DC1 & DC4: Overlay Gateway configuration on DCI tier 11, 12, 41, & 42
NOTE
Using map vlan auto will generate VNIs with the same ID as the VLAN ID; for example VLAN 203 will have VNI 203.
DC1 & DC4: EVPN instance configuration on DCI tier 11, 12, 41, & 42

214 53-1004313-04
VLAN Configuration & Verification
DC1: VLAN verification on Leaf 5 & 6
DC1: VLAN verification on border leaf

53-1004313-04 215
DC1: DCI tier VLAN verification
DC2: DCI tier VLAN verification (DCE 41 & 42)
DC4: VLAN verification on leaf nodes - site 4
DC4: IP fabric gateway verification on leafs in site 4

216 53-1004313-04
EVPN Route Verification
DC1: BGP EVPN IMR route verification (Leaf 5 & 6)

53-1004313-04 217
DC1: BGP EVPN IMR route verification border leaf

218 53-1004313-04
DC1: BGP EVPN IMR route DCI tier 11 & 12

53-1004313-04 219

220 53-1004313-04
DC4: BGP EVPN IMR route on DCI tier 41 & 42

53-1004313-04 221

222 53-1004313-04
ARP Verification
DC1: ARP-suppression table on Leaf 5 & 6
DC1: ARP-suppression cache table on border leaf

53-1004313-04 223
DC4: ARP learned on leaf node in data center site 4
Ping from Server in Site 1 to Site 4 (VLAN 203 to VLAN 204)
DC1: Server
Ping from Server in Site 4 to Site 1 (VLAN 204 to VLAN 203)
DC4: Server

224 53-1004313-04
Validated Design: EVPN DCI with BGP-
through Spines
• Topology Description....................................................................................................................................................................................225
• Hardware/Software Matrix ......................................................................................................................................................................... 226
• Configuration Steps....................................................................................................................................................................................... 226
• Configuration: Spine to Spine Layer 3...................................................................................................................................................226
• Example 1—DCI L2 Extension.................................................................................................................................................................228
• Example 2—DCI VLAN Routing .............................................................................................................................................................241
The following section provides a step by step configuration examples for the BGP-EVPN-based L2 and L3 extension deployment
model based on a test topology and walks through common use cases with selected show commands to demonstrate intended
functions.
• In Data Center Site1, all Leaf nodes are connected to two spine nodes (with IPv4 address configured on interfaces in /31
subnet) using IPv4 EBGP adjacency with both spine nodes in same AS 64610. Leaf C and Leaf D are single and LeafE-LeafF,

53-1004313-04 225
LeafG-LeafH are vLAG-pair. Leaf C is in AS 64630, Leaf D is in 64650, Leaf E-Leaf F are in 64640, Leaf G-Leaf H are in
64670. ECMP is achieved using multipath eBGP.
subnet) using IPv4 EBGP adjacency with both spine nodes in same AS 64710. Leaf I, Leaf J and Leaf L are single. Leaf I is in
AS 64720, Leaf J is in AS 64730, Leaf L is in AS 64750. ECMP is achieved using multipath eBGP.
subnet) using IPv4 EBGP adjacency with both spine nodes in same AS 64810. Leaf 1, Leaf 2, Leaf 3, Leaf 4 are single. Leaf 1
is in AS 64820, Leaf 2 is in AS 64830, Leaf 3 is in AS 64840 and Leaf 4 is in AS 64850. ECMP is achieved using multipath
eBGP.
• Leaf-Spine adjacencies are activated under L2VPN EVPN address-family on all Leaf and Spine Switches. Leaf-Spine
adjacencies are configured with next-hop-unchanged to advertise routes from EVPN peers to other EVPN peers without
changing the next-hop.
• In Spine switches, retain route-target all is configured under EVPN address-family. This is to prevent stripping of RTs when
EVPN instance, RT advertised by each Leaf node should be maintained before reflecting to other Leaf nodes.
• Leaf to Host interfaces are configured as an Active-Active vLAG (aggregation of multiple physical links across multiple switches
from a single fabric forming single logical interface). The interfaces can be in access or trunk VLANs with IPv4, IPv6 any cast
address configured to allow VM mobility within or across DCS.
• Overlay Gateway is configured in global context on all leaf nodes (applies to both nodes in case of two node vLAG pair) with
type of overlay to be used, respective VLAN VNI mapping, VTEP membership, switches membership, and VXLAN monitoring
like VLAN stats and SFLOW.
• EVPN instance is configured under rbridge mode for each leaf with RD, RT, VNIs to be extended.
NOTE
Connections between Leaf-to-Spine and Spine-to-Spine are all 40G connections.
Leaf BR-VDX6740T BR-VDX6740 NOS 7.0.1 and up

Spine BR-VDX8770-4/8 BR-VDX6940-36Q BR-VDX6940-144S NOS 7.0.1 and up
Configuration Steps
The BGP-EVPN-based L2 and L3 extension deployment model is characterized by the following:
• Layer 3 reachability between Spine nodes in different data centers via EBGP neighbor ship.
• BGP neighborship between Spine nodes in different data centers with EVPN AF enabled.
Configuration: Spine to Spine Layer 3

Interface configuration on Spine A to Spine G (Similar configurations are needed on interfaces between other Spines).

226 53-1004313-04
Configuration: Spine to Spine Layer 3
BGP Configuration on Spine A to Spines (Similar configurations on all Spines in other DC's with respective IP addresses)
Verify EBGP neighborship is established on Spine A to Spine G.
Verify EVPN neighborship is established on Spine A to Spine G.

53-1004313-04 227

In the following example for the BGP-EVPN-based L2 and L3 extension deployment model:
• VLAN 203 is extended between Data Center Site1 (Leaf G & H), Data Center Site2 (Leaf J) and Data Center Site3 (Leaf 1) with
VNI 20003.
• Traffic between Leaf G & H, Leaf J and Leaf 1 is verified using traceroute from servers attached to the leaf nodes.
• Configuration examples of servers, interfaces, BGP, overlay-gateway, and EVPN instance on leaf nodes are discussed in the

228 53-1004313-04
Server 1 Bond interface configuration for CentOS server attached to Leaf G and Leaf H of Data Center Site1.
Server 2 interface configuration for Win-VM attached to Leaf J of Data Center Site2.
Port-channel interface configuration to Server 1 on Leaf G & H.

53-1004313-04 229
VLAN interface configuration on Leaf G & H.
VE interface configuration to Server 1 on Leaf G & H.

230 53-1004313-04
Loopback interface configuration (VTEP address) on Leaf G & H.
BGP Configurations on Leaf G.
rbridge-id 1
router bgp
local-as 64670
neighbor Leaf-Spine-EBGP-Peer peer-group
neighbor Leaf-Spine-EBGP-Peer remote-as 64610
neighbor 10.1.1.8 peer-group Leaf-Spine-EBGP-Peer
address-family ipv4 unicast
network 10.10.10.4/32
maximum-paths 8
multipath ebgp
!
address-family l2vpn evpn
graceful-restart
neighbor Leaf-Spine-EBGP-Peer activate
neighbor Leaf-Spine-EBGP-Peer allowas-in 1
neighbor Leaf-Spine-EBGP-Peer next-hop-unchanged

53-1004313-04 231
BGP Configuration on Leaf H.
rbridge-id 2
router bgp
local-as 64670
neighbor Leaf-Spine-EBGP-Peer remote-as 64610
network 10.10.10.4/32
maximum-paths 8
multipath ebgp
!
graceful-restart
Overlay gateway configuration on Leaf G and Leaf H.
NOTE
VLAN-to-VNI mapping can be done manually or automatically. If automatic mapping is enabled, the VNI-to-VLAN mapping is
1:1, i.e. VLAN 201 maps to VNI 201.
EVPN instance configuration on Leaf G and Leaf H (per rbridge).

232 53-1004313-04
Port-channel verification on Leaf G and Leaf H.
BGP and EVPN verification on Leaf G can be done similar to Border-Leaf1 to spine (Leaf H can be verified using the same command).
vLAG-pair verification on Leaf G (Leaf H can be verified using the same command).
Leaf_G_6740# show vcs

Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 90
VCS GUID : 25415c12-5399-4425-8396-985e4ae15637
Total Number of Nodes : 2
Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
Anycast gateway verification on Leaf G (the same command can be used to verify on other leaf nodes).
Interface configuration to Server 2 on Leaf J.

53-1004313-04 233
VLAN interface configuration on Leaf J.
VE interface configuration to Server 2 on Leaf J.
Loopback interface configuration (VTEP address) on Leaf J.
BGP Configuration on Leaf J.
rbridge-id 1
router bgp
local-as 64730

234 53-1004313-04

neighbor 10.1.5.2 remote-as 64710
network 10.10.10.6/32
maximum-paths 8
multipath ebgp
!
!
graceful-restart
Overlay gateway configuration on Leaf J (under config mode).
EVPN instance configuration on Leaf J (under rbridge mode).

53-1004313-04 235
rbridge-id 1
router bgp

236 53-1004313-04
local-as 64820
network 10.10.10.70/32
neighbor Leaf-Spine-EBGP-Peer capability additional-paths
maximum-paths 8
multipath ebgp
!
!
graceful-restart
Overlay Gateway configuration on Leaf 1 (under config mode).
EVPN instance configuration on Leaf 1 (under r-bridge mode).
Inclusive-multicast route verification on Leaf G for VNI associated with VLAN 203 (same command can be used to verify on other
nodes).

53-1004313-04 237
Tunnel status verification on Leaf G (same command can be used to verify on other nodes).
Individual Tunnel verification on Leaf G (same command can be used to verify on other nodes).

238 53-1004313-04
VLAN verification on Leaf G and Leaf H for 203 (same command can be used to verify on other nodes).
Server 1 Attached to Leaf G and H issuing ARP.
[root@DCI_Centos ~]# arping -c 4 -A -I bond0 172.17.18.171

ARPING 172.17.18.171 from 172.17.18.171 bond0
Broadcast reply from 172.17.18.171 [00:50:56:bb:4a:fa] 8.128ms

53-1004313-04 239
Sent 4 probes (1 broadcast(s))

Received 1 response(s) (1 broadcast(s))
ARP verification on Leaf G (Locally learnt arp entries can be verified using this command).
ARP suppression verification on Leaf J (Remote ARP learnt via BGP EVPN can be verified using show ip arp suppression-cache).
Local and Remote MAC verification on Leaf G for VLAN 203 (Same command can be used to verify on other leaf nodes).

240 53-1004313-04

Inter VLAN traffic
Asymmetric Routing
• VLAN 203 & 204 is extended between Data Center Site1 (Leaf G & H), Data Center Site2 (Leaf J) and Data Center Site3 (Leaf
1) with VNIs 20003 and 20004 respectively.
• For documentation purpose traffic between Leaf G & H and Leaf J is verified using traceroute from servers attached to the leaf
nodes (between VLAN 203 and 204).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on leaf nodes are discussed in the below
section.
• Refer Example 1 for the configurations and verifications of VLAN 203, corresponding port-channel in that VLAN, Overlay-
Gateway, EVPN instance, BGP and tunnel.

53-1004313-04 241
Server 1 bond interface configuration for server attached to Leaf G and Leaf H of Data Center Site1.
Server 2 interface configuration for Windows VM attached to Leaf J of Data Center Site2.

242 53-1004313-04
VLAN interface configuration to Server 1 on Leaf G & H.
Overlay Gateway configuration on Leaf G and Leaf H under config mode.

53-1004313-04 243
EVPN instance configuration on Leaf G and Leaf H (under rbridge mode).
Port-channel verification on Leaf G and Leaf H.

244 53-1004313-04
VE interface configuration to Server 2 on Leaf J.
Overlay Gateway configuration on Leaf J (under config mode).
EVPN instance configuration on Leaf J (under rbridge mode).

53-1004313-04 245
Inclusive-multicast route verification on Leaf G for VNI associated with VLAN 204 (same command can be used to verify on Leaf H and
Leaf J).
VLAN verification on Leaf G and Leaf H for 204 (same command can be used to verify on Leaf 8).
Server 1 Attached to Leaf G and H issuing ARP.

246 53-1004313-04

ARPING 172.17.18.171 from 172.17.18.171 bond0
Broadcast reply from 172.17.18.171 [00:50:56:bb:4a:fa] 6.448ms
ARP suppression verification on Leaf J (Remote ARP learnt via BGP EVPN can be verified using show ip arp suppression-cache).
ARP verification on Leaf J from Server 2 (Locally learnt arp entries can be verified using this command).
ARP suppression verification on Leaf G (Remote ARP learnt via BGP EVPN can be verified using show ip arp suppression-cache).

53-1004313-04 247
Local and Remote MAC verification on Leaf G for VLAN 204 (Same command can be used to verify on other leaf nodes).
Conversational ARP verification on Leaf G by sending continuous traffic between Server 1 and Server 2.

248 53-1004313-04
Symmetric Routing
• VLAN 203 is configured on Data center Site1 (LeafG & H), VLAN 204 is configured on Data Center Site2 (Leaf J) and VLAN
205 is configured on Data Center Site3 (Leaf 1).
• VRF vpn1 is configured on Leaf G, Leaf H, Leaf J and Leaf 1 with respective import, export route-targets and with common L3
VNI 2005.
• This VNI 2005 is not needed to add under EVPN instance. But VLAN to VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces of 203, 204, 205 and VNI VLAN VE will be configured under VRF vpn1.
• For demonstration purpose traffic between Leaf G & H and Leaf J is verified using traceroute from servers attached to the leaf
nodes (between VLAN 203 & 204).
below section.
• Refer Example 1 for tunnel, port-channel, and VLAN verifications

53-1004313-04 249
Server 1 bond interface configuration for CentOS server attached to Leaf G and Leaf H of Data Center Site1.
Server 2 interface configuration for for Win-VM attached to Leaf J of Data Center Site2.

250 53-1004313-04
Server 3 interface configuration for CentOS attached to Leaf 1 of Data Center Site3.

53-1004313-04 251
VRF configuration on Leaf G & H.

252 53-1004313-04
VRF VNI VLAN and VE interface configuration on Leaf G & H.
VE interface configuration to server on Leaf G & H.

53-1004313-04 253
Overlay Gateway configuration on Leaf G and Leaf H (under config mode).
BGP configuration on Leaf G (similar configuration is needed on Leaf H).

254 53-1004313-04
VRF configuration on Leaf J.
VRF VNI VLAN and VE interface configuration on Leaf J.

53-1004313-04 255
VE interface configuration towards server on Leaf J.
Overlay Gateway configuration on Leaf J (under config mode).
BGP configuration on Leaf J.

256 53-1004313-04
NOTE
Similar Configurations are followed for section "Leaf Node 1 Configurations on DCS3" with Leaf 1 in local AS64820, using
VLAN205, and VRF vpn1 RD 601:601.
VRF VNI verification on Leaf G (Same command can be used to verify on other leaf nodes).
L3 prefixes (Type 5 routes) verification on Leaf G (Same command can be used to verify on other leaf nodes).

53-1004313-04 257
VRF route verification on Leaf G (Same command can be used to verify on other leaf nodes).

258 53-1004313-04
Server 1 Attached to Leaf G and H issuing arp.

ARPING 172.17.18.171 from 172.17.18.171 bond0
Broadcast reply from 172.17.18.171 [00:50:56:BB.4A:FA] 4.729ms
Server 2 Attached to Leaf J
ARP verification on Leaf J (Locally learnt arp entries can be verified using this command).

53-1004313-04 259

260 53-1004313-04
Design Considerations
Tunnel Scale
The Extreme implementation is designed to minimize the number of VXLAN tunnels required at a given leaf switch by allowing for
multiple VLAN-to-VNI mappings on each. With an extended control plane between two different data centers, a general rule of thumb for
calculating the number of tunnels originating from a given leaf switch is to count the number of leaf switches sharing a common VNI. An
illustrative example follows:
Scenario: Multiple leaf nodes in different data centers with a varying number of VLAN/VNI mappings:
• Leaf node 1 in DC1 has VLANs 100–199 mapped to VNI 10100–10199.
• Leaf node 2 in DC1 has VLANs 100–199 mapped to VNI 10100–10199.
• Leaf node 3 in DC3 has VLAN 100 mapped to VNI 10100.
• Leaf node 4 in DC3 has VLAN 200 mapped to VNI 20000.
Two tunnels will be created on the following leaf switches:

• Leaf 1 to Leaf 2 / Leaf 3
No tunnels will be created on Leaf 4 because there are no other leaf switches sharing a common VNI.
Tunnels * VLANs
The tunnels x VLAN scale is calculated as the sum of all VLANs extended across VXLAN tunnels. For example:
Leaf node 1 has 10 VXLAN tunnels (i.e., 10 remote leaf nodes with a common VNI mapping), 5 of which are providing extension for 5
VLANs and the other 5 are extending 10 VLANs. VXLAN * tunnels is 75.
The following tables provide a brief summary of the key scale parameters validated in the test topologies in this document. It should be
noted that these values are not a measure of the maximum scale that can be supported with Extreme switches for DCI.
BGP-EVPN-Based L2 and L3 Extension Validated

Scale
Description Scale Number
Number of VTEPs 512

Number of VNIs per VTEP 8
Number of VRFs 32
Number of IPv4 routes in VRF 2048
Number of IPv6 routes in VRF 1k
Scaling of ARP in HW 16k

53-1004313-04 261
BGP-EVPN-Based L2 Extension Validated Scale
Description Scale Number

Scaling of ND in HW 2k
Scaling of MAC 64k
BGP-EVPN-Based L2 Extension Validated Scale

Number of VXLAN Tunnels on DCI VLANs Extended per Tunnel Tunnels * VLANs (DCI Tier) VNIs per Tunnel at DCI Tier
Tier Nodes (Between DCI Tier Nodes)
4 4000 16000 4000

262 53-1004313-04
References
1. Brocade Data Center Fabric Architectures
http://www.brocade.com/content/dam/common/documents/content-types/whitepaper/brocade-data-center-fabric-
architectures-wp.pdf
2. Configuring a Brocade IP Fabric with Optional BGP EVPN Overlay
http://www.brocade.com/content/html/en/configuration-guide/nos-700-ipfabrics/GUID-C490DC0B-
BEE0-46A8-9A6B-294035E19834.html
3. Brocade IP Fabric and Network Virtualization with BGP EVPN
http://www.brocade.com/content/brocade/en/backend-content/pdf-page.html?/content/dam/common/documents/content-
types/brocade-validated-design/brocade-ip-fabric-bvd.pdf
4. Brocade NetIron Multiprotocol Label Switch Configuration Guide
http://www.brocade.com/content/html/en/configuration-guide/netiron-05900-mplsguide/index.html
5. RFC 7342: BGP EVPN

https://tools.ietf.org/html/rfc7432
6. RFC 3209: MPLS Signaling RSVP-TE
7. RFC 4364: BGP/MPLS IP Virtual Private Networks
8. RFC 5036: LDP Specification

53-1004313-04 263

Data Center Interconnect Evd

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Data Center Interconnect Evd

Hochgeladen von

Copyright:

Verfügbare Formate

EXTREME VALIDATED DESIGN

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based Data Center Interconnect

Extreme EVPN-Based DCI Deployment Model—Overview................................................................................................................................... 11

EVPN DCI Deployment Model 1—BGP-EVPN-Based L2 and L3 Extension ...................................................................................................13

EVPN DCI Deployment Model 2—BGP-EVPN-Based L2 Extension..................................................................................................................21

Validated Design—EVPN DCI with BGP-EVPN-Based L2 and L3 Extension...................................................................................................25

Validated Design—EVPN DCI with BGP-EVPN-Based L2 Extension.............................................................................................................. 113

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based Data Center Interconnect

Extreme Validated Designs

Purpose of the Document

April 2016 53-1004313-01 Initial version.

BGP-EVPN-Based Data Center Interconnect

Date Part Number Description

Layer 2 extension in EVPN DCI with BGP-EVPN-based L2 extension.

VLAN routing in EVPN DCI with BGP-EVPN-based L2 extension.

BGP-EVPN-Based Data Center Interconnect

Active-Active vLAG Active-Active Virtual Link Aggregation Group

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based Data Center Interconnect

This document describes the following architectures:

BGP-EVPN-Based Data Center Interconnect

TABLE 1 BGP-EVPN-Based DCI Model Comparison

Layer 2 extension Yes Yes

Scale consideration at DC leaf (VXLAN tunnel Tunnels contained inside EVPN-based IP

Scale consideration at border leaf (BL will have a

BGP-EVPN-Based Data Center Interconnect

IP Fabric DC Component Review

BGP-EVPN-Based Data Center Interconnect

FIGURE 1 A 3-Stage Folded Clos Topology with Border Leaf

The basic IP fabric topology consists of the following elements:

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based L2 and L3 Extension—Extending

BGP-EVPN-Based Data Center Interconnect

FIGURE 2 DC1-DC2 eBGP Multihop Peering Between Border Leaf Nodes

BGP-EVPN-Based Data Center Interconnect

FIGURE 3 DC1-DC2 DCI with Extended Control Plane (EVPN)

BGP-EVPN-Based Data Center Interconnect

FIGURE 4 DC1-to-DC2 VXLAN Tunnel Formation

BGP-EVPN-Based Data Center Interconnect

FIGURE 5 VLAN Reuse Between Tenants in DC1 to DC2

Shared Control Plane

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based Data Center Interconnect

FIGURE 6 DCI Tier Network Placement

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based L2 Extension—DCI Tier to DCI

FIGURE 7 DC1-DC2 DCI Tier eBGP Multihop Peering

BGP-EVPN-Based Data Center Interconnect

FIGURE 8 Packet Path Between Two Data Center Sites

BGP-EVPN-Based Data Center Interconnect

Separated Control Plane/Administrative Control

BGP-EVPN-Based Data Center Interconnect

BGP-EVPN-Based Data Center Interconnect

Leaf BR-VDX6940-36Q Network OS 7.0 and later

BGP-EVPN-Based Data Center Interconnect

Configuration—Border Leaf to Spine Layer 3

BGP-EVPN-Based Data Center Interconnect

Verify that eBGP neighborship is established on Border Leaf 1 to Spine A.

Verify that EVPN neighborship is established on Border Leaf 1 to Spine A.

BGP-EVPN-Based Data Center Interconnect

Verification can be done similar to verification of Border-Leaf1 to Spine.

BGP-EVPN-Based Data Center Interconnect