Building Data Center Networks With VXLAN EVPN Overlays - Part II

BRKDCT-3378
Building Data Center

Networks with VXLAN EVPN
Overlays – Part II
Lukas Krattiger, Principal Engineer

Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKDCT-3378
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objective
• A short Overview on Overlays
• Standards and Implementation on VXLAN BGP EVPN
• A short repeat on Control- & Data-Plane
• Details around Tenant Routed Multicast (TRM)
• Overview and Details around EVPN Multi-Site
• VXLAN OAM – Operation, Administration and Management
BRKDCT-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Introduction to Overlays
• VXLAN with BGP EVPN
• Standards and Implementation
• Control & Data Plane
• Tenant Routed Multicast (TRM)

• Multi-Site
• VXLAN OAM
Introduction to Overlays
Overlay Taxonomy - Underlay
Layer-3
Interface Spine Spine Spine Spine
Peering
Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Virtual
Server Physical
Server
Overlay Taxonomy - Overlay
Spine Spine Spine Spine
Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
LAN
Segment
Virtual
Server Physical VTEP: VXLAN Tunnel End-Point
Server VNI/VNID: VXLAN Network Identifier
Overlay Taxonomy - Overlay
Tunnel Encapsulation
Spine
(VNI Namespace)
Spine Spine Spine
Overlay
LAN
Segment
Virtual
Server Physical VTEP: VXLAN Tunnel End-Point
Server VNI/VNID: VXLAN Network Identifier
Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3
Data-Plane
Control-Plane • Overlay Layer-2/Layer-3 Unicast Traffic
• Peer-Discovery • Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution Multicast traffic (BUM traffic)
• Local Learning forwarding
• Remote Learning • Ingress Replication (Unicast)
• Multicast
Agenda

• Multi-Site
• VXLAN OAM
Standards and Implementation
What is … ?
• VXLAN • EVPN
• Standards based Encapsulation • Standards based Control-Plane
• RFC 7348 • RFC 7432
• Uses UDP-Encapsulation • Uses Multiprotocol BGP
• Transport Independent • Uses Various Data-Planes

• Layer-3 Transport (Underlay) • VXLAN (EVPN-Overlay), MPLS,
Provider Backbone (PBB)
• Flexible Namespace
• 24-bit field (VNID) provides ~16M • Many Use-Cases Covered
unique identifier • Bridging, MAC Mobility, First-Hop &
• Allows Segmentations Prefix Routing, Multi-Tenancy (VPN)
Introducing Ethernet VPN (EVPN)
EVPN MP-BGP – RFC 7432
MPLS Provider Backbone Bridges Overlay (NVO3)
(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)
VXLAN and EVPN related RFCs & Drafts (IETF)
ID Title Category
RFC 7348 Virtual Extensible Local Area Network Data Plane
RFC 7432 BGP MPLS based Ethernet VPNs Control Plane
draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane
draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane
draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane
draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane
Different Type of EVPN Use-Cases
Use-
Case
Layer-2 &
Layer-2
Layer-3
Asymmetric IRB Symmetric IRB

(VLAN-Aware) (VLAN-Based)
VRF to VRF
Use-
Case
Layer-2 &
Layer-2
Layer-3

VRF-to-VRF
Integrated Routing and Bridging in EVPN
• Symmetric Inter-Subnet
Forwarding
• Bridge->Route/Route->Bridge
• Symmetric VNI in both
directions
• Adjacency contains Remote
VTEP,VRF
• Optimal for Scale
• Flexible Configuration
VTEP = VXLAN Tunnel End-Point

VRF = Virtual Routing and Forwarding
VNI = VXLAN Network Identifier
• Asymmetric Inter-Subnet
Forwarding
• Bridge->Route->Bridge
• Different (Asymmetric) VNI
depending on directions
• Adjacency contains Remote
VTEP,VRF and End-Points
• Potential Sub-Optimal for Scale
• Consistent Configuration

• Symmetric Inter-Subnet • Asymmetric Inter-Subnet

Forwarding Forwarding
• Bridge->Route/Route->Bridge • Bridge->Route->Bridge
• Symmetric VNI in both • Different (Asymmetric) VNI
directions depending on directions
• Adjacency contains Remote • Adjacency contains Remote
VTEP,VRF VTEP,VRF and End-Points
• Optimal for Scale • Potential Sub-Optimal for Scale
• Flexible Configuration • Consistent Configuration

Operational Models for Asymmetric Inter-Subnet Forwarding
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4)
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
Bridge -> Route -> Bridge
ARP and Adjacency Table
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
a.a.a b.b.b
192.168.22.33 192.168.33.44
ARP table V1 ARP table V2

a.a.a, 192.168.22.33, VLAN 30 b.b.b, 192.168.33.44, VLAN 40
b.b.b, 192.168.33.44, VNI 40000 a.a.a, 192.168.22.33, VNI 30000
Routing Table
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
192.168.22.33
a.a.a 192.168.33.44
b.b.b
192.168.22.33 192.168.33.44
Routing table V1 Routing table V2

192.168.22.33, local, VLAN 30 192.168.33.44, local, VLAN 40
192.168.33.44, local, VNI 40000 192.168.22.33, local, VNI 30000
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4.1)
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4.1)
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
Operational Models for Symmetric Inter-Subnet Forwarding
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
Bridge -> Route -> Route -> Bridge
ARP and Adjacency Table
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
a.a.a b.b.b
192.168.22.33 192.168.33.44
ARP table V1 ARP table V2

a.a.a, 192.168.22.33, VLAN 30 b.b.b, 192.168.33.44, VLAN 40
Routing Table
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
a.a.a b.b.b
192.168.22.33 192.168.33.44
Routing table V1 Routing table V2

192.168.22.33, V2, VNI 50000 192.168.33.44, V1, VNI 50000
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1.1)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
L3VNI: 50000
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1.1)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
L3VNI: 50000
Use-
Case
Layer-2 &
Layer-2
Layer-3

VRF-to-VRF
EVPN Layer-2 Service Interface
• Single Subnet per EVI

• VLAN-based
• Per EVI BGP Route Distinguisher / Router Target per EVI / VNI
• BGP Route-Target constrain mechanism to limit propagation (import/export)
• 1:1 mapping
• EVI to Single Broadcast Domain
(Bridge Domain)
• Ethernet Tag ID must be 0
VID = VLAN ID
EVI = EVPN Virtual Instance BRKDCT-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
EVPN Layer-2 Service Interface
• Single Subnet per EVI • Multiple Subnets per EVI

• VLAN-based • VLAN-aware
• Per EVI BGP Route Distinguisher / Router Target per EVI / VNI
• BGP Route-Target constrain mechanism to limit propagation (import/export)
• 1:1 mapping • 1:N mapping

• EVI to Single Broadcast Domain • EVI to Multiple Broadcast Domains
(Bridge Domain) (Bridge Domains)
• Ethernet Tag ID must be 0 • Ethernet Tag ID is to differentiate
Bridge Domains
VID = VLAN ID
Virtual Identifiers to EVI Mapping
(draft-ietf-bess-evpn-overlay – Section 5.1.2)
• VLAN-based
VID
EVI
10
Route Target: 65000:30000
VID = VLAN ID
Virtual Identifiers to EVI Mapping
(draft-ietf-bess-evpn-overlay – Section 5.1.2)
• VLAN-based
VID
EVI
10
VID = VLAN ID
VLAN-Based Service Interface
(RFC7432 – Section 6.1)
• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VID
EVI
10
VID = VLAN ID
VLAN-Aware Bundle Service Interface
(RFC 7432 – Section 6.3)
• VLAN-aware
VID
10
VID
20
EVI
VID
30
[2]:[0]:[20]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VLAN-Aware Bundle Service Interface
(RFC 7432 – Section 6.3)
• VLAN-aware
VID
10
VID
20
EVI
VID
30
[2]:[0]:[20]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VLAN-Based / VLAN-Aware Bundle Service Interface
• VLAN-based • VLAN-aware
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VID
10
VID VID
EVI 20
EVI
10
VID
30
[2]:[0]:[20]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
Use-
Case
Layer-2 &
Layer-2
Layer-3

VRF-to-VRF
IP-VRF-to-IP-VRF Model in EVPN
• Interface-Less Model
• Route-Type 5 only
• Next-Hop is remote VTEP
• Two extended communities
• Encapsulation Extended
Community
• Router’s MAC Address (remote
VTEP)
Route Type 2 = MAC/IP Route

Route Type 5 = IP Prefix Route
IP-VRF-to-IP-VRF Model in EVPN
• Interface-Less Model • Interface-Ful Model (2 Modes)

• Core-facing IRB
• Route-Type 5 only • Unnumbered Core-facing IRB (Optional)
• Next-Hop is remote VTEP
• Route-Type 5
• Two extended communities
• Next-Hop is remote IRB
• Encapsulation Extended
Community • One or two extended communities
• Router’s MAC Address (remote • Encapsulation Extended Community
VTEP) • Router’s MAC Address (remote VTEP)
• Route-Type 2
• Containing Router MAC or MAC/IP
Route Type 2 = MAC/IP Route

Route Type 5 = IP Prefix Route
Interface-less IP-VRF-to-IP-VRF Model
(draft-ietf-bess-evpn-prefix-advertisement – Section 4.4.1)
• Interface-Less
NVE IP: 10.22.22.34
EVPN Router MAC: 0200.0ADE.DE22
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22
Interface-less IP-VRF-to-IP-VRF Model
• Interface-Less
NVE IP: 10.22.22.34
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Interface-ful IP-VRF-to-IP-VRF with SBD IRB
• Interface-Ful (Core-facing IRB)
NVE IP: 10.22.22.34
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[10.22.22.34]
BGP
10.22.22.34 (Next-Hop)
Update
Encap:8 (VXLAN)
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
Interface-ful IP-VRF-to-IP-VRF with SBD IRB
• Interface-Ful (Core-facing IRB)
NVE IP: 10.22.22.34
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[10.22.22.34]
BGP
10.22.22.34 (Next-Hop)
Update
Encap:8 (VXLAN)
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB
• Interface-Ful
(Unnumbered Core-facing IRB)
NVE IP: 10.22.22.34
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB
• Interface-Ful
(Unnumbered Core-facing IRB)
NVE IP: 10.22.22.34
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Control- & Data-Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

• Host MAC (Route Type 2)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 • MAC
10.200.200.107
• MPLS Label1 (L2VNI*)

Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
• Route Target for MAC-VRF
• MAC attributes are Mandatory
Baremetal Baremetal Baremetal
Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
*L2VNI: VNI for all Bridging operation (”VLAN-VNI”) BRKDCT-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L2VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label1)
10.200.200.101 (metric 3) fromL2VNI
10.10.10.201 (10.10.10.201)
Encap:8
Route Target VXLAN
Origin IGP, MED not set, localpref 100, weight 0
Received label 3001
Extcommunity: RT:65500:3001 ENCAP:8
Originator: 10.10.10.101 Cluster list: 10.10.10.201
Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 • Host

5000, 65500:5000 MAC+IP (Route Type 2)
10.200.200.101
•
MAC and IP
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000
• 10.200.200.104
MPLS Label1 (L2VNI)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 192.168.20.101 /32 5000, 65500:5000• Route Target for MAC-VRF
10.200.200.107
• MPLS Label2 (L3VNI*)
Leaf
101010110101
Leaf Leaf Leaf
101010110101
Leaf Leaf Leaf
101010110101
• Route Target for IP-VRF
•
01010101010 01010101010 01010101010
Router MAC
• IP Attributes are Optional

Baremetal Baremetal Baremetal • Populated through ARP/ND
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
*L3VNI: VNI for all Routing operation (”VRF-VNI”) BRKDCT-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
MAC/IP for VRF default, address family L2VPN EVPN
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,
version 4
Paths: (1 available, best #1) IP Address
Length
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked IP Address
Next-Hop L3VNI
IP Address
Path type: internal,L2VNI
path is (MPLS
valid, is best path, no labeled nexthop
Label2)
AS-Path: NONE, path
(MPLSsourced
Label1) internal to AS
10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 VXLAN
Received label 3001 5000
Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
L2VNI L3VNI
Route Target Router MAC
Route Target
Subnet Route Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq.
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

• Internal and External Subnet
Prefixes (Route Type 5)
Overlay • IP Prefix
• MPLS Label (L3VNI)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Route Target for IP-VRF
•
101010110101
01010101010
Router MAC
• Populated through External

Routing Protocol
Subnet A
192.168.10.0/24
Ethernet Tag
Ethernet Identifier
Segment
V2# show bgp l2vpn evpn 192.168.10.0 (Ethtag)
Identifier (ESI) IP Address
Route Type: IP Address
Length family
IP Prefix for VRF default, address L2VPN EVPN
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Next-Hop
IP Address
Path type: internal, path
L3VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label)
10.200.200.101 (metric 3) fromL3VNI
10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 Router MAC
Route Target VXLAN
Received label 5000
Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Routing and the Router MAC – Ethernet
Router MAC
SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1
interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Routing and the Router MAC – VXLAN
Router MAC
SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1
interface: NVE1 interface: NVE1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Packet Walk – Symmetric IRB (A to C)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107
SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA
Baremetal 192.168.10.101 192.168.20.101 Baremetal Baremetal

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Symmetric IRB (C to A)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.107 10.200.200.101 5000 0200.0ade.de07 0200.0ade.de01 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101


MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Asymmetric IRB (A to C)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 10.200.200.107

Payload
10.200.200.101 10.200.200.107 3002 2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA
Baremetal 192.168.10.101 192.168.20.101 Baremetal Baremetal

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Asymmetric IRB (C to A)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 10.200.200.107

Payload
10.200.200.107 10.200.200.101 3001 0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101


MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Agenda

• Multi-Site
• VXLAN OAM
Multicast Forwarding
Tenant Routed Multicast (TRM)
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays
Spine Spine
Site-External DCI
(IP Routing and Increased
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal Baremetal Baremetal
SRC-10 RCVR-10 RCVR-11 RCVR-14

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Same Subnet Forwarding no IGMP Snooping
Spine Spine
Site-External •DCI
”Single Copy” in Core – Treated as BUM
MTU Support)• Same Subnet Only
• No Pruning on Local Interface or Remote VTEP
VXLAN EVPNInterface
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Same Subnet Forwarding with IGMP Snooping
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Spine Spine
• Pruning on Local Interface
VXLAN EVPN
• VXLAN is ”pruned off” if no interest Receiver exists
behind any Remote VTEP
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Spine Spine
VXLAN EVPN
• VXLAN is NOT pruned if interest Receiver exists behind
one Remote VTEP
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Different Subnet Forwarding – Router on-a-Stick
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal 10.10.10.254 Baremetal Baremetal
20.20.20.254
224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Different Subnet Forwarding – Router on-a-Stick
Spine Spine
Multiple Copy in Core – Treated as BUM
MTU Support)• Different Subnet possible – RPF Challenges
VXLAN EVPN
• VXLAN is NOT pruned if interest Receiver exists behind
one Remote VTEP
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal 10.10.10.254 Baremetal Baremetal
20.20.20.254
224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Functional Components
https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-
Tenant Routed Multicast (TRM) seamless-interop
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR
SRC-10 RCVR-10 RCVR-20 RCVR-30 RCVR-11

224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100
Spine Spine
Site-External DCI
Underlay: MTU Support)
• PIM-based Underlay Transport (PIM ASM)
• Separate Multicast Groups from Layer-2 VNI
• Leveraging same redundant Underlay Rendezvous-
VXLAN EVPN
Point (i.e. PIM Anycast-RP) Single Packet in Core
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100
Spine Spine
Site-External DCI
Underlay: MTU Support)
• PIM-based Underlay Transport (PIM ASM)
• Separate Multicast Groups from Layer-2 VNI
• Leveraging same redundant Underlay Rendezvous-
VXLAN EVPN
Point (i.e. PIM Anycast-RP) Single Packet in Core
VTEP VTEP VTEP VTEP
DR DR DR DR
Overlay:
• BGP-based Control-Plane using ngMVPN (Next-
Generation Multicast VPN)
• Using existing BGP Route-Reflector
• Rendezvous-Point-less
Baremetal Baremetal
• Efficient Single Copy in Multicast Underlay

224.10.10.10 10.10.10.10 20.20.20.20 • Always-Route approach
30.30.30.30 (per-VLAN config)
10.10.10.11
10.10.10.100 • Distributed Anycast Designated Router (DR)
• VPC – Virtual Port-Channel
• Integration with non-TRM VTEP
Forwarding Behavior
Tenant Routed Multicast (TRM)
Same Subnet Forwarding – Local and Remote Snooping
TRM Forwarding (Layer-2 only mode)
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Spine Spine
Local IGMP Termination (needs Querier)
• Bridge approach only – in Layer-2 VNI
MTU Support)
• ”Single Copy” in Core
VXLAN EVPN
• Local and Remote IGMP Snooping
• Uses BGP EVPN Route-Type 6 (SMET)
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Different and Same Subnet Forwarding
TRM Forwarding (Layer-3 Mode)
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Different and Same Subnet Forwarding
Spine Spine
Distributed Designated Router (DR)
• Always Route approach – in Layer-3 VNI (VRF)
MTU Support)
• ”Single Copy” in Core
VXLAN EVPN
• Egress replication - closest to the fan-out.
• Single Default MDT (I-PMSI)
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Local and Remote Forwarding
Spine Spine
Site-External DCI
TTL Decrement
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR
No TTL Decrement
(bridged) TTL Decrement
(routed)

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Local vs. Remote Forwarding
Spine Spine
Site-External Tenant
DCI Routed Multicast routes between different IP
Subnets on the same Switch (Leaf). As a result, the TTL is
TTL Decrement
MTU Support) decremented in this routed forwarding operation.
Local to the same Switch (Leaf), Multicast is bridged within

VXLAN EVPN
the same IP Subnet. During this forwarding operation, TTL
is not decremented.
VTEP VTEP Tenant

VTEP Routed Multicast
VTEP uses an always route, single
DR DR copy
DR approach across
DR the VXLAN EVPN Fabric. This is
No TTL Decrement true if Sources and Receivers reside in the same or in
(bridged) TTL Decrement different IP Subnets. As a result, the TTL is decremented
(routed) during this forwarding operation.
All IP Subnets must be in the same Routing Domain or

Tenant (= VRF)
Baremetal Baremetal

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Overlay Rendezvous Point
Spine Spine
Site-External DCI
• RP-less
MTU Support) • Distributed Anycast RP (NGMVPN-based)
• Shortest Path Tress (SPT only)
VXLAN EVPN • Requires per-Tenant Loopback, Multicast enabled
• External RP
VTEP VTEP VTEP • Centralized
VTEP RP (PIM-based)
DR DR DR • Shared Tree
DR and Shortest Path Tree (cut over)
• Requires External PIM-based RP

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
TRM Control- & Data-Plane
Underlay Multicast Tree – PIM ASM
Spine Spine
Layer-3 VNI: 50001
Default MDT: 239.1.1.2
Site-External DCI
Route-Target: 65502:50001
MTU Support)
S,G – (10.0.0.1, 239.1.1.2/32)
VRF IIF: NVE-Loopback S,G – (10.0.0.2, 239.1.1.2/32)
(Underlay)
Tenant1 VXLAN EVPNS,G
OIF: Uplink (Underlay)
IIF: NVE-Loopback – (10.0.0.3, 239.1.1.2/32)
(Underlay)
*,G – (*, 239.1.1.2/32) OIF: Uplink (Underlay)
IIF: NVE-Loopback S,G – (10.0.0.4, 239.1.1.2/32)
(Underlay)
IIF: Uplink (Underlay) OIF: Uplink (Underlay)
IIF: NVE-Loopback (Underlay)
OIF: NVE1 (Underlay) OIF: Uplink (Underlay)
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Underlay Multicast Tree – PIM ASM
Spine Spine
Layer-3 VNI: 50001
PIM ASM required for Underlay
Route-Target: 65502:50001 • Separate Groups for BUM and MDT
MTU Support)
S,G – (10.0.0.1, 239.1.1.2/32)
VRF IIF: NVE-Loopback S,G
• Default MDT initiates on VTEP startup (*, G)
– (10.0.0.2, 239.1.1.2/32)
(Underlay)
Tenant1 VXLAN
OIF: Uplink (Underlay) EVPN • Per-VTEP
IIF: NVE-Loopback S,G – (10.0.0.3,
(Underlay) (S,G)239.1.1.2/32)
imitated on VTEP startup
*,G – (*, 239.1.1.2/32) OIF: Uplink (Underlay)
IIF: NVE-Loopback S,G – (10.0.0.4, 239.1.1.2/32)
(Underlay)
IIF: Uplink (Underlay) OIF: Uplink (Underlay)
IIF: NVE-Loopback (Underlay)
OIF: NVE1 (Underlay) OIF: Uplink (Underlay)
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
NGMVPN – Source Active Advertisement (MVPN Type 5)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32) Source Active (NGMVPN Type 5)
IIF: Uplink (Underlay)
Originator: Leaf #1
OIF: NVE1 (Underlay)
S,G: 10.10.10.100,
VTEP VTEP 224.10.10.10
VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
MRoute – Overlay Multicast Tree (Source Join)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32)
S,G – (10.10.10.100, 224.10.10.10) VTEP VTEP VTEP VTEP

S,G – (10.10.10.100, 224.10.10.10)
IIF: VLAN100 (Host-facing) DR DR DR DR IIF: VRF-L3VNI (Overlay)
OIF: None OIF: None

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
NGMVPN – Source Tree Join (MVPN Type 7)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32) Source Tree Join (NGMVPN Type 5)
Originator: Leaf #4
S,G: 10.10.10.100, 224.10.10.10
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
MRoute – Overlay Multicast Tree (Receiver Join)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32)
S,G – (10.10.10.100, 224.10.10.10) VTEP VTEP VTEP VTEP

S,G – (10.10.10.100, 224.10.10.10)
IIF: VLAN100 (Host-facing) DR DR DR DR IIF: VRF-L3VNI (Overlay)
OIF: VRF-L3VNI (Overlay), VLAN100 (Host-facing) OIF: VLAN100 (Host-facing)

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Agenda

• Multi-Site
• VXLAN OAM
VXLAN BGP EVPN Multi-Site
Inter-X Connectivity
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site
EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal
DCI DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane
 Single Fabric with End-to-End  Multiple Fabrics – Normalized  Multiple Fabrics with
Encapsulation through Ethernet Integrated DCI
 Build Hierarchy in the Underlay  Multiple Fabrics Interconnect  Integrated DCI – Scaling
– Flatten it in the Overlay using DCI (Layer 2 and Layer 3) within and between Fabrics
Functional Components https://tools.ietf.org/html/draft-sharma-multi-site-evpn
Site-External DCI
Border Gateways MTU Support)
(Key Functional Components of
VXLAN Multi-Site Architecture)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
Site-Internal Fabric
(Common VXLAN and
96
BGP-EVPN Functions) BRKDCT-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hierarchical Overlay Domains
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine
Site 1 Site n
Anycast Border Gateway (1)
Anycast Border Gateway
 Up to 4 Border Gateways
 Border Gateway
BGW BGW BGW BGW • Deploying at Leaf – 7.0(3)I7(1)
VTEP VTEP VTEP VTEP
• Deploying at Spine – 7.0(3)I7(2)
Site 1
 Common Virtual IP (VIP) across BGW
Border VIP
10.1.1.111 • VIP for communication between the Border
BGW BGW BGW BGW
Gateways in different Sites
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4 • VIP for communication between Border
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
Gateway and Leaf within a Site
Border VIP
 Individual Primary IP (PIP) per BGW
10.1.1.111
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
endpoints (routed only), intra- and inter-Site
Site 1
Type: 03 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:03:09 VNI: 30010  Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a
BGW BGW BGW BGW
set of Layer-2 VNIs
VTEP VTEP VTEP VTEP
• DF election and assignment is automatic
DF
30010
DF
30011
DF
30012
DF
30099
 Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 03)
BGP EVPN
• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
RR RR
Spine Spine
• Multi-Site Discriminator (Ethernet-Segment:
00:03:09)
• Originators IP Address (PIP): 10.1.1.101
• Layer-2 VNI: 30010
Site 1
Failure Detection on BGWs – Fabric Isolation (1)
 The Site-Internal interfaces on BGW nodes are

constantly tracked to determine their status (‘evpn
Site-External
Multi-Site VIP
multisite fabric-tracking’ command)
10.111.111.1
BGW BGW BGW BGW

VTEP VTEP VTEP VTEP
PIP-BGW2 PIP-BGW3 PIP-BGW4

10.200.200.22 10.200.200.23 10.200.200.24
Site-Internal
Spine Spine
Site 1

Site-External
Multi-Site VIP
10.111.111.1
 If all the Site-Internal interfaces are detected as
BGW BGW BGW BGW down:
VTEP VTEP VTEP VTEP
1. The isolated BGW stops advertising PIP/VIP
addresses toward the Site-External network
10.200.200.22 10.200.200.23 10.200.200.24
The remaining BGWs perform new DF elections for
Site-Internal
2.
the L2VNIs owned by the isolated BGW
Spine Spine
Site 1

Site-External
Multi-Site VIP
10.111.111.1
 If all the Site-Internal interfaces are detected as
BGW BGW BGW BGW down:
VTEP VTEP VTEP VTEP
1. The isolated BGW stops advertising PIP/VIP
addresses toward the Site-External network
10.200.200.22 10.200.200.23 10.200.200.24
The remaining BGWs perform new DF elections for
Site-Internal
2.
the L2VNIs owned by the isolated BGW
Spine Spine  As a result, the BGW becomes isolated from both
the Site-Internal and Site-External networks
Site 1
 Seamless BGW node reinsertion using a “delay-
restore” timer for the VIP address
Failure Detection on BGWs – DCI Isolation
DC Core
(Layer-3 Unicast)  The Site-External interfaces on BGW nodes are
also tracked to determine their status (‘evpn
Site-External
multisite dci-tracking’ command)

 If all the Site-External interfaces are detected as
BGW BGW BGW BGW down, the isolated BGW node:
VTEP VTEP VTEP VTEP
1. Stops advertising VIP VTEP address toward the Site-
PIP-BGW1 PIP-BGW2 PIP-BGW3 PIP-BGW4
Internal network
10.200.200.21 10.200.200.22 10.200.200.23 10.200.200.24 Withdraws BGP EVPN Type-4 advertisements
Site-Internal
2.
Multi-Site VIP (triggering a new DF election between other BGWs)
10.111.111.1
3. Starts functioning as a regular VTEP (PIP still up)
 As a result, the BGW continues to operate as a

Site 1 Site-Internal VTEP
 Seamless BGW node reinsertion using a “delay-
restore” timer for the VIP address
Multi-Site Control- & Data-Plane
Multi-Site Control Plane Deployment Considerations
 MP-eBGP EVPN only inter-Sites
• Next-hop behavior (VXLAN tunnel termination and reorigination) and loop protection (as-
path attribute)
 Two main options for underlay and overlay control plane deployment
1. I-E-I (Recommended)
• Intra-Site: IGP (OSPF, IS-IS) as underlay CP, iBGP as overlay CP
• Inter-Sites: eBGP for both underlay and overlay CPs
2. E-E-E
• Intra-Site and Inter-Sites: eBGP for both underlay and overlay CPs
 Full mesh of MP-eBGP EVPN adjacencies across sites

• Recommended to deploy a couple of Route-Servers with 3 or more sites
• RS in a separate AS only perform control plane functions (“eBGP Route-Reflectors”,
IETF RFC 7947)
• RS functions: EVPN routes reflection, next-hop-unchanged, route-target rewrite
Multi-Site Overlay Control Plane – back-to-back
Site 3
Site N
Site 2
eBGP EVPN
Full mesh
Site 1
Multi-Site Overlay Control Plane – Route-Server
Site 3
eBGP EVPN
RS
Site N
Site 2
RS
eBGP EVPN
Site 1
Multi-Site Overlay Control Plane – Tenants
RS
L3VNI: 50001 L3VNI: 50001

Route-Target: 65501:50001 DC Core Route-Target: 65502:50001
(Layer-3 Unicast)
VRF VRF
Tenant1 Tenant1
DCI
…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW
VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Control Plane – Site1
RS
L3VNI: 50001 L3VNI: 50001

(Layer-3 Unicast)
2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1

VRF VRF
Tenant1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 Tenant1 10.1.1.111
50001, 65501:50001
DCI 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111
…. ….
10.1.1.111 10.2.2.222
VXLAN EVPN VXLAN EVPN

RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Control Plane – Site2
RS
L3VNI: 50001 L3VNI: 50001

(Layer-3 Unicast)
2 VRF 30010, 65502:30010

0000.3010.1101/48 192.168.10.101/32 50001, 65502:50001 10.2.2.222 VRF
2 Tenant1 30020, 65502:30020
0000.3020.2101/48 192.168.20.101/32 50001, 65502:50001 10.2.2.1 Tenant1
DCI
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3
…. ….
10.1.1.111 10.2.2.222

RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Control Plane – Between Sites
RS
L3VNI: 50001 L3VNI: 50001

(Layer-3 Unicast)
VRF VRF
Tenant1 Tenant1
DCI
…. ….
Type MAC / Length
10.1.1.111 L2VNI / RT IP / Length L3VNI / RT Next-Hop
10.2.2.222 Seq.
Fabric BGW 2 0000.3010.1101/48BGW30010, 65599:30010 192.168.10.101/32 BGW
50001, 65599:50001 10.1.1.111 BGW
2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222

VXLAN
2
EVPN
0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32
VXLAN
50001, 65599:50001
EVPN
10.2.2.222
RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Data Plane – Overview
Inter-site VXLAN
Data Plane
De-capsulation and DC Core

Re-encapsulation on De-capsulation and
BGW (L2 or L3 lookup)
(Layer-3 Unicast)
Re-encapsulation on
BGW (L2 or L3 lookup)
DCI
…. ….
10.1.1.111 10.2.2.222

Intra-site VXLAN
Data Plane
Site1 Site2
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Packet Walk (BUM)
Multi-Site – BUM Traffic Distribution
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
BUM
Site 1 Site n
Baremetal
Multi-Site – BUM Replication Modes (Multicast Sites)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Multicast Multicast
Site 1 Site n
Multi-Site – BUM Replication Modes (All Ingress Replication)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Ingress Replication
Site 1 Site n
Multi-Site – BUM Replication Modes (Mixed Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Multicast
Site 1 Site n
Multi-Site – BUM Traffic Enforcement
Overlay Multi-Site
Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP
BGW BGW
Unknown Unicast 0-100% BGW BGW
Multicast 0-100%
Spine Spine Spine
BUM
Site 1 Site n
Baremetal
Layer 2 (BUM) – Site 1
Bridge

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
Leaf10 replicates VTEP 30010 VTEP
traffic intra-Site BGW11 BGW21
2
VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20
DF
VTEP 30010 VTEP
1 Host 1 sends a
BGW12 BGW22
L2 BUM frame
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – Site 1
Bridge

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP

DF
VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102
Drop due to Split-Horizon rule

Layer 2 (BUM) – DCI
Bridge
BGW11 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 Payload
BGW11 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010
BGW11 replicates traffic VTEP
BGW11
inter-Sites toward remote BGW21
BGW nodes
VTEP BUM Forward 3 VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – DCI
Bridge
BGW11 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 Payload
BGW11 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP

DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
192.168.10.101 192.168.10.102

Bridge

Payload
BGW22 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21 BGW22 replicates

traffic intra-Site
VTEP
4 VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – Site 2
Bridge

Payload
BGW22 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
192.168.10.101 192.168.10.102

Bridge
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
5
Leaf20 sends traffic to
local Host 2
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Multi-Site and External Layer 3 Connectivity
 The BGW nodes can also be used to provide Layer-3 external

connectivity to each site
 Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers (Shared
Border deployment model, aka GOLF)
• Dedicated or shared pair of WAN Edge routers across sites
 External Layer-3 network may be different from the DCI network

used for inter-site communication
Border Gateway and VRF-Lite
Separate routing peering

Dedicated physical interfaces / VRF-A VRF-B VRF-C
for each VRF (IGP or
sub-interfaces* for each VRF, External eBGP)
separate from interfaces used Connectivity
for Multi-Site traffic
Site-External
Multi-Site
Overlay
BGW BGW BGW BGW BGW BGW BGW BGW

Site-Internal
Site 1 Site N
*No current SVIs support on BGWs
Border Gateway and Shared External Connectivity
Various hand-off options
depending on deployed HW Shared Border operates like a
VRF-A VRF-B VRF-C traditional VXLAN EVPN
(VRF-Lite, MPLS-VPN, LISP)
External VTEP (Layer 3 only)
Connectivity
Single MP-BGP EVPN VXLAN Data Plane
instance to exchange VTEP VTEP
between BGW and WAN
Site-External
routes for all VRFs Edge Router

Shared Border
Multi-Site
Overlay
BGW BGW BGW BGW BGW BGW BGW BGW

Site-Internal
Site 1 Site N
Agenda
• Multicast Forwarding
• Multi-Site
• VXLAN OAM
Operations, Administration and Management (OAM)
• OAM – processes, activities,

tools and standards
• Various Mode of Operation
• Pro-Active
• Controlling a Situation
• Re-Active
• Responding to a Situation
VXLAN OAM - OAM Model of Operation
V V V V V V V
Endpoint Locator Ping / Path MTU Pathtrace Pro-Active Monitoring

• Locate End-Host and • Check liveliness of End- • Trace paths to End-Host • Proactive Monitoring with
Segment Identifier Host and Tunnel-Endpoint Threshold and State
Notifications
• Track History of End- • Option to specify Payload • Get Path, Interface and
Host Parameters Error statistics along path
• Provide Fabric Host- • Specify Payload
Count and Activity Parameters for Path
Selection
NGOAM or VXLAN OAM
• Next Generation OAM for Data

Center Fabrics
• Running on Nexus 9000, Nexus
7000 and Nexus 5600
• VXLAN Today
• All IP Tomorrow
• Various Methods to Execute and

Retrieve Data
• Command Line Interface (CLI)
• NX-API
• DCNM (using NX-API)
Endpoint Reachability – VXLAN OAM
• Endpoint Reachability
• Uses ICMP
• VTEP to Endpoint reachability
Overlay Is Host A alive? • VTEP to VTEP reachability

• Validates ECMP Path
• Single Random Path
• Multiple, Random/Specified Path
• Provides VXLAN Outer UDP

Baremetal Baremetal Baremetal Source Port (SPORT) as output
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Endpoint Traceroute – VXLAN OAM
• Endpoint Reachability
• Uses ICMP
• VTEP to Endpoint reachability
Overlay What is the Path • VTEP to VTEP reachability
to Host A?

• Validates Overlay Path
• Single Specified Path
• Multiple, Specified Path
• Provides Overlay to Underlay

Baremetal Baremetal Baremetal correlation
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
How would a normal Traceroute look alike?
Eth1/5
10.1.1.17
Spine
NVE1
10.200.200.18 What is the Path
to Host A?
VTEP VTEP
Baremetal
AS#65500
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Which Path did my Traceroute take?

Spine
Eth1/5
10.1.2.17
How would a normal Traceroute look alike?
Eth1/5
10.1.1.17
Spine
NVE1
10.200.200.18 What is the Path
to Host A?
VTEP VTEP
L15# traceroute 192.168.10.101 source 10.50.1.15 vrf BLUE
Baremetal
AS#65500
traceroute to 192.168.10.101 (192.168.10.101) from 10.50.1.15 (10.50.1.15), 30 hops max, 40 byte packets
Host(10.50.1.18)
1 10.50.1.18 A 0.96 ms 0.817 ms 0.746 ms
MAC: 0000.3001.1101
2 2 192.168.10.101 (192.168.10.101) 4.751 ms 0.69 ms 0.697 ms
IP: 192.168.10.101
Which Path did my Traceroute take?

Spine
Eth1/5
10.1.2.17
Endpoint Traceroute – VXLAN OAM – Close-Up
L15# traceroute nve ip 192.168.10.101 vrf BLUE source 10.50.1.15 sport 35977 verbose
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem),
'c' - Corrupted Data/Test, '#' - Duplicate response
Traceroute Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 94
1 !Reply from 10.1.1.17,time = 1 ms Spine Ingress Interface IP
2 !Reply from 10.200.200.18,time = 1 ms
Destination VTEP IP
3 !Reply from 192.168.10.101,time = 4 ms
Host A IP
Spine Ingress Interface and Destination VTEP IP Address
are Underlay Information – additions vs. standard Traceroute
Pathtrace for Enhanced Network Visibility
What is the Path

• Application Specific Pathtrace
from Host C to • Uses “draft-tissa-nvo3-oam-fm”
Host A for HTTP?
•
Endpoint to Endpoint Pathtrace
• Adds Interface Load and Error
Overlay Statistic of the Path
• Uses Protocol Information
• Validates Specific or All Path

• Provides Overlay to Underlay
correlation
Host A Host B Host C • Superset of NVE Ping/Traceroute

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Pathtrace – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE Known or Unknown VTEP IP Address
payload
ip 192.168.10.101 192.168.20.101 Dst Endpoint IP / Src Endpoint IP
port 54321 80 Source Port / Destination Port
proto 6
payload-end TCP (IANA Protocol Number 6)

'D' - Destination Unreachable, 'X' - unknown return code,
Path trace Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 142
Hop Code ReplyIP IngressI/f EgressI/f State

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP
2 !Reply from 10.200.200.18, Eth1/54 Unknown UP / DOWN
Why are we specifying Payload information?
101010110101010
10101010
• VXLAN provides variable UDP

Baremetal Baremetal
Host A Spine Host B

MAC: 0000.3001.1101
IP: 192.168.10.101
Source Port in Outer Header
MAC: 0000.3001.1102
IP: 192.168.10.102
• Hash of the inner Layer-2/Layer-

VTEP Spine
3/Layer-4 Headers of the original
VTEP
Ethernet Frame.
• Enables entropy for ECMP Load
balancing in the Network
Which Path did your Application Traffic took?
Pathtrace – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE payload ip 192.168.10.101 ...

'D' - Destination Unreachable,
Spine Ingress 'X' Interface
Interface, Egress - unknown return
and code,VTEP IP Address
Destination
are Underlay Information – additions vs. standard and NVE Traceroute
Path trace Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 142
Spine Ingress Interface IP Spine Ingress Interface Spine Egress Interface
Hop Code ReplyIP IngressI/f EgressI/f State

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP Interface Status
2 !Reply from 10.200.200.18, Eth1/54 Unknown UP / DOWN
Destination VTEP IP Destination Leaf Ingress Interface
Database Output – VXLAN OAM – Close-Up
L15# show ngoam pathtrace database session 168 detail
Pathtrace entry for session id 168 OAM Session ID

================================
Start time: Tue Jun 13 01:18:39.710 PDT

End time: Tue Jun 13 01:18:39.735 PDT
Last Clear of Summary Statistics: Never

Pathtrace Requests: sent (2)/received (0)/timeout (0)/unsent (0)
Pathtrace Replies: sent (0)/received (2)/unsent (0)/Duplicate (0)
! Reply from 10.1.1.17 on Eth1/5, state UP. Sent on Eth1/8, state UP.
Interface stats for interface: Eth1/5
-------------------------------
Rx Len : 84
Rx Bytes : 66113123 Interface Statistics
Rx Pkt rate : 0
Rx Byte rate : 0
Rx Load : 0 BRKDCT-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 178
Summary
Summary
• Overview on VXLAN Overlay
• Control- and Data-Plane interactions
• Some info around Multicast forwarding
• How Multi-Site enhanced VXLAN EVPN
• Operations is key – VXLAN OAM
If you haven’t
had enough
VXLAN BGP
EVPN
Links & Resources
• VXLAN Multi-Site Intro
• https://blogs.cisco.com/datacenter/vxlan-innovations-vxlan-evpn-multi-site-part-2-of-2
• VXLAN Multi-Site @ Cisco Live online
• https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035#/
• ”eBGP” for EVPN
• https://learningnetwork.cisco.com/blogs/community_cafe/2017/11/02/vxlan-ebgp-evpn-
the-incarnation-of-a-hybrid-guest-post
• Configuration Example
• https://communities.cisco.com/community/technology/datacenter/data-center-
networking/blog/2015/05/19/vxlanevpn-configuration-example
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKDCT-3378
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
www.ciscolive.com/global/on-demand-library/.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

Building Data Center Networks With VXLAN EVPN Overlays - Part II

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Building Data Center Networks With VXLAN EVPN Overlays - Part II

Hochgeladen von

Copyright:

Verfügbare Formate

BRKDCT-3378

Building Data Center

Lukas Krattiger, Principal Engineer

• Tenant Routed Multicast (TRM)

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Spine Spine Spine Spine

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

• Tenant Routed Multicast (TRM)

• Transport Independent • Uses Various Data-Planes

EVPN MP-BGP – RFC 7432

MPLS Provider Backbone Bridges Overlay (NVO3)

(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

Asymmetric IRB Symmetric IRB

Asymmetric IRB Symmetric IRB

VTEP = VXLAN Tunnel End-Point

VTEP = VXLAN Tunnel End-Point

• Symmetric Inter-Subnet • Asymmetric Inter-Subnet

VTEP = VXLAN Tunnel End-Point

VNI 30000 (L2VNI)

Bridge -> Route -> Bridge

VNI 30000 (L2VNI)

ARP table V1 ARP table V2

VNI 30000 (L2VNI)

Routing table V1 Routing table V2

VNI 30000 (L2VNI)

VNI 30000 (L2VNI)

Bridge -> Route -> Route -> Bridge

ARP table V1 ARP table V2

Routing table V1 Routing table V2

Asymmetric IRB Symmetric IRB

• Single Subnet per EVI

• Single Subnet per EVI • Multiple Subnets per EVI

• 1:1 mapping • 1:N mapping

Route Target: 65000:30000

Route Target: 65000:30000

Route Target: 65000:30000

Route Target: 65000:30000

Route Target: 65000:30000

Asymmetric IRB Symmetric IRB

Route Type 2 = MAC/IP Route

• Interface-Less Model • Interface-Ful Model (2 Modes)

Route Type 2 = MAC/IP Route

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

• Interface-Ful (Core-facing IRB)

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

• Interface-Ful (Core-facing IRB)

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22