Sie sind auf Seite 1von 54
ACI Multisite Control plane and Data plane troubleshooting Partner DC VT Mar 2019 Roland Ducomble

ACI Multisite Control plane and

Data plane troubleshooting

Partner DC VT Mar 2019

Roland Ducomble

CX Technical Leader ACI Solutions Team

Introduction

© 2019 Cisco and/or its affiliates. All rights reserved.

ACI Multi-Site

Overview

IP Network MP-BGP - EVPN ACI Multi Site Orchestrator (MSO) REST
IP Network
MP-BGP - EVPN
ACI
Multi Site
Orchestrator
(MSO)
REST
GUI API Availability Zone ‘B’ Availability Zone ‘A’ Region ‘C’
GUI
API
Availability Zone ‘B’
Availability Zone ‘A’
Region ‘C’

Separate ACI Fabrics with independent APIC clusters

ACI MSO pushes cross-fabric configuration to multiple APIC clusters providing scoping of all configuration

changes

© 2019 Cisco and/or its affiliates. All rights reserved.

MP-BGP EVPN control plane between sites

Data Plane VXLAN encapsulation across sites

End-to-end policy definition and enforcement

ACI Multi-Site And translation

Network information carried across Site Identity information carried across site (Class-id aka pcTag (VNID for
Network information carried across Site
Identity information carried across site
(Class-id aka pcTag
(VNID for VRF or BD)
Tenant
VTEP IP
VNID
Class-ID
No Multicast Requirement in
Packet
Backbone, Head-End Replication
IP Network
(HER) for any Layer 2 BUM
traffic)
MP-BGP - EVPN
ACI
MSO

A same VRF, BD or EPG created on two different site will have different VNID and Class-Id as those are allocated by the local APIC cluster (not by MSO) Need for a translation !

In ACI multisite, data packet are across site are vxlan encapsulated with the

vnid/class of the source site The Destination site spine will have the role of translating those value

ACI Multi-Site

Namespace translation

Translation of Source VTEP address IP Network Translation of Class-ID, VNID (scoping of name spaces)
Translation of Source
VTEP address
IP Network
Translation of Class-ID, VNID
(scoping of name spaces)
MP-BGP - EVPN
ACI
MSO
VNID (scoping of name spaces) MP-BGP - EVPN … ACI MSO Site 1 Site n Site

Site 1

of name spaces) MP-BGP - EVPN … ACI MSO Site 1 Site n Site to Site
of name spaces) MP-BGP - EVPN … ACI MSO Site 1 Site n Site to Site
of name spaces) MP-BGP - EVPN … ACI MSO Site 1 Site n Site to Site
of name spaces) MP-BGP - EVPN … ACI MSO Site 1 Site n Site to Site
of name spaces) MP-BGP - EVPN … ACI MSO Site 1 Site n Site to Site

Site n

Site to Site VTEP traffic (VTEPs, VNID and Class-ID are mapped on spine) Leaf to
Site to Site VTEP traffic (VTEPs, VNID
and Class-ID are mapped on spine)
Leaf to Leaf VTEP, Class-ID is local to the Fabric
Leaf to Leaf VTEP, Class-ID is local to the Fabric
VTEP
VNID
Class-ID
Tenant Packet
IP
VTEP
VNID
Class-ID
Tenant Packet
VTEP
IP
VNID
Class-ID
Tenant Packet
IP

© 2019 Cisco and/or its affiliates. All rights reserved.

ACI Multipod versus Multisite

Multipod
Multipod

Controller : Single APIC controller for all pod

Multisite
Multisite

Controller : separate APIC cluster per site, MSO talks to each APIC cluster

Namespace : No need of any translation (all come from same APIC cluster)

Control Plane : BGP EVPN used to sync COOP DB

Unicast Data : VXLAN encapsulated

Leaf to Leaf tunnel for established flow

BUM traffic in the IPN: multicast encapsulated in BD GIPo

Control plane : PIM BiDir

© 2019 Cisco and/or its affiliates. All rights reserved.

Namespace : need to translate vnid and pcTag across site (done on target spine)

Control Plane : BGP EVPN used to sync COOP DB

Unicast Data : VXLAN encapsulated in source site VNID

Leaf to target site spine tunnel for establish flow

(Always need to hop by Spine for Xlate)

BUM traffic in the ISN : unicast copy to each target site (HREP Head end replication)

Control plane: no need of multicast control plane

Namespace Translation

troubleshooting

© 2019 Cisco and/or its affiliates. All rights reserved.

Object Model

For every VRF (fvCtx), BD

(fvBD)or EPG (fvAEPg) that needs to be extended.

Each APIC creates additional

object to represent the vrf/bd/epg on each of the

remote site

© 2019 Cisco and/or its affiliates. All rights reserved.

fvCtx fvSiteAssociated fvRemoteId
fvCtx
fvSiteAssociated
fvRemoteId
All rights reserved. fvCtx fvSiteAssociated fvRemoteId Site-Id Remote-Id fvBD fvAEPg fvSiteAssociated

Site-Id

Remote-Id

fvBD fvAEPg fvSiteAssociated fvSiteAssociated fvPeerContext fvRemoteId fvRemoteId
fvBD
fvAEPg
fvSiteAssociated
fvSiteAssociated
fvPeerContext
fvRemoteId
fvRemoteId
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ctx-L2 Total Objects shown: 1 # fv.Ctx name : L2 annotation :
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ctx-L2
Total Objects shown: 1
# fv.Ctx
name
: L2
annotation
: orchestrator:msc
bdEnforcedEnable : no
childAction
:
descr
:
dn
extMngdBy
knwMcastAct
lcOwn
: uni/tn-RD-L2/ctx-L2
: msc
: permit
: local
modTs
: 2018-08-27T01:42:04.727+01:00
monPolDn
: uni/tn-common/monepg-default
pcEnfDir
: ingress
pcEnfDirUpdated : yes
pcEnfPref
: enforced
pcTag
: 32770
rn
: ctx-L2
scope
: 2457600
seg
: 2162688

Example for VRF Site1 APIC view

On site 1 APIC Under the Dn of the VRF tn-RD-

L2/ctx-L2

We have fvRemoteId for site-2 Containing the remote vrf Vnid

(called for whaterver reason remote PcTag

fvCtx fvSiteAssociated fvRemoteId Site-Id Remote-Id
fvCtx
fvSiteAssociated
fvRemoteId
Site-Id
Remote-Id

This is enough to push through Object model translation on spine

From Local VNID 2457600 to remote VNID site2

2162688

This will be pushed to APIC in site1 here

bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ctx-L2/stAsc/site-2 Total Objects shown: 1

# fv.RemoteId

siteId

:

2

childAction

:

descr

:

dn

: uni/tn-RD-L2/ctx-L2/stAsc/site-2

lcOwn

: local

modTs

: 2018-05-03T03:14:40.895+00:00

monPolDn

: uni/tn-common/monepg-default

name

:

nameAlias

:

ownerKey

:

ownerTag

:

Similar construct will exist on APIC site2 to push

reverse translation on site2 spine

© 2019 Cisco and/or its affiliates. All rights reserved.

remoteCtxPcTag : 32770

remotePcTag

: 2162688

- Actually this is remote vrf PcTag

rn

: site-2

status

:

uid

: 15374

Logical BD site 1

Similar construct for BD
Similar construct for BD
fvBD fvSiteAssociated fvRemoteId
fvBD
fvSiteAssociated
fvRemoteId
Site 2 BD
Site 2 BD

dsol-aci36-apic1# moquery -c fvBD -f 'fv.BD.seg

="15073234"' | egrep "dn|scope|seg"

n

: uni/tn-RD-L2/BD-Web

cope

: 2162688

eg

: 15073234

© 2019 Cisco and/or its affiliates. All rights reserved.

bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web/stAsc/ Total Objects shown: 1 bdsol-aci35-apic1# moquery -d
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web/stAsc/
Total Objects shown: 1
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web | egrep
"annot|dn|seg|scope"
annotation
: orchestrator:msc
dn
: uni/tn-RD-L2/BD-Web
scope
: 2457600
seg
: 15204288
Translation of BD vnid to spine site 1 From 15204288 to 150723234
Translation of BD vnid to spine site 1
From 15204288 to 150723234
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web/stAsc/site-2 Total Objects shown: 1 # fv.RemoteId siteId : 2
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web/stAsc/site-2
Total Objects shown: 1
# fv.RemoteId
siteId
:
2
childAction
:
descr
:
dn
lcOwn
modTs
monPolDn
: uni/tn-RD-L2/BD-Web/stAsc/site-2
: local
: 2018-05-03T03:14:40.895+00:00
: uni/tn-common/monepg-default
name
:
nameAlias
:
ownerKey
:
ownerTag
:
remoteCtxPcTag : any
remotePcTag
: 15073234
Actually remote BD VNID
rn
: site-2
status
:
uid
: 15374

bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ap-App/epg-Web/stAsc/

Total Objects shown: 1

Logical AEPg site 1

In case of EPG we have the pcTag translation

# fv.SiteAssociated childAction :

descr

:

dn

: uni/tn-RD-L2/ap-App/epg-Web/stAsc/stAsc

lcOwn

: local

modTs

: 2018-05-03T03:14:39.572+00:00

monPolDn

: uni/tn-common/monepg-default

name

: msc-local

nameAlias

:

ownerKey

:

ownerTag

:

rn

: stAsc

siteId

:

1

status

:

uid

: 15374

fvAEPg fvSiteAssociated fvPeerContext fvRemoteId
fvAEPg
fvSiteAssociated
fvPeerContext
fvRemoteId
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ap-App/epg- Web/stAsc/site-2 Total Objects shown: 1 # fv.RemoteId siteId :
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ap-App/epg-
Web/stAsc/site-2
Total Objects shown: 1
# fv.RemoteId
siteId
:
2
childAction
:
descr
:
dn
lcOwn
modTs
monPolDn
name
: uni/tn-RD-L2/ap-App/epg-Web/stAsc/site-2
: local
: 2018-05-03T03:14:40.895+00:00
: uni/tn-common/monepg-default
:
nameAlias
:
ownerKey
:
ownerTag
:
remoteCtxPcTag : any
remotePcTag
: 49155
rn
: site-2
status
:
uid
: 15374
Site 2 EPG
Site 2 EPG
bdsol-aci36-apic1# moquery -c fvAEPg -f 'fv.AEPg.pcTag =="49155"' | egrep "dn|scope|pcTag" dn
bdsol-aci36-apic1# moquery -c fvAEPg -f 'fv.AEPg.pcTag
=="49155"' | egrep "dn|scope|pcTag"
dn
pcTag
scope
: uni/tn-RD-L2/ap-App/epg-Web
: 49155
: 2162688
© 2019 Cisco and/or its affiliates. All rights reserved.

Process involved for vnidmap/sclass/site-etep

on Spine only

Dcimgr This run on sup. Listen for events from PE and send it Dcimc on
Dcimgr
This run on
sup. Listen
for events
from PE and
send it Dcimc
on Lc
This runs on sup-lc. Call Dcimc sdk hal api to program site- etep, vnidmap sclassmap
This runs on
sup-lc. Call
Dcimc
sdk hal api to
program site-
etep,
vnidmap
sclassmap
Sdk Programs the Hal hardware
Sdk
Programs the
Hal
hardware
DCImgr is the NXOS process On spine that will program the Translation based on the
DCImgr is the NXOS process
On spine that will program the
Translation based on the objectmodel
This is what you need to check first (easiest to check ☺)

© 2019 Cisco and/or its affiliates. All rights reserved.

Dcimgr/dcimc/sdkTraces for sclass/vnid map

Dcimgr (on sup)

show dcimgr internal event-history events

And log file :

pod35-spine1# ls -al /var/sysmgr/tmp_logs/dcimgr.log -rw-rw-rw- 1 root root 3162338 May 2 16:37 /var/sysmgr/tmp_logs/dcimgr.log

pod35-spine1#

HAL CLI :

module-2# show platform internal hal objects dci ?

all

Dump All HAL objects

remotesite

Remotesite or wan instance

remotesiteetep Unicast etep that belongs to this remotesite

remotevrfvnid

Vrf for remotesite object

sclassmap

Sclass mapping for remotesite vrf

vnidmap

Vnid mapping for remotesite object

© 2019 Cisco and/or its affiliates. All rights reserved.

DCI mgr xlate

Vnid translate (vrf and bd)
Vnid translate (vrf and bd)
pod36-spine1# show dcimgr repo vnid-maps detail -------------------------------------------------------------- Remote |
pod36-spine1# show dcimgr repo vnid-maps detail
--------------------------------------------------------------
Remote
|
Local
site Vrf
Bd
|
Vrf
Bd
Rel-state
--------------------------------------------------------------
1
2981888
|
2293760
[formed]
0x2d8000
| 0x230000
--------------------------------------------------------------
1
2981888 16678778 |
2293760 16154554 [formed]
0x2d8000 0xfe7f7a | 0x230000 0xf67fba
--------------------------------------------------------------
1
3014656
|
2457600
[formed]
0x2e0000
| 0x258000

© 2019 Cisco and/or its affiliates. All rights reserved.

pcTag (sclass) translate
pcTag (sclass) translate
pod36-spine1# show dcimgr repo sclass-maps detail ---------------------------------------------------------- Remote |
pod36-spine1# show dcimgr repo sclass-maps detail
----------------------------------------------------------
Remote
|
Local
site Vrf
PcTag
|
Vrf
PcTag
Rel-state
----------------------------------------------------------
1 2981888
49153
| 2293760
49153
[formed]
0x2d8000 0xc001 | 0x230000 0xc001
-------------------------------------------------------------------
1 2981888
49154
| 2293760
49155
[formed]
0x2d8000 0xc002 | 0x230000 0xc003
-------------------------------------------------------------------
1 2981888
16387
| 2293760
16386
[formed]
0x2d8000 0x4003 | 0x230000 0x4002
-------------------------------------------------------------------
1 3014656
49153
| 2457600
49153
[formed]
0x2e0000 0xc001 | 0x258000 0xc001
-------------------------------------------------------------------
1 3014656
16387
| 2457600
32772
[formed]
0x2e0000 0x4003 | 0x258000 0x8004

Unicast Control Plane

BGP route exchange detail

© 2019 Cisco and/or its affiliates. All rights reserved.

ACI Multi-Site

Inter-Site MP-BGP EVPN Control Plane

MP-BGP EVPN used to communicate Endpoint (EP) information across Sites

MP-iBGP or MP-EBGP peering supported across sites

Remote host route entries (EVPN Type-2) are associated to the remote site Anycast DP-ETEP address

Automatic filtering of endpoint

information across Sites

Host routes are exchanged only if there is

a cross-site contract requiring

communication between endpoints

© 2019 Cisco and/or its affiliates. All rights reserved.

S3-S4 Table

S5-S8 Table

EP1 Leaf 1 EP2 Leaf 4 MP-BGP EVPN EP2 DP-ETEP B EP1 DP-ETEP A EP3
EP1
Leaf 1
EP2
Leaf 4
MP-BGP EVPN
EP2
DP-ETEP B
EP1
DP-ETEP A
EP3
Leaf 4
EP4
Leaf 6
IP
Network
DP-ETEP A
DP-ETEP B
S1
S2
S3
S4
S5
S6
S7
S8
COOP
COOP
ACI
Multi-Zone
EP2
EP4
EP1
EP3
Site 1
Site 2
Define and push inter-site policy EP1 EP2 C EPG EPG
Define and push inter-site policy
EP1
EP2
C
EPG
EPG

MP-BGP in ACI summary

So for we use BGP in infra (vrf overlay-1) for many reasons:

Intra Pod : VPNv4 AF for L3 out routes only

Multipod : VPNv4 AF for L3 out routes across Pod and l2vpn evpn type2 for End point synchro across Pod GOLF : l2vpn evpn (type2 and type5) for L3 subnet between GOLF and ACI (routes in VRF)

Multisite : L2vpn evpn for end point synchro across site

© 2019 Cisco and/or its affiliates. All rights reserved.

Route-Target and Route Distinguisher in ACI

bdsol-aci32-leaf1# show bgp process vrf RD-BGP:RD • For VPNv4 : VRF RD Export RT list:
bdsol-aci32-leaf1# show bgp process vrf RD-BGP:RD
• For VPNv4 :
VRF RD
Export RT list:
: 10.0.88.95:6
• RD is typically the PTEP of the origination:vrf_id
132:2654211
Import RT list:
• RT is typically the BGP_ASN:VRF_VNID
132:2654211
• In single Pod, or multipod, every switches have the same RT and RD in
• In single Pod, or multipod, every switches have the same
RT and RD in the VRF so the route-target import/export
are done automatically
• And can be seen in show bgp process vrf XXX

In multisite, RT will be different across site for the same vrf, so we must configure cross route-target import/export for correct BGP path exchange to happen

This is done using bgp EVI (EVPN instance) (show bgp internal evi XXXX) . Similar to pcTag and VNID translation this is also pushed using object model

© 2019 Cisco and/or its affiliates. All rights reserved.

Note that here we need route-target per BD (not per VRF)

BGP VNI

Route Exchange issues can be seen either in the source or on the remote site.

Check if the BGP MOs are created for VNIs/RTs and RDs correctly. These MOs are created only on spines in every site. These MOs are created

when the VRF/BD/EPGs are stretched or the contracts are created at EPG level

Following shows mapping of BGP VNIDs and what routes are requested from COOP and why they are used:

BGP Route Target for a BD

Site 2 spine Import RT Site 1 spine Import RT # bgp.RttEntry # bgp.RttEntry rtt
Site 2 spine Import RT
Site 1 spine Import RT
# bgp.RttEntry
# bgp.RttEntry
rtt
: route-target:as2-nn4:136:49676223
rtt
: route-target:as2-nn4:135:33128354
childAction :
childAction :
dn
: sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan-
dn
: sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan-2457600]-
3014656]-bd-[vxlan-16351138]-epg-[unknown]/rtp-import/ent-route-
bd-[vxlan-16121791]-epg-[unknown]/rtp-import/ent-route-target:as2-
target:as2-nn4:136:49676223
nn4:135:33128354
lcOwn
: local
lcOwn
: local
modTs
modTs
rn
rn
status
: 2018-04-11T04:28:21.600+00:00
: ent-route-target:as2-nn4:136:49676223
:
status
: 2018-04-11T04:28:16.142+00:00
: ent-route-target:as2-nn4:135:33128354
:
Site 1 spine export RT
Site 2 spine export RT
# bgp.RttEntry
# bgp.RttEntry
rtt
: route-target:as2-nn4:135:33128354
rtt
: route-target:as2-nn4:136:49676223
childAction :
childAction :
dn
: sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan-
dn
: sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan-2457600]-
3014656]-bd-[vxlan-16351138]-epg-[unknown]/rtp-export/ent-route-
bd-[vxlan-16121791]-epg-[unknown]/rtp-export/ent-route-target:as2-
target:as2-nn4:135:33128354
nn4:136:49676223
lcOwn
: local
lcOwn
modTs
: 2018-04-11T04:28:21.600+00:00
modTs
rn
rn
: local
: 2018-04-11T04:28:16.142+00:00
: ent-route-target:as2-nn4:136:49676223
status
: ent-route-target:as2-nn4:135:33128354
:
status
:

© 2019 Cisco and/or its affiliates. All rights reserved.

BGP EVI check (NXOS) BD on site 1

Use show bgp internal evi xx to verify RD and RT exp/import (where xx is
Use show bgp internal evi xx to verify
RD and RT exp/import (where xx is BD VNID)
(kind of similar to show bgp process for GOLF)
pod35-spine1# show bgp internal evi 16351138 ************************************************* BGP L2VPN/EVPN RD
pod35-spine1# show bgp internal evi 16351138
*************************************************
BGP L2VPN/EVPN RD Information for 1:33128354
L2VNI ID
#Prefixes Local/BRIB
: 16351138 (vni_16351138)
:
2
/ 2
#Paths L3VPN->EVPN/EVPN->L3VPN : 0 / 0
*************************************************
==============================================
BGP Configured VNI Information:
VNI ID (Index)
RD
: 16351138 (0)
: 1:33128354
Export RTs
:
1
Export RT cfg list: 135:33128354(refcount:1
Import RTs
:
1
Import RT cfg list: 136:49676223(refcount:1
Topo Id
VTEP IP
VTEP VPC IP
Enabled
Delete Pending
: 16351138
: 0.0.0.0
: 0.0.0.0
: Yes
: No
RD/Import RT/Export RT
: Yes/Yes/Yes
Type
:
3
Usage
L2 stretch enabled
VRF Vnid
Refcount
Encap
:
2
:
1
: 3014656
: 00000003
: VxLAN
==============================================
++++++++++++++++++++++++++++++++++++++++++
© 2019 Cisco and/or its affiliates. All rights reserved.
Note the EVI

++++++++++++++++++++++++++++++++++++++++++ BGP VNI Information for vni_16351138

L2VNI ID

: 16351138 (vni_16351138)

RD

: 1:33128354

VRF Vnid

: 3014656

Prefixes (local/total)

: 2/2

VNID registered with COOP

: Yes

Enabled

: Yes

Delete pending

:

0

Stale

: No

Import pending

:

0

Import in progress

:

0

Encap

: VxLAN

Topo Id

: 16351138

VTEP IP

: 0.0.0.0

VTEP VPC IP

: 0.0.0.0

Active Export RTs

:

1

Active Export RT list

: 135:33128354

Config Export RTs

:

1

Export RT cfg list: 135:33128354(refcount:1

Export RT chg/chg-pending Active Import RTs Active Import RT list

Config Import RTs

: 0/0

:

: 136:49676223

:

1

1

Import RT cfg list: 136:49676223(refcount:1

Import RT chg/chg-pending

: 0/0

IMET Reg/Unreg from L2RIB

: 1/0

MAC Reg/Unreg from L2RIB

: 1/0

MAC IP Reg/Unreg from L2RIB : 1/0 IP-only Reg/Unreg from L2RIB : 0/0

SMAD Reg/Unreg from L2RIB

: 1/0

IMET Add/Del from L2RIB

: 0/0

MAC Add/Del from L2RIB

: 3/2

MAC IP Add/Del from L2RIB

: 3/2

SMAD Add/Del from L2RIB

: 0/0

IMET Dnld/Wdraw to L2RIB

: 0/0

IMET Dnld/Wdraw to L2RIB failures : 0/0

MAC Dnld/Wdraw to L2RIB

MAC Dnld/Wdraw to L2RIB failures : 0/0

SMAD Dnld/Wdraw to L2RIB

SMAD Dnld/Wdraw to L2RIB failures : 0/0

: 0/0

: 0/0

to L2RIB SMAD Dnld/Wdraw to L2RIB failures : 0/0 : 0/0 : 0/0 MAC-IP/SMAD Msite-RD routes

MAC-IP/SMAD Msite-RD routes : 2

number if the BD VNID we are looking for

MAC-IP WAN-RD routes

MAC-IP network host routes

:

:

0

0

Type

:

3

Unicast forwarding across site

© 2019 Cisco and/or its affiliates. All rights reserved.

Overview

1. Unicast TX proxy/(local to remote site)

Leaf has not learned the remote site ep. Leaf sends the traffic to local spine proxy. Local spine looks up the route. The route for remote site Ep is programmed with next hop of remote site’s ETEP. Dipo is re-written with remote site ETEP. Sipo is re-written with local site ETEP

2. Unicast TX (local to remote site)

Leaf has learned the remote site ep against remote site ETEP. Leaf sends the traffic to remote site ETEP. Local site

spine will intercept this packet and re-write the sipo with Local site ETEP

3. Unicast RX (remote to local site)

Incoming traffic destined to the local site’s unicast ETEP goes through vnid and sclass translations. The receiving spine looks up the route for destination ep and sends the traffic to correct leaf.

1. Proxy – Spine COOP lookup 3. In all case Rx on spine does vnid/sclass
1.
Proxy – Spine COOP lookup
3. In all case Rx on spine does vnid/sclass translation
IP Network
SIP and DIP outer rewritten
1
3
MP-BGP - EVPN
2
Network SIP and DIP outer rewritten 1 3 MP-BGP - EVPN 2 2. Known EP –

2. Known EP EPM lookup on leaf

© 2019 Cisco and/or its affiliates. All rights reserved.

DIP outer set on lef SIP outer Rewritten when passing by ingress spine

LAB Stretched VRF

Pod35 Pod36 Aci-35-interconnect BD GW 1/51-52 172.16.1.254/24 1/49-50 172.16.3.254/24 Route 2/5-6 2/5-6
Pod35
Pod36
Aci-35-interconnect
BD GW
1/51-52
172.16.1.254/24
1/49-50
172.16.3.254/24
Route
2/5-6
2/5-6
172.16.[1-4].0/24 aci35-spine1
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1
Web-EPG1 172.16.1.1/24 vlan-101
App-EPG1 172.16.3.2/24 VM
BD GW 172.16.2.254/24 172.16.4.254/24 Route 172.16.[1-4].0/24
BD GW
172.16.2.254/24
172.16.4.254/24
Route
172.16.[1-4].0/24

aci36-spine1

aci36-leaf1

Web-EPG2 172.16.2.2/24 VM App-EPG2 172.16.4.1/24 vlan-104
Web-EPG2 172.16.2.2/24 VM
App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

ACI Multi-Site

and/or its affiliates. All rights reserved. ACI Multi-Site APIC Site 1 APIC Site 2 Tenant IPA
and/or its affiliates. All rights reserved. ACI Multi-Site APIC Site 1 APIC Site 2 Tenant IPA
APIC Site 1 APIC Site 2 Tenant IPA VRF DC:DC1 BD2 172.16.2.54/24 BD1 172.16.1.54/24 Web-EPG2
APIC Site 1
APIC Site 2
Tenant IPA
VRF DC:DC1
BD2 172.16.2.54/24
BD1 172.16.1.54/24
Web-EPG2
Web-EPG1
C1
C1
C2
C2
BD4 172.16.4.254/24
BD3 172.16.3.254/24
App-EPG1
App-EPG2
Test : 172.16.3.2 to 172.16.2.2
Test :
172.16.3.2 to 172.16.2.2

Control Plane EP in site 2 to COOP in site 1

BGP EVPN type2 For 172.16.2.2 Pod35 Pod36 BD GW Aci-35-interconnect 172.16.2.254/24 BD GW 172.16.4.254/24 1/51-52
BGP EVPN type2
For 172.16.2.2
Pod35
Pod36
BD GW
Aci-35-interconnect
172.16.2.254/24
BD GW
172.16.4.254/24
1/51-52
172.16.1.254/24
1/49-50
Route
172.16.3.254/24
172.16.[1-4].0/24
Route
2/5-6
2/5-6
172.16.[1-4].0/24 aci35-spine1
aci36-spine1
2/1-2
2/1-2
Epm local Learn
1/49-50
1/49-50
aci35-leaf1
aci36-leaf1
Web-EPG1 172.16.1.1/24 vlan-101
Web-EPG2 172.16.2.2/24 VM
App-EPG1 172.16.3.2/24 VM
App-EPG2 172.16.4.1/24 vlan-104

BGP to COOP 172.16.2.2 to DP-ETEP Site 2

Control plane to reach 172.16.2.2

Note that dataplane to 172.16.2.2 (site1 to site2 is translated in site2 spine

Leaf to spine COOP
Leaf to spine COOP

© 2019 Cisco and/or its affiliates. All rights reserved.

Control plane EP in Site 2 Local COOP site 2 Publisher id is the local
Control plane EP in Site 2
Local COOP site 2
Publisher id is the local leaf in site2
pod36-spine1# show coop internal
info ip-db key 2457600 172.16.2.2
Local EPM
pod36-leaf1# show system internal epm endpoint ip 172.16.2.2
IP address : 172.16.2.2
Vrf : 2457600
Flags : 0
MAC : 0050.56b1.4403 ::: Num IPs : 1
IP# 0 : 172.16.2.2 ::: IP# 0 flags :
EP bd vnid : 16220082
EP mac : 00:50:56:B1:44:03
Publisher Id : 10.1.48.64
Vlan id : 21 ::: Vlan vnid : 8194 ::: VRF name : DC:DC1
BD vnid : 16220082 ::: VRF vnid : 2457600
Phy If : 0x1a001000 ::: Tunnel If : 0
Interface : Ethernet1/2
Flags : 0x80004c04 ::: sclass : 32771 ::: Ref count : 5
EP Create Timestamp : 04/19/2018 07:03:23.999543
EP Update Timestamp : 05/02/2018 02:33:29.507208
EP Flags : local|IP|MAC|sclass|timer|
::::
Record timestamp : 05 02 2018 02:29:12 339899902
Publish timestamp : 05 02 2018 02:29:12 340145880
Seq No: 0
Remote publish timestamp: 01 01 1970 00:00:00 0
URIB Tunnel Info
Num tunnels : 1
Tunnel address : 10.1.48.64
Tunnel ref count : 1
Remote COOP entry site 1
Extract BGP table site 2
Publisher id is the spine DP TEP in site 2
pod36-spine1# show bgp l2vpn evpn vrf overlay-1 | egrep "Route Dis|172.16.2.2\]”
Route Distinguisher: 1:49774514
(L2VNI 16220082)
pod35-spine1# show coop internal info ip-db | egrep -A
15 -B 1 "172.16.2.2$"
------------------------------
*>l[2]:[0]:[16220082]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272
Route Distinguisher: 10.10.35.102:136
(L2VNI 1)
*>l[2]:[0]:[16220082]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272
Extract BGP table site 1
IP address : 172.16.2.2
Vrf : 3014656
Flags : 0x4
EP bd vnid : 15925206
EP mac : 00:50:56:B1:44:03
Publisher Id : 10.10.35.102
Record timestamp : 01 01 1970 00:00:00 0
pod35-spine1# show bgp l2vpn evpn vrf overlay-1 | egrep "Route Dis|172.16.2.2\]"
Publish timestamp : 01 01 1970 00:00:00 0
Route Distinguisher: 1:49774514
*>e[2]:[0]:[0]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272
Route Distinguisher: 1:32702422
(L2VNI 15925206)
*>e[2]:[0]:[15925206]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272
© 2019 Cisco and/or its affiliates. All rights reserved.
Route Distinguisher: 10.10.35.101:135
(L2VNI 1)
*>e[2]:[0]:[15925206]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272
Seq No: 0
Remote publish timestamp: 04 24 2018 05:05:34 611613733
URIB Tunnel Info
Num tunnels : 1
Tunnel address : 10.10.35.102
Tunnel ref count : 1

DCI Mgr on spine pod 36 (site 2)

Remote Site DP-ETEP and Mcast ETEP(dcimgr and Object model)
Remote Site
DP-ETEP and
Mcast ETEP(dcimgr and Object model)
pod36-spine1# show dcimgr repo eteps Remote site=1 : Rem Etep=10.10.35.101/32, is_ucast=yes Rem Etep=10.10.35.121/32,
pod36-spine1# show dcimgr repo eteps
Remote site=1 :
Rem Etep=10.10.35.101/32, is_ucast=yes
Rem Etep=10.10.35.121/32, is_ucast=no
pod36-spine1#

© 2019 Cisco and/or its affiliates. All rights reserved.

pod36-spine1# moquery -c dciAnycastExtn Total Objects shown: 2 # dci.AnycastExtn etep : 10.10.35.101/32 childAction :
pod36-spine1# moquery -c dciAnycastExtn
Total Objects shown: 2
# dci.AnycastExtn
etep
: 10.10.35.101/32
childAction :
dn
is_ucast
lcOwn
modTs
rn
: sys/inst-overlay-1/remoteSite-1/anycastExtn-[10.10.35.101/32]
: yes
: local
: 2018-03-30T05:50:34.562+00:00
: anycastExtn-[10.10.35.101/32]
status
:
# dci.AnycastExtn
etep
: 10.10.35.121/32
childAction :
dn
: sys/inst-overlay-1/remoteSite-1/anycastExtn-[10.10.35.121/32]
is_ucast
: no
lcOwn
: local
modTs
rn
: 2018-03-30T05:50:34.562+00:00
: anycastExtn-[10.10.35.121/32]
status
:

DCI Mgr on spine pod 36 (site 2) VNID MAP

DCI mgr vnid map pod35-spine1# show dcimgr repo vnid-maps --------------------------------------------------------------
DCI mgr vnid map
pod35-spine1# show dcimgr repo vnid-maps
--------------------------------------------------------------
Remote
|
Local
site Vrf
Bd
|
Vrf
Bd
Rel-state
--------------------------------------------------------------
1
3014656
|
2457600
[formed]
1
3014656 16056263 |
2457600 16121790 [formed]
1
3014656 16351138 |
2457600 16121791 [formed]
1
3014656 15925206 | 2457600 16220082 [formed]
1
3014656 16056262 |
2457600 15794151 [formed]

Aci-35-interconnect

1/51-52 1/49-50 2/5-6 2/5-6 aci35-spine1 2/1-2 2/1-2 1/49-50 1/49-50 aci35-leaf1
1/51-52
1/49-50
2/5-6
2/5-6
aci35-spine1
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1

aci36-spine1

aci36-leaf1

Translation for packet to 172.16.2.2

aci36-leaf1 Translation for packet to 172.16.2.2 Web-EPG2 172.16.2.2/24 VM App-EPG2 172.16.4.1/24 vlan- 104
Web-EPG2 172.16.2.2/24 VM App-EPG2 172.16.4.1/24 vlan- 104
Web-EPG2 172.16.2.2/24 VM
App-EPG2 172.16.4.1/24 vlan-
104
Web-EPG1 172.16.1.1/24 vlan- App-EPG1 172.16.3.2/24 VM 101 Here packet received on site2 From site1 (l2
Web-EPG1 172.16.1.1/24 vlan-
App-EPG1 172.16.3.2/24 VM
101
Here packet received on site2 From site1
(l2 case)with BD VNID 159250206 will be Xlated to 16220082
(L3 case) with VRF VNID 3014656 will be xlated to 2457600
© 2019 Cisco and/or its affiliates. All rights reserved.

DCI Mgr on spine pod 36 (site 2) SCLASS MAP

DCI mgr vnid map pod36-spine1# show dcimgr repo sclass-maps ----------------------------------------------------------
DCI mgr vnid map
pod36-spine1# show dcimgr repo sclass-maps
----------------------------------------------------------
Remote
|
Local
site Vrf
PcTag
|
Vrf
PcTag
Rel-state
----------------------------------------------------------
1
3014656
49153
|
2457600
49153
[formed]
1
3014656
16387
|
2457600
32772
[formed]
1
3014656
16388
|
2457600
16387
[formed]
1
3014656
32770
|
2457600
16390
[formed]
1
3014656
32772
|
2457600
32771
[formed]

© 2019 Cisco and/or its affiliates. All rights reserved.

ivxlan header review

© 2019 Cisco and/or its affiliates. All rights reserved.
© 2019 Cisco and/or its affiliates. All rights reserved.
Note in Outer L4 header you can Get : VNID (BD or VRF) Sclass (src
Note in Outer L4 header you can
Get :
VNID (BD or VRF)
Sclass (src sclass) as part of
Nounce field (last 4 nibble):
Ex :
hom_elam_in_l4v_tn.tn_nonce_info: 0x188002
Sclass of Rx frame is 0x8002

Data path known EP Site 1 to Site 2 (Known

unicast on ingress leaf)

Outer : 10.10.35.101 (site1 DP-ETEP to 10.10.35.102 (site2 dp-etep) Outer L4 : site1 vrf VNID
Outer : 10.10.35.101 (site1 DP-ETEP to 10.10.35.102 (site2 dp-etep)
Outer L4 : site1 vrf VNID –
Sclass App-EPG1
Pod36
Inner : 172.16.3.2 to 172.16.2.2
Aci-35-interconnect

Pod35

1/51-52 1/49-50 Spine just RW Outer Sip 2/5-6 2/5-6 Outer Dst IP is my DP-etep
1/51-52
1/49-50
Spine just RW Outer Sip
2/5-6
2/5-6
Outer Dst IP is my DP-etep  DCI-rx
aci35-spine1
aci36-spine1
Translate Vnid and sclass
Send to pod36-leaf1 tep per coop
2/1-2
2/1-2
Outer : pod35-leaf1 PTEP to 10.10.35.102 (site2 dp-etep)
Outer L4 : site1 vrf VNID – Sclas App-EPG1
Inner : 172.16.3.2 to 172.16.2.2
Outer : 10.10.35.101 (site1 DP-ETEP) to pod36-leaf1 PTEP
Outer L4 : site2 vrf VNID – Sclass App-EPG1 Translated
Inner : 172.16.3.2 to 172.16.2.2
Epm entry for Dest
1/49-50
1/49-50
EPM learning for 172.16.3.2 to tunnel to site1 DP-ETEP
going to tunnel to
aci36-spine aci35-leaf1 DP-ETEP
aci36-leaf1
to tunnel to aci36-spine aci35-leaf1 DP-ETEP aci36-leaf1 Web-EPG1 172.16.1.1/24 vlan-101 App-EPG1 172.16.3.2/24 VM
to tunnel to aci36-spine aci35-leaf1 DP-ETEP aci36-leaf1 Web-EPG1 172.16.1.1/24 vlan-101 App-EPG1 172.16.3.2/24 VM
to tunnel to aci36-spine aci35-leaf1 DP-ETEP aci36-leaf1 Web-EPG1 172.16.1.1/24 vlan-101 App-EPG1 172.16.3.2/24 VM
Web-EPG1 172.16.1.1/24 vlan-101 App-EPG1 172.16.3.2/24 VM
Web-EPG1 172.16.1.1/24 vlan-101
App-EPG1 172.16.3.2/24 VM
Web-EPG2 172.16.2.2/24 VM App-EPG2 172.16.4.1/24 vlan-104
Web-EPG2 172.16.2.2/24 VM
App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

Ingress Leaf Known EP

Pod35

Pod36

Aci-35-interconnect

1/51-52 1/49-50 2/5-6 2/5-6 2/1-2 2/1-2 1/49-50 1/49-50
1/51-52
1/49-50
2/5-6
2/5-6
2/1-2
2/1-2
1/49-50
1/49-50
pod35-leaf1# show system internal epm endpoint ip 172.16.2.2 MAC : 0000.0000.0000 ::: Num IPs :
pod35-leaf1# show system internal epm endpoint ip 172.16.2.2
MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 172.16.2.2 ::: IP# 0 flags :
Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : DC:DC1
BD vnid : 0 ::: VRF vnid : 3014656
Phy If : 0 ::: Tunnel If : 0x18010007
Interface : Tunnel7
Flags : 0x80004400 ::: sclass : 32772 ::: Ref count : 3
EP Create Timestamp : 04/24/2018 05:05:32.831665
EP Update Timestamp : 04/25/2018 04:58:50.374323
EP Flags : IP|sclass|timer|
::::

aci35-spine1

aci36-spine1

aci36-leaf1
aci36-leaf1

aci35-leaf1

:::: aci35-spine1 aci36-spine1 aci36-leaf1 aci35-leaf1 pod35-leaf1# show interface tunnel 7 Tunnel7 is up MTU
pod35-leaf1# show interface tunnel 7 Tunnel7 is up MTU 9000 bytes, BW 0 Kbit Transport
pod35-leaf1# show interface tunnel 7
Tunnel7 is up
MTU 9000 bytes, BW 0 Kbit
Transport protocol is in VRF "overlay-1"
Tunnel protocol/transport is ivxlan
Tunnel source 10.0.112.64/32 (lo0)
Tunnel destination 10.10.35.102/32
Last clearing of "show interface" counters never
Tx
0
packets output, 1 minute output rate 0 packets/sec
Rx
0
packets input, 1 minute input rate 0 packets/sec

© 2019 Cisco and/or its affiliates. All rights reserved.

ELAM Ingress LC Spine Site 1 EP known

module-2# debug platform internal roc elam asic 0 module-2(DBG-elam)# trigger init in-select 14 out-select 1
module-2# debug platform internal roc elam asic 0
module-2(DBG-elam)# trigger init in-select 14 out-select 1
module-2(DBG-elam-insel15)# set inner ipv4 src_ip 172.16.3.2 dst_ip 172.16.2.2
#########################HOMEWOOD ELAM REPORT START#########################
Pod35
Pod36
Dumping report for asic type 8 inst 0 slice 0 a_to_d 1 insel 15 outsel 1
LUA captured data with :
Aci-35-interconnect
1/51-52
1/49-50
2/5-6
2/5-6
aci35-spine1
aci36-spine1
SRCID: 20
*** Parsed Outer l2 vector
hom_elam_in_l2v_da_sa_qtag0.qtag0_vlan: 0x2
*** Parsed Outer l3 vector
- 10.0.35.102 (Dp-ETEP site2)
- 10.0.112.64 (leaf1 pod35 PTEP)
2/1-2
2/1-2
hom_elam_in_l3v_ipv4.da: 0xA0A2366
hom_elam_in_l3v_ipv4.sa: 0xA007040
*** Parsed Outer l4 vector
hom_elam_in_l4v_tn.tn_seg_id: 0x2E0000
- 3014656
hom_elam_in_l4v_tn.tn_nonce_info: 0x8002
1/49-50
1/49-50
aci35-leaf1
aci36-leaf1
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_rx: 0x0
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_tx: 0x0
module-2(DBG-elam-insel15)# show platform internal hal l2 port gpd | egrep "Eth2/1|==|IfId|Uc|Xla"
module-2(DBG-elam-insel15)# show platform internal hal l2 port gpd | egrep "Eth2/1|==|IfId|Uc|Xla"
============================================================================================================================= ===============
Uc
Uc
|
Reprogram
I PC
Pc
L
|
R I R D
R
|
U U X | L Xla Ovx N
NI Vif
RwV
Ing
Egr
| Rep |
| V R | PROF H
IfId
Ifname
P Cfg
MbrID As AP Sl Sp Ss Ovec S | P P P S P Sp Sp C M L | 3 Idx Idx L3 L3 Tid
Tid
Lbl
Lbl
|
S
V | ID
I
============================================================================================================================= ================
1a080000 Eth2/1
0 9a
28
0
11 0
10 20 20
1
0
0
0
0
0 0
0
0
0 0
1 1
1
1
1
D -f3
D-61
100
0
0 0
4
0
pod35-spine1# show lldp neighbors | egrep "Eth2/1"
pod35-leaf1
Eth2/1
120
BR
Eth1/49
pod35-spine1#

© 2019 Cisco and/or its affiliates. All rights reserved.

ELAM Ingress LC Spine Site 2 Proxy

module-2# debug platform internal roc elam asic 0 module-2(DBG-elam)# trigger reset module-2(DBG-elam)# trigger init
module-2# debug platform internal roc elam asic 0
module-2(DBG-elam)# trigger reset
module-2(DBG-elam)# trigger init in-select 15 out-select 1
module-2(DBG-elam-insel15)# set inner ipv4 src_ip 172.16.3.2 dst_ip 172.16.2.2
aci36-spine1
Pod35 Pod36 Aci-35-interconnect 1/51-52 1/49-50 2/5-6 2/5-6 aci35-spine1 2/1-2 2/1-2 1/49-50 1/49-50
Pod35
Pod36
Aci-35-interconnect
1/51-52
1/49-50
2/5-6
2/5-6
aci35-spine1
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1
#########################HOMEWOOD ELAM REPORT START######################### Dumping report for asic type 8 inst 0 slice
#########################HOMEWOOD ELAM REPORT START#########################
Dumping report for asic type 8 inst 0 slice 0 a_to_d 1 insel 15 outsel 1
LUA captured data with :
SRCID: 0
*** Parsed Outer l2 vector
hom_elam_in_l2v_da_sa_qtag0.qtag0_vlan: 0x4
*** Parsed Outer l3 vector
hom_elam_in_l3v_ipv6_da_only.da: 0x000000000000000000A0A2366 – 10.10.35.102 (site2 – DP-ETEP)
hom_elam_in_l3v_ipv6_da_only.sa: 0xA0A2365
*** Parsed Outer l4 vector
hom_elam_in_l4v_tn.tn_nonce_info: 0x188002
hom_elam_in_l4v_tn.tn_seg_id: 0x2E0000
- 10.10.35.101 (site1 – DP-ETEP)
- Rx sclass is 0x8002 = 16387
- 3014656 (vnid before rewrite)
hom_elam_out_sidebnd_no_spare_vec.ovector_idx: 0x78
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_rx: 0x1
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_tx: 0x0
(useless internal port to FC)
hom_lurw_vec.info.ifabric_spine.vnid: 0x258000
hom_lurw_vec.info.ifabric_spine.sclass: 0x4006
- Vnid after rewrite = 2457600
- rewritten Sclass is 16390
======== lux_fwd_mode = 0x09516040
LUX_FWD_MODE: ISPINE_LC
LUX_FWD_MODE: ISPINE_DCI
bit is set
bit is set
ingress LC

aci36-leaf1

ISPINE_DCI bit is set bit is set ingress LC aci36-leaf1 pod36-spine1# show dcimgr repo sclass-maps |
pod36-spine1# show dcimgr repo sclass-maps | egrep "3014656.*16387" 1 3014656 16387 | 2457600 32772 [formed]
pod36-spine1# show dcimgr repo sclass-maps | egrep "3014656.*16387"
1 3014656
16387
|
2457600
32772
[formed]
© 2019 Cisco and/or its affiliates. All rights reserved.
pod36-spine1# show dcimgr repo vnid-maps | egrep 3014656
1 3014656
|
2457600
[formed]

Data path unknown EP on leaf Site 1 to Site 2 -

Proxy

Only differences is that ingress leaf does send to Local Proxy spine (like in single
Only differences is that ingress
leaf does send to Local Proxy
spine (like in single site pod
case). Ingress spine does Rw
Outer IP
Outer : 10.10.35.101 (site1 DP-ETEP to 10.10.35.102 (site2 dp-etep) Outer L4 : site1 vrf VNID
Outer : 10.10.35.101 (site1 DP-ETEP to 10.10.35.102 (site2 dp-etep)
Outer L4 : site1 vrf VNID –
Sclass App-EPG1
Pod36
Inner : 172.16.3.2 to 172.16.2.2
Aci-35-interconnect

Pod35

1/51-52 1/49-50 Spine just RW Outer Sip 2/5-6 2/5-6 And Outer aci35-spine1 Dest IP 2/1-2
1/51-52
1/49-50
Spine just RW Outer Sip
2/5-6
2/5-6
And Outer aci35-spine1 Dest IP
2/1-2
2/1-2
Outer : pod35-leaf1 PTEP to 10.0.88.66 (site1
anycast proxy)
Outer L4 : site1 vrf VNID – Sclas App-EPG1
Inner : 172.16.3.2 to 172.16.2.2
No EPM entry relying on
1/49-50
1/49-50
EPM learning for 172.16.3.2 to tunnel to site1 DP-ETEP
BD subnet route to
SPine
aci35-leaf1
aci36-leaf1
Outer Src IP is my DP-etep  DCI-rx aci36-spine1  Translate Vnid and sclass Send
Outer Src IP is my DP-etep  DCI-rx
aci36-spine1
Translate Vnid and sclass
Send to pod36-leaf1 tep per coop
Outer : 10.10.35.101 (site1 DP-ETEP) to pod36-leaf1 PTEP Outer L4 : site2 vrf VNID –
Outer : 10.10.35.101 (site1 DP-ETEP) to pod36-leaf1 PTEP
Outer L4 : site2 vrf VNID – Sclass App-EPG1 Translated
Inner : 172.16.3.2 to 172.16.2.2
Sclass App-EPG1 Translated Inner : 172.16.3.2 to 172.16.2.2 Web-EPG1 172.16.1.1/24 vlan-101 App-EPG1 172.16.3.2/24 VM
Web-EPG1 172.16.1.1/24 vlan-101 App-EPG1 172.16.3.2/24 VM
Web-EPG1 172.16.1.1/24 vlan-101
App-EPG1 172.16.3.2/24 VM
Web-EPG2 172.16.2.2/24 VM App-EPG2 172.16.4.1/24 vlan-104
Web-EPG2 172.16.2.2/24 VM
App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

Policy enforcement

© 2019 Cisco and/or its affiliates. All rights reserved.

Sclass Translation

Pod35 Pod36 Aci-35-interconnect DCI mgr translation Vrf vnid 2457600 -> 3014656 1/51-52 1/49-50 Sclass 32771
Pod35
Pod36
Aci-35-interconnect
DCI mgr translation
Vrf vnid 2457600 -> 3014656
1/51-52
1/49-50
Sclass 32771 -> 32772
2/5-6
2/5-6
aci35-spine1
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1
App-EPG1 172.16.3.2/24 VM
Sclass 32770 in VRF vnid 3014656
DCI mgr translation Vrf vnid 3014656 -> 2457600 Sclass 32770 -> 16390
DCI mgr translation
Vrf vnid 3014656 -> 2457600
Sclass 32770 -> 16390

aci36-spine1

Policy Enforcement

- Ingress leaf derives sclass and vnid based on local EPM

- If Remote EPM is populated Enforce Policy (as usual)

- Transmit to Remote Spine Site

- Remote spine site translate sclass and VNId

- - sent it to Dest leaf

aci36-leaf1

- Dest leaf learn remote EP entry in translated sclass

- Enforce policy if not done on ingress

in translated sclass - Enforce policy if not done on ingress Web-EPG2 172.16.2.2/24 VM Sclass 32771
Web-EPG2 172.16.2.2/24 VM Sclass 32771 in vrf VNID 2457600
Web-EPG2 172.16.2.2/24 VM
Sclass 32771 in vrf VNID 2457600

© 2019 Cisco and/or its affiliates. All rights reserved.

Policy from Site 2 (172.16.2.2) to Site 1 (172.16.3.2)

Packet Data in IPN VNID 2457600 sclass 32771 pod35-leaf1# show system internal epm endpoint ip
Packet Data in IPN
VNID 2457600 sclass 32771
pod35-leaf1# show system internal epm endpoint ip 172.16.3.2
MAC : 0050.56b1.4b52 ::: Num IPs : 1
IP# 0 : 172.16.3.2 ::: IP# 0 flags :
BD vnid : 16351138 ::: VRF vnid : 3014656
Phy If : 0x1a001000 ::: Tunnel If : 0
Flags : 0x80004c04 ::: sclass : 32770 ::: Ref count : 5
- Local EP learn with 32770
Aci-35-interconnect
DCI mgr translation
Vrf vnid 2457600 -> 3014656
1/51-52
pod35-leaf1# show system internal epm endpoint ip 172.16.2.2
MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 172.16.2.2 ::: IP# 0 flags :
1/49-50
Sclass 32771 -> 32772
BD vnid : 0 ::: VRF vnid : 3014656
Phy If : 0 ::: Tunnel If : 0x18010007
2/5-6
2/5-6
aci35-spine1
Interface : Tunnel7
Flags : 0x80004400 ::: sclass : 32772 ::: Ref count : 3
- We learn Remote EP with translated Sclass
aci36-spine1
pod35-leaf1# show zoning-rule| egrep "32770.*3014656"
4123
32770
32772
10
enabled
3014656
permit
fully_qual(7)
4124
32772
32770
10
enabled
3014656
permit
fully_qual(7)
2/1-2
2/1-2
pod35-leaf1# show system internal policy-mgr stats | egrep "3014656.*32770“
Rule(4123)DN(sys/actrl/scope-3014656/rule-3014656-s-32770-d-32772-f-10) , Pkts: 495659 RevPkts: 0
Rule(4124)DN(sys/actrl/scope-3014656/rule-3014656-s-32772-d-32770-f-10) , Pkts: 6 RevPkts: 0
Packet Data in site1
VNID 3014656 sclass 32772
Packet enforcement is mostly done in Ingress (if XR remote EP is learn), in egress otherwise
pod35-spine1# show dcimgr repo sclass-maps | egrep
1/49-50
1/49-50
"Remote|Vrf|32771"
Remote
|
Local
aci35-leaf1
aci36-leaf1
site Vrf
PcTag
|
Vrf
PcTag
Rel-state
2
2457600
32771
|
3014656
32772
[formed]
pod36-leaf1# pod36-leaf1# show show system system internal internal epm epm endpoint endpoint ip ip 172.16.2.2 172.16.2.2
MAC MAC : : 0050.56b1.4403 0050.56b1.4403 ::: ::: Num Num IPs IPs : : 1 1
IP# IP# 0 0 : : 172.16.2.2 172.16.2.2 ::: ::: IP# IP# 0 0 flags flags : :
Vlan Vlan id id : : 21 21 ::: ::: Vlan Vlan vnid vnid : : 8194 8194 ::: ::: VRF VRF name name : : DC:DC1 DC:DC1
BD BD vnid vnid : : 16220082 16220082 ::: ::: VRF VRF vnid vnid : : 2457600 2457600
App-EPG1 172.16.3.2/24 VM
Web-EPG2 172.16.2.2/24 VM
Phy If : 0x1a001000 ::: Tunnel If : 0
Phy If : 0x1a001000 ::: Tunnel If : 0
Sclass 32770 in VRF vnid 3014656
Sclass 32771 in vrf VNID 2457600
Interface Interface : : Ethernet1/2 Ethernet1/2
Flags : 0x80004c04 ::: sclass : 32771 ::: Ref count : 5
Flags : 0x80004c04 ::: sclass : 32771 ::: Ref count : 5
© 2019 Cisco and/or its affiliates. All rights reserved.

Multicast Multisite

© 2019 Cisco and/or its affiliates. All rights reserved.

Overview - Layer 2 BUM traffic across Sites

TX (local to remote site)

GIPo (BUM) traffic sourced from the local site is Head-end replicated (HREP) to each remote site from the Spine. DIPo is rewritten to a unicast address called as Multicast HREP TEP IP (also called Multicast DP-TEP IP) of the remote site. SIPo is rewritten with the Unicast ETEP IP

RX (remote to local site)

Incoming traffic destined to the local site’s Multicast HREP TEP IP gets

translated, derives the local site’s BD-GIPo, and follows the regular GIPo

lookup path from there

© 2019 Cisco and/or its affiliates. All rights reserved.

Multi-Site

Stretched BD with L2 Broadcast Extension

ACI Multi-Site

L2 flooding

BD with L2 Broadcast Extension ACI Multi-Site L2 flooding APIC Site 1 APIC Site 2 Tenant
APIC Site 1 APIC Site 2 Tenant IPA VRF Stone-IPA BD1/Subnet1 Web-EPG C1 BD2/Subnet2 App-EPG
APIC Site 1
APIC Site 2
Tenant IPA
VRF Stone-IPA
BD1/Subnet1
Web-EPG
C1
BD2/Subnet2
App-EPG
© 2019 Cisco and/or its affiliates. All rights reserved.
Use Case Properties ▪ Active/Active deployment with inter-site Layer 2 extension ▪ Objects stretched across
Use Case Properties
▪ Active/Active deployment with inter-site Layer 2
extension
▪ Objects stretched across sites:
• Tenant ID
• VRF context
• BD/Subnet
• Provider and Consumer EPGs
• Policy between EPGs
▪ L2 flooding enabled at the BD level
• L2 BUM traffic forwarded over head-end
replicated VXLAN tunnels

Use case lab VRF RD-L2:L2

Pod35

Pod36

Aci-35-interconnect

1/51-52 BD GW 1/49-50 10.1.1.254/24 10.2.2.254/24 2/5-6 2/5-6 aci35-spine1 2/1-2 2/1-2 1/49-50 1/49-50
1/51-52
BD GW
1/49-50
10.1.1.254/24
10.2.2.254/24
2/5-6
2/5-6
aci35-spine1
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1
BD GW 10.1.1.254/24 10.2.2.254/24
BD GW
10.1.1.254/24
10.2.2.254/24

aci36-spine1

aci36-leaf1

Vm 10.1.1.35
Vm 10.1.1.35
Vm 10.1.1.36
Vm 10.1.1.36

© 2019 Cisco and/or its affiliates. All rights reserved.

ACI Multi-Site L2 flooding APIC Site 1 APIC Site 2 Tenant RD-L2 VRF L2 BD1/10.1.1.254/24
ACI Multi-Site
L2 flooding
APIC Site 1
APIC Site 2
Tenant RD-L2
VRF L2
BD1/10.1.1.254/24
Web-EPG
Test
BD2/10.2.2.254/24
App-EPG

Config Check

BD must be set with intersite BUM allow flag

© 2019 Cisco and/or its affiliates. All rights reserved.

© 2019 Cisco and/or its affiliates. All rights reserved. admin@bdsol-aci35-apic1:~> moquery -d
admin@bdsol-aci35-apic1:~> moquery -d uni/tn-RD-L2/BD-Web Total Objects shown: 1 # fv.BD name OptimizeWanBandwidth
admin@bdsol-aci35-apic1:~> moquery -d uni/tn-RD-L2/BD-Web
Total Objects shown: 1
# fv.BD
name
OptimizeWanBandwidth
: Web
: yes
arpFlood
: yes
bcastP
: 225.0.216.80
childAction
:
configIssues
:
descr
:
dn
epClear
epMoveDetectMode
: uni/tn-RD-L2/BD-Web
: no
:
extMngdBy
: msc
intersiteBumTrafficAllow : yes
intersiteL2Stretch
ipLearning
lcOwn
limitIpLearnToSubnets
llAddr
: yes
: yes
: local
: yes
:
::
mac
mcastAllow
: 00:22:BD:F8:19:FF
: no
modTs
: 2018-05-03T03:14:39.650+00:00
monPolDn
mtu
multiDstPktAct
nameAlias
: uni/tn-common/monepg-default
: inherit
: bd-flood
:
ownerKey
:
ownerTag
:
pcTag
: 32770
rn
: BD-Web
scope
: 2457600
seg
: 15204288
status
:
type
: regular
uid
: 15374
unicastRoute
: yes
unkMacUcastAct
: flood
unkMcastAct
: flood
vmac
: not-applicable

Config check

Multicast HREP TEP IP per Site

Tunnel to each Remote site’s Multicast HREP TEP

pod35-spine1# show ip interface vrf overlay-1 | egrep -A 1 mcast-hrep loopback14, Interface status:
pod35-spine1# show ip interface vrf overlay-1 | egrep -A 1 mcast-hrep
loopback14, Interface status: protocol-up/link-up/admin-up, iod: 120, mode: mcast-hrep, vrf_vnid: 16777199
IP address: 10.10.35.121, IP subnet: 10.10.35.121/32
pod35-spine1# show interface tunnel 5
Tunnel5 is up
MTU 9000 bytes, BW 9 Kbit
Transport protocol is in VRF "overlay-1"
Tunnel protocol/transport ivxlan
Tunnel source 10.0.112.65, destination 10.10.35.122
© 2019 Cisco and/or its affiliates. All rights reserved.

Control Plane interaction

ISIS

For the stretched BDs (with intersiteBUMTrafficAllow), based on HREP-TEP configuration,

ISIS adds the Remote sites HREP Tunnel If to the BD-GIPO of the Stretched BD.

BD-GIPOs are striped across the Multisite-capable Spines meaning HREP Tunnel If is added to BD-GIPo only on one of the Multi-site capable Spines in a site

Unlike Multi-pod, no IGMP joins are sent out towards IPN, since native multicast is not

used for forwarding BUM traffic across the sites

pod35-spine1# show isis internal mcast routes gipo | egrep -A 6 "225.0.216.80" GIPo: 225.0.216.80 [LOCAL]
pod35-spine1# show isis internal mcast routes gipo | egrep -A 6 "225.0.216.80"
GIPo: 225.0.216.80 [LOCAL]
OIF List:
Ethernet2/1.35
Ethernet2/2.36
Tunnel5

© 2019 Cisco and/or its affiliates. All rights reserved.

One spine per site Should have Tunnel Interface as BD GIPo OIL
One spine per site
Should have Tunnel
Interface as BD GIPo
OIL

Use case lab VRF RD-L2:L2

ACI Multi-Site L2 flooding APIC Site 1 APIC Site 2 Tenant RD-L2 VRF L2 BD1/10.1.1.254/24
ACI Multi-Site
L2 flooding
APIC Site 1
APIC Site 2
Tenant RD-L2
VRF L2
BD1/10.1.1.254/24
Web-EPG
Test
BD2/10.2.2.254/24
App-EPG

Pod35

Pod36

Aci-35-interconnect

1/51-52 BD GW 1/49-50 10.1.1.254/24 10.2.2.254/24 2/5-6 2/5-6 aci35-spine1 2/1-2 2/1-2 1/49-50 1/49-50
1/51-52
BD GW
1/49-50
10.1.1.254/24
10.2.2.254/24
2/5-6
2/5-6
aci35-spine1
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1
BD GW 10.1.1.254/24 10.2.2.254/24
BD GW
10.1.1.254/24
10.2.2.254/24

aci36-spine1

aci36-leaf1

Vm 10.1.1.35
Vm 10.1.1.35
Lab Test ARP broadcast Vm 10.1.1.36 From 10.1.1.35 to 10.1.1.32
Lab Test ARP broadcast
Vm 10.1.1.36
From 10.1.1.35 to 10.1.1.32

© 2019 Cisco and/or its affiliates. All rights reserved.

Use case lab VRF RD-L2:L2

Outer : 10.10.35.101 to 10.10.35.122 (site2 dci-mcast) VNID 0xE7FFC0 Pod35 Pod36 Inner : arp from
Outer : 10.10.35.101 to
10.10.35.122 (site2
dci-mcast) VNID 0xE7FFC0
Pod35
Pod36
Inner : arp from 10.1.1.25 Aci-35-interconnect to 10.1.1.32
1/51-52
BD GW
BD GW
1/49-50
10.1.1.254/24
10.1.1.254/24
10.2.2.254/24
10.2.2.254/24
2/5-6
2/5-6
aci35-spine1
aci36-spine1
2/1-2
2/1-2
Outer : 10.10.35.101 to 225.0.191.0 + ftag
(site2 gipo) VNID 0xe5ffd2
Outer : pod35-leaf1 PTEP to GIPo (site1
Inner : arp from 10.1.1.25 to 10.1.1.32
225.0.216.90 +FTAG) VNID 0xE7FFC0
Inner : arp from 10.1.1.25 to 10.1.1.32
1/49-50
1/49-50
aci35-leaf1
aci36-leaf1
Vm 10.1.1.35
Vm 10.1.1.35
Lab Test ARP broadcast Vm 10.1.1.36 From 10.1.1.35 to 10.1.1.32
Lab Test ARP broadcast
Vm 10.1.1.36
From 10.1.1.35 to 10.1.1.32

© 2019 Cisco and/or its affiliates. All rights reserved.

GIPo route on line card site 1 spine

module-2# show forwarding multicast route group 225.0.216.80 vrf all (*, 225.0.216.80/32), RPF Interface: NULL, flags:
module-2# show forwarding multicast route group 225.0.216.80 vrf all
(*, 225.0.216.80/32), RPF Interface: NULL, flags: Dc
Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 3
Outgoing Interface List Index: 15
Ethernet2/1.35 Outgoing Packets:0 Bytes:0
Ethernet2/2.36 Outgoing Packets:0 Bytes:0
Tunnel5 Outgoing Packets:0 Bytes:0

© 2019 Cisco and/or its affiliates. All rights reserved.

L3 out and Multisite

© 2019 Cisco and/or its affiliates. All rights reserved.

Use case 3 lab VRF RD-L2:L2

Pod35

Pod36

BD GW 10.1.1.254/24 10.2.2.254/24
BD GW
10.1.1.254/24
10.2.2.254/24

Aci-35-interconnect

BD GW 10.1.1.254/24 10.2.2.254/24
BD GW
10.1.1.254/24
10.2.2.254/24
1/51-52 1/49-50 2/5-6 2/5-6 2/1-2 2/1-2 1/49-50 1/49-50 OSPF
1/51-52
1/49-50
2/5-6
2/5-6
2/1-2
2/1-2
1/49-50
1/49-50
OSPF

aci35-spine1

aci35-leaf1

OSPF

Vm 10.1.1.35
Vm 10.1.1.35

aci36-spine1

aci36-leaf1

Vm 10.1.1.36 Lo1 10.30.2.1/24
Vm 10.1.1.36
Lo1 10.30.2.1/24
Lo1 - 10.30.1.1/24
Lo1 - 10.30.1.1/24

© 2019 Cisco and/or its affiliates. All rights reserved.

L3 out lab VRF RD-L2:L2

Pod35

Pod36

L3 out – lab VRF RD-L2:L2 Pod35 Pod36 Aci-35-interconnect BD GW 10.1.1.254/24 10.2.2.254/24 BD GW 1/51-52

Aci-35-interconnect

BD GW 10.1.1.254/24 10.2.2.254/24
BD GW
10.1.1.254/24
10.2.2.254/24
BD GW 1/51-52 10.1.1.254/24 1/49-50 10.2.2.254/24 2/5-6 2/5-6 2/1-2 2/1-2 1/49-50 1/49-50 aci35-leaf1 OSPF Vm
BD GW
1/51-52
10.1.1.254/24
1/49-50
10.2.2.254/24
2/5-6
2/5-6
2/1-2
2/1-2
1/49-50
1/49-50
aci35-leaf1
OSPF
Vm 10.1.1.35

aci35-spine1

OSPF

1/49-50 aci35-leaf1 OSPF Vm 10.1.1.35 aci35-spine1 OSPF aci36-spine1 aci36-leaf1 Vm 10.1.1.36 Lo1 10.30.2.1/24 Lo1

aci36-spine1

aci36-leaf1

Vm 10.1.1.36 Lo1 10.30.2.1/24
Vm 10.1.1.36
Lo1 10.30.2.1/24
Lo1 - 10.30.1.1/24
Lo1 - 10.30.1.1/24

© 2019 Cisco and/or its affiliates. All rights reserved.

• Working Session : • in EPG web : 10.1.1.35 to 10.1.1.36 • Local L3
• Working Session :
• in EPG web : 10.1.1.35 to 10.1.1.36
• Local L3 out :
• 10.1.1.35 to 10.30.1.1
• 10.1.1.36 to 10.30.2.1
• Non working connection (expected)
• 10.1.1.35 to 10.30.2.1
• 10.1.1.36 to 10.30.1.1
• Might or not be Working Direction (return from L3 out):
• 10.30.2.1 can reach 10.1.1.35
• 10.30.1.1 can reach 10.1.1.36
• Non working direction (from VM to L3 remote):
• 10.1.1.35 to 10.30.2.1
• 10.1.1.36 to 10.30.1.1
In summary: we do not support traffic From EPG in siteX to L3 out in
In summary: we do not support traffic
From EPG in siteX to L3 out in siteY
(planned for 4.2)

Why EP to remote L3 out do not work

No VPNv4 route exchange across multisite BGP session

No l2vpn evpn type 5 neither

Site 2 never got route from Site 1 L3 out

Only l2vpn evpn capa nego with Peer on the intersite) pod35-spine1# show bgp l2vpn evpn
Only l2vpn evpn capa nego with Peer on the intersite)
pod35-spine1# show bgp l2vpn evpn neigh 10.10.35.112 vrf overlay-1 | egrep -A 1
"capabili"
Additional Paths capability: advertised received
Additional Paths Capability Parameters:
Send capability advertised to Peer for AF:
L2VPN EVPN
Receive capability advertised to Peer for AF:
L2VPN EVPN
No l2vpn evpn type5 for subnet neither is advert
pod35-spine1# show bgp l2vpn evpn neigh 10.10.35.112 adver vrf overlay-1 | egrep
"10.30"
pod35-spine1#
© 2019 Cisco and/or its affiliates. All rights reserved.

Traffic return from L3 out

Will work if no subnet configured on l3 out epg (or 0.0.0.0/0)

Will not work if any subnet is configured under the l3 out with ext subnet for external EPG

See MSC release notes :

NOTE: The subnet in the L3extInstP must be the same for all inter-

related sites (and variable length network masks are not supported).

© 2019 Cisco and/or its affiliates. All rights reserved.

Thanks Q&A

© 2019 Cisco and/or its affiliates. All rights reserved.