Beruflich Dokumente
Kultur Dokumente
Configuration is Deployed
Phillip Ferrell, Technical Leader Insieme BU Escalation Team
Andy Gossett, Technical Leader Insieme BU Escalation Team
BRKACI-3101
Agenda
• Introduction
• Building the Overlay
• Access Policies
• VRFs, Bridge Domains, and Endpoint Groups
• L2Outs and Loop Prevention
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Acronyms/Definitions
Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure LPM Longest Prefix Match
ACL Access Control List MDT Multicast Distribution Tree
Reference Slide
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Introduction
Introduction
What are our basic network requirements?
L2 L3
EP1 VLAN EP2 EP3 VLAN EP4
1 2 External External
VRF-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Introduction
What are our basic network requirements?
6) Allow security policies in order to limit communication to between endpoints to allowed protocols
ip access-list web-in
VLAN 1 VRF1 VLAN 2 permit tcp Subnet1 Subnet2 eq 80
Subnet1 Subnet2 ip access-list web-out
permit tcp Subnet2 eq 80 Subnet1
EP1
80 EP3
ip access-group web1 in
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
What physical topology is required?
Physical topology must support our endpoint communication (layer-2 / layer-3)
and the location of endpoints within the physical network will affect the supporting
design/configuration.
L2 L3
EP1 VLAN EP2 EP3 VLAN EP4
1 2 External External
VRF-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Traditional Topology – Routing at Core/Spine
STP results in unused links / limits scale / slower convergence
L2 L3
EP1 VLAN EP2 EP3 VLAN
1 2 External External
VRF-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Traditional Topology – Routing at Access
Restricts L2 endpoint locations / requires separate links for L2 / segmented STP
L2 L3
EP1 VLAN EP2 EP3 VLAN
1 2 External External
VRF-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
ACI Infrastructure
Physical links
ISIS is run on links between spines / leaves
ISIS / MDT
L2 L3
EP1 EP2 EP3
External External
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
ACI Infrastructure
Physical links
APICs communicate to fabric over infra vlan
ISIS / MDT
L2 L3
EP1 EP2 EP3 APIC
External External
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
ACI Infrastructure
Physical links
Leaves/spines advertise TEP via ISIS
ISIS / MDT
T T T T T
T
L2 L3
EP1 EP2 EP3 APIC
External External
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
ACI Infrastructure
Physical links
Leaves advertise learned EP to spines via COOP
ISIS / MDT
COOP Oracles T Tunnel Endpoint (TEP)
L2 v4 v6 L2 v4 v6
L2 v4 v6 Anycast Spine Proxy TEPs
10.1.1.57
COOP Citizens
T T T T T
10.1.1.57
L2 L3
EP1 EP2 EP3 APIC
External External
10.1.1.57
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
ACI Infrastructure
Physical links
BL advertises external routes to fabric through MP-BGP
ISIS / MDT
MP-BGP RRs T Tunnel Endpoint (TEP)
L2 v4 v6 L2 v4 v6
0.0.0.0/0 L2 v4 v6 Anycast Spine Proxy TEPs
RR-Clients
0.0.0.0/0
T T T T T
0.0.0.0/0
L2 L3
EP1 EP2 EP3 APIC
External External
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ACI Infrastructure
APIC provisions BD/VRF VXLAN overlays based on EPG attachments
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
VXLAN
VXLAN differentiates tunneled traffic based on VNID field.
OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
Flags
Reserved
I
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
iVXLAN
In addition to differentiating traffic based on VNID, iVXLAN allows the source EPG of traffic to be identified
by the source group (PCTAG) bits and to determine if policy was applied by source (SP) / destination (DP).
OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header
iVXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
Flags S D
Reserved Source Group
I P P
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
ACI Infrastructure
Policy is implemented through contracts / filters specifying allowed traffic
cons prov
EPG1 EPG2
HTTP (80)
VRF-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
MAC Header
Ethernet Frame
MAC Header PAYLOAD FCS
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
EtherType
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
MAC w/802.1Q Header
Ethernet Frame
MAC Header PAYLOAD FCS
802.1Q
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
PCP / D
Tag Protocol Identifier (0x8100) E VLAN Identifier
COS I
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
EtherType
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
IPv4 Header
Ethernet frame containing IP packet
MAC Header IPv4 Header PAYLOAD FCS
802.1Q
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
D M
Identification R Fragment Offset
F F
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
Source IP Address
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
Destination IP Address
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
TCP Header
Ethernet Frame containing TCP packet
MAC Header IPv4 Header TCP Header FCS
PAYLOAD FCS
802.1Q
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Sequence Number
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
Acknowledgement Number
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
UDP Header
Ethernet Frame containing UDP packet
MAC Header IPv4 Header UDP Header FCS
PAYLOAD FCS
802.1Q
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Length Checksum
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Access / 802.1q Trunked Hosts
Ethernet Frame
MAC Header PAYLOAD FCS
Ethernet Frame
MAC Header PAYLOAD FCS
802.1Q
802.1Q
Trunk
Ethernet frame containing IP packet
MAC Header IPv4 Header PAYLOAD FCS
802.1Q
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Hypervisor Host w/AVS
VXLAN Tunnel
of the leaf.
OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
ACI Intra-fabric
802.1Q
Trunk
VXLAN Tunnel
OUTER INNER
MAC Header IPv4 Header UDP Header iVXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Access Policies
Access Policies
Access policies refer to the configuration that is applied for physical and virtual
(hypervisors/VMs) devices attached to the fabric.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Global Policy
Pools (Vlan / VXLAN) Pool1 Pool2
A resource pool of encapsulations that
can be allocated within the fabric.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Global Policy - Attachable Entity Profiles
Configuration:
• Create a VLAN/VXLAN pool with a range
of encapsulations Pool1 Pool2 Pool3 Pool4
• Create a domain (physical, l2/l3 external, DomPhy1 DomVm1 DomL2 DomL3
or VMM) and associate pool
• Associate domain to AEP AEP AEP AEP
• Associate interface policy group to AEP Statics VMs External
switch/interface selectors will apply the
config through the interface policy group
assign to specific ports
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
What have we accomplished?
• Specified what domains and
corresponding pools are allowed per
interface in the fabric!
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Access Policies SWITCH POLICY
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Interface Policy Groups
Used to specify which interface policies to be applied to a particular interface type.
It also associates an AEP (which defines which domains are allowed on the interface).
Types:
VPC Domain 1
Access port (EP1)
Access Bundle Groups
• Virtual Port-channel (EP2)
• Port-channel (EP3)
Note: Separate policy groups should be created for each port-channel (standard or VPC) that you
need to configure. All interfaces on leaf that are associated with a particular access bundle group reside
in same channel.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Port-Channel Policies
interface Ethernet1/5-6
lacp port-priority 32768
lacp rate normal
channel-group 10 mode on
interface Ethernet1/10-11
lacp port-priority 32768
lacp rate fast
channel-group 20 mode active
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Access Policy Example
General Configuration (reused for many interfaces): AEP Pool1
1) Configure a physical domain and vlan pool Vandalay
DomPhy1
2) Create an AEP and associate physical domain
3) Create switch/interfaces profiles for leaf (LEAF101) Switch Profile
LLDP Rx / Tx enabled
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Creating Physical Domain / AEP / Vlan Pool
In dropdown:
Click Create Attachable Entity Profile
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Creating Physical Domain / AEP / Vlan Pool
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Creating Physical Domain / AEP / Vlan Pool
In dropdown:
Click Create VLAN Pool
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Create Interface Profile for each leaf / VPC domain
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Create Switch Profile for each leaf / VPC domain
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Create Switch Profile for each leaf / VPC domain
Enter name
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Create common protocol configurations
Example demonstrates a common lacp port-channel policy
Configure options/knobs
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Access Policy Example
Interface specific (each time you add a new interface): AEP Pool1
1) Create policy group for device (VPC / PC / Access) Vandalay
DomPhy1
2) Within the policy group, select the desired policies /
AEP Switch Profile
3) Associate interfaces to policy group via desired leaf LEAF101
profile
• use specific leaf profile if access or PC blk_101
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Create policy groups
Note:
A separate policy
Descriptive name
group should be
created for each
PC/VPC that you will
deploy
Associate your desired
interface policies
(otherwise default)
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Create interface selectors / associate policy group
Specify interface/range
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Example policy scheme
Switch Profile Leaf101 Leaf101_102
1/1-4
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
vPC Protection Group Policy
vPC Domain 1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
VPC Protection Group (example configuration)
GUI sequence:
Tabs:
Fabric -> Access Policies
Navigation Tree:
Switch Policies -> Policies ->
VPC Domain -> Default
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
VRFs, Bridge Domains, and
Endpoint Groups
VRF/BD/EPG Logical Configuration
VRF-Vandalay
BD-Importers BD-Exporters
Importer-1 Importer-2 Exporters
IM1 IM2 IM3 EX1 EX2
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
ACI Logical Configuration
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Overlay Fabric Allocations
VRF-VNID – allocated per VRF
Tenant: Vandalay Industries • (unique within fabric)
Networking App Profile: Operations
BD-VNID – allocated per BD
VRF: Vandalay EPG: Importer-1 • (unique within fabric)
Subnet: 10.20.0.1/24
PCTAG – allocated per EPG
BD: Importers • FABRIC-global if shared service
EPG: Importer-2 provider
Subnet: 10.10.0.1/24
• VRF-local otherwise
EPG: Exporters
BD: Exporters EPG-VNID – allocated from vlan pool
Subnet: 10.30.0.1/24 (domain specific) and is unique within
fabric
• Used for STP BPDU flooding and
Domain: DomPhy1 flood in encap for unknown unicast
traffic
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Creating a Tenant
Create a tenant by clicking the
Tenant Tab and ‘Add Tenant’
icon.
Provide a name for the new
Provide a name for the new tenant
tenant.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Creating a VRF in the Tenant
Right click on the VRF’s under
the networking folder and
choose ‘Create VRF’. Provide a name for
the new VRF
Provide a name for the new
VRF
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creating a BD and associate with VRF
Create a new BD by right clicking on the ‘Bridge Domain’ folder under the Networking tab and choose
‘Create Bridge Domain’
Provide a name for the new BD and associate it to the previously created VRF.
Click ‘Next’ and leave the L3 Configurations and Advanced/Troubleshooting with default values
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Adding a Subnet to a BD
Create a new Subnet under the
bridge domain by right clicking
the subnets folder and choose
‘Create Subnet’
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Creating an Application Profile
Create a new application profile by right
clicking the folder and choose ‘Create
Application Profile’
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Creating an Application EPG
Create the EPG and associate it
with the correct BD.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Adding a Domain to the EPG
After the EPG has been creating, associate a physical
domain by right clicking on the Domains folder and
choosing ‘Add a Physical Domain’.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Adding a Static Path to the EPG
To add a static path, under the Static Bindings folder
right click and choose ‘Deploy Static EPG’.
Specify the static path port, port-channel, or VPC
along with the VLAN encap
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
EPG Static Path Deployment
EPG are deployed through:
VRF-Vandalay • Static binding to port/PC/VPC
BD-Importers BD-Exporters • Static binding to node
• VM attachment
Importer-1 Importer-2 Exporters
IM1 IM2 IM3 EX1 EX2 To successfully deploy an EPG
configuration on a leaf:
1. AEP of target interface must allow
same domain as assigned to EPG
2. encapsulation/vlan must be allowed
in the target domain
vPC Domain
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
DomPhy1
IM1 IM2 IM3 EX1 EX2
vlan 100-200
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
EPG Static Path Deployment Leaf101 BD-Importers vlan-101
vPC Domain
VRF-Vandalay 10.10.0.1/24
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
External EPG –
LegacyApache EPG – Apache
BD – Webservers
BD – Webservers
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Extending Layer-2 domain outside of ACI
Extend an EPG to legacy switches
Spanning Tree R
D
Root port
Designated port
Classical behavior B (Blk) Blocking port
• STP BDPUs (PVST or MST) are
generated by each switch in the topology. Root Bridge
• STP root is elected and interface
forwarding is calculated to prevent loops
by blocking some interfaces. D D
• All interfaces with best-path (highest
bandwidth) towards root bridge will be
forwarding.
• Backup paths will be put in a blocking R R
state by the switch with worst path
towards root on the affected path
(usually based on either the bridge
identifier or port priority) D B
• Topology changes (TC) trigger MAC
addresses to be flushed in received vlan,
allowing traffic reconvergence based on
new topology
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Spanning Tree
ACI floods BPDUs in the fabric encap
• ACI leaves don’t participate in spanning
tree (generate BPDUs or block any ports)
• STP BDPUs (PVST or MST) are flooded
within the fabric/EPG encap (allocated per
vlan encap in a domain)
• Leaves flush endpoints in the EPG if a TC
BPDU is received. EPG - Web
• Spanning Tree Domain policy
determines which EPGs to flush for
MST domain TCs BPDU BPDU
D D
NOTE: MST BPDUs are untagged and
require an untagged/native EPG to be Root
Root Bridge
deployed on all interfaces connected to MST Bridge
domain (this includes L3outs using SVIs)
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Spanning-Tree Policy
Classical MST Configuration
Requires configuration of STP mode, MST region,
MST revision, and vlan assignments to MST instances
Root Bridge
Note: MST configuration must match for all switches
within a specified region. If they do not, any port
receiving conflicting or legacy BPDUs will be treated as D D
part of the IST instance.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Spanning Tree Domain Policy
ACI MST Configuration
Configuration is fabric-wide and supports multiple
regions for use within different tenants/domains.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Common mistakes that cause loops
Missing untagged/native EPG in MST region
EPG - Web
All interfaces connected to a common MST
vlan-100 vlan-100
region should have the same EPG deployed
(this is to ensure BPDU is flooded to all of the D D
MST switches connected to fabric).
LOOP!!
BPDUD R BPDU
BPDU BPDU
Root
Bridge
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Common mistakes that cause loops
Multiple fabric encaps used for same EPG
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Agenda
• Introduction
• Building the Overlay
• Access Policies
• VRFs, Bridge Domains, and Endpoint Groups
• L2Outs and Loop Prevention
SMAC
LPM Routes
Eth: 0x0806
• Connected/direct routes manually Route Adj
configured 10.1.1.101/32 … Hdr/Opcode
• Static/dynamic routing protocols to 10.1.1.0/24
20.1.1.101/32 Glean
… Sender MAC
learn prefixes 20.1.1.0/24 Glean
Sender IP
Host Routes
A Target MAC
• Glean adjacency for connected routes P
to punt frame and generate ARP A Target IP
request
• ARP/ND used to create MAC to IP
binding and install host route into 10.1.1.101/24 20.1.1.101/24
routing table
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
ACI Learning and Forwarding (Physical Local - PL)
NEW
Encap + Interface => EPG
EPG => BD EPGs and L3
BD => VRF Learning
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC
ARP L3 Forwarding
L3 forwarding based on ARP target IP field
(VRF, ARP Target IP) Miss => Proxy
with miss sent to spine proxy
(VRF, ARP Target IP) Hit=> Adjacency
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
ACI Learning (Virtual Local - VL) Fabric TEP
AVS VTEP
Inner Header VXLAN Outer Header
Infra VLAN
Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID Rsvd DIP SIP 802.1Q SMAC DMAC
UDP
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
ACI Learning (Remote - XR) Dst Leaf VTEP
Src Leaf VTEP
Inner Header iVXLAN Outer Header
Fabric QoS
flags Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID DIP SIP 802.1Q SMAC DMAC
EPG UDP
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
ACI Learning (COOP vs. EP Sync)
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
ACI Learning (EP) Leaf Endpoint Database
Endpoint Entry
What is an EP (Endpoint)? - EPG (pcTag)
• MAC Remote IP Entries - Interface/Tunnel
VRF (VRF, IP)
• IPv4 (/32) or IPv6 (/128) host route - Control flags
VXLAN Tunnel
• No IP learning of shared service
prefixes outside of our VRF between AVS and
fabric on Infra VLAN
LPM Routes (Same as Classical)
• Pervasive SVI Routes (BD Subnets) Static/Dynamic WAN/Inte
Routing on L3Out rnet
• Static and dynamic routing protocols
on L3Out AVS
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
ARP has resolved on both hosts.
ACI Forwarding L1 does not have H2 in EP database
Hardware Proxy enabled on BD-B1
Unknown Layer2 Unicast: Hardware Proxy 1. H1 sends layer2 unicast frame to H2.
Layer2 Spine Proxy
2. L1 performs layer2 lookup on H2
Policy Applied destination MAC and misses. Frame is
S1 S2 on egress L3 sent to Spine Anycast MAC Proxy
VTEP. EPG-E1 and BD-B1 VNID set in
2 3 VXLAN header. No policy applied since
destination EPG is unknown
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Hardware Proxy Enabled
under the BD
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
L2 Unknown Unicast flood
with ARP flooding enabled
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
ACI Forwarding
BD Multicast Settings
Layer 2 Multicast Layer 3 Multicast (IANA range)
• Flood in BD • Known multicast traffic will have
flood to all ports in bridge domain IGMP/MLD snooping entry and
• Flood in Encapsulation forwarded to appropriate ports
flood to all ports matching ingress
• Unknown multicast
encapsulation. This may be a subset of
ports in the bridge domain • Flood
flood to ports in bridge domain
• Drop • Optimize Flood
send only to router ports detected by
PIM hellos
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
ARP has resolved on both hosts.
ACI Forwarding L1 has learned H2 from L3
1. H1 sends layer2 unicast frame to H2.
Known Layer2 Unicast
2. L1 performs layer2 lookup on H2
destination MAC and finds endpoint with
Policy Applied destination EPG-E2 and VTEP of L3.
S1 S2 on ingress L1 L1 applies policy between EPG-E1 and
EPG-E2. If permitted, frame is sent to L3
2 3 VTEP with EPG-E1 and BD-B1 VNID set
in VXLAN header.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
ARP has resolved on hosts for ACI GW
ACI Forwarding L1 has learned H3 from L6
1. H1 sends layer3 unicast frame to H3
Known Layer3 Unicast (destination MAC of BD-B1).
2. L1 performs layer3 lookup on H3
Policy Applied destination IP and finds endpoint with
S1 S2 on ingress L1 destination EPG-E3 and VTEP of L6.
L1 applies policy between EPG-E1 and
2 3 EPG-E3. If permitted, frame is sent to L6
VTEP with EPG-E1 and VRF-V1 set in
VXLAN header.
4
L1 L2 L3 L4 L5 L6 3. Spine receives frame with outer
destination IP of L6 and routes packet.
5
1 4. L6 does layer3 lookup on H3 destination
IP in VRF-V1.Hit in local EP database and
H1 H2 H3 derives destination EPG-E3. Since policy
BD-B1 VRF-V1 BD-B2 already applied on L1, no policy check
EPG-E1 EPG-E2 EPG-E3 on L6.
5. L6 forwards traffic to H3 with appropriate
encap
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
No endpoints initially learned
ACI Forwarding 1. H1 sends ARP broadcast request for H2.
L1 learns MAC and IP for H1
ARP/IP Unknown Layer3 Endpoint (glean) L1 performs layer3 forwarding lookup
Layer3 Spine Proxy Miss based on ARP target IP address for H2
2. H2 not present on L1, send to Spine
S1 S2 Anycast IPv4 Proxy VTEP. VRF-V1
VNID set in VXLAN header.
2 3 No policy applied since destination EPG
is unknown
3. Spine does not have H2 IP, sends special
L1 L2 L3 L4 L5 L6 glean packet to all leafs on reserved GIPo
with VRF-V1 VNID set in VXLAN header
4 4
5
1 4. L1 and L3 have BD-B1 subnet present,
generate ARP request for H2 (sourced
H1 H2 H3 from fabric pervasive SVI)
BD-B1 VRF-V1 BD-B2
EPG-E1 EPG-E2 EPG-E3 5. H2 sends ARP response, L3 learns H2
MAC and IP and syncs to Spines
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Broken Traffic Flow Example
• A Layer3 gateway device (GW) is connected to the fabric via a
normal BD/EPG. Host H3 is using GW as its gateway for a
L3Out subset of traffic.
• The initial EP database show the IP’s and MACs learned in the
correct locations.
Subnet E1 E2 Subnet
int-S1 int-S2 MAC EP Database
BD-B1 BD-B2
BD MAC EPG Port
1/1 1/2 1/3
BD-B1 mac:G1 E1 1/1
BD-B2 mac:G2 E2 1/2
GW H3
IP:G1 IP:G2 BD-B2 mac:H3 E2 1/3
mac:G1 mac:G2 IP:H3
IP EP Database
H3 gateway mac:H3
FW, LB, Router, etc. Vrf IP MAC EPG Port
v1 IP:G1 mac:G1 E1 1/1
v1 IP:G2 mac:G2 E2 1/2
v1 IP:H3 mac:H3 E2 1/3
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Broken Traffic Flow Example
• H3 sends a frame to GW on BD-B2 (L2 switched through the
fabric). GW routes the frame and sends it toward the fabric to
L3Out be routed out.
• Fabric performs IP learning on routed traffic, IP:H3 moves to
mac:G1 on EGP E1, port 1/1
Subnet E1 E2 Subnet
int-S1 int-S2 MAC EP Database
BD-B1 BD-B2
BD MAC EPG Port
1/1 1/2 1/3
BD-B1 mac:G1 E1 1/1
BD-B2 mac:G2 E2 1/2
GW H3
IP:G1 IP:G2 BD-B2 mac:H3 E2 1/3
mac:G1 mac:G2 IP:H3
IP EP Database
H3 gateway mac:H3
FW, LB, Router, etc. Vrf IP MAC EPG Port
v1 IP:G1 mac:G1 E1 1/1
v1 IP:G2 mac:G2 E2 1/2
v1 IP:H3 mac:H3
mac:G1 E2
E1 1/3
1/1
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Broken Traffic Flow Example
ARP for IP:H3 What’s Broken?
sent out EPG-E1
L3Out • ARP to IP:H3 may fail since the IP is pointing to the wrong port
• Routed traffic to IP:H3 may be policy dropped since it’s
classified in EPG-E1 instead of EPG-E2
Subnet E1 E2 Subnet
int-S2 • IP:H3 may rapidly move within the fabric.
int-S1
BD-B1 BD-B2
1/1 1/2 1/3 IP EP Database
ARP for Vrf IP MAC EPG Port
IP:H3H3
GW v1 IP:G1 mac:G1 E1 1/1
IP:G1 IP:G2
mac:G1 mac:G2 IP:H3 v1 IP:G2 mac:G2 E2 1/2
H3 gateway mac:H3 v1 IP:H3 mac:H3
mac:G1 E2
E1 1/3
1/1
FW, LB, Router, etc.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Broken Traffic Flow Example
Solutions
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Broken Traffic Flow Example #2 • H1 in EPG-E1 with gateway
configured on BD-B1
ext2
• H2 in EPG-E2 is in layer2 only BD-B2
L3Out with gateway outside the fabric via an
subnet L2Out
L1 L2 ext-S2 H2 subnet is not configured in fabric
Common for during brownfield migration
L2Out
• Traffic from H1 to H2 is routed outside
the fabric via the L3Out and then bridged
H1 H2
BD-B1 BD-B2 back in from an external router via the
subnet L2Out
EPG-E1 int-S1 EPG-E2
• A contract C1 is configured to allow
traffic from EPG-E1 to the L3Out.
VRF-V1
• A contract C2 is configured to allow
traffic from EPG-E2 to its gateway on the
L2Out.
EPG-E1 ext2 L2Out EPG-E2 • Traffic from EPG-E1 to EPG-E2 works
C1 subnet C2 fine but return traffic fails, why?
ext-S2
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Broken Traffic Flow Example #2 1. H2 sends ARP request for
external gateway. L2 learns IP
No contract between ext2 from ARP for H2 in EPG-E2
EPG-E2 and EPG-E1 When traffic is received from L3Out on
L3Out 2.
subnet L2 with source IP of H2, L2 derives
L1 L2 ext-S2 source EPG of EPG-E2 instead of the
L3Out external EPG-ext2
L2Out
3. Policy enforcement on L2 is between
EPG-E2 and EPG-E1 instead of L3Out
H1 H2
BD-B1 BD-B2 EPG-Ext2 and EPG-E1.
subnet Since there is no contract defined
EPG-E1 int-S1 EPG-E2 between these EPGs, traffic is dropped
How to fix this issue?
VRF-V1
• Disable Unicast Routing on BD-B2. This
will prevent Layer2 only BD’s from
Enabling ‘Enforce Subnet Check’ on the learning endpoint IP’s from host ARP
BD is recommended for preventing the • OR, enable ‘Enforce Subnet Check’ on
fabric from learning rogue/misconfigured BD-B2
hosts on layer3 BD’s
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
enable/disable unicast
routing under the BD
enable/disable subnet
check under the BD
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
Classical Policy Enforcement
Ingress Egress Type Access Control Entry (ACE) Format
Pipeline Pipeline MAC action src/mask dst/mask ethertype [PD filters]
ARP action opcode srcIp/mask dstIp/mask srcMac/mask
1 2 3 4 5 dstMac/mask [PD filters]
IP/IPv6 action protocol srcIp/mask srcPort/mask dstIp/mask
dstPort/mask [PD filters]
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
ACI Policy Enforcement • Policy is created based on contract between EPGs
with support for L2/L3/L4 filters similar to traditional
ACLs.
• Leaf derives source EPG pcTag based on:
Scope Access Control Entry (ACE) Format • match in EP database
VRF action src-EPG dst-EPG [filters] src MAC for L2 traffic or src IP for L3 traffic
VRF permit any any (unenforced mode) • longest-prefix match against src IP
(IP-based EPG or L3Out external EPG)
• ingress port + encap
1
• Leaf derives destination EPG pcTag based on:
• match in EP database
dst MAC for L2 traffic or dst IP for L3 traffic
Apply Policy • longest-prefix match against dst IP
(L3Out external EPG or shared-services)
Derive destination EPG pcTag
EP lookup, IP Prefix • Rules are programmed with scope of VRF. Policy
lookup is always (VRF, src-EPG, dst-EPG, filter).
Derive source EPG pcTag
• Allow traffic between all EPGs without a contract
local EP, IP Prefix, or Encap
by setting the VRF to unenforced mode
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
ACI Policy Enforcement SYN
Web Server
(S1)
Reference TCP Packet SYN+ACK
H1
Data
Seq#, Ack# Dst Src Proto
DIP SIP ethtype SMAC DMAC
ACK
flags, etc.. Port Port TCP
port x data… port 80
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Option 1 – Unidirectional filters
ACI Policy Enforcement Apply both flt-1 and flt-2 to subject
Identify Provider (P) EPG and Consumer (C) EPG
flt-1 (C to P) and flt-2 (P to C)
src-port dst-port
permit tcp Consumer Provider eq 80
H1 C P
permit tcp Provider eq 80 Consumer
• With a bidirectional contract, the ‘provider’ will be permit tcp Consumer Provider eq 80
the dst-port filters and the ‘consumer’ will be the
src-port filters (opposite of contract arrows) flt-1 + apply both directions
Create Filters permit tcp Consumer Provider eq 80
Name EthType Proto Src Port Dst Port permit tcp Provider Consumer eq 80 Only flt-1
flt-1 IP TCP Any 80 needed!
flt-2 IP TCP 80 Any flt-1 + apply both directions + reverse ports
Create a contract, subject, and filter(s). Apply to EPGs permit tcp Consumer Provider eq 80
EGP-Web as provider and EPG-Client as consumer permit tcp Provider eq 80 Consumer
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
filter flt-1 created matching
TCP port with any source port
to destination port 80 (http)
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
filter flt-2 created matching
TCP port with source port 80
(http) to any destination port
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Contract Scope
Create a contract The contract scope will limit which providers and
consumers can participate within the same contract.
Specify contract name • VRF
The contract can be applied between EPGs within
Contract Scope the same VRF.
(default to VRF)
• Application Profile
The contract can be applied between EPGs within
the same application profile
• Tenant
The contract can be applied between EPGs within
the same tenant.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
Option 1
Unidirectional filters to
Unidirectional requires specifying explicitly specify rule from
both Consumer to Provider AND consumer to provider AND
provider to consumer filters from provider to consumer.
Consumer to Provider
users filter flt-1
Consumer to Provider
users filter flt-2
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
Option 2
Bidirectional filter with
reverse port enabled.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Add EPG provider to contract Add EPG consumer to contract
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
High Policy CAM Utilization Example
• 100 EPGs all providing a basic management
E1 E1 E1 E1 E0
contract to a single consumer EPG.
E2 E2 E2 E2
E3 E3 E3 E3 mgmt- mgmt- • TCAM Utilization Calculation (Approximate)
E2 E2 E2 E2 contract EPG ~= (entries in contract)(# of Cons)(# of Providers)(2)
E1 E2 E3 E4 ~= 2 * 1 * 100 * 2
100 EPGs ~= 400 entries in hardware
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
High Policy CAM Utilization Example
Name EthType Proto Src Port Dst Port permit tcp E1 eq 1 E0 eq 22
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
ACI Contracts and Resource Utilization
Contract created between E2 and E3 • BD-B1 and BD-B2 each have a subnet
defined. Subnet int-S1 on BD-B1 exists on
E2 E3 L1 and L3, while subnet int-S2 for BD-B2
exists on L6
Add contract and S1 S2 Add contract and When creating the contract between E2 and E3:
route to int-S2 route to int-S1 • Program contract rule between E2 and E3
in TCAM. Add Static route for int-S1
created on L6 pointing to spine proxy.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
Create Global Contract and Export to
Tenant-T2
Create a contract C1 with Global scope in Tenant-T1.
Contract defined in Tenant-T1 Create a subject with appropriate filters.
with scope of Global.
epg-E1 provides this contract Ensure EPG-E1 is a provider for C1
C1-export present in
Tenant-T2
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
choose the imported
contract
Shared Service
Consumer
Consume the imported contract via
‘Consumed Contract Interface’
Add Consumed Contract
Interface to consume Routes are now leaked between
imported contract VRFs.
fab2-leaf101# show ip route vrf Tenant-T1:VRF-V1 fab2-leaf101# show ip route vrf Tenant-T2:VRF-V2
IP Route Table for VRF "Tenant-T1:VRF-V1" IP Route Table for VRF "Tenant-T2:VRF-V2"
'*' denotes best ucast next-hop '*' denotes best ucast next-hop
'**' denotes best mcast next-hop '**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric] '[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string> '%<string>' in via output denotes VRF <string>
10.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive 10.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.1.1%overlay-1, [1/0], 00:06:06, static *via 10.0.1.1%overlay-1, [1/0], 00:06:12, static
10.1.1.1/32, ubest/mbest: 1/0, attached, pervasive 10.2.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.1.1.1, vlan24, [1/0], 00:38:56, local, local *via 10.0.1.1%overlay-1, [1/0], 00:10:40, static
10.2.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive 10.2.1.1/32, ubest/mbest: 1/0, attached, pervasive
*via 10.0.1.1%overlay-1, [1/0], 00:06:06, static *via 10.2.1.1, vlan26, [1/0], 00:10:40, local, local
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
1. H1 sends packet toward gateway in
Shared Service Forwarding EPG-E1 with destination IP of H3
2. L1 performs layer3 lookup for H3 in
From Provider E1 to Consumer E4 VRF-V1 and hits LPM entry for H3 subnet.
LPM entry points to proxy with VNID
rewrite info for VRF-V2.
Policy Applied on Packet is sent to Spine Anycast IPv4
S1 S2 egress L6 Proxy VTEP with VRF-V2 VNID and
(consumer VRF) EPG-E1 set in VXLAN header.
2 3 No policy applied in provider VRF
Workaround?
1. Configure subnet under the provider EPG and the provider BD
Supported in 1.2 and above
2. Force one-way leaking in both directions by making both EPGs shared service providers and shared
service consumers
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
export to T2
Consume
ACI Shared Services Provide
Interface
C1 C1-export
Subnet on Provider BD scope: global
Workaround 2
E1 E2 E3 E4
Both EPG-E1 and EPG-E4 are shared service
providers and shared service consumers BD-B1 BD-B2
Contract C1
VRF-V1 VRF-V2
• leaks subnet S2 into VRF-V1
Tenant-T1 Tenant-T2
• programs policy into VRF-V2 BD-B1 Subnet: S1 BD-B2 Subnet: S2
Contract C2 scope: shared scope: shared
export to T1
• leaks subnet S1 into VRF-V2
Consume Provide
• programs policy into VRF-V1 Interface C2-export C2
scope: global
Advantages/Disadvantages?
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
Policy TCAM
Contract Review Contract VRF Action Src Dst Filter
C1 V2 permit E2 E1 flt1
• Shared Service EPGs V2 permit E1 E2 *flt1
EPGs that provide contract consumed by C2 V2 permit E4 E3 flt2
EPG in a different VRF: E1, E2*
V2 permit E3 E4 *flt2
• Application EPGs V2 permit ext2 E3 flt2
E1, E2, E3, E4 V2 permit E3 ext2 *flt2
C3 V2 permit ext1 any flt3
• External EPGs
configured on L3Out and classified based on V2 permit any ext1 *flt3
IP prefix: ext1, ext2
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
Configure L3Out Basic
Connectivity
• Right click the External Routed
Networks and choose Create
Routed Outside
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
Configure L3Out Basic
Name of Logical Node Profile Click + to add Nodes to Connectivity
this Logical Node Profile
• Configure the Logical Node Profile
by first providing the name.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Select vpc path and
Name of Logical vlan encap for L3Out
Interface Profile
Configuring an
external SVI in Configure path
this example as Trunk
• Configure the name of the Logical Interface Profile and add a path attribute. In this example, the path is configured as a vPC with encap of vlan-151.
For vPC path, the IP address will be configured on both leafs in the vPC pair. Therefore, there’s a side-A and a side-B IP address.
• Multiple other interface options can be configured under the LIF such as MTU, secondary IP’s, IPv6 Link local, etc…
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
Verify L3Out Basic
L3Out ‘L3Out-1’ Connectivity Configuration
• Verify that the logical node profile
Node profile ‘node- is correctly configured
103-104’
• Verify that the logical interface
profile is configured with correct
Interface profile IP’s, path, and VLAN
external SVI path configured with encapsulation
‘ipv4-lif’
IP’s assigned to both nodes on the
vPC with encap of vlan-151
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
Basic Connectivity Pool4
DomL3
Layer3 Out: L3Out-1
VRF: VRF-V1
Layer-3 Domain: DomL3 AEP
External
Logical Node Profile: node-103-104
path: topology/pod-1/…vpcX
topology/pod-1/…vpcX Remember AEP!
path:
SVI encap: vlan-x
IP-A, IP-B, MTU, MAC, mode • interfaces from path must be member of DomL3 AEP.
• The vlan encapsulation must be within encap blocks
defined in DomL3 vlan pool
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
Verify L3Out Path
Path attribute from logical Configuration
interface profile references
interface policy group In this example, a vpc is
configured as the path attribute
on the L3out. The vpc is is
configured on node-103 and
node-104 with a VLAN encap
of vlan-151.
In this slide:
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
Verify L3Out Path
Configuration
In this example, a vpc is
Domain is associated to configured as the path attribute
the correct AEP on the L3out. The vpc is is
configured on node-103 and
Domain is associated to
node-104 with a VLAN encap
VLAN pool4 of vlan-151.
In this slide:
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
Configuring Routing Protocols
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
Enabling BGP
1. Under the L3Out, enable
the BGP process.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
Enabling BGP
2. Under the logical Node
profile, create BGP peer
connectivity profile under
loopback or previously
configured logical interface
profile.
• BGP controls
• BGP Credentials
• EBGP multi-hop
MP-BGP
overlay-1
L3Out-1 L3Out-2
E1 E2
ext1 ext2
BD-B1 BD-B2
subnet subnet subnet subnet
ext-S1 int-S1 int-S2 ext-S2
• Internal Routes: Subnets defined under BD are internal routes and create static pervasive routes
within the fabric.
• External Routes: Routes learned via a routing protocol or static routes configured under an L3Out.
These routes are redistributed into MP-BGP and advertise to all leafs that contain the VRF
• Transit Routes – Routes advertised between L3Outs.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
Configure Fabric MP-BGP
To enable MP-BGP through the fabric
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
Types of Fabric Routes – Internal Routes
Subnet: int-S2 3
Subnet int-S2 installed on border leaf
Scope: MP-BGP
when creating contract between EPG
Private to VRF E2 and external overlay-1
EPG ext2
Advertise Externally
Share Between VRFs
L3Out-1 L3Out-2
1
E1 E2
ext1 ext2
BD-B1 BD-B2 2
subnet subnet subnet C1 subnet
ext-S1 int-S1 int-S2 ext-S2
There are three requirements to advertise Internal Routes out an L3Out:
1. The BD must be associated with the L3Out*
The association adds prefix entry to route map controlling advertised routes
2. A contract must exists between an EPG within the BD and an external EPG on the L3Out.
The contract creates internal BD route on border leaf (cannot advertise route until it exists locally)
3. The subnet must have a public scope (Advertise Externally)
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Advertise Internal BD’s
Under L3 Configurations for BD,
ensure
• BD is associated to L3Out
Associate BD to L3Out
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
Types of Fabric Routes – External Routes
ext-S1
ospf, eigrp, static route RD: L1:V1
redistributed into bgp and RT: ASN:V1 MP-BGP
exported into mp-bgp
overlay-1
ext-S1
L3Out-1 L3Out-2
• External Routes from ospf, eigrp, or static are redistributed on the border leaf into the local bgp process.
• The bgp route is exported into MP-BGP with a route-target (RT) of the corresponding VRF. Each leaf in
the fabric with the VRF present will import the RT and install the route. External routes on the non-
originating border leaf will be seen as bgp learned routes.
• External Routes are controlled via Import Route Control flag
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
Types of Fabric Routes – Transit Routes
MP-BGP
overlay-1
ext-S1
ext-S1
L3Out-1 L3Out-2
• In this example, external route ext-S1 is a Transit Route when advertised out L3Out-2.
• If OSPF or EIGRP on L3Out-2, ext-S1 is redistributed from BGP into the IGP and advertised.
• Transit Routes are controlled via Export Route Control flag
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Configure L3Out External Network node-103
RID: #
node-104
RID: #
IP: A IP: B
Define an External Network, ext1 in this example
• Note: At least one external network required to
bring up L3Out interfaces on border leaf
vlan-x
• Add Subnet to External Network
Add a subnet
Specify
scope/controls
• Configure the subnet prefix and choose appropriate scope flags. Further definition of each flag in upcoming slides.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
External Subnets for the External EPG
Previously: Import-Security Subnet: ext-S2/mask
Scope:
External Subnet for the External EPG is used to classify External Subnets for the External EPG
dataplane packets into external EPG for policy enforcement.
EPG to pcTag
• An IP prefix is installed into leaf TCAM to classify traffic VRF EPG pcTag
to/from the external network and assign correct pcTag for
policy enforcement V1 E1 49156
neighbor neighbor-1
Inbound route-map imp-l3out-vrf
L3Out-1
Outbound route-map exp-l3out-vrf
Allow
Advertisement:
route-map imp-l3out-peer-vrf permit - ext-S1/mask
match: prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst - ext-S2/mask
- ext-S3/mask
ip prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst BGP Ignore
permit ext-S1/mask Neighbor-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 153
Aggregate Import Subnet: 0.0.0.0/0
Scope:
*Aggregate Import supported only for 0.0.0.0/0 or ::/0 Import Route Control Subnet
Aggregate:
Import Route Control allows fabric to permit a Aggregate Import
specific prefix. Instead of creating each prefix
advertised by a neighbor, multiple prefixes can be
aggregated together by using the Aggregate Import
flag.
neighbor neighbor-1
Inbound route-map imp-l3out-vrf
Outbound route-map exp-l3out-vrf
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
Export Route Control & Aggregate Export
*Aggregate Export supported only for 0.0.0.0/0 or ::/0
Subnet: ext-S1/mask
Export Route Control allows Transit Routes to be Scope:
advertised out of the fabric. Export Route Control Subnet
• Export control does NOT affect pervasive BD SVIs,
they are only advertised when the BD is associated
with the L3Out.
• Similar to import route control subnet, a prefix list
with corresponding exported subnets is created to
allow routes to be advertised out
Subnet: 0.0.0.0/0
Scope: Export all Advertisement: Export:
Export Route Control Subnet Transit Routes - ext-S1/mask - ext-S1/mask
Aggregate: within VRF - ext-S2/mask
- ext-S3/mask
Aggregate Export
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
Shared L3Out
Similar to Shared Services, a Shared L3Out uses
contracts to leak routes between VRFs. The leaked
routes can be:
• int-S1 subnet from VRF-V1 to VRF-V2
• ext-S2 subnet from VRF-V2 into VRF-V1
Similar Restrictions as Shared Services
E1 L3Out-1
• If the application EPG is providing the contract for
shared L3Out, the internal subnet must be defined
under the EPG. BD-B1
• If the external EPG is providing the contract for shared ext2
VRF-V1
L3Out, then internal subnet can be defined either under C1
the EPG or the BD EPG-subnet subnet
• Internal subnet must have shared and Advertise int-S1 ext-S2
Externally(public) scope. VRF-V2
• Contract must be appropriately scoped. Scope:
• For shared L3Out, shared subnet must be globally Private to VRF
unique within the entire ACI fabric. Advertise Externally
Share Between VRFs
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
Shared L3Out E1 L3Out-1
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
Shared L3Out E1 L3Out-1
• In this example, adding shared security import to the VRF Route pcTag Flags
external subnet created a prefix-based EPG in any- V1 int-S1 1 proxy
VRF* for the external subnet ext-S2 with pcTag of EPG-
ext2. V2 ext-S2 ext2 L3Out
V2 int-S1 E1 proxy, leak->V1
V1
V1 ext-S2 ext2
deny-tag L3Out, leak->V2
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
Subnet: 8.8.0.0/16
ip prefix-list IPv4-V2-V1-shared-svc-leak
seq 3 permit 8.8.0.0/16 le 32
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 160
L3 External Subnet Review
o External Subnets for the External EPG (Security Import)
Used to classify dataplane packets into external EPG for policy enforcement
o Aggregate Export - allows prefixes to be aggregated together in export direction (0/0 or ::/0 only)
o Aggregate Import - allows prefixes to be aggregated together in import direction (0/0 or ::/0 only)
o Aggregate Shared Route - allows prefixes to be aggregated together for shared route control
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
Extra L3Out FAQ
How to Advertise Transit Static Route MP-BGP
ext-S1 overlay-1
In this example, a static route ext-S1 is configured on leaf L1 with
next-hop out L3Out-1. A second L3Out-2 is running OSPF. The
intention is to advertise the static route out L3Out-2.
L1 L4
An external network is configured under L3Out-2 with export flag for
0.0.0.0/0 along with aggregate to allow ALL routes to be advertised
In this topology, L4 advertises the static route while L1 does not. Static Route on L1
ext-S1
with next-hop on
Why? L3Out-1 L3Out-1 L3Out-2
By default, static routes configured within the fabric are not OSPF
advertised out L3Outs and will not match aggregate 0/0 prefixes. Static Static Route is
On L4, route to ext-S1 is technically a BGP route and internal route- advertised out L4
map will match 0/0 aggregate prefix for static route. On L1, route to
but not L1
ext-S1 is a static route that will not match aggregate 0/0 (by design).
ext-S1
Subnet: 0.0.0.0/0
Export Route Control Subnet
Fix: Aggregate Export
To properly advertise a static route, create an external network
matching the static route prefix and enable the export flag
Subnet: ext-S1/mask
Export Route Control Subnet
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
How to use the Route Tag Policy
To prevent potential route loops, transit routes are sent with the
VRF route-tag. External routes that are advertised with the same
route-tag are denied.
By default, all VRFs resolve to the same Route Tag Policy with
default value of 4294967295. As a result, transit routes advertised
between VRFs may be denied.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 164
How to use the Route Control Policy
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
route tag Yes Yes Supported only for BD (internal) BD-B1 BD-B2 L3Out-1
subnets. Transit prefixes are always
set according to VRF route-tag policy subnet subnet
preference Yes BGP local preference
int-S1 int-S2
int-S1
metric Yes Yes Sets MED for BGP. Will change the community: 65535:2
metric for EIGRP but you cannot
specify the EIGRP composite metric.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 165
How to use the Route Control Policy
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
For fabric routes, the route attributes can be set at the L3Out level, at the BD
level, or at the subnet level. If a route control policy is set at each level, the
most specific policy will be applied. I.e.,
• Tenant BD Subnet
• Tenant BD
• L3Out
There are two reserved policies that can be used at the L3Out level:
• default-import
• default-export
In this example, the default-export route control policy will set the community
and an additional route control policy applied at the BD will be configured to
set both the community and the MED.
Steps
1. Create an action rule to set
the community
2. Create a second action rule
that sets both the Under External Routed
community and the MED Networks, create action
rule
3. (continue to next slide)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
How to use the Route Control Policy
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
Steps
3. Right-click the ‘Route Profiles’ under the L3Out-1 and create a new route-
profile
4. From the drop-down list, chose the reserved route profile default-export
5. Add a route control context to set the community
6. (continue to next slide)
At the point, all traffic advertised out L3Out-1 will have the community set. A
second route profile is needed to tag BD-B2 subnets differently.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
How to use the Route Control Policy
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
Steps
6. Right-click the ‘Route Profiles’ under the L3Out-1 and create a new route-
profile
7. Chose a unique name for the route control profile
8. Add a route control context to set both the community and the MED
9. (continue to next slide)
NOTE, only the default-export policy affects routes advertised at the L3Out level.
Custom route profiles still need to be applied at the BD or BD-subnet level.
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 168
How to use the Route Control Policy
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
Steps
9. Set the L3Out for
Route Profile under
the BD to L3Out-1
10. Set the Route Profile
to the previously
configured profile
11. Complete!
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 170
Agenda
• Introduction
• Building the Overlay
• Access Policies
• VRFs, Bridge Domains, and Endpoint Groups
• L2Outs and Loop Prevention
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 172
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services