Beruflich Dokumente
Kultur Dokumente
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
APIC
Application Policy
ACI Fabric Infrastructure Controller
Integrated GBP VXLAN Overlay
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
ACI MultiPod/MultiSite Use Cases
Single Site Multi-Fabric
Multiple Fabrics connected within the same DC (between halls, buildings, … within the
same Campus location)
Cabling limitations, HA requirements, Scaling requirements
L2/L3
MP-BGP - EVPN
… MP-BGP - EVPN
APIC Cluster
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Agenda
DC Site 1 DC Site 2
vCenter
Fabric stretched between two or three sites works as Work with one or more transit leaf per site any leaf
a single fabric deployed within a DC node can be a transit leaf
One APIC cluster one management and configuration Number of transit leaf and links dictated by
point redundancy and bandwidth capacity decision
Anycast GW on all leaf switches Different options for Inter-site links (dark fiber, DWDM,
EoMPLS PWs)
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Stretched ACI Fabric
Option 1 – Dark Fiber
DC Site 1 DC Site 2
vCenter
40G links currently required between Transit Leafs and remote Spines
QSA adapters required for deploying 10G connections
Roadmap item for CY17
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Stretched ACI Fabric
Option 2 – DWDM Circuits
DC Site 1 DC Site 2
10ms RTT
QSFP-40G-SR4
DWDM
40G 4x10G
40G
MTP-LC
breakout cable
10 ms RTT
800 KM / 500 miles
QSFP-40G-SR4
40G
10G/40G/100G
40G
EoMPLS Pseudowire
10G/40G/100G
40G 40G
WAN
Port mode EoMPLS used to stretch the ACI 1.0(3f) release or later, 10ms max RTT between
fabric over long distance sites
DC Interconnect links could be 10G (minimum) or Under normal conditions 10 ms allows us to support two
higher with 40G facing the Leaf/Spine nodes DCs up to 800 Km apart
DWDM or Dark Fiber provide connectivity between Other ports on the Router used for connecting to the WAN
two sites via L3Out
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Stretched ACI Fabric
Support for 3 Interconnected Sites (Q2CY16) Site 2
Site 1
Site 3
Transit Leaf
2x40G or 4x40G
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Agenda
MP-BGP - EVPN
Multiple ACI Pods connected by an IP Inter-Pod Forwarding control plane (IS-IS, COOP)
L3 network, each Pod consists of leaf and spine fault isolation
nodes Data Plane VXLAN encapsulation between
Managed by a single APIC Cluster Pods for seamless L2 or L3 connectivity
Single Management and Policy Domain End-to-end policy enforcement 16
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ACI Multi-Pod Solution
Use Cases
Handling 3-tiers physical Pod
cabling layout Inter-Pod
Leaf Nodes Network
Cable constrain (multiple
buildings, campus, metro)
requires a second tier of “spines” Spine Nodes
Preferred option when compared
to ToR FEX deployment
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
ACI Multi-Pod Solution
Supported Topologies
Intra-DC Two DC sites directly connected
10G/40G/100G
40G/100G 40G/100G
POD 1 40G/100G 40G/100G
POD n POD 1 Dark fiber/DWDM POD 2
(up to 10 msec RTT)
…
POD 3 BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
SW/HW Roadmap and Scalability
Values
ACI Multi-Pod Solution
SW and HW Requirements
Software Software
2.0 Release 2.0 MR Release (Q3CY16)
Hardware Hardware
1st Generation Leafs 2nd Generation Leafs
9396PX/TX, 9372PX/TX, 93120TX, 93128TX, 93108-EX, 93180-EX
9332PQ
2nd Generation Spines
1st Generation Spines 9732C-EX Line Card
9736PQ Line Card, 9336PQ
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ACI Multi-Pod Solution
Scalability Considerations
Those scalability values may change without warning before the Multi-Pod
solution gets officially released
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Inter-Pod Connectivity Deployment
Considerations
ACI Multi-Pod Solution
Inter-Pod Network (IPN) Requirements
MP-BGP - EVPN
DB Web/App
APIC Cluster Web/App
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Inter-Pod Connectivity
Frequently Asked Questions
Nexus 9200s, 9300-EX, but also any other
switch or router supporting all the IPN
requirements
What platforms can or should I
deploy in the IPN? First generation Nexus 9300s/9500s not initially
supported as IPN nodes
SW fix is being scoped for 2HCY16 timeframe
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Inter-Pod Connectivity
Frequently Asked Questions (2)
back?
No, because of multicast requirement for L2 multi-
destination inter-Pod traffic
40G/100G
IPN Devices
connections
POD 1 POD 2
Can I use a Layer 2 only No, the IPN nodes should be deployed as L3
infrastructure as IPN? network devices
Any Protocol
OSPF OSPF
L3
POD 1 POD 2
IPN
Is OSPF the only protocol Devices
supported in the IPN network?
APIC Cluster
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
APIC Cluster Deployment Considerations
Single Pod Scenario
X X
APIC APIC APIC Shards in
‘read-only’
mode
X X
APIC APIC APIC APIC APIC
Shards in Shards in
‘read-only’ ‘read-write’ mode
APIC will allow read-only access to the DB mode
Additional APIC will increase the system scale (today
when only one node remains active (standard up to 5 nodes supported) but does not add more
DB quorum) redundancy
Hard failure of two nodes cause all shards to Hard failure of two nodes would cause inconsistent
be in ‘read-only’ mode (of course reboot etc. behaviour across shards (some will be in ‘read-only’
heals the cluster after APIC nodes are up) mode, some in ‘read-write’ mode)
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
APIC Cluster Deployment Considerations
Multi-Pod Scenario
X X
Pod 1 Pod 2 Pod 1 Pod 2
X X
Up to 10 msec Up to 10 msec
X X
APIC APIC APIC APIC
X X X
APIC APIC APIC APIC APIC
Pod isolation scenario: changes still possible Pod isolation scenario: same considerations as with
on APIC nodes in Pod1 but not in Pod2. single Pod (inconsistent behaviour across shards)
Cluster nodes fully re-join once Pods are Pod failure scenario: Pod1 major failure may cause
reconnected the loss of information for the shards replicated across
Pod failure scenario: under major Pod failure, the 3 local nodes
recommendation is to activate a standby node Possible to restore the whole fabric state to the latest taken
to make the cluster fully functional again configuration snapshot (‘ID Recovery’ procedure – needs BU
and TAC involvement)
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
APIC Cluster Deployment Considerations
Deployment Recommendations
Main recommendation: deploy a 3 nodes APIC cluster, independently from the total
number of Pods (plus 1 backup node in a 2 Pods scenario)
Avoiding loss of information under any failure scenario
Pods can be centrally managed even without a locally connected APIC node
When 5 nodes are needed for scalability reasons, follow the recommendations below:
Pod1 Pod2 Pod3 Pod4 Pod5 Pod6
Infra policies are currently the only policies following zone deployment
configuration
Tenant polices are immediately propagated to all nodes independently of zone
configuration
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Reducing the Impact of Configuration Errors
Introducing Configuration Zones
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Multi-Pod Control and Data Planes
For more information on how to setup an
ACI Multi-Pod Solution ACI Fabric from scratch:
Auto-Provisioning of Pods BRKACI-2004
3
1 4
1
6
1
DHCP response reaches Spine 1
allowing its full provisioning
2
1 7
1
10
1 Discover other Pods following the same procedure
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
ACI Multi-Pod Solution
IPN Control Plane
IPN Global VRF
IP Prefix Next-Hop
10.0.0.0/16 Pod1-S1, Pod1-S2, Pod1-S3, Pod1-S4
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
ACI Fabric – Integrated Overlay
Decoupled Identity, Location & Policy
APIC
ACI Fabric decouples the tenant end-point address, it’s “identifier”, from the location of that end-
point which is defined by it’s “locator” or VTEP address
Forwarding within the Fabric is between VTEPs (ACI VXLAN tunnel endpoints) and leverages an
extender VXLAN header format referred to as the ACI VXLAN policy header
The mapping of the internal tenant MAC or IP address to location is performed by the VTEP using
a distributed mapping database
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Host Routing - Inside
Inline Hardware Mapping DB - 1,000,000+ hosts
10.1.3.35 Leaf 3
10.1.3.11 Leaf 1
Global Station Table Proxy Proxy Proxy Proxy fe80::8e5e Leaf 4
contains a local cache fe80::5b1a Leaf 6
of the fabric endpoints
10.1.3.35 Leaf 3
Proxy Station Table contains
addresses of ‘all’ hosts attached
* Proxy A to the fabric
10.1.3.11 Port 9
Local Station Table The Forwarding Table on the Leaf Switch is divided between local (directly attached) and
contains addresses of global entries
‘all’ hosts attached The Leaf global table is a cached portion of the full global table
directly to the Leaf
If an endpoint is not found in the local cache the packet is forwarded to the ‘default’
forwarding table in the spine switches (1,000,000+ entries in the spine forwarding table)
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
ACI Multi-Pod Solution
Inter-PODs MP-BGP EVPN Control Plane
All remote Pod entries associated to a Proxy EP3 Proxy B EP3 Leaf 4
VTEP next-hop address EP4 Proxy B EP4 Leaf 6
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
ACI Multi-Pod Solution
Inter-PODs Data Plane Policy information
carried across Pods = VXLAN Encap/Decap
Group
VTEP IP VNID Tenant Packet
Policy
Spine encapsulates
EP1 Leaf 4 Leaf 4
EP2 Proxy B traffic to remote Proxy EP2
Proxy A
EP1
B Spine VTEP Spine encapsulates
traffic to local leaf
3 4
Proxy A Proxy B
EP2 e1/1
EP1 e1/3 EP1 Pod1 L4
5 * Proxy B
* Proxy A
Leaf learns remote VM1
Single APIC Cluster location and enforces policy
VM2 unknown, traffic is 2
EP1 EP2
encapsulated to the local Proxy
A Spine VTEP (adding S_Class 1 6
information) VM1 sends traffic destined If policy allows it, VM2
to remote VM2 receives the packet
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
ACI Multi-Pod Solution
Inter-PODs Data Plane (2) = VXLAN Encap/Decap
Group
VTEP IP VNID Tenant Packet
Policy
EP1 e1/3
EP2 Pod2 L4 EP1 Pod1 L4
** Proxy A
8 * Proxy B
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
ACI Multi-Pod Solution
Handling of Multi-Destination Traffic (BUM*)
IPN replicates traffic to all
the Pods that joined GIPo 1
(optimized delivery to Pods)
Spine 2 is responsible to 1
4
send GIPo 1 traffic toward
the IPN
1
3
*L2 Broadcast, Unknown Unicast and Multicast BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Establishing Layer 3 External
Connectivity
Connecting ACI to Layer 3 Domain
Traditional L3Out Design
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Connecting ACI to Layer 3 Domain For more information refer to:
BRKACI-2020
’GOLF’ Design (ACI 2.0 Release)
WAN
WAN Edge Connect an ACI Fabric to the external L3
Routers
domain
Wan Edge devices functionally behave as ACI ‘border leafs’
VXLAN Data Plane L3Out at
spines Control plane and data plane scale
MP-BGP
OpFlex for automating the exchange of config parameters
EVPN
(VRF names, BGP Route-Targets, etc.)
= VXLAN Encap/Decap BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
ACI Integration with WAN at Scale
Supported WAN Edge Platforms
WAN
WAN Edge Router initial choices
Nexus 7000/7700: F3 line card in
7.3.0.DX(1)ES. M3 support in Q4CY16
MP-BGP
IP Network ASR 9000: IOS-XR 6.1.1 for platforms with
EVPN
RSP2*, RSP440 and RSP880 supervisors
ASR 1000: 16.4 release (Q4CY16), including
also CSR1Kv support
High level whitepaper available on CCO:
http://www.cisco.com/c/en/us/solutions/collateral
/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-736899.html
Software Software
2.0 Release (July 2016) 2.1 Release (Q4CY16)
Hardware Hardware
1st and 2nd Generation Leafs 2nd Generation Spines
9396PX/TX, 9372PX/TX, 93120TX, 93128TX, 9332PQ, 9732C-EX Line Card
93108-EX, 93180-EX
1st Gen Spines
9736PQ Line Card, 9336PQ
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
ACI Integration with WAN at Scale
Supported Topologies
Directly Connected Indirectly Connected
WAN Edge Routers WAN Edge Routers Multi-Pod*
MP-BGP
MP-BGP
EVPN IP Network IP Network
EVPN
MP-BGP
EVPN
MP-BGP
EVPN
MP-BGP
EVPN
Inter-Pod VXLAN traffic exchanged via the IPN WAN Edge devices perform a dual function:
devices Pure L3 routing for Inter-Pod VXLAN traffic
WAN to DC traffic VXLAN encapsulated between VXLAN Encap/Decap for WAN to DC traffic
ACI fabric and WAN Edge devices flows
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Multi-Pod and Scalable WAN Integration
Intra-DC Deployment – Control Plane
WAN Edge
Devices WAN
WAN routes received on the Pod
spines as EVPN routes and translated
MP-BGP EVPN Control Plane to VPNv4/VPNv6 routes with the spine
proxy TEP as Next-Hop
IPN
Public BD subnets advertised to WAN
Edge devices with the external spine-
proxy TEP as Next-Hop
Multiple
Pods
...
Web/App Web/App Web/App
DB DB
Single
Single APIC Cluster
APIC Domain
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Multi-Pod and Scalable WAN Integration
Intra-DC Deployment – Control Plane
Multiple
Pods
...
Single
Single APIC Cluster
APIC Domain
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Multi-Pod and Scalable WAN Integration
Intra-DC Deployment – Data Plane
WAN Edge devices have host
routes information mapping WAN Edge
endpoints to each Pod location Devices WAN Traffic from an external user
is steered toward the WAN
Edge devices
WAN Edge devices VXLAN
encapsulate traffic and send it to
the Spine Proxy VTEP address in IPN
the ‘right’ Pod
Single
Single APIC Cluster
APIC Domain
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Multi-Pod and Scalable WAN Integration
Intra-DC Deployment – Data Plane (2)
WAN Edge
Devices WAN
WAN Edge devices decapsulate Traffic is received by
traffic and send it to the WAN the external user
IPN
Single
Single APIC Cluster
APIC Domain
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Multi-Pod and Scalable WAN Integration
Inter-DC Scenario
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Multi-Pod and Scalable WAN Integration
Inter-DC Scenario (2)
Remote Router Table Granular inbound path
10.10.10.10/32 optimization( host route
G1,G2
advertisement into the WAN or
10.10.10.11/32 G3,G4
integration with LISP)
IPN
Proxy A Proxy B
10.10.10.10 10.10.10.11
Single APIC Cluster
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Migration Scenarios
Migration Scenarios
Adding Pods to an Existing ACI
Add connections to the Connect and auto-provision
1 IPN network the other Pod(s)
Pod1 Pod2
MP-BGP - EVPN
MP-BGP EVPN
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Migration Scenarios
Converting Stretched Fabric to Multi-Pod
3
Pod1 Pod2
MP-BGP EVPN
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
ACI Multi-Pod Solution
Summary
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Agenda
• Multi-Fabric Scenarios
Primary use case is to support multiple “Availability Zones”
Use Cases
Multiple Fabrics within a single site (includes Multi-Floor, Multi-Room Data Centers)
Multi-Building cross campus and metro distances (Majority of larger customers
require a dual site active/active design)
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
For more information on ACI Dual
ACI Multi-Site Solutions Fabric Deployment:
Current Deployment Option 1 – Dual Fabric BRKACI-3503
Orchestrator
(UCSD, etc…)
Site ‘A’ Site ‘B’
L2 Transport
Direct/vPC
OTV/VXLAN/PBB
WAN
L2 Outside
VRF-Lite (iBGP, OSPF, Static Routes)
L2 and L3 Traffic
WAN
‘Register for EP
notification in ‘WEB1’ EPG Ext-WEB1 App
APP1 WEB1
‘Ext-WEB1 EPG created in
Subnet/BD ‘A’ ACI
Toolkit remote L3Out
‘Intersite’ Application of ACI Toolkit allows to peer with local and remote APIC
clusters and specifies:
What local EPG needs to be “exported” to a remote site (‘WEB1’) and its name in the
remote location (‘Ext-WEB1’)
What contracts will be consumed/provided by that EPG in the remote site
The L3Out in the remote sites where to program host routes
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
ACI Multi-Site Solutions
Current Deployment Option 2 – ACI Toolkit Use Case
Site ‘A’ Site ‘B’
IP == EPG classification on
the Border Leaf (scaling is
limited to LST 24K entries)
Policy applied on the
egress VTEP
WAN
Notifies about endpoints
APP1 WEB1 connecting to the specified Ext-WEB1 App
Subnet/BD ‘A’ ACI ‘WEB1’ EPG
Toolkit Program host routes into the
‘Ext-WEB1’ EPG in remote L3Out
The local APIC notifies the toolkit every time an endpoint connects to the ‘WEB1’ EPG in the
“local” site
The ACI Toolkit communicate to the remote APIC to program the host route in the L3Out of the
“remote” site (part of the ‘Ext-WEB1’ EPG)
This allows to properly classify traffic received in DC site 2 specified contract is also applied
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
ACI Multi-Site Solutions Future
Future Deployment Option (CY17)
Web1 Web2 Import Web & App Export Web & App Web1 Web2
from Fabric ‘B’ to Fabric ‘A’
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
ACI Multi-Site Solutions Future
Future Deployment Option (CY17)
mBGP - EVPN Translation of VTEP, GBP-ID, VNID
(scoping of name spaces)
VTEP Group
VNID Tenant Packet
VTEP Group IP Policy
VNID Tenant Packet VTEP Group
IP Policy VNID Tenant Packet
IP Policy
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Where to Go for More Information
ACI Stretched Fabric White Paper
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-
aci-stretched-fabric.html#concept_524263C54D8749F2AD248FAEBA7DAD78
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Thank you