Beruflich Dokumente
Kultur Dokumente
Table of Contents
Deploying the Cisco Intelligent WAN................................................................................................. 1
For design details, see Intelligent WAN Design Summary. For configuration details, see Intelligent WAN Configuration
Files Guide.
For an automated way to deploy IWAN, use the APIC-EM IWAN Application. For more information, see the Cisco
IWAN Application on APIC-EM User Guide.
If want to use TrustSec with your IWAN deployment, see Configuring SGT Propagation in the User-to-Data-
Center Access Control Using TrustSec Deployment Guide.
Web Server-1 DC
(Shared Services) Hub Site
Default VRF INET1 Employees
DHY-M2I2I3- Tunnel 200 RS11-4451-1 RS-ACCESS Default VRF
Contractor WAN-D3750X ASR1002X-2
Server-1
VRF 102
Contractor
DHY-MC-
CSR1000v-1 Client
VRF 102
MPLS2
Tunnel 300
DCI
Remote-Site
Transit-Site (Dual Router)
IoT Client
IoT Server-2
DHY-M1I1- RS32-4451-1 VRF 101
VRF 101 ASR1002X-T1
INET2
Web Server-2 DC Tunnel 400
(Shared Services) Transit Site
Default VRF
DHY-M2I2I3-
Employees
RS-ACCESS Default VRF
Contractor WAN-D3750X-T ASR1002X-T2
Server-2
VRF 102
7089F
ASR1002X-T1
Tunnel 500
VRF 102
This guide uses the following conventions for Commands at a CLI or script prompt:
commands that you enter at the command-line Router# enable
interface (CLI).
Long commands that line wrap are underlined.
Commands to enter at a CLI prompt: Enter them as one command:
configure terminal police rate 10000 pps burst 10000
packets conform-action
Commands that specify a value for a variable:
ntp server 10.10.48.17 Noteworthy parts of system output (or of device
configuration files) are highlighted:
Commands with variables that you must define: interface Vlan64
class-map [highest class name] ip address 10.5.204.5 255.255.255.0
This process assumes that the distribution switch has already been configured following the guidance in the
Campus LAN Layer 2 Access with Simplified Distribution Deployment Guide. Only the procedures required to support
the integration of the WAN aggregation router into the deployment are included.
The hub-site WAN distribution switch is the path to the organizations main campus and DC. It is also an aggre-
gation point between the firewall and rest of the network for multi-VRF traffic. This design uses the virtual local
area network (VLAN) trunking with Layer 2 port-channel interfaces between the distribution switch and connected
devices to carry multi-VRF traffic.
Tech Tip
As a best practice, use the same channel numbering on both sides of the link where possible.
The following figure describes the physical network connections for inside the hub-site.
Core Layer
Firewall
(Inter-VRF
Route Leaking)
WAN Distribution
Layer
Hub-Site Data Center
Servers Interconnect
7090F
Border Router 2 Border Router 1 Hub Master
Controller
The diagram below shows three different colored lines (green, blue and red)one for each of the VRFs described
in the previous section.
IoT-VRF-101
Default VRF Core Layer
CONT-VRF-102 Firewall
(Inter-VRF
Route Leaking)
WAN Distribution
Layer
Hub-Site Data Center
Servers Interconnect
7091F
In this procedure, you configure the definitions for the two new IVRFs and their associated loopbacks.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.21.32.240 255.255.255.255
ip pim sparse-mode
interface Loopback102
description CONT-VRF-102
vrf forwarding CONT-VRF-102
ip address 10.25.32.240 255.255.255.255
ip pim sparse-mode
In this procedure, you create the VLANs and switched virtual interfaces (SVI) interfaces.
Step 1: Create VLANs for the default VRF. You should repeat this step for each VLAN using the parameters in the
table below.
Step 2: Create SVI interfaces for the default VRF. You should repeat this step for each SVI interface using the
parameters in the table below.
Step 3: Create SVI interfaces for IoT-VRF-101 using the parameters in the table below.
Step 4: Create SVI interfaces for CONT-VRF-102 using the parameters in the table below.
In this procedure, you create the port-channels and assign the interface members to the port-channels. This pro-
cedure should be repeated for all the port-channels listed in the table below.
Physical
Connected devices Port-channel VLAN-ID range interface
DHY-MC-CSR1000v-1 (MC) Port-channel 21 VLAN 1100 - 1102 Gig 1/0/15
Gig 2/0/15
DHY-M1I1-ASR1002X-1 (BR1) Port-channel 1 VLAN 1110 - 1112 Gig 1/0/1
Gig 2/0/1
DHY-M2I2I3-ASR1002X-2 (BR2) Port-channel 2 VLAN 1120 - 1122 Gig 1/0/2
Gig 2/0/2
DCI Port-channel 10 VLAN 10 - 12 Gig 1/0/24
Gig 2/0/24
Firewall (for route leaking) Port-channel 20 VLAN 20 - 22 Gig 1/0/23
Gig 2/0/23
Hub-Site Shared Services Port-channel 30 VLAN 30 - 32 Gig 1/0/18
Gig 2/0/18
interface Port-channel21
description DHY-MC-CSR1000v-1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1100-1102
switchport mode trunk
Using static routes, route leaking between the global routing table (GRT) and VRF table is quite easy. You either
provide the next-hop IP address (for multi-access segment) or point the route out of an interface (point-to-point
interface).
In this procedure, you define the next-hop IP address.
Step 1: Define next-hop IP address for multi VRF routes from the GRT.
This design also uses the VLAN trunking with Layer 2 port-channel interfaces between the distribution switch and
connected devices to carry multi-VRF traffic.
The following figure describes the physical network connections for the transit-site.
Core Layer
WAN Distribution
Layer
Transit-Site Data Center
Servers Interconnect
7092F
The diagram below shows three different colored lines (green, blue and red)one for each of the VRFs described
in the previous section.
IoT-VRF-101
Default VRF Core Layer
CONT-VRF-102
WAN Distribution
Layer
Transit-Site Data Center
Servers Interconnect
7093F
Border Router 2 Border Router 1 Hub Master
Controller
Table 7 VRFs and loopback interfaces for transit-site WAN distribution switch
In this procedure, you configure the definitions for the two new IVRFs and their associated loopbacks.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.23.32.240 255.255.255.255
ip pim sparse-mode
interface Loopback102
description CONT-VRF-102
vrf forwarding CONT-VRF-102
ip address 10.27.32.240 255.255.255.255
ip pim sparse-mode
Step 1: Create VLANs for the default VRF. You should repeat this step for each VLAN by using the parameters in
the table below.
Step 2: Create SVI interfaces for the default VRF. You should repeat this step for each SVI interface by using the
parameters in the table below.
Step 3: Create SVI interfaces for IoT-VRF-101 by using the parameters in the table below.
Step 4: Create SVI interfaces for CONT-VRF-102 by using the parameters in the table below.
In this procedure, you create the port-channels and assign the interface members to the port-channels. This pro-
cedure should be repeated for all the port-channels listed in the table.
Physical
Connected devices Port-channel VLAN-ID range interface
DHY-MC-ASR1002X-T1 (MC) Port-channel 21 VLAN 1100 - 1102 Gig 1/0/3
Gig 2/0/3
DHY-M1I1-ASR1002X-T1 (BR1) Port-channel 1 VLAN 1110 - 1112 Gig 1/0/1
Gig 2/0/1
DHY-M2I2I3-ASR1002X-T2 (BR2) Port-channel 2 VLAN 1120 - 1122 Gig 1/0/2
Gig 2/0/2
DCI Port-channel 10 VLAN 10 - 12 Gig 1/0/24
Gig 2/0/24
Transit-site shared services Port-channel 30 VLAN 30 - 32 Gig 1/0/18
Gig 2/0/18
interface Port-channel21
description DHY-MC-ASR1002X-T1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1100-1102
switchport mode trunk
Use this process to configure the DMVPN hub border routers and repeat it for each DMVPN hub router.
The diagram below shows the physical connections between the WAN distribution switch and the hub router, as
well as the connections for the MPLS and Internet transports.
Internet Edge
INET
DMVPN 2
Hub
WAN Distribution Border
Layer Router
DMVPN 1 MPLS
7094F
The diagram below shows the Multi-VRF connections between the WAN distribution switch and the hub router, as
well as the connections for the MPLS and Internet transports.
IoT-VRF-101
Internet Edge
Default VRF
CONT-VRF-102 INET
DMVPN 2
Hub
WAN Distribution Border
Layer Router
DMVPN 1
7095F
MPLS
Table 13 VRFs and loopback interfaces for hub-site border router 1 (DHY-M1I1-ASR1002X-1)
Table 14 VRFs and loopback interfaces for hub-site border router 2 (DHY-M2I2I3-ASR1002X-2)
Table 15 VRFs and loopback interfaces for transit-site border router 1 (DHY-M1I1-ASR1002X-T1)
Table 16 VRFs and loopback interfaces for transit-site border router 2 (DHY-M2I2I3-ASR1002X-T2)
In this procedure, you configure the definitions for the two new IVRFs and their associated loopbacks.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.21.32.241 255.255.255.255
ip pim sparse-mode
interface Loopback102
description CONT-VRF-102
vrf forwarding CONT-VRF-102
ip address 10.25.32.241 255.255.255.255
Step 3: Using the parameters in the tables above, repeat the previous step for all border routers.
In this procedure, you configure the port-channels and port-channel sub-interfaces, then add the local interface
members to the respective port-channels.
interface Port-channel1
description IWAN-D3750X
no ip address
ip nbar protocol-discovery
ip pim sparse-mode
no negotiation auto
interface GigabitEthernet0/0/0
description IWAN-D3750X Gig1/0/1
no ip address
negotiation auto
cdp enable
channel-group 1
interface GigabitEthernet0/0/1
description IWAN-D3750X Gig2/0/1
no ip address
negotiation auto
cdp enable
channel-group 1
interface Port-channel1.1110
encapsulation dot1Q 1110
ip address 10.6.32.2 255.255.255.252
ip nbar protocol-discovery
ip pim sparse-mode
interface Port-channel1.1111
description WAN-D3750X VRF-101
encapsulation dot1Q 1111
vrf forwarding IoT-VRF-101
ip address 10.21.32.2 255.255.255.252
ip nbar protocol-discovery
ip pim sparse-mode
interface Port-channel1.1112
description WAN-D3750X VRF-102
encapsulation dot1Q 1112
vrf forwarding CONT-VRF-102
ip address 10.25.32.2 255.255.255.252
ip nbar protocol-discovery
ip pim sparse-mode
Step 4: Using the parameters in the tables above, repeat the previous steps for all border routers.
To complete the IKEv2 and IPsec configuration for this router, follow the steps in Configuring IKEv2 and IPsec for
a remote site router in Appendix B.
In this procedure, you create mGRE tunnels for DMVPN in each of the new VRFs. This type of tunnel requires a
source interface only. Use the same source interface that you used to connect to the default VRF.
In a multi-VRF deployment, all tunnels with the same tunnel source interface must use the same IPsec profile and
must have the tunnel protection shared command configured. To differentiate tunnels after decryption, a different
tunnel key is used per VRF.
Tunnel
Transport name IPSEC profile Bandwidth FVRF source
MPLS1 DMVPN-IPSEC-PROFILE- 600000 IWAN-TRANSPORT-1 Gig 0/0/3
MPLS1
INET1 DMVPN-IPSEC-PROFILE- 900000 IWAN-TRANSPORT-2 Gig 0/0/4
INET1
MPLS2 DMVPN-IPSEC-PROFILE- 500000 IWAN-TRANSPORT-3 Gig 0/0/3
MPLS2
INET2 DMVPN-IPSEC-PROFILE- 1000000 IWAN-TRANSPORT-4 Gig 0/0/4
INET2
INET4G DMVPN-IPSEC-PROFILE- 400000 IWAN-TRANSPORT-5 Gig 0/0/5
INET4G
Step 1: Create mGRE tunnels for each VRF on border router DHY-M1I1-ASR1002X-1.
Example: DHY-M1I1-ASR1002X-1
interface Tunnel101
description IoT-VRF-101 tunnel via MPLS
bandwidth 600000
vrf forwarding IoT-VRF-101
ip address 10.21.34.1 255.255.254.0
no ip redirects
ip mtu 1400
ip pim nbma-mode
ip pim sparse-mode
interface Tunnel102
description CONT-VRF-102 tunnel via MPLS1
bandwidth 600000
vrf forwarding CONT-VRF-102
ip address 10.25.34.1 255.255.254.0
no ip redirects
ip mtu 1400
ip pim nbma-mode
ip pim sparse-mode
ip nhrp authentication cisco123
ip nhrp network-id 1102
ip nhrp redirect
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/3
tunnel mode gre multipoint
tunnel key 1102
tunnel vrf IWAN-TRANSPORT-1
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-MPLS1 shared
hold-queue 4096 in
hold-queue 4096 out
Step 2: Using the parameters in the tables below, create mGRE tunnels for border router DHY-M2I2-AS-
R1002X-2.
Step 3: Using the parameters in the tables below, create mGRE tunnels for border router DHY-M1I1-ASR1002X-
T1.
Step 4: Using the parameters in the tables below, create mGRE tunnels for border router DHY-M2I2I3-AS-
R1002X-T2.
Use the parameters from the tables below to configure OSPF on the hub border routers.
Example: DHY-M1I1-ASR1002X-1
router ospf 101 vrf IoT-VRF-101
router-id 10.21.32.241
capability vrf-lite
passive-interface default
no passive-interface Port-channel1.1111
network 10.20.0.0 0.1.255.255 area 0
Step 2: Using the parameters in the tables above, repeat the previous step for all border routers.
Use the parameters from the tables below to configure BGP on the hub border routers.
Example: DHY-M1I1-ASR1002X-1
router bgp 65100
bgp listen range 10.21.34.0/23 peer-group MPLS1-SPOKES-VRF101
Example: DHY-M1I1-ASR1002X-1
router bgp 65100
bgp listen range 10.25.34.0/23 peer-group MPLS1-SPOKES-VRF102
Step 5: Using the parameters in the tables below, repeat the previous steps to configure the BGP routing proto-
col for the other border routers.
Step 1: Create the default route and site-specific prefix-lists for all border routers.
Step 2: Using the parameters in the tables below, create the local DC prefix-list for the border routers.
Example: DHY-M1I1-ASR1002X-1
ip prefix-list LOCALDC-PREFIX-101 seq 20 permit 10.21.0.0/16
ip prefix-list LOCALDC-PREFIX-102 seq 20 permit 10.25.0.0/16
Step 3: Using the parameters in the tables below, create the local MC loopback prefix-list for the border routers.
Example: DHY-M1I1-ASR1002X-1
ip prefix-list LOCALMCLOOPBACK-101 seq 10 permit 10.21.32.251/32
ip prefix-list LOCALMCLOOPBACK-102 seq 10 permit 10.25.32.251/32
Procedure 8 Configure the prefix route maps for OSPF and BGP
Step 1: Using the parameters in the tables, create and apply the prefix route-map for BGP on border router
DHY-M1I1-ASR1002X-1.
Route-map IN OUT
Default VRF (MPLS1) MPLS1-IN MPLS1-OUT
IoT-VRF-101 (MPLS1) MPLS1-VRF101-IN MPLS1-VRF101-OUT
CONT-VRF-102 (MPLS1) MPLS1-VRF102-IN MPLS1-VRF102-OUT
Default VRF (INET1) INET1-IN INET1-OUT
IoT-VRF-101 (INET1) INET1-VRF101-IN INET1-VRF101-OUT
CONT-VRF-102 (INET1) INET1-VRF102-IN INET1-VRF102-OUT
Example: DHY-M1I1-ASR1002X-1
ip bgp-community new-format
Step 2: Using the parameters in the tables, repeat the previous step for border router DHY-M2I2I3-ASR1002X-2.
Route-map IN OUT
Default VRF (MPLS2) MPLS2-IN MPLS2-OUT
IoT-VRF-101 (MPLS2) MPLS2-VRF101-IN MPLS2-VRF101-OUT
CONT-VRF-102 (MPLS2) MPLS2-VRF102-IN MPLS2-VRF102-OUT
Default VRF (INET2) INET2-IN INET2-OUT
IoT-VRF-101 (INET2) INET2-VRF101-IN INET2-VRF101-OUT
CONT-VRF-102 (INET2) INET2-VRF102-IN INET2-VRF102-OUT
Default VRF (INET4G) INET4G-IN INET4G-OUT
IoT-VRF-101 (INET4G) INET4G-VRF101-IN INET4G-VRF101-OUT
CONT-VRF-102 (INET4G) INET4G-VRF102-IN INET4G-VRF102-OUT
Step 3: Using the parameters in the tables, repeat the previous step for border router DHY-M1I1-ASR1002X-T1.
Route-map IN OUT
Default VRF (MPLS1) MPLS1-IN MPLS1-OUT
IoT-VRF-101 (MPLS1) MPLS1-VRF101-IN MPLS1-VRF101-OUT
CONT-VRF-102 (MPLS1) MPLS1-VRF102-IN MPLS1-VRF102-OUT
Default VRF (INET1) INET1-IN INET1-OUT
IoT-VRF-101 (INET1) INET1-VRF101-IN INET1-VRF101-OUT
CONT-VRF-102 (INET1) INET1-VRF102-IN INET1-VRF102-OUT
Step 4: Using the parameters in the tables, repeat the previous step for border router DHY-M2I2I3-ASR1002X-
T2.
Route-map IN OUT
Default VRF (MPLS2) MPLS2-IN MPLS2-OUT
IoT-VRF-101 (MPLS2) MPLS2-VRF101-IN MPLS2-VRF101-OUT
CONT-VRF-102 (MPLS2) MPLS2-VRF102-IN MPLS2-VRF102-OUT
Default VRF (INET2) INET2-IN INET2-OUT
IoT-VRF-101 (INET2) INET2-VRF101-IN INET2-VRF101-OUT
CONT-VRF-102 (INET2) INET2-VRF102-IN INET2-VRF102-OUT
Default VRF (INET4G) INET4G-IN INET4G-OUT
IoT-VRF-101 (INET4G) INET4G-VRF101-IN INET4G-VRF101-OUT
CONT-VRF-102 (INET4G) INET4G-VRF102-IN INET4G-VRF102-OUT
Step 5: Using the parameters in the table below, create and apply the prefix route-map for OSPF on hub-site
border router DHY-M1I1-ASR1002X-1.
Table 58 OSPF route map information for both hub-site border routers
Example: DHY-M1I1-ASR1002X-1
route-map REDIST-BGP-TO-OSPF-101 permit 10
description Secondary POP2 with higher Metric
match community POP2-SPOKES
set metric 2000
set metric-type type-1
set tag 0
Step 6: Repeat the previous step for hub-site border router DHY-M2I2I3-ASR1002X-2.
Step 7: Using the parameters in the table below, repeat the previous step for transit-site border router DHY-
M1I1-ASR1002X-T1.
Table 59 OSPF route map information for both transit-site border routers
Step 8: Repeat the previous step for transit-site border router DHY-M2I2I3-ASR1002X-T2.
Step 1: Configure the static null routes and EOT for both hub-site border routers.
Step 2: Configure the static null routes and EOT for both transit-site border routers.
8. Create and apply the prefix route maps for OSPF and BGP
Use this process to connect remote site router to the WAN transports.
The following diagram shows the physical connection between the remote site border-router and the MPLS and
Internet transports.
DMVPN 1 DMVPN 2
MPLS Internet
Branch Master
Controller/
Branch
7096F
Border Router
The following diagram shows the logical view of the multi-VRF traffic between the remote site border-router and
the tunnels over MPLS and Internet transports.
MPLS Internet
Branch Master
Controller/
Branch
Border Router
7097F
In this procedure, you configure the IVRF definition and their associated loopback interfaces for the remote-site
border routers.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
Example: RS11-2921
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.201.241.11 255.255.255.255
ip pim sparse-mode
hold-queue 1024 in
hold-queue 1024 out
interface Loopback102
description CONT-VRF-102
In this procedure, you configure the router with connectivity to the access layer switch. In the access layer design,
the remote sites use collapsed routing, with 802.1Q trunk interfaces to the LAN access layer.
Step 1: Create VLANs for data and voice traffic for the all VRFs.
vlan 521
name IoT-VRF-101 Data
vlan 522
name CONT-VRF-102 Data
vlan 531
name IoT-VRF-101 Voice
vlan 532
name CONT-VRF-102 Voice
Use an 802.1Q trunk for the connection, which allows the router to provide the Layer 3 services to all the VLANs
defined on the access layer switch. The VLANs allowed on the trunk are pruned to only the VLANs that are active
on the access switch. DHCP snooping and ARP inspection are set to trust.
interface GigabitEthernet1/0/48
description RS11-2921 Gig0/2
switchport trunk allowed vlan 521-522,531-532
switchport mode trunk
ip arp inspection trust
spanning-tree portfast trunk
logging event link-status
logging event trunk-status
ip dhcp snooping trust
load-interval 30
no shutdown
The Cisco Catalyst 3750 Series Switch requires the switchport trunk encapsulation dot1q command.
This design uses an IP addressing convention with the default gateway router assigned an IP address and IP mask
combination of N.N.N.1 255.255.255.0 where N.N.N is the IP network and 1 is the IP host.
When you are using a centralized DHCP server, your routers with LAN interfaces connected to a LAN using DHCP
for end-station IP addressing must use an IP helper.
Example: RS11-2911
interface GigabitEthernet0/2.521
description IoT-VRF-101 Data
encapsulation dot1Q 521
vrf forwarding IoT-VRF-101
ip address 10.22.2.1 255.255.255.0
ip helper-address global 10.4.48.10
ip pim sparse-mode
interface GigabitEthernet0/2.522
description CONT-VRF-102 Data
encapsulation dot1Q 522
vrf forwarding CONT-VRF-102
ip address 10.26.2.1 255.255.255.0
ip helper-address global 10.4.48.10
ip pim sparse-mode
interface GigabitEthernet0/2.531
description IoT-VRF-101 Voice
encapsulation dot1Q 531
vrf forwarding IoT-VRF-101
ip address 10.22.3.1 255.255.255.0
ip helper-address global 10.4.48.10
ip pim sparse-mode
interface GigabitEthernet0/2.532
description CONT-VRF-102 Voice
encapsulation dot1Q 532
vrf forwarding CONT-VRF-102
ip address 10.26.3.1 255.255.255.0
ip helper-address global 10.4.48.10
ip pim sparse-mode
To complete the IKEv2 and IPsec configuration for this router, follow the steps in Configuring IKEv2 and IPsec for
a remote site router in Appendix B.
In this procedure, you configure mGRE tunnels for DMVPN in each VRF. This type of tunnel requires a source
interface only. Use the same source interface that you use to connect to the MPLS or Internet.
In a multi-VRF deployment, all tunnels with the same tunnel source interface must use the same IPsec profile and
must have the tunnel protection shared command configured. To differentiate tunnels after decryption, a different
tunnel key is used per VRF.
Tunnel
Transport name IPSEC profile Bandwidth FVRF source
MPLS1 DMVPN-IPSEC-PROFILE-MPLS1 20000 IWAN-TRANSPORT-1 Gig 0/0
INET1 DMVPN-IPSEC-PROFILE-INET1 50000 IWAN-TRANSPORT-2 Gig 0/1
Example: RS11-2911
interface Tunnel101
description IoT-VRF-101 tunnel via MPLS1
bandwidth 20000
vrf forwarding IoT-VRF-101
interface Tunnel102
description CONT-VRF-102 tunnel via MPLS1
bandwidth 20000
vrf forwarding CONT-VRF-102
ip address 10.25.34.11 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
if-state nhrp
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 1102
tunnel vrf IWAN-TRANSPORT-1
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-MPLS1 shared
interface Tunnel201
description IoT-VRF-101 tunnel via INET1
bandwidth 50000
vrf forwarding IoT-VRF-101
ip address 10.21.36.11 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key 1201
tunnel vrf IWAN-TRANSPORT-2
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-INET1 shared
interface Tunnel202
description CONT-VRF-102 tunnel via INET1
bandwidth 50000
vrf forwarding CONT-VRF-102
ip address 10.25.36.11 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key 1202
tunnel vrf IWAN-TRANSPORT-2
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-INET1 shared
The DMVPN hub router is the NHRP server for all of the spokes. NHRP is used by remote routers to determine the
tunnel destinations for peers attached to the mGRE tunnel.
The spoke router requires several additional configuration statements in order to define the NHRP server and
NHRP map statements for the DMVPN hub router mGRE tunnel IP address. Spoke routers require the NHRP static
multicast mapping.
When hub BRs are added for horizontal scaling or a second DC is added as a transit site, spoke routers require
additional NHS statements for each BR in their environment. The configuration details are covered in subsequent
sections of this guide.
The value used for the NHS is the mGRE tunnel address for the DMVPN hub router. The map entries must be set
to the outside NAT value of the DMVPN hub, as configured on the Cisco ASA 5500 appliance. This design uses
the values shown in the following tables.
Table 64 DMVPN tunnel NHRP parameters for border router RS11-2921 (MPLS1)
Table 65 DMVPN tunnel NHRP parameters for border router RS11-2921 (INET1)
NHRP requires all devices within a DMVPN cloud to use the same network ID and authentication key. The NHRP
cache holdtime should be configured to 600 seconds.
This design supports DMVPN spoke routers that receive their external IP addresses through DHCP. It is pos-
sible for these routers to acquire different IP addresses after a reload. When the router attempts to register with
the NHRP server, it may appear as a duplicate to an entry already in the cache and be rejected. The registration
no-unique option allows you to overwrite existing cache entries. This feature is only required on NHRP clients
(DMVPN spoke routers). The if-state nhrp option ties the tunnel line-protocol state to the reachability of the
NHRP NHS, and if the NHS is unreachable, the tunnel line-protocol state changes to down. This feature is used in
conjunction with EOT.
Example: RS11-2921
interface Tunnel101
ip nhrp authentication cisco123
ip nhrp network-id 1101
ip nhrp holdtime 600
ip nhrp nhs 10.21.34.1 nbma 192.168.6.1 multicast
ip nhrp nhs 10.21.34.2 nbma 192.168.6.41 multicast
ip nhrp registration no-unique
ip nhrp shortcut
no nhrp route-watch
if-state nhrp
interface Tunnel102
ip nhrp authentication cisco123
ip nhrp network-id 1102
ip nhrp holdtime 600
ip nhrp nhs 10.25.34.1 nbma 192.168.6.1 multicast
ip nhrp nhs 10.25.34.2 nbma 192.168.6.41 multicast
ip nhrp registration no-unique
ip nhrp shortcut
no nhrp route-watch
if-state nhrp
interface Tunnel201
ip nhrp authentication cisco123
ip nhrp network-id 1201
ip nhrp holdtime 600
ip nhrp nhs 10.21.36.1 nbma 172.16.140.1 multicast
ip nhrp nhs 10.21.36.2 nbma 172.16.140.2 multicast
ip nhrp registration no-unique
ip nhrp shortcut
no nhrp route-watch
if-state nhrp
interface Tunnel202
ip nhrp authentication cisco123
Step 1: Configure an OSPF routing protocol on the spoke router for the new VRFs.
Example: RS11-2921
router ospf 101 vrf IoT-VRF-101
router-id 10.201.241.11
redistribute bgp 65100 subnets route-map REDIST-BGP-TO-OSPF
passive-interface default
no passive-interface GigabitEthernet0/2.521
no passive-interface GigabitEthernet0/2.531
network 10.22.0.0 0.0.7.255 area 0
network 10.201.241.11 0.0.0.0 area 0
default-information originate
Hub tunnel
VRF name Router ID Aggregate-subnet Peer-group address Tunnel ID
Default VRF 10.255.241.11 10.7.0.0/21 MPLS1-HUB 10.6.34.1 100
10.6.34.2
INET1-HUB 10.6.36.1 200
10.6.36.2
IoT-VRF-101 10.201.241.11 10.22.0.0/21 MPLS1-HUB- 10.21.34.1 101
VRF101
10.21.34.2
INET1-HUB- 10.21.36.1 201
VRF101
10.21.36.2
Step 1: Using the parameters in the table above, configure BGP routing protocol for VRFs IoT-VRF-101 and
CONT-VRF-102.
Example: IoT-VRF-101RS11-2921
router bgp 65100
address-family ipv4 vrf IoT-VRF-101
bgp router-id 10.201.241.11
aggregate-address 10.22.0.0 255.255.248.0 summary-only
redistribute connected
neighbor INET1-HUB-VRF101 peer-group
neighbor INET1-HUB-VRF101 remote-as 65100
neighbor INET1-HUB-VRF101 description To IWAN INET1 Hub Router
neighbor INET1-HUB-VRF101 update-source Tunnel201
neighbor INET1-HUB-VRF101 timers 20 60
neighbor INET1-HUB-VRF101 send-community
neighbor INET1-HUB-VRF101 next-hop-self all
neighbor INET1-HUB-VRF101 weight 50000
neighbor INET1-HUB-VRF101 soft-reconfiguration inbound
Example: CONT-VRF-102RS11-2921
router bgp 65100
address-family ipv4 vrf CONT-VRF-102
bgp router-id 10.202.241.11
aggregate-address 10.26.0.0 255.255.248.0 summary-only
redistribute connected
neighbor INET1-HUB-VRF102 peer-group
neighbor INET1-HUB-VRF102 remote-as 65100
neighbor INET1-HUB-VRF102 description To IWAN INET1 Hub Router
neighbor INET1-HUB-VRF102 update-source Tunnel202
neighbor INET1-HUB-VRF102 timers 20 60
neighbor INET1-HUB-VRF102 send-community
neighbor INET1-HUB-VRF102 next-hop-self all
neighbor INET1-HUB-VRF102 weight 50000
neighbor INET1-HUB-VRF102 soft-reconfiguration inbound
neighbor MPLS1-HUB-VRF102 peer-group
Step 1: Create the local loopback prefix-list for each VRF on remote-site border router.
Example: RS11-2921
ip prefix-list LOCAL-LOOPBACKS-101 seq 10 permit 10.201.241.11/32
ip prefix-list LOCAL-LOOPBACKS-102 seq 10 permit 10.202.241.11/32
ip prefix-list LOCAL-SUBNETS-101 seq 10 permit 10.22.0.0/21
ip prefix-list LOCAL-SUBNETS-102 seq 10 permit 10.26.0.0/21
Procedure 8 Create and apply the prefix route maps for OSPF and BGP
Step 1: Create the prefix route-map SPOKE-OUT for BGP on remote-site border router.
3. Enable EOT
The following diagram shows the logical view of the multi-VRF traffic between both of the remote-site border
routers and up to five transports.
7099F
Procedure 1 Configure the access layer HSRP
You need to configure HSRP to enable the use of a virtual IP (VIP) as a default gateway that is shared between
two routers. The HSRP active router is the router connected to the primary WAN link and the HSRP standby router
is the router connected to the secondary WAN link
In this procedure, you configure the router with connectivity to the access layer switch. In the access layer design,
the remote sites use collapsed routing, with 802.1Q trunk interfaces to the LAN access layer.
The router with the higher standby priority value is elected as the HSRP active router. The preempt option allows a
router with a higher priority to become the HSRP active, without waiting for a scenario where there is no router in
the HSRP active state. The following table shows the relevant HSRP parameters for the router configuration.
Router HSRP role VIP Real IP address HSRP priority PIM DR priority
Primary Active .1 .2 110 110
Secondary Standby .1 .3 105 105
Step 1: Using the parameters in the table above, create the subinterfaces for all data or voice traffic.
interface GigabitEthernet0/0/2.521
ip address 10.22.146.2 255.255.255.0
ip pim dr-priority 110
standby version 2
standby 1 ip 10.22.146.1
standby 1 priority 110
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.522
ip address 10.26.146.2 255.255.255.0
ip pim dr-priority 110
standby version 2
standby 1 ip 10.26.146.1
standby 1 priority 110
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.531
ip address 10.22.147.2 255.255.255.0
ip pim dr-priority 110
standby version 2
standby 1 ip 10.22.147.1
standby 1 priority 110
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.532
ip address 10.26.147.2 255.255.255.0
ip pim dr-priority 110
standby version 2
standby 1 ip 10.26.147.1
standby 1 priority 110
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.511
description IoT-VRF-101 Transit Net
encapsulation dot1Q 511
vrf forwarding IoT-VRF-101
ip address 10.22.144.9 255.255.255.252
ip pim sparse-mode
interface GigabitEthernet0/0/2.512
description CONT-VRF-102 Transit Net
encapsulation dot1Q 512
vrf forwarding CONT-VRF-102
ip address 10.26.144.9 255.255.255.252
ip pim sparse-mode
If the VLAN does not already exist on the access layer switch, configure it now.
vlan 511
name VRF-101_TransitNet
vlan 512
name VRF-102_TransitNet
Step 4: Add transit network VLANs to the existing access layer switch trunk.
interface GigabitEthernet0/2
switchport trunk allowed vlan add 511,512
Step 1: Turn off passive-interface for the transit network LAN interfaces.
The HSRP active router remains the active router unless the router is reloaded or fails. Having the HSRP router
remain as the active router can lead to undesired behavior. If the primary WAN transport were to fail, the HSRP
active router would learn an alternate path through the transit network to the HSRP standby router and begin to
forward traffic across the alternate path. This is sub-optimal routing, and you can address it by using EOT.
The HSRP active router can track the state of its DMVPN tunnel interface. If the tunnel line-protocol state changes
to down, this implies that the path to the primary site is no longer viable. This is a benefit of using the if-state
nhrp feature with a DMVPN tunnel configuration.
This procedure is valid only on the router connected to the primary transport.
A tracked object is created based on tunnel line-protocol state. If the tunnel is up, the tracked object status is Up;
if the tunnel is down, the tracked object status is Down. A short delay is added after the tunnel interface comes
back up in order to ensure that routing has converged properly before changing the HSRP active router.
Step 2: Link HSRP with the tracked object. All data or voice subinterfaces should enable HSRP tracking. HSRP
can monitor the tracked object status. If the status is down, the HSRP priority is decremented by the configured
priority. If the decrease is large enough, the HSRP standby router preempts.
interface GigabitEthernet0/0/2.521
standby 1 track 50 decrement 10
interface GigabitEthernet0/0/2.522
standby 1 track 50 decrement 10
interface GigabitEthernet0/0/2.531
standby 1 track 50 decrement 10
interface GigabitEthernet0/0/2.532
standby 1 track 50 decrement 10
9. Create and apply the prefix route maps for OSPF and BGP
Use these procedures when configuring the second router of a dual-router design.
This set of procedures includes the additional steps necessary to configure a second router as a DMVPN spoke
router when the first router has already been configured with the process Configuring Remote-Site DMVPN
Spoke Router.
If you have not done so already, the previous process, Modifying the First Router for Dual Router Design, must
be completed.
The parameters in the tables below are used in the following procedures.
In this procedure, you configure the IVRF definitions and their associated loopbacks for the remote-site border
router.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
Example: RS32-4451x-2
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.201.243.32 255.255.255.255
ip pim sparse-mode
hold-queue 1024 in
hold-queue 1024 out
interface Loopback102
description CONT-VRF-102
vrf forwarding CONT-VRF-102
ip address 10.202.243.32 255.255.255.255
hold-queue 1024 in
hold-queue 1024 out
In this procedure, you configure the router with connectivity to the access layer switch. In the access layer design,
the remote sites use collapsed routing, with 802.1Q trunk interfaces to the LAN access layer.
This design uses an IP addressing convention with the default gateway router assigned an IP address and IP mask
combination of N.N.N.1 255.255.255.0 where N.N.N is the IP network and 1 is the IP host.
When you are using a centralized DHCP server, your routers with LAN interfaces connected to a LAN using DHCP
for end-station IP addressing must use an IP helper.
This remote-site DMVPN spoke router is the second router of a dual-router design and HSRP is configured at the
access layer. The actual interface IP assignments are configured in the following procedure.
interface GigabitEthernet0/0/2.521
encapsulation dot1Q 521
vrf forwarding IoT-VRF-101
ip helper-address global 10.4.48.10
ip pim sparse-mode
interface GigabitEthernet0/0/2.522
encapsulation dot1Q 522
vrf forwarding CONT-VRF-102
ip helper-address global 10.4.48.10
ip pim sparse-mode
interface GigabitEthernet0/0/2.531
encapsulation dot1Q 531
vrf forwarding IoT-VRF-101
ip helper-address global 10.4.48.10
ip pim sparse-mode
interface GigabitEthernet0/0/2.532
encapsulation dot1Q 532
vrf forwarding CONT-VRF-102
ip helper-address global 10.4.48.10
ip pim sparse-mode
You configure HSRP to enable a VIP that you use as a default gateway that is shared between two routers. The
HSRP active router is the router connected to the primary carrier and the HSRP standby router is the router con-
nected to the secondary carrier or backup link.
Step 1: Configure the HSRP standby router with a standby priority that is lower than the HSRP active router.
The router with the higher standby priority value is elected as the HSRP active router. The preempt option allows
a router with a higher priority to become the HSRP active, without waiting for a scenario where there is no router
in the HSRP active state. The relevant HSRP parameters for the router configuration are shown in the following
table.
Router HSRP role VIP Real IP address HSRP priority PIM DR priority
Primary Active .1 .2 110 110
Secondary Standby .1 .3 105 105
Step 2: Create the subinterfaces for both data and voice VLANs for each VRF.
interface GigabitEthernet0/0/2.521
description IoT-VRF-101 Data
ip address 10.22.146.3 255.255.255.0
ip pim dr-priority 105
standby version 2
standby 1 ip 10.22.146.1
standby 1 priority 105
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.522
description CONT-VRF-102 Data
ip address 10.26.146.3 255.255.255.0
ip pim dr-priority 105
standby version 2
standby 1 ip 10.26.146.1
standby 1 priority 105
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.531
description IoT-VRF-101 Voice
ip address 10.22.147.3 255.255.255.0
ip pim dr-priority 105
standby version 2
standby 1 ip 10.22.147.1
standby 1 priority 105
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.532
description CONT-VRF-102 Voice
ip address 10.26.147.3 255.255.255.0
ip pim dr-priority 105
standby version 2
standby 1 ip 10.26.147.1
standby 1 priority 105
standby 1 preempt
standby 1 authentication md5 key-string c1sco123
interface GigabitEthernet0/0/2.511
description IoT-VRF-101 Transit Net
encapsulation dot1Q 511
vrf forwarding IoT-VRF-101
ip address 10.22.144.10 255.255.255.252
ip pim sparse-mode
interface GigabitEthernet0/0/2.512
description CONT-VRF-102 Transit Net
encapsulation dot1Q 512
vrf forwarding CONT-VRF-102
ip address 10.26.144.10 255.255.255.252
ip pim sparse-mode
To complete the IKEv2 and IPsec configuration for this router, follow the steps in Configuring IKEv2 and IPsec for
a remote site router in Appendix B.
In this procedure, you configure mGRE tunnels for DMVPN in each VRF. This type of tunnel requires a source
interface only. Use the same source interface that you use to connect to the MPLS or Internet.
In a multi-VRF deployment, all tunnels with the same tunnel source interface must use the same IPsec profile and
must have the tunnel protection shared command configured. To differentiate tunnels after decryption, a different
tunnel key is used per VRF.
Tunnel
Transport name IPSEC profile Bandwidth FVRF source
MPLS2 DMVPN-IPSEC-PROFILE-MPLS2 50000 IWAN-TRANSPORT-3 Gig 0/0/0
INET2 DMVPN-IPSEC-PROFILE-INET2 300000 IWAN-TRANSPORT-4 Gig 0/0/1
Step 1: Using the parameters in the tables above, configure the tunnel interfaces for each tunnel ID.
Example: RS32-4451x-2
interface Tunnel301
description IoT-VRF-101 tunnel via MPLS2
bandwidth 50000
vrf forwarding IoT-VRF-101
ip address 10.21.38.32 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
if-state nhrp
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1301
tunnel vrf IWAN-TRANSPORT-3
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-MPLS2 shared
interface Tunnel302
description CONT-VRF-102 tunnel via MPLS2
bandwidth 50000
vrf forwarding CONT-VRF-102
ip address 10.25.38.32 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
if-state nhrp
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1302
tunnel vrf IWAN-TRANSPORT-3
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-MPLS2 shared
interface Tunnel401
description IoT-VRF-101 tunnel via INET2
bandwidth 300000
vrf forwarding IoT-VRF-101
ip address 10.21.40.32 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
interface Tunnel402
description CONT-VRF-102 tunnel via INET2
bandwidth 300000
vrf forwarding CONT-VRF-102
ip address 10.25.40.32 255.255.254.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/1
tunnel mode gre multipoint
tunnel key 1402
tunnel vrf IWAN-TRANSPORT-4
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE-INET2 shared
The DMVPN hub router is the NHRP server for all of the spokes. NHRP is used by remote routers to determine the
tunnel destinations for peers attached to the mGRE tunnel.
The spoke router requires several additional configuration statements in order to define the NHRP server and
NHRP map statements for the DMVPN hub router mGRE tunnel IP address. Spoke routers require the NHRP static
multicast mapping.
When hub BRs are added for horizontal scaling or a second DC is added as a transit site, spoke routers require
additional NHS statements for each BR in their environment. The configuration details are covered in subsequent
sections of this guide.
The value used for the NHS is the mGRE tunnel address for the DMVPN hub router. The map entries must be set
to the outside NAT value of the DMVPN hub, as configured on the Cisco ASA 5500. This design uses the values
shown in the following tables.
Table 77 DMVPN tunnel NHRP parameters for border router RS32-4451x-2 (MPLS2)
Table 78 DMVPN tunnel NHRP parameters for border router RS32-4451x-2 (INET2)
NHRP requires all devices within a DMVPN cloud to use the same network ID and authentication key. The NHRP
cache holdtime should be configured to 600 seconds.
This design supports DMVPN spoke routers that receive their external IP addresses through DHCP. It is possible
for these routers to acquire different IP addresses after a reload. When the router attempts to register with the
NHRP server, the router may appear as a duplicate to an entry already in the cache and be rejected. The reg-
istration no-unique option allows you to overwrite existing cache entries. This feature is only required on NHRP
clients (DMVPN spoke routers). The if-state nhrp option ties the tunnel line-protocol state to the reachability of
the NHRP NHS, and if the NHS is unreachable, the tunnel line-protocol state changes to down. This feature is
used in conjunction with EOT.
Example: RS32-4451x-2
interface Tunnel301
ip nhrp authentication cisco123
ip nhrp network-id 1301
ip nhrp holdtime 600
ip nhrp nhs 10.21.38.1 nbma 192.168.7.1 multicast
ip nhrp nhs 10.21.38.2 nbma 192.168.7.41 multicast
ip nhrp registration no-unique
ip nhrp shortcut
no nhrp route-watch
if-state nhrp
interface Tunnel302
ip nhrp authentication cisco123
ip nhrp network-id 1302
ip nhrp holdtime 600
ip nhrp nhs 10.25.38.1 nbma 192.168.7.1 multicast
ip nhrp nhs 10.25.38.2 nbma 192.168.7.41 multicast
ip nhrp registration no-unique
ip nhrp shortcut
no nhrp route-watch
if-state nhrp
interface Tunnel401
ip nhrp authentication cisco123
ip nhrp network-id 1401
ip nhrp holdtime 600
ip nhrp nhs 10.21.40.1 nbma 172.17.140.1 multicast
ip nhrp nhs 10.21.40.2 nbma 172.17.140.2 multicast
ip nhrp registration no-unique
ip nhrp shortcut
no nhrp route-watch
if-state nhrp
interface Tunnel402
ip nhrp authentication cisco123
Example: RS32-4451x-2
router ospf 101 vrf IoT-VRF-101
router-id 10.201.243.32
redistribute bgp 65100 subnets route-map REDIST-BGP-TO-OSPF
passive-interface default
no passive-interface GigabitEthernet0/0/2.521
no passive-interface GigabitEthernet0/0/2.531
no passive-interface GigabitEthernet0/0/2.511
network 10.22.144.0 0.0.7.255 area 0
network 10.201.243.32 0.0.0.0 area 0
default-information originate
Hub tunnel
VRF name Router ID Aggregate-subnet Peer-group address Tunnel ID
Default VRF 10.255.243.32 10.7.144.0/21 MPLS2-HUB 10.6.38.1 300
10.6.38.2
INET2-HUB 10.6.40.1 400
10.6.40.2
IoT-VRF-101 10.201.243.32 10.22.144.0/21 MPLS2-HUB- 10.21.38.1 301
VRF101
10.21.38.2
INET2-HUB- 10.21.40.1 401
VRF101
10.21.40.2
Step 1: Configure BGP routing protocol for VRFs IoT-VRF-101 and CONT-VRF-102.
Example: IoT-VRF-101RS32-4451x-2
router bgp 65100
address-family ipv4 vrf IoT-VRF-101
bgp router-id 10.201.243.32
aggregate-address 10.22.144.0 255.255.248.0 summary-only
redistribute connected
neighbor INET2-HUB-VRF101 peer-group
neighbor INET2-HUB-VRF101 remote-as 65100
neighbor INET2-HUB-VRF101 description To IWAN INET2 Hub Router
neighbor INET2-HUB-VRF101 update-source Tunnel401
neighbor INET2-HUB-VRF101 timers 20 60
neighbor INET2-HUB-VRF101 send-community
neighbor INET2-HUB-VRF101 next-hop-self all
neighbor INET2-HUB-VRF101 weight 50000
neighbor INET2-HUB-VRF101 soft-reconfiguration inbound
neighbor MPLS2-HUB-VRF101 peer-group
neighbor MPLS2-HUB-VRF101 remote-as 65100
neighbor MPLS2-HUB-VRF101 description To IWAN MPLS2 Hub Router
neighbor MPLS2-HUB-VRF101 update-source Tunnel301
neighbor MPLS2-HUB-VRF101 timers 20 60
neighbor MPLS2-HUB-VRF101 send-community
neighbor MPLS2-HUB-VRF101 next-hop-self all
neighbor MPLS2-HUB-VRF101 weight 50000
neighbor MPLS2-HUB-VRF101 soft-reconfiguration inbound
neighbor 10.21.38.1 peer-group MPLS2-HUB-VRF101
neighbor 10.21.38.1 activate
neighbor 10.21.38.2 peer-group MPLS2-HUB-VRF101
neighbor 10.21.38.2 activate
neighbor 10.21.40.1 peer-group INET2-HUB-VRF101
neighbor 10.21.40.1 activate
neighbor 10.21.40.2 peer-group INET2-HUB-VRF101
neighbor 10.21.40.2 activate
distance bgp 201 89 89
exit-address-family
Example: CONT-VRF-102RS32-4451x-2
router bgp 65100
address-family ipv4 vrf IoT-VRF-101
bgp router-id 10.202.243.32
aggregate-address 10.26.144.0 255.255.248.0 summary-only
redistribute connected
neighbor INET2-HUB-VRF102 peer-group
neighbor INET2-HUB-VRF102 remote-as 65100
neighbor INET2-HUB-VRF102 description To IWAN INET2 Hub Router
neighbor INET2-HUB-VRF102 update-source Tunnel402
neighbor INET2-HUB-VRF102 timers 20 60
neighbor INET2-HUB-VRF102 send-community
neighbor INET2-HUB-VRF102 next-hop-self all
neighbor INET2-HUB-VRF102 weight 50000
neighbor INET2-HUB-VRF102 soft-reconfiguration inbound
neighbor MPLS2-HUB-VRF102 peer-group
neighbor MPLS2-HUB-VRF102 remote-as 65100
neighbor MPLS2-HUB-VRF102 description To IWAN MPLS2 Hub Router
neighbor MPLS2-HUB-VRF102 update-source Tunnel302
neighbor MPLS2-HUB-VRF102 timers 20 60
neighbor MPLS2-HUB-VRF102 send-community
neighbor MPLS2-HUB-VRF102 next-hop-self all
neighbor MPLS2-HUB-VRF102 weight 50000
neighbor MPLS2-HUB-VRF102 soft-reconfiguration inbound
neighbor 10.25.38.1 peer-group MPLS2-HUB-VRF102
neighbor 10.25.38.1 activate
neighbor 10.25.38.2 peer-group MPLS2-HUB-VRF102
neighbor 10.25.38.2 activate
neighbor 10.25.40.1 peer-group INET2-HUB-VRF102
neighbor 10.25.40.1 activate
neighbor 10.25.40.2 peer-group INET2-HUB-VRF102
neighbor 10.25.40.2 activate
distance bgp 201 89 89
exit-address-family
Step 1: Create the local loopback prefix-list for each VRF on remote-site border router.
Example: RS32-4451x-2
ip prefix-list LOCAL-LOOPBACKS-101 seq 10 permit 10.201.241.32/32
ip prefix-list LOCAL-LOOPBACKS-101 seq 20 permit 10.201.243.32/32
Step 2: Create the local subnet prefix-list for each VRF on remote-site border router.
Example: RS32-4451x-2
ip prefix-list LOCAL-SUBNETS-101 seq 10 permit 10.22.144.0/21
ip prefix-list LOCAL-SUBNETS-102 seq 10 permit 10.26.144.0/21
Procedure 9 Create and apply the prefix route maps for OSPF and BGP
Step 1: Create the prefix route-map SPOKE-OUT for BGP on remote-site border router.
Table 83 Prefix route-map SPOKE-OUT information for remote-site border router RS32-4451x-2
Step 3: Create the route-map REDIST-BGP-TO-OSPF for OSPF on remote-site border router.
The following diagram uses dotted lines to show the end-to-end traffic flow from remote sites to the hub-site.
The dotted-line color indicates how the firewall directs the traffic flow on two sides:
The blue dotted line shows the firewall directing the example IoT VRF traffic to the shared services in the
global network.
The green dotted line shows the traffic from the example IoT client to/from the WAN distribution switch,
routed to the firewall.
Figure 12 Firewall managing the traffic flow between the hub and remote sites
IoT-VRF-101 Default VRF Traffic Path
Default VRF IoT-VRF-101 Traffic Path
CONT-VRF-102
MPLS1
Hub-Site Tunnel 100 Remote-Site
(Single Router)
WAN-DIST-Firewall IoT Client
IoT Server-1 VRF 101
DHY-M1I1-
VRF 101 ASR1002X-1
Web Server-1 DC
(Shared Services) Hub Site
Default VRF INET1 Employees
DHY-M2I2I3- Tunnel 200 RS11-4451-1 RS-ACCESS Default VRF
Contractor WAN-D3750X ASR1002X-2
Server-1
VRF 102
Contractor
DHY-MC-
CSR1000v-1 Client
VRF 102
MPLS2
Tunnel 300
DCI
Remote-Site
Transit-Site (Dual Router)
IoT Client
IoT Server-2
DHY-M1I1- RS32-4451-1 VRF 101
VRF 101 ASR1002X-T1
INET2
Web Server-2 DC Tunnel 400
(Shared Services) Transit Site
Default VRF
DHY-M2I2I3-
Employees
RS-ACCESS Default VRF
Contractor WAN-D3750X-T ASR1002X-T2
Server-2
VRF 102
7100F
ASR1002X-T1
Tunnel 500
VRF 102
The following diagram uses dotted lines to show the end-to-end traffic flow from remote sites to the transit-site.
The dotted-line color indicates how the firewall directs the traffic flow on two sides:
The blue dotted line shows the firewall directing the example IoT VRF traffic to the shared services in the
global network in the transit-site.
The green dotted line shows the traffic from the example IoT client to/from the WAN distribution switch,
routed to the firewall.
Figure 13 Firewall at the hub-site managing the traffic flow between the transit and remote sites
IoT-VRF-101 Default VRF Traffic Path
Default VRF IoT-VRF-101 Traffic Path
CONT-VRF-102
MPLS1
Hub-Site Tunnel 100 Remote-Site
(Single Router)
WAN-DIST-Firewall IoT Client
IoT Server-1 VRF 101
DHY-M1I1-
VRF 101 ASR1002X-1
Web Server-1 DC
(Shared Services) Hub Site
Default VRF INET1 Employees
DHY-M2I2I3- Tunnel 200 RS11-4451-1 RS-ACCESS Default VRF
Contractor WAN-D3750X ASR1002X-2
Server-1
VRF 102
Contractor
DHY-MC-
CSR1000v-1 Client
VRF 102
MPLS2
Tunnel 300
DCI
Remote-Site
Transit-Site (Dual Router)
IoT Client
IoT Server-2
DHY-M1I1- RS32-4451-1 VRF 101
VRF 101 ASR1002X-T1
INET2
Web Server-2 DC Tunnel 400
(Shared Services) Transit Site
Default VRF
DHY-M2I2I3-
Employees
RS-ACCESS Default VRF
Contractor WAN-D3750X-T ASR1002X-T2
Server-2
VRF 102
7101F
ASR1002X-T1
Tunnel 500
VRF 102
In this procedure, you configure the connection between the Cisco ASA firewall and the hub-site distribution
switch.
interface GigabitEthernet0/0
description WAN-D3750X (gig 1/0/23)
channel-group 20 mode on
no nameif
no security-level
no ip address
no shutdown
interface GigabitEthernet0/1
description WAN-D3750X (gig 2/0/23)
channel-group 20 mode on
no nameif
no security-level
no ip address
no shutdown
interface Port-channel20
description WAN-D3750X
lacp max-bundle 8
no nameif
no security-level
no ip address
no shutdown
interface Port-channel20.20
vlan 20
nameif DC-INSIDE
security-level 100
ip address 10.6.32.26 255.255.255.248
interface Port-channel20.21
vlan 21
nameif VRF101-OUTSIDE
security-level 100
ip address 10.21.32.26 255.255.255.248
interface Port-channel20.22
vlan 22
nameif VRF102-OUTSIDE
security-level 100
ip address 10.25.32.26 255.255.255.248
In this procedure, you define the next-hop IP address for global and both VRFs.
Step 1: Define next-hop IP address for multi VRF routes from GRT.
In this procedure, you configure the access lists in order to enable the IP network security between global and
both VRFs.
This section describes configuring the performance routing (PfR) hub-site MC.
In this procedure, you configure the definitions for two VRFs and their associated loopback interfaces.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.21.32.251 255.255.255.255
hold-queue 1024 in
hold-queue 1024 out
interface Loopback102
description CONT-VRF-102
vrf forwarding CONT-VRF-102
ip address 10.25.32.251 255.255.255.255
hold-queue 1024 in
hold-queue 1024 out
interface Port-channel21.1100
description IW-WAN-D3750X
encapsulation dot1Q 1100
ip address 10.6.32.151 255.255.255.224
interface Port-channel21.1101
description IW-WAN-D3750X IoT-VRF-101
encapsulation dot1Q 1101
vrf forwarding IoT-VRF-101
ip address 10.21.32.151 255.255.255.224
interface Port-channel21.1102
description IW-WAN-D3750X CONT-VRF-102
encapsulation dot1Q 1102
vrf forwarding CONT-VRF-102
ip address 10.25.32.151 255.255.255.224
All sites belong to a PfR domain where the remote site MCs are peered together. Peering has been greatly en-
hanced in PfRv3, which allows site information exchange and single touch provisioning.
PfRv3 has simplified policies with pre-existing templates. The policy configuration for the PfR domain is done in
the hub MC and the information is distributed to all sites via MC peering. This not only simplifies provisioning sub-
stantially but also makes the policy consistent across the entire IWAN network.
PfRv3 uses Unified Monitor (also called Performance Monitor) to monitor traffic going into WAN links and traf-
fic coming from the WAN links. It monitors performance metrics per differentiated service code point (DSCP)
rather than monitoring on per-flow or per-prefix basis. When application-based policies are used, the MC uses
a mapping table between the application name and the DSCP discovered. This reduces the number of records
significantly. PfRv3 relies on performance data measured on the existing data traffic on all paths whenever it can,
thereby reducing the need of synthetic traffic. Furthermore, the measurement data is not exported unless there is
a violation, which further reduces control traffic and processing of those records.
PfRv3 is also VRF-aware and instances of the MC work under a VRF.
It is mandatory to use loopback interfaces for the peering traffic between the BR and MC routers. For this design,
you put the loopback addresses into a specific subnet range, so they are easily identified in the routing table. The
loopback address ranges for the remote sites are as follows:
Step 1: Verify that the loopback 0 interfaces on each of your remote sites are reachable from the hub MC by us-
ing the show ip route command.
This example shows a loopback address range of 10.255.241.0/24 for six remote site primary routers.
Before the configuration of PfRv3 on the hub MC, you must create a prefix list for the enterprise and DC. The
enterprise-prefix list covers the range of IP addresses to be controlled and optimized within this IWAN domain.
Prefixes outside of the enterprise-prefix list are not be controlled by application policies, but they are load-bal-
anced.
The site-prefix range includes the prefixes at this specific site, which is normally a WAN aggregation or DC site.
Site-prefixes are typically statically defined at WAN aggregation and DC sites and discovered automatically at
remote sites.
Tech Tip
Step 1: Create the enterprise prefix list for each new VRF.
Step 2: Create the primary site prefix list for each new VRF.
Domain policies are configured on the hub MC. These policies are distributed to branch MCs by using the peering
infrastructure. All sites that are in the same domain share the same set of PfR policies. Policies can be based on
DSCP or on application names.
Policies are created using preexisting templates, or they can be customized with manually defined thresholds for
delay, loss, and jitter.
Tech Tip
Loss is not calculated for UDP traffic that is not RTP. Traffic loss for RTP voice and video packets is
calculated using the sequence numbers in the RTP header.
To avoid unwanted channel unreachable messages, it is recommended that you change the value of the channel-
unreachable-timer command from its default setting of 1 second. The command is under the advanced setting
and the value is specified in seconds.
The NMS collector IP address and port number are defined in the hub MC. The information is automatically
propagated to devices in the IWAN domain. If you do not want to use a single collector for your entire network,
you can specify a different IP address and port number in the IWAN domain for each device.
domain iwan
vrf IoT-VRF-101
master hub
source-interface Loopback101
site-prefixes prefix-list DC1-PREFIXES-101
password c1sco123
enterprise-prefix prefix-list ENTERPRISE-PREFIXES-101
collector 10.4.48.36 port 9991
vrf CONT-VRF-102
master hub
source-interface Loopback102
site-prefixes prefix-list DC1-PREFIXES-102
password c1sco123
enterprise-prefix prefix-list ENTERPRISE-PREFIXES-102
collector 10.4.48.36 port 9991
The policies use the PfR predefined templates. The path preference for voice, real time video, and low latency
data is to use MPLS1 and MPLS2 unless the delay, jitter, and loss values on the path fall outside the values speci-
fied in the templates. The bulk data and default classes use INET1 and INET2 with fallback to MPLS1 and MPLS2,
and the scavenger class uses INET1 and INET2 with fallback to blackhole. The rest of the traffic is load-balanced
between the two paths.
Tech Tip
With this recommended policy, PfR does not manage Internetwork Control (DSCP CS6) traffic. CS6
traffic should always follow the normal routing path.
domain iwan
vrf IoT-VRF-101
master hub
load-balance advanced
path-preference INET1 INET2 fallback routing
class VOICE sequence 10
match dscp ef policy voice
path-preference MPLS1 MPLS2 fallback INET1 INET2
path-last-resort INET4G
class REAL_TIME_VIDEO sequence 20
The hub-site BRs are also the DMVPN hub WAN aggregation routers for the network. The PfRv3 configurations
for standalone BRs are much simpler because they dynamically learn their policy information from the hub-site
MC. The hub-site BR routers are also used to advertise the path names and path-ids specified in the hub-site
MC configuration.
Local MC
Path loopback IP
Host name VRF name Tunnel ID Path ID address
DHY-M1I1-ASR1002X-1 Default VRF 100 MPLS1 1 10.6.32.251
IoT-VRF-101 101 10.21.32.251
CONT-VRF-102 102 10.25.32.251
Default VRF 200 INET1 2 10.6.32.251
IoT-VRF-101 201 10.21.32.251
CONT-VRF-102 202 10.25.32.251
DHY-M1I1-ASR1002X-2 Default VRF 300 MPLS2 3 10.6.32.251
IoT-VRF-101 301 10.21.32.251
CONT-VRF-102 302 10.25.32.251
Default VRF 400 INET2 4 10.6.32.251
IoT-VRF-101 401 10.21.32.251
CONT-VRF-102 402 10.25.32.251
Default VRF 500 INET4G 5 10.6.32.251
IoT-VRF-101 501 10.21.32.251
CONT-VRF-102 502 10.25.32.251
When using multiple VRFs, the border router monitor cache is shared across the VRFs and requires additional
configuration. If you do not change the values, a warning message is displayed on the console. The recommend-
ed value for three VRFs is 33%, which is 100 divided by 3, but the value can be adjusted if the number of flows or
amount of traffic is not equal between the VRFs.
domain iwan
vrf default
border
advanced
monitor-cache-percent 33
vrf IoT-VRF-101
border
source-interface Loopback101
master 10.21.32.251
password c1sco123
advanced
monitor-cache-percent 33
vrf CONT-VRF-102
border
source-interface Loopback102
master 10.25.32.251
password c1sco123
advanced
monitor-cache-percent 33
Step 2: Add the path names and path-ids to the tunnel interfaces of the hub-site BR.
This example is the primary hub-site BR using tunnels with MPLS as the provider for each VRF.
interface Tunnel101
domain iwan path MPLS1 path-id 1
interface Tunnel102
domain iwan path MPLS1 path-id 1
This example is the primary hub-site BR using tunnels with INET as the provider for each VRF.
interface Tunnel201
domain iwan path INET1 path-id 2
interface Tunnel202
domain iwan path INET1 path-id 2
Step 3: Verify the border is operational for VRFs IoT-VRF-101 and CONT-VRF-102 by using the show domain
[name] vrf [VRF name] border status command.
Step 4: Repeat this procedure for each hub BR by using the appropriate path name and path-id.
The PfR path names and path-ids are automatically discovered at the remote site routers from the configuration
entered into the tunnel interfaces at the hub site. The hub-site MC uses the path names and path-ids to deter-
mine where traffic should be sent according to its policies.
Step 1: Verify the domain is operational from the hub-site MC using the show domain [name] master status
command for VRFs IoT-VRF-101 and CONT-VRF-102.
This section describes configuring the PfR transit-site MC as a new router. Only the core relevant features are
included.
In this procedure, you configure the definitions for two VRFs and their associated loopbacks.
address-family ipv4
exit-address-family
address-family ipv4
exit-address-family
interface Loopback101
description IoT-VRF-101
vrf forwarding IoT-VRF-101
ip address 10.23.32.251 255.255.255.255
hold-queue 1024 in
hold-queue 1024 out
interface Loopback102
description CONT-VRF-102
vrf forwarding CONT-VRF-102
ip address 10.27.32.251 255.255.255.255
hold-queue 1024 in
hold-queue 1024 out
interface Port-channel21.1100
description IW-WAN-D3750X-T
encapsulation dot1Q 1100
ip address 10.8.32.151 255.255.255.224
interface Port-channel21.1101
description IW-WAN-D3750X-T IoT-VRF-101
description IoT-VRF-101
encapsulation dot1Q 1101
vrf forwarding IoT-VRF-101
ip address 10.23.32.151 255.255.255.224
interface Port-channel21.1102
description IW-WAN-D3750X-T CONT-VRF-102
encapsulation dot1Q 1102
vrf forwarding CONT-VRF-102
ip address 10.27.32.151 255.255.255.224
These procedures are similar to previously configured PfR for the hub-site location.
To verify the remote-site loopback interfaces reachabilities on the transit-site MC, in the previous process Con-
figuring PfR for the Hub-site Location, follow Procedure 1, Verify IP connectivity to remote-site loopback inter-
faces.
Before the configuration of PfRv3 on the transit MC, you must create prefix lists for the DC. The enterprise-prefix
list is only configured on the hub MC and you do not configure one on the transit MC.
The site-prefix range for the transit site includes the prefixes at this specific site, which is normally a WAN aggre-
gation or DC site. Site-prefixes are typically statically defined at WAN aggregation and DC sites and discovered
automatically at remote sites.
Tech Tip
Domain policies are configured on the hub MC. These policies are distributed to branch MCs and the transit MC
by using the peering infrastructure. All sites that are in the same domain share the same set of PfR policies. The
transit MC must peer to the hub MC to get the policy information.
Example
domain iwan
vrf IoT-VRF-101
master transit 1
source-interface Loopback101
site-prefixes prefix-list DC2-PREFIXES-101
password c1sco123
hub 10.21.32.251
vrf CONT-VRF-102
master transit 1
source-interface Loopback102
site-prefixes prefix-list DC2-PREFIXES-102
password c1sco123
hub 10.25.32.251
Step 2: Verify the hub-site MC policy configuration is available by using the show domain [name] master policy
command.
The output from this command should look the same as the output on the hub-site MC.
The transit-site BRs are also the DMVPN hub WAN aggregation routers for the transit-site network. The PfRv3
configurations for standalone BRs are much simpler because they dynamically learn their policy information from
the transit-site MC. The transit-site BR routers are also used to advertise the path names and path-ids specified
in the hub-site MC configuration.
Local MC
Path loopback IP
Host name VRF name Tunnel ID Path ID address
DHY-M1I1-ASR1002X-T1 Default VRF 100 MPLS1 1 10.8.32.251
IoT-VRF-101 101 10.23.32.251
CONT-VRF-102 102 10.27.32.251
Default VRF 200 INET1 2 10.8.32.251
IoT-VRF-101 201 10.23.32.251
CONT-VRF-102 202 10.27.32.251
DHY-M1I1-ASR1002X-T2 Default VRF 300 MPLS2 3 10.8.32.251
IoT-VRF-101 301 10.23.32.251
CONT-VRF-102 302 10.27.32.251
Default VRF 400 INET2 4 10.8.32.251
IoT-VRF-101 401 10.23.32.251
CONT-VRF-102 402 10.27.32.251
Default VRF 500 INET4G 5 10.8.32.251
IoT-VRF-101 501 10.23.32.251
CONT-VRF-102 502 10.27.32.251
When using multiple VRFs, the border router monitor cache is shared across the VRFs and requires additional
configuration. If you do not change the values, a warning message is displayed on the console. The recommend-
ed value for three VRFs is 33%, which is 100 divided by 3, but the value can be adjusted if the number of flows or
amount of traffic is not equal between the VRFs.
This example is the primary transit-site BR.
domain iwan
vrf default
border
advanced
monitor-cache-percent 33
vrf IoT-VRF-101
border
source-interface Loopback101
master 10.23.32.251
password c1sco123
advanced
monitor-cache-percent 33
vrf CONT-VRF-102
border
source-interface Loopback102
master 10.27.32.251
password c1sco123
advanced
monitor-cache-percent 33
Step 2: Add the path names and path-ids to the tunnel interfaces of the transit-site BR.
This example is the primary transit-site BR using tunnels with MPLS as the provider for each VRF.
interface Tunnel101
domain iwan path MPLS1 path-id 1
interface Tunnel102
domain iwan path MPLS1 path-id 1
This example is the primary transit-site BR using tunnels with INET as the provider for each VRF.
interface Tunnel201
domain iwan path INET1 path-id 2
interface Tunnel202
domain iwan path INET1 path-id 2
Step 3: Verify the border is operational for VRFs IoT-VRF-101 and CONT-VRF-102 by using the show domain
[name] vrf [VRF name] border status command.
Step 4: Repeat this procedure for each transit BR by using the appropriate path name and path-id.
The PfR path names and path-ids are automatically discovered at the remote site routers from the configuration
entered into the tunnel interfaces at the hub and transit sites. The hub-site MC uses the path names and path-ids
to determine where traffic should be sent according to its policies.
Step 1: Verify the domain is operational from the transit-site MC by using the show domain [name] master sta-
tus command for VRFs IoT-VRF-101 and CONT-VRF-102.
Remote sites are discovered using peering. Each remote site MC peers with the hub MC. The remote site MC ad-
vertises local site information and learns information about every other site. Prefixes specific to sites are adver-
tised along with the site-id. The site-prefix to site-id mapping is used in monitoring and optimization. This map-
ping is also used for creating reports for specific sites.
WAN interfaces at each site are discovered using a special probing mechanism referred to as smart probes. This
further reduces provisioning on the remote sites. The WAN interface discovery also creates mapping of the inter-
face to a particular service provider. The mapping is used in monitoring and optimization.
PfRv3 requires loopback interfaces for the peering traffic between the BR and MC routers. For this design, you
put the hub MC loop back interface into the subnet range of the hub location. The following table shows the loop-
back addresses for the hub MC.
Each remote site must have a route to the hub over each exit path. You can have more than two paths. You can
also have two routes and Equal Cost Multiple Paths.
Step 1: Verify that there are at least two available paths to the loopback 0 interface on the hub MC from each
remote site router by using the show ip bgp [Hub MC Loopback address] command.
Step 2: Repeat this procedure to verify the available path for each VRF.
Each remote site must have a branch MC and branch BR configured. At dual-router sites it is recommended that
you configure the primary router as both an MC and BR and the secondary router as only a BR.
The domain name, VRF, and password must match the hub MC configuration. Use the respective loopback inter-
faces as the source for each VRFs. Configure the hub MC IP address.
Step 1: If you are not on the router console port, turn on terminal monitoring with the terminal monitor command
from the global command line interface.
terminal monitor
This example configures the branch MC and points to the IP address of the hub MC in the associated VRF of the
IWAN dual hybrid design model.
domain iwan
vrf IoT-VRF-101
master branch
source-interface Loopback101
password c1sco123
hub 10.21.32.251
vrf CONT-VRF-102
master branch
source-interface Loopback102
password c1sco123
hub 10.25.32.251
Step 3: After approximately two minutes, the console displays an EIGRP SAF message similar to the one below,
which indicates the branch MC has created an adjacency with the loopback interface of the hub MC.
Step 4: Verify the PfR policy from the hub MC has been propagated to the branch MC by using the show domain
[name] master policy command.
When using multiple VRFs, the border router monitor cache is shared across the VRFs and requires additional
configuration. If you do not change the values, a warning message is displayed on the console. The recommend-
ed value for three VRFs is 33%, which is 100 divided by 3, but the value can be adjusted if the number of flows or
amount of traffic is not equal between the VRFs.
This example configures the branch BR and points it to the local branch MC, which is running on the same router
platform.
domain iwan
vrf default
border
advanced
monitor-cache-percent 33
vrf IoT-VRF-101
border
source-interface Loopback101
master local
password c1sco123
advanced
monitor-cache-percent 33
vrf CONT-VRF-102
border
source-interface Loopback102
master local
password c1sco123
advanced
monitor-cache-percent 33
Step 6: After approximately thirty seconds, the console displays a line protocol up/down message similar to the
one below, which indicates the automatically generated tunnel interface has been created.
Step 7: Verify that the branch BR is operational by using the show domain [name] vrf [VRF name] border status
command for non-default VRFs.
Step 8: Verify that the branch MC is operational by using the show domain [name] vrf [VRF name] master sta-
tus command for non-default VRFs.
Use this procedure only when there is a secondary remote site router. If you have a single router at a remote
location, skip this procedure.
PfRv3 requires loopback interfaces for the peering traffic between the BR and MC routers. For this design, you
put the hub MC loop back interface into the subnet range of the hub location.
Each remote site must have a route to the hub MC in the BGP routing table over each exit path. You can have
more than two paths. You can also have two routes and Equal Cost Multiple Paths.
Step 1: Verify that there are at least two available paths to the loopback 0 interface on the hub MC from each
remote site router by using the show ip bgp [Hub MC Loopback address] command.
Step 2: Repeat the previous step to verify the available path for each VRF.
When using multiple VRFs, the border router monitor cache is shared across the VRFs and requires additional
configuration. If you do not change the values, a warning message is displayed on the console. The recommend-
ed value for three VRFs is 33%, which is 100 divided by 3, but the value can be adjusted if the number of flows or
amount of traffic is not equal between the VRFs.
domain iwan
vrf default
border
advanced
monitor-cache-percent 33
This example configures the branch BR and points it to the branch MC, which is running on the primary remote
site router.
domain iwan
vrf default
border
advanced
monitor-cache-percent 33
vrf IoT-VRF-101
border
source-interface Loopback101
master 10.201.241.32
password c1sco123
advanced
monitor-cache-percent 33
vrf CONT-VRF-102
border
source-interface Loopback102
master 10.202.241.32
password c1sco123
advanced
monitor-cache-percent 33
Step 4: After approximately thirty seconds, the console displays several EIGRP messages and a line protocol up/
down message similar to those shown below. These messages indicate the branch BR has neighbored with the
branch MC and automatically generated the tunnel interface from the loopback of the branch BR to the loopback
of the branch MC.
Step 5: Verify that the branch BR is operational by using the show domain [name] vrf [VRF name] border status
command for non-default VRFs.
Step 6: Repeat Procedure 1 through Procedure 3 for each remote site in your network.
The final procedure is to verify that the configured and default traffic classes are controlled by the MC at the hub
and branch locations.
For more information about verifying PfR traffic classes, see the main Intelligent WAN Deployment Guide.
For multi-VRF deployments, the QoS configuration has to take into account each tunnel sending traffic to the
same remote site router at the same time. In a multi-VRF environment, the following rules are applied:
Total bandwidth for VRFs should not exceed 200% of remote-site inbound service rate, because oversub-
scription traffic will be dropped in the SP cloud.
Bandwidth for guest traffic is normally about half the value of employee traffic.
Bandwidth for low volume VRFs such as IoT and PCI should use a much smaller percentage, closer to one-
tenth the value of employee traffic.
As the number of VRFs increases, the percentages need to come down accordingly based on the network
administrators knowledge of their traffic patterns.
The QoS policy on a tunnel instance allows you to shape the tunnel traffic to individual spokes and to differentiate
between traffic classes within the tunnel for appropriate treatment.
The QoS policy on the tunnel instance is defined and applied only to the DMVPN hub routers at the central site.
The remote-site router signals the QoS group policy information to the hub router with a command in the NHRP
configuration, which greatly reduces QoS configuration and complexity. The hub router applies the signaled policy
in the egress direction for each remote site.
In the example below, configure the default VRF traffic at 80% of the remote site inbound service rate bandwidth,
the IoT VRF at 10% of the bandwidth, and the contractor VRF at 80% of the bandwidth.
The bandwidth remaining ratio command is used to provide each site with their fair share of the remaining
bandwidth when the outbound interface is experiencing congestion. If you do not use this command, the lower-
bandwidth sites get all of their assigned bandwidth, while the higher bandwidth sites get less than their fair share.
In the example below, divide the shape average bandwidth by 1 Mbps to come up with the value for the ratio. If
you have sites with less than 5 Mbps of shape average bandwidth, you should divide the shape average for all of
your sites by 100 Kbps to ensure they all get a reasonable ratio greater than 1.
Tech Tip
With Per-Tunnel QoS for DMVPN, the queuing and shaping is performed at the outbound physical
interface for the GRE/IPsec tunnel packets. This means that the GRE header, the IPsec header and
the layer2 (for the physical interface) header are included in the packet-size calculations for shaping
and bandwidth queuing of packets under QoS.
The values in the table are examples; make sure to adjust these values for your specific needs and
remote-site bandwidth provisioned with your ISP.
Table 89 Per-tunnel QoS policies with 80% employee, 10% IoT and 80% contractor
policy-map [policy-map-name]
Step 2: Define a shaper and bandwidth remaining ratio for the default-class and apply the WAN QoS queuing
child service policy created in Procedure 2, Create policy map with queuing policy.
The shape average value is entered in bits per second (bps). If all of your bandwidth values are greater than 5
Mbps, enter the bandwidth remaining ratio as shape average bandwidth/1 Mbps. If any of your bandwidth values
are 5 Mbps or less, enter the bandwidth remaining ratio as shape average bandwidth/100 Kbps.
policy-map [policy-map-name]
description [percentage] of inbound service rate
class class-default
shape average [bandwidth (bps)]
bandwidth remaining ratio [shape average bandwidth/1 Mbps]
service-policy [policy-map name]
class class-default
shape average 40000000
bandwidth remaining ratio 40
service-policy WAN
policy-map RS-GROUP-30MBPS-80-POLICY
description 80% of inbound service rate
class class-default
shape average 24000000
bandwidth remaining ratio 24
service-policy WAN
policy-map RS-GROUP-20MBPS-80-POLICY
description 80% of inbound service rate
class class-default
shape average 16000000
bandwidth remaining ratio 16
service-policy WAN
policy-map RS-GROUP-10MBPS-80-POLICY
description 80% of inbound service rate
class class-default
shape average 8000000
bandwidth remaining ratio 8
service-policy WAN
policy-map RS-GROUP-4G-80-POLICY
description 80% of inbound service rate
class class-default
shape average 6000000
bandwidth remaining ratio 6
service-policy WAN
policy-map RS-GROUP-300MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 30000000
bandwidth remaining ratio 30
service-policy WAN
policy-map RS-GROUP-200MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 20000000
bandwidth remaining ratio 20
service-policy WAN
policy-map RS-GROUP-100MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 10000000
bandwidth remaining ratio 10
service-policy WAN
policy-map RS-GROUP-50MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 5000000
bandwidth remaining ratio 5
service-policy WAN
policy-map RS-GROUP-30MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 3000000
bandwidth remaining ratio 3
service-policy WAN
policy-map RS-GROUP-20MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 2000000
bandwidth remaining ratio 2
service-policy WAN
policy-map RS-GROUP-10MBPS-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 1000000
bandwidth remaining ratio 1
service-policy WAN
policy-map RS-GROUP-4G-10-POLICY-VRF101
description 10% of inbound service rate
class class-default
shape average 2000000
bandwidth remaining ratio 2
service-policy WAN
policy-map RS-GROUP-300MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 240000000
bandwidth remaining ratio 240
service-policy WAN
policy-map RS-GROUP-200MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 160000000
bandwidth remaining ratio 160
service-policy WAN
policy-map RS-GROUP-100MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 80000000
bandwidth remaining ratio 80
service-policy WAN
policy-map RS-GROUP-50MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 40000000
bandwidth remaining ratio 40
service-policy WAN
policy-map RS-GROUP-30MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 24000000
bandwidth remaining ratio 24
service-policy WAN
policy-map RS-GROUP-20MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 16000000
bandwidth remaining ratio 16
service-policy WAN
policy-map RS-GROUP-10MBPS-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 8000000
bandwidth remaining ratio 8
service-policy WAN
policy-map RS-GROUP-4G-80-POLICY-VRF102
description 80% of inbound service rate
class class-default
shape average 6000000
bandwidth remaining ratio 6
service-policy WAN
The QoS policy that the hub uses for a particular endpoint or spoke is selected by the NHRP group in which the
spoke is configured.
Prerequisites and important caveats:
DMVPN must be fully configured and operational before you can configure an NHRP group on a spoke or
map the NHRP group to a QoS policy on a hub.
Although you may configure multiple spokes as part of the same NHRP group, the tunnel traffic for each
spoke is measured individually for shaping and policing.
Only output NHRP policies are supported. These apply to per-site traffic egressing the router towards the
WAN.
Step 1: Create NHRP group policy name mapping and apply the policies configured in the previous procedure to
the DMVPN tunnel interface on the hub router.
interface tunnel[number]
ip nhrp map group [NHRP GROUP Policy Name] service-policy output [policy-map
name]
interface Tunnel101
ip nhrp map group RS-GROUP-300MBPS-10-VRF101 service-policy output RS-GROUP-
300MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-200MBPS-10-VRF101 service-policy output RS-GROUP-
200MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-100MBPS-10-VRF101 service-policy output RS-GROUP-
100MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-50MBPS-10-VRF101 service-policy output RS-GROUP-
50MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-30MBPS-10-VRF101 service-policy output RS-GROUP-
30MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-20MBPS-10-VRF101 service-policy output RS-GROUP-
20MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-10MBPS-10-VRF101 service-policy output RS-GROUP-
10MBPS-10-POLICY-VRF101
ip nhrp map group RS-GROUP-4G-10-VRF101 service-policy output RS-GROUP-4G-
10-POLICY-VRF101
interface Tunnel102
ip nhrp map group RS-GROUP-300MBPS-80-VRF102 service-policy output RS-GROUP-
300MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-200MBPS-80-VRF102 service-policy output RS-GROUP-
200MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-100MBPS-80-VRF102 service-policy output RS-GROUP-
100MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-50MBPS-80-VRF102 service-policy output RS-GROUP-
50MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-30MBPS-80-VRF102 service-policy output RS-GROUP-
30MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-20MBPS-80-VRF102 service-policy output RS-GROUP-
20MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-10MBPS-80-VRF102 service-policy output RS-GROUP-
10MBPS-80-POLICY-VRF102
ip nhrp map group RS-GROUP-4G-80-VRF101 service-policy output RS-GROUP-4G-
80-POLICY-VRF101
This process completes the remote-site QoS configuration and applies to all DMVPN spoke routers.
This procedure configures the remote-site router to reference the QoS policy configured on the hub site routers.
Step 1: Apply the NHRP group policy to the DMVPN tunnel interface on the corresponding remote-site router.
Use the NHRP group name as defined on the hub router in Procedure 2, Configure per tunnel QoS policies for
DMVPN hub router, above.
Configure the bandwidth statement on the interface to match the outbound service rate. Even though we are
shaping from the hub site to a lower percentage of bandwidth, configure the bandwidth receive statement on the
interface to match the inbound service rate and the NHRP group policy chosen. The bandwidth value is entered in
kilobits per second (Kbps).
interface Tunnel[value]
bandwidth [outbound service rate value in Kbps]
bandwidth receive [inbound service rate value in Kbps]
ip nhrp group [NHRP GROUP Policy Name]
interface Tunnel100
bandwidth 20000
bandwidth receive 20000
ip nhrp group RS-GROUP-20MBPS-80
interface Tunnel101
bandwidth 20000
bandwidth receive 20000
ip nhrp group RS-GROUP-20MBPS-10-VRF101
interface Tunnel102
bandwidth 20000
bandwidth receive 20000
ip nhrp group RS-GROUP-20MBPS-80-VRF102
interface Tunnel200
bandwidth 50000
bandwidth receive 50000
ip nhrp group RS-GROUP-50MBPS-80
interface Tunnel201
bandwidth 50000
bandwidth receive 50000
ip nhrp group RS-GROUP-50MBPS-10-VRF101
interface Tunnel202
bandwidth 50000
bandwidth receive 50000
ip nhrp group RS-GROUP-50MBPS-80-VRF102
After all of the physical interfaces on a router are configured, you can verify each one before moving to the next
remote site.
Step 1: Verify the QoS output policy on each interface is correct by using the show policy-map interface com-
mand.
Step 2: Repeat the previous step for each interface configured with QoS.
Tech Tip
If you experience tail-drops in your class class-default, a potential work-around is to increase the size
of the queue-limit.
On an interface with bandwidth of less than 15 Mbps, the default queue-limit is 64 packets. Increas-
ing this value adds latency to the traffic in the default-class but also reduces the number of tail-drops.
policy-map WAN
class class-default
queue-limit 512 packets
After the all of the DMVPN routers are configured for Per-Tunnel QoS, you can verify the configurations from the
hub router.
Step 1: Verify the Per-Tunnel QoS output policy to each remote-site is correct by using the show dmvpn detail
command.
Step 2: Repeat the previous step for each DMVPN hub router.
IPsec uses a key exchange between the routers in order to encrypt/decrypt the traffic. These keys are ex-
changed using pre-shared keys.
The crypto keyring defines a pre-shared key (or password) valid for IP sources that are reachable within a par-
ticular VRF. This key is a wildcard pre-shared key if it applies to any IP source. A wildcard key is configured using
the 0.0.0.0 0.0.0.0 as the network/mask combination.
Example
crypto ikev2 keyring DMVPN-KEYRING
peer ANY
address 0.0.0.0 0.0.0.0
pre-shared-key c1sco123
Diffie-Hellman group: 19
Example
crypto ikev2 proposal AES/GCM/256
encryption aes-gcm-256
prf sha512
group 19
The default IKEv2 proposal is also used.
A show crypto ikev2 proposal displays the details of the two proposals.
The crypto policy includes the proposal you created in the previous step. This policy matches any FVRF defined
on the router.
Example
crypto ikev2 policy AES/GCM/256
match fvrf any
proposal AES/GCM/256
The default IKEv2 policy is also used.
A show crypto ikev2 policy displays the details of the two policies.
The IKEv2 profile creates an association between an identity address, a VRF, and a crypto keyring. A wildcard
address within a VRF is referenced with 0.0.0.0. The identity local address must match the loopback address of
this router.
Tech Tip
Identity local address is needed for customers who use Carrier Grade NAT (CGN) which requires a
unique identity per remote site router even if the same pre-NAT IP address is used for other loca-
tions. The command does not affect customers who are not using CGN, so it is a recommended best
practice to use the command all of the time.
The profile also defines what method of key sharing are used on this router with authentication local and what
methods are accepted from remote locations with authentication remote. The pre-share keyword is used with
the keyring defined above.
A transform set is an acceptable combination of security protocols, algorithms, and other settings to apply to
IPsec-protected traffic. Peers agree to use a particular transform set when protecting a particular data flow.
The IPsec transform set for DMVPN uses the following:
ESP with the 256-bit GCM encryption algorithm
Because the DMVPN hub router is behind a NAT device, the IPsec transform must be configured for transport
mode.
Example
crypto ipsec transform-set AES256/GCM/TRANSPORT esp-gcm 256
mode transport
The IPsec profile creates an association between an IKEv2 profile and an IPsec transform-set.
Example
IPSec profile for WAN transport MPLS1.
Example
crypto ipsec security-association replay window-size 1024
Tech Tip
QoS queuing delays can cause anti-replay packet drops, so it is important to extend the window size
in order to prevent the drops from occurring.
Increasing the anti-replay window size has no impact on throughput and security. The impact on
memory is insignificant because only an extra 128 bytes per incoming IPsec SA is needed.
It is recommended that you use the maximum window size in order to minimize future anti-replay
problems. On the ASR1K, ISR4K and ISRG2 router platforms, the maximum replay window size is
1024.
If you do not increase the window size, the router may drop packets and you may see the following
error messages on the router CLI and in the log:
A show crypto ipsec sa displays the transform and anti-replay window size.
interface: Tunnel102
Crypto map tag: DMVPN-IPSEC-PROFILE-MPLS1-head-1, local addr 192.168.6.1
PERMIT, flags={origin_is_acl,}
#pkts encaps: 97284, #pkts encrypt: 97284, #pkts digest: 97284
#pkts decaps: 96016, #pkts decrypt: 96016, #pkts verify: 96016
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
IPsec uses a key exchange between the routers in order to encrypt/decrypt the traffic. These keys are ex-
changed using pre-shared keys.
The crypto keyring defines a pre-shared key (or password) valid for IP sources that are reachable within a par-
ticular VRF. This key is a wildcard pre-shared key if it applies to any IP source. A wildcard key is configured using
the 0.0.0.0 0.0.0.0 network/mask combination.
Example
crypto ikev2 keyring DMVPN-KEYRING
peer ANY
address 0.0.0.0 0.0.0.0
pre-shared-key c1sco123
Diffie-Hellman group: 19
Example
crypto ikev2 proposal AES/GCM/256
encryption aes-gcm-256
prf sha512
group 19
The default IKEv2 proposal is also used.
A show crypto ikev2 proposal displays the details of the two proposals.
The crypto policy includes the proposal you created in the previous step. This policy matches any FVRF defined
on the router.
Example
crypto ikev2 policy AES/GCM/256
match fvrf any
proposal AES/GCM/256
The default IKEv2 policy is also used.
A show crypto ikev2 policy displays the details of the two policies.
The IKEv2 profile creates an association between an identity address, a VRF, and a crypto keyring. A wildcard
address within a VRF is referenced with 0.0.0.0. The identity local address must match the loopback address of
this router.
Tech Tip
Identity local address is needed for customers who use CGN, which requires a unique identity per
remote site router even if the same pre-NAT IP address is used for other locations. The command
does not affect customers who are not using CGN, so it is a recommended best practice to use the
command all of the time.
The profile also defines what method of key sharing are used on this router with authentication local and what
methods are accepted from remote locations with authentication remote. The pre-share keyword is used with
the keyring defined above.
DPD is essential in order to facilitate fast re-convergence and for spoke registration to function properly in case
a DMVPN hub is restarted. The IWAN design recommends you set the remote site DPD timer to 40 seconds with
a 5 second retry. Moving the DPD timer into the crypto ikev2 profile ensures the command is used immediately,
rather than waiting for the first 24 hour refresh cycle if the command is entered in the global configuration.
A transform set is an acceptable combination of security protocols, algorithms, and other settings to apply to
IPsec-protected traffic. Peers agree to use a particular transform set when protecting a particular data flow.
The IPsec transform set for DMVPN uses the following:
ESP with the 256-bit GCM encryption algorithm
Because the DMVPN hub router is behind a NAT device, the IPsec transform must be configured for transport
mode.
Example
crypto ipsec transform-set AES256/GCM/TRANSPORT esp-gcm 256
mode transport
The IPsec profile creates an association between an IKEv2 profile and an IPsec transform-set.
Example
IPSec profile for WAN transport MPLS1.
Example
crypto ipsec security-association replay window-size 1024
Tech Tip
QoS queuing delays can cause anti-replay packet drops, so it is important to extend the window size
in order to prevent the drops from occurring.
Increasing the anti-replay window size has no impact on throughput and security. The impact on
memory is insignificant because only an extra 128 bytes per incoming IPsec SA is needed.
It is recommended that you use the maximum window size in order to minimize future anti-replay
problems. On the ASR1K, ISR4K and ISRG2 router platforms, the maximum replay window size is
1024.
If you do not increase the window size, the router may drop packets and you may see the following
error messages on the router CLI and in the log:
Appendix C: Changes
This appendix summarizes the changes Cisco made to this guide since its last edition.
Guide updates:
Replaced the connection between border routers, WAN-Distribution switches, remote-site access switch-
es with Layer 3 trunk interfaces and added the subinterfaces to the port-channels.
Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco Website at www.cisco.com/go/offices.
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, DESIGNS) IN THIS MANUAL ARE PRESENTED AS
IS, WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT
SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION,
LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN
ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR
THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS
OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON
FACTORS NOT TESTED BY CISCO.
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included
in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go
to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not
imply a partnership relationship between Cisco and any other company. (1110R)