W HITE P APER

Gigabit Campus Design

Configuration and Recovery Analysis

Introduction
Some of the most useful variations of the large-scale multilayer campus internetwork design were tested for failure recovery. Please refer to
the companion document Gigabit Campus Design—Principles and Architecture for an explanation of several implementations of the
multilayer design. In all cases, Gigabit EtherChannel® links were used between distribution-layer switches and core-layer switches.
Redundant Gigabit Ethernet uplinks were used to connect the access-layer switches to the distribution-layer switches. The Catalyst® 4000,
5500, and 6000 family switches were used in the wiring closet configurations. The Catalyst 6500 with Multilayer Switch Feature Card
(MSFC) and Catalyst 8540 were both used as the Layer 3 switch. The Catalyst 6500 is used as the Layer 2 switch in the core.

This paper documents configuration and recovery with three fundamental designs:

1. Layer 3 distribution with dual-path Layer 2 core (good)

a) Catalyst 6500 standard building block
2. Layer 3 distribution with Layer 3 core (better)
a) Catalyst 6500 standard building block
3. Layer 3 distribution with dual-path Layer 3 core (best)
a) Catalyst 6500 standard building block
b) Catalyst 6500 virtual LAN(VLAN) building block

The following two design variations are described in the appendix:

A1. Layer 3 distribution with Layer 3 core

a) Catalyst 8540 standard building block
A2. Layer 3 distribution with dual-path Layer 3 core
a) Catalyst 8540 standard building block

Layer 2 spanning-tree loops were eliminated in the core and in the standard building block. Avoidance of spanning-tree loops in the core is
important for high availability. Within the core and the standard building block all Layer 3 switches were configured with native routed
interfaces. VLAN trunks were only used within the VLAN building block. The VLAN building block is typically used in a server farm to
provide for redundant server connection with dual Network Interface Cards (NICs).
The Enhanced Interior Gateway Routing Protocol (EIGRP) was tuned for fast convergence with a one-second-hello timer and a
three-second-hold timer. The Hot Standby Routing Protocol (HSRP) was tuned for fast convergence with a one-second-hello timer and a
three-second-hold timer. When using protocol hello timers of one second, the limit of accuracy for recovery measurement is about one second.

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 1 of 26
Thus, all results are rounded to units of whole seconds, representing the worst case. Network recovery times were very predictable, with some
variation due to the state of things such as the Address Resolution Protocol (ARP) cache. OSPF was configured with the same timer values
as EIGRP, with similar results.
Every router and switch in the network was provided an out-of-band Ethernet management interface on VLAN 99. One Ethernet port on
each device was configured in VLAN 99 and wired outside the test network to a separate switched management network. In addition to the
management IP address on VLAN 99, we also configured Cisco 2511 terminal servers with RJ-45 serial cables to access the console port of
every router and switch in the layout. With this arrangement, we have two out-of-band paths to every switch and router, as well as a number
of in-band paths.
Wherever possible Gigabit EtherChannel was configured with ports on two different cards to increase availability. For example,
port-channel 1 on CT85 (core top 8540) used interfaces gigabit 0/0/0 and gigabit 1/0/0, which are the first physical ports on two different
cards. EtherChannel recovery was not tested, as it is much faster than one second and falls below the measurement threshold of this testing.

Configuration of the Standard Building Block

The standard building block appears on the left side (west block) of Figures 3a, 3b, and 3c. Configuration of the standard building block is
very simple because all spanning-tree loops are eliminated. Hence, there is no Layer 2 tuning, such as selecting the best root switch for a given
VLAN. No VLAN trunks are used and uplinks are connected to native routed interfaces on the Layer 3 switches in the distribution layer.

Catalyst 6000 Access Switch: Standard Building Block

Set prompt aw6 (access layer, west block, Catalyst 6000)
Set vtp domain west
Set vtp mode transparent
(no VLAN trunks, use transparent mode)
Set vlan 99 (used for out of band management)
Set vlan 99 3/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
Set port channel 1/1-2 mode off
(turn off channel negotiation on uplinks)
(not using EtherChannel in this configuration)
set trunk 1/1-2 off(turn off VLAN trunking on uplinks)
set vlan 10(VLAN 10 corresponds to subnet “10” ie 10.10.0.0)
set vlan 10 1/1-2(all other ports are part of VLAN 10)
set vlan 10 3/1-47
We disable channel negotiation on the uplinks to make connection faster after a failure has been restored. This is appropriate because the
uplinks are routed connections to Layer 3 switches in the distribution layer, not Layer 2 connections that require spanning tree. (Nothing in
the configuration indicates that all clients and uplinks attached to the switch are in subnet “10” (10.10.0.0) with mask 255.255.0.0. VLAN 10
is configured everywhere to match subnet “10” [10.10.0.0]).

Configuration of the Standard Building Block with Load Balancing

To achieve load balancing with the standard building block, a couple of changes are required. Configure two VLANs (two subnets) on a wiring
closet switch and use VLAN trunks for uplinks. For example, on switch aw6, configure VLAN 10 and VLAN 11. All outbound VLAN 10
traffic will take one uplink to the HSRP primary gateway router for subnet 10.10.0.0. All outbound VLAN 11 traffic will take the other uplink
to the HSRP primary gateway router for subnet 10.11.0.0. This load balancing configuration maintains all the advantages of the standard
building block and is only marginally more complex.

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 2 of 26
Catalyst 6000 Access Switch: Standard Building Block with Load Balancing
Set prompt aw6 (access layer, west block, Catalyst 6000)
Set vtp domain west
Set vtp mode transparent
(use transparent mode, configure VLANs explicitly)
Set vlan 99 (used for out of band management)
Set vlan 99 5/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
Set port channel 1/1-2 mode off
(turn off channel negotiation on uplinks)
(not using EtherChannel on this switch)
set trunk 1/1-2 on 10,11 dot1q
(VLAN trunking on uplinks, for VLAN 10 and VLAN 11)
set vlan 10 3/1-48(VLAN 10 corresponds to subnet “10” ie 10.10.0.0)
set vlan 11 4/1-48(VLAN 11 corresponds to subnet “11” ie 10.11.0.0)

Configuration of the VLAN Building Block

The VLAN building block appears on the right side (east block) of Figures 3a, 3b, and 3c. If dual-attached servers are used, then a VLAN
trunking configuration is required within the server distribution block. The two distribution-layer switches are the root bridges of the even and
odd numbered VLANs. UplinkFast is configured on the access-layer switches. BackboneFast is configured on the access and distribution
switches for faster spanning tree recovery. For maximum determinism, VTP transparent mode is used and all VLANs are configured explicitly.
HSRP at Layer 3 is configured to match the Layer 2 spanning tree configuration. This way, the HSRP primary gateway router for
even-numbered subnets is also the spanning tree root for even-numbered VLANs. The HSRP primary gateway router for odd-numbered
subnets is also the spanning tree root for odd-numbered VLANs.

Configuration of Catalyst 6000 Access Switch:

Set prompt ae6 (access layer, east block, Catalyst 6000)
Set vtp domain east
Set vtp mode transparent
(use transparent mode, configure all VLANs explicitly)
Set vlan 99 (used for out of band management)
Set vlan 99 4/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
set spantree uplinkfast enable
(set uplinkfast on the access switch only)
set spantree backbonefast enable
(enable backbonefast on all switches in block)
Set port channel 1/1-2 mode off
(turn off channel negotiation on uplinks)
(not using EtherChannel on this switch)
set trunk 1/1-2 on 50,51,52,53 dot1q
(dot1q VLAN trunking on uplinks, state VLANs explicitly)
(these four VLANs are used in this building block)
set vlan 50 2/1-48(VLAN 50 corresponds to subnet “50” ie 10.50.0.0)
set vlan 51 3/1-48(VLAN 51 corresponds to subnet “51” ie 10.51.0.0)

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 3 of 26
Configuration of Catalyst 6500 Distribution Switch:
The ARP cache timeout on the MSFC is four hours. However, the Layer 2 CAM table times out in 300 seconds by default. This may result
in some IP unicast traffic being flooded. In the following configuration the CAM timeout “agingtime” is set to 4hours*60min/hour*60sec/
min. = 14400 seconds to match the ARP cache timeout.
Set prompt det65 (distribution layer, east block, top, Catalyst 6500)
Set vtp domain east
Set vtp mode transparent
(use transparent mode, configure all VLANs explicitly)
Set vlan 99 (used for out of band management)
Set vlan 99 3/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
set cam agingtime 1-1000 14400
(set CAM timeout to 4 hours to match ARP timeout)
set spantree backbonefast enable
(enable backbonefast on all switches in block)
Set port channel 1/1-2 mode off
(turn off channel negotiation on non-Etherchannel links)
set port channel 3/7-8 mode off
set port channel 4/7-8 mode off
set port channel 3/1-2 mode on
(EtherChannel used for routed links to the core)
set port channel 4/1-2 mode on
(EtherChannel used for routed links to the core)
set vlan 40 3/1,4/1
(routed etherchannel link VLAN40=10.40.0.0 spans two cards)
set vlan 42 3/2,4/2
(routed etherchannel link VLAN42=10.42.0.0 spans two cards)
set trunk 3/7 on 50,51,52,53 dot1q
(set VLANs and dot1q trunking explicitly)
(3/7-8 and 4/7-8 are uplinks to wiring closet switches)
set trunk 3/8 on 50,51,52,53 dot1q
set trunk 4/7 on 50,51,52,53 dot1q
set trunk 4/8 on 50,51,52,53 dot1q
set trunk 1/1 on 50,51,52,53 dot1q
(this is the backup trunk to other distribution switch)
set spantree root 50,52
(make this root bridge for even VLANs)
set spantree root secondary 51,53
(make this backup root bridge odd VLANs)
set trunk 3/1-2 off
(no VLAN trunking on routed links to core)
set trunk 4/1-2 off
(no VLAN trunking on routed links to core)
As shown in Figures 3b and 3c, VLANs 50, 51, 52, and 53 correspond to the wiring closet subnets. VLANs 40 and 42 correspond to routed
links to the core switches. Refer to section three of the test results for the corresponding MSFC (router) configuration.

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 4 of 26
Notes on Test Procedures
Test results are measured by recovery of multiple two-way PING traffic flows. In all cases, test traffic flows are between clients attached to
the network, not to and from the switches within the network. We measure the outage when the failure is created in terms of seconds of lost
traffic. Then we measure the outage when the failure is restored in terms of seconds of lost traffic. Results are rounded up to the next second,
which is the limit of accuracy with one-second-hello protocols. If recovery is given as three seconds in the table, that means we measured two
seconds and three seconds over several tests. The “small” routing table just reflects the subnets within the test bed. To stress routing protocol
convergence, the “large” routing table includes 3000 static host routes injected into the test bed from the six Layer 3 switches.

Summary of IP Routing with Small Routing Table

ctmsfc#sho ip route summary

Route Source Networks Subnets Overhead Memory (bytes)
connected 1 5 336 864
static 1 0 56 144
eigrp 1 0 11 616 1584
internal 2 2328
Total 4 16 1008 4920

Summary of IP Routing with Large Routing Table

ctmsfc#sho ip ro sum
Route Source Networks Subnets Overhead Memory (bytes)
connected 1 5 336 864
static 1 500 28056 72144
eigrp 1 0 2511 140616 361584
internal 2 2328
Total 4 3016 169008 436920

To stress the control plane (routing software running on the CPU) static routes are added to each of the routers (Layer 3 switches) in the test
bed by Trivial File Transfer Protocol (TFTP). TFTP configuration is accomplished with the “configure network” command. The files on the
TFTP server consist of a series of commands of the form:
ip route 10.0.1.001 255.255.255.255 null 0 1
ip route 10.0.1.002 255.255.255.255 null 0 1
ip route 10.0.1.003 255.255.255.255 null 0 1
ip route 10.0.1.004 255.255.255.255 null 0 1
…
ip route 10.0.1.254 255.255.255.255 null 0 1
end

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 5 of 26
1. Layer 3 Distribution with Dual-Path Layer 2 Core—Catalyst 6500
Please refer to Figures 1a, 1b, and 1c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to
test network recovery times as different failures were induced. This campus design features redundant Layer 3 switches in each
distribution layer building block and redundant Layer 2 switches in the core. The dual-path Layer 2 core consists of two separate switched
VLANs with no loops and no VLAN trunks. The links into the core VLANs are native routed interfaces on the distribution-layer switches,
and VLAN trunks are not used.

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 6 of 26
Configuration of MSFC: Client-Side Interface on HSRP Primary Distribution Switch
interface Vlan10
ip address 10.10.0.81 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
(set eigrp hello timer 1 second)
ip hold-time eigrp 1 3
(set eigrp hold timer 3 seconds)
standby 10 timers 1 3
(set HSRP hello timer 1 hold timer 3 seconds)
(convention - HSRP group number 10 matches VLAN number)
standby 10 priority 200 preempt delay 60
(this is the primary gateway router for subnet 10)
(preempt delay 60 seconds allows EIGRP to stabilize before
HSRP switches back upon power recovery)
standby 10 ip 10.10.0.200
(10.10.0.200 is the HSRP gateway router address)
standby 10 track Vlan31 75
standby 10 track Vlan32 75
(if you lose both links to the backbone, drop priority
by 150 to initiate HSRP recovery)

Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch

interface Vlan10
ip address 10.10.0.82 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby 10 timers 1 3
standby 10 priority 100 preempt delay 60
(HSRP secondary or backup gateway router for subnet 10)
standby 10 ip 10.10.0.200
standby 10 track Vlan31 25
standby 10 track Vlan32 25

Additional MSFC Configuration—Interface to Backbone VLAN 31

interface Vlan31
ip address 10.31.0.81 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3

Additional MSFC Configuration—Interface on Management VLAN

interface Vlan99
ip address 172.26.196.81 255.255.254.0
no ip directed-broadcast

Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets

router eigrp 1
passive-interface Vlan10
passive-interface Vlan11
passive-interface Vlan12
passive-interface Vlan13
passive-interface Vlan99
network 10.0.0.0

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 7 of 26
2. Layer 3 Distribution with Layer 3 Core—Catalyst 6500
Please refer to Figures 2a, 2b, and 2c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to
test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the core, but
each building block as a whole has redundant connectivity into the core.

This campus design features redundant Layer 3 switches in the distribution layer. For building block “west” the redundant switches are
dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has a two-port Gigabit
EtherChannel connection to a Layer 3 switch in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP recovery
will take place if the routed link to the core is broken.

There is one important caveat with this design: because each distribution-layer switch has a single path into the core, a redundant routed
path must be provided in case this fails. If the physical link breaks, then HSRP will handle the recovery. However, in the event of some
logical failure of the MSFC within the core switch, HSRP will not be triggered. An example of this is when the MSFC in the core router
is reloaded. Eliminate the passive interface command on two of the wiring closet VLAN interfaces to provide two routed backup paths
between the distribution-layer switches.

Table 3 Results with Small Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 4s HSRP

Restore 1s HSRP

Distribution Switch

Fail 4s HSRP

Restore 1s HSRP

Core Switch

Fail 4s HSRP

Restore 14s (See note below)

Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore,
HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a
routing table built; hence the 14-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the
HSRP switch itself, and does not affect this value.

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 8 of 26
Table 4 Results with Large Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 4s HSRP

Restore 1s HSRP

Distribution Switch

Fail 4s HSRP

Restore 1s HSRP

Core Switch

Fail 4s HSRP

Restore 30s (See note below)

Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore,
HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a
routing table built; hence the 30-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the
switch running HSRP in the distribution layer, and is not effective when the core switch is powered up.

Configuration of MSFC: Client-Side Interface on HSRP Primary Distribution Switch

interface Vlan10
ip address 10.10.0.81 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
(set eigrp hello timer 1 second)
ip hold-time eigrp 1 3
(set eigrp hold timer 3 seconds)
standby 10 timers 1 3
(set HSRP hello timer 1 hold timer 3 seconds)
(convention - HSRP group number 10 matches VLAN number)
standby 10 priority 200 preempt delay 60
(this is the primary gateway router for subnet 10)
(preempt delay 60 seconds allows EIGRP to stabilize before
HSRP switches back upon power recovery)
standby 10 ip 10.10.0.200
(10.10.0.200 is the HSRP gateway router address)
standby 10 track Vlan20 150
(if you lose the link to the backbone, drop priority
by 150 to initiate HSRP recovery)

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 9 of 26
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10
ip address 10.10.0.82 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby 10 timers 1 3
standby 10 priority 100 preempt delay 60
(HSRP secondary or backup gateway router for subnet 10)
standby 10 ip 10.10.0.200
standby 10 track Vlan23 50

Additional MSFC Configuration—Any Routed Interface to Backbone

interface Vlan21
ip address 10.21.0.81 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3

Additional MSFC Configuration—Interface on Management VLAN

interface Vlan99
ip address 172.26.196.81 255.255.254.0
no ip directed-broadcast

Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets

router eigrp 1
passive-interface Vlan10
passive-interface Vlan11
(no passive interface on VLAN 12 or VLAN 13)
(two VLANs are kept as redundant routed paths)
passive-interface Vlan99
network 10.0.0.0
3. Layer 3 Distribution with Dual-Path Layer 3 Core—Catalyst 6500
Please refer to Figures 3a, 3b, and 3c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to
test network recovery times as different failures were induced. This design has the highest level of redundancy and the highest link capacity
into the core.

This campus design features redundant Layer 3 switches in the distribution layer. For building block “west” the redundant switches are
dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has redundant two-port Gigabit
EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP
recovery will take place if both routed links to the core are broken.

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 10 of 26
Table 5 Testing with the Standard Building Block—Small Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 3s HSRP

Restore 0s HSRP

Distribution Switch

Fail 3s HSRP

Restore 0s HSRP

Core Switch

Fail 0s Dual-path redundancy

Restore 6s EIGRP

Table 6 Testing with the Standard Building Block—Large Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 3s HSRP

Restore 0s HSRP

Distribution Switch

Fail 3s HSRP

Restore 0s HSRP

Core Switch

Fail 0s Dual-path redundancy

Restore 20s EIGRP

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 11 of 26
Table 7 Testing with the Standard Building Block—Small Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 2s UplinkFast

Restore 0s UplinkFast

Distribution Switch

Fail 3s UplinkFast/HSRP

Restore 15s (See note below)

Core Switch

Fail 3s Dual-path redundancy

Restore 0s Dual-path redundancy

VLAN Backup Trunk

Fail 31s Spanning tree

Restore 28s Spanning tree

Note: Restoration of the distribution-layer switch causes a 15-second outage. This is a result of the fact that the Layer 2 and Layer 3 path
determination functions on the switch recover at slightly different moments. When the routing table is increased to 3000 routes in the next
test, this anomaly is eliminated.

Table 8 Testing with the VLAN Building Block—Large Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 2s UplinkFast

Restore 0s UplinkFast

Distribution Switch

Fail 3s UplinkFast/HSRP

Restore 5s (See note below)

Core Switch

Fail 3s Dual-path redundancy

Restore 0s Dual-path redundancy

VLAN Backup Trunk

Fail 31s Spanning tree

Restore 28s Spanning tree

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 12 of 26
Note: When the distribution switch is restored, a five-second outage results because the Layer 2 topology stabilizes at a slightly different
moment than the Layer 3 topology. With the large routing table this difference is reduced from the previous test.

Configuration of MSFC: Client-Side Interface on Distribution Switch—HSRP Primary

interface Vlan10
ip address 10.10.0.81 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
(set eigrp hello timer 1 second)
ip hold-time eigrp 1 3
(set eigrp hold timer 3 seconds)
standby 10 timers 1 3
(set HSRP hello timer 1 hold timer 3 seconds)
(convention - HSRP group number 10 matches VLAN number)
standby 10 priority 200 preempt delay 60
(this is the primary gateway router for subnet 10)
(preempt delay 60 seconds allows EIGRP to stabilize before
HSRP switches back upon power recovery)
standby 10 ip 10.10.0.200
(10.10.0.200 is the HSRP gateway router address)
standby 10 track Vlan20 75
standby 10 track Vlan21 75
(if you lose both links to the backbone, drop priority
by 150 to initiate HSRP recovery)

Configuration of MSFC: Client-Side Interface on Distribution Switch—HSRP Secondary

interface Vlan10
ip address 10.10.0.82 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby 10 timers 1 3
standby 10 priority 100 preempt delay 60
(HSRP secondary or backup gateway router for subnet 10)
standby 10 ip 10.10.0.200
standby 10 track Vlan22 25
standby 10 track Vlan23 25
(track both links to the backbone)

Additional MSFC Configuration—Any Routed Interface to Backbone

interface Vlan21
ip address 10.21.0.81 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3

Additional MSFC Configuration—Interface on Management VLAN

interface Vlan99
ip address 172.26.196.81 255.255.254.0
no ip directed-broadcast

Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets

router eigrp 1
passive-interface Vlan10
passive-interface Vlan11
passive-interface Vlan12
passive-interface Vlan13
passive-interface Vlan99
network 10.0.0.0

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 13 of 26
4. Layer 3 Distribution with Layer 3 Core—Catalyst 8540
Please refer to Figures A1a, A1b, and A1c for this section. Clients target1 through target6 were attached to the wiring closet switches in
order to test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the
core, but each building block as a whole has redundant connectivity into the core.

This campus design features redundant Layer 3 switches in the distribution layer. For building block “left” the redundant switches are
dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has a two-port Gigabit EtherChannel
connection to a Layer 3 switch in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP recovery will take place
if the routed link to the core is broken.

Table 9 Results with Small Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 4s HSRP

Restore 1s HSRP

Distribution Switch

Fail 3s HSRP

Restore 1s HSRP

Core Switch

Fail 6s EIGRP

Restore 3s HSRP

Table 10 Results with Large Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 4s HSRP

Restore 1s HSRP

Distribution Switch

Fail 3s HSRP

Restore 2s HSRP

Core Switch

Fail 7s EIGRP

Restore 3s HSRP

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 14 of 26
Configuration Details of HSRP Primary Distribution Router dlt85
interface Port-channel1
ip address 10.70.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface Port-channel2
ip address 10.71.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface GigabitEthernet0/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet1/0/0
ip address 10.60.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.60.0.200
standby track Port-channel1 150
!
interface GigabitEthernet1/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet2/0/0
ip address 10.61.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.61.0.200
standby track Port-channel1 150
!
interface GigabitEthernet2/0/1
no ip address
no ip directed-broadcast
channel-group 2
!
interface GigabitEthernet3/0/0
ip address 10.62.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 15 of 26
 standby timers 1 3
standby priority 200
standby preempt
standby ip 10.62.0.200
standby track Port-channel1 150
!
interface GigabitEthernet3/0/1
no ip address
no ip directed-broadcast
channel-group 2
!
interface Ethernet0 (management interface)
ip address 172.26.196.53 255.255.254.0
no ip directed-broadcast
!
………
!
router eigrp 1
passive-interface GigabitEthernet0/0/0
passive-interface GigabitEthernet1/0/0
passive-interface GigabitEthernet2/0/0
passive-interface GigabitEthernet3/0/0
network 10.0.0.0
!
………
end
5. Layer 3 Distribution with Dual-Path Layer 3 Core—Catalyst 8540
Please refer to Figures A2a, A2b, and A2c for this section. Clients target1 through target6 were attached to the wiring closet switches in
order to test network recovery times as different failures were induced. The dual-path design has the highest level of redundancy and the
highest link capacity into the core.

This campus design features redundant Layer 3 switches in the distribution layer. For building block “left” the redundant switches are
dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has redundant two-port Gigabit
EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP
recovery will take place if both routed links to the core are broken.

Table 11 Results with Small Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 3s HSRP

Restore 0s HSRP

Distribution Switch

Fail 3s HSRP

Restore 0s HSRP

Core Switch

Fail 3s EIGRP

Restore 1s EIGRP

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 16 of 26
Table 12 Results with Large Routing Table

Time of Recovery
Test Failure Interruption Mechanism

Wiring Closet Uplink

Fail 3s HSRP

Restore 0s HSRP

Distribution Switch

Fail 3s HSRP

Restore 11s (See note below)

Core Switch

Fail 10s EIGRP

Restore 1s EIGRP

Note: Restoring the distribution switch results in an outage between the time that HSRP switches and the EIGRP routing table is built. This
can be corrected by adding a “preempt delay” as in the following command:

standby 10 priority 100 preempt delay 60

Configuration details of HSRP primary distribution router dlt85

interface Port-channel1
ip address 10.70.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface Port-channel2
ip address 10.71.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface GigabitEthernet0/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet1/0/0
ip address 10.60.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.60.0.200
standby track Port-channel1 75
standby track Port-channel2 75
!

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 17 of 26
interface GigabitEthernet1/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet2/0/0
ip address 10.61.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.61.0.200
standby track Port-channel1 75
standby track Port-channel2 75
!
interface GigabitEthernet2/0/1
no ip address
no ip directed-broadcast
channel-group 2
!
interface GigabitEthernet3/0/0
ip address 10.62.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.62.0.200
standby track Port-channel1 75
standby track Port-channel2 75
!
interface GigabitEthernet3/0/1
no ip address
no ip directed-broadcast
channel-group 2
!
interface Ethernet0
ip address 172.26.196.53 255.255.254.0
no ip directed-broadcast
!
………
!
router eigrp 1
passive-interface GigabitEthernet0/0/0
passive-interface GigabitEthernet1/0/0
passive-interface GigabitEthernet2/0/0
passive-interface GigabitEthernet3/0/0
network 10.0.0.0
!
………
end

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 18 of 26
1A: Dual-Path Layer 2 Core—Layout, Catalyst 6500

aw3 ae3

West Block East Block

Standard VLAN
Test1 Test4
aw4 ae4
dwt65 ct65 det65

Test2 Test5
aw5 ae5

Test3 Test6
aw6 ae6

dwb65 cb65 deb65

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

1B: Dual-Path Layer 2 Core—Logical, Catalyst 6500

aw3 ae3
10.10.0.101 10.10.0.0 10.5x.0.0 10.50.0.101
172.26.196.11 West Block East Block 172.26.196.21
Standard VLAN

Test1 aw4 ae4

Test4
10.11.0.101 10.11.0.0 dwt65 ct65 det65 10.5x.0.0 10.51.0.101
172.26.196.12 10.x.0.25 10.x.0.29 172.26.196.22
10.x.0.200 Pri 10.31.0.0 10.x.0.200 Pri
172.26.196.25 172.26.196.27 172.26.196.29

Test2 aw5 ae5 Test5

10.12.0.101 10.12.0.0 10.5x.0.0 10.52.0.101
172.26.196.13 172.26.196.23
10.x.0.26 10.x.0.30
10.32.0.0
10.x.0.200 Sec 10.x.0.200 Sec
aw6 172.26.196.28 ae6
Test3 172.26.196.26 172.26.196.30 Test6
10.13.0.101 10.13.0.0 dwb65 cb65 deb65 10.5x.0.0 10.53.0.101
172.26.196.14 172.26.196.24

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 19 of 26
1C: Dual-Path Layer 2 Core—Backbone, Catalyst 6500

West Block East Block

dwt65 ct65 det65
VLAN 31 VLAN 31
10.31.0.0

VLAN 32 VLAN 32

VLAN 31 VLAN 31

10.32.0.0
VLAN 32 VLAN 32
dwb65 cb65 deb65

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

2A: Layer 3 Core—Layout, Catalyst 6500

aw3 ae3

West Block East Block

Standard VLAN
Test1 Test4
aw4 ae4
dwt65 ct65 det65

Test2 Test5
aw5 ae5

Test3 Test6
aw6 ae6

dwb65 cb65 deb65

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 20 of 26
2B: Layer 3 Core—Logical, Catalyst 6500

aw3 ae3
10.10.0.101 10.10.0.0 10.5x.0.0 10.50.0.101
172.26.196.11 West Block East Block 172.26.196.21
Standard VLAN

Test1 aw4 ae4

Test4
10.11.0.101 10.11.0.0 dwt65 ct65 det65 10.5x.0.0 10.51.0.101
172.26.196.12 10.x.0.25 10.x.0.29 172.26.196.22
10.x.0.200 Pri 10.x.0.27 10.x.0.200 Pri
172.26.196.25 172.26.196.27 172.26.196.29

Test2 aw5 ae5 Test5

10.12.0.101 10.12.0.0 10.5x.0.0 10.52.0.101
172.26.196.13 172.26.196.23
10.x.0.26 10.x.0.30
10.x.0.28
10.x.0.200 Sec 10.x.0.200 Sec
aw6 172.26.196.28 ae6
Test3 172.26.196.26 172.26.196.30 Test6
10.13.0.101 10.13.0.0 dwb65 cb65 deb65 10.5x.0.0 10.53.0.101
172.26.196.14 172.26.196.24

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

2C: Layer 3 Core—Backbone, Catalyst 6500

West Block East Block

dwt65 ct65 det65
10.20.0.0 10.40.0.0
VLAN 20 VLAN 20 VLAN 40 VLAN 40
VLAN 30

10.30.0.0

VLAN 30
10.23.0.0 10.43.0.0
VLAN 23 VLAN 23 VLAN 43 VLAN 43
dwb65 cb65 deb65

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 21 of 26
3A: Dual-Path Layer 3 Core—Layout, Catalyst 6500

aw3 ae3

West Block East Block

Standard VLAN
Test1 Test4
aw4 ae4
dwt65 ct65 det65

Test2 Test5
aw5 ae5

Test3 Test6
aw6 ae6

dwb65 cb65 deb65

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

3B: Dual-Path Layer 3 Core—Logical, Catalyst 6500

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 22 of 26
3C: Dual-Path Layer 3 Core—Backbone, Catalyst 6500

West Block East Block

dwt65 ct65 det65
VLAN 20 10.20.0.0 VLAN 20 VLAN 40 10.40.0.0 VLAN 40
VLAN 22 VLAN 41
VLAN 21 VLAN 42
VLAN 30
10.21.0.0 10.42.0.0

10.30.0.0

10.22.0.0 10.41.0.0
VLAN 30
VLAN 22 VLAN 41

10.23.0.0 VLAN 21 VLAN 42 10.43.0.0

VLAN 23 VLAN 23 VLAN 43 VLAN 43
dwb65 cb65 deb65

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

A1A: Layer 3 Core—Layout, Catalyst 8540

Left Block Right Block

al4 ar4
dlt85 ct85 drt85

Target1 Target3
al5 ar5

Target2 Target4
al6 ar6

dlb85 cb85 drb85

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 23 of 26
A1B: Layer 3 Core—Logical, Catalyst 8540

Left Block Right Block

al4 ae4
10.60.0.101 10.60.0.0 dlt85 ct65 drt85 10.100.0.0 10.100.0.101
172.26.196.50 10.x.0.53 10.x.0.59 172.26.196.61
10.x.0.200 Pri 10.x.0.57 10.x.0.200 Pri
172.26.196.53 172.26.196.57 172.26.196.59

Target1 al5 ae5 Target3

10.61.0.101 10.61.0.0 10.101.0.0 10.101.0.101
172.26.196.51 172.26.196.62
10.x.0.54 10.x.0.60
10.x.0.58
10.x.0.200 Sec 10.x.0.200 Sec
al6 172.26.196.58 ae6
Target2 172.26.196.54 172.26.196.60 Target4
10.62.0.101 10.62.0.0 dlb85 cb85 drb85 10.102.0.0 10.102.0.101
172.26.196.52 172.26.196.63

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

A1C: Layer 3 Core—Topology, Catalyst 8540

Left Block Right Block

dlt85 ct85 drt85
10.70.0.0 10.90.0.0

Port-chan 1 Port-chan 1 Port-chan 3 Port-chan 1

int gig 0/0/1 int gig 0/0/0 int gig 0/0/1 int gig 0/0/0
int gig 1/0/1 int gig 1/0/0 int gig 1/0/1 int gig 1/0/0
Port-chan 5
int gig 9/0/0
int gig 9/0/1

10.80.0.0

Port-chan 5
int gig 9/0/0
int gig 9/0/1
10.73.0.0 10.93.0.0

Port-chan 2 Port-chan 2 Port-chan 4 Port-chan 2

int gig 2/0/1 int gig 2/0/0 int gig 2/0/1 int gig 2/0/0
dlb85 int gig 3/0/1 int gig 3/0/0 cb85 int gig 3/0/1 int gig 3/0/0 drb85

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 24 of 26
A2A: Dual-Path Layer 3 Core—Layout, Catalyst 8540

Left Block Right Block

al4 ar4
dlt85 ct85 drt85

Target1 Target3
al5 ar5

Target2 Target4
al6 ae6

dlb85 cb85 drb85

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

A2B: Dual-Path Layer 3 Core—Logical, Catalyst 8540

Left Block Right Block

al4 ar4
10.60.0.101 10.60.0.0 dlt85 ct85 drt85 10.100.0.0 10.100.0.101
172.26.196.50 10.x.0.53 10.x.0.59 172.26.196.61
10.x.0.200 Pri 10.x.0.57 10.x.0.200 Pri
172.26.196.53 172.26.196.57 172.26.196.59

Target1 al5 ar5 Target3

10.61.0.101 10.61.0.0 10.101.0.0 10.101.0.101
172.26.196.51 172.26.196.62
10.x.0.54 10.x.0.60
10.x.0.58
10.x.0.200 Sec 10.x.0.200 Sec
al6 172.26.196.58 ar6
Target2 172.26.196.54 172.26.196.60 Target4
10.62.0.101 10.62.0.0 dlb85 cb85 drb85 10.102.0.0 10.102.0.101
172.26.196.52 172.26.196.63

Access Distribution Core Distribution Access

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Public
Copyright © 2000 Cisco Systems, Inc. All Rights Reserved.
Page 25 of 26
A2C: Dual-Path Layer 3 Core—Backbone, Catalyst 8540

Left Block Right Block

dlt85 ct85 drt85
Port 1 10.70.0.0 Port 1 Port 3 10.90.0.0 VLAN 40
Port 2 Port 4
Port 2 Port 2
Port 5
10.71.0.0 10.92.0.0

10.80.0.0

10.72.0.0 10.91.0.0
Port 5
Port 1 Port 1

10.73.0.0 Port 1 Port 3 10.93.0.0

Port 2 Port 2 Port 4 Port 2
dlb85 cb85 drb85

Gigabit VLAN Trunk

Gigabit Ethernet
Gigabit EtherChannel—Dual

Conclusion Geoff Haviland (haviland@cisco.com)—Network Design

Choose a deterministic, structured design model to achieve high Engineer.
availability in your enterprise network. Apply redundancy in the
mission-critical parts of the network. Scale the enterprise
network to the size required by choosing the appropriate
building block model and combining with the right backbone
model. For a better understanding of the different choices
referred to in this paper refer to the paper Gigabit Campus
Network Design—Principles and Architecture.

Corporate Headquarters European Headquarters Americas Asia Headquarters

Cisco Systems, Inc. Cisco Systems Europe s.a.r.l. Headquarters Nihon Cisco Systems K.K.
170 West Tasman Drive Parc Evolic, Batiment L1/L2 Cisco Systems, Inc. Fuji Building, 9th Floor
San Jose, CA 95134-1706 16 Avenue du Quebec 170 West Tasman Drive 3-2-3 Marunouchi
USA Villebon, BP 706 San Jose, CA 95134-1706 Chiyoda-ku, Tokyo 100
http://www.cisco.com 91961 Courtaboeuf Cedex USA Japan
Tel: 408 526-4000 France http://www.cisco.com http://www.cisco.com
800 553-NETS (6387) http://www-europe.cisco.com Tel: 408 526-7660 Tel: 81 3 5219 6250
Fax: 408 526-4100 Tel: 33 1 69 18 61 00 Fax: 408 527-0883 Fax: 81 3 5219 6001
Fax: 33 1 69 28 83 26

Cisco Systems has more than 200 offices in the following countries. Addresses, phone numbers, and fax numbers are listed on the
C i s c o C o n n e c t i o n O n l i n e We b s i t e a t h t t p : / / w w w. c i s c o . c o m / o f f i c e s .
Argentina • Australia • Austria • Belgium • Brazil • Canada • Chile • China • Colombia • Costa Rica • Croatia • Czech Republic • Denmark • Dubai, UAE Finland • France
• Germany • Greece • Hong Kong • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia Mexico • The Netherlands • New
Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Singapore Slovakia • Slovenia • South Africa • Spain •
Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela
Copyright © 2000 Cisco Systems, Inc. All rights reserved. Printed in the USA. Catalyst, Cisco, Cisco IOS, Cisco Systems, and the Cisco Systems logo are registered trademarks of Cisco Systems, Inc. or its affiliates in the U.S. and
certain other countries. All other trademarks mentioned in this document are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any of its resellers.
(9912R) 02/00 BW5858