Beruflich Dokumente
Kultur Dokumente
1
UCP Network Architecture and Configuration Guide
MK-92UCP084-00
Contents
Preface
vii
Intended audience
Product version
Accessing product documentation
Related documents
Comments
Getting help
vii
vii
vii
viii
ix
ix
Contents
UCP Network Architecture and Configuration Guide
iii
iv
Contents
UCP Network Architecture and Configuration Guide
21
21
21
23
23
24
24
25
27
27
29
32
34
34
34
36
40
41
41
42
43
43
44
51
54
55
56
57
57
63
76
76
77
79
79
81
89
Contents
UCP Network Architecture and Configuration Guide
89
89
90
91
92
93
95
95
97
98
vi
Contents
UCP Network Architecture and Configuration Guide
Preface
This book provides Hitachi Unified Compute Platform (UCP) IP
network architecture and configuration examples that are needed to plan
and prepare for a UCP installation.
Intended audience
This book is intended for network administrators and Hitachi Data Systems
(HDS) representatives who are involved in installing and configuring UCP. It
assumes that you are familiar with the network technologies, network
hardware and its command line interfaces.
Product version
This guide applies to UCP 4000 Revision 3 and UCP4000ERevision 2
systems. These systems were first introduced with UCPDirector 4.1.
Preface
UCP Network Architecture and Configuration Guide
vii
Related documents
You can also download the most current UCP 4.1-related PDFs
(http://ucp.io/docs/4.1/#PDFDownloads.htm).
Related documents
The following documents contain additional information about UCP:
viii
UCP Director API Reference Guide Describes how to use the UCP
Director API.
UCP DOC API ReferenceGuide Describes how to use the UCP DOC
API.
UCP DOC CLI Reference Guide Describes how to use the UCP DOC
CLI.
Preface
UCP Network Architecture and Configuration Guide
Comments
Comments
Please send us your comments on this document:
UCPDocumentationFeedback@hds.com
Include the document title and number, including the revision (for
example, -01), and refer to specific sections and paragraphs whenever
possible. All comments become the property of Hitachi Data Systems.
Thank you!
Getting help
Hitachi Data Systems Support Portal is the destination for technical
support for your current or previously sold storage systems, midrange and
enterprise servers, and combined solution offerings. The Hitachi Data
Systems customer support staff is available 24 hours a day, seven days a
week. If you need technical support, log into the Hitachi Data Systems
Support Portal for contact information: http://portal.hds.com
Hitachi Data Systems Community is a global online community for
HDScustomers, partners, independent software vendors, employees, and
prospects. It is an open discussion among these groups about the
HDSportfolio of products and services. It is the destination to get answers,
discover insights, and make connections. The HDSCommunity
complements our Support Portal and support services by providing an area
where you can get answers to noncritical issues and questions. Join the
conversation today! Go to http://community.hds.com, register, and
complete your profile.
Preface
UCP Network Architecture and Configuration Guide
ix
Preface
UCP Network Architecture and Configuration Guide
1
UCP Systems High Level Design
The high level design of the UCP system includes server, network, storage,
and software components designed to deliver a complete virtualized data
center as an end-to-end solution.
The following table lists the components for a UCP4000 Rev3 system.
UCP4000
w/Brocade
UCP4000 w/ Cisco
UCP 4000E
1-8
(1-16) - 2port
(1-11) - 4port
1-3
Blades - Hitachi
CB520H B3
2-64
(2-128) - 2port
(2-88) - 4port
2-24
In-chassis Ethernet
Switches
2 x PassThru per
chassis
2 x PassThru per
chassis
In-chassis Fibre
Channel Switches
N/A
Management Servers
(2,4,6,8) x Cisco
Nexus9372PX
2 x Cisco Nexus
9332PQ
2 x Cisco Nexus
5548UP
Management Ethernet
2 x Brocade ICX 7450
switches
N/A
Storage
2 x Brocade 6510
Hitachi VSPG200
Hitachi VSPG400
Hitachi VSPG600
Hitachi VSPG800
Hitachi VSPG1000
2
UCP4000 Brocade Systems
This chapter will cover the network architecture and uplink configuration
examples for UCP 4000 Brocade systems.
Physical Configurations
The following tables show port utilization of the Brocade VDX 6740 and ICX
7450 switches that can be used on the UCP 4000 Brocade model.
Port Utilization of Brocade VDX 6740 Switch
Usage
Speed
10Gbps
10Gbps
10Gbps
10Gbps
10Gbps
Note: In addition to the above list, 4 x 40G (QSFP) ports are available with
additional 40G port license.
Speed
1Gbps
10Gbps
1Gbps
1Gbps
1Gbps
1Gbps
10Gbps
Note: For routed traffic or northbound traffic, the uplink ports (port-channel)
trunk VLANs need to be configured manually.
Uplink Configurations
Finally, the port is set to allow specific vlans which are configured by UCP
Director through the switchport trunk allowed vlan add command. Once
user needs to modify the allowed vlans they can perform a Configure
host/cluster vlan operation and UCP Director will modify the port
configuration accordingly.
The "spanning-tree shutdown", "no fabric isl enable", and "no fabric trunk
enable" are configured on all of server facing ports during UCP Deployment.
UCP Director Software will not touch these configurations. HDS
recommends not to change these configurations.
Uplink Configurations
If the UCP system is based on Brocade network devices, understand which
of the following scenarios can be applied, and use the procedure described
in that section to connect the UCP system to the datacenter core network.
Uplink Configurations
Note: In addition to the port 1 to 8, 4 x 40G (QSFP) ports are available with
additional 40G port license.
Typically the following 3 scenarios can be applied.
#
Scenario
10
Uplink Configurations
Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.
11
Uplink Configurations
12
Uplink Configurations
13
Uplink Configurations
13. Connect cables to the customer switch, and verify the both port
channels are up, and all the ports are synchronized status.
Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.
14
Uplink Configurations
Important: Please make sure no fabric isl enable and no fabric trunk
enable are configured on ten gigabit interfaces. These settings will prevent to
form single fabric between customer VCS fabric and UCP VCS fabric.
15
Uplink Configurations
Scenario
16
Uplink Configurations
19. Connect cables to the customer switch, and verify all the ports are up.
Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.
SSH@R1-BR-ICX7450-Stacking#show lag id 10
Total number of LAGs: 2
Total number of deployed LAGs: 2
Total number of trunks created:2 (121 available)
LACP System Priority / ID: 1 / 748e.f880.5cc0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "FCX_ISL" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/47 to 1/1/48 e 2/1/47 to 2/1/48
Port Count: 4
Primary Port: 1/1/47
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC
Name
1/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
1/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
17
Uplink Configurations
18
Uplink Configurations
24. Connect cables to the customer switch, and verify all the ports are up.
Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.
SSH@R1-BR-ICX7450-Stacking#show lag id 10
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (121 available)
LACP System Priority / ID: 1 / 748e.f880.5cc0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "FCX_ISL" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/47 e 2/1/47
Port Count: 2
Primary Port: 1/1/47
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC
Name
1/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
2/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
SSH@R1-BR-ICX7450-Stacking#show lag id 11
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (121 available)
LACP System Priority / ID: 1 / 748e.f880.5cc0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "FCX_ISL" ID 11 (dynamic Deployed) ===
19
Uplink Configurations
LAG Configuration:
Ports: e 1/1/48 e
Port Count: 2
Primary Port: 1/1/48
Trunk Type: hash-based
LACP Key: 20011
Deployment: HW Trunk ID 3
Port Link State Dupl
Name
1/1/48 Up Forward Full
2/1/48 Up Forward Full
20
2/1/48
3
UCP4000 Cisco Systems
This chapter will cover the network architecture, internal network
configuration, and uplink configuration examples for UCP 4000 Cisco
Systems.
Physical Configurations
The following table shows configurations of the UCP 4000 with Cisco
Networking model.
Component
Configuration Detail
1G Management
10G Interfaces
ISL Interfaces
40G Interfaces
21
The following table shows port usage of the Cisco Nexus 9332PQ,
Nexus9372PX and Nexus 3048 switches that can be used on the UCP 4000
with Cisco Networking model.
Cisco Nexus 9332PQ Port Usage
Usage
Speed
40Gbps
40Gbps
16
40Gbps
Speed
10Gbps
40Gbps
40Gbps
10Gbps
10Gbps
22
Usage
Speed
1Gbps
10Gbps
10Gbps
1Gbps
1Gbps
1Gbps
1Gbps
Layer 2 Mode
This UCP configuration utilizes 9372PX switches in the access layer and
9332PQ switches in the aggregation layer. vPC is configured between
9372PX access switch pairs and the 9332PQ aggregation switch pair. Each
9372PX switch has two 40G interfaces connected to each 9332PQ switch. All
interfaces between the access and aggregation are layer 2 trunk-ports
configured in back to back vPC port-channels. All vlans are allowed on the
vpc port-channels. The spanning tree root is configured on the aggregation
layer.
During purchase of the UCP system, there is an option to connect the UCP
system to the customer network via 8x10G links offered per 9372PX base
rack switch or 6x40G links offered per 9332PQ aggregation switch. Those
links can be configured as L2 interfaces or L3 interfaces depending on
preference.
23
Layer 3 Mode
This configuration utilizes 9372PX switches in the leaf layer and 9332PQ
switches in the spine layer. vPC is configured on all the leaf switch peers.
The leaf switches are where the layer 2 and layer 3 boundary is formed.
Additionally IGP peering is also configured on a vlan interface for
redundancy purposes on each leaf switch pair in the event there is an uplink
failure. Individual layer 3 links are configured between the spine and leaf
switches.
During purchase of the UCP system there is an option to connect the UCP
system to the customer network via 8x10G links offered per 9372PX base
rack switch or 6x40G links offered per 9332PQ spine switch. Those links can
be configured in as L2 links or L3 links depending on preference.
24
Switch Health Monitoring The UCP Director monitors the health of the
onboard switches and notifies if there are any warnings or errors detected
on the switches.
Configuration Backup UCP Director will periodically take backup
configurations of the switches in its inventory. UCP Director can also
manually backup/apply the switch configurations through user request.
Configure Host/Cluster Network This allows the user to configure
the Nexus 9372PX server facing interface on a blade by blade basis. The
native vlan which is configured and managed by UCP for the management
network and additional vlans such as vMotion or Compute vlans can also be
managed through this UCP Director feature.
VLAN DB management (Layer 2 mode) - If a new vlan is added
through the Configure Host/Cluster Network feature, the vlan is added to
the pair of Nexus 9372PX server facing interfaces as an allowed vlan and is
created in the vlan db. In addition, the vlan is added to the Nexus 9332PQ
vlan db and any/all additional Nexus 9372PX pairs vlan database in other
racks.
VLAN DB management (Layer 3 mode) - If a new vlan is added
through the Configure Host/Cluster Network feature, the vlan is added to
the pair of Nexus 9372PX server facing interfaces as an allowed vlan and is
created in the vlan db.
25
vPC peer-switch Allows both vPC peers to act as Root Bridge for the
vlan.
vPC ip arp synchronize Allows both vPC peers to synchronize arp
entries for faster ARP learning and convergence.
vPC delay restore 240 In the event of switch reboot, the vPC bring-up
is delayed by 240 seconds to allow network re-converge before bringing the
vPC peer up.
Rapid-PVST Mode (Layer 2 mode) Spanning tree rapid-PVST mode
is configured with the spanning tree root configured on the 9332PQ
aggregation switches with priority 57344. The access 9372PX switches have
priority setting 61440.
Rapid-PVST Mode (Layer 3 mode) Spanning tree rapid-PVST mode
is configured with the priority setting 61440 on the 9372PX leaf switches.
vPC port-channels (Layer 2 mode) - Back-to-Back vPC trunk portchannel is configured between aggregation and access switches, enabled all
vlans by default. Also, Back-to-Back vPC trunk port-channel is configured
between 1G Ethernet switch (Nexus3048) and access switches on compute
rack #1, enabled management vlan by default.
vPC port-channels (Layer 3 mode) - Back-to-Back vPC trunk portchannel is configured between 1G Ethernet switch (Nexus3048) and access
switches on compute rack #1, enabled management vlan by default.
Sample vPC Configuration
vpc domain 901
peer-switch
role priority 100
peer-keepalive destination 10.21.101.213 source 10.21.101.212
delay restore 240
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize
26
This section describes how to configure the UCP Internal Network after UCP
Initial Deployment in Layer 3 mode.
27
Design Options
Routing Protocol
HSRP, VRRP
Overlay Network
VM Migration
VXLAN (Overlay)
OSPF
Yes
eBGP
Yes
OSFP
VXLAN EVPN
Yes
eBGP
VXLAN EVPN
Not documented
Steps
28
(Run this step if Layer 2 VM Migration chosen and/or any overlay network needed for VM
traffics.)
Note: In reality, you may not want to spread out the Spine 9332 to Leaf 9372
subnets across such a large address space 10.1.1.10 to 10.2.8.0 etc. This was
used in the example configurations for simplicity. They may provide a /24 address
space and ask to break it up into /31 networks for each interface.
Network Design Requirements
Subnets
Interface
Host
(Example used in
Mask Size
this document)
Hosts Usage
(Example used in
this document)
Notes
Loopback 0
10.0.0.x/32
/32
10
hosts
Loopback 1
10.0.1.x/32
/32
10
hosts
9332A to 9372A
10.1.1.10/31,
10.1.1.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332A to 9372B
10.1.2.10/31,
10.1.2.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332A to 9372C
10.1.3.10/31,
10.1.3.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332A to 9372D
10.1.4.10/31,
10.1.4.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332A to 9372E
10.1.5.10/31,
10.1.5.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332A to 9372F
10.1.6.10/31,
10.1.6.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332A to 9372G
10.1.7.10/31,
10.1.7.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
29
Subnets
Interface
30
Host
(Example used in
Mask Size
this document)
Hosts Usage
(Example used in
this document)
Notes
9332A to 9372H
10.1.8.10/31,
10.1.8.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372A
10.2.1.10/31,
10.2.1.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372B
10.2.2.10/31,
10.2.2.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372C
10.2.3.10/31,
10.2.3.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372D
10.2.4.10/31,
10.2.4.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372E
10.2.5.10/31,
10.2.5.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372F
10.2.6.10/31,
10.2.6.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372G
10.2.7.10/31,
10.2.7.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9332B to 9372H
10.2.8.10/31,
10.2.8.20/31
/31
4
10,11,20,21
hosts
L3 interfaces
9372A to 9372B
10.0.10.212/31
/31
2
212,213
hosts
IGP between
Leafs
9372C to 9372D
10.0.10.214/31
/31
2
214,215
hosts
IGP between
Leafs
9372E to 9372F
10.0.10.216/31
/31
2
216,217
hosts
IGP between
Leafs
9372G to 9372H
10.0.10.218/31
/31
2
218,219
hosts
IGP between
Leafs
Compute Rack 1
SVI
ucpmanagement
172.21.100.x/24
/24
N/A
1,2,3
GW, HSRP
Compute Rack 2
SVI
ucpmanagement
172.22.100.x/24
/24
N/A
1,2,3
GW, HSRP
Compute Rack 3
SVI
ucpmanagement
172.23.100.x/24
/24
N/A
1,2,3
GW, HSRP
Subnets
Host
(Example used in
Mask Size
this document)
Hosts Usage
(Example used in
this document)
Compute Rack 4
SVI
ucpmanagement
172.24.100.x/24
/24
N/A
1,2,3
GW, HSRP
Loopback 0
Secondary
10.0.101.x/32
/32
1
host
101
VXLAN only
Loopback 0
Secondary
10.0.102.x/32
/32
1
host
102
VXLAN only
Loopback 0
Secondary
10.0.103.x/32
/32
1
host
103
VXLAN only
Loopback 0
Secondary
10.0.104.x/32
/32
1
host
104
VXLAN only
Loopback 254
10.254.254.254/32 /32
1
host
254
VXLAN only
Multicast Group
230.1.1.x/24
N/A
N/A
VXLAN only
Interface
/32
Notes
If the customer is using ebgp as the global routing protocol gather the
AS#s as well and fill into the column.
Example of eBGP AS Assignment
Rack Location
Switch
Router AS #
(Example used in this document)
Router-id
Compute Rack #1
9332A-U41
65001
loopback0
Compute Rack #1
9332B-U40
65001
loopback0
Compute Rack #1
9372A-U39
65101
loopback0
Compute Rack #1
9372B-U38
65101
loopback0
Compute Rack #2
9372C-U40
65102
loopback0
Compute Rack #2
9372D-U39
65102
loopback0
Compute Rack #3
9372E-U40
65103
loopback0
Compute Rack #3
9372F-U39
65103
loopback0
Compute Rack #4
9372G-U40
65104
loopback0
Compute Rack #4
9372H-U39
65104
loopback0
31
Policy-Based Routing
VXLAN
Please install the Layer 3 license on all Nexus 9332PQ and Nexus 9372PX
Switches before configuring layer 3 features. It should be included in the
order with the UCP purchase.
1. Login to the all Nexus 9372 and 9332 switches using SSH.
2. Verify "LAN_ENTERPRISE_SERVICES_PKG" license is not installed. If it
is already installed, skip rest of the steps for this switch, and move to
next switch.
switch# show license usage
Feature Ins Lic Status Expiry Date Comments
Count
------------------------------------------------------------------------------LAN_ENTERPRISE_SERVICES_PKG No - Unused -------------------------------------------------------------------------------
32
3. Obtain the serial number of the Nexus 9300 switch through the
following command.
switch# show license host-id
License hostid: VDH=FOX064317SQ
33
34
Feature OSPF
To enable OSPF on the switch we first need to enable the feature ospf. This
needs to be done on all spine and leaf switches.
3. Run following command on all Nexus 9332PQ and Nexus 9372PX
switches.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature ospf
Configure Loopback 1
The loopback 1 interface should be configured on each device, a suggested
configuration scheme is listed below. This loopback interface will be used as
the router id as well as for other uses.
Configuration for Loopback 1 needs to be configured on all spine and leaf
switches according to the table below.
5. Run following commands on all Nexus 9332PQ and Nexus 9372PX
switches.
R1-CS-9332-A-U42(config)# interface loopback1 <- Creates the
interface
35
Rack Location
Switch
Loopback 1 IP/
Router-id
VRF
OSPF Process
Compute Rack #1
9332A-U41
10.0.1.220/32
ucpmanagement ucpmanagement
Compute Rack #1
9332B-U40
10.0.1.221/32
ucpmanagement ucpmanagement
Compute Rack #1
9372A-U39
10.0.1.212/32
ucpmanagement ucpmanagement
Compute Rack #1
9372B-U38
10.0.1.213/32
ucpmanagement ucpmanagement
Compute Rack #2
9372C-U40
10.0.1.214/32
ucpmanagement ucpmanagement
Compute Rack #2
9372D-U39
10.0.1.215/32
ucpmanagement ucpmanagement
Compute Rack #3
9372E-U40
10.0.1.216/32
ucpmanagement ucpmanagement
Compute Rack #3
9372F-U39
10.0.1.217/32
ucpmanagement ucpmanagement
Compute Rack #4
9372G-U40
10.0.1.218/32
ucpmanagement ucpmanagement
Compute Rack #4
9372H-U39
10.0.1.219/32
ucpmanagement ucpmanagement
Subinterfaces
Subinterfaces are a division of a physical interface into multiple logical
interfaces. We use the subinterfaces for the vrf ucpmanagement so that the
entire interface does not need to be allocated for ucpmanagement. The
physical interface as well as remaining sub-interfaces can we utilized for
other purposes.
6. Configure the subinterfaces on each spine and leaf switch according to
the IP Scheme table below.
R1-CS-9372-A-U39(config)# interface Ethernet1/49.1 <- creates the
interface
R1-CS-9372-A-U39(config-subif)# mtu 9216
R1-CS-9372-A-U39(config-subif)# encapsulation dot1q 101 <- a unique
encapsulation for the subinterface
R1-CS-9372-A-U39(config-subif)# vrf member ucpmanagement
R1-CS-9372-A-U39(config-subif)# ip address 10.1.1.11/31 <- set the ip
via scheme below
R1-CS-9372-A-U39(config-subif)# ip ospf network point-to-point
36
Note: If configuring the mtu 9216 gives an error configure the mtu 9216 on
the parent interface first. Parent interface of Ethernet 1/49.1 would be Ethernet
1/49.
In the configuration example, we use encapsulation dot1q 101 as the tag
for the subinterface. The subinterface is then added to the vrf
ucpmanagement and ip address and ospf peering is configured. The
interface is now part of the vrf ucpmanagement and isolated from the global
routing table. In addition, OSPF peering has been enabled so all
neighboring routers will learn the routes.
Cisco Nexus9332-A IP Scheme
Link
#
Switch
Interface
IP
Encapsulation Speed
Neighbor
9332A-U41
Ethernet 1/1.1
40G
9372A-U39
9332A-U41
Ethernet 1/2.1
40G
9372A-U39
9332A-U41
Ethernet 1/3.1
40G
9372B-U38
9332A-U41
Ethernet 1/4.1
40G
9372B-U38
9332A-U41
Ethernet 1/5.1
40G
9372C-U40
9332A-U41
Ethernet 1/6.1
40G
9372C-U40
9332A-U41
Ethernet 1/7.1
40G
9372D-U39
9332A-U41
Ethernet 1/8.1
40G
9372D-U39
9332A-U41
Ethernet 1/9.1
40G
9372E-U40
10
9332A-U41
Ethernet 1/10.1
40G
9372E-U40
11
9332A-U41
Ethernet 1/11.1
40G
9372F-U39
12
9332A-U41
Ethernet 1/12.1
40G
9372F-U39
13
9332A-U41
Ethernet 1/13.1
40G
9372G-U40
14
9332A-U41
Ethernet 1/14.1
40G
9372G-U40
15
9332A-U41
Ethernet 1/15.1
40G
9372H-U39
16
9332A-U41
Ethernet 1/16.1
40G
9372H-U39
37
Switch
Interface
IP
Encapsulation Speed
Neighbor
9332B-U40
Ethernet 1/1.1
40G
9372A-U39
9332B-U40
Ethernet 1/2.1
40G
9372A-U39
9332B-U40
Ethernet 1/3.1
40G
9372B-U38
9332B-U40
Ethernet 1/4.1
40G
9372B-U38
9332B-U40
Ethernet 1/5.1
40G
9372C-U40
9332B-U40
Ethernet 1/6.1
40G
9372C-U40
9332B-U40
Ethernet 1/7.1
40G
9372D-U39
9332B-U40
Ethernet 1/8.1
40G
9372D-U39
9332B-U40
Ethernet 1/9.1
40G
9372E-U40
10
9332B-U40
Ethernet 1/10.1
40G
9372E-U40
11
9332B-U40
Ethernet 1/11.1
40G
9372F-U39
12
9332B-U40
Ethernet 1/12.1
40G
9372F-U39
13
9332B-U40
Ethernet 1/13.1
40G
9372G-U40
14
9332B-U40
Ethernet 1/14.1
40G
9372G-U40
15
9332B-U40
Ethernet 1/15.1
40G
9372H-U39
16
9332B-U40
Ethernet 1/16.1
40G
9372H-U39
Switch
Interface
49
9372A-U39
Ethernet 1/49.1
40G
9332A-U41
50
9372A-U39
Ethernet 1/50.1
40G
9332A-U41
51
9372A-U39
Ethernet 1/51.1
40G
9332B-U40
52
9372A-U39
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
38
Switch
Interface
49
9372B-U38
Ethernet 1/49.1
40G
9332A-U41
50
9372B-U38
Ethernet 1/50.1
40G
9332A-U41
51
9372B-U38
Ethernet 1/51.1
40G
9332B-U40
52
9372B-U38
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
Switch
Interface
49
9372C-U40
Ethernet 1/49.1
40G
9332A-U41
50
9372C-U40
Ethernet 1/50.1
40G
9332A-U41
51
9372C-U40
Ethernet 1/51.1
40G
9332B-U40
52
9372C-U40
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
Switch
Interface
49
9372D-U39
Ethernet 1/49.1
40G
9332A-U41
50
9372D-U39
Ethernet 1/50.1
40G
9332A-U41
51
9372D-U39
Ethernet 1/51.1
40G
9332B-U40
52
9372D-U39
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
Switch
Interface
49
9372E-U40
Ethernet 1/49.1
40G
9332A-U41
50
9372E-U40
Ethernet 1/50.1
40G
9332A-U41
51
9372E-U40
Ethernet 1/51.1
40G
9332B-U40
52
9372E-U40
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
Switch
Interface
49
9372F-U39
Ethernet 1/49.1
40G
9332A-U41
50
9372F-U39
Ethernet 1/50.1
40G
9332A-U41
51
9372F-U39
Ethernet 1/51.1
40G
9332B-U40
52
9372F-U39
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
39
Switch
Interface
49
9372G-U40
Ethernet 1/49.1
40G
9332A-U41
50
9372G-U40
Ethernet 1/50.1
40G
9332A-U41
51
9372G-U40
Ethernet 1/51.1
40G
9332B-U40
52
9372G-U40
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
Switch
Interface
49
9372H-U39
Ethernet 1/49.1
40G
9332A-U41
50
9372H-U39
Ethernet 1/50.1
40G
9332A-U41
51
9372H-U39
Ethernet 1/51.1
40G
9332B-U40
52
9372H-U39
Ethernet 1/52.1
40G
9332B-U40
IP
Encapsulation Speed
Neighbor
40
Switch
SVI IP
172.21.100.2/24
100
41
Rack Location
Switch
SVI IP
172.21.100.3/24
172.21.100.1
100
172.22.100.2/24
172.22.100.1
100
172.22.100.3/24
172.22.100.1
100
172.23.100.2/24
172.23.100.1
100
172.23.100.2/24
172.23.100.1
100
172.24.100.2/24
172.24.100.1
100
172.24.100.3/24
172.24.100.1
100
42
Note: Please decide now whether OSPF or eBGP will be the routing protocol of
choice and skip to related section.
OSPF
OSPF is a layer 3 interior gateway protocol (IGP). The feature must be
enabled and a process must be created. Finally, OSPF must be enabled on
IPv4/v6 individual interfaces which need to be advertised to neighbors.
Feature OSPF
1. To enable OSPF on the switch we first need to enable the feature ospf.
This needs to be done on all Nexus 9332 spine and Nexus 9372 leaf
switches.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature ospf
43
2. Configure the OSPF Process for all Nexus9332 and Nexus9372 switches,
use the table below to complete.
R1-CS-9332-A-U42(config)# router ospf 100
R1-CS-9332-A-U42(config-router)# router-id 10.0.0.220
Rack Location
Switch
OSPF ID
Router-id
100
10.0.0.220
100
10.0.0.221
100
10.0.0.212
100
10.0.0.213
100
10.0.0.214
100
10.0.0.215
100
10.0.0.216
100
10.0.0.217
100
10.0.0.218
100
10.0.0.219
eBGP
Note: If OSPF was chosen as the routing protocol of choice, please skip this
section. Otherwise, please continue.
BGP is becoming commonly used in Spine Leaf topologies because of its
scalability and flexibility.
Feature BGP
4. To enable BGP on the switch we first need to enable the feature BGP.
This needs to be done on all Nexus 9332 and Nexus 9372 switches.
44
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature bgp
5. Configure the BGP Process for all Nexus9332 and Nexus9372 switches,
use the table below to complete.
R1-CS-9332-A-U42(config)# router bgp 65001
R1-CS-9332-A-U42(config-router)# router-id 10.0.0.220 <- assign the
router id
Rack Location
Switch
Router AS #
Router-id
Compute Rack #1
9332A-U41
65001
10.0.0.220
Compute Rack #1
9332B-U40
65001
10.0.0.221
Compute Rack #1
9372A-U39
65101
10.0.0.212
Compute Rack #1
9372B-U38
65101
10.0.0.213
45
Rack Location
Switch
Router AS #
Router-id
Compute Rack #2
9372C-U40
65102
10.0.0.214
Compute Rack #2
9372D-U39
65102
10.0.0.215
Compute Rack #3
9372E-U40
65103
10.0.0.216
Compute Rack #3
9372F-U39
65103
10.0.0.217
Compute Rack #4
9372G-U40
65104
10.0.0.218
Compute Rack #4
9372H-U39
65104
10.0.0.219
46
Networks
Neighbor
Remote-AS
Neighbor
9332A-U41
10.1.1.10/31
10.1.1.11
65101
9372A-U39
9332A-U41
10.1.1.20/31
10.1.1.21
65101
9372A-U39
9332A-U41
10.1.2.10/31
10.1.2.11
65101
9372B-U38
9332A-U41
10.1.2.20/31
10.1.2.21
65101
9372B-U38
9332A-U41
10.1.3.10/31
10.1.3.11
65102
9372C-U40
9332A-U41
10.1.3.20/31
10.1.3.21
65102
9372C-U40
9332A-U41
10.1.4.10/31
10.1.4.11
65102
9372D-U39
9332A-U41
10.1.4.20/31
10.1.4.21
65102
9372D-U39
9332A-U41
10.1.5.10/31
10.1.5.11
65103
9372E-U40
9332A-U41
10.1.5.20/31
10.1.5.21
65103
9372E-U40
9332A-U41
10.1.6.10/31
10.1.6.11
65103
9372F-U39
9332A-U41
10.1.6.20/31
10.1.6.21
65103
9372F-U39
9332A-U41
10.1.7.10/31
10.1.7.11
65104
9372G-U40
9332A-U41
10.1.7.20/31
10.1.7.21
65104
9372G-U40
9332A-U41
10.1.8.10/31
10.1.8.11
65104
9372H-U39
9332A-U41
10.1.8.20/31
10.1.8.21
65104
9372H-U39
9332A-U41
10.0.10.220/31
10.0.10.221
65001
9332B-U40
9332A-U41
10.254.254.254/32
N/A
N/A
N/A
9332A-U41
10.0.0.220/32
N/A
N/A
N/A
Network Statement
Neighbor
Remote-AS
Neighbor
9332B-U40
10.2.1.10/31
10.2.1.11
65101
9372A-U39
9332B-U40
10.2.1.20/31
10.2.1.21
65101
9372A-U39
9332B-U40
10.2.2.10/31
10.2.2.11
65101
9372B-U38
9332B-U40
10.2.2.20/31
10.2.2.21
65101
9372B-U38
9332B-U40
10.2.3.10/31
10.2.3.11
65102
9372C-U40
9332B-U40
10.2.3.20/31
10.2.3.21
65102
9372C-U40
9332B-U40
10.2.4.10/31
10.2.4.11
65102
9372D-U39
9332B-U40
10.2.4.20/31
10.2.4.21
65102
9372D-U39
9332B-U40
10.2.5.10/31
10.2.5.11
65103
9372E-U40
9332B-U40
10.2.5.20/31
10.2.5.21
65103
9372E-U40
47
Switch
Network Statement
Neighbor
Remote-AS
Neighbor
9332B-U40
10.2.6.10/31
10.2.6.11
65103
9372F-U39
9332B-U40
10.2.6.20/31
10.2.6.21
65103
9372F-U39
9332B-U40
10.2.7.10/31
10.2.7.11
65104
9372G-U40
9332B-U40
10.2.7.20/31
10.2.7.21
65104
9372G-U40
9332B-U40
10.2.8.10/31
10.2.8.10
65104
9372H-U39
9332B-U40
10.2.8.20/31
10.2.8.20
65104
9372H-U39
9332B-U40
10.0.10.220/31
10.0.10.220
65001
9332A-U41
9332B-U40
10.254.254.254/32
N/A
N/A
N/A
9332B-U40
10.0.0.221/32
N/A
N/A
N/A
48
R1-CS-9372-A-U39(config-router-neighbor)#
R1-CS-9372-A-U39(config-router-neighbor)#
as 65001
R1-CS-9372-A-U39(config-router-neighbor)#
R1-CS-9372-A-U39(config-router-neighbor)#
as 65001
R1-CS-9372-A-U39(config-router-neighbor)#
R1-CS-9372-A-U39(config-router-neighbor)#
as 65001
R1-CS-9372-A-U39(config-router-neighbor)#
Network Statement
Neighbor
Remote-AS
Neighbor
9372A-U39
10.1.1.10/31
10.1.1.10
65001
9332A-U41
9372A-U39
10.1.1.20/31
10.1.1.20
65001
9332A-U41
9372A-U39
10.2.1.10/31
10.2.1.10
65001
9332B-U40
9372A-U39
10.2.1.20/31
10.2.1.20
65001
9332B-U40
9372A-U39
10.0.10.212/31
10.0.10.213
65001
9372B-U38
9372A-U39
10.0.10.212/32
N/A
N/A
N/A
Network Statement
Neighbor
Remote-AS
Neighbor
9372B-U38
10.1.2.10/31
10.1.2.10
65001
9332A-U41
9372B-U38
10.1.2.20/31
10.1.2.20
65001
9332A-U41
9372B-U38
10.2.2.10/31
10.2.2.10
65001
9332B-U40
9372B-U38
10.2.2.20/31
10.2.2.20
65001
9332B-U40
9372B-U38
10.0.10.212/31
10.0.10.212
65001
9372A-U39
9372B-U38
10.0.10.213/32
N/A
N/A
N/A
Network Statement
Neighbor
Remote-AS
Neighbor
9372C-U40
10.1.3.10/31
10.1.3.10
65001
9332A-U41
9372C-U40
10.1.3.20/31
10.1.3.20
65001
9332A-U41
9372C-U40
10.2.3.10/31
10.2.3.10
65001
9332B-U40
9372C-U40
10.2.3.20/31
10.2.3.20
65001
9332B-U40
9372C-U40
10.0.10.214/31
10.0.10.213
65001
9372B-U38
9372C-U40
10.0.10.214/32
N/A
N/A
N/A
49
Network Statement
Neighbor
Remote-AS
Neighbor
9372D-U39
10.1.4.10/31
10.1.4.10
65001
9332A-U41
9372D-U39
10.1.4.20/31
10.1.4.20
65001
9332A-U41
9372D-U39
10.2.4.10/31
10.2.4.10
65001
9332B-U40
9372D-U39
10.2.4.20/31
10.2.4.20
65001
9332B-U40
9372D-U39
10.0.10.214/31
10.0.10.214
65001
9332A-U41
9372D-U39
10.0.10.215/32
N/A
N/A
N/A
Network Statement
Neighbor
Remote-AS
Neighbor
9372E-U40
10.1.5.10/31
10.1.5.10
65001
9332A-U41
9372E-U40
10.1.5.20/31
10.1.5.20
65001
9332A-U41
9372E-U40
10.2.5.10/31
10.2.5.10
65001
9332B-U40
9372E-U40
10.2.5.20/31
10.2.5.20
65001
9332B-U40
9372E-U40
10.0.10.216/31
10.0.10.216
65001
9372F-U39
9372E-U40
10.0.10.216/32
N/A
N/A
N/A
Network Statement
Neighbor
Remote-AS
Neighbor
9372F-U39
10.1.6.10/31
10.1.6.10
65001
9332A-U41
9372F-U39
10.1.6.20/31
10.1.6.20
65001
9332A-U41
9372F-U39
10.2.6.10/31
10.2.6.10
65001
9332B-U40
9372F-U39
10.2.6.20/31
10.2.6.20
65001
9332B-U40
9372F-U39
10.0.10.216/31
10.0.10.215
65001
9372E-U40
9372F-U39
10.0.10.217/32
N/A
N/A
N/A
50
Network Statement
Neighbor
Remote-AS
Neighbor
9372G-U40
10.1.7.10/31
10.1.7.10
65001
9332A-U41
9372G-U40
10.1.7.20/31
10.1.7.20
65001
9332A-U41
9372G-U40
10.2.7.10/31
10.2.7.10
65001
9332B-U40
9372G-U40
10.2.7.20/31
10.2.7.20
65001
9332B-U40
9372G-U40
10.0.10.218/31
10.0.10.219
65001
9372H-U39
9372G-U40
10.0.10.218/32
N/A
N/A
N/A
Network Statement
Neighbor
Remote-AS
Neighbor
9372H-U39
10.1.8.10/31
10.1.8.10
65001
9332A-U41
9372H-U39
10.1.8.20/31
10.1.8.20
65001
9332A-U41
9372H-U39
10.2.8.10/31
10.2.8.10
65001
9332B-U40
9372H-U39
10.2.8.20/31
10.2.8.20
65001
9332B-U40
9372H-U39
10.0.10.218/31
10.0.10.218
65001
9372G-U40
9372H-U39
10.0.10.219/32
N/A
N/A
N/A
Switch
IP
Interface
Neighbor
9332A-U41
Ethernet 1/1
10.1.1.10/31 9372A-U39
9332A-U41
Ethernet 1/2
10.1.1.20/31 9372A-U39
9332A-U41
Ethernet 1/3
10.1.2.10/31 9372B-U38
9332A-U41
Ethernet 1/4
10.1.2.20/31 9372B-U38
9332A-U41
Ethernet 1/5
10.1.3.10/31 9372C-U40
9332A-U41
Ethernet 1/6
10.1.3.20/31 9372C-U40
9332A-U41
Ethernet 1/7
10.1.4.10/31 9372D-U39
51
Link #
Switch
IP
Interface
Neighbor
9332A-U41
Ethernet 1/8
10.1.4.20/31 9372D-U39
9332A-U41
Ethernet 1/9
10.1.5.10/31 9372E-U40
10
9332A-U41
Ethernet 1/10
10.1.5.20/31 9372E-U40
11
9332A-U41
Ethernet 1/11
10.1.6.10/31 9372F-U39
12
9332A-U41
Ethernet 1/12
10.1.6.20/31 9372F-U39
13
9332A-U41
Ethernet 1/13
10.1.7.10/31 9372G-U40
14
9332A-U41
Ethernet 1/14
10.1.7.20/31 9372G-U40
15
9332A-U41
Ethernet 1/15
10.1.8.10/31 9372H-U39
16
9332A-U41
Ethernet 1/16
10.1.8.20/31 9372H-U39
Switch
IP
Interface
Neighbor
9332B-U40
Ethernet 1/1
10.2.1.10/31 9372A-U39
9332B-U40
Ethernet 1/2
10.2.1.20/31 9372A-U39
9332B-U40
Ethernet 1/3
10.2.2.10/31 9372B-U38
9332B-U40
Ethernet 1/4
10.2.2.20/31 9372B-U38
9332B-U40
Ethernet 1/5
10.2.3.10/31 9372C-U40
9332B-U40
Ethernet 1/6
10.2.3.20/31 9372C-U40
9332B-U40
Ethernet 1/7
10.2.4.10/31 9372D-U39
9332B-U40
Ethernet 1/8
10.2.4.20/31 9372D-U39
9332B-U40
Ethernet 1/9
10.2.5.10/31 9372E-U40
10
9332B-U40
Ethernet 1/10
10.2.5.20/31 9372E-U40
11
9332B-U40
Ethernet 1/11
10.2.6.10/31 9372F-U39
12
9332B-U40
Ethernet 1/12
10.2.6.20/31 9372F-U39
13
9332B-U40
Ethernet 1/13
10.2.7.10/31 9372G-U40
14
9332B-U40
Ethernet 1/14
10.2.7.20/31 9372G-U40
15
9332B-U40
Ethernet 1/15
10.2.8.10/31 9372H-U39
16
9332B-U40
Ethernet 1/16
10.2.8.20/31 9372H-U39
52
Switch
9372A-U39
IP
Interface
Ethernet 1/49
Neighbor
10.1.1.11/31 9332A-U41
Link #
Switch
IP
Interface
Neighbor
50
9372A-U39
Ethernet 1/50
10.1.1.21/31 9332A-U41
51
9372A-U39
Ethernet 1/51
10.2.1.11/31 9332B-U40
52
9372A-U39
Ethernet 1/52
10.2.1.21/31 9332B-U40
Switch
IP
Interface
Neighbor
49
9372B-U38
Ethernet 1/49
10.1.2.11/31 9332A-U41
50
9372B-U38
Ethernet 1/50
10.1.2.21/31 9332A-U41
51
9372B-U38
Ethernet 1/51
10.2.2.11/31 9332B-U40
52
9372B-U38
Ethernet 1/52
10.2.2.21/31 9332B-U40
Switch
IP
Interface
Neighbor
49
9372C-U40
Ethernet 1/49
10.1.3.11/31 9332A-U41
50
9372C-U40
Ethernet 1/50
10.1.3.21/31 9332A-U41
51
9372C-U40
Ethernet 1/51
10.2.3.11/31 9332B-U40
52
9372C-U40
Ethernet 1/52
10.2.3.21/31 9332B-U40
Switch
Interface
IP
Neighbor
49
9372D-U39
Ethernet 1/49
10.1.4.11/31 9332A-U41
50
9372D-U39
Ethernet 1/50
10.1.4.21/31 9332A-U41
51
9372D-U39
Ethernet 1/51
10.2.4.11/31 9332B-U40
52
9372D-U39
Ethernet 1/52
10.2.4.21/31 9332B-U40
Switch
Interface
IP
Neighbor
49
9372E-U40
Ethernet 1/49
10.1.5.11/31 9332A-U41
50
9372E-U40
Ethernet 1/50
10.1.5.21/31 9332A-U41
51
9372E-U40
Ethernet 1/51
10.2.5.11/31 9332B-U40
52
9372E-U40
Ethernet 1/52
10.2.5.21/31 9332B-U40
53
Neighbor
Link # Switch
Interface
49
9372F-U39
Ethernet 1/49
10.1.6.11/31 9332A-U41
50
9372F-U39
Ethernet 1/50
10.1.6.21/31 9332A-U41
51
9372F-U39
Ethernet 1/51
10.2.6.11/31 9332B-U40
52
9372F-U39
Ethernet 1/52
10.2.6.21/31 9332B-U40
Switch
IP
Interface
Neighbor
49
9372G-U40
Ethernet 1/49
10.1.7.11/31 9332A-U41
50
9372G-U40
Ethernet 1/50
10.1.7.21/31 9332A-U41
51
9372G-U40
Ethernet 1/51
10.2.7.11/31 9332B-U40
52
9372G-U40
Ethernet 1/52
10.2.7.21/31 9332B-U40
Switch
IP
Interface
Neighbor
49
9372H-U39
Ethernet 1/49
10.1.8.11/31 9332A-U41
50
9372H-U39
Ethernet 1/50
10.1.8.21/31 9332A-U41
51
9372H-U39
Ethernet 1/51
10.2.8.11/31 9332B-U40
52
9372H-U39
Ethernet 1/52
10.2.8.21/31 9332B-U40
54
Switch
Int Vlan 10
Compute Rack #1
9372A-U39
10.0.10.212/31
Compute Rack #1
9372B-U38
10.0.10.213/31
Compute Rack #2
9372C-U40
10.0.10.214/31
Compute Rack #2
9372D-U39
10.0.10.215/31
Compute Rack #3
9372E-U40
10.0.10.216/31
Compute Rack #3
9372F-U39
10.0.10.217/31
Compute Rack #4
9372G-U40
10.0.10.218/31
Compute Rack #4
9372H-U39
10.0.10.219/31
Configure Loopback 0
10. Configure the loopback 0 interface so the switch loopback interface is
reachable. Repeat for 9332A,B,9372A,B,C,D,E,F,G,H.
R1-CS-9332-A-U42(config-if)# interface loopback 0
R1-CS-9332-A-U42(config-if)# ip address 10.0.0.220/32 <- Loopback 0
address below.
R1-CS-9332-A-U42(config-if)# ip router ospf 100 area 0.0.0.0 <-- Add
this if OSPF is used
Switch
Loopback 0 Primary
Compute Rack #1
9332A-U41
10.0.0.220/32
Compute Rack #1
9332B-U40
10.0.0.221/32
Compute Rack #1
9372A-U39
10.0.0.212/32
Compute Rack #1
9372B-U38
10.0.0.213/32
Compute Rack #2
9372C-U40
10.0.0.214/32
Compute Rack #2
9372D-U39
10.0.0.215/32
Compute Rack #3
9372E-U40
10.0.0.216/32
Compute Rack #3
9372F-U39
10.0.0.217/32
Compute Rack #4
9372G-U40
10.0.0.218/32
Compute Rack #4
9372H-U39
10.0.0.219/32
55
Verify Neighborships
Verify OSPF neighborships
11. Verify that the L3 ospf neighborships are in FULL state on all L3
interfaces that are connected to neighboring switches.
R1-CS-9332-A-U42# sh ip ospf neighbors vrf ucpmanagement
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 8
Neighbor ID Pri State Up Time Address Interface
10.0.0.212 1 FULL/ - 1d05h 10.1.1.11 Eth1/1
10.0.0.212 1 FULL/ - 1d05h 10.1.1.21 Eth1/2
10.0.0.213 1 FULL/ - 2d04h 10.1.2.11 Eth1/3
10.0.0.213 1 FULL/ - 2d04h 10.1.2.21 Eth1/4
10.0.0.214 1 FULL/ - 2d04h 10.1.3.11 Eth1/5
10.0.0.214 1 FULL/ - 2d04h 10.1.3.21 Eth1/6
10.0.0.215 1 FULL/ - 2d04h 10.1.4.11 Eth1/7
10.0.0.215 1 FULL/ - 2d04h 10.1.4.21 Eth1/8
10.0.0.216 1 FULL/ - 1d05h 10.1.5.11 Eth1/9
10.0.0.216 1 FULL/ - 1d05h 10.1.5.21 Eth1/10
10.0.0.217 1 FULL/ - 2d04h 10.1.6.11 Eth1/11
10.0.0.217 1 FULL/ - 2d04h 10.1.6.21 Eth1/12
10.0.0.218 1 FULL/ - 2d04h 10.1.7.11 Eth1/13
10.0.0.218 1 FULL/ - 2d04h 10.1.7.21 Eth1/14
10.0.0.219 1 FULL/ - 2d04h 10.1.8.11 Eth1/15
10.0.0.219 1 FULL/ - 2d04h 10.1.8.21 Eth1/16
56
10.1.5.21
10.1.6.11
10.1.6.21
10.1.7.11
10.1.7.21
10.1.8.11
10.1.9.21
4
4
4
4
4
4
4
65103
65103
65103
65104
65104
65104
65104
4378
4378
4378
4378
4378
4378
4378
4420
4420
4420
4420
4420
4420
4420
7
7
7
7
7
7
7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1d05h
1d05h
1d05h
1d05h
1d05h
1d05h
1d05h
10
10
10
10
10
10
10
Configure VXLAN
VXLAN is an overlay technology that encapsulates packets with a VXLAN
network identifier (VNI) to connect disjoint layer 2 networks within a layer 3
connected IP network. Traditionally layer 2 networks had to be connected
directly through a series of layer 2 devices. However, the need to connect
disjoint layer 2 networks for reasons such as vmotion for high availability or
flexibility to add hosts anywhere in the data center; VXLAN allows us the
freedom to tunnel traffic over a traditional Layer 3 network and scale with
multi-tenancy in mind.
Important: Please decide now whether the customer will use VXLAN Flood and
Learn or VXLAN/EVPN and skip to section accordingly. Only one VXLAN variant
can be configured at a time.
57
Important: This VXLAN flood and learn configuration guide assumes the Unicast
Routing has been already configured. If not, go back to "Configure Global Unicast
Routing" on page 43 section and complete either the OSPF or eBGP configuration.
Enable Required Features
Configure tcam-size
3. Change tcam-size for region arp-ether on all Leaf switches
9372A,B,C,D,E,F,G,H.
Note: To allocate tcam space for region arp-ether, we need to reallocate tcam
space. In this example, we reduce the tcam space from the vacl region, and reallocate to the arp-ether region. Please select an appropriate region to re-allocate
for arp-ether.
58
Switch
Loopback 0 Secondary
Compute Rack #1
9332A-U41
N/A
Compute Rack #1
9332B-U40
N/A
Compute Rack #1
9372A-U39
10.0.101.101/32
Compute Rack #1
9372B-U38
10.0.101.101/32
Compute Rack #2
9372C-U40
10.0.102.102/32
Compute Rack #2
9372D-U39
10.0.102.102/32
Compute Rack #3
9372E-U40
10.0.103.103/32
Compute Rack #3
9372F-U39
10.0.103.103/32
Compute Rack #4
9372G-U40
10.0.104.104/32
Compute Rack #4
9372H-U39
10.0.104.104/32
59
Here we specify the RP address and multicast group list associated with it.
In addition, anycast rp is configured for RP redundancy on both Spines.
9. Configure exactly the same on 9332A and 9332B.
60
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#ip pim rp-address 10.254.254.254 group-list
230.1.1.0/24 <- 10.254.254.254 is the anycast RP.
R1-CS-9332-A-U42(config)#ip pim anycast-rp 10.254.254.254 10.0.0.220
<- 10.0.0.220 is the local RP address.
R1-CS-9332-A-U42(config)#ip pim anycast-rp 10.254.254.254 10.0.0.221
<- 10.0.0.221 is the peer RP address.
Rack Location
Local RP
Peer RP
Switch
Anycast RP
Compute Rack #1
9332A-U41
10.254.254.254
10.0.0.220
10.0.0.221
Compute Rack #1
9332B-U40
10.254.254.254
10.0.0.221
10.0.0.220
61
R1-CS-9372-A-U40(config)#Vlan 50
R1-CS-9372-A-U40(config-vlan)# name migration_vlan
R1-CS-9372-A-U40(config-vlan)# vn-segment 10000 <- make vlan 50 part
of vni 10000
R1-CS-9372-A-U40(config-vlan)# interface nve1
R1-CS-9372-A-U40(config-if-nve)# member vni 10000
R1-CS-9372-A-U40(config-if-nve-vni)# mcast-group 230.1.1.1 <- assign
vni 10000 (vlan50) to multicast group 230.1.1.1
Note: Ideally, one VXLAN segment mapping to one IP multicast group is the way
to provide the optimal multicast forwarding. However, it is possible to have
multiple VXLAN segments share a single IP multicast group to achive the desired
VXLAN scalability. Having multiple-tenant VXLAN networks to share a multicast
group does not bring any implications to the Layer 2 isolation between the tenant
networks.
Verify VXLAN Configuration
13. Displays the nve peer status
R1-CS-9372-A-U40# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- ----------------nve1 10.0.0.101 Up CP 00:01:19 84b8.02ca.9625
62
10.1.1.11
10.1.1.21
10.1.2.11
10.1.2.21
10.1.3.11
10.1.3.21
10.1.4.11
10.1.4.21
Ethernet1/1
Ethernet1/2
Ethernet1/3
Ethernet1/4
Ethernet1/5
Ethernet1/6
Ethernet1/7
Ethernet1/8
00:02:54
00:00:19
00:00:03
00:00:03
00:00:02
00:00:02
00:00:02
00:00:02
00:01:37
00:01:25
00:01:41
00:01:41
00:01:42
00:01:42
00:01:42
00:01:42
1
1
1
1
1
1
1
1
no
no
no
no
no
no
no
no
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Note: If you can ping between vms but cannot ssh or send traffic, make sure that
jumbo frames are configured along the entire traffic path. This includes the
vswitch or vsphere distributed switch is using mtu 9000. In addition, make sure
the mtu is set to 9000 on the vm itself such as ifcfg-eth0 configuration if the vm is
a linux os.
Note: In VxLAN flood and learn mode (7.0(3)I1(2) and earlier), the default
gateway for VXLAN VLANs should be provisioned on external routing devices. In
VXLAN flood and learn mode (7.0(3)I2(1) and later), the default gateway for
VXLAN VLAN is recommended to be a centralized gateway on a pair of VPC devices
with FHRP (First Hop Redundancy Protocol) running between them. In BGP EVPN,
it is recommended to use the anycast gateway feature on all VTEPs. For more
information on configuring the default gateway on external routing devices please
reference this Cisco whitepaper.
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000series-switches/white-paper-c11-732453.html
VXLAN / EVPN
VXLAN over EVPN uses any layer 3 routing protocol as the underlay and
multicast for Broadcast, unknown unicast, and multicast. Arp flood and
learn is avoided and localized with ARP supression. VXLAN EVPN introduces
the idea of traffic segregation for mult-tenancy support.
63
Important: This VXLAN over EVPN configuration guide assumes that OSPF has
been chosen as the unicast routing protocol and has been configured already. This
section does not cover eBGP as the unicast routing protocol.
Also, this VXLAN over EVPN configuration requires deep networking knowledge.
For more information, please refer to the "Cisco Nexus 9000 Series NX-OS VXLAN
Configuration Guide".
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/
7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_
Configuration_Guide_7x.html
OSPF Underlay is used as the routing protocol of the underlying network
infrastructure. This is the IP Network that edge devices will use to
communicate. Although any unicast routing protocol is supported.
MP-iBGP L2VPN EVPN is used as the overlay network or virtual network
built over the underlying network infrastructure. This allows traffic
segregation for multi-tenancy support.
PIM-SM is the multicast routing protocol used for multi-destination traffic
such as broadcast, unknown unicast, and multicast.
Anycast-RP is used for rendezvous-point redundancy. The Anycast-rps
are configured on the spine switches with the same ip.
Anycast gateway is used to have the same gateway and mac address on
all leaf layer switches for a locally defined subnet. This is useful when there
are multiple VXLAN VTEPs to facilitate a centralized gateway as opposed to
having separate gateways.
Enable Required Features
16. Run following commands to enable required feature on all Nexus 9332 A
and B Spine switches.
R1-CS-9332-A-U42(config)#
R1-CS-9332-A-U42(config)#
R1-CS-9332-A-U42(config)#
R1-CS-9332-A-U42(config)#
nv overlay evpn
feature ospf
feature bgp
feature pim
17. Run following commands to enable required feature on all Nexus 9372
A,B,C,D,E,F,G,H Leaf switches.
R1-CS-9372-A-U40(config)#nv overlay evpn
R1-CS-9372-A-U40(config)#feature ospf
64
R1-CS-9372-A-U40(config)#feature
R1-CS-9372-A-U40(config)#feature
R1-CS-9372-A-U40(config)#feature
R1-CS-9372-A-U40(config)#feature
bgp
pim
vn-segment-vlan-based
nv overlay
Important: If ospf was configured the ip router ospf command should already
be present.
Rack Location
Switch
Loopback 0 Secondary
Compute Rack #1
9332A-U41
N/A
Compute Rack #1
9332B-U40
N/A
Compute Rack #1
9372A-U39
10.0.101.101/32
Compute Rack #1
9372B-U38
10.0.101.101/32
Compute Rack #2
9372C-U40
10.0.102.102/32
Compute Rack #2
9372D-U39
10.0.102.102/32
Compute Rack #3
9372E-U40
10.0.103.103/32
Compute Rack #3
9372F-U39
10.0.103.103/32
Compute Rack #4
9372G-U40
10.0.104.104/32
Compute Rack #4
9372H-U39
10.0.104.104/32
65
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#interface loopback0
R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9332-A-U42(config-if)#interface Ethernet1/1-16
R1-CS-9332-A-U42(config-if-range)# ip pim sparse-mode <- Enable
multicast on the interface
22. Here we specify the RP address and multicast group list associated with
it. In addition, anycast rp is configured for RP redundancy on both
Spines. Configure exactly the same on 9332A and 9332B.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
66
Rack Location
Local RP
Peer RP
Switch
Anycast RP
Compute Rack #1
9332A-U41
10.254.254.254
10.0.0.220
10.0.0.221
Compute Rack #1
9332B-U40
10.254.254.254
10.0.0.221
10.0.0.220
67
Rack Location
Switch
Router-id
Compute Rack #1
9332A-U42
10.0.0.220
Compute Rack #1
9332B-U41
10.0.0.221
Rack Location
Switch
Remote-AS #
Neighbor
65001
10.0.0.212
65001
10.0.0.213
65001
10.0.0.214
65001
10.0.0.215
65001
10.0.0.216
65001
10.0.0.217
65001
10.0.0.218
65001
10.0.0.219
68
Rack Location
Switch
Router-id
Compute Rack #1
9372A-U40
10.0.0.212
Compute Rack #1
9372B-U39
10.0.0.213
Compute Rack #2
9372C-U40
10.0.0.214
Compute Rack #2
9372D-U39
10.0.0.215
Compute Rack #3
9372E-U40
10.0.0.216
Compute Rack #3
9372F-U39
10.0.0.217
Compute Rack #4
9372G-U40
10.0.0.218
Compute Rack #4
9372H-U39
10.0.0.219
28. Create Vlans and associate to VN-Segment Vlan 2500 is used for
assigning to VRF for multi-tenancy purposes, Vlan 99 is a L2 only
69
30. Configure on all switches 9372A,B,C,D. Vlan 100 has svi created in
anycast gateway mode. Vlan 99 does not have one because it is L2 mode
only.
R1-CS-9372-A-U40(config)# interface vlan 100
R1-CS-9372-A-U40(config-if)# no shutdown
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# vrf member TENANT1
Warning: Deleted all L3 config on interface Vlan100
R1-CS-9372-A-U40(config-if)# ip address 192.168.100.1/24
R1-CS-9372-A-U40(config-if)# fabric forwarding mode anycast-gateway
70
nve #
30099 l2
rd auto
route-target
route-target
vni 30100 l2
rd auto
route-target
route-target
VNI
import auto
export auto
import auto
export auto
Vlan
Mcast Group
30099
230.1.1.99
99
30100
230.1.1.100
100
50000
N/A
2500
71
nve #
1
VNI
30101
Mcast Group
230.1.1.101
Vlan
101
72
73
74
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.4970]:[32]:[192.168.101.45]/272
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.77a0]:[32]:[10.0.50.40]/272
10.0.0.100 100 0 i
75
Uplink Configurations
Note: If you can ping between vms but cannot ssh or send traffic, make sure that
jumbo frames are configured along the entire traffic path. This includes the
vswitch or vsphere distributed switch is using mtu 9000. In addition, make sure
the mtu is set to 9000 on the vm itself such as ifcfg-eth0 configuration if the vm is
a linux os.
Uplink Configurations
With UCP 4000 with Cisco Networking model, 8 x 10G ports on Nexus
9372PX or 6 x 40G ports on Nexus 9332PQ can be utilized for uplink
connectivity. Also, 2 x 1G ports on Nexus 3048 are allocated for uplink
connectivity.
The following tables and diagrams show default port allocations for uplink
connectivity.
Switch
76
Uplink Ports
Total Uplink
Bandwidth
Speed
Nexus 9372PX
Port #33-40
(8 ports per switch)
10Gbps
160Gbps
Nexus 9332PQ
Port #26-32
(6 ports per switch)
40Gbps
480Gbps
Nexus 3048
Port #47-48
(2 ports per switch)
1Gbps
4Gbps
Note
Compute Rack
#1 only
Layer 2 mode
only
Uplink Configurations
Also, UCP 4000 with Cisco Networking model supports Layer 2 connectivity
and Layer 3 connectivity to the core network.
Scenario
77
Uplink Configurations
Scenario
&
Multiple Core Networks
78
Uplink Configurations
79
Uplink Configurations
5. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
6. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to core switch A
switchport mode trunk
switchport trunk allowed vlan <VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
speed 40000 <-- Configure this if ports are 40G ports
vpc 10
interface Port-channel 11
description port-channel to core switch B
switchport mode trunk
switchport trunk allowed vlan <VLANs>
spanning-tree port type normal
speed 40000 <-- Configure this if ports are 40G ports
vpc 11
80
Uplink Configurations
Scenario 3: Connecting to multiple core networks & multichassis ether channel technology enabled switches
Configure the uplink ports with two vPCs on the Nexus switches. Ensure
cross-connectivity of the physical connections to provide redundancy.
10. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
11. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to management core network
switchport mode trunk
switchport trunk allowed vlan <VLANs for management>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
speed 40000 <-- Configure this if ports are 40G ports
vpc 10
interface Port-channel 11
description port-channel to data core network
switchport mode trunk
switchport trunk allowed vlan <VLANs for data>
spanning-tree port type normal
speed 40000 <-- Configure this if ports are 40G ports
vpc 11
12. Configure the upstream ports for 1st vPC on both switches
interface Ethernet1/26
switchport mode trunk
switchport trunk allowed vlan <VLANs for management>
speed 40000 <-- Configure this if ports are 40G ports
channel-group 10 mode active
...
13. Configure the upstream ports for 2nd vPC on both switches
interface Ethernet1/28
switchport mode trunk
switchport trunk allowed vlan <VLANs for data>
81
Uplink Configurations
14. Ensure that the vPC is correctly configured by checking vPC status
#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <VLANs for Management>
11 Po11 up success success <VLANs for Data>
82
Uplink Configurations
17. Configure the upstream ports for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <VLANs for Management>
channel-group 10 mode active
18. Configure the upstream ports for 2nd vPC on both switches
interface Ethernet1/2
switchport mode trunk
switchport trunk allowed vlan <VLANs for Management>
channel-group 11 mode active
19. Configure the upstream ports for 3rd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <VLANs for Data>
channel-group 12 mode active
20. Configure the upstream ports for 4th vPC on both switches
interface Ethernet1/4
switchport mode trunk
switchport trunk allowed vlan <VLANs for Data>
channel-group 13 mode active
21. Ensure that the vPC is correctly configured by checking vPC status
#show vpc
vPC status
-----------------------------------------------------------------
83
Uplink Configurations
Scenario
OSPF Peering to Core Network
1
2
84
Uplink Configurations
R1-CS-9372-A-U40(config-if)# no shut
24. Verify that the ospf neighborships are in FULL state on all L3 interfaces
that are connected to neighboring switches.
R1-CS-9372-A-U40# sh ip ospf neighbors vrf ucpmanagement
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 8
Neighbor ID Pri State Up Time Address Interface
10.0.0.212 1 FULL/ - 1d05h 10.1.1.11 Eth1/1
10.0.0.212 1 FULL/ - 1d05h 10.1.1.21 Eth1/2
...
85
Uplink Configurations
28. Verify that the BGP neighborships are in FULL state on all L3 interfaces
that are connected to neighboring switches.
R1-CS-9372-A-U40# sh ip bgp summary
BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.0.0.208, local AS number 65001
BGP table version is 7, IPv4 Unicast config peers 5, capable peers 4
0 network entries and 0 paths using 0 bytes of memory
BGP attribute entries [0/0], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.1.11 4 65101 4378 4420 7 0 0 1d05h 10
10.1.1.21 4 65101 4378 4420 7 0 0 1d05h 10
...
31. [BGP case]Add the default route into routing bgp configuration.
R1-CS-9372-A-U40(config)#router bgp 65101 <- AS number
R1-CS-9372-A-U40(config-router-af)# network 0.0.0.0/0
86
Uplink Configurations
33. Verify that the BGP neighborships are in FULL state on all L3 interfaces
that are connected to neighboring switches.
R1-CS-9372-A-U40# ip route 0.0.0.0/0 10.20.1.11 <- fill in the ip
addresss with the customer's ip address of the L3 interface.
87
88
4
UCP4000E Systems
This chapter will cover the network architecture and uplink configuration
examples for UCP 4000E Systems.
Physical Configurations
The following table shows port usage of the Cisco Nexus 5548UP switches.
89
Speed
10Gbps
10Gbps
Max 24
10Gbps
1Gbps
10Gbps
1Gbps
8 or 16Gbps
90
91
92
Uplink Configurations
Uplink Configurations
The UCP 4000E system shares a pair of Cisco Nexus 5548UP switches for
data traffic and management traffic.
The Cisco 5548UP switches act as a single switch in Virtual Port Channel
(vPC) configuration for spanning tree topology determination. The different
options for upstream connectivity to the production network infrastructure
are dependent on whether the immediate upstream pair of switches is in
vPC or spanning-tree configuration.
By default, port 1 to 4 on both Cisco Nexus 5548UP switches are dedicated
for connecting core network. And, port channel 10 is configured for these
ports as a vPC.
Scenario
&
Single Core Network
93
Uplink Configurations
Scenario
&
Separated Core Network
&
Separated Core Network
94
Uplink Configurations
3. Configure the upstream ports 1/1-1/4 for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
channel-group 10 mode active
...
95
Uplink Configurations
to provide redundancy.
5. Open SSHclient software, and login to the Both Nexus 5548UP A and B
6. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to core switch A
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
vpc 10
interface Port-channel 11
description port-channel to core switch B
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
spanning-tree port type normal
vpc 11
7. Configure the upstream ports 1/1-1/2 for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
channel-group 10 mode active
...
8. Configure the upstream ports 1/3-1/4 for 2nd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
channel-group 11 mode active
...
96
Uplink Configurations
12. Configure the upstream ports 1/1-1/2 for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
channel-group 10 mode active
...
13. Configure the upstream ports 1/3-1/4 for 2nd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
channel-group 11 mode active
...
14. Ensure that the vPC is correctly configured by checking vPC status
97
Uplink Configurations
#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Management VLANs>
11 Po11 up success success <Data VLANs>
16. Configure the upstream port 1/1 for 1st vPC on both switches
98
Uplink Configurations
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
channel-group 10 mode active
17. Configure the upstream port 1/2 for 2nd vPC on both switches
interface Ethernet1/2
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
channel-group 11 mode active
18. Configure the upstream port 1/3 for 3rd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
channel-group 12 mode active
19. Configure the upstream port 1/4 for 4th vPC on both switches
interface Ethernet1/4
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
channel-group 13 mode active
20. Ensure that the vPC is correctly configured by checking vPC status
#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Management VLANs>
11 Po11 up success success <Management VLANs>
12 Po12 up success success <Data VLANs>
13 Po13 up success success <Data VLANs>
99
100
MK-92UCP084-00