Sie sind auf Seite 1von 112

Unified Compute Platform 4.

1
UCP Network Architecture and Configuration Guide

MK-92UCP084-00

2012-2015 Hitachi Data Systems Corporation. All rights reserved.


No part of this publication may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying and recording, or stored in a database or retrieval system for any
purpose without the express written permission of Hitachi Data Systems Corporation (hereinafter referred
to as Hitachi Data Systems).
Hitachi Data Systems reserves the right to make changes to this document at any time without notice and
assumes no responsibility for its use. This document contains the most current information available at the
time of publication. When new or revised information becomes available, this entire document will be
updated and distributed to all registered users.
Some of the features described in this document might not be currently available. Refer to the most recent
product announcement for information about feature and product availability, or contact Hitachi Data
Systems at https://support.hds.com/en_us/contact-us.html.
Notice: Hitachi Data Systems products and services can be ordered only under the terms and conditions of
the applicable Hitachi Data Systems agreements. The use of Hitachi Data Systems products is governed by
the terms of your agreements with Hitachi Data Systems.
By using this software, you agree that you are responsible for:
a) Acquiring the relevant consents as may be required under local privacy laws or otherwise from
employees and other individuals to access relevant data; and
b) Ensuring that data continues to be held, retrieved, deleted, or otherwise processed in accordance with
relevant laws.

Contents
Preface

vii

Intended audience
Product version
Accessing product documentation
Related documents
Comments
Getting help

vii
vii
vii
viii
ix
ix

Chapter 1: UCP Systems High Level Design

Chapter 2: UCP4000 Brocade Systems

Network Architecture and Configurations Overview


5
Physical Configurations
6
UCP Director Network Management Features
7
UCP Appliance Initial VCS and Spanning Tree Configuration
8
Server Facing Port Configurations
8
Uplink Configurations
9
Brocade VDX 6740
9
Scenario 1: Connecting to multi chassis ether channel technology
enabled switches
10
Scenario 2: Connecting to standard Ethernet switches (non-multi chassis
ether channel technology enabled switches)
12
Scenario 3: Connecting to Brocade VCS enabled core network using
vLAG
15
Brocade ICX 7450
16
Scenario 1: Connecting to multi chassis ether channel technology
enabled switches
16
Scenario 2: Connecting to standard Ethernet switches (non-multi chassis
ether channel technology enabled switches)
18

Contents
UCP Network Architecture and Configuration Guide

iii

Chapter 3: UCP4000 Cisco Systems


Network Architecture and Configurations Overview
Physical Configurations
Layer 2 / Layer 3 Mode
Layer 2 Mode
Layer 3 Mode
UCP Director Network Management Features
UCP Appliance Initial vPC and Spanning Tree Configuration
Server Facing Port Configurations
Internal Network Configurations (Layer 3 mode)
Network Design Table
Add Layer 3 License
Configure Unicast Routing for VRF ucpmanagement
Verify Virtual Routing and Forwarding (VRF)
OSPF for VRF ucpmanagement
Subinterfaces
Verify OSPF neighborships
Configure OSPF on the UCP Management SVI
Verify UCP Management SVI
Configure VM Traffic SVI
Configure Global Unicast Routing
OSPF
eBGP
Configure the Layer 3 interfaces
Configure L3 Peer Adjacency
Configure Loopback 0
Verify Neighborships
Configure VXLAN
VXLAN Flood and Learn
VXLAN / EVPN
Save All Switch Configurations
Uplink Configurations
Layer 2 Uplink Configurations
Scenario 1: Connecting to single core network & multi chassis ether
channel technology enabled switches
Scenario 2: Connecting to single core network & non-multi chassis ether
channel technology enabled switches
Scenario 3: Connecting to multiple core networks & multi-chassis ether
channel technology enabled switches

iv

Contents
UCP Network Architecture and Configuration Guide

21
21
21
23
23
24
24
25
27
27
29
32
34
34
34
36
40
41
41
42
43
43
44
51
54
55
56
57
57
63
76
76
77
79
79
81

Scenario 4: Connecting to multiple core networks & non-multi chassis ether


channel technology enabled switches
82
Layer 3 Uplink Configurations
84
Scenario 1: OSPF Peering to Core Network
84
Scenario 2: BGP Peering to Core Network
85
Scenario 3: Static Routing
86

Chapter 4: UCP4000E Systems

89

Network Architecture and Configurations Overview


Physical Configurations
UCP Director Network Management Features
UCP Appliance Initial vPC and Spanning Tree Configuration
Server Facing Port Configurations
Uplink Configurations
Scenario 1: Connecting to single core network & multi chassis ether
channel technology enabled switches
Scenario 2: Connecting to single core network & non-multi chassis ether
channel technology enabled switches
Scenario 3: Connecting to separate core networks & multi chassis ether
channel technology enabled switches
Scenario 4: Connecting to separate core networks & non-multi chassis
ether channel technology enabled switches

Contents
UCP Network Architecture and Configuration Guide

89
89
90
91
92
93
95
95
97
98

vi

Contents
UCP Network Architecture and Configuration Guide

Preface
This book provides Hitachi Unified Compute Platform (UCP) IP
network architecture and configuration examples that are needed to plan
and prepare for a UCP installation.

Intended audience
This book is intended for network administrators and Hitachi Data Systems
(HDS) representatives who are involved in installing and configuring UCP. It
assumes that you are familiar with the network technologies, network
hardware and its command line interfaces.

Product version
This guide applies to UCP 4000 Revision 3 and UCP4000ERevision 2
systems. These systems were first introduced with UCPDirector 4.1.

Accessing product documentation


In this release, we are introducing the new UCP Documentation Portal
(http://ucp.io/docs/4.1/). The documentation portal is a search-enabled
online resource containing the complete set of product documentation for
UCP 4000 and 4000E systems running either UCP Director 4.1 or UCP
Director Operations Center 4.1 software.
To ensure you are reading the most up-to-date UCP 4.1 product
documentation, we highly recommend that you search the online portal
first, before reviewing any offline documentation. Check this site for the
most current documentation, including important updates that may have
been made after the release of the product.

Preface
UCP Network Architecture and Configuration Guide

vii

Related documents

You can also download the most current UCP 4.1-related PDFs
(http://ucp.io/docs/4.1/#PDFDownloads.htm).

Related documents
The following documents contain additional information about UCP:

viii

UCP Pre-Installation Requirements and Configuration Guide


Contains information and procedures you need to be aware of for a
successful UCP installation.

UCP Network Architecture and Configuration Guide Contains


technical reference information about the networking architecture of
UCPsystems and provides procedures to help you configure thesystem
to work in your network environment.

UCP Director Administrator's Guide Contains technical and usage


information for UCP and UCP Director. Describes how to administer UCP
Director through UCP Director Console[[[Undefined variable
UCPLibrary.UCPDirectorWebInterface]]] with both VMware vCenter and
Microsoft SCVMM.

UCP Director API Reference Guide Describes how to use the UCP
Director API.

UCP Director CLI ReferenceGuide Describes how to use the UCP


Director CLI.

UCP Director Third-Party Copyrights and Licenses Contains


copyright and license information for the third-party software
distributed with or embedded in UCP Director.

UCP DOC Administrator's Guide Contains technical and usage


information for Unified Compute Platform Director Operations Center
(UCP DOC). Describes how to administer UCP DOC through UCP DOC
Console.

UCP DOC API ReferenceGuide Describes how to use the UCP DOC
API.

UCP DOC CLI Reference Guide Describes how to use the UCP DOC
CLI.

Preface
UCP Network Architecture and Configuration Guide

Comments

Comments
Please send us your comments on this document:

UCPDocumentationFeedback@hds.com
Include the document title and number, including the revision (for
example, -01), and refer to specific sections and paragraphs whenever
possible. All comments become the property of Hitachi Data Systems.

Thank you!

Getting help
Hitachi Data Systems Support Portal is the destination for technical
support for your current or previously sold storage systems, midrange and
enterprise servers, and combined solution offerings. The Hitachi Data
Systems customer support staff is available 24 hours a day, seven days a
week. If you need technical support, log into the Hitachi Data Systems
Support Portal for contact information: http://portal.hds.com
Hitachi Data Systems Community is a global online community for
HDScustomers, partners, independent software vendors, employees, and
prospects. It is an open discussion among these groups about the
HDSportfolio of products and services. It is the destination to get answers,
discover insights, and make connections. The HDSCommunity
complements our Support Portal and support services by providing an area
where you can get answers to noncritical issues and questions. Join the
conversation today! Go to http://community.hds.com, register, and
complete your profile.

Preface
UCP Network Architecture and Configuration Guide

ix

Preface
UCP Network Architecture and Configuration Guide

1
UCP Systems High Level Design
The high level design of the UCP system includes server, network, storage,
and software components designed to deliver a complete virtualized data
center as an end-to-end solution.
The following table lists the components for a UCP4000 Rev3 system.
UCP4000
w/Brocade

UCP4000 w/ Cisco

UCP 4000E

Hitachi CB500 chassis

1-8

(1-16) - 2port
(1-11) - 4port

1-3

Blades - Hitachi
CB520H B3

2-64

(2-128) - 2port
(2-88) - 4port

2-24

In-chassis Ethernet
Switches

2 x Brocade VDX 6746


per chassis

2 x PassThru per
chassis

2 x PassThru per
chassis

In-chassis Fibre
Channel Switches

2 x Brocade 6546 per chassis

N/A

Management Servers

(2/3) x T41S-2U nodes

Ethernet TOR switches 2 x Brocade VDX 6740

(2,4,6,8) x Cisco
Nexus9372PX
2 x Cisco Nexus
9332PQ

2 x Cisco Nexus
5548UP

Management Ethernet
2 x Brocade ICX 7450
switches

2 x Cisco Nexus 3048

N/A

Fibre Channel TOR


Switches

(2,4) x Broacade 6510 N/A

Storage

2 x Brocade 6510

Hitachi VSPG200
Hitachi VSPG400
Hitachi VSPG600
Hitachi VSPG800
Hitachi VSPG1000

Chapter 1: UCP Systems High Level Design


UCP Network Architecture and Configuration Guide

Hitachi VSP G200


Hitachi VSPG400
Hitachi VSPG600
Hitachi VSPG800
Hitachi VSPG1000
(Bolt-on only)

UCP 4000 Brocade Appliance Overview

UCP 4000 Cisco Appliance Overview (Front View)

Chapter 1: UCP Systems High Level Design


UCP Network Architecture and Configuration Guide

UCP 4000 Cisco Appliance Overview (Rear View)

UCP 4000E Appliance Overview

Chapter 1: UCP Systems High Level Design


UCP Network Architecture and Configuration Guide

Chapter 1: UCP Systems High Level Design


UCP Network Architecture and Configuration Guide

2
UCP4000 Brocade Systems
This chapter will cover the network architecture and uplink configuration
examples for UCP 4000 Brocade systems.

Network Architecture and Configurations Overview


In the Brocade Networking model, UCP utilizes Brocade Virtual Cluster
Switching (VCS) Technology to eliminate spanning tree, and optimize for
east west traffic. All VDX switches in a single UCP instance are configured
as VCS Fabric Cluster mode, and form single VCS fabric.
UCP also utilizes Virtual LAG (vLAG) technology which provides multi-path
networking between VCS fabric and your network.
The following diagram illustrates the LAN Architecture for the UCP 4000
Brocade model.

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Network Architecture and Configurations Overview

UCP 4000 Brocade Network Architecture

Physical Configurations
The following tables show port utilization of the Brocade VDX 6740 and ICX
7450 switches that can be used on the UCP 4000 Brocade model.
Port Utilization of Brocade VDX 6740 Switch
Usage

Ethernet Port Counts

Speed

Uplinks to customer core

10Gbps

ISL to other Brocade VDX 6740

10Gbps

ISL to ICX 7450 switches

10Gbps

Management server connectivity

10Gbps

ISL to CB500 in-chassis switches (Brocade


VDX 6746)

4 per CB500 chassis, up to 32 (8 x


CB500 chassis)

10Gbps

Note: In addition to the above list, 4 x 40G (QSFP) ports are available with
additional 40G port license.

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Network Architecture and Configurations Overview

Port Utilization of Brocade ICX 7450 Switch


Usage

Ethernet Port Counts

Speed

Uplinks to customer core

1Gbps

ISL to other Brocade ICX 7450


(Stacking)

10Gbps

CB500 SVP connectivity

1 per CB500 chassis, up to 8 (8 x CB500


chassis)

1Gbps

Management server BMC


connectivity

3 (ICX7450 A side only)

1Gbps

Storage system management


connectivity

1Gbps

Switch management connectivity

1 per switch, up to 4 (ICX 7450 A side only)

1Gbps

ISL to Brocade VDX 6740 switches 1

10Gbps

UCP Director Network Management Features


Onboarding Switches The UCP Director supports onboarding of the
Brocade VDX 6740 switches and Brocade VDX 6746 In-Chassis switches for
management and health monitoring.
Switch Health Monitoring The UCP Director monitors the health of the
onboard switches and notifies if there are any warnings or errors detected
on the switches.
Configuration Backup UCP Director will periodically take backup
configurations of the switches in its inventory. UCP Director can also
manually backup/apply the switch configurations through user request.
Configure Host/Cluster Network This allows the user to configure
the Brocade VDX 6746 server facing interfaces on a blade by blade basis.
The native vlan which is configured and managed by UCP for the
management network and additional vlans such as vMotion or Compute
vlans can also be managed through this UCP Director feature.
UCP utilizes Brocade VCS technology. By adding VLANs to server facing
ports on the VDX 6746, blades with the same VLAN can communicate with
each other immediately. You don't need to configure the VDX 6740 switch
for internal VLAN communication.

Note: For routed traffic or northbound traffic, the uplink ports (port-channel)
trunk VLANs need to be configured manually.

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Network Architecture and Configurations Overview

VLAN DB management - If a new vlan is added through the Configure


Host/Cluster Network feature, the vlan is added to the VDX 6746.

UCP Appliance Initial VCS and Spanning Tree Configuration


VCS Mode UCP 4000 supports Fabric Cluster mode only.
Important: UCP 4000 Brocade Revision 3 doesn't support VCS Logical Chassis
mode and related features.
VCS ID UCP Default VCS ID is "10" across entire UCPstack. HDS
representatives configure/change VCS ID based on customer engagement
information.
Important: Forming a single VCS fabric across the customer core network and
UCP network is NOT supported. If VCS is enabled on the core network and
Brocade VDX switches, please use a different VCS ID. Also, please make sure to
use vLAG technology to connect them.
Rbridge ID HDS representatives assign Rbridge ID with last octet of the
switch management IP address.
Rapid-PVST Mode Spanning tree rapid-PVST mode is configured with
priority setting 61440.
Sample Spanning tree configuration
protocol spanning-tree rpvst
bridge-priority 61440
!

Server Facing Port Configurations


The UCP Director manages the server facing ports of Brocade VDX 6746 InChassis switch which are connected to the blades.
When managing the server facing ports, UCP Director sets the port in
switchport trunk mode. In addition, it sets the native vlan on the port to the
specified management vlan id which is chosen during deployment. It is
critical these settings are never modified because the native vlan is used by
UCP Director for management functions.

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Finally, the port is set to allow specific vlans which are configured by UCP
Director through the switchport trunk allowed vlan add command. Once
user needs to modify the allowed vlans they can perform a Configure
host/cluster vlan operation and UCP Director will modify the port
configuration accordingly.
The "spanning-tree shutdown", "no fabric isl enable", and "no fabric trunk
enable" are configured on all of server facing ports during UCP Deployment.
UCP Director Software will not touch these configurations. HDS
recommends not to change these configurations.

Sample Port Configuration


interface TenGigabitEthernet 214/0/9
description Connecting to blade 0
no fabric isl enable
no fabric trunk enable
switchport
switchport mode trunk
switchport trunk allowed vlan add 57,59
switchport trunk tag native-vlan
switchport trunk native-vlan 57
spanning-tree shutdown
no shutdown
!

Uplink Configurations
If the UCP system is based on Brocade network devices, understand which
of the following scenarios can be applied, and use the procedure described
in that section to connect the UCP system to the datacenter core network.

Brocade VDX 6740


UCP system with Brocade VDX 6740 switch supports Brocade Virtual Link
Aggregation (vLAG) technology. It enables high-performance / redundant
networking across the UCP system and datacenter core networks.
By default, port 1 to 8 on both Brocade VDX 6740 switches are dedicated
for connecting core network. And, port channel 10 is configured for these
ports as a virtual LAG(vLAG).

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Note: In addition to the port 1 to 8, 4 x 40G (QSFP) ports are available with
additional 40G port license.
Typically the following 3 scenarios can be applied.
#

Scenario

Example of Network Topology

Connecting to multi chassis ether channel


technology enabled switches

Connecting to standard Ethernet switches


2

(non multi chassis Ethernet channel


technology switch)

Connecting to Brocade VCS enabled core


network using vLAG

Important: Forming single VCS


fabric across datacenter core network and
UCP network is NOT supported. If the VCS
is enabled on core network Brocade VDX
switches, please use vLAG technology to
connect them.

Scenario 1: Connecting to multi chassis ether channel technology enabled


switches
1. Login to the Brocade VDX6740 A using SSH.

10

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

2. Configure interface port-channel.


R1-BR-6740-A-U42# configure terminal
Entering configuration mode terminal
R1-BR-6740-A-U42(config)# interface Port-channel 10
R1-BR-6740-A-U42(config-Port-channel-10)# vlag ignore-split
R1-BR-6740-A-U42(config-Port-channel-10)# switchport
R1-BR-6740-A-U42(config-Port-channel-10)# switchport mode trunk
R1-BR-6740-A-U42(config-Port-channel-10)# switchport trunk allowed
vlan add 103
R1-BR-6740-A-U42(config-Port-channel-10)# spanning-tree bpdu-mac
0100.0ccc.cccdThis is required in case of connecting to Cisco
switch or Brocade FCX/ICX switch
R1-BR-6740-A-U42(config-Port-channel-10)# no spanning-tree shutdown
R1-BR-6740-A-U42(config-Port-channel-10)# no shutdown
R1-BR-6740-A-U42(config-Port-channel-10)# exit

Important: If core network switch is Cisco switch or Brocade ICX/FCX switch,


you must configure "spanning-tree bpdu-mac 0100.0ccc.cccd" on the portchannel for Spanning Tree interoperability .

3. Add Ten Gigabit interfaces into the port-channel, which connecting to


the core switch.
R1-BR-6740-A-U42(config)# interface TenGigabitEthernet 212/0/1-8
R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# no fabric isl enable
R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# no fabric trunk enable
R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# channel-group 10 mode active
type standard
R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# no shutdown

4. Repeat previous steps to the Brocade VDX6740 B

Important: Must use same port-channel number on both VDX6740 A & B


switches. By default, port-channel 10 is configured on both switches as vLAG.
5. Connect cables to the customer switch, and verify the both port
channels are up, and all the ports are synchronized status.

Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.

R1-BR-6740-A-U42# show interface port-channel 10

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

11

Uplink Configurations

Port-channel 10 is up, line protocol is up


Hardware is AGGREGATE, address is 0027.f8b3.1945
Current address is 0027.f8b3.1945
Description: Connecting to management switches
.

6. Verify all the ports are synchronized status.


R1-BR-6740-A-U42# show port-channel 10
LACP Aggregator: Po 10
Aggregator type: Standard
Ignore-split is enabled
Admin Key: 0010 - Oper Key 0010
Partner System ID - 0x0001,74-8e-f8-80-5c-c0
Partner Oper Key 20002
Member ports on rbridge-id 212:
Link: Te 212/0/1 (0xD418008000) sync: 1 *
Link: Te 212/0/2 (0xD418010001) sync: 1
Link: Te 212/0/3 (0xD418018002) sync: 1
Link: Te 212/0/4 (0xD418020003) sync: 1
Link: Te 212/0/5 (0xD418028004) sync: 1
Link: Te 212/0/6 (0xD418030005) sync: 1
Link: Te 212/0/7 (0xD418038006) sync: 1
Link: Te 212/0/8 (0xD418040007) sync: 1

Scenario 2: Connecting to standard Ethernet switches (non-multi chassis


ether channel technology enabled switches)
7. Login to the Brocade VDX6740 A using SSH.
8. Configure first interface port channel (for example, port-channel 10)
R1-BR-6740-A-U42# configure terminal
Entering configuration mode terminal
R1-BR-6740-A-U42(config)# interface Port-channel 10
R1-BR-6740-A-U42(config-Port-channel-10)# vlag ignore-split
R1-BR-6740-A-U42(config-Port-channel-10)# switchport
R1-BR-6740-A-U42(config-Port-channel-10)# switchport mode trunk
R1-BR-6740-A-U42(config-Port-channel-10)# switchport trunk allowed
vlan add 103
R1-BR-6740-A-U42(config-Port-channel-10)# spanning-tree bpdu-mac
0100.0ccc.cccdThis is required in case of connecting to Cisco
switch or Brocade FCX/ICX switch
R1-BR-6740-A-U42(config-Port-channel-10)# no spanning-tree shutdown
R1-BR-6740-A-U42(config-Port-channel-10)# no shutdown
R1-BR-6740-A-U42(config-Port-channel-10)# exit

12

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Important: If core network switch is Cisco switch or Brocade ICX/FCX switch,


you must configure "spanning-tree bpdu-mac 0100.0ccc.cccd" on the portchannel for Spanning Tree interoperability .
9. Add Ten Gigabit interfaces into the port-channel, which will be
connecting to core network Ethernet switch A
R1-BR-6740-A-U42(config)# interface TenGigabitEthernet 212/0/1-4
R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# no fabric isl enable
R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# no fabric trunk enable
R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# channel-group 10 mode active
type standard
R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# no shutdown

10. Configure second interface port-channel (for example, port-channel 11)


R1-BR-6740-A-U42(config)# interface Port-channel 11
R1-BR-6740-A-U42(config-Port-channel-11)# vlag ignore-split
R1-BR-6740-A-U42(config-Port-channel-11)# switchport
R1-BR-6740-A-U42(config-Port-channel-11)# switchport mode trunk
R1-BR-6740-A-U42(config-Port-channel-11)# switchport trunk allowed
vlan add 103
R1-BR-6740-A-U42(config-Port-channel-11)# spanning-tree bpdu-mac
0100.0ccc.cccdThis is required in case of connecting to Cisco
switch or Brocade FCX/ICX switch
R1-BR-6740-A-U42(config-Port-channel-11)# no spanning-tree shutdown
R1-BR-6740-A-U42(config-Port-channel-11)# no shutdown
R1-BR-6740-A-U42(config-Port-channel-11)# exit
Add Ten Gigabit interfaces into the port-channel, which connecting to
customers Ethernet switch B
R1-BR-6740-A-U42(config)# interface TenGigabitEthernet 212/0/5-8
R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# no fabric isl enable
R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# no fabric trunk enable
R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# channel-group 11 mode active
type standard
R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# no shutdown
R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# end

11. Repeat previous steps to the Brocade VDX6740 B

Important: Must use same port-channel numbers on both VDX6740 A & B


switches.
12. Verify the spanning-tree is running

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

13

Uplink Configurations

R1-BR-6740-A-U42# show spanning-tree brief


VLAN 1
Spanning-tree Mode: Rapid Per-VLAN Spanning Tree Protocol
Root ID Priority 61441
Address 01e0.5200.0a46
Hello Time 2, Max Age 20, Forward Delay 15
Bridge ID Priority 61441
Address 01e0.5200.0a46
Hello Time 2, Max Age 20, Forward Delay 15, TxHoldCount 6
Migrate Time 3 sec
Interface Role Sts Cost Prio Link-type Edge
--------------------------------------------------------------------Po 1 DES FWD 20000 128 P2P No
Po 2 DES FWD 20000 128 P2P No
Po 10 DIS DSC 20000000 128 P2P No
Po 11 DIS DSC 20000000 128 P2P No

13. Connect cables to the customer switch, and verify the both port
channels are up, and all the ports are synchronized status.

Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.

R1-BR-6740-A-U42# show interface port-channel 10


Port-channel 10 is up, line protocol is up
Hardware is AGGREGATE, address is 0027.f8b3.1945
Current address is 0027.f8b3.1945
Description: Connecting to management switches
.
R1-BR-6740-A-U42# show interface port-channel 11
Port-channel 11 is up, line protocol is up
Hardware is AGGREGATE, address is 0027.f8b3.1945
Current address is 0027.f8b3.1945
Description: Connecting to management switches
.

14. Verify all the ports are synchronized status.


R1-BR-6740-A-U42# show port-channel 10
LACP Aggregator: Po 10
Aggregator type: Standard
Ignore-split is enabled

14

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Admin Key: 0010 - Oper Key 0010


Partner System ID - 0x0001,74-8e-f8-80-5c-c0
Partner Oper Key 20002
Member ports on rbridge-id 212:
Link: Te 212/0/1 (0xD418008000) sync: 1 *
Link: Te 212/0/2 (0xD418010001) sync: 1
Link: Te 212/0/3 (0xD418018002) sync: 1
Link: Te 212/0/4 (0xD418020003) sync: 1
R1-BR-6740-A-U42# show port-channel 11
LACP Aggregator: Po 11
Aggregator type: Standard
Ignore-split is enabled
Admin Key: 0011 - Oper Key 0011
Partner System ID - 0x0001,74-8e-f8-80-5c-c1
Partner Oper Key 20003
Member ports on rbridge-id 212:
Link: Te 212/0/5 (0xD418028004) sync: 1 *
Link: Te 212/0/6 (0xD418030005) sync: 1
Link: Te 212/0/7 (0xD418038006) sync: 1
Link: Te 212/0/8 (0xD418040007) sync: 1

Scenario 3: Connecting to Brocade VCS enabled core network using vLAG


Important: Forming a single VCS fabric across the customer core network and
UCP network is NOT supported. If VCS is enabled on the core network and
Brocade VDX switches, please use a different VCS ID. Also, please make sure to
use vLAG technology to connect them.
15. Follow the same steps with Scenario 1: Connecting to multi chassis
ether channel technology enabled switches.

Important: Please make sure no fabric isl enable and no fabric trunk
enable are configured on ten gigabit interfaces. These settings will prevent to
form single fabric between customer VCS fabric and UCP VCS fabric.

R1-BR-6740-A-U42# show running-config interface TenGigabitEthernet


212/0/1
interface TenGigabitEthernet 212/0/1
description Port-Channel connecting to customer network
no fabric isl enable
no fabric trunk enable
channel-group 10 mode active type standard
lacp timeout long
shutdown

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

15

Uplink Configurations

Brocade ICX 7450


UCP system with Brocade networking model utilizes stacking technology of
Brocade ICX 7450 switch. It enables highly redundant networking across
UCP system and datacenter core networks.
Typically the following 2 scenarios can be applied. By default, port 47 to 48
on both Brocade ICX 7450 switches are dedicated for connecting core
network.
#

Scenario

Example of Network Topology

Connecting to multi chassis ether channel


technology enabled switches

Connecting to standard Ethernet switches


2

(non multi chassis Ethernet channel


technology switch)

Scenario 1: Connecting to multi chassis ether channel technology enabled


switches
16. Login to the Brocade ICX 7450 using SSH.
17. Configure port channel (Link Aggregation)
SSH@R1-BR-ICX7450-Stacking#configure terminal
SSH@R1-BR-ICX7450-Stacking(config)#lag customer_isl dynamic id 10
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#ports ethernet
1/1/47 to 1/1/48 ethernet 2/1/47 to 2/1/48
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#primary-port
1/1/47
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#deploy
LAG customer_isl deployed successfully!
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#exit

16

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

If ports on the customers switch are configured using trunk VLAN


(tagged VLAN), configure tagged VLAN on uplink ports.
SSH@R1-BR-ICX7450-Stacking(config)#vlan 100 UCP Management VLAN
ID
SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#no untagged ethernet
1/1/47
SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#tagged ethernet 1/1/47
Added tagged port(s) ethe 1/1/47 to 1/1/48 ethe 2/1/47 to 2/1/48 to
port-vlan 100.

18. Verify spanning tree is running


SSH@R1-BR-ICX7450-Stacking#show 802-1w
--- VLAN 1 [STP Instance owned by VLAN 1 ] --------------------------Bridge IEEE 802.1W Parameters:
Bridge Bridge Bridge Bridge Force tx
Identifier MaxAge Hello FwdDly Version Hold
hex sec sec sec cnt
ffff748ef8805cc0 20 2 15 Default 3
.

19. Connect cables to the customer switch, and verify all the ports are up.

Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.

SSH@R1-BR-ICX7450-Stacking#show lag id 10
Total number of LAGs: 2
Total number of deployed LAGs: 2
Total number of trunks created:2 (121 available)
LACP System Priority / ID: 1 / 748e.f880.5cc0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "FCX_ISL" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/47 to 1/1/48 e 2/1/47 to 2/1/48
Port Count: 4
Primary Port: 1/1/47
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC
Name
1/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
1/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

17

Uplink Configurations

2/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec


2/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec

Scenario 2: Connecting to standard Ethernet switches (non-multi chassis


ether channel technology enabled switches)
20. Login to the Brocade ICX 7450 using SSH.
21. Configure first port channel (Link Aggregation)
SSH@R1-BR-ICX7450-Stacking#configure terminal
SSH@R1-BR-ICX7450-Stacking(config)#lag customer_isl dynamic id 10
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#ports ethernet
1/1/47 ethernet 2/1/47
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#primary-port
1/1/47
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#deploy
LAG customer_isl deployed successfully!
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#exit
If ports on the customers switch are configured using trunk VLAN
(tagged VLAN), configure tagged VLAN on uplink ports.
SSH@R1-BR-ICX7450-Stacking(config)#vlan 100 UCP Management VLAN
ID
SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#no untagged ethernet
1/1/47
SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#tagged ethernet 1/1/47
Added tagged port(s) ethe 1/1/47 ethe 2/1/47 to port-vlan 100.

22. Configure second port channel (Link Aggregation)


SSH@R1-BR-ICX7450-Stacking#configure terminal
SSH@R1-BR-ICX7450-Stacking(config)#lag customer_isl dynamic id 11
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#ports ethernet
1/1/48 ethernet 2/1/48
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#primary-port
1/1/48
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#deploy
LAG customer_isl deployed successfully!
SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#exit
If ports on the customers switch are configured using trunk VLAN
(tagged VLAN), configure tagged VLAN on uplink ports.
SSH@R1-BR-ICX7450-Stacking(config)#vlan 100 UCP Management VLAN
ID
SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#no untagged ethernet
1/1/48
SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#tagged ethernet 1/1/48
Added tagged port(s) ethe 1/1/48 ethe 2/1/48 to port-vlan 100.

18

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

23. Verify spanning tree is running


SSH@R1-BR-ICX7450-Stacking#show 802-1w
--- VLAN 1 [STP Instance owned by VLAN 1 ] --------------------------Bridge IEEE 802.1W Parameters:
Bridge Bridge Bridge Bridge Force tx
Identifier MaxAge Hello FwdDly Version Hold
hex sec sec sec cnt
ffff748ef8805cc0 20 2 15 Default 3
.

24. Connect cables to the customer switch, and verify all the ports are up.

Important: Please make sure to connect each cable one by one, and each time a
connection is made verify the status of the port-channel.

SSH@R1-BR-ICX7450-Stacking#show lag id 10
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (121 available)
LACP System Priority / ID: 1 / 748e.f880.5cc0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "FCX_ISL" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/47 e 2/1/47
Port Count: 2
Primary Port: 1/1/47
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC
Name
1/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
2/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec
SSH@R1-BR-ICX7450-Stacking#show lag id 11
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (121 available)
LACP System Priority / ID: 1 / 748e.f880.5cc0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "FCX_ISL" ID 11 (dynamic Deployed) ===

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

19

Uplink Configurations

LAG Configuration:
Ports: e 1/1/48 e
Port Count: 2
Primary Port: 1/1/48
Trunk Type: hash-based
LACP Key: 20011
Deployment: HW Trunk ID 3
Port Link State Dupl
Name
1/1/48 Up Forward Full
2/1/48 Up Forward Full

20

2/1/48

Speed Trunk Tag Pvid Pri MAC


1G 1 Yes N/A 0 748e.f880.5cec
1G 1 Yes N/A 0 748e.f880.5cec

Chapter 2: UCP4000 Brocade Systems


UCP Network Architecture and Configuration Guide

3
UCP4000 Cisco Systems
This chapter will cover the network architecture, internal network
configuration, and uplink configuration examples for UCP 4000 Cisco
Systems.

Network Architecture and Configurations Overview


In the Cisco Networking model, UCP utilizes Cisco Nexus 9300 series, and its
advanced network technologies.

Physical Configurations
The following table shows configurations of the UCP 4000 with Cisco
Networking model.
Component

Configuration Detail

Spine / Aggregation layer

Cisco Nexus 9332PQ

Leaf / Access layer

Cisco Nexus 9372PX

1G Management

Cisco Nexus 3048

Server Facing Interfaces

10G Interfaces

ISL Interfaces

40G Interfaces

Max CB500 Chassis with 2-Nic


server

16 CB500 chassis (4 chassis per 9372PX pairs)

Max CB500 Chassis with 4-Nic


server

11 CB500 chassis (2 chassis base 9372PX pair, 3 chassis


remaining 3 pairs)

Uplinks per switch

8 x 10G on Cisco Nexus 9372PX or 6 x 40G on Cisco Nexus


9332PQ

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

21

Network Architecture and Configurations Overview

The following table shows port usage of the Cisco Nexus 9332PQ,
Nexus9372PX and Nexus 3048 switches that can be used on the UCP 4000
with Cisco Networking model.
Cisco Nexus 9332PQ Port Usage
Usage

Ethernet Port Counts

Speed

Uplinks to Core Network

40Gbps

ISL to other Nexus 9332PQ

2 (L2 Mode Only)

40Gbps

ISL to Nexus 9372PX

16

40Gbps

Cisco Nexus 9372PX Port Usage


Usage

Ethernet Port Counts

Speed

Uplinks to Core Network

8 (Compute Rack #1 Only)

10Gbps

ISL to other Nexus 9372PQ

40Gbps

ISL to Nexus 9332PX

40Gbps

ISL to Nexus 3048

1 (Compute Rack #1 Only)

10Gbps

Management Server Connectivity

3 (Compute Rack #1 Only)

10Gbps

CB500 Blade Server Connectivity


(Connecting to CB500 In-Chassis
Pass-Throu module)

Max 32 (Compute Rack #1)


10Gbps
Max 48 (Compute Rack #2-4)

Cisco Nexus 3048 Port Usage

22

Usage

Ethernet Port Counts

Speed

Uplinks to core network

1Gbps

ISL to other Nexus 3048

10Gbps

ISL to Nexus 9372PX

10Gbps

CB500 SVP connectivity

1 per CB500 chassis, up to 16 (16 x CB500


chassis)

1Gbps

Management server BMC connectivity

3 (Nexus 3048 A side only)

1Gbps

Storage system management


connectivity

1Gbps

Switch management connectivity

1 per switch, up to 10 (Nexus 3048 A side


only)

1Gbps

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Network Architecture and Configurations Overview

Layer 2 / Layer 3 Mode


UCP 4000 with Cisco offers 2 types of network layer configurations (Layer 2
/ Layer 3 mode).

Layer 2 Mode
This UCP configuration utilizes 9372PX switches in the access layer and
9332PQ switches in the aggregation layer. vPC is configured between
9372PX access switch pairs and the 9332PQ aggregation switch pair. Each
9372PX switch has two 40G interfaces connected to each 9332PQ switch. All
interfaces between the access and aggregation are layer 2 trunk-ports
configured in back to back vPC port-channels. All vlans are allowed on the
vpc port-channels. The spanning tree root is configured on the aggregation
layer.
During purchase of the UCP system, there is an option to connect the UCP
system to the customer network via 8x10G links offered per 9372PX base
rack switch or 6x40G links offered per 9332PQ aggregation switch. Those
links can be configured as L2 interfaces or L3 interfaces depending on
preference.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

23

Network Architecture and Configurations Overview

Layer 3 Mode
This configuration utilizes 9372PX switches in the leaf layer and 9332PQ
switches in the spine layer. vPC is configured on all the leaf switch peers.
The leaf switches are where the layer 2 and layer 3 boundary is formed.
Additionally IGP peering is also configured on a vlan interface for
redundancy purposes on each leaf switch pair in the event there is an uplink
failure. Individual layer 3 links are configured between the spine and leaf
switches.
During purchase of the UCP system there is an option to connect the UCP
system to the customer network via 8x10G links offered per 9372PX base
rack switch or 6x40G links offered per 9332PQ spine switch. Those links can
be configured in as L2 links or L3 links depending on preference.

UCP Director Network Management Features


Onboarding Switches (Layer 2 Mode) The UCP Director supports
onboarding of the Nexus 9372PX access switches and 9332PQ aggregation
switches for management and health monitoring.
Onboarding Switches (Layer 3 mode) The UCP Director supports
onboarding of the Nexus 9372PX leaf switch only for management and
health monitoring. Currently the 9332PQ spine switch onboarding is not
supported.

24

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Network Architecture and Configurations Overview

Switch Health Monitoring The UCP Director monitors the health of the
onboard switches and notifies if there are any warnings or errors detected
on the switches.
Configuration Backup UCP Director will periodically take backup
configurations of the switches in its inventory. UCP Director can also
manually backup/apply the switch configurations through user request.
Configure Host/Cluster Network This allows the user to configure
the Nexus 9372PX server facing interface on a blade by blade basis. The
native vlan which is configured and managed by UCP for the management
network and additional vlans such as vMotion or Compute vlans can also be
managed through this UCP Director feature.
VLAN DB management (Layer 2 mode) - If a new vlan is added
through the Configure Host/Cluster Network feature, the vlan is added to
the pair of Nexus 9372PX server facing interfaces as an allowed vlan and is
created in the vlan db. In addition, the vlan is added to the Nexus 9332PQ
vlan db and any/all additional Nexus 9372PX pairs vlan database in other
racks.
VLAN DB management (Layer 3 mode) - If a new vlan is added
through the Configure Host/Cluster Network feature, the vlan is added to
the pair of Nexus 9372PX server facing interfaces as an allowed vlan and is
created in the vlan db.

UCP Appliance Initial vPC and Spanning Tree Configuration


vPC peer-link The vPC peerlink trunk port-channel interface which
allows traffic to ingress/egress between vPC peers also allows all vlans by
default.
vPC keep-alive The vPC keep-alive link is established through the
management vrf with the source as MGMT 0 IP and destination as the
MGMT 0 IP on the peer.
vPC orphan-port suspend In the event that there is a vPC peer-link
failure, all server facing vPC orphan ports will be disabled to avoid a vPC
dual-active scenario.
vPC peer-gateway Peer gateway is configured on vPC peers to act as
the gateway even when packets are destined to the vPC peers MAC address.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

25

Network Architecture and Configurations Overview

vPC peer-switch Allows both vPC peers to act as Root Bridge for the
vlan.
vPC ip arp synchronize Allows both vPC peers to synchronize arp
entries for faster ARP learning and convergence.
vPC delay restore 240 In the event of switch reboot, the vPC bring-up
is delayed by 240 seconds to allow network re-converge before bringing the
vPC peer up.
Rapid-PVST Mode (Layer 2 mode) Spanning tree rapid-PVST mode
is configured with the spanning tree root configured on the 9332PQ
aggregation switches with priority 57344. The access 9372PX switches have
priority setting 61440.
Rapid-PVST Mode (Layer 3 mode) Spanning tree rapid-PVST mode
is configured with the priority setting 61440 on the 9372PX leaf switches.
vPC port-channels (Layer 2 mode) - Back-to-Back vPC trunk portchannel is configured between aggregation and access switches, enabled all
vlans by default. Also, Back-to-Back vPC trunk port-channel is configured
between 1G Ethernet switch (Nexus3048) and access switches on compute
rack #1, enabled management vlan by default.
vPC port-channels (Layer 3 mode) - Back-to-Back vPC trunk portchannel is configured between 1G Ethernet switch (Nexus3048) and access
switches on compute rack #1, enabled management vlan by default.
Sample vPC Configuration
vpc domain 901
peer-switch
role priority 100
peer-keepalive destination 10.21.101.213 source 10.21.101.212
delay restore 240
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize

Sample Spanning tree configuration


spanning-tree vlan 1-3967 priority 61440

26

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Server Facing Port Configurations


The UCP Director manages the server facing ports which are connected to
the CB500 chassis via passthru modules that connect to blades. These
server facing ports are connected via the 10G ports on the Nexus 9372PX
switches.
When managing the server facing ports, UCP Director sets the port in
switchport trunk mode. In addition, it sets the native vlan on the port to the
specified management vlan id which is chosen during deployment. It is
critical these settings are never modified because the native vlan is used by
UCP Director for management functions.
Finally, the port is set to allow specific vlans which are configured by UCP
Director through the switchport trunk allowed vlan command. Once user
needs to modify the allowed vlans they can perform a Configure
host/cluster vlan operation and UCP Director will modify the port
configuration accordingly.
The "spanning-tree port type edge trunk" and "vpc orphan-port suspend"
are configured on all of server facing ports during UCP Deployment. UCP
Director Software will not touch these configuration. HDS recommends not
to change these configuration.

Sample Port Configuration


interface Ethernet1/1
description to_blade_server
switchport mode trunk
switchport trunk native vlan 92
switchport trunk allowed vlan 92-94
spanning-tree port type edge trunk
vpc orphan-port suspend

Internal Network Configurations (Layer 3 mode)


Note: Skip this section if the UCP 4000 System is running in Layer 2 mode.

This section describes how to configure the UCP Internal Network after UCP
Initial Deployment in Layer 3 mode.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

27

Internal Network Configurations (Layer 3 mode)

In Layer 3 mode, there are a couple of design choices, and configuration


options as shown below. These design choices should be determined before
starting UCP Internal Network Configuration.
Design Options in Layer 3 mode (*This chapter provides sample
configurations for bold highlighted options.)
Item

Design Options

Routing Protocol

OSPF, eBGP, iBGP, EIGRP, Static Route

First Hop Redundancy Protocol

HSRP, VRRP

Isolation / Access Control

ACL, VRF for UCP Management Traffic

Overlay Network

None, VXLAN, VMware NSX

VM Migration

Layer 2 VM Migration with VXLAN, Layer 3 VM Migration

Important: Layer 3 VM Migration has limited support in UCPv4.1. (Cluster


Service Template feature doesn't support ESXi Cluster creation with Layer 3
vMotion vmkernel.)
Note: This guide covers following combination of Routing Protocl and VXLAN
method listed in below matrix table.
Routing Protocol (underlay)

VXLAN (Overlay)

Covered in this document

OSPF

VXLAN Flood and Learn

Yes

eBGP

VXLAN Flood and Learn

Yes

OSFP

VXLAN EVPN

Yes

eBGP

VXLAN EVPN

Not documented

Following table shows overall configuration workflow.


#

Steps

Configure VRF ucpmanagment

Configure UCP Management SVI

Configure VM Traffic SVI

Configure Unicast Routing


Configure VXLAN

28

(Run this step if Layer 2 VM Migration chosen and/or any overlay network needed for VM
traffics.)

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Network Design Table


Before continuing further please print out the following table of network
design requirements and host counts to configure the future sections. Two
columns are filled with subnet and host default values and used in the
configuration examples below. Gather the customer preference of available
subnets and hosts in their data-center and substitute accordingly. Also,
take note of whether VXLAN will be configured or not. If not, skip past those
fields. The notes column lets you know what will be used where.

Note: In reality, you may not want to spread out the Spine 9332 to Leaf 9372
subnets across such a large address space 10.1.1.10 to 10.2.8.0 etc. This was
used in the example configurations for simplicity. They may provide a /24 address
space and ask to break it up into /31 networks for each interface.
Network Design Requirements
Subnets
Interface

Host
(Example used in
Mask Size
this document)

Hosts Usage
(Example used in
this document)

Notes

Loopback 0

10.0.0.x/32

/32

10
hosts

212, 213, 214, 215, Router-id for


216, 217, 218,
global ospf or
219, 220, 221
ebgp

Loopback 1

10.0.1.x/32

/32

10
hosts

212, 213, 214, 215,


Router-id for vrf
216, 217, 218,
ucpmanagement
219, 220, 221

9332A to 9372A

10.1.1.10/31,
10.1.1.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332A to 9372B

10.1.2.10/31,
10.1.2.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332A to 9372C

10.1.3.10/31,
10.1.3.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332A to 9372D

10.1.4.10/31,
10.1.4.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332A to 9372E

10.1.5.10/31,
10.1.5.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332A to 9372F

10.1.6.10/31,
10.1.6.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332A to 9372G

10.1.7.10/31,
10.1.7.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

29

Internal Network Configurations (Layer 3 mode)

Subnets
Interface

30

Host
(Example used in
Mask Size
this document)

Hosts Usage
(Example used in
this document)

Notes

9332A to 9372H

10.1.8.10/31,
10.1.8.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372A

10.2.1.10/31,
10.2.1.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372B

10.2.2.10/31,
10.2.2.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372C

10.2.3.10/31,
10.2.3.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372D

10.2.4.10/31,
10.2.4.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372E

10.2.5.10/31,
10.2.5.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372F

10.2.6.10/31,
10.2.6.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372G

10.2.7.10/31,
10.2.7.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9332B to 9372H

10.2.8.10/31,
10.2.8.20/31

/31

4
10,11,20,21
hosts

L3 interfaces

9372A to 9372B

10.0.10.212/31

/31

2
212,213
hosts

IGP between
Leafs

9372C to 9372D

10.0.10.214/31

/31

2
214,215
hosts

IGP between
Leafs

9372E to 9372F

10.0.10.216/31

/31

2
216,217
hosts

IGP between
Leafs

9372G to 9372H

10.0.10.218/31

/31

2
218,219
hosts

IGP between
Leafs

Compute Rack 1
SVI
ucpmanagement

172.21.100.x/24

/24

N/A

1,2,3

GW, HSRP

Compute Rack 2
SVI
ucpmanagement

172.22.100.x/24

/24

N/A

1,2,3

GW, HSRP

Compute Rack 3
SVI
ucpmanagement

172.23.100.x/24

/24

N/A

1,2,3

GW, HSRP

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Subnets

Host
(Example used in
Mask Size
this document)

Hosts Usage
(Example used in
this document)

Compute Rack 4
SVI
ucpmanagement

172.24.100.x/24

/24

N/A

1,2,3

GW, HSRP

Loopback 0
Secondary

10.0.101.x/32

/32

1
host

101

VXLAN only

Loopback 0
Secondary

10.0.102.x/32

/32

1
host

102

VXLAN only

Loopback 0
Secondary

10.0.103.x/32

/32

1
host

103

VXLAN only

Loopback 0
Secondary

10.0.104.x/32

/32

1
host

104

VXLAN only

Loopback 254

10.254.254.254/32 /32

1
host

254

VXLAN only

Multicast Group

230.1.1.x/24

N/A

N/A

VXLAN only

Interface

/32

Notes

If the customer is using ebgp as the global routing protocol gather the
AS#s as well and fill into the column.
Example of eBGP AS Assignment
Rack Location

Switch

Router AS #
(Example used in this document)

Router-id

Compute Rack #1

9332A-U41

65001

loopback0

Compute Rack #1

9332B-U40

65001

loopback0

Compute Rack #1

9372A-U39

65101

loopback0

Compute Rack #1

9372B-U38

65101

loopback0

Compute Rack #2

9372C-U40

65102

loopback0

Compute Rack #2

9372D-U39

65102

loopback0

Compute Rack #3

9372E-U40

65103

loopback0

Compute Rack #3

9372F-U39

65103

loopback0

Compute Rack #4

9372G-U40

65104

loopback0

Compute Rack #4

9372H-U39

65104

loopback0

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

31

Internal Network Configurations (Layer 3 mode)

Add Layer 3 License


Optional license (N93-LAN1K9 LAN_ENTERPRISE_SERVICES_PKG) is
required for enterprise Layer 3 features on Cisco Nexus 9000 switch. The
following features are included in this license.

Open Shortest Path First (OSPF) Protocol

Border Gateway Protocol (BGP)

Intermediate System-to-Intermediate System (IS-IS) Protocol (Layer 3


only)

Protocol Independent Multicast (PIM), which includes sparse mode,


bidirectional mode, and Source-Specific Multicast (SSM) mode

Multicast Source Discovery Protocol (MSDP)

Policy-Based Routing

Generic routing encapsulation (GRE) tunnels

Enhanced Interior Gateway Routing Protocol (EIGRP)

VXLAN

BGP eVPN control plane

Please install the Layer 3 license on all Nexus 9332PQ and Nexus 9372PX
Switches before configuring layer 3 features. It should be included in the
order with the UCP purchase.
1. Login to the all Nexus 9372 and 9332 switches using SSH.
2. Verify "LAN_ENTERPRISE_SERVICES_PKG" license is not installed. If it
is already installed, skip rest of the steps for this switch, and move to
next switch.
switch# show license usage
Feature Ins Lic Status Expiry Date Comments
Count
------------------------------------------------------------------------------LAN_ENTERPRISE_SERVICES_PKG No - Unused -------------------------------------------------------------------------------

32

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

3. Obtain the serial number of the Nexus 9300 switch through the
following command.
switch# show license host-id
License hostid: VDH=FOX064317SQ

4. Take the serial # that follows the = sign.


5. Obtain the PAK(Product Authorization Key)
6. Locate the Website URL from the PAK key or go to
https://tools.cisco.com/SWIFT/LicensingUI/Home
7. Input the serial # and PAK key and other information the webpage asks.
A license file will be generated and emailed to you.
8. Copy the license file to the bootflash of the Nexus 9300 device using
supported file transfer command. Below is example using scp.
copy scp://username@172.17.92.242/license.lic bootflash://license.lic

9. Login to the switch, and install the license


switch# install license bootflash:license.lic
Installing license ..done

10. Verify license is installed


switch# show license usage
Feature Ins Lic Status Expiry Date Comments
Count
------------------------------------------------------------------------------LAN_ENTERPRISE_SERVICES_PKG Yes - In use Never -------------------------------------------------------------------------------

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

33

Internal Network Configurations (Layer 3 mode)

Configure Unicast Routing for VRF ucpmanagement


In this section, we go over the Nexus 9300 series switches and
configurations to bring up OSPF routing protocol for the VRF
ucpmanagement. This section explains how to enable and configure OSPF
routing between the Nexus 9332PQ spine switch and Nexus 9372PX leaf
switches.

Important: After the initial deployment by UCP deployment tool,


"ucpmanagement" vrf and SVI for ucpmanagement is pre-defined on Cisco Nexus
9372PX and 9332PQ switches.
If ACL, Firewall, or other function will be used for network isolation / access
control to UCP management network stack, please remove "ucpmanagement" vrf,
reconfigure ucpmanagement SVI, and skip this section.

Verify Virtual Routing and Forwarding (VRF)


VRFs are used to isolate Layer 3 traffic from the default vrf or global routed
traffic. This is done so that hypervisor management and UCP management
traffic routes are not reachable from the global routing table and makes the
management network more secure.
By UCP default, "ucpmanagement" vrf is pre-defined on Cisco Nexus
9372PX and 9332PQ switches.
1. Login to the all Nexus 9372 and 9332 switches using SSH.
2. Verify vrf 'ucpmanagement' is configured on all Nexus 9372PX and
9332PQ switches by running following command.
R1-CS-9372-A-U40# show vrf
VRF-Name VRF-ID State Reason
default 1 Up -management 2 Up -ucpmanagement 3 Up --

OSPF for VRF ucpmanagement


OSPF is a layer 3 interior gateway protocol (IGP). The feature must be
enabled and a process must be created. Finally, OSPF must be enabled on
IPv4/v6 individual interfaces which need to be advertised to neighbors.

34

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Feature OSPF
To enable OSPF on the switch we first need to enable the feature ospf. This
needs to be done on all spine and leaf switches.
3. Run following command on all Nexus 9332PQ and Nexus 9372PX
switches.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature ospf

Router OSPF Process


We need to create the router ospf process next. We have named it
ucpmanagement and specified the loopback 1 interface ip address as the
router-id. The router-id is what neighboring routers will see when uniquely
identifying the router. This needs to be done on all spine and leaf switches.
Here we create the ospf process, setup the vrf ucpmanagment in the ospf
process, assign the router id for peering, and assign the maximum paths.
4. Run following commands on all Nexus 9332PQ and Nexus 9372PX
switches.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42 (config)# router ospf ucpmanagement
R1-CS-9332-A-U42 (config-router)# vrf ucpmanagement
R1-CS-9332-A-U42 (config-router-vrf)# router-id 10.0.1.220 <- assign
via the table below
R1-CS-9332-A-U42 (config-router-vrf)# maximum-paths 32

Configure Loopback 1
The loopback 1 interface should be configured on each device, a suggested
configuration scheme is listed below. This loopback interface will be used as
the router id as well as for other uses.
Configuration for Loopback 1 needs to be configured on all spine and leaf
switches according to the table below.
5. Run following commands on all Nexus 9332PQ and Nexus 9372PX
switches.
R1-CS-9332-A-U42(config)# interface loopback1 <- Creates the
interface

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

35

Internal Network Configurations (Layer 3 mode)

R1-CS-9332-A-U42(config-if)# vrf member ucpmanagement <- assigns to


vrf ucpmanagement
R1-CS-9332-A-U42(config-if)# ip address 10.0.1.220/32 <- assigns the
ip address
R1-CS-9332-A-U42(config-if)# ip router ospf ucpmanagement area
0.0.0.0 <- advertise loopback 1 in the vrf
R1-CS-9332-A-U42(config-if)# no shut

Rack Location

Switch

Loopback 1 IP/
Router-id

VRF

OSPF Process

Compute Rack #1

9332A-U41

10.0.1.220/32

ucpmanagement ucpmanagement

Compute Rack #1

9332B-U40

10.0.1.221/32

ucpmanagement ucpmanagement

Compute Rack #1

9372A-U39

10.0.1.212/32

ucpmanagement ucpmanagement

Compute Rack #1

9372B-U38

10.0.1.213/32

ucpmanagement ucpmanagement

Compute Rack #2

9372C-U40

10.0.1.214/32

ucpmanagement ucpmanagement

Compute Rack #2

9372D-U39

10.0.1.215/32

ucpmanagement ucpmanagement

Compute Rack #3

9372E-U40

10.0.1.216/32

ucpmanagement ucpmanagement

Compute Rack #3

9372F-U39

10.0.1.217/32

ucpmanagement ucpmanagement

Compute Rack #4

9372G-U40

10.0.1.218/32

ucpmanagement ucpmanagement

Compute Rack #4

9372H-U39

10.0.1.219/32

ucpmanagement ucpmanagement

Subinterfaces
Subinterfaces are a division of a physical interface into multiple logical
interfaces. We use the subinterfaces for the vrf ucpmanagement so that the
entire interface does not need to be allocated for ucpmanagement. The
physical interface as well as remaining sub-interfaces can we utilized for
other purposes.
6. Configure the subinterfaces on each spine and leaf switch according to
the IP Scheme table below.
R1-CS-9372-A-U39(config)# interface Ethernet1/49.1 <- creates the
interface
R1-CS-9372-A-U39(config-subif)# mtu 9216
R1-CS-9372-A-U39(config-subif)# encapsulation dot1q 101 <- a unique
encapsulation for the subinterface
R1-CS-9372-A-U39(config-subif)# vrf member ucpmanagement
R1-CS-9372-A-U39(config-subif)# ip address 10.1.1.11/31 <- set the ip
via scheme below
R1-CS-9372-A-U39(config-subif)# ip ospf network point-to-point

36

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-A-U39(config-subif)# ip router ospf ucpmanagement area


0.0.0.0 <- advertise via area 0
R1-CS-9372-A-U39(config-subif)# no shutdown

Note: If configuring the mtu 9216 gives an error configure the mtu 9216 on
the parent interface first. Parent interface of Ethernet 1/49.1 would be Ethernet
1/49.
In the configuration example, we use encapsulation dot1q 101 as the tag
for the subinterface. The subinterface is then added to the vrf
ucpmanagement and ip address and ospf peering is configured. The
interface is now part of the vrf ucpmanagement and isolated from the global
routing table. In addition, OSPF peering has been enabled so all
neighboring routers will learn the routes.
Cisco Nexus9332-A IP Scheme
Link
#

Switch

Interface

IP

Encapsulation Speed

Neighbor

9332A-U41

Ethernet 1/1.1

10.1.1.10/31 dot1q 101

40G

9372A-U39

9332A-U41

Ethernet 1/2.1

10.1.1.20/31 dot1q 101

40G

9372A-U39

9332A-U41

Ethernet 1/3.1

10.1.2.10/31 dot1q 101

40G

9372B-U38

9332A-U41

Ethernet 1/4.1

10.1.2.20/31 dot1q 101

40G

9372B-U38

9332A-U41

Ethernet 1/5.1

10.1.3.10/31 dot1q 101

40G

9372C-U40

9332A-U41

Ethernet 1/6.1

10.1.3.20/31 dot1q 101

40G

9372C-U40

9332A-U41

Ethernet 1/7.1

10.1.4.10/31 dot1q 101

40G

9372D-U39

9332A-U41

Ethernet 1/8.1

10.1.4.20/31 dot1q 101

40G

9372D-U39

9332A-U41

Ethernet 1/9.1

10.1.5.10/31 dot1q 101

40G

9372E-U40

10

9332A-U41

Ethernet 1/10.1

10.1.5.20/31 dot1q 101

40G

9372E-U40

11

9332A-U41

Ethernet 1/11.1

10.1.6.10/31 dot1q 101

40G

9372F-U39

12

9332A-U41

Ethernet 1/12.1

10.1.6.20/31 dot1q 101

40G

9372F-U39

13

9332A-U41

Ethernet 1/13.1

10.1.7.10/31 dot1q 101

40G

9372G-U40

14

9332A-U41

Ethernet 1/14.1

10.1.7.20/31 dot1q 101

40G

9372G-U40

15

9332A-U41

Ethernet 1/15.1

10.1.8.10/31 dot1q 101

40G

9372H-U39

16

9332A-U41

Ethernet 1/16.1

10.1.8.20/31 dot1q 101

40G

9372H-U39

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

37

Internal Network Configurations (Layer 3 mode)

Cisco Nexus9332-B IP Scheme


Link
#

Switch

Interface

IP

Encapsulation Speed

Neighbor

9332B-U40

Ethernet 1/1.1

10.2.1.10/31 dot1q 101

40G

9372A-U39

9332B-U40

Ethernet 1/2.1

10.2.1.20/31 dot1q 101

40G

9372A-U39

9332B-U40

Ethernet 1/3.1

10.2.2.10/31 dot1q 101

40G

9372B-U38

9332B-U40

Ethernet 1/4.1

10.2.2.20/31 dot1q 101

40G

9372B-U38

9332B-U40

Ethernet 1/5.1

10.2.3.10/31 dot1q 101

40G

9372C-U40

9332B-U40

Ethernet 1/6.1

10.2.3.20/31 dot1q 101

40G

9372C-U40

9332B-U40

Ethernet 1/7.1

10.2.4.10/31 dot1q 101

40G

9372D-U39

9332B-U40

Ethernet 1/8.1

10.2.4.20/31 dot1q 101

40G

9372D-U39

9332B-U40

Ethernet 1/9.1

10.2.5.10/31 dot1q 101

40G

9372E-U40

10

9332B-U40

Ethernet 1/10.1

10.2.5.20/31 dot1q 101

40G

9372E-U40

11

9332B-U40

Ethernet 1/11.1

10.2.6.10/31 dot1q 101

40G

9372F-U39

12

9332B-U40

Ethernet 1/12.1

10.2.6.20/31 dot1q 101

40G

9372F-U39

13

9332B-U40

Ethernet 1/13.1

10.2.7.10/31 dot1q 101

40G

9372G-U40

14

9332B-U40

Ethernet 1/14.1

10.2.7.20/31 dot1q 101

40G

9372G-U40

15

9332B-U40

Ethernet 1/15.1

10.2.8.10/31 dot1q 101

40G

9372H-U39

16

9332B-U40

Ethernet 1/16.1

10.2.8.20/31 dot1q 101

40G

9372H-U39

Cisco Nexus9372-A IP Scheme


Link
#

Switch

Interface

49

9372A-U39

Ethernet 1/49.1

10.1.1.11/31 dot1q 101

40G

9332A-U41

50

9372A-U39

Ethernet 1/50.1

10.1.1.21/31 dot1q 101

40G

9332A-U41

51

9372A-U39

Ethernet 1/51.1

10.2.1.11/31 dot1q 101

40G

9332B-U40

52

9372A-U39

Ethernet 1/52.1

10.2.1.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Neighbor

Cisco Nexus9372-B IP Scheme


Link
#

38

Switch

Interface

49

9372B-U38

Ethernet 1/49.1

10.1.2.11/31 dot1q 101

40G

9332A-U41

50

9372B-U38

Ethernet 1/50.1

10.1.2.21/31 dot1q 101

40G

9332A-U41

51

9372B-U38

Ethernet 1/51.1

10.2.2.11/31 dot1q 101

40G

9332B-U40

52

9372B-U38

Ethernet 1/52.1

10.2.2.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Neighbor

Internal Network Configurations (Layer 3 mode)

Cisco Nexus9372-C IP Scheme


Link
#

Switch

Interface

49

9372C-U40

Ethernet 1/49.1

10.1.3.11/31 dot1q 101

40G

9332A-U41

50

9372C-U40

Ethernet 1/50.1

10.1.3.21/31 dot1q 101

40G

9332A-U41

51

9372C-U40

Ethernet 1/51.1

10.2.3.11/31 dot1q 101

40G

9332B-U40

52

9372C-U40

Ethernet 1/52.1

10.2.3.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Neighbor

Cisco Nexus9372-D IP Scheme


Link
#

Switch

Interface

49

9372D-U39

Ethernet 1/49.1

10.1.4.11/31 dot1q 101

40G

9332A-U41

50

9372D-U39

Ethernet 1/50.1

10.1.4.21/31 dot1q 101

40G

9332A-U41

51

9372D-U39

Ethernet 1/51.1

10.2.4.11/31 dot1q 101

40G

9332B-U40

52

9372D-U39

Ethernet 1/52.1

10.2.4.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Neighbor

Cisco Nexus9372-E IP Scheme


Link
#

Switch

Interface

49

9372E-U40

Ethernet 1/49.1

10.1.5.11/31 dot1q 101

40G

9332A-U41

50

9372E-U40

Ethernet 1/50.1

10.1.5.21/31 dot1q 101

40G

9332A-U41

51

9372E-U40

Ethernet 1/51.1

10.2.5.11/31 dot1q 101

40G

9332B-U40

52

9372E-U40

Ethernet 1/52.1

10.2.5.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Neighbor

Cisco Nexus9372-F IP Scheme


Link
#

Switch

Interface

49

9372F-U39

Ethernet 1/49.1

10.1.6.11/31 dot1q 101

40G

9332A-U41

50

9372F-U39

Ethernet 1/50.1

10.1.6.21/31 dot1q 101

40G

9332A-U41

51

9372F-U39

Ethernet 1/51.1

10.2.6.11/31 dot1q 101

40G

9332B-U40

52

9372F-U39

Ethernet 1/52.1

10.2.6.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Neighbor

39

Internal Network Configurations (Layer 3 mode)

Cisco Nexus9372-G IP Scheme


Link
#

Switch

Interface

49

9372G-U40

Ethernet 1/49.1

10.1.7.11/31 dot1q 101

40G

9332A-U41

50

9372G-U40

Ethernet 1/50.1

10.1.7.21/31 dot1q 101

40G

9332A-U41

51

9372G-U40

Ethernet 1/51.1

10.2.7.11/31 dot1q 101

40G

9332B-U40

52

9372G-U40

Ethernet 1/52.1

10.2.7.21/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Neighbor

Cisco Nexus9372-H IP Scheme


Link
#

Switch

Interface

49

9372H-U39

Ethernet 1/49.1

10.1.8.11/31 dot1q 101

40G

9332A-U41

50

9372H-U39

Ethernet 1/50.1

10.1.8.21/31 dot1q 101

40G

9332A-U41

51

9372H-U39

Ethernet 1/51.1

10.2.8.10/31 dot1q 101

40G

9332B-U40

52

9372H-U39

Ethernet 1/52.1

10.2.8.20/31 dot1q 101

40G

9332B-U40

IP

Encapsulation Speed

Neighbor

Verify OSPF neighborships


7. Verify that the L3 ospf neighborships are in FULL state on all L3
interfaces that are connected to neighboring switches.
R1-CS-9332-A-U42# sh ip ospf neighbors vrf ucpmanagement
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 8
Neighbor ID Pri State Up Time Address Interface
10.0.1.212 1 FULL/ - 1d05h 10.1.1.11 Eth1/1.1
10.0.1.212 1 FULL/ - 1d05h 10.1.1.21 Eth1/2.1
10.0.1.213 1 FULL/ - 2d04h 10.1.2.11 Eth1/3.1
10.0.1.213 1 FULL/ - 2d04h 10.1.2.21 Eth1/4.1
10.0.1.214 1 FULL/ - 2d04h 10.1.3.11 Eth1/5.1
10.0.1.214 1 FULL/ - 2d04h 10.1.3.21 Eth1/6.1
10.0.1.215 1 FULL/ - 2d04h 10.1.4.11 Eth1/7.1
10.0.1.215 1 FULL/ - 2d04h 10.1.4.21 Eth1/8.1
10.0.1.216 1 FULL/ - 1d05h 10.1.5.11 Eth1/9.1
10.0.1.216 1 FULL/ - 1d05h 10.1.5.21 Eth1/10.1
10.0.1.217 1 FULL/ - 2d04h 10.1.6.11 Eth1/11.1
10.0.1.217 1 FULL/ - 2d04h 10.1.6.21 Eth1/12.1
10.0.1.218 1 FULL/ - 2d04h 10.1.7.11 Eth1/13.1
10.0.1.218 1 FULL/ - 2d04h 10.1.7.21 Eth1/14.1
10.0.1.219 1 FULL/ - 2d04h 10.1.8.11 Eth1/15.1
10.0.1.219 1 FULL/ - 2d04h 10.1.8.21 Eth1/16.1

40

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Configure OSPF on the UCP Management SVI


8. Configure ospf on ucp management svi of all Nexus9372PX leaf switches
(A, B, C, D, E, F, G, H).
R1-CS-9372-A-U39(config)# interface vlan 100 <- Use "UCP Management
VLAN ID"
R1-CS-9372-A-U39(config-if)# ip router ospf ucpmanagement area
0.0.0.0 <- Advertise the SVI

Verify UCP Management SVI


Basic UCP Management SVI configurations will be configured after running
UCP Deployment Tool (except OSPF).
9. Verify UCP Management SVI by running below commands on all
Nexus9372PX switch.
R1-CS-9372-A-U40# show run interface vlan 100 <- Use "UCP Management
VLAN ID"
!Command: show running-config interface Vlan100
!Time: Tue Oct 13 04:34:46 2015
version 7.0(3)I1(2)
interface Vlan100 <- Must be same "UCP Management VLAN ID" across all
racks.
no shutdown
mtu 9216
vrf member ucpmanagement <- vrf "ucpmanagment" by UCP default
ip address 172.21.100.2/24 <- SVI IP Address. See below table.
ip router ospf ucpmanagement area 0.0.0.0
hsrp version 2
hsrp 100
preempt
ip 172.17.131.1 <- Gateway IP Address. See below table.
ip dhcp relay address 172.21.100.242 use-vrf management <- UCP
Utility VM (DHCP Server) IP Address

Cisco Nexus9372 Management SVI IP Scheme


Rack Location

Switch

Compute Rack #1 9372A-U39

SVI IP
172.21.100.2/24

HSRP Gateway Mgmt Vlan ID


172.21.100.1

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

100

41

Internal Network Configurations (Layer 3 mode)

Rack Location

Switch

SVI IP

HSRP Gateway Mgmt Vlan ID

Compute Rack #1 9372B-U38

172.21.100.3/24

172.21.100.1

100

Compute Rack #2 9372C-U40

172.22.100.2/24

172.22.100.1

100

Compute Rack #2 9372D-U39

172.22.100.3/24

172.22.100.1

100

Compute Rack #3 9372E-U40

172.23.100.2/24

172.23.100.1

100

Compute Rack #3 9372F-U39

172.23.100.2/24

172.23.100.1

100

Compute Rack #4 9372G-U40

172.24.100.2/24

172.24.100.1

100

Compute Rack #4 9372H-U39

172.24.100.3/24

172.24.100.1

100

Configure VM Traffic SVI


10. Configure SVI for VM Traffic as needed, by running below commands on
all Nexus9372PX switch.

Configure SVI on the 9372 A, C, E, G Leaf switch.


R1-CS-9372-C-U40(config)# int vlan 103
R1-CS-9372-C-U40(config-if)# description VM Data subnet on rack 2
R1-CS-9372-C-U40(config-if)# mtu 9216
R1-CS-9372-C-U40(config-if)# no shut
R1-CS-9372-C-U40(config-if)# ip address 172.17.103.2/24 <- SVI IP
Address
R1-CS-9372-C-U40(config-if)# hsrp version 2
R1-CS-9372-C-U40(config-if)# hsrp 103
R1-CS-9372-C-U40(config-if-hsrp)# preempt
R1-CS-9372-C-U40(config-if-hsrp)# priority 100 <- Must be higher
priority (smaller value) than B, D, F, H switch to match vPC primary
R1-CS-9372-C-U40(config-if-hsrp)# ip 172.17.103.1 <- Gateway IP
address

Configure SVI on the 9372 B, D, F, H Leaf switch.


R1-CS-9372-D-U39(config)# int vlan 103 <- Must use same UCP
Management VLAN ID for all racks.
R1-CS-9372-D-U39(config-if)# description VM Data subnet on rack 2
R1-CS-9372-D-U39(config-if)# mtu 9216
R1-CS-9372-D-U39(config-if)# no shut
R1-CS-9372-D-U39(config-if)# ip address 172.17.103.3/24<- SVI IP
Address
R1-CS-9372-D-U39(config-if)# hsrp version 2
R1-CS-9372-D-U39(config-if)# hsrp 103
R1-CS-9372-D-D39(config-if-hsrp)# preempt

42

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-D-U39(config-if-hsrp)# priority 110 <- Must be lower


priority (higher value) than A, C, E, G switch to match vPC seconday
R1-CS-9372-D-U39(config-if-hsrp)# ip 172.17.103.1 <- Gateway IP
address

Note: The priority value is different on 9372 A, C, E, G and B, D, F, H. Cisco


recommends to match HSRP active/standby and vPC primary/seconday.

Configure Global Unicast Routing


In this section, we go over the Nexus 9300 series switches and
configurations to bring up OSPF or BGP routing protocols. This section
explains how to enable and configure OSPF and eBGP routing between the
Nexus 9332PQ spine switch and Nexus 9372PX leaf switch.

Note: Please decide now whether OSPF or eBGP will be the routing protocol of
choice and skip to related section.

OSPF
OSPF is a layer 3 interior gateway protocol (IGP). The feature must be
enabled and a process must be created. Finally, OSPF must be enabled on
IPv4/v6 individual interfaces which need to be advertised to neighbors.
Feature OSPF
1. To enable OSPF on the switch we first need to enable the feature ospf.
This needs to be done on all Nexus 9332 spine and Nexus 9372 leaf
switches.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature ospf

Router OSPF Process


We need to create the router ospf process next. We have named it 100 in
this example and specify the loopback 0 primary ip address as the router-id.
The router-id identifies the OSPF instance. This needs to be done on all
spine and leaf switches.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

43

Internal Network Configurations (Layer 3 mode)

2. Configure the OSPF Process for all Nexus9332 and Nexus9372 switches,
use the table below to complete.
R1-CS-9332-A-U42(config)# router ospf 100
R1-CS-9332-A-U42(config-router)# router-id 10.0.0.220

Rack Location

Switch

OSPF ID

Router-id

Compute Rack #1 9332A-U41

100

10.0.0.220

Compute Rack #1 9332B-U40

100

10.0.0.221

Compute Rack #1 9372A-U39

100

10.0.0.212

Compute Rack #1 9372B-U38

100

10.0.0.213

Compute Rack #2 9372C-U40

100

10.0.0.214

Compute Rack #2 9372D-U39

100

10.0.0.215

Compute Rack #3 9372E-U40

100

10.0.0.216

Compute Rack #3 9372F-U39

100

10.0.0.217

Compute Rack #4 9372G-U40

100

10.0.0.218

Compute Rack #4 9372H-U39

100

10.0.0.219

Configure OSPF on VM Traffic SVI


3. If you need VM Traffic SVI to be advertised, please run the following
commands on a switch by switch and svi by svi basis.
R1-CS-9372-A-U39(config-if)#interface Vlan101
R1-CS-9372-A-U39(config-if)# ip router ospf 100 area 0.0.0.0

eBGP
Note: If OSPF was chosen as the routing protocol of choice, please skip this
section. Otherwise, please continue.
BGP is becoming commonly used in Spine Leaf topologies because of its
scalability and flexibility.
Feature BGP
4. To enable BGP on the switch we first need to enable the feature BGP.
This needs to be done on all Nexus 9332 and Nexus 9372 switches.

44

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature bgp

Router BGP Process


We need to create the router bgp process next. The Spine (Nexus 9332) is
configured in its own Autonomous System (AS) and each of the Leafs
(Nexus 9372) are configured in separate ASs thus external BGP is used.
The router-id identifies the BGP instance.

5. Configure the BGP Process for all Nexus9332 and Nexus9372 switches,
use the table below to complete.
R1-CS-9332-A-U42(config)# router bgp 65001
R1-CS-9332-A-U42(config-router)# router-id 10.0.0.220 <- assign the
router id

Rack Location

Switch

Router AS #

Router-id

Compute Rack #1

9332A-U41

65001

10.0.0.220

Compute Rack #1

9332B-U40

65001

10.0.0.221

Compute Rack #1

9372A-U39

65101

10.0.0.212

Compute Rack #1

9372B-U38

65101

10.0.0.213

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

45

Internal Network Configurations (Layer 3 mode)

Rack Location

Switch

Router AS #

Router-id

Compute Rack #2

9372C-U40

65102

10.0.0.214

Compute Rack #2

9372D-U39

65102

10.0.0.215

Compute Rack #3

9372E-U40

65103

10.0.0.216

Compute Rack #3

9372F-U39

65103

10.0.0.217

Compute Rack #4

9372G-U40

65104

10.0.0.218

Compute Rack #4

9372H-U39

65104

10.0.0.219

Configure BGP Routing


6. On the Nexus 9332A and Nexus 9332B spine switches, we create the
BGP process and configure the advertised networks as well as
neighborships.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#router bgp 65001 <- AS number will be same
for both Spine switches
R1-CS-9332-A-U42(config-router)# router-id 10.0.0.220 <- primary
loopback 0 ip address
R1-CS-9332-A-U42(config-router)# graceful-restart-helper
R1-CS-9332-A-U42(config-router)# log-neighbor-changes
R1-CS-9332-A-U42(config-router-af)# address-family ipv4 unicast
R1-CS-9332-A-U42(config-router-af)# network 10.0.0.220/32 <- add
primary loopback 0 ip address
R1-CS-9332-A-U42(config-router-af)# network 10.254.254.254/32 <advertise anycast rp address if VXLAN will be used
R1-CS-9332-A-U42(config-router-af)# network 10.1.1.10/31
<- advertise interface ip as needed
...
R1-CS-9332-A-U42(config-router-af)# maximum-paths 32
R1-CS-9332-A-U42(config-router-neighbor)# template peer BGPLEAF <template to save time on neighborships configuration
R1-CS-9332-A-U42(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9332-A-U42(config-router-neighbor)# default-originate
R1-CS-9332-A-U42(config-router-neighbor)# next-hop-self
R1-CS-9332-A-U42(config-router-neighbor)# soft-reconfiguration
inbound
R1-CS-9332-A-U42(config-router-neighbor)# neighbor 10.1.1.11 remoteas 65101 <- create neighborships on all L3 links between spine and
leaf layer. Follow the table below.
R1-CS-9332-A-U42(config-router-neighbor)# inherit peer BGPLEAF
R1-CS-9332-A-U42(config-router-neighbor)# neighbor 10.1.1.21 remoteas 65101
R1-CS-9332-A-U42(config-router-neighbor)# inherit peer BGPLEAF
...

46

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Nexus 9332 A BGP Routing


Switch

Networks

Neighbor

Remote-AS

Neighbor

9332A-U41

10.1.1.10/31

10.1.1.11

65101

9372A-U39

9332A-U41

10.1.1.20/31

10.1.1.21

65101

9372A-U39

9332A-U41

10.1.2.10/31

10.1.2.11

65101

9372B-U38

9332A-U41

10.1.2.20/31

10.1.2.21

65101

9372B-U38

9332A-U41

10.1.3.10/31

10.1.3.11

65102

9372C-U40

9332A-U41

10.1.3.20/31

10.1.3.21

65102

9372C-U40

9332A-U41

10.1.4.10/31

10.1.4.11

65102

9372D-U39

9332A-U41

10.1.4.20/31

10.1.4.21

65102

9372D-U39

9332A-U41

10.1.5.10/31

10.1.5.11

65103

9372E-U40

9332A-U41

10.1.5.20/31

10.1.5.21

65103

9372E-U40

9332A-U41

10.1.6.10/31

10.1.6.11

65103

9372F-U39

9332A-U41

10.1.6.20/31

10.1.6.21

65103

9372F-U39

9332A-U41

10.1.7.10/31

10.1.7.11

65104

9372G-U40

9332A-U41

10.1.7.20/31

10.1.7.21

65104

9372G-U40

9332A-U41

10.1.8.10/31

10.1.8.11

65104

9372H-U39

9332A-U41

10.1.8.20/31

10.1.8.21

65104

9372H-U39

9332A-U41

10.0.10.220/31

10.0.10.221

65001

9332B-U40

9332A-U41

10.254.254.254/32

N/A

N/A

N/A

9332A-U41

10.0.0.220/32

N/A

N/A

N/A

Nexus 9332 B BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9332B-U40

10.2.1.10/31

10.2.1.11

65101

9372A-U39

9332B-U40

10.2.1.20/31

10.2.1.21

65101

9372A-U39

9332B-U40

10.2.2.10/31

10.2.2.11

65101

9372B-U38

9332B-U40

10.2.2.20/31

10.2.2.21

65101

9372B-U38

9332B-U40

10.2.3.10/31

10.2.3.11

65102

9372C-U40

9332B-U40

10.2.3.20/31

10.2.3.21

65102

9372C-U40

9332B-U40

10.2.4.10/31

10.2.4.11

65102

9372D-U39

9332B-U40

10.2.4.20/31

10.2.4.21

65102

9372D-U39

9332B-U40

10.2.5.10/31

10.2.5.11

65103

9372E-U40

9332B-U40

10.2.5.20/31

10.2.5.21

65103

9372E-U40

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

47

Internal Network Configurations (Layer 3 mode)

Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9332B-U40

10.2.6.10/31

10.2.6.11

65103

9372F-U39

9332B-U40

10.2.6.20/31

10.2.6.21

65103

9372F-U39

9332B-U40

10.2.7.10/31

10.2.7.11

65104

9372G-U40

9332B-U40

10.2.7.20/31

10.2.7.21

65104

9372G-U40

9332B-U40

10.2.8.10/31

10.2.8.10

65104

9372H-U39

9332B-U40

10.2.8.20/31

10.2.8.20

65104

9372H-U39

9332B-U40

10.0.10.220/31

10.0.10.220

65001

9332A-U41

9332B-U40

10.254.254.254/32

N/A

N/A

N/A

9332B-U40

10.0.0.221/32

N/A

N/A

N/A

7. Configure BGP process and configure the advertised networks as well as


neighborships on all Nexus 9372 A,B,C,D,E,F,G,H leaf switches.
R1-CS-9372-A-U39# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U39(config)# router bgp 65101 <- AS number will be
different for each rack.
R1-CS-9372-A-U39(config-router)# router-id 10.0.0.212 <- primary
loopback 0 ip address
R1-CS-9372-A-U39(config-router)# address-family ipv4 unicast
R1-CS-9372-A-U39(config-router-af)# network 10.0.0.212/32 <- add
primary loopback 0 ip address
R1-CS-9372-A-U39(config-router-af)# network 10.0.10.212/31 <- add
network statements for L3 interfaces created beforehand
R1-CS-9372-A-U39(config-router-af)# network 10.1.1.10/31
...
R1-CS-9372-A-U39(config-router-af)# maximum-paths 32
R1-CS-9372-A-U39(config-router-af)# template peer BGPSPINE <template to save time on neighborships configuration
R1-CS-9372-A-U39(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9372-A-U39(config-router-neighbor)# next-hop-self
R1-CS-9372-A-U39(config-router-neighbor)# soft-reconfiguration
inbound
R1-CS-9372-A-U39(config-router-neighbor)# neighbor 10.0.10.213
remote-as 65101 <- ibgp neighborship between vpc peers on leaf
R1-CS-9372-A-U39(config-router-neighbor)# update-source Vlan10
R1-CS-9372-A-U39(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9372-A-U39(config-router-neighbor)# next-hop-self
R1-CS-9372-A-U39(config-router-neighbor)# soft-reconfiguration
inbound
R1-CS-9372-A-U39(config-router-neighbor)# neighbor 10.1.1.10 remoteas 65001 <- create neighborships on all L3 links between spine and
leaf layer

48

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-A-U39(config-router-neighbor)#
R1-CS-9372-A-U39(config-router-neighbor)#
as 65001
R1-CS-9372-A-U39(config-router-neighbor)#
R1-CS-9372-A-U39(config-router-neighbor)#
as 65001
R1-CS-9372-A-U39(config-router-neighbor)#
R1-CS-9372-A-U39(config-router-neighbor)#
as 65001
R1-CS-9372-A-U39(config-router-neighbor)#

inherit peer BGPSPINE


neighbor 10.1.1.20 remoteinherit peer BGPSPINE
neighbor 10.2.1.10 remoteinherit peer BGPSPINE
neighbor 10.2.1.20 remoteinherit peer BGPSPINE

Nexus 9372 A BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372A-U39

10.1.1.10/31

10.1.1.10

65001

9332A-U41

9372A-U39

10.1.1.20/31

10.1.1.20

65001

9332A-U41

9372A-U39

10.2.1.10/31

10.2.1.10

65001

9332B-U40

9372A-U39

10.2.1.20/31

10.2.1.20

65001

9332B-U40

9372A-U39

10.0.10.212/31

10.0.10.213

65001

9372B-U38

9372A-U39

10.0.10.212/32

N/A

N/A

N/A

Nexus 9372 B BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372B-U38

10.1.2.10/31

10.1.2.10

65001

9332A-U41

9372B-U38

10.1.2.20/31

10.1.2.20

65001

9332A-U41

9372B-U38

10.2.2.10/31

10.2.2.10

65001

9332B-U40

9372B-U38

10.2.2.20/31

10.2.2.20

65001

9332B-U40

9372B-U38

10.0.10.212/31

10.0.10.212

65001

9372A-U39

9372B-U38

10.0.10.213/32

N/A

N/A

N/A

Nexus 9372 C BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372C-U40

10.1.3.10/31

10.1.3.10

65001

9332A-U41

9372C-U40

10.1.3.20/31

10.1.3.20

65001

9332A-U41

9372C-U40

10.2.3.10/31

10.2.3.10

65001

9332B-U40

9372C-U40

10.2.3.20/31

10.2.3.20

65001

9332B-U40

9372C-U40

10.0.10.214/31

10.0.10.213

65001

9372B-U38

9372C-U40

10.0.10.214/32

N/A

N/A

N/A

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

49

Internal Network Configurations (Layer 3 mode)

Nexus 9372 D BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372D-U39

10.1.4.10/31

10.1.4.10

65001

9332A-U41

9372D-U39

10.1.4.20/31

10.1.4.20

65001

9332A-U41

9372D-U39

10.2.4.10/31

10.2.4.10

65001

9332B-U40

9372D-U39

10.2.4.20/31

10.2.4.20

65001

9332B-U40

9372D-U39

10.0.10.214/31

10.0.10.214

65001

9332A-U41

9372D-U39

10.0.10.215/32

N/A

N/A

N/A

Nexus 9372 E BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372E-U40

10.1.5.10/31

10.1.5.10

65001

9332A-U41

9372E-U40

10.1.5.20/31

10.1.5.20

65001

9332A-U41

9372E-U40

10.2.5.10/31

10.2.5.10

65001

9332B-U40

9372E-U40

10.2.5.20/31

10.2.5.20

65001

9332B-U40

9372E-U40

10.0.10.216/31

10.0.10.216

65001

9372F-U39

9372E-U40

10.0.10.216/32

N/A

N/A

N/A

Nexus 9372 F BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372F-U39

10.1.6.10/31

10.1.6.10

65001

9332A-U41

9372F-U39

10.1.6.20/31

10.1.6.20

65001

9332A-U41

9372F-U39

10.2.6.10/31

10.2.6.10

65001

9332B-U40

9372F-U39

10.2.6.20/31

10.2.6.20

65001

9332B-U40

9372F-U39

10.0.10.216/31

10.0.10.215

65001

9372E-U40

9372F-U39

10.0.10.217/32

N/A

N/A

N/A

Nexus 9372 G BGP Routing


Switch

50

Network Statement

Neighbor

Remote-AS

Neighbor

9372G-U40

10.1.7.10/31

10.1.7.10

65001

9332A-U41

9372G-U40

10.1.7.20/31

10.1.7.20

65001

9332A-U41

9372G-U40

10.2.7.10/31

10.2.7.10

65001

9332B-U40

9372G-U40

10.2.7.20/31

10.2.7.20

65001

9332B-U40

9372G-U40

10.0.10.218/31

10.0.10.219

65001

9372H-U39

9372G-U40

10.0.10.218/32

N/A

N/A

N/A

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Nexus 9372 H BGP Routing


Switch

Network Statement

Neighbor

Remote-AS

Neighbor

9372H-U39

10.1.8.10/31

10.1.8.10

65001

9332A-U41

9372H-U39

10.1.8.20/31

10.1.8.20

65001

9332A-U41

9372H-U39

10.2.8.10/31

10.2.8.10

65001

9332B-U40

9372H-U39

10.2.8.20/31

10.2.8.20

65001

9332B-U40

9372H-U39

10.0.10.218/31

10.0.10.218

65001

9372G-U40

9372H-U39

10.0.10.219/32

N/A

N/A

N/A

Configure the Layer 3 interfaces


Note: Please run this section for both OSPF and BGP.

8. Add Layer 3 configurations to the interfaces on all 9332 and 9372


switches according to tables below.
R1-CS-9332-A-U42(config)# interface Ethernet1/1
R1-CS-9332-A-U42(config-if)# speed 40000
R1-CS-9332-A-U42(config-if)# mtu 9216
R1-CS-9332-A-U42(config-if)# ip address 10.1.1.10/31 <- Refer IP
address of each interface in below tables
R1-CS-9332-A-U42(config-if)# ip ospf network point-to-point <-- Add
this if OSPF is used
R1-CS-9332-A-U42(config-if)# ip router ospf 100 area 0.0.0.0 <-- Add
this if OSPF is used
R1-CS-9332-A-U42(config-if)# no shutdown

Cisco Nexus9332-A IP Scheme


Link #

Switch

IP

Interface

Neighbor

9332A-U41

Ethernet 1/1

10.1.1.10/31 9372A-U39

9332A-U41

Ethernet 1/2

10.1.1.20/31 9372A-U39

9332A-U41

Ethernet 1/3

10.1.2.10/31 9372B-U38

9332A-U41

Ethernet 1/4

10.1.2.20/31 9372B-U38

9332A-U41

Ethernet 1/5

10.1.3.10/31 9372C-U40

9332A-U41

Ethernet 1/6

10.1.3.20/31 9372C-U40

9332A-U41

Ethernet 1/7

10.1.4.10/31 9372D-U39

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

51

Internal Network Configurations (Layer 3 mode)

Link #

Switch

IP

Interface

Neighbor

9332A-U41

Ethernet 1/8

10.1.4.20/31 9372D-U39

9332A-U41

Ethernet 1/9

10.1.5.10/31 9372E-U40

10

9332A-U41

Ethernet 1/10

10.1.5.20/31 9372E-U40

11

9332A-U41

Ethernet 1/11

10.1.6.10/31 9372F-U39

12

9332A-U41

Ethernet 1/12

10.1.6.20/31 9372F-U39

13

9332A-U41

Ethernet 1/13

10.1.7.10/31 9372G-U40

14

9332A-U41

Ethernet 1/14

10.1.7.20/31 9372G-U40

15

9332A-U41

Ethernet 1/15

10.1.8.10/31 9372H-U39

16

9332A-U41

Ethernet 1/16

10.1.8.20/31 9372H-U39

Cisco Nexus9332-B IP Scheme


Link #

Switch

IP

Interface

Neighbor

9332B-U40

Ethernet 1/1

10.2.1.10/31 9372A-U39

9332B-U40

Ethernet 1/2

10.2.1.20/31 9372A-U39

9332B-U40

Ethernet 1/3

10.2.2.10/31 9372B-U38

9332B-U40

Ethernet 1/4

10.2.2.20/31 9372B-U38

9332B-U40

Ethernet 1/5

10.2.3.10/31 9372C-U40

9332B-U40

Ethernet 1/6

10.2.3.20/31 9372C-U40

9332B-U40

Ethernet 1/7

10.2.4.10/31 9372D-U39

9332B-U40

Ethernet 1/8

10.2.4.20/31 9372D-U39

9332B-U40

Ethernet 1/9

10.2.5.10/31 9372E-U40

10

9332B-U40

Ethernet 1/10

10.2.5.20/31 9372E-U40

11

9332B-U40

Ethernet 1/11

10.2.6.10/31 9372F-U39

12

9332B-U40

Ethernet 1/12

10.2.6.20/31 9372F-U39

13

9332B-U40

Ethernet 1/13

10.2.7.10/31 9372G-U40

14

9332B-U40

Ethernet 1/14

10.2.7.20/31 9372G-U40

15

9332B-U40

Ethernet 1/15

10.2.8.10/31 9372H-U39

16

9332B-U40

Ethernet 1/16

10.2.8.20/31 9372H-U39

Cisco Nexus9372-A IP Scheme


Link #
49

52

Switch
9372A-U39

IP

Interface
Ethernet 1/49

Neighbor

10.1.1.11/31 9332A-U41

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Link #

Switch

IP

Interface

Neighbor

50

9372A-U39

Ethernet 1/50

10.1.1.21/31 9332A-U41

51

9372A-U39

Ethernet 1/51

10.2.1.11/31 9332B-U40

52

9372A-U39

Ethernet 1/52

10.2.1.21/31 9332B-U40

Cisco Nexus9372-B IP Scheme


Link #

Switch

IP

Interface

Neighbor

49

9372B-U38

Ethernet 1/49

10.1.2.11/31 9332A-U41

50

9372B-U38

Ethernet 1/50

10.1.2.21/31 9332A-U41

51

9372B-U38

Ethernet 1/51

10.2.2.11/31 9332B-U40

52

9372B-U38

Ethernet 1/52

10.2.2.21/31 9332B-U40

Cisco Nexus9372-C IP Scheme


Link #

Switch

IP

Interface

Neighbor

49

9372C-U40

Ethernet 1/49

10.1.3.11/31 9332A-U41

50

9372C-U40

Ethernet 1/50

10.1.3.21/31 9332A-U41

51

9372C-U40

Ethernet 1/51

10.2.3.11/31 9332B-U40

52

9372C-U40

Ethernet 1/52

10.2.3.21/31 9332B-U40

Cisco Nexus9372-D IP Scheme


Link #

Switch

Interface

IP

Neighbor

49

9372D-U39

Ethernet 1/49

10.1.4.11/31 9332A-U41

50

9372D-U39

Ethernet 1/50

10.1.4.21/31 9332A-U41

51

9372D-U39

Ethernet 1/51

10.2.4.11/31 9332B-U40

52

9372D-U39

Ethernet 1/52

10.2.4.21/31 9332B-U40

Cisco Nexus9372-E IP Scheme


Link #

Switch

Interface

IP

Neighbor

49

9372E-U40

Ethernet 1/49

10.1.5.11/31 9332A-U41

50

9372E-U40

Ethernet 1/50

10.1.5.21/31 9332A-U41

51

9372E-U40

Ethernet 1/51

10.2.5.11/31 9332B-U40

52

9372E-U40

Ethernet 1/52

10.2.5.21/31 9332B-U40

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

53

Internal Network Configurations (Layer 3 mode)

Cisco Nexus9372-F IP Scheme


IP

Neighbor

Link # Switch

Interface

49

9372F-U39

Ethernet 1/49

10.1.6.11/31 9332A-U41

50

9372F-U39

Ethernet 1/50

10.1.6.21/31 9332A-U41

51

9372F-U39

Ethernet 1/51

10.2.6.11/31 9332B-U40

52

9372F-U39

Ethernet 1/52

10.2.6.21/31 9332B-U40

Cisco Nexus9372-G IP Scheme


Link #

Switch

IP

Interface

Neighbor

49

9372G-U40

Ethernet 1/49

10.1.7.11/31 9332A-U41

50

9372G-U40

Ethernet 1/50

10.1.7.21/31 9332A-U41

51

9372G-U40

Ethernet 1/51

10.2.7.11/31 9332B-U40

52

9372G-U40

Ethernet 1/52

10.2.7.21/31 9332B-U40

Cisco Nexus9372-H IP Scheme


Link #

Switch

IP

Interface

Neighbor

49

9372H-U39

Ethernet 1/49

10.1.8.11/31 9332A-U41

50

9372H-U39

Ethernet 1/50

10.1.8.21/31 9332A-U41

51

9372H-U39

Ethernet 1/51

10.2.8.11/31 9332B-U40

52

9372H-U39

Ethernet 1/52

10.2.8.21/31 9332B-U40

Configure L3 Peer Adjacency


9. Add vlan, ip and ospf on the interface vlan for the L3 adjacency between
vPC peers according to below table on 9372A,B,C,D,E,F,G,H switches .
R1-CS-9372-A-U39(config)# Vlan 10 <- VLAN for L3 peer adjacency
R1-CS-9372-A-U39(config-vlan)# interface Vlan10 <- ospf/iBgp
interface between vpc peers
R1-CS-9372-A-U39(config-if)# no shutdown
R1-CS-9372-A-U39(config-if)# ip address 10.0.10.212/31 <- L3 Peer
Adjacency SVI IP Address
R1-CS-9372-A-U39(config-if)# mtu 9216
R1-CS-9372-A-U39(config-if)# ip router ospf 100 area 0.0.0.0 <-- Add
this if OSPF is used

54

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

L3 adjacency between vPC peers


Rack Location

Switch

Int Vlan 10

Compute Rack #1

9372A-U39

10.0.10.212/31

Compute Rack #1

9372B-U38

10.0.10.213/31

Compute Rack #2

9372C-U40

10.0.10.214/31

Compute Rack #2

9372D-U39

10.0.10.215/31

Compute Rack #3

9372E-U40

10.0.10.216/31

Compute Rack #3

9372F-U39

10.0.10.217/31

Compute Rack #4

9372G-U40

10.0.10.218/31

Compute Rack #4

9372H-U39

10.0.10.219/31

Configure Loopback 0
10. Configure the loopback 0 interface so the switch loopback interface is
reachable. Repeat for 9332A,B,9372A,B,C,D,E,F,G,H.
R1-CS-9332-A-U42(config-if)# interface loopback 0
R1-CS-9332-A-U42(config-if)# ip address 10.0.0.220/32 <- Loopback 0
address below.
R1-CS-9332-A-U42(config-if)# ip router ospf 100 area 0.0.0.0 <-- Add
this if OSPF is used

Loopback 0 Interface IP Address


Rack Location

Switch

Loopback 0 Primary

Compute Rack #1

9332A-U41

10.0.0.220/32

Compute Rack #1

9332B-U40

10.0.0.221/32

Compute Rack #1

9372A-U39

10.0.0.212/32

Compute Rack #1

9372B-U38

10.0.0.213/32

Compute Rack #2

9372C-U40

10.0.0.214/32

Compute Rack #2

9372D-U39

10.0.0.215/32

Compute Rack #3

9372E-U40

10.0.0.216/32

Compute Rack #3

9372F-U39

10.0.0.217/32

Compute Rack #4

9372G-U40

10.0.0.218/32

Compute Rack #4

9372H-U39

10.0.0.219/32

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

55

Internal Network Configurations (Layer 3 mode)

Verify Neighborships
Verify OSPF neighborships
11. Verify that the L3 ospf neighborships are in FULL state on all L3
interfaces that are connected to neighboring switches.
R1-CS-9332-A-U42# sh ip ospf neighbors vrf ucpmanagement
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 8
Neighbor ID Pri State Up Time Address Interface
10.0.0.212 1 FULL/ - 1d05h 10.1.1.11 Eth1/1
10.0.0.212 1 FULL/ - 1d05h 10.1.1.21 Eth1/2
10.0.0.213 1 FULL/ - 2d04h 10.1.2.11 Eth1/3
10.0.0.213 1 FULL/ - 2d04h 10.1.2.21 Eth1/4
10.0.0.214 1 FULL/ - 2d04h 10.1.3.11 Eth1/5
10.0.0.214 1 FULL/ - 2d04h 10.1.3.21 Eth1/6
10.0.0.215 1 FULL/ - 2d04h 10.1.4.11 Eth1/7
10.0.0.215 1 FULL/ - 2d04h 10.1.4.21 Eth1/8
10.0.0.216 1 FULL/ - 1d05h 10.1.5.11 Eth1/9
10.0.0.216 1 FULL/ - 1d05h 10.1.5.21 Eth1/10
10.0.0.217 1 FULL/ - 2d04h 10.1.6.11 Eth1/11
10.0.0.217 1 FULL/ - 2d04h 10.1.6.21 Eth1/12
10.0.0.218 1 FULL/ - 2d04h 10.1.7.11 Eth1/13
10.0.0.218 1 FULL/ - 2d04h 10.1.7.21 Eth1/14
10.0.0.219 1 FULL/ - 2d04h 10.1.8.11 Eth1/15
10.0.0.219 1 FULL/ - 2d04h 10.1.8.21 Eth1/16

Verify BGP neighborships


12. Verify that the L3 BGP neighborships are in FULL state on all L3
interfaces that are connected to neighboring switches.
R1-CS-9332-A-U42# sh ip bgp summary
BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.0.0.208, local AS number 65001
BGP table version is 7, IPv4 Unicast config peers 5, capable peers 4
0 network entries and 0 paths using 0 bytes of memory
BGP attribute entries [0/0], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.1.11 4 65101 4378 4420 7 0 0 1d05h 10
10.1.1.21 4 65101 4378 4420 7 0 0 1d05h 10
10.1.2.11 4 65101 4378 4420 7 0 0 1d05h 10
10.1.2.21 4 65101 4378 4420 7 0 0 1d05h 10
10.1.3.11 4 65102 4378 4420 7 0 0 1d05h 10
10.1.3.21 4 65102 4378 4420 7 0 0 1d05h 10
10.1.4.11 4 65102 4378 4420 7 0 0 1d05h 10
10.1.4.21 4 65102 4378 4420 7 0 0 1d05h 10
10.1.5.11 4 65103 4378 4420 7 0 0 1d05h 10

56

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

10.1.5.21
10.1.6.11
10.1.6.21
10.1.7.11
10.1.7.21
10.1.8.11
10.1.9.21

4
4
4
4
4
4
4

65103
65103
65103
65104
65104
65104
65104

4378
4378
4378
4378
4378
4378
4378

4420
4420
4420
4420
4420
4420
4420

7
7
7
7
7
7
7

0
0
0
0
0
0
0

0
0
0
0
0
0
0

1d05h
1d05h
1d05h
1d05h
1d05h
1d05h
1d05h

10
10
10
10
10
10
10

Configure VXLAN
VXLAN is an overlay technology that encapsulates packets with a VXLAN
network identifier (VNI) to connect disjoint layer 2 networks within a layer 3
connected IP network. Traditionally layer 2 networks had to be connected
directly through a series of layer 2 devices. However, the need to connect
disjoint layer 2 networks for reasons such as vmotion for high availability or
flexibility to add hosts anywhere in the data center; VXLAN allows us the
freedom to tunnel traffic over a traditional Layer 3 network and scale with
multi-tenancy in mind.

Important: Please decide now whether the customer will use VXLAN Flood and
Learn or VXLAN/EVPN and skip to section accordingly. Only one VXLAN variant
can be configured at a time.

VXLAN Flood and Learn


Traditional VXLAN uses any layer 3 routing protocol as the underlay and
multicast for broadcast, unknown unicast, multicast, and arp flood and
learn traffic. VXLAN flood and learn does not support the idea of traffic
segregation for mult-tenancy support.

eBGP or OSPF underlay is used as the routing protocol of the underlying


network infrastructure. This is the IP Network that edge devices will use to
communicate.
PIM-SM is the multicast routing protocol used for multi-destination traffic
such as broadcast, unknown unicast, multicast, and flood and learn for endhost detection.
Anycast-RP is used for rendezvous-point redundancy. The Anycast-RPs
are configured on the spine switches with the same ip.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

57

Internal Network Configurations (Layer 3 mode)

Important: This VXLAN flood and learn configuration guide assumes the Unicast
Routing has been already configured. If not, go back to "Configure Global Unicast
Routing" on page 43 section and complete either the OSPF or eBGP configuration.
Enable Required Features

1. Enable pim, the multicast routing protocol. This needs to be configured


on all Spine switches 9332A and 9332B.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)# feature pim

2. Enable pim, nv overlay, and vn-segment-vlan-based on all Leaf switches


9372A,B,C,D,E,F,G,H.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one
R1-CS-9372-A-U40(config)# feature
R1-CS-9372-A-U40(config)# feature
R1-CS-9372-A-U40(config)# feature

per line. End with CNTL/Z.


pim
nv overlay
vn-segment-vlan-based

Configure tcam-size
3. Change tcam-size for region arp-ether on all Leaf switches
9372A,B,C,D,E,F,G,H.

Note: To allocate tcam space for region arp-ether, we need to reallocate tcam
space. In this example, we reduce the tcam space from the vacl region, and reallocate to the arp-ether region. Please select an appropriate region to re-allocate
for arp-ether.

R1-CS-9372-A-U40(config)# hardware access-list tcam region vacl 0


Warning: Please save config and reload the system for the
configuration to take effect
R1-CS-9372-A-U40(config)# hardware access-list tcam region arp-ether
256
Warning: Please save config and reload the system for the
configuration to take effect
R1-CS-9372-A-U40(config)# copy running-config startup-config
[########################################] 100%
Copy complete.

58

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

4. Reload (reboot) all Leaf switches 9372A,B,C,D,E,F,G,H.


R1-CS-9332-B-U41# reload
!!!WARNING! there is unsaved configuration!!!
This command will reboot the system. (y/n)? [n] y

Configure Loopback 0 Secondary Address


5. The secondary loopback 0 address is used by VXLAN for peering
between VTEPs when vPC is configured. Configure secondary IP address
on loopback 0 of all Nexus 9372 leaf switches.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)# interface loopback0
R1-CS-9372-A-U40(config-if)# ip address 10.0.101.101/32 secondary <this address should be the same on both vpc peers follow ip scheme.

Important: If eBGP was configured as the routing protocol make sure to


advertise the loopback0 secondary in a network statement. If ospf was configured
the ip router ospf command should already be present.
Rack Location

Switch

Loopback 0 Secondary

Compute Rack #1

9332A-U41

N/A

Compute Rack #1

9332B-U40

N/A

Compute Rack #1

9372A-U39

10.0.101.101/32

Compute Rack #1

9372B-U38

10.0.101.101/32

Compute Rack #2

9372C-U40

10.0.102.102/32

Compute Rack #2

9372D-U39

10.0.102.102/32

Compute Rack #3

9372E-U40

10.0.103.103/32

Compute Rack #3

9372F-U39

10.0.103.103/32

Compute Rack #4

9372G-U40

10.0.104.104/32

Compute Rack #4

9372H-U39

10.0.104.104/32

Configure PIM on L3 Interfaces


Configure on all layer 3 interfaces on the Nexus 9332 and Nexus 9372
switches. PIM must be enabled on all the Spine or Leaf facing interfaces and
loopback 0.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

59

Internal Network Configurations (Layer 3 mode)

6. Login to the Nexus 9332 A and B switches, and run following


commands.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#interface loopback0
R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9332-A-U42(config-if)#interface Ethernet1/1-16
R1-CS-9332-A-U42(config-if-range)# ip pim sparse-mode <- Enable
multicast on the interface

7. Login to the Nexus 9372 A, B, C, D, E, F, G, H switches, and run


following commands.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)#interface loopback0
R1-CS-9372-A-U40(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9372-A-U40(config-if)#interface Ethernet1/49-52
R1-CS-9372-A-U40(config-if)# ip pim sparse-mode <- Enable multicast
on the interface

Configure Anycast-RP (Nexus 9332PQ Spine Switch Only)


8. Interface Loopback 254 is configured with the RP Address. PIM must be
enabled for multicast to work. Configure exactly the same on 9332A and
9332B.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#interface loopback254
R1-CS-9332-A-U42(config-if)# ip address 10.254.254.254/32
R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9332-A-U42(config-if)# ip router ospf 100 area 0.0.0.0 <- Only
configure if ospf is the global routing protocol, if ebgp ignore.

Here we specify the RP address and multicast group list associated with it.
In addition, anycast rp is configured for RP redundancy on both Spines.
9. Configure exactly the same on 9332A and 9332B.

60

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#ip pim rp-address 10.254.254.254 group-list
230.1.1.0/24 <- 10.254.254.254 is the anycast RP.
R1-CS-9332-A-U42(config)#ip pim anycast-rp 10.254.254.254 10.0.0.220
<- 10.0.0.220 is the local RP address.
R1-CS-9332-A-U42(config)#ip pim anycast-rp 10.254.254.254 10.0.0.221
<- 10.0.0.221 is the peer RP address.

Rack Location

Local RP

Peer RP

Switch

Anycast RP

Compute Rack #1

9332A-U41

10.254.254.254

10.0.0.220

10.0.0.221

Compute Rack #1

9332B-U40

10.254.254.254

10.0.0.221

10.0.0.220

Configure RP Address (Nexus 9372PX Leaf Switch Only)


10. Configure the RP address and group-list associated with the RP on Nexus
9372 A,B,C,D,E,F,G,H switches.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)# ip pim rp-address 10.254.254.254 group-list
230.1.1.0/24

Configure VXLAN VTEP (Nexus9372PX Leaf Switch Only)


11. Configure the nve1 interface on all Leafs.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z
R1-CS-9372-A-U40(config)# interface nve1
R1-CS-9372-A-U40(config-if)# source-interface loopback0
R1-CS-9372-A-U40(config-if)# no shutdown

Configure VLAN ID and VXLANIDmapping


12. To configure an VLAN to be part of VXLAN VTEP, use following
commands on all Leaf VXLAN switches that are to be part of VXLAN.
Repeat following steps for all necessary VXLAN-VLAN mappings. In this
example, we map vlan 50 (L2 Migration VLAN) to vni 10000, and assign
vni 10000 to multicast group 230.1.1.1.

Note: For L2 VM Migration configuration, must use same migration VLAN ID on


all Nexus 9372 leaf switches.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

61

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-A-U40(config)#Vlan 50
R1-CS-9372-A-U40(config-vlan)# name migration_vlan
R1-CS-9372-A-U40(config-vlan)# vn-segment 10000 <- make vlan 50 part
of vni 10000
R1-CS-9372-A-U40(config-vlan)# interface nve1
R1-CS-9372-A-U40(config-if-nve)# member vni 10000
R1-CS-9372-A-U40(config-if-nve-vni)# mcast-group 230.1.1.1 <- assign
vni 10000 (vlan50) to multicast group 230.1.1.1

Note: Ideally, one VXLAN segment mapping to one IP multicast group is the way
to provide the optimal multicast forwarding. However, it is possible to have
multiple VXLAN segments share a single IP multicast group to achive the desired
VXLAN scalability. Having multiple-tenant VXLAN networks to share a multicast
group does not bring any implications to the Layer 2 isolation between the tenant
networks.
Verify VXLAN Configuration
13. Displays the nve peer status
R1-CS-9372-A-U40# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- ----------------nve1 10.0.0.101 Up CP 00:01:19 84b8.02ca.9625

14. Displays the vni and relevant info


R1-CS-9372-A-U40# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ ---nve1 10000 230.1.1.1 Up CP L2 [50] SA
nve1 10001 230.1.1.2 Up CP L2 [60] SA

15. Verify PIM neighbors are formed


R1-CS-9332-A-U42# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State

62

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

10.1.1.11
10.1.1.21
10.1.2.11
10.1.2.21
10.1.3.11
10.1.3.21
10.1.4.11
10.1.4.21

Ethernet1/1
Ethernet1/2
Ethernet1/3
Ethernet1/4
Ethernet1/5
Ethernet1/6
Ethernet1/7
Ethernet1/8

00:02:54
00:00:19
00:00:03
00:00:03
00:00:02
00:00:02
00:00:02
00:00:02

00:01:37
00:01:25
00:01:41
00:01:41
00:01:42
00:01:42
00:01:42
00:01:42

1
1
1
1
1
1
1
1

no
no
no
no
no
no
no
no

n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a

R1-CS-9372-A-U40# sh ip pim neighbor


PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.1.1.10 Ethernet1/49 00:03:25 00:01:39 1 no n/a
10.1.1.20 Ethernet1/50 00:00:50 00:01:23 1 no n/a
10.2.1.10 Ethernet1/51 00:00:43 00:01:31 1 no n/a
10.2.1.20 Ethernet1/52 00:00:43 00:01:44 1 no n/a

Note: If you can ping between vms but cannot ssh or send traffic, make sure that
jumbo frames are configured along the entire traffic path. This includes the
vswitch or vsphere distributed switch is using mtu 9000. In addition, make sure
the mtu is set to 9000 on the vm itself such as ifcfg-eth0 configuration if the vm is
a linux os.
Note: In VxLAN flood and learn mode (7.0(3)I1(2) and earlier), the default
gateway for VXLAN VLANs should be provisioned on external routing devices. In
VXLAN flood and learn mode (7.0(3)I2(1) and later), the default gateway for
VXLAN VLAN is recommended to be a centralized gateway on a pair of VPC devices
with FHRP (First Hop Redundancy Protocol) running between them. In BGP EVPN,
it is recommended to use the anycast gateway feature on all VTEPs. For more
information on configuring the default gateway on external routing devices please
reference this Cisco whitepaper.
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000series-switches/white-paper-c11-732453.html

VXLAN / EVPN
VXLAN over EVPN uses any layer 3 routing protocol as the underlay and
multicast for Broadcast, unknown unicast, and multicast. Arp flood and
learn is avoided and localized with ARP supression. VXLAN EVPN introduces
the idea of traffic segregation for mult-tenancy support.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

63

Internal Network Configurations (Layer 3 mode)

Important: This VXLAN over EVPN configuration guide assumes that OSPF has
been chosen as the unicast routing protocol and has been configured already. This
section does not cover eBGP as the unicast routing protocol.
Also, this VXLAN over EVPN configuration requires deep networking knowledge.
For more information, please refer to the "Cisco Nexus 9000 Series NX-OS VXLAN
Configuration Guide".
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/
7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_
Configuration_Guide_7x.html
OSPF Underlay is used as the routing protocol of the underlying network
infrastructure. This is the IP Network that edge devices will use to
communicate. Although any unicast routing protocol is supported.
MP-iBGP L2VPN EVPN is used as the overlay network or virtual network
built over the underlying network infrastructure. This allows traffic
segregation for multi-tenancy support.
PIM-SM is the multicast routing protocol used for multi-destination traffic
such as broadcast, unknown unicast, and multicast.
Anycast-RP is used for rendezvous-point redundancy. The Anycast-rps
are configured on the spine switches with the same ip.
Anycast gateway is used to have the same gateway and mac address on
all leaf layer switches for a locally defined subnet. This is useful when there
are multiple VXLAN VTEPs to facilitate a centralized gateway as opposed to
having separate gateways.
Enable Required Features
16. Run following commands to enable required feature on all Nexus 9332 A
and B Spine switches.
R1-CS-9332-A-U42(config)#
R1-CS-9332-A-U42(config)#
R1-CS-9332-A-U42(config)#
R1-CS-9332-A-U42(config)#

nv overlay evpn
feature ospf
feature bgp
feature pim

17. Run following commands to enable required feature on all Nexus 9372
A,B,C,D,E,F,G,H Leaf switches.
R1-CS-9372-A-U40(config)#nv overlay evpn
R1-CS-9372-A-U40(config)#feature ospf

64

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-A-U40(config)#feature
R1-CS-9372-A-U40(config)#feature
R1-CS-9372-A-U40(config)#feature
R1-CS-9372-A-U40(config)#feature

bgp
pim
vn-segment-vlan-based
nv overlay

Configure Loopback 0 Secondary Address


18. The secondary loopback 0 address is used by VXLAN for peering
between VTEPs when vPC is configured. Configure secondary IP address
on loopback 0 of all Nexus 9372 leaf switches.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)# interface loopback0
R1-CS-9372-A-U40(config-if)# ip address 10.0.101.101/32 secondary <this address should be the same on both vpc peers follow ip scheme.

Important: If ospf was configured the ip router ospf command should already
be present.
Rack Location

Switch

Loopback 0 Secondary

Compute Rack #1

9332A-U41

N/A

Compute Rack #1

9332B-U40

N/A

Compute Rack #1

9372A-U39

10.0.101.101/32

Compute Rack #1

9372B-U38

10.0.101.101/32

Compute Rack #2

9372C-U40

10.0.102.102/32

Compute Rack #2

9372D-U39

10.0.102.102/32

Compute Rack #3

9372E-U40

10.0.103.103/32

Compute Rack #3

9372F-U39

10.0.103.103/32

Compute Rack #4

9372G-U40

10.0.104.104/32

Compute Rack #4

9372H-U39

10.0.104.104/32

Configure PIM on L3 Interfaces


Configure on all layer 3 interfaces on the Nexus 9332 and Nexus 9372
switches. PIM must be enabled on all the Spine or Leaf facing interfaces and
loopback 0.
19. Login to the Nexus 9332 A and B switches, and run following
commands.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

65

Internal Network Configurations (Layer 3 mode)

R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#interface loopback0
R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9332-A-U42(config-if)#interface Ethernet1/1-16
R1-CS-9332-A-U42(config-if-range)# ip pim sparse-mode <- Enable
multicast on the interface

20. Login to the Nexus 9372 A, B, C, D, E, F, G, H switches, and run


following commands.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)#interface loopback0
R1-CS-9372-A-U40(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9372-A-U40(config-if)#interface Ethernet1/49-52
R1-CS-9372-A-U40(config-if)# ip pim sparse-mode <- Enable multicast
on the interface

Configure Anycast-RP (Nexus 9332PQ Spine Switch Only)


21. Interface Loopback 254 is configured with the RP Address. PIM must be
enabled for multicast to work. Configure exactly the same on 9332A and
9332B.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9332-A-U42(config)#interface loopback254
R1-CS-9332-A-U42(config-if)# ip address 10.254.254.254/32
R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast
on the interface
R1-CS-9332-A-U42(config-if)# ip router ospf 100 area 0.0.0.0 <- Only
configure if ospf is the global routing protocol.

22. Here we specify the RP address and multicast group list associated with
it. In addition, anycast rp is configured for RP redundancy on both
Spines. Configure exactly the same on 9332A and 9332B.
R1-CS-9332-A-U42# conf t
Enter configuration commands, one per line. End with CNTL/Z.

66

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9332-A-U42(config)#ip pim rp-address 10.254.254.254 group-list


230.1.1.0/24 <- 10.254.254.254 is the anycast RP.
R1-CS-9332-A-U42(config)#ip pim anycast-rp 10.254.254.254 10.0.0.220
<- 10.0.0.220 is the local RP address.
R1-CS-9332-A-U42(config)#ip pim anycast-rp 10.254.254.254 10.0.0.221
<- 10.0.0.221 is the peer RP address.

Rack Location

Local RP

Peer RP

Switch

Anycast RP

Compute Rack #1

9332A-U41

10.254.254.254

10.0.0.220

10.0.0.221

Compute Rack #1

9332B-U40

10.254.254.254

10.0.0.221

10.0.0.220

Configure RP Address (Nexus 9372PX Leaf Switch Only)


23. Configure the RP address and group-list associated with the RP on Nexus
9372 A,B,C,D,E,F,G,H switches.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)# ip pim rp-address 10.254.254.254 group-list
230.1.1.0/24

Configure BGP neighborships


24. Configure the following BGP configuration on the spine switches 9332A
and 9332B
R1-CS-9332-A-U42(config)# router bgp 65001
R1-CS-9332-A-U42(config-router)# router-id 10.0.0.220 <- Use
appropriate router-id according to table below
R1-CS-9332-A-U42(config-router)# address-family ipv4 unicast
R1-CS-9332-A-U42(config-router-af)# neighbor 10.0.0.212 remote-as
65001 <- repeat from here to "send-community both" for all 9372 leaf
switches according to table below
R1-CS-9332-A-U42(config-router-neighbor)# update-source loopback 0
R1-CS-9332-A-U42(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9332-A-U42(config-router-neighbor-af)# address-family l2vpn
evpn
R1-CS-9332-A-U42(config-router-neighbor-af)# send-community both
R1-CS-9332-A-U42(config-router-neighbor-af)# route-reflector-client

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

67

Internal Network Configurations (Layer 3 mode)

Rack Location

Switch

Router-id

Compute Rack #1

9332A-U42

10.0.0.220

Compute Rack #1

9332B-U41

10.0.0.221

Rack Location

Switch

Remote-AS #

Neighbor

Compute Rack #1 9372A-U40

65001

10.0.0.212

Compute Rack #1 9372B-U39

65001

10.0.0.213

Compute Rack #2 9372C-U40

65001

10.0.0.214

Compute Rack #2 9372D-U39

65001

10.0.0.215

Compute Rack #3 9372E-U40

65001

10.0.0.216

Compute Rack #3 9372F-U39

65001

10.0.0.217

Compute Rack #4 9372G-U40

65001

10.0.0.218

Compute Rack #4 9372H-U39

65001

10.0.0.219

25. Configure the following BGP configuration on the Nexus 9372


A,B,C,D,E,F,G,H leaf switches
R1-CS-9372-A-U40(config)#router bgp 65001
R1-CS-9372-A-U40(config-router)# router-id 10.0.0.212 <- modify
according to table below.
R1-CS-9372-A-U40(config-router)# address-family ipv4 unicast
R1-CS-9372-A-U40(config-router-af)# neighbor 10.0.0.220 remote-as
65001 <- modify according to Router ID of Nexus 9332A Spine switch.
R1-CS-9372-A-U40(config-router-neighbor)# update-source loopback0
R1-CS-9372-A-U40(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9372-A-U40(config-router-neighbor)# address-family l2vpn evpn
R1-CS-9372-A-U40(config-router-neighbor)# send-community both
R1-CS-9372-A-U40(config-router-neighbor)# neighbor 10.0.0.221 remoteas 65001<- modify according to Router ID of Nexus 9332B Spine switch.
R1-CS-9372-A-U40(config-router-neighbor)# update-source loopback0
R1-CS-9372-A-U40(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9372-A-U40(config-router-neighbor)# address-family l2vpn evpn
R1-CS-9372-A-U40(config-router-neighbor)# send-community both
R1-CS-9372-A-U40(config-router-neighbor)# vrf TENANT1
R1-CS-9372-A-U40(config-router-vrf)# address-family ipv4 unicast
R1-CS-9372-A-U40(config-router-vrf)# advertise l2vpn evpn

68

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

Rack Location

Switch

Router-id

Compute Rack #1

9372A-U40

10.0.0.212

Compute Rack #1

9372B-U39

10.0.0.213

Compute Rack #2

9372C-U40

10.0.0.214

Compute Rack #2

9372D-U39

10.0.0.215

Compute Rack #3

9372E-U40

10.0.0.216

Compute Rack #3

9372F-U39

10.0.0.217

Compute Rack #4

9372G-U40

10.0.0.218

Compute Rack #4

9372H-U39

10.0.0.219

Configure Anycast Gateway MAC (Nexus9372PX Leaf Switch Only)


26. Configure anycast gateway mac address on Nexus 9372
A,B,C,D,E,F,G,H leaf switches.
R1-CS-9372-A-U40(config)# fabric forwarding anycast-gateway-mac
2020.DEAD.BEEF

Configure VXLAN VTEP (Nexus9372PX Leaf Switch Only)


Configure the nve1 interface on all Nexus 9372 Leaf switches. Source
interface loopback 0 will pick up the secondary ip which is the same as the
secondary ip on the vPC peer. VNIs are added with arp suppression and
associated to a multicast group.
27. Create the nve interface and make vni 30099, 30100, 50000 as a
member on all Leaf VTEP switches 9372A,B,C,D.
R1-CS-9372-A-U40(config)# interface nve1
R1-CS-9372-A-U40(config-if-nve)# no shutdown
R1-CS-9372-A-U40(config-if-nve)# source-interface loopback 0
R1-CS-9372-A-U40(config-if-nve)# host-reachability protocol bgp
R1-CS-9372-A-U40(config-if-nve)# member vni 30099
R1-CS-9372-A-U40(config-if-nve-vni)# mcast-group 230.1.1.99
R1-CS-9372-A-U40(config-if-nve-vni)# member vni 30100
R1-CS-9372-A-U40(config-if-nve-vni)# suppress-arp
R1-CS-9372-A-U40(config-if-nve-vni)# mcast-group 230.1.1.100
R1-CS-9372-A-U40(config-if-nve-vni)# member vni 50000 associate-vrf

28. Create Vlans and associate to VN-Segment Vlan 2500 is used for
assigning to VRF for multi-tenancy purposes, Vlan 99 is a L2 only

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

69

Internal Network Configurations (Layer 3 mode)

segment while Vlan 100 is L2/L3 segment. Configure on all switches


9372A,B,C,D.
R1-CS-9372-A-U40(config)# vlan 99
R1-CS-9372-A-U40(config-vlan)# name L2onlyHostSegment
R1-CS-9372-A-U40(config-vlan)# vn-segment 30099
R1-CS-9372-A-U40(config-vlan)# vlan 100
Warning: Disabling IGMP snooping for VLAN 99.
R1-CS-9372-A-U40(config-vlan)# name L2L3HostSegment
R1-CS-9372-A-U40(config-vlan)# vn-segment 30100
R1-CS-9372-A-U40(config-vlan)# vlan 2500
Warning: Disabling IGMP snooping for VLAN 100.
R1-CS-9372-A-U40(config-vlan)# name FabricBD
R1-CS-9372-A-U40(config-vlan)# vn-segment 50000

29. Configure on all switches 9372A,B,C,D. VRF TENANT1 is assigned vni


segment 50000 which is same as vlan 2500s configured above.
R1-CS-9372-A-U40(config)# vrf context TENANT1
R1-CS-9372-A-U40(config-vrf)# vni 50000
R1-CS-9372-A-U40(config-vrf)# rd auto
R1-CS-9372-A-U40(config-vrf)# address-family ipv4 unicast
R1-CS-9372-A-U40(config-vrf-af-ipv4)# route-target both auto
R1-CS-9372-A-U40(config-vrf-af-ipv4)# route-target both auto evpn
R1-CS-9372-A-U40(config-vrf-af-ipv4)# address-family ipv6 unicast
R1-CS-9372-A-U40(config-vrf-af-ipv6)# route-target both auto
R1-CS-9372-A-U40(config-vrf-af-ipv6)# route-target both auto evpn

30. Configure on all switches 9372A,B,C,D. Vlan 100 has svi created in
anycast gateway mode. Vlan 99 does not have one because it is L2 mode
only.
R1-CS-9372-A-U40(config)# interface vlan 100
R1-CS-9372-A-U40(config-if)# no shutdown
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# vrf member TENANT1
Warning: Deleted all L3 config on interface Vlan100
R1-CS-9372-A-U40(config-if)# ip address 192.168.100.1/24
R1-CS-9372-A-U40(config-if)# fabric forwarding mode anycast-gateway

31. Configure on all switches 9372A,B,C,D. Vlan 2500 is created as well.


R1-CS-9372-A-U40(config)# interface vlan 2500

70

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-A-U40(config-if)# description FabricBD


R1-CS-9372-A-U40(config-if)# no shutdown
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# vrf member TENANT1
Warning: Deleted all L3 config on interface Vlan2500
R1-CS-9372-A-U40(config-if)# ip forward

32. Configure on all switches 9372A,B,C,D. EVPN configurations.


R1-CS-9372-A-U40(config)# evpn
R1-CS-9372-A-U40(config-evpn)# vni
R1-CS-9372-A-U40(config-evpn-evi)#
R1-CS-9372-A-U40(config-evpn-evi)#
R1-CS-9372-A-U40(config-evpn-evi)#
R1-CS-9372-A-U40(config-evpn-evi)#
R1-CS-9372-A-U40(config-evpn-evi)#
R1-CS-9372-A-U40(config-evpn-evi)#
R1-CS-9372-A-U40(config-evpn-evi)#

nve #

30099 l2
rd auto
route-target
route-target
vni 30100 l2
rd auto
route-target
route-target

VNI

import auto
export auto

import auto
export auto

Vlan

Mcast Group

30099

230.1.1.99

99

30100

230.1.1.100

100

50000

N/A

2500

Configure Additional VLAN


Configure following configurations on all switches 9372A,B,C,D exactly the
same.
33. Add the vni to the nve interface.
R1-CS-9372-A-U40(config)# interface nve 1
R1-CS-9372-A-U40(config-if-nve)# member vni 30101
R1-CS-9372-A-U40(config-if-nve-vni)# suppress-arp
R1-CS-9372-A-U40(config-if-nve-vni)# mcast-group 230.1.1.101

34. EVPN configurations.


R1-CS-9372-A-U40(config-if-nve-vni)# evpn
R1-CS-9372-A-U40(config-evpn)# vni 30001 l2
R1-CS-9372-A-U40(config-evpn-evi)# rd auto
No VLAN id configured, unable to generate auto RD

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

71

Internal Network Configurations (Layer 3 mode)

R1-CS-9372-A-U40(config-evpn-evi)# route-target import auto


R1-CS-9372-A-U40(config-evpn-evi)# route-target export auto

35. Create the vlan.


R1-CS-9372-A-U40(config-evpn-evi)# vlan 101
R1-CS-9372-A-U40(config-vlan)# name L2L3HostSegment2
R1-CS-9372-A-U40(config-vlan)# vn-segment 30101

Create the anycast gateway.


R1-CS-9372-A-U40(config-vlan)# interface vlan 101
Warning: Disabling IGMP snooping for VLAN 101.
R1-CS-9372-A-U40(config-if)# no shutdown
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# vrf member TENANT1
Warning: Deleted all L3 config on interface Vlan101
R1-CS-9372-A-U40(config-if)# ip address 192.168.101.1/24
R1-CS-9372-A-U40(config-if)# fabric forwarding mode anycast-gateway

nve #
1

VNI
30101

Mcast Group
230.1.1.101

Vlan
101

Verify VXLAN Configuration


36. Displays the vrf and corresponding vni
R1-CS-9372-A-U40# show nve vrf
VRF-Name VNI Interface Gateway-MAC
------------ ---------- --------- ----------------TENANT1 50000 nve1 78ba.f9ad.87f3

37. Displays the nve peer status


R1-CS-9372-A-U40# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- ----------------nve1 10.0.102.102 Up CP 00:01:19 84b8.02ca.9625

72

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

38. Displays the vni and relevant info


R1-CS-9372-A-U40# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ ---nve1 30099 230.1.1.99 Up CP L2 [99]
nve1 30100 230.1.1.100 Up CP L2 [100] SA
nve1 30101 230.1.1.101 Up CP L2 [101] SA
nve1 50000 n/a Up CP L3 [TENANT1]

39. Displays the vxlan interfaces


R1-CS-9372-A-U40# show vxlan interface
Interface Vlan VPL Ifindex LTL HW VP
========= ==== =========== === =====
Po1 99 0x50062000 0x10004 5701
Po1 100 0x50063000 0x10005 5702
Po1 101 0x50064000 0x10006 5703
Po1 2500 0x509c3000 0x10007 5704

40. show bgp l2vpn evpn summary


R1-CS-9372-A-U40# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.0.0.212, local AS number 65001
BGP table version is 139, L2VPN EVPN config peers 2, capable peers 2
19 network entries and 37 paths using 2824 bytes of memory
BGP attribute entries [22/3168], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [6/24]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.208 4 65001 11817 11796 139 0 0 1w1d 10
10.0.0.209 4 65001 11817 11796 139 0 0 1w1d 10

41. show bgp l2vpn evpn


R1-CS-9372-A-U40# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN
EVPN
BGP table version is 139, local router ID is 10.0.0.212

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

73

Internal Network Configurations (Layer 3 mode)

Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *valid, >-best


Path type: i-internal, e-external, c-confed, l-local, a-aggregate, rredist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.212:32868 (L2VNI 30001)
* i[2]:[0]:[0]:[48]:[0050.5694.4970]:[0]:[0.0.0.0]/216
10.0.0.101 100 0 i
*>i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.4970]:[32]:[192.168.101.45]/272
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
Route Distinguisher: 10.0.0.213:32817
* i[2]:[0]:[0]:[48]:[0050.5694.77a0]:[0]:[0.0.0.0]/216
10.0.0.100 100 0 i
*>i 10.0.0.100 100 0 i
* i[2]:[0]:[0]:[48]:[0050.5694.77a0]:[32]:[10.0.50.40]/272
10.0.0.100 100 0 i
*>i 10.0.0.100 100 0 i
Route Distinguisher: 10.0.0.214:32817
* i[2]:[0]:[0]:[48]:[0050.5694.0d60]:[0]:[0.0.0.0]/216
10.0.0.101 100 0 i
*>i 10.0.0.101 100 0 i
* i[2]:[0]:[0]:[48]:[0050.5694.0d60]:[32]:[10.0.50.45]/272
10.0.0.101 100 0 i
*>i 10.0.0.101 100 0 i
Route Distinguisher: 10.0.0.214:32868
*>i[2]:[0]:[0]:[48]:[0050.5694.4970]:[0]:[0.0.0.0]/216
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.4970]:[32]:[192.168.101.45]/272
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
Route Distinguisher: 10.0.0.215:32817
*>i[2]:[0]:[0]:[48]:[0050.5694.0d60]:[0]:[0.0.0.0]/216
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.0d60]:[32]:[10.0.50.45]/272
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
Route Distinguisher: 10.0.0.215:32868
*>i[2]:[0]:[0]:[48]:[0050.5694.4970]:[0]:[0.0.0.0]/216
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
* i[2]:[0]:[0]:[48]:[0050.5694.4970]:[32]:[192.168.101.45]/272
10.0.0.101 100 0 i
*>i 10.0.0.101 100 0 i
Route Distinguisher: 10.0.0.212:3 (L3VNI 50000)
*>i[2]:[0]:[0]:[48]:[0050.5694.0d60]:[32]:[10.0.50.45]/272

74

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Internal Network Configurations (Layer 3 mode)

10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.4970]:[32]:[192.168.101.45]/272
10.0.0.101 100 0 i
* i 10.0.0.101 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.5694.77a0]:[32]:[10.0.50.40]/272
10.0.0.100 100 0 i

42. show l2route evpn mac all


R1-CS-9372-A-U40# show l2route evpn mac all
Topology Mac Address Prod Next Hop (s)
----------- -------------- ------ --------------50 0050.5694.0d60 BGP 10.0.0.101
50 0050.5694.77a0 Local Eth1/3
101 0050.5694.4970 BGP 10.0.0.101
2500 84b8.02ca.9625 VXLAN 10.0.0.101

43. show l2route evpn mac-ip all


R1-CS-9372-A-U40# show l2route evpn mac-ip all
Topology ID Mac Address Prod Host IP Next Hop (s)
----------- -------------- ---- -------------------------------------- --------------101 0050.5694.4970 BGP 192.168.101.45 10.0.102.102

44. Verify PIM neighbors are formed


9332A(config-if-range)# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
10.1.1.11 Ethernet1/1 00:02:54 00:01:37 1 no n/a
10.1.1.21 Ethernet1/2 00:00:19 00:01:25 1 no n/a
10.1.2.11 Ethernet1/3 00:00:03 00:01:41 1 no n/a
10.1.2.21 Ethernet1/4 00:00:03 00:01:41 1 no n/a
10.1.3.11 Ethernet1/5 00:00:02 00:01:42 1 no n/a
10.1.3.21 Ethernet1/6 00:00:02 00:01:42 1 no n/a
10.1.4.11 Ethernet1/7 00:00:02 00:01:42 1 no n/a
10.1.4.21 Ethernet1/8 00:00:02 00:01:42 1 no n/a

R1-CS-9372-A-U40# sh ip pim neighbor


PIM Neighbor Status for VRF "default"

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

75

Uplink Configurations

Neighbor Interface Uptime Expires DR Bidir- BFD


Priority Capable State
10.1.1.10 Ethernet1/49 00:03:25 00:01:39 1 no n/a
10.1.1.20 Ethernet1/50 00:00:50 00:01:23 1 no n/a
10.2.1.10 Ethernet1/51 00:00:43 00:01:31 1 no n/a
10.2.1.20 Ethernet1/52 00:00:43 00:01:44 1 no n/a

Note: If you can ping between vms but cannot ssh or send traffic, make sure that
jumbo frames are configured along the entire traffic path. This includes the
vswitch or vsphere distributed switch is using mtu 9000. In addition, make sure
the mtu is set to 9000 on the vm itself such as ifcfg-eth0 configuration if the vm is
a linux os.

Save All Switch Configurations


45. Once all configurations are complete, please save all switch
configurations on Nexus 9332 A,B, Nexus 9372 A,B,C,D,E,F,G,H
switches to prevent configuration loss in the future.
R1-CS-9332-A-U42# copy running-config startup-config

Uplink Configurations
With UCP 4000 with Cisco Networking model, 8 x 10G ports on Nexus
9372PX or 6 x 40G ports on Nexus 9332PQ can be utilized for uplink
connectivity. Also, 2 x 1G ports on Nexus 3048 are allocated for uplink
connectivity.
The following tables and diagrams show default port allocations for uplink
connectivity.

Switch

76

Uplink Ports

Total Uplink
Bandwidth

Speed

Nexus 9372PX

Port #33-40
(8 ports per switch)

10Gbps

160Gbps

Nexus 9332PQ

Port #26-32
(6 ports per switch)

40Gbps

480Gbps

Nexus 3048

Port #47-48
(2 ports per switch)

1Gbps

4Gbps

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Note
Compute Rack
#1 only

Layer 2 mode
only

Uplink Configurations

Nexus 9372PX Uplink Ports

Nexus 9332PQ Uplink Ports

Nexus 3048 Uplink Ports

Also, UCP 4000 with Cisco Networking model supports Layer 2 connectivity
and Layer 3 connectivity to the core network.

Layer 2 Uplink Configurations


Typically the following 4 scenarios can be applied.
#

Scenario

Example of Network Topology

Connecting to multi chassis ether channel


technology enabled switches.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

77

Uplink Configurations

Scenario

Example of Network Topology

Connecting to standard Ethernet switches


2

(non multi chassis Ethernet channel


technology switch)

Connecting to multi chassis ether channel


technology enabled switches
3

&
Multiple Core Networks

Connecting to standard Ethernet switches

(non multi chassis Ethernet channel


technology switch)
&
Multiple Core Networks

The following scenarios show the example of detail configuration.

Important: In the Double-sided vPC configuration, the vPC domain identifiers


must be different between the upstream switches vPC domain and the UCP vPC
domain. During UCP Deployment, HDS support will configure the vPC domain ID
using the ID which HDS collects during pre-engagement. If the domain ID needs
to be changed later, please contact HDS support.
Note: Uplink configurations are identical between Nexus 9372PX (10G port),
Nexus 9332PQ (40G port), and Nexus 3048 (1G port) except port speed setting
and port number.

78

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Scenario 1: Connecting to single core network & multi chassis


ether channel technology enabled switches
Configure the uplink ports with one vPCs on the Nexus switches. Ensure
cross-connectivity of the physical connections to provide redundancy.
1. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
2. Configure the interface port-channel on both switches
interface Port-channel 10
description port-channel to core switch
switchport mode trunk
switchport trunk allowed vlan <VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
speed 40000 <-- Configure this if ports are 40G ports
vpc 10

3. Configure the upstream ports for 1st vPC on both switches


interface Ethernet1/26
switchport mode trunk
switchport trunk allowed vlan <VLANs>
speed 40000 <-- Configure this if ports are 40G ports
channel-group 10 mode active
...

4. Ensure that the vPC is correctly configured by checking vPC status


#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Data & Management VLANs>

Scenario 2: Connecting to single core network & non-multi


chassis ether channel technology enabled switches
Configure the uplink ports with two vPCs on the Nexus switches. Ensure
cross-connectivity of the physical connections to provide redundancy.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

79

Uplink Configurations

5. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
6. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to core switch A
switchport mode trunk
switchport trunk allowed vlan <VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
speed 40000 <-- Configure this if ports are 40G ports
vpc 10
interface Port-channel 11
description port-channel to core switch B
switchport mode trunk
switchport trunk allowed vlan <VLANs>
spanning-tree port type normal
speed 40000 <-- Configure this if ports are 40G ports
vpc 11

7. Configure the upstream ports for 1st vPC on both switches


interface Ethernet1/26
switchport mode trunk
switchport trunk allowed vlan <VLANs>
speed 40000 <-- Configure this if ports are 40G ports
channel-group 10 mode active
...

8. Configure the upstream ports for 2nd vPC on both switches


interface Ethernet1/28
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
speed 40000 <-- Configure this if ports are 40G ports
channel-group 11 mode active
...

9. Ensure that the vPC is correctly configured by checking vPC status


#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- --------------------------

80

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

10 Po10 up success success <Data & Management VLANs>


11 Po11 up success success <Data & Management VLANs>

Scenario 3: Connecting to multiple core networks & multichassis ether channel technology enabled switches
Configure the uplink ports with two vPCs on the Nexus switches. Ensure
cross-connectivity of the physical connections to provide redundancy.
10. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
11. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to management core network
switchport mode trunk
switchport trunk allowed vlan <VLANs for management>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
speed 40000 <-- Configure this if ports are 40G ports
vpc 10
interface Port-channel 11
description port-channel to data core network
switchport mode trunk
switchport trunk allowed vlan <VLANs for data>
spanning-tree port type normal
speed 40000 <-- Configure this if ports are 40G ports
vpc 11

12. Configure the upstream ports for 1st vPC on both switches
interface Ethernet1/26
switchport mode trunk
switchport trunk allowed vlan <VLANs for management>
speed 40000 <-- Configure this if ports are 40G ports
channel-group 10 mode active
...

13. Configure the upstream ports for 2nd vPC on both switches
interface Ethernet1/28
switchport mode trunk
switchport trunk allowed vlan <VLANs for data>

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

81

Uplink Configurations

speed 40000 <-- Configure this if ports are 40G ports


channel-group 11 mode active
...

14. Ensure that the vPC is correctly configured by checking vPC status
#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <VLANs for Management>
11 Po11 up success success <VLANs for Data>

Scenario 4: Connecting to multiple core networks & non-multi


chassis ether channel technology enabled switches
Configure the uplink ports with four vPCs on the Nexus switches. Ensure
cross-connectivity of the physical connections to provide redundancy.
15. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
16. Configure the four interface port-channels on both switches
interface Port-channel 10
description port-channel to management core switch A
switchport mode trunk
switchport trunk allowed vlan <VLANs for Management>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
speed 40000 <-- Configure this if ports are 40G ports
vpc 10
interface Port-channel 11
description port-channel to management core switch B
switchport mode trunk
switchport trunk allowed vlan <VLANs for Management>
spanning-tree port type normal
speed 40000 <-- Configure this if ports are 40G ports
vpc 11
interface Port-channel 12
description port-channel to data core switch A
switchport mode trunk
switchport trunk allowed vlan <VLANs for Data>
spanning-tree port type normal

82

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

speed 40000 <-- Configure this if ports are 40G ports


vpc 12
interface Port-channel 13
description port-channel to data core switch B
switchport mode trunk
switchport trunk allowed vlan <VLANs for Data>
spanning-tree port type normal
speed 40000 <-- Configure this if ports are 40G ports
vpc 13

17. Configure the upstream ports for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <VLANs for Management>
channel-group 10 mode active

18. Configure the upstream ports for 2nd vPC on both switches
interface Ethernet1/2
switchport mode trunk
switchport trunk allowed vlan <VLANs for Management>
channel-group 11 mode active

19. Configure the upstream ports for 3rd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <VLANs for Data>
channel-group 12 mode active

20. Configure the upstream ports for 4th vPC on both switches
interface Ethernet1/4
switchport mode trunk
switchport trunk allowed vlan <VLANs for Data>
channel-group 13 mode active

21. Ensure that the vPC is correctly configured by checking vPC status
#show vpc
vPC status
-----------------------------------------------------------------

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

83

Uplink Configurations

id Port Status Consistency Reason Active vlans


------ ----------- ------ ----------- -------------------------10 Po10 up success success <Management VLANs>
11 Po11 up success success <Management VLANs>
12 Po12 up success success <Data VLANs>
13 Po13 up success success <Data VLANs>

Layer 3 Uplink Configurations


Typically the following 3 scenarios can be applied.
#

Scenario
OSPF Peering to Core Network

1
2

BGP Peering to Core Network


Static Routing

Important: Uplink connection on Nexus 3048 switch is not covered in Layer 3


mode. If uplink on Nexus 3048 configuration in layer 3 mode is required, please
contact HDS support.
Note: Uplink configurations are identical between Nexus 9372PX (10G port) and
Nexus 9332PQ (40G port) except port speed setting and port number.

Scenario 1: OSPF Peering to Core Network


Configure the uplink ports with OSPF Peering.
22. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
23. Configure layer 3 interfaces on Nexus 9372 A and B (or Nexus 9332 A
and B)
R1-CS-9372-A-U40(config)# int e 1/33
R1-CS-9372-A-U40(config-if)# description L3 link to customer network
R1-CS-9372-A-U40(config-if)# ip address 192.168.1.2/24 <- enter ip
and netmask provided by customer
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# ip router ospf 100 area 0.0.0.0

84

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

R1-CS-9372-A-U40(config-if)# no shut

24. Verify that the ospf neighborships are in FULL state on all L3 interfaces
that are connected to neighboring switches.
R1-CS-9372-A-U40# sh ip ospf neighbors vrf ucpmanagement
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 8
Neighbor ID Pri State Up Time Address Interface
10.0.0.212 1 FULL/ - 1d05h 10.1.1.11 Eth1/1
10.0.0.212 1 FULL/ - 1d05h 10.1.1.21 Eth1/2
...

Scenario 2: BGP Peering to Core Network


25. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
26. Configure layer 3 interfaces on Nexus 9372 A and B (or Nexus 9332 A
and B)
R1-CS-9372-A-U40(config)# int e 1/33
R1-CS-9372-A-U40(config-if)# description L3 link to customer network
R1-CS-9372-A-U40(config-if)# ip address 192.168.1.2/31 <- enter ip
and netmask provided by customer
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# no shut

27. Add the network statement and neighbor statement to bgp


configuration.
R1-CS-9372-A-U40# conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-CS-9372-A-U40(config)#router bgp 65101 <- AS number
R1-CS-9372-A-U40(config-router-af)# network 192.168.1.2/31 <- add
network statement for L3 interfaces that connects to customer core
network.
R1-CS-9372-A-U40(config-router-neighbor)# template peer BGPCORE <template to save time on neighborships configuration
R1-CS-9372-A-U40(config-router-neighbor)# address-family ipv4 unicast
R1-CS-9372-A-U40(config-router-neighbor)# next-hop-self
R1-CS-9372-A-U40(config-router-neighbor)# soft-reconfiguration
inbound

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

85

Uplink Configurations

R1-CS-9372-A-U40(config-router-neighbor)# neighbor 192.168.1.3/31


remote-as 65002 <- establish the neighborship with the ip address on
directly connected customers core link.
R1-CS-9372-A-U40(config-router-neighbor)# inherit peer BGPCORE

28. Verify that the BGP neighborships are in FULL state on all L3 interfaces
that are connected to neighboring switches.
R1-CS-9372-A-U40# sh ip bgp summary
BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.0.0.208, local AS number 65001
BGP table version is 7, IPv4 Unicast config peers 5, capable peers 4
0 network entries and 0 paths using 0 bytes of memory
BGP attribute entries [0/0], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.1.11 4 65101 4378 4420 7 0 0 1d05h 10
10.1.1.21 4 65101 4378 4420 7 0 0 1d05h 10
...

Scenario 3: Static Routing


If customer has decided not to use L3 peering through a routing protocol
with the 9332 or 9372, a default route can be added on the switches that
are connected to the customer network via a L3 interface. This way the
customer network can be reached.
29. Login to the Nexus 9372 A and B, or 9332 A and B which be used for
uplink connection using SSH.
30. Configure layer 3 interfaces on Nexus 9372 A and B (or Nexus 9332 A
and B)
R1-CS-9372-A-U40(config)# int e 1/33
R1-CS-9372-A-U40(config-if)# description L3 link to customer network
R1-CS-9372-A-U40(config-if)# ip address 10.20.1.10/31 <- enter ip and
netmask provided by customer
R1-CS-9372-A-U40(config-if)# mtu 9216
R1-CS-9372-A-U40(config-if)# no shut

31. [BGP case]Add the default route into routing bgp configuration.
R1-CS-9372-A-U40(config)#router bgp 65101 <- AS number
R1-CS-9372-A-U40(config-router-af)# network 0.0.0.0/0

86

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

32. [OSPF case]Add the default route into routing ospfconfiguration.


R1-CS-9372-A-U40(config)#router ospf 100
R1-CS-9372-A-U40(config-router)# default-information originate

33. Verify that the BGP neighborships are in FULL state on all L3 interfaces
that are connected to neighboring switches.
R1-CS-9372-A-U40# ip route 0.0.0.0/0 10.20.1.11 <- fill in the ip
addresss with the customer's ip address of the L3 interface.

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

87

88

Chapter 3: UCP4000 Cisco Systems


UCP Network Architecture and Configuration Guide

4
UCP4000E Systems
This chapter will cover the network architecture and uplink configuration
examples for UCP 4000E Systems.

Network Architecture and Configurations Overview


In the UCP 4000E model, UCP utilizes Cisco Nexus 5548UP switch, and
leverages its Storage Network technology (FCoE).
UCP 4000E Network Architecture

Physical Configurations
The following table shows port usage of the Cisco Nexus 5548UP switches.

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

89

Network Architecture and Configurations Overview

Cisco Nexus 5548UP Port Usage


Usage

Ethernet Port Counts

Speed

Uplinks to Core Network

10Gbps

ISL to other Nexus 5548UP

10Gbps

Max 24

10Gbps

CB500 SVP Connectivity

1Gbps

Management Server Connectivity

10Gbps

Storage System Management Connectivity

1Gbps

Storage System Connectivity (Native FC)

8 or 16Gbps

CB500 Blade Server Connectivity (FCoE)


(Connecting to CB500 In-Chassis Pass-Throu module)

UCP Director Network Management Features


Onboarding Switches The UCP Director supports onboarding of the
Nexus 5548UP for management and health monitoring.
Switch Health Monitoring The UCP Director monitors the health of the
onboard switches and notifies if there are any warnings or errors detected
on the switches.
Configuration Backup UCP Director will periodically take backup
configurations of the switches in its inventory. UCP Director can also
manually backup/apply the switch configurations through user request.
Configure Host/Cluster Network This allows the user to configure
the Nexus 5548UP server facing interface on a blade by blade basis. The
native vlan which is configured and managed by UCP for the management
network and additional vlans such as vMotion or Compute vlans can also be
managed through this UCP Director feature.
VLAN DB management - If a new vlan is added through the Configure
Host/Cluster Network feature, the vlan is added to the pair of Nexus
5548UP server facing interfaces as an allowed vlan and is created in the vlan
db.
FC Zone management - UCP Director will configure FCZone for storage
facing FC ports and server facing vFC ports.

90

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

Network Architecture and Configurations Overview

UCP Appliance Initial vPC and Spanning Tree Configuration


vPC peer-link The vPC peerlink trunk port-channel interface which
allows traffic to ingress/egress between vPC peers also allows all vlans by
default.
vPC keep-alive The vPC keep-alive link is established through the
management vrf with the source as MGMT 0 IP and destination as the
MGMT 0 IP on the peer.
vPC orphan-port suspend In the event that there is a vPC peer-link
failure, all server facing vPC orphan ports will be disabled to avoid a vPC
dual-active scenario.
vPC peer-gateway Peer gateway is configured on vPC peers to act as
the gateway even when packets are destined to the vPC peers MAC address.
vPC peer-switch Allows both vPC peers to act as Root Bridge for the
vlan.
vPC ip arp synchronize Allows both vPC peers to synchronize arp
entries for faster ARP learning and convergence.
vPC delay restore 240 In the event of switch reboot, the vPC bring-up
is delayed by 240 seconds to allow network re-converge before bringing the
vPC peer up.
Rapid-PVST Mode Layer 2 mode) Spanning tree rapid-PVST mode is
configured with the priority 61440.
vPC port-channels - vPC trunk port-channel is configured on uplink ports
during initial deployment.
Sample vPC Configuration
vpc domain 901
peer-switch
role priority 100
peer-keepalive destination 10.21.101.213 source 10.21.101.212
delay restore 240
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize

Sample Spanning tree configuration

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

91

Network Architecture and Configurations Overview

spanning-tree vlan 1-3967 priority 61440

Server Facing Port Configurations


The UCP Director manages the server facing ports which are connected to
the CB500 chassis via passthru modules that connect to blades.
When managing the server facing ports, UCP Director sets the port in
switchport trunk mode. In addition, it sets the native vlan on the port to the
specified management vlan id which is chosen during deployment. It is
critical these settings are never modified because the native vlan is used by
UCP Director for management functions.
Finally, the port is set to spanning-tree port type edge trunk and
whichever vlans are allowed are configured by UCP Director through the
switchport trunk allowed vlan command. Once user needs to modify the
allowed vlans they can perform a Configure host/cluster vlan operation
and UCP Director will modify the port configuration accordingly.
The "spanning-tree port type edge trunk" and "vpc orphan-port suspend"
are configured on all of server facing ports during UCP Deployment. UCP
Director Software will not touch these configuration. HDS recommends not
to change these configurations.
Also, Virtual FC interfaces are created and configured during UCP
Deployment. This vfc port is bound to server facing ports, and used for FCoE
traffic.

Sample Port Configuration


interface Ethernet1/17
description to_blade_server
switchport mode trunk
switchport trunk native vlan 92
switchport trunk allowed vlan 92-94
spanning-tree port type edge trunk
vpc orphan-port suspend
interface vfc17
bind interface Ethernet1/17
no shutdown

92

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Uplink Configurations
The UCP 4000E system shares a pair of Cisco Nexus 5548UP switches for
data traffic and management traffic.
The Cisco 5548UP switches act as a single switch in Virtual Port Channel
(vPC) configuration for spanning tree topology determination. The different
options for upstream connectivity to the production network infrastructure
are dependent on whether the immediate upstream pair of switches is in
vPC or spanning-tree configuration.
By default, port 1 to 4 on both Cisco Nexus 5548UP switches are dedicated
for connecting core network. And, port channel 10 is configured for these
ports as a vPC.

Typically the following 4 scenarios can be applied.


#

Scenario

Example of Network Topology

Connecting to multi chassis ether channel


technology enabled switches.
1

&
Single Core Network

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

93

Uplink Configurations

Scenario

Example of Network Topology

Connecting to standard Ethernet switches

(non multi chassis Ethernet channel


technology switch)
&
Single Core Network

Connecting to multi chassis ether channel


technology enabled switches.
3

&
Separated Core Network

Connecting to standard Ethernet switches


4

&
Separated Core Network

The following scenarios show the example of detail configuration.

Important: In the Double-sided vPC configuration, vPC domain identifiers must


be different between upstream switches vPC domain and UCP vPC domain.
During UCP Deployment, HDS support will configure vPC domain ID using the ID
which HDS corrects during pre-engagement. If the domain ID need to be changed
later, please contact HDS support.

94

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Scenario 1: Connecting to single core network & multi chassis


ether channel technology enabled switches
Configure the uplink ports with one vPCs on the Nexus 5548UP. Depending
on bandwidth requirements, up to 4 ports per switch can be added to these
vPCs. Ensure cross-connectivity of the physical connections to provide
redundancy.
1. Open SSHclient software, and login to the Both Nexus 5548UP A and B
2. Configure the interface port-channel on both switches
interface Port-channel 10
description port-channel to core switch
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
vpc 10

3. Configure the upstream ports 1/1-1/4 for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
channel-group 10 mode active
...

4. Ensure that the vPC is correctly configured by checking vPC status


#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Data & Management VLANs>

Scenario 2: Connecting to single core network & non-multi


chassis ether channel technology enabled switches
Configure the uplink ports with two vPCs on the Nexus 5548UP switches.
Depending on bandwidth requirements, up to 2 ports per switch can be
added to these vPCs. Ensure cross-connectivity of the physical connections

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

95

Uplink Configurations

to provide redundancy.
5. Open SSHclient software, and login to the Both Nexus 5548UP A and B
6. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to core switch A
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
vpc 10
interface Port-channel 11
description port-channel to core switch B
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
spanning-tree port type normal
vpc 11

7. Configure the upstream ports 1/1-1/2 for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
channel-group 10 mode active
...

8. Configure the upstream ports 1/3-1/4 for 2nd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <Data & Management VLANs>
channel-group 11 mode active
...

9. Ensure that the vPC is correctly configured by checking vPC status


#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Data & Management VLANs>
11 Po11 up success success <Data & Management VLANs>

96

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

Scenario 3: Connecting to separate core networks & multi


chassis ether channel technology enabled switches
Configure the uplink ports with two vPCs on the Nexus 5548UP switches.
Depending on bandwidth requirements, up to 2 ports per switch can be
added to these vPCs. Ensure cross-connectivity of the physical connections
to provide redundancy.
10. Open SSHclient software, and login to the Both Nexus 5548UP A and B
11. Configure the two interface port-channels on both switches
interface Port-channel 10
description port-channel to management core switch
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
vpc 10
interface Port-channel 11
description port-channel to data core switch
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
spanning-tree port type normal
vpc 11

12. Configure the upstream ports 1/1-1/2 for 1st vPC on both switches
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
channel-group 10 mode active
...

13. Configure the upstream ports 1/3-1/4 for 2nd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
channel-group 11 mode active
...

14. Ensure that the vPC is correctly configured by checking vPC status

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

97

Uplink Configurations

#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Management VLANs>
11 Po11 up success success <Data VLANs>

Scenario 4: Connecting to separate core networks & non-multi


chassis ether channel technology enabled switches
Configure the uplink ports with four vPCs on the Nexus 5548UP switches. 1
ports per switch can be added to these vPCs. Ensure cross-connectivity of
the physical connections to provide redundancy.
15. Configure the four interface port-channels on both switches
interface Port-channel 10
description port-channel to management core switch A
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
spanning-tree port type normal <-- Cisco Recommendation :Do not
enable Bridge Assurance (BA) on vPC.
vpc 10
interface Port-channel 11
description port-channel to management core switch B
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
spanning-tree port type normal
vpc 11
interface Port-channel 12
description port-channel to data core switch A
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
spanning-tree port type normal
vpc 12
interface Port-channel 13
description port-channel to data core switch B
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
spanning-tree port type normal
vpc 13

16. Configure the upstream port 1/1 for 1st vPC on both switches

98

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

Uplink Configurations

interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
channel-group 10 mode active

17. Configure the upstream port 1/2 for 2nd vPC on both switches
interface Ethernet1/2
switchport mode trunk
switchport trunk allowed vlan <Management VLANs>
channel-group 11 mode active

18. Configure the upstream port 1/3 for 3rd vPC on both switches
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
channel-group 12 mode active

19. Configure the upstream port 1/4 for 4th vPC on both switches
interface Ethernet1/4
switchport mode trunk
switchport trunk allowed vlan <Data VLANs>
channel-group 13 mode active

20. Ensure that the vPC is correctly configured by checking vPC status
#show vpc
vPC status
----------------------------------------------------------------id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------10 Po10 up success success <Management VLANs>
11 Po11 up success success <Management VLANs>
12 Po12 up success success <Data VLANs>
13 Po13 up success success <Data VLANs>

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

99

100

Chapter 4: UCP4000E Systems


UCP Network Architecture and Configuration Guide

UCP Network Architecture and Configuration Guide

Hitachi Data Systems


Corporate Headquarters
2845 Lafayette Street
Santa Clara, California 95050-2627
U.S.A.
www.hds.com
Regional Contact Information
Americas
+1 408 970 1000
info@hds.com
Europe, Middle East, and Africa
+44 (0) 1753 618000
info.emea@hds.com
Asia Pacific
+852 3189 7900
hds.marketing.apac@hds.com

MK-92UCP084-00

Das könnte Ihnen auch gefallen