Beruflich Dokumente
Kultur Dokumente
Updated: 7/7/2015
Marcus DAndrea, HP DCA
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Table of contents
Introduction ............................................................................................................................................................................................... 3
Testing Topologies .................................................................................................................................................................................. 4
Cisco ACI Overview ................................................................................................................................................................................... 7
Understanding the ACI Policy Model and Keeping it Simple .................................................................................................... 7
Cisco ACI Switching and APIC Key Integration Elements for HP ............................................................................................10
VLAN Tagging in ACI ................................................................................................................................................................. 10
Virtual Machine Manager Domain(VMM) with ACI and HP ............................................................................................... 11
Configuring Hypervisors in ACI without VMM integration ................................................................................................ 15
Server Infrastructure .............................................................................................................................................................................18
Virtual Connect FlexFabric 20/40 F8 with Cisco ACI ...................................................................................................................18
FlexFabric 20/40 F8 Overview ......................................................................................................................................................18
Ports and Port Numbering ..................................................................................................................................................... 18
FlexFabric 20/40 F8 with OneView Key Integrations and Best Practices with ACI ............................................................19
FlexFabric 20/40 F8 with Virtual Connect Manager Key Integrations and Best Practices with ACI ...............................21
Cisco Nexus HP B22 Module with Cisco ACI .....................................................................................................................................25
Cisco Nexus HP B22 Overview ......................................................................................................................................................25
Cisco Nexus HP B22 with OneView Key Integrations with ACI ...............................................................................................25
Cisco Nexus HP B22 Supported Topologies with ACI ..............................................................................................................26
HP Operations Orchestration with Cisco ACI and OneView ...........................................................................................................27
HP Operations Orchestration Overview ......................................................................................................................................27
HP OneView REST API .....................................................................................................................................................................27
Cisco ACI REST API ...........................................................................................................................................................................27
ACI References .......................................................................................................................................................................................29
Conclusion ...............................................................................................................................................................................................30
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Introduction
This document provides a basis for understanding and demystifying Cisco ACI with HP solutions. This guide provides information
to show how the solutions work together but does not detail all of the implementation tasks necessary to deploy Cisco ACI. . For
detailed installation steps, please see the ACI References section at the end of this document.
The hardware and software for Cisco ACI testing included the following:
Infrastructure
Components
Products
Version
HP BladeSystem C7000 Chassis with ProLiant BL460 Gen8 and Gen 9 servers
SPP2014.11
4.31/4.45
Networking
Cisco ACI Fabric Nexus 9504 Spine and Nexus 9396 PX Leaf
Networking
1.0(4h)
Networking
Management S/W
Products
Version
Virtual Machine
Management
5.5
Bare Metal
Microsoft Windows
2012 R2
Virtual Machine
14.04
Infrastructure
Management
1.20
Virtual Connect
Manager
4.45
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Testing Topologies
Figure 1: Virtual Connect FlexFabric 20/40 with OneView
The above topology reflects an HP C7000 blade enclosure managed by HP OneView 1.20 and connected to a pair of ACI leaf
switches with Virtual Connect FlexFabric 20/40 F8 Modules. The blades inside the systems are loaded with Windows and VSphere
along with a test Ubuntu VM.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
The above topology reflects an HP C7000 blade enclosure managed by HP OneView 1.20 and connected to a pair of ACI leaf
switches with Cisco Nexus B22 for HP. The blades inside the systems are loaded with Windows and VSphere along with a test
Ubuntu VM.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
The above topology reflects an HP C7000 blade enclosure managed by HP Virtual Connect Manager 4.45 and connected to a pair
of ACI leaf switches with Virtual Connect FlexFabric 20/40 F8 Modules. The blades inside the systems are loaded with Windows
and VSphere along with a test Ubuntu VM.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Bridge Domain A bridge domain represents a Layer 2 construct. Think of this as a layer 2 switch without any routing
functionality.
Private Network/Context A context is a simple layer 3 domain definition and is analogous to a Virtual Route Forwarder
(VRF) from a networking point of view.
Endpoints Endpoints are just what they sound like and can be anything from Servers, Storage, Routers, or other
devices.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Endpoint Group The Endpoint Group construct contains a collection of Endpoints. A simple way to think of this this is
group nesting in email systems in which you have a distribution lists that contains smaller subsets of distribution lists.
Contracts Contracts define who can communicate with whom in ACI. EPGs can only communicate with other EPGs
according to contract rules. If you want to communicate in the same EPG you do not need a contract but if you want to
communicate between EPGs both parties need to agree.
Filters- Provide a construct to manipulate Layer 2 thru Layer 4 protocol types and ports. Think Access Control Lists
(ACL) if you have a security background.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
External Connectivity External connectivity is just what it sounds like and surfing the web will require some external
connectivity from the ACI fabric to the outside world.
Another aspect of ACI that is not covered in this paper but is key to getting an ACI fabric provisioned is fabric/access policies.
Fabric/access policies leverage things like interfaces, Virtual Port Channels, Link Aggregation Control Protocol, Network Time
Protocol, DHCP etc. This can be found in most of the Cisco documentation and anything that is modified from default perspective
to make ACI work with HP will be covered in this paper.
Access Polices are defined under the Fabric menu. A sample HP B22 FEX Interface Profile Definition is shown below.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Tagged When this option is used the leaf node is expecting the incoming traffic to be tagged with a specified VLAN ID.
This is similar to traditional Nexus command Switchport mode trunk.
2.
Untagged When this option is used the leaf node expects untagged traffic which is traffic that does not have a VLAN ID.
This is similar to traditional Nexus command Switchport mode access.
3.
802.1P When this option is used the leaf node expects tagged traffic with 802.1P headers. This is similar to the
traditional Nexus command Switchport trunk native vlan.
When using tagged VLAN/VXLAN ID in multiple EPGs with the same interface the VLAN/VXLAN ID must be unique.
2.
When using untagged VLAN/VXLAN ID against an interface this interface can only be bound in one EPG.
3.
When using 802.1P option on an interface this designation can exist in only one EPG. However, the same interface can
exist in another EPG and have a tagged option applied against it.
Encap Mode
Use Case
Untagged
No
Tagged
Yes
802.1P
Yes
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
vSphere DVS functionality controlled and maintained by ACI. The topic of LLDP is important in this deployment as the endpoint
information that sits on the downlinks of HP Virtual Connect must be presented to the DVS. Virtual Connect Manager version 4.45
will provide the ability to pass LLDP information to the DVS for consumption by the ACI Fabric.
Note: OneView 1.20 does not support VMM integrated deployment.
LLDP Information dynamically learned in ACI from the VMM Integration as well as what the Virtual Connect downlink port is
reporting back to the DVS which is illustrated below.
Verification of the DVS deployment will be shown from a VC perspective in the upcoming figures and will correlate with the above
graphic to help stich together the pieces of the integration.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Below is the integration of ACI VMM with a VSphere 5.5 U2 VCenter deployed in a topology outline in Figure 3: leveraging Virtual
Connect 4.45.
DVS in VCenter for Virtual Connect 4.45 Deployment:
LLDP-VC Cluster with Ubuntu-1 VM and MAC address 00:50:56:a5:f7:fe and IP Address 10.17.60.100.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
On a leaf node of the ACI Fabric, endpoints can be displayed by issuing the show endpoint command. The green arrow shows
the MAC address of the Ubuntu-1 Virtual Machine which was learned dynamically across port eth 1/11.
Ubuntu-1 VM is pinging gateway 10.17.60.1 to verify proper operation of DVS and dynamic mapping in ACI.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Ubuntu-2 Virtual Machine on VLAN-55 of VSwitch with MAC Address 00:50:56:a5:16:9c and IP Address 10.17.60.101.
On a leaf node of the ACI Fabric, endpoints can be displayed by issuing the show endpoint command. The green arrow shows
the MAC address of the Ubuntu-2 Virtual Machine.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Ubuntu-2 VM is pinging gateway 10.17.60.1 and 10.16.1.15 to verify proper operation of VSwitch.
This section has covered the key ACI Switching and APIC Integration elements for making Cisco ACI and HP interoperability
successful.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Server Infrastructure
The HP BladeSystem c7000 enclosure houses the VMware hosts. Server infrastructure management is provided by HP OneView
software. HP OneView is HPs converged infrastructure platform managing server provisioning, monitoring, and LAN/SAN
networking configuration tasks.
HP OneView integrates functions of all traditional HP management software such as Onboard Administrator, Virtual Connect
Manager, Virtual Connect Enterprise Manager and Insight Control software.
OneView software is packaged as VMware virtual appliance ova format. Users can download software at
http://www.hp.com/go/oneview. Detailed HP OneView installation and initial setup procedure are discussed in HP OneView 1.20
Deployment and Management Guide.
HP Virtual Connect FlexFabric-20/40 F8 Modules are the simplest, most flexible way to connect virtualized server blades to data or
storage networks. VC FlexFabric-20/40 F8 modules eliminate network sprawl at the server edge with one device that converges
traffic inside enclosures and directly connects to external LANs and SANs. Using Flex-20 technology with Fiber Channel over
Ethernet and accelerated iSCSI, these modules converge traffic over high-speed 10Gb/20Gb connections to servers with HP
FlexFabric Adapters. Each redundant pair of Virtual Connect FlexFabric modules provide eight adjustable downlink connections ( six
Ethernet and two Fiber Channel, or six Ethernet and two iSCSI or eight Ethernet) to dual-port 10Gb/20Gb FlexFabric Adapters on
each server. Up to twelve uplinks with eight Flexport and four QSFP+ interfaces, without splitter cables, are available for connection
to upstream Ethernet and Fiber Channel switches. Including splitter cables up to 24 uplinks are available for connection to upstream
Ethernet and Fiber Channel. VC FlexFabric-20/40 F8 modules avoid the confusion of traditional and other converged network
solutions by eliminating the need for multiple Ethernet and Fiber Channel switches, extension modules, cables and software
licenses. Also, Virtual Connect wire-once connection management is built-in enabling server adds, moves, and replacements in
minutes instead of days or weeks.
Ports and Port Numbering
The HP Virtual Connect FlexFabric20/40 F8 Module has 4x40G QSPF+ and 8x1/10G SFP+ Ethernet or 2/4/8GB FC interfaces as
uplink ports. For the downlink ports, you have 16x10/20Gb Ethernet links towards HP BladeSystem C7000 server bays with a
single downlink to a server support up to 3 FlexNICs and 1 FlexHBA or 4 FlexNICs. A FlexHBA can be configured to transport either
FCOE or Accelerated iSCSI.
The 4xQSFP port numbers start with Q1.1 and end with Q4.4. If you decide to leverage a breakout cable you will have 4 10Gb
ports per QSFP slot so Q1.1 through Q1.4 would be available for 10Gb. If a 40Gb QSFP is placed in Q1 then Q1.1 will appear as
populated and port Q1.2-Q1.4 will be unavailable. Below is an example of how a 40Gb port would appear in OneView with the
20/40 Module.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
The 8xSFP+ port numbers start from X1 and end with X8. These ports can accommodate 1/10Gb Ethernet or 2/4/8GB FC
connections. Below you can see a variety of different connections options and how they would appear in OneView include FC,
Ethernet, and stacking between Virtual Connect Modules.
Ports X5-X6 and X7-X8 are called paired ports and each paired port can be set to Ethernet or Fiber Channel but mixing of both
traffic types in the same paired set is not allowed. An example of a configuration that would not be allowed is putting a 10Gb
Ethernet in port X7 and then trying to enable Fiber Channel on port X8. However, you could have FCOE on port X8 because it
would be running at 10Gb. You cannot mix 1Gb and 10Gb on the same paired port set as well. They are paired in terms of internal
clocking source but there is no performance penalty by using paired ports comparing with individual ports X1-X4.
Ports X9 and X10 are dedicated internal horizontal cross-connects which provide 2x20Gb connectivity.
FlexFabric 20/40 F8 with OneView Key Integrations and Best Practices with ACI
Understanding LLDP is instrumental in making FlexFabric 20/40 F8 module with OneView a working solution with ACI since LLDP
provides needed information from both uplinks and downlinks to integrate ACI and the VMM elements. However the focus for
OneView is a static configuration with ACI and not an integrated configuration with ACI and VMM. The topology leveraged for this is
depicted in Figure 1:
Note: OneView 1.20 does not support VMM integration into ACI as LLDP downlink information is not available.
Since Virtual Connect passes VLAN information it is important to extend and configure any VLANS that will be used in regards to
ACI encapsulation.
The Virtual Connect Module can operate in three modes: Private Network, Tunnel Mode, or Mapped Mode). Details and
information providing these modes of operation can be located in the Virtual Connect Cookbook. In regards to ACI and Virtual
Connect we will focus on two modes, Tunnel Mode and Mapped Mode and the caveats and restrictions they have regarding ACI.
Tunnel Mode: Tunnel Mode allows packets with VLANs tagged to pass through Virtual Connect without modification but more
importantly for large scale designs thinking about the number of VLANs to extend to each VC Domain is and after thought versus
the competition as they may need to extend all 4096 vlans.
Mapped Mode: In Mapped Mode all VLAN Tags are examined. The big limitation is that 162 virtual networks are allowed to be
extend at the server profile layer which is not the case of Tunnel Mode. This mode also requires all of the VLAN that would be
required by ACI to be extended down to VC which is not the case in Tunnel Mode.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
ACI Static Endpoint definition with encapsulating VLAN-55 for one VM.
A Network Set was created in OneView to carry the management encapsulating VLAN 60 and the VM VLAN-55. Notice that VLAN
60 is set to untagged which translated into an 802.1p vlan-60 interface definition in ACI.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
ACI configuration for connection going into Virtual Connect FlexFabric 20/40 with encapsulating vlan-60 and 802.1p Tag.
In OneView 1.20 running VC 4.31 the only LLDP information exposed is the exchange between VC and the ACI leaf node shown
below. Currently OneView 1.20 does not support LLDP on the downlinks facing the servers.
FlexFabric 20/40 F8 with Virtual Connect Manager Key Integrations and Best Practices with
ACI
In order to make the FlexFabric 20/40 F8 module with Virtual Connect Manager 4.45 a working solution with ACI, an
understanding of LLDP was key as it is important since LLDP from an uplink perspective and downlink perspective provides
needed information to take advantage of full advantage of ACI and the VMM integration elements. The topology leveraged for
this is depicted in Figure 3:
Since Virtual Connect passes VLAN information it is important to extend and configure any VLANS that will be used in regards to
ACI encapsulation.
The Virtual Connect Module can operate in a couple of modes included (Private Network, Tunneling, or Mapped Mode). Details
and information providing these modes of operation can be located in the Virtual Connect Cookbook. In regards to ACI and Virtual
Connect we will focus on two modes, Tunnel Mode and Mapped Mode and the caveats and restrictions they may have regarding
ACI.
Tunnel Mode: Tunnel Mode allows packets with VLANs tagged to pass through Virtual Connect without modification but more
importantly for large scale designs thinking about the number of VLANs to extend to each VC Domain is and after thought versus
the competition as they may need to extend all 4096 vlans.
Mapped Mode: In Mapped Mode all VLAN Tags are examined. The big limitation is that 162 virtual networks are allowed to be
extend at the server profile layer which is not the case of Tunnel Mode. This mode also requires all of the VLAN that would be
required by ACI to be extended down to VC which is not the case in Tunnel Mode.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Virtual Connect allows for an optional domain IP address which creates a consistent IP address that is independent of the
interconnect module on which it is running. If set, this IP address must be unique within the network and must be different than
the IP address of the module itself. If this IP address is not set, the VC Manager can still be reached through the IP address of the
host VC-Enet module.
To use an optional domain IP address, select the Use Domain IP address check box, and then enter the IP Address, Subnet Mask,
and Default Gateway.
NOTE: Even if a domain IP address is provided, the normal IP address assigned to the interconnect bay can still be used.
In regards to ACI the key information that is leveraged is LLDP and the Management IP Address of VC. If a failover occurs to ensure a
timely transition of interfaces and prevent a loss of connectivity it is recommend to set the VC Domain IP address.
Virtual Connect CLI tagged LLDP enabled is set equal to true in the testing below. The default option is false. This option can only
be set in Virtual Connect Manager 4.45 in CLI mode at this time.
Below are the Virtual Connect uplink set definitions for ACI leaf nodes.
Encapsulating VLAN 65 was used as a management VLAN and Encapsulating VLAN 63 was extending into the Virtual Machine
world for VMM consumption. VLAN 65 was set native for management, PXE etc.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
In ACI the definitions for the links connecting into VC have been defined as 802.1P Tag with encapsulating vlan-65. This also
depicts how and individual interface is mapped which is different than how a VPC is mapped out.
In ACI the definitions for the links connecting into VC have been defined as 802.1P Tag with encapsulating vlan-65. This also
depicts how a VPC mapping is handled versus the mapping above.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
The VMM domain was created and the VCenter was populated with a DVS integrated into ACI for consumption of the VMWare
Hosts and VMs sitting VC.
The VMM Integration with DVS and LLDP Information learned from Virtual Connect downlink ports shows the dynamic learning in
ACI which is illustrated below.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
The HP B22 is a monitored entity in HP OneView so no configuration was required to make this work in an ACI fabric.
This is the CLI verification of the HP B22 appearing on the ACI leaf node.
The static mapping for the interfaces going over to the HP B22 from the ACI leaf have been defined and to keep it simple only
VLAN 64 was extended and marked as untagged.
The diagram is showing what is called a single-homed Fabric Extender (FEX) topology.
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Example: Establishes external route for VMM with port connect to leaf node.
<!-- api/policymgr/mo/.xml -->
<polUni>
<fvTenant name="mgmt">
<l3extOut name="vmm">
<l3extInstP name="vmmMgmt">
<l3extSubnet ip="10.16.0.0/16" />
<fvRsCons tnVzBrCPName="default" />
</l3extInstP>
<l3extLNodeP name="bLeaf">
<l3extRsNodeL3OutAtt tDn="topology/pod-1/node-101" rtrId="1.1.1.1">
<ipRouteP ip="0.0.0.0/0">
<ipNexthopP nhAddr="10.16.200.1" />
</ipRouteP>
</l3extRsNodeL3OutAtt>
<l3extLIfP name="portProfile">
<l3extRsPathL3OutAtt tDn="topology/pod-1/paths-101/pathep-[eth1/1]"
ifInstT="l3-port" addr="10.16.200.2/30" />
</l3extLIfP>
</l3extLNodeP>
<l3extRsEctx tnFvCtxName="inb" />
</l3extOut>
</fvTenant>
</polUni>
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
ACI References
During the process of building out ACI and testing various HP solutions a number of key reference documents were leveraged.
Below is a list of some of the most useful documents to help in regards to understanding ACI.
Cisco Application Centric Infrastructure Fundamentals
Cisco APIC Getting Started Guide
Operating Cisco Application Infrastructure
Troubleshooting Cisco Application Centric Infrastructure
Demystifying Cisco ACI for HP Servers with OneView, Virtual Connect and B22 Modules
Conclusion
This paper is a technical understanding document that should allow the reader to successfully interpret and demystify an ACI
fabric that is leveraged with HP solutions.