IBM Cloud for VMware Solutions

NSX Edge Services Gateway

Solution Architecture

Date: 2017-03-29
Version: 1.0

 Copyright IBM Corporation 2017 Page 1 of 16

Table of Contents
1 Introduction ............................................................................................................................... 4
1.1 About NSX Edge Services Gateway .................................................................................. 4
2 Design ...................................................................................................................................... 5
2.1 Overview ............................................................................................................................. 5
2.1.1 Internal Architecture ................................................................................................... 5
2.1.2 Dedicated Architecture............................................................................................... 5
2.1.3 IBM Cloud IP address space vs BYOIP..................................................................... 5
2.2 Cloud Networking Services ................................................................................................ 6
2.2.1 IBM Management Edge ............................................................................................. 7
2.2.2 IBM Workload Edge ................................................................................................. 11
2.3 Multi-Site ........................................................................................................................... 14
2.3.1 Overview .................................................................................................................. 14
2.3.2 NSX Cross vCenter .................................................................................................. 14
2.3.3 Multi-Site example ................................................................................................... 15
3 References ............................................................................................................................. 16

 Copyright IBM Corporation 2017 Page 2 of 16

Summary of Changes
This section records the history of significant changes to this document. Only the most significant changes
are described here.
Version Date Author Description of Change
1.0 2017-03-29 IBM Cloud Initial Release

 Copyright IBM Corporation 2017 Page 3 of 16

1 Introduction
The purpose of this document is to define and describe the solution architecture for the VMware NSX Edge
Services Gateway (ESG) solution deployed on the IBM Cloud. Specifically, it will detail the baseline
components of the management edge and workload edge solution as well as the configuration of each
component in the design. This solution is considered an extension of VMware Cloud Foundation and
vCenter Server on IBM Cloud offerings that are currently available on the IBM Cloud. The configuration
of the VMware Cloud Foundation (VCF) or vCenter Server (VCS) on IBM Cloud are not covered in this
document. Instead it is highly recommended to review and understand the aforementioned architectures
located on the IBM Architecture Center before reading this document.

Figure 1 Cloud Networking Services on IBM Cloud

1.1 About NSX Edge Services Gateway

The NSX Edge gateway connects isolated, stub networks to shared (uplink) networks by providing
common gateway services such as DHCP, VPN, NAT, dynamic routing, and load balancing. Common
deployments of NSX Edge include DMZ, VPN Extranets, and multi-tenant Cloud environments where the
NSX Edge creates virtual boundaries for each tenant, workload, or management component. The following
features are available within the NSX Edge Service Gateway.
 NSX Edge Service Routing provides the necessary forwarding information between layer 2
broadcast domains, thereby allowing you to decrease layer 2 broadcast domains and improve
network efficiency and scale. NSX extends this intelligence to where the workloads reside for
doing East-West routing. This allows more direct virtual machine to virtual machine
communication without the costly or timely need to extend hops. At the same time, NSX also
provides North-South connectivity, thereby enabling tenants to access public networks.
 Firewall - Supported rules include IP 5-tuple configuration with IP and port ranges for stateful
inspection for all protocols.
 Network Address Translation - Separate controls for Source and Destination IP addresses, as well
as port translation
 Dynamic Host Configuration Protocol (DHCP) - Configuration of IP pools, gateways, DNS
servers, and search domains.
 Site-to-Site Virtual Private Network (VPN) - Uses standardized IPsec protocol settings to
interoperate with all major VPN vendors.
 L2VPN - Provides the ability to stretch L2 networks across L3 topologies.
 SS VPN-Plus - SSL VPN-Plus enables remote users to connect securely to private networks
behind a NSX Edge gateway.
 Load Balancing - Simple and dynamically configurable virtual IP addresses and server groups.
 High Availability - High availability ensures an active NSX Edge on the network in case the
primary NSX Edge virtual machine is unavailable.

 Copyright IBM Corporation 2017 Page 4 of 16

2 Design
2.1 Overview
2.1.1 Internal Architecture
The NSX Edge Gateway Services on IBM Cloud solution provides VMware technology deployed in this
document’s prescribed architecture within IBM Cloud datacenter across the globe. Note that this document
describes two solution architectures relating to the NSX Edge Services Gateway. The first of these
architectures is known as the internal architecture and specifies the deployment of the necessary NSX Edge
components in a resource pool in either a VMware Cloud Foundation converged cluster or vCenter Server
cluster. The second architecture, known as the dedicated architecture, deploys the necessary NSX Edge
components in a separate two-node cluster dedicated for the use of the NSX Edge.

Figure 2 Cloud Networking Services

2.1.2 Dedicated Architecture

The dedicated architecture design dedicates a vSphere cluster for the purposes of providing critical
interaction with the physical network infrastructure. The dedicated design has following characteristics and
functions:
 Provide on-ramp and off-ramp connectivity to physical networks (i.e., north-south L3 routing on
NSX Edge virtual appliances).
 Allow communication with physical devices connected to VLANs in the physical networks
through NSX L2 bridging. Hosts the Control VM for DLR routing.
 May have centralized logical or physical services (e.g., firewall, load balancers, VPN monitoring
components, log insight VMs).
 NSX Controllers can be hosted in an edge cluster, when a dedicated vCenter is used to manage the
compute and edge resources.
 Edge cluster resources have anti-affinity requirement to protect the active standby configuration or
to maintain the bandwidth availability during failure.

2.1.3 IBM Cloud IP address space vs BYOIP

RFC1918 (private IP range) specifically reserves the use of certain network ranges for organization internal
use and cannot be used on the internet. IBM Cloud physical network infrastructure utilizes a specific
RFC1918 private address space across all of its worldwide locations (10.x.x.x/8). These IP address ranges
do not overlap across customer accounts or within an IBM Cloud customer account. Within a customer
account, any IBM Cloud allocated private IP address space can, with VLAN Spanning enabled, route to

 Copyright IBM Corporation 2017 Page 5 of 16

any other IBM Cloud private IP address range in any IBM Cloud data center. While this makes it very
simple to setup worldwide connected infrastructure within a customer account, because the IP address
space is fixed, it can be problematic when a customer wishes to extend their data center into IBM cloud via
routing if they are using the same private address space as IBM cloud. The solution is to utilize NSX to
create an “overlay” topology on top of VCF/VCS, isolating the customer IP address space from interacting
with IBM Cloud assigned private IP address space. NSX is capable of providing a L2 VPN would serve to
“span” internal IP address space within the tunnel across external possibly overlapping IP address spaces.

Figure 3 L2 vpn tunnel - IP isolation

2.2 Cloud Networking Services

The Cloud Networking Services on IBM Cloud consists of two pair of VMware of VMware NSX Edge
Services Gateways (ESG) for communication between the IBM Cloud and either the public internet or
customer on-premises network via VPN. These ESGs are segregated to support internal IBM Cloud
management function and egress, ingress of customer related network traffic.

 Copyright IBM Corporation 2017 Page 6 of 16

 Figure 4 Cloud Networking Services Diagram (VCF Shown)

Figure 4 is a simplified network diagram which shows the management and workload ESGs. It also shows
a NSX Distributed Logical Router (DLR) and workload VXLAN. These components are intended to be an
initial landing point for customer workloads without requiring the specific knowledge to set them up within
NSX. This can facilitate a decreased time to value when deploying VCS/VCF for demonstration purposes
or customer POCs. A DLR is typically employed to route inter-VCF/VCS traffic which is typically termed
“east-west” traffic between separate layer 2 networks within the instance. This is in contrast to a ESG
which functions to facilitate “north-south” network traffic traversing in and out of the VCS/VCF instance.
While a single ESG could suffice for both management and customer traffic, the separation of management
and customer traffic is a design decision made to primarily to keep from accidental misconfiguration of the
management edge. Note that misconfiguration or disabling the management ESG does not keep the
VCF/VCS instance from functioning, but would disable all portal management functions.

2.2.1 IBM Management Edge

The first NSX ESG discussed in this architecture is the IBM Management Edge. The management ESG is a
dedicated NSX edge cluster for IBM Cloud management network traffic only. It is not intended for traffic
traversal of any component not deployed and managed by the VCF/VCS automation.
The purpose of this ESG is to provide a communication path between IBM VCF deployed virtual machines
residing within VMware Cloud Foundation and/or vCenter Server instances and the IBM Automation
infrastructure in the IBM Cloud as shown in Figure 5 IBM Management VM Communication via
Management Edge.

 Copyright IBM Corporation 2017 Page 7 of 16

 Figure 5 IBM Management VM Communication via Management Edge
As a result of the light communication between the IBM Management VMs and IBM Automation, the NSX
ESGs are sized in a “Large” configuration in an Active/Passive HA Pair and deployed on the management
resource pool of the VCF converged cluster or the vCenter Server cluster. A summary of the IBM
Management Edge deployment is shown in Table 1 IBM Management NSX ESG Specifications.
vCPU Memory Disk Size Storage Location
IBM Management 2 1 GB 1 GB vSAN Datastore (VCF)
NSX ESG 1 Shared Attached Storage for Management
(VCS)
IBM Management 2 1 GB 1 GB vSAN Datastore (VCF)
NSX ESG 2 Shared Attached Storage for Management
(VCS)
Table 1 IBM Management NSX ESG Specifications

2.2.1.1 Management Services

As of this writing, only outbound access is required to the following services:
 Cloud Driver VM. This is a VCF deployed image that interfaces with the SDDC manager VM
for life cycle management of the VCF instance. It periodically polls for requests against an
instance of IBM Cloudant™ setup for the management of VCF.
 Zerto™ Management VM, if installed. Zerto requires outbound access to the internet for
licensing activation.

 Copyright IBM Corporation 2017 Page 8 of 16

 2.2.1.2 Edge interfaces
Configuration of the ESG interfaces define what L2 networks the ESG has access to. For the
current purposes of VCF/VCS life cycle management, it is currently required that specific VMs
placed on the management VLAN, be allowed to traverse to the public VLAN. As such, the
following interfaces are defined on deployment:
Interface Interface Connected To Description
Type
Public Uplink Uplink SDDC- Public internet facing interface
DportGroup-
External
Private Uplink Uplink SDDC- Internal private network facing
DportGroup- interface
Mgmt
Internal Internal Workload HA Internal interface used for ESG HA
VXLAN pair heartbeat – portgroup on SDDC-
Dswitch-Private
Table 2 NSX ESG Interface Configuration

2.2.1.3 Subnets
The following subnets will be utilized for the purposes of the Management ESG:
Interface Interface IP v4 subnet Range Description
Type type
Public Uplink IBM Cloud /30 – renders Public internet facing interface
Uplink portable public one assignable
IP address.
Private Uplink IBM Cloud /26 – renders Internal private network facing
Uplink portable 61 assignable interface
private IP addresses
(existing
management)
Internal Internal Link local 169.254.0.0/16 Internal interface used for ESG
HA pair communication
Table 3 NSX ESX IP Configuration

2.2.1.4 NAT definitions

Network Address Translation is employed on the Management ESG for the means of allowing
network traffic to traverse between one IP address space and another. This is typically done to
conserve internet routable IPs or to conceal internal IPs from public ones for security reasons.
NAT can also be used to allow for TCP and UDP port redirection. As of this writing, management
traffic will always be initiated from inside the VCF/VCS instance. As such, only a SNAT (source
NAT) need be defined on the Management ESG. An individual SNAT will be created for each
internal VM hosting a service which needs to egress from the VCF instance.
Applied on Interface Source IP range Translated Source IP
Public Uplink Individual IP addresses IBM Cloud portable public IP
on the Management
Portable /26
Table 4 NSX ESG NAT Configuration

 Copyright IBM Corporation 2017 Page 9 of 16

 2.2.1.5 Routing
Since services within VMs required to traverse through the Management ESG may also have need
to get to IBM Cloud services within the customer IBM Cloud private network, the following is the
configuration required to achieve this communication.
While it is difficult to predict which destination IP range is needed as a destination for internet
facing connections, any service which will be deployed by and managed by IBM Cloud will point
to the Management ESG as its default gateway. A static route will be required to force traffic
across the IBM Cloud BCR for the services required external network connections.
For any service that will be using the management ESG to traverse out of a VCF/VCS instance, it
is recommended that the following be configured.
 Default Gateway is Management ESG
 Static route is required for internal IBM Cloud destinations.
If there is a need for the service or VM to access the customer ESG, then static routes will need to
be maintained within the individual service or VM as well as pointing to the customer ESG.
No automatic routing protocols are configured for the Management ESG at this time.

2.2.1.6 VXLAN definitions

The Management HA pair requires a network for the connection of the “internal” interfaces. This
could be done on an existing vSwitch, port group or VXLAN. For this design, a dedicated
VXLAN will be created for the HA heartbeat communication of the Management ESG HA pair.
VXLAN Name Transport Type
Zone
Mgmt HA transport-1 global
Table 5 NSX ESG VXLAN Definitions

2.2.1.7 Firewall rules

By default, the Management ESG is configured to drop (Deny) all traffic. “Deny” drops all traffic
with no response when that traffic is not allowed to traverse the firewall by any previous (higher in
the order) rule or rule set. Automatic rule generation will be selected to allow for control traffic to
the ESG pair.
The following firewall rules are set (in addition to the automatically generated rules):
Service Source Destination Protocol Action
VCF/VCS management Cloud driver VM Any Cloudant port Allow
(TBD)
Zerto Zerto Any Licensing port Allow
Management VM (TBD)
Any Any Any Any Deny
Table 6 NSX ESG Firewall Configuration

2.2.1.8 ESG settings

By default, logging is enabled on all new NSX Edge appliances. The default logging level is
NOTICE.

 Copyright IBM Corporation 2017 Page 10 of 16

2.2.2 IBM Workload Edge
As described in section 2.2, the Workload Edge (ESG) described in this design is part of a simple topology
intended for workload network communication. Prior versions of VCF and VCS deploy networks and
network subnets, none of which are intended for workload traffic. This section now serves to describe
the design intent of where to attach workloads to a network within a VCF/VCS instance. This is a starting
point for attaching on-premises networks and IP spaces to a particular VCF/VCS instance and is the basis
for a true hybrid cloud architecture.

Figure 6 Example Customer network flow diagram

As Figure 6 Example Customer network flow diagram shows, the Workload ESG is attached to both the
public and private IBM Cloud networks. This allows for workload access to and from internet facing
traffic, but also allows for a site-to-site VPN to be created from either public or private IBM Cloud
networks. This is useful especially in a POC process as it allows for drastically decreased time to value
with regards to connecting to on-premises networks since it can take months to bring up a dedicated WAN
due to particular customer security requirements. However, once a dedicated link is in place, the VPN can
be “flipped” over to traverse that link without affecting the overlay network inside the VPN tunnel or
within the VCF/VCS instance. Once this is done, the public interface for the workload ESG can simply be
removed if it is desired from a security perspective.
The topology portrayed in the diagram above consists of the following NSX components:
 NSX Edge appliance (ESG)
 Distributed Logical Router (DLR)
 VXLAN (L2 over L3)

2.2.2.1 Edge interfaces

As explained in the Management ESG section, configuration of the ESG interfaces define what
adjacent L2 networks the ESG has access to. Part of the design intent of the workload topology is
to achieve a SDN overlay to isolate workloads from the underlying IBM Cloud address space.

 Copyright IBM Corporation 2017 Page 11 of 16

 This is the basis for allowing a “Bring Your Own IP” (BYOIP) design to be achieved. As such, the
following interfaces need to be defined:
Interface Interface Type Connected Description
To
Public Uplink Uplink SDDC- Public internet facing interface
DportGroup-
External
Private Uplink Uplink SDDC- Internal private network facing interface
DportGroup-
Mgmt
Transit Uplink Uplink Workload- Transit VXLAN between the Workload
Transit ESG and the Workload DLR

Internal Internal Workload Internal interface used for ESG HA pair

HA VXLAN heartbeat

Table 7 Workload ESG Interfaces

In this design, a DLR is employed to allow for potential “east-west” routing between local
workload connected L2 networks. As this topology is intended to be a simple example, only one
L2 network intended for workloads is described. Adding additional security zones can be achieved
by simply adding additional VXLANs attached to new interfaces on the DLR. The following are
the DLR interfaces to be configured:
Interface Interface Connected Description
Type To
Transit Uplink Workload- Transit VXLAN between the Workload ESG
Uplink Transit and the Workload DLR

Workload Uplink Workload VXLAN for Workload connections

Uplink

Internal Internal Workload HA Internal interface used for ESG HA pair

VXLAN heartbeat

Table 8 DLR Interfaces

2.2.2.2 Subnets
The following are example subnets to be utilized for the purposes of the Workload ESG:
Interface Interface IP v4 subnet Range Description
Type type
Public Uplink Uplink IBM Cloud /30 – renders one Public internet
(ESG) portable public assignable IP facing interface.
address. (Customer can
order additional IPs
separately)
Private Uplink Uplink IBM Cloud /26 – renders 61 Internal private
(ESG) portable private assignable IP network facing
(existing addresses interface
management)

 Copyright IBM Corporation 2017 Page 12 of 16

 Interface Interface IP v4 subnet Range Description
Type type
Internal (ESG Internal Link local 169.254.0.0/16 Internal interface
and DLR) used for ESG HA
pair
communication
Transit Uplink Uplink Assigned by TBD transit network
(ESG and DLR) Customer connection for
ESG to DLR
Workload Uplink Assigned by TBD Workload subnet
(DLR) customer
Table 9 DLR and Workload ESG IP Configuration

2.2.2.3 NAT definitions

Network Address Translation is employed on the Workload ESG for the means of allowing
network traffic to traverse between one IP address space and another. For the workload ESG, NAT
is required not only to allow for communication to internet destinations, but also to communicate
to any IBM Cloud sourced IP ranges. For this design, workload traffic will be allowed to exit to
the internet, but not to the management or any IBM Cloud networks. As such, only a SNAT
(source NAT) need be defined on the Workload ESG. Note that the entire workload portable
subnet is configured to traverse thru the SNAT.
Note that while it is possible to use NAT to allow for workload communication across multiple
instances of VCF/VCS, this becomes impractical when many workloads need to be connected
across instances. See the Multi-Site section for examples of using advanced NSX capabilities to
create an L2 overly transit network across VCF/VCS instances.
Applied on Source IP range Translated Source IP NAT Enabled or
Interface Disabled
Public Customer defined IBM Cloud portable public IP Customer defined
Uplink default disabled
(Workload
ESG)
Table 10 Workload ESG NAT Rules

2.2.2.4 Routing
Within this design, the only requirement for Workloads traversing the DLR to the Workload ESG
is to access the internet. The Workload ESG needs to understand the path to the workload
VXLAN and any future workload VXLAN/subnets created behind the DLR. While this could be
achieved via static routes on the ESG, the intent of the workload topology is that of a
demonstrated best practice design. As such, OSPF will be configured between the Workload ESG
and the downstream DLR. See https://pubs.vmware.com/NSX-
6/index.jsp?topic=%2Fcom.vmware.nsx.admin.doc%2FGUID-6E985577-3629-42FE-AC22-
C4B56EFA8C9B.html
For configuration procedure.
Area OSPF OSPF interface IP OSPF authentication
type
51 stub Assign an IP for each the DLR and ESG on the None
transit RFC1918 network
Table 11 Dynamic Routing

 Copyright IBM Corporation 2017 Page 13 of 16

 2.2.2.5 Firewall rules
By default, the Workload ESG is configured to drop (Deny) all traffic. “Deny” drops all traffic
with no response when that traffic is not allowed to traverse the firewall by any previous (higher in
the order) rule or rule set. Automatic rule generation will be selected to allow for control traffic to
the ESG pair.
The following firewall rules are set (in addition to the automatically generated rules):
Service Source Destination Protocol Action
Workloads Workload Any Any Allow
subnet
Any Any Any Any Deny
Table 12 Workload ESG Firewall Rules

2.2.2.6 VXLAN definitions

The Workload topology ESG and DLR HA pairs require L2 segments (VXLAN) for the
connection of the “internal” interfaces, data transit between the two, and finally, for workloads.
VXLAN Name VCF/VCS Transport Type
Zone
Workload HA transport-1 global
Workload Transit transport-1 global
Workload transport-1 global
Table 13 Workload ESG Internal Interfaces

2.2.2.7 ESG DLR Settings

By default, logging is enabled on all new NSX Edge appliances. The default logging level is
NOTICE.

2.3 Multi-Site
2.3.1 Overview
One key differentiator between IBM Cloud and other cloud offerings is the ability to provision dedicated
compute capability across the globe and have that on demand infrastructure automatically network
connected within the customer’s private IBM Cloud account. The software defined network capabilities of
Cloud Foundation in concert with IBM Cloud provide a granular defined global infrastructure. Capable of
being built within days within just a few mouse clicks. The following describes a multi-site architecture
example of what can be achieved with the out of the box capability of Cloud Foundation.

2.3.2 NSX Cross vCenter

NSX Cross vCenter capability allows for linking in a primary, secondary relationship of up to 9 NSX
managers. (1 primary and 8 secondary NSX managers) While it is not required to have vCenter servers in
an Enhanced Linked Mode relationship for NSX Cross vCenter to function, it helps a great deal in the
following ways:
 Simplified primary, secondary relationship creation using SSO credentials.
 Cloud Foundation automation configures DNS name resolution for all sites linked together
 Single pane of glass management across all sites for both NSX and normal vCenter functions.

 Copyright IBM Corporation 2017 Page 14 of 16

2.3.3 Multi-Site example
The following example simply adds a NSX universal transport zone to the basic Management and
Workload topologies discussed in previous sections. This universal transport zone spans two IBM Cloud
data centers or PODs within a datacenter. Once that is added, multiple VXLANs are added and a Universal
Distributed Router that spans this new VXLANs and has uplinks to the workload ESGs in both sites must
be configured. This causes VMs in the local site to traverse to its local ESG. Of course, for inbound traffic,
a global load balancer is required. That can be achieved with IBM Cloud’s global load balancing offerings.
The following is an example would require Enterprise edition of NSX which is part of Cloud Foundation.

Figure 7 Multi-site topology

 Copyright IBM Corporation 2017 Page 15 of 16

3 References
NSX 6 Documentation
https://pubs.vmware.com/NSX-6/index.jsp#com.vmware.nsx.install.doc/GUID-D8578F6E-A40C-493A-
9B43-877C2B75ED52.html
NSX edition features (Standard, Advanced, Enterprise)
http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-nsx-
datasheet.pdf
vSphere™ 6 Configuration Maximums
https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf
IBM Cloud global load balancing
http://knowledgelayer.softlayer.com/articles/global-load-balancing-options

 Copyright IBM Corporation 2017 Page 16 of 16