VMware Validated Design

for Software-Defined Data Center 4.1

Logical Component Architecture

In a dual-region Software-Defined Data Center, the two Platform Service Controllers and two vCenter Server instances are deployed in each region. In a dual-region Software-Defined Data Center, a vRealize Log Insight cluster is deployed in each region.
In a dual-region Software-Defined Data Center, two primary NSX Manager instances are deployed in Region A. The design integrates solutions for compute, storage, network, cloud operations, and cloud management.
Core vSphere Management This includes a vCenter Server for the management pod and a vCenter Server for the shared edge and compute pods.
Each vCenter Server instance is connected to load-balanced pair of Platform Services Controllers using an NSX Edge Services Gateway. NSX One for the management pod and one for the shared edge and compute pods, along with associated NSX Universal Controller Clusters.
In Region B, secondary NSX Manager instances automatically import the configurations of the NSX Universal Controller Clusters from Region A.
vRealize Operations vRealize Operations analytics clusters monitor and perform diagnostics across the Software-Defined Data Center
by using a series of remote collectors and solution management packs.
vRealize Log Insight Each cluster consists of three nodes, enabling continued availability and increased log ingestion rates.
vRealize Log Insight collects and analyzes log data across the stack using the syslog protocol and the ingestion API.
To enable enhanced linked mode, the design joins the Platform Services Controller instances into a unified Single Sign-On domain. vRealize Log Insight also integrates with vRealize Operations Manager to facilitate root cause analysis.

Region A Region B Region A Region B Region A Region B

Management / Management /
Common vCenter Single Sign-On Domain Management /
vRealize
Management /
Compute
vRealize
Compute
vRealize
Automation
Management Stack Management Stack Compute
Automation
Compute Automation
(Ring Topology) vCenter Servers vCenter Servers
vCenter Servers vCenter Servers Proxy Agents
vCenter Server vCenter Server
Appliance Appliance
NSX vRealize Log Insight Cluster vRealize Log Insight Cluster NSX
NSX Remote Collectors Remote Collectors NSX
Event
Region A
Region B Management Stack Management Stack Master Worker Worker
Forwarding
via Ingestion Master Worker Worker
Clctr Clctr Clctr Clctr
NSX Manager Pairing Shared Node Node Node API Node Node Node
NSX Manager NSX Manager Storage
Node Node Node Node Shared
Storage vSAN vSAN
(Primary) (Secondary) Systems Systems

vRealize

Platform Services Platform Services SDPlatform Services Platform Services NSX Edge Management Stack Import of Management Stack NSX Edge Analytics Cluster
vRealize
Operations
Any Supported NFS Any Supported NFS Operations
Remote
Controller Controller Controller Controller Services Gateways NSX Universal NSX Controller Configuration Services Gateways
Collectors
Appliance Appliance Appliance Appliance from Primary NSX Manager Master Replica Data
(N/S Routing) Controller Cluster (N/S Routing) Node Node Node

Primary Storage Log Archives Primary Storage Log Archives

NSX Edge Services Compute Stack Compute Stack NSX Edge Services Region A Management Pod Region B Management Pod Region A Management Pod Region B Management Pod
NSX Edge Services NSX Edge Services Gateway w/ HA vCenter Server vCenter Server Gateway w/ HA
Gateway w/ HA Gateway w/ HA (One-Arm Load Balancer) Appliance Appliance (One-Arm Load Balancer)
(Load Balancer) (Load Balancer)

Compute Stack NSX Manager Pairing

Compute Stack The design establishes a Cloud Management Platform with vRealize Automation to provide a The design implements a single vRealize Automation tenant. Business groups can be created to fit your needs.
NSX Manager NSX Manager vRealize Automation, vRealize Orchestrator service catalog and self-service portal to deploy, update, and manage the workloads. Its embedded instance of vRealize Automation Within each business group the tenant administrators are able to manage users and groups, apply tenant-specific
(Primary) (Secondary) and vRealize Business for Cloud vRealize Orchestrator provides a repository of extensible workflows and integrations. vRealize Business for Cloud Business Groups & Reservations branding, enable notifications, configure business policies, and manage the service catalog.
Management Stack Compute Stack Management Stack Compute Stack provides visibility into the financial aspects of the cloud infrastructure, allowing cost to be tracked and optimized. The IT Automating IT Use Case documenation provides implementation steps for a set of scenarios.
vCenter Server vCenter Server vCenter Server vCenter Server
Appliance Appliance Appliance Appliance
Business Business
Region A Management Pod Region B Management Pod Tenant
Admin Group Group
Tenant
Admin
vRealize vRealize vRealize vRealize
Sign In

Manager Manager
Automation Business Automation Business
https://my.sddc.local/vcac/org/company

vSphere Update vSphere Update NSX Edge Compute Stack NSX Import of Compute Stack NSX Edge IWS IMS DEM IAS SQL BUS IAS BUC
VRA
Manager Download Manager Download Services Gateways Universal Controller NSX Controller Configuration Services Gateways Edge Business Group Business Group Business Group Business Group Edge
Service Service (N/S Routing) Cluster
from Primary NSX Manager
(N/S Routing) Reservation Reservation Reservation Reservation Reservation Reservation
VRA IWS IMS DEM IAS BUC IAS Fabric Fabric
Region A Fabric Group Region B Fabric Group
Admin Admin

IaaS Region A Data Center Infrastructure Fabric Region B Data Center Infrastructure Fabric IaaS
Region A Management Pod Region B Management Pod Region A Shared Edge and Compute Pod Region B Shared Edge and Compute Pod Admin Admin
Shared Edge/Compute Pod Additional Compute Pod(s) Shared Edge/Compute Pod Additional Compute Pod(s)
(Edge Resource Pool) (Edge Resource Pool)
Region A Management Pod Region B Management Pod

Core and Pod Architecture Pod and Clusters

Network Transport Pods The design uses standardized building blocks called pods.
Host Connectivity
Universal Logical Switch Universal Logical Switch

Application Virtual Networks

Workload Virtual Networks

Below is the standard architecture based on two pod functions. L2 L2
UDLR External UDLR & DLR

for SDDC Solutions

APP APP APP APP APP APP
L3 Networks APP APP APP
L3

OS OS OS OS OS OS OS OS OS
Universal Compute Transport Zone Management Pod ESXi Host Shared Edge and Compute Pod ESXi Host
Spine Spine Spine Universal Logical Switch Universal Logical Switch
Universal Management Transport Zone Local Compute Transport Zone

North/South
L2
10 GigE nic0 nic1 10 GigE 10 GigE nic0 nic1 10 GigE L2
UDLR UDLR & DLR

Routing
L3 L3 L3 L3
Spine Spine
Management Distributed Switch Compute Distributed Switch L3 L3
APP APP APP APP APP APP APP APP APP APP
Routed Uplinks (ECMP) vDS MTU 9000 vDS MTU 9000
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP OS OS OS OS OS OS OS OS OS OS
L3 L3
Layer 3 ToR Switch
Management Distributed Switch Compute Distributed Switch
VDP vCenter vCenter L2 L2

Core Platform
(Management)
ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi L3
(Management) (Compute)

Services
VMK MTU VMK MTU
VLAN NFS VLAN NFS SRM PSC PSC
40 GigE 40 GigE L2 9000 9000 Edge
(Management) (Compute) (Management) NSX Controllers N/S NSX EDGE N/S NSX EDGE NSX Controllers
Leaf Leaf Leaf Leaf Leaf Leaf Resource
Leaf Leaf (Management) (Management) (Compute) (Compute)
VMK MTU VMK MTU Pool
VLAN Management VLAN Management VR NSX Manager NSX Manager
L3 L3 L3 Any Supported Storage Any Supported Storage 9000 9000
(Management) (Compute) (Management)
vSAN Recommended vSAN Recommended VLAN 1611 VLAN 1612 VLAN 1613 VLAN 1614 VMK MTU VMK MTU
UDLR UDLR & DLR
IGMP IGMP IGMP IGMP IGMP IGMP
L2 IGMP IGMP VLAN vMotion 9000 VLAN vMotion 9000
L2 L2

Span of VLANs
plus NFS plus NFS VMK MTU VMK MTU Management Management
10 GigE 10 GigE VLAN VTEP (VXLAN) 9000 VLAN VTEP (VXLAN) 9000

Distributed
Management Pod
Shared Edge and Compute Cluster

Switches
North/South Uplink(s)

Span of VLANs
North/South Uplink(s)
(4+ Hosts) Management Cluster
& Compute Cluster n
VMK MTU VMK MTU vMotion vMotion
VLAN vSAN 9000 VLAN vSAN 9000
Minimum 4 Nodes | vSAN ReadyNodes Recommended Minimum 4 Nodes | vSAN ReadyNodes Recommended VLAN Trunk (802.1Q) vSAN vSAN
Server vSphere HA and DRS Enabled vSphere HA and DRS Enabled | Business Workload Requirements Region Dependent VXLAN VXLAN xxxx
ESXi Host VLAN External Management VLAN Uplink 01
Region Independent VXLAN VXLAN xxxx
Shared Edge and Compute Pod
Management Pod VMK MTU
& Compute Pod n VLAN vSphere Replication 9000 VLAN Uplink 02
The management pod hosts the infrastructure components used to instantiate,
Management vMotion VXLAN vSAN
manage and monitor the SDDC. This includes the core infrastructure
Shared Edge and Compute Pod Workloads running in the SDDC do not have direct access to external networks.
172.16.11.0/24 172.16.12.0/24 172.16.13.0/24 172.16.14.0/24 VLAN Uplink 01
components, such as the Platform Services Controllers, vCenter Server instances,

Edge/Compute Cluster
To access external networks, tra c is routed through distributed routing ESXi-MGMT-01 ESXi-MGMT-02 ESXi-MGMT-03

Transport Zones
(4+ Hosts) VTEPs VTEPs VTEPs VTEPs ESX-COMP-01

Management Custer
NSX Managers, NSX Controllers for the management stack, vSphere Replication, to the NSX Edge Services Gateways in the shared edge and compute pod. DGW: DGW:
Site Recovery Manager, as well as the SDDC monitoring and automation 172.16.11.253 172.16.12.253 UDLR UDLR UDLR UDLR & DLR

Universal
solutions like vRealize Operations, vRealize Log Insight and vRealize Automation. Expansions beyond the initial shared pod are simply compute pods. VLAN Uplink 02

Any Supported Any Supported

Additional Compute Pods
Storage Storage
Management Stack Compute Stack When using the recommended L3 network transport, the top-of-rack leaf switches of each rack act as the The two 10GbE NICs on each host are connected across the top-of-rack leaf switches and teamed on the vSphere Distributed Switch via an active-active configuration.
The design supports L3 or L2 network transport services. For a scalable and vendor-neutral data center network, use an L3 transport. Managed by Management Stack vCenter Server Managed by Compute Stack vCenter Server corresponding L3 interface for the associated subnets. The management pod and the shared edge and compute pod All port groups, except for the ones that carry VXLAN tra c, are configured for the 'Route based on physical NIC load' teaming algorithm.
are provided with externally accessible VLANs to access to the Internet and corporate networks. VTEP kernel ports and VXLAN tra c use the ’Route based on SRC-ID' algorithm.
Management Pod Edge/Compute Pod
All design documentation is provided for an L3 transport. Adjust the deployment and operations guidance under the context of an L2 transport. A new Consolidated Management and Workload architecture is also available in the 4.1 documentation. The vSphere Distributed Switch has a MTU of 9000 configured for Jumbo Frames along with with necessary VMkernel ports.
.

Distributed Logical Routing and Application Virtual Networks for Management, Operations and Automation Solutions Storage

Distributed Logical Routing All design documentation for is provided for an L3 transport with BGP based peering. vRealize Operations vRealize Automation Primary Storage Secondary Storage
and Application Virtual Networks A TechNote is provided for the alternative mixed-use or end-to-end use of OSPF. and vRealize Log Insight and vRealize Business for Cloud

Region A Region B
Caching
SSD PCIe NVMe Tier
Read and Write Cache NFS Storage Array NFS Storage Array
ECMP
NSX Edge
Region A Region B ECMP
NSX Edge
Region A Region B Region A Region B
ECMP ECMP Services Gateways Services Gateways
NSX Edge NSX Edge
Internet or Internet or Services Gateways Services Gateways BGP Universal Transit Network Universal Transit Network BGP
Peering Universal Logical Switch / VXLAN Segment 192.168.10.0/24 192.168.10.0/24 Universal Logical Switch / VXLAN Segment Peering
Enterprise Enterprise BGP Universal Transit Network Universal Transit Network BGP
WAN/MPLS WAN/MPLS Peering Universal Logical Switch / VXLAN Segment 192.168.10.0/24 192.168.10.0/24 Universal Logical Switch / VXLAN Segment Peering

Management Universal Distributed Logical Router

Management Universal Distributed Logical Router
Internet or Enterprise WAN/MPLS

Internet or Enterprise WAN/MPLS

Spine To Shared Edge and Compute Pod To Shared Edge and Compute Pod Spine Region Independent Application Virtual Network Region Independent Application Virtual Network
Switches Switches Region Independent Application Virtual Network Region Independent Application Virtual Network Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment Volume 1 Volume 2 Volume 1 Volume 2
To Additional Compute Pods To Additional Compute Pods
Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment Capacity
NSX Edge Services Gateway NSX Edge Services Gateway
NSX Edge Services Gateway
One-Arm Load Balancer One-Arm Load Balancer Capacity Export for Export for Export for Export for Export for Export for
NSX Edge Services Gateway
192.168.11.0/24 192.168.11.0/24
One-Arm Load Balancer One-Arm Load Balancer Tier Content Library Log Archives Backups Content Library Log Archives Backups
L3 L3 192.168.11.0/24 192.168.11.0/24
Top-of-Rack Top-of-Rack and Templates and Templates
Leaf Switches 172.16.11.0/24 172.17.11.0/24 Leaf Switches
L2 L2 APP APP APP APP
VRA VRA VRA VRA
OS OS OS OS
APP APP APP APP APP APP APP
BGP Peering BGP Peering
OS OS OS OS OS OS OS All design documentation and validation is provided using vSAN as the primary storage system.
NSXM VDP PSC VC VC PSC VDP NSXM vSAN enables both all-flash and hybrid architectures. Adjust deployment and operations for supported storage systems.
Master Replica Data Master Replica Data
IWS APP APP
IWS IWS APP APP
IWS The design used NFS as a secondary storage tier.
ECMP OS OS OS OS OS OS OS OS ECMP
Node Node Node Node Node Node
NSX Edge NSX Edge OS OS OS OS NFS is used for the content library and templates consumed by vRealize Automation blueprints and for vRealize Log Insight log archives.
Services Gateways Services Gateways
Use of vSAN ReadyNodes is recommended to ensure seamless compatibility and support. NFS is also used by any vSphere APIs for Data Protection compatible solution to store backups.
BGP Universal Transit Network Universal Transit Network vRealize Operations Replicated for Disaster Recovery
Peering Universal Logical Switch / VXLAN Segment 192.168.10.0/24 192.168.10.0/24 Universal Logical Switch / VXLAN Segment
BGP
Peering
APP APP APP APP
The configuration and assembly of the components are standardized to eliminate system variability.
IMS IMS IMS IMS
OS OS OS OS

Region Dependent Application Virtual Network Region Dependent Application Virtual Network
Management Universal Distributed Logical Router
Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment APP APP APP APP
DEM DEM DEM DEM
OS OS OS OS
192.168.31.0/24
192.168.32.0/24

Reference
Region Independent Application Virtual Network Region Independent Application Virtual Network

Region Protection and Disaster Recovery

Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment APP APP APP APP
SQL BUS SQL BUS
S
OS OS OS OS
NSX Edge Services Gateway NSX Edge Services Gateway
One-Arm Load Balancer One-Arm Load Balancer APP APP APP APP
192.168.11.0/24 192.168.11.0/24 OS OS OS
OS vRealize Automation / vRealize Orchestrator Replicated for Disaster Recovery
Collector Collector vRealize Business for Cloud
Collector Collector
APP APP APP APP APP
Reserved for Disaster Recovery
Node Node Node Node Region A Region A Replicated Region B Replicated Region B Networks Notable Acronyms
OS OS OS OS OS
vRealize Operations vRealize Operations
Non-Replicated Non-Replicated
BUC vRealize Business Data Collector
Region Dependent Application Virtual Network Region Dependent Application Virtual Network vRealize Automation vRealize Automation External Transit Network(s) BUS vRealize Business Appliance
Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment Region Dependent Application Virtual Network Region Dependent Application Virtual Network vRealize Business for Cloud vRealize Business for Cloud DEM vRealize Automation Distributed Execution Manager
Universal Logical Switch / VXLAN Segment 192.168.31.0/24 192.168.32.0/24 Universal Logical Switch / VXLAN Segment
192.168.31.0/24 192.168.32.0/24 vRealize Log Insight vRealize Operations vRealize Operations vRealize Log Insight IAS vRealize Automation IaaS vSphere Proxy Agent
APP APP APP APP APP APP IMS vRealize Automation IaaS Manager Service
OS OS OS IWS vRealize Automation IaaS Web Server
SRM SRM
OS OS OS
APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP (Use vSphere Replication (Use vSphere Replication Universal Transit Network VXLAN
Cluster Master Worker Worker Cluster Master Worker Worker OS OS NSXM NSX Manager
OS OS OS OS OS OS OS OS OS OS VIP Node Node Node VIP Node
OS OS OS OS
when using vSAN) when using vSAN)
Node Node PSC Platform Services Controller
IAS IAS BUC IAS IAS BUC
vRealize Log Insight
SQL Microsoft SQL Server Database
vRealize Log Insight
Application Virtual Networks for SDDC Management Solutions in Region A Application Virtual Networks for SDDC Management Solutions in Region B Region A Infrastructure Management Region B Infrastructure Management SRM Site Recovery Manager
Management Distributed Port Group UDLR Universal Distributed Logical Router
vSphere Update Manager Download Service, vSphere Update Manager Download Service,
vRealize Operations Analytics Cluster and Remote Collectors, Regional vRealize Log Insight Cluster, vRealize Operations Remote Collectors, Regional vRealize Log Insight Cluster, 192.168.11.51 > Active Node 192.168.11.57 > Active Node VDP vSphere Data Protection
Distributed vRealize Automation and Proxy Agents, and vRealize Business for Cloud Server and Collector. vRealize Automation Proxy Agents and vRealize Business for Cloud Collector. VRA VIP: 192.168.11.53 IMS VIP: 192.168.11.59 vSphere vSphere
192.168.11.58 > Passive Node VR vSphere Replication
192.168.11.52 > Active Node NSX NSX
Disaster Recovery vRealize Operations Analytics Cluster, VRA vRealize Automation Appliance
Distributed vRealize Automation, and vRealize Business for Cloud Server. Site Recovery Manager Site Recovery Manager VTEP VXLAN Tunnel Endpoint
192.168.11.54 > Active Node Management Application Virtual Network VXLAN
IWS VIP: 192.168.11.56
192.168.11.55 > Active Node
One region is designated as the primary region and the other as the secondary region. SDDC management, automation and operations solutions are
deployed in the primary region and configured to migrate to the secondary region in the event of a disaster. All regions actively run business workloads.

Copyright © 2017 VMware, Inc. All rights reserved. Refer to the design release notes for products and versions included in the design. @tenthirtyam | vmware.com/go/vvd-docs