VCE VxRAIL™ 3.

5 NETWORK GUIDE
Physical and Logical Network Considerations
and Planning

Document Version: H15300

July 2016
Table of Contents

INTENDED USE AND AUDIENCE 4

INTRODUCTION TO VXRAIL 4

Planning Your Network 4

Physical Network 5

VxRail Clusters, Appliances and Nodes ................................................ 5

Network Switch ................................................................................ 6

Decision Regarding vCenter Server ..................................................... 6

Topology and Connections ................................................................. 6

Workstation/Laptop .......................................................................... 7

Out-of-Band Management (optional) ................................................... 7

VxRail Setup Workflow 8

BEFORE CABLING VXRAIL 8

Step 1: Plan Logical Network 8

Step 1A. Reserve VLANs (Best Practice) 9

Step 1B. System 9

Time Zone, NTP Server, Proxy Server ................................................. 9

DNS Server ................................................................................... 10

Step 1C. Management 11

ESXi Hostnames and IP Addresses .................................................... 11

vCenter Server ............................................................................... 11

VxRail Manager and Networking ....................................................... 13

Passwords ..................................................................................... 14

Step 1D. vMotion and Virtual SAN 14

Step 1E. Solutions 15

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 2

Step 1F. Workstation/Laptop 15

Step 2: Set Up Switch 16

Step 2A. Understanding Switch Configuration 16

Network Traffic .............................................................................. 16

Multicast Traffic .............................................................................. 17

Inter-switch Communication ............................................................ 17

Disable Link Aggregation ................................................................. 17

vSphere Security Recommendations ................................................. 18

Step 2B. Configure VLANs on your switch(es) 18

Step 2C. Confirm Your Configuration 19

AFTER PLANNING AND SWITCH SETUP 19

Step 3: Cable & Power On 19

Management VLAN ......................................................................... 20

Step 4: Connect & Configure 20

VXRAIL INITIAL CONFIGURATION 21

ADDING NODES TO A VXRAIL CLUSTER 22

Step 1: Plan Logical Network 22

Step 2: Set Up Switch 22

Step 3: Cable & On 22

Step 4: Add VxRail Node 22

VXRAIL NETWORK CONFIGURATION TABLE 23

VXRAIL SETUP CHECKLIST 24

APPENDIX A: CUSTOMIZING THE VXRAIL INITIAL IP ADDRESS 26

APPENDIX B: NSX SUPPORT ON VXRAIL 27

APPENDIX C: JSON CONFIGURATION FILE 29

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 3

INTENDED USE AND AUDIENCE
This guide discusses the essential network details for VxRail deployment planning purposes only. It also introduces
best practices, recommendations, and requirements for both physical and virtual network environments. The guide
has been prepared for anyone involved in planning, installing, and maintaining VxRail, including EMC field
engineers and customer system and network administrators. This guide should not be used to perform the actual
installation and set-up of VxRail. Please work with your EMC or Partner implementation representative to perform
the actual installation.

INTRODUCTION TO VXRAIL
VxRail™ is a hyper-converged infrastructure (HCI) solution that consolidates compute and storage into a single,
highly available, network-ready unit. With careful planning, VxRail can be rapidly deployed into an existing
environment and the infrastructure is immediately available to deploy applications and services.

VxRail is not a server. It is an appliance that consists of four nodes. You will need 2U rack space in a 19x30-inch
cabinet for each VxRail appliance (4 server nodes). A 10GbE switch (or a 1GbE switch for certain models of VxRail)
is required. A workstation/laptop for the VxRail user interface is also required.

VxRail has a simple, scale-out architecture, leveraging VMware vSphere and Virtual SAN to provide server
virtualization and software-defined storage. Fundamental to the VxRail clustered architecture is network
connectivity. It is through the logical and physical networks that individual nodes act as a single system providing
scalability, resiliency and workload balance.

The VxRail software bundle is preloaded onto hardware and consists of the following components:

 VxRail Manager
 VMware vCenter Server™
 VMware vRealize Log Insight™
 VMware Virtual SAN™
 EMC Secure Remote Support (ESRS)/VE
 EMC Recover Point for Virtual Machines (RP4VM) - 15 Full Licenses per appliance
 EMC CloudArray- 1 TB local cache/10 TB cloud storage License
 VMware vSphere® licenses are also required and can be purchased through EMC, VMware or your preferred
VMware reseller partner
VxRail is fully compatible with other software in the VMware ecosystem, including VMware NSX.

Planning Your Network

The network considerations are no different from those of any enterprise IT infrastructure: availability,
performance, and extensibility. Generally, VxRail appliances are delivered ready to deploy and attach to any 10GbE
network infrastructure and use IPv4 and IPv6. (VxRail Model 60 uses a 1GbE switch.) Most production VxRail
network topologies use dual top-of-the-rack (ToR) switches to eliminate the switch as a single point of failure.

Follow all of the network prerequisites described in this document; otherwise VxRail will not be installed properly,
and it will not function correctly in the future. You must fill in the VxRail Network Configuration Table. Review
the VxRail Setup Checklist to ensure smooth deployment and configuration. Both are included in this guide. If
you have separate teams for network and servers in your data center, you will need to work together to design the
network and configure the switch(es).

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 4

Physical Network
This section describes the physical components found in a VxRail cluster:

 VxRail clusters, appliances and nodes

 Network switch
 Decision regarding vCenter Server
 Topology and connections
 Workstation/laptop
 Out-of-band management (optional)

VxRail Clusters, Appliances and Nodes

VxRail starts with one appliance with four nodes connected to one or more network switches, deployed to form a
VxRail cluster that contains the Virtual SAN environment. Up to 64 VxRail nodes can be added to the cluster. The
internal disks on each node combine to create a VxRail datastore that is shared across all the nodes in the cluster,
whether it’s a cluster of four nodes or 64 nodes. Within the cluster, multiple networks may service different
functions or types of traffic.

The cluster is managed by a single instance of VxRail Manager and vCenter Server. A logical tag in each node and
chassis is used to display the identity of the appliance in VxRail Manager. These tags are 11 alphanumeric
characters that uniquely identify the appliance.

Please review the physical power, space and cooling requirements for your expected resiliency level.

The following illustrations show possible configurations of a VxRail appliance with four nodes:

Figure 1. VxRail appliance with four nodes, showing the 10GbE

ports on each node

BMC 1GbE
port ports

Figure 2. VxRail Model 60 appliance with four nodes, showing the

1GbE ports on each node

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 5

Network Switch
VxRail is broadly compatible with most customer networks and switches. VxRail nodes communicate over one or
more customer-provided network switch(es), typically a top-of-rack switch. One example is the EMC Connectrix
VDX-6740 switch, available through your EMC representative.

Switch requirements:

 The switch(es) connected directly to VxRail must support IPv4 and IPv6 multicast on 10GbE ports for all
models of VxRail except for the VxRail Model 60 (which uses 1GbE ports).
 Be sure to have access to the manufacturer’s documentation for your specific switch(es).
 Keep in mind that while one switch can work, it is a potential single point of failure.
Port availability:

 Each VxRail node with 10GbE ports ships with either two SFP+ or RJ-45 NIC ports. Two corresponding ports
are required for each VxRail node on one or more 10GbE switch(es). Eight ports are needed for a four-node
initial configuration.
 Each VxRail node with 1GbE ports ships with four RJ-45 NIC ports. Four corresponding ports are required for
each VxRail node on one or more 1GbE switch(es). Sixteen ports are needed for a four-node initial
configuration.
 One additional port on the switch or one logical path on the VxRail management VLAN is required for a
workstation/laptop to access the VxRail user interface for the cluster.
Cable requirements:

 VxRail nodes with RJ-45 ports require CAT5 or CAT6 cables. CAT6 cables are included with every VxRail
 VxRail nodes with SFP+ ports require optical cables or Twinax Direct-Attach-Copper (DAC) cables. These
cables are not included; you must supply your own. The NIC and switch connectors and cables must be on the
same wavelength.

Please review the logical switch configuration requirements in the next section of this document.

Decision Regarding vCenter Server

The VxRail virtual infrastructure is managed by vCenter Server. Either VxRail connects to an existing vCenter
Server, or vCenter Server is installed and configured during VxRail initial configuration. Whether to use an internal
vCenter Server that is part of the VxRail infrastructure or to connect to an existing external vCenter Server
instance is an important decision point. If VxRail in in a standalone environment, then configuring an internal
vCenter Server is the easiest approach. On the other hand, if the new VxRail cluster will be added to an existing
VMware environment, integrating into the existing vCenter Server offers a consolidated view and management
point for the virtualized environment.

Topology and Connections

Various network topologies for switch(es) and VLANs are possible with VxRail. Complex production environments
will have multiple core switches and VLANs. A site diagram showing the proposed network components and
connectivity is highly recommended before cabling and powering on VxRail.

Be sure to follow your switch vendor’s best practices for performance and availability. For example, packet buffer
banks may provide a way to optimize your network with your wiring layout.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 6

Decide if you plan to use one or two switches for VxRail. One switch is acceptable and is often seen in
test/development or remote/branch office (ROBO) environments. However, two or more switches are used for high
availability and failover in production environments. Because VxRail is an entire software-defined data center in a
box, if one switch fails you are at risk of losing availability of hundreds of virtual machines.

Figure 3. Rear view of one deployment of VxRail connected to two

10GbE switches and a separate switch for out-of-band
management

Workstation/Laptop
A workstation/laptop with a web browser for the VxRail user interface is required. It must be either plugged into
the switch or able to logically reach the VxRail management VLAN from elsewhere on your network; for example, a
jump server (https://en.wikipedia.org/wiki/Jump_server).

Don’t try to plug your workstation/laptop directly into a server node on VxRail; plug it into your network or
switch and make sure that it is logically configured to reach VxRail.

You will use a browser for the VxRail user interface. The latest versions of Firefox, Chrome, and Internet Explorer
10+ are all supported. If you are using Internet Explorer 10+ and an administrator has set your browser to
“compatibility mode” for all internal websites (local web addresses), you will get a warning message from VxRail.
Contact your administrator to whitelist URLs mapping to the VxRail user interface.

Out-of-Band Management (optional)

If VxRail will be located at a data center that you cannot access easily, we recommend setting up an out-of-band
management switch to facilitate direct communication with each node. To use out-of-band management, connect
the BMC port on each node to a separate switch to provide physical network separation.

Default values, capabilities, and recommendations for out-of-band management are provided with server hardware
information. The default configuration is via DHCP with:

Username: UserId Password: Passw0rd!

NOTE: Case sensitive and using a zero in place of a lowercase ‘o’ in the password

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 7

The <ApplianceID> can be found on a pull out tag located in front of the chassis. The default hostnames should be
as follows:

BMC interface node 1: hostname = <ApplianceID>-01

BMC interface node 2: hostname = <ApplianceID>-02
BMC interface node 3: hostname = <ApplianceID>-03
BMC interface node 4: hostname = <ApplianceID>-04

VxRail Setup Workflow

To ensure the correct functioning of VxRail, understanding the recommendations and requirements in this guide is
essential. You must complete the steps in this workflow in this order to successfully set up VxRail.

Before cabling VxRail:

1. Plan logical network: Meet with your team to plan the network architecture including switch configuration,
VLANs, and IP addresses.

2. Set up switch: Configure your 10GbE or 1GbE switch. This must be done BEFORE you connect or power on.

After planning and switch setup:

3. Cable & on: Cable nodes to switch(es), then turn on all four VxRail nodes.

4. Connect & configure: Connect to VxRail’s initial IP address via workstation/laptop. Point your browser to VxRail
initial configuration user interface to create your software defined data center.

BEFORE CABLING VXRAIL

Step 1: Plan Logical Network
VxRail is not a simple server but is an entire data center in a box. Consequently, the network and virtualization
teams need to meet in advance to plan VxRail’s network architecture.

Use the VxRail Setup Checklist and the VxRail Network Configuration Table to document your network plan.
References to rows in this document are to rows in this table. Work with your EMC implementation rep or partner
who has access to these tools.

Once you set up VxRail, the configuration cannot be changed easily. Consequently, we strongly
recommend that you take care during this planning phase to decide on the configurations that will
work most effectively for your organization. We want you to set up VxRail correctly when it arrives.

A VxRail cluster consists of four or more VxRail nodes. Your plan can include up to 64 server nodes that can be
joined together in one VxRail cluster. If you have already configured enough IP addresses for expansion (which we
recommend), all you do is supply the passwords that you created for the VxRail cluster. If you do not have enough
IP addresses, just follow the section at the end of this document, Adding Nodes to a VxRail Cluster. VxRail
Manager will prompt you to add the new IP addresses and the passwords – nothing else!

You will be making decisions in the following areas:

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 8

 Step 1A. Reserve VLANs (best practice)
 Step 1B. System
 Step 1C. Management
 Step 1D. vMotion and Virtual SAN
 Step 1E. Solutions
 Step 1F. Workstation/laptop

Step 1A. Reserve VLANs (Best Practice)

VxRail groups traffic in the following categories: management, vSphere vMotion, Virtual SAN, and Virtual Machine.
Traffic isolation on separate VLANs is highly recommended (but not required) in VxRail. If you are using multiple
switches, connect them via VLAN trunked interfaces and ensure that all VLANs used for VxRail are carried across
the trunk following the requirements in this user guide.

Management traffic includes all VxRail, vCenter Server, and ESXi communication. The management VLAN also
carries traffic for vRealize Log Insight. By default, all management traffic is untagged and must be able to go
over a Native VLAN on your switch or you will not be able to build VxRail and configure the ESXi hosts. However,
you can tag management traffic in one of two ways:

1. Configure each VxRail port on your switch to tag the management traffic and route it to the desired VLAN.

2. Alternately, you can configure a custom management VLAN to allow tagged management traffic. After you
power on each node, but before your run VxRail initial configuration, please follow the instructions in Step 3:
Cable & Power On to change the management VLAN.

vSphere vMotion and Virtual SAN traffic cannot be routed. This traffic will be tagged for the VLANs you specify
in VxRail initial configuration.

Dedicated VLANs are preferred to divide virtual machine traffic. VxRail will create one or more VM Networks for
you, based on the name and VLAN ID pairs that you specify. Then when you create VMs in vSphere Web Client,
you can easily assign the virtual machine to the VM Network(s) of your choice. For example, you could have one
VLAN for Development, one for Production, and one for Staging.

Network Configuration Enter the management VLAN ID for VxRail, ESXi, and vCenter Server. If you do
Table not plan to have a dedicated management VLAN and will accept this traffic as
 Row 1 untagged, enter “0” or “Native VLAN”.

Network Configuration
Enter a VLAN ID for vSphere vMotion.
Table
(Enter a 0 in the VLAN ID field for untagged traffic)
 Row 33

Network Configuration
Enter a VLAN ID for Virtual SAN.
Table
(Enter a 0 in the VLAN ID field for untagged traffic)
 Row 37

Network Configuration Enter a Name and VLAN ID pair for each VM network you want to create.
Table You must create at least one VM Network.
 Rows 38-40 (Enter a 0 in the VLAN ID field for untagged traffic)

Step 1B. System

VxRail can configure connections to external servers in your network.

Time Zone, NTP Server, Proxy Server

A time zone is required. It is configured on vCenter Server and each ESXi host.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 9

An NTP server is not required, but it is recommended. If you provide an NTP server, vCenter Server will be
configured to use it. If you do not provide at least one NTP server, VxRail uses the time that is set on ESXi host #1
(regardless of whether the time is correct or not).

A proxy server is optional. If you have a proxy server on your network and vCenter Server needs to access
services outside of your network, supply the IP address, port, username, and password.

Network Configuration
Table Enter your time zone.
 Row 3

Network Configuration
Table Enter the hostname(s) or IP address(es) of your NTP server(s).
 Row 4

Network Configuration
Table Enter the proxy server IP address, port, username, and password.
 Rows 6 and 7

DNS Server
One or more external DNS servers are required for production use (it is not required in a completely isolated
environment). DNS is used for some VxRail management operations, such as importing an OVA file, which requires
a FQDN for direct host access. During initial configuration, VxRail sets up vCenter Server to resolve hostnames to
the DNS server.

If you are in an isolated environment, you will need to use the DNS server that is built into vCenter
Server. To manage VxRail via your workstation/laptop, configure your laptop’s network settings to use the vCenter
Server IP address (Row 15) for DNS. VxRail’s IP addresses and hostnames are configured for you.

Make sure that the DNS IP address is accessible from the network to which VxRail is connected and
functioning properly. If the DNS server requires access via a gateway that is not reachable during
initial configuration, do not enter a DNS IP address. Instead, add a DNS server after you have
configured VxRail using VMware KB (http://kb.vmware.com/kb/2107249).

Network Configuration
Enter the IP address(es) for your DNS server(s). Leave blank if you are in an
Table
isolated environment. Required when an external vCenter Server is used.
 Row 5

If you are using your corporate DNS server(s) for VxRail, be sure to add the hostnames and IP addresses for VxRail
Manager, vCenter Server, Log Insight, and each ESXi host (see the naming scheme in ESXi Hostnames and IP
Addresses). vMotion and Virtual SAN IP addresses are not configured for routing by VxRail and there are no
hostnames

Example of VxRail hostnames and IP addresses configured on a DNS server:

esxi-host01.localdomain.local 192.168.10.1
esxi-host02.localdomain.local 192.168.10.2
esxi-host03.localdomain.local 192.168.10.3
esxi-host04.localdomain.local 192.168.10.4
vxrail.localdomain.local 192.168.10.100
vcserver.localdomain.local 192.168.10.101
loginsight.localdomain.local 192.168.10.102

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 10

Step 1C. Management
VxRail does not have a single hostname. You must configure the hostnames for each ESXi host, VxRail Manager,
and vCenter Server.

You must configure the IP addresses for VxRail, vCenter Server, and your ESXi hosts. When selecting your IP
addresses, you must make sure that none of them conflict with existing IP addresses in your network. Also make
sure that these IP addresses can reach other hosts in your network.

You cannot easily change the IP addresses after you have configured VxRail.

ESXi Hostnames and IP Addresses

All ESXi hostnames in a VxRail cluster are defined by a naming scheme that comprise: an ESXi hostname prefix (an
alphanumeric string), a separator (“None” or a dash ”-“), an iterator (Alpha, Num X, or Num 0X), and a domain.
The Preview field shown during VxRail initial configuration is an example of the hostname of the first ESXi host. For
example, if the prefix is “host”, the separator is “None”, the iterator is “Num 0X”, and the domain is “local”, the
first ESXi hostname would be “host01.local”. The domain is also automatically applied to the vCenter Server and
VxRail virtual machines. (Example: my-vcenter.local)

Examples:

Example 1 Example 2 Example 3

Prefix host myname esxi-host
Separator None - -
Iterator Num 0X Num X Alpha
Domain local college.edu company.com
Resulting hostname host01.local myname-1.college.edu esxi-host-a.company.com

There are four ESXi hosts in your initial cluster and each requires an IP address. We recommend that you consider
allocating additional ESXi IP addresses for future nodes to join your VxRail cluster. Because VxRail supports up to
64 nodes in a cluster, you can allocate up to 64 ESXi IP addresses.

Network Configuration
Enter an example of your desired ESXi host-naming scheme. Be sure to show
Table
your desired prefix, separator, iterator, and domain.
 Rows 8-11

Network Configuration
Enter the starting and ending IP addresses for the ESXi hosts - a continuous IP
Table
range is required, with a minimum of 4 IPs.
 Rows 12 and 13

vCenter Server
A new feature in VxRail 3.5 is the ability to join an existing vCenter Server instead of deploying a new vCenter
Server for the VxRail cluster you will build. This allows a remote central vCenter Server to manage multiple VxRail
clusters in a single pane of glass.

If you want VxRail to create a new vCenter Server, you will need to specify a hostname and IP address for your
new vCenter Server and Platform Services Controller (PSC) virtual machines. (Rows 14-17)

If you want VxRail to join an existing vCenter Server, you will need to:

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 11

 Know whether your external vCenter Server has an embedded or external Platform Services Controller. If the
PSC is external, enter the PSC FQDN (Row 18).
 Know the external vCenter Server FQDN (Row 19) and the administrative username and password (Row 20).
 Create a VxRail management user and password (Row 21) for this VxRail cluster on the external vCenter
Server. This user must be created with no permissions and it must be unique for each VxRail cluster on this
external vCenter Server.
 Create or select an existing datacenter (Row 22) on the external vCenter Server.
 Specify the name of the cluster (Row 23) that will be created by VxRail in the selected datacenter when the
cluster is built. This name must be unique and not used anywhere in the datacenter on the external vCenter
Server.
In Release 3.5, the top-level domain of the external vCenter Server and PSC must be publicly known,
such as .com, .net, .edu, .local, and many country-specific suffixes. Most of those listed in this reference
are supported: https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains

VxRail Manager leverages the same database as vCenter Server, so any changes in VxRail are reflected in vCenter
Server and vice-versa.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 12

 Internal vCenter Server (deployed when VxRail is built)

Network Configuration
Enter an alphanumeric string for the new vCenter Server hostname. The domain
Table
specified in Row 11 will be appended.
 Row 14

Network Configuration
Table Enter the IP address for new vCenter Server.
 Row 15

Network Configuration
Enter an alphanumeric string for the new Platform Services Controller hostname.
Table
The domain specified in Row 11 will be appended.
 Row 16

Network Configuration
Table Enter the IP address for new Platform Services Controller.
 Row 17

External vCenter Server

Network Configuration Enter the FQDN of the external Platform Services Controller (PSC) in the
Table hostname. In the user interface, there is a checkbox for external PSC.
 Row 18 Leave this row blank if the PSC is embedded in the external vCenter Server.

Network Configuration
Table Enter the FQDN of the external vCenter Server in the hostname field.
 Row 19

Network Configuration
Enter the full administrative username and password for the external vCenter
Table
Server. (For example, administrator@vpshere.local)
 Row 20

Network Configuration Go to the external vCenter Server and create a new, unique user and password
Table with no permissions for this cluster.
 Row 21 (For example, cluster1-manager@vsphere.local)
Enter the full VxRail management username and password that you created.

Network Configuration Go to the external vCenter Server and select or create a datacenter.
Table
 Row 22 Enter the name of a datacenter on the external vCenter Server.

Network Configuration
Table Enter the name of the cluster that will be created by VxRail.
 Row 23

VxRail Manager and Networking

You must specify the hostname and IP address for the VxRail Manager virtual machine. In addition, you must
specify the subnet mask and gateway that VxRail Manager, vCenter Server, and the ESXi hosts all share.

We do not recommend using the default VxRail initial IP address (192.168.10.200/24) as your
permanent VxRail IP address (Row 25), because if you later add more nodes to the VxRail cluster or if
you create more clusters, the initial IP addresses will conflict with the existing cluster’s IP address.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 13

 Network Configuration
Table Enter an alphanumeric string for the VxRail Manager hostname.
 Row 24

Network Configuration
Table Enter the IP address for VxRail Manager after it is configured. We recommend
that you do not use the default 192.168.10.200/24
 Row 25

Network Configuration
Table Enter the subnet mask and gateway for all management IP addresses.
 Rows 26 and 27

Passwords
You must specify one root password for all ESXi hosts in the cluster. You must also specify one password for the
VxRail Manager virtual machine. Unless you are using an external vCenter Server, the VxRail Manager and vCenter
Server virtual machines will have the same administrative password.

Passwords must contain between 8 and 20 characters with at least one lowercase letter, one uppercase letter, one
numeric character, and one special character. For more information about password requirements, see the vSphere
password documentation and vCenter Server password documentation.

For ESXi hosts, the username is root; the pre-configuration password is Passw0rd! and the post-configuration
password is the one you set in VxRail initial configuration (Row 28).

For VxRail Manager and an internal vCenter Server, the username for both user interfaces is
administrator@vsphere.local and the console username is root. The pre-configuration password for VxRail is
Passw0rd! and the post-configuration password is the one you set in VxRail initial configuration (Row 29).

Network Configuration
Please check that you know your passwords in these rows, but for security
Table
reasons, we suggest that you do not write them down.
 Rows 28 and 29

Step 1D. vMotion and Virtual SAN

vSphere vMotion and Virtual SAN each require at least four IP addresses for the initial cluster. We recommend that
you consider allocating additional IP addresses for future nodes to join your VxRail cluster. If you have already
configured enough IP addresses for expansion, all you do is supply the passwords that you created when VxRail
was built.

Because VxRail supports up to 64 nodes in a cluster, you can allocate up to 64 vMotion IP addresses and 64 Virtual
SAN IP addresses.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 14

 Network Configuration Enter the starting and ending IP addresses for vSphere vMotion – a continuous
Table IP range is required, with a minimum of 4 IPs. Routing is not configured for
 Rows 30 and 31 vMotion.

Network Configuration
Table Enter the subnet mask for vMotion.
 Row 32

Network Configuration Enter the starting and ending IP addresses for Virtual SAN – a continuous IP
Table range is required, with a minimum of 4 IPs. Routing is not configured for Virtual
 Rows 34 and 35 SAN.

Network Configuration
Table Enter the subnet mask for Virtual SAN.
 Row 36

Step 1E. Solutions

VxRail is deployed with vRealize Log Insight. Alternately, you may choose to use your own third-party syslog
server(s). If you choose to use vRealize Log Insight, it will always be available by pointing a browser to the
configured IP address with the username, admin. (If you ssh to Log Insight instead of pointing your browser to it,
the username is root.) The password, in either case, is the same password that you specified for vCenter
Server/VxRail (Row 29).

NOTE: The IP address for Log Insight must be on the same subnet as VxRail and vCenter Server.

Network Configuration
Table Enter the hostname and IP address for vRealize Log Insight or the hostname(s)
 Rows 41 and 42 or of your existing third-party syslog server(s).
 Row 43

Step 1F. Workstation/Laptop

To access the VxRail for the first time, you must use the temporary VxRail initial IP address that was pre-
configured, typically 192.168.10.200/24. You will change this IP address during VxRail initial configuration to your
desired permanent address for your new VxRail cluster.

VxRail Workstation/laptop
Example
Configuration IP address/netmask IP address Subnet mask Gateway
Initial
192.168.10.200/24 192.168.10.150 255.255.255.0 192.168.10.254
(temporary)
Post-
configuration 10.10.10.100/24 10.10.10.150 255.255.255.0 10.10.10.254
(permanent)

Your workstation/laptop will need to be able to reach both the VxRail initial IP address (Row 2) and your selected
permanent VxRail IP address (Row 25). VxRail initial configuration will remind you that you may need to
reconfigure your workstation/laptop network settings to access the new IP address.

It may be possible to give your workstation/laptop or your jump server two IP addresses, which allows for a
smoother experience. Depending on your workstation/laptop, this can be implemented in several ways (such as

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 15

dual-homing or multi-homing). Otherwise, change the IP address on your workstation/laptop when instructed to
and then return to VxRail Manager.

If you cannot reach the VxRail initial IP address, you will need to follow the instructions in Appendix A to configure
a custom IP address, subnet mask, and gateway.

Furthermore, if a custom management VLAN ID will be used for VxRail other than VLAN 1 (VLAN 1
default management VLAN ID for most of switches), make sure the workstation/laptop can also access
this management VLAN.

Network Configuration Please enter the VxRail initial IP address.

Table Enter 192.168.10.200/24 if you can reach this address on your network.
 Row 2
Otherwise, enter your custom IP address, subnet mask, and gateway.

Step 2: Set Up Switch

In order for VxRail to function properly, you must configure the ports the that VxRail will use on your switch
before you plug in VxRail and turn it on.

Set up your switch by following these steps:

 Step 2A. Understanding switch configuration

 Step 2B. Configure VLANs on your switch(es)
 Step 2C. Confirm your configuration

Step 2A. Understanding Switch Configuration

Be sure to follow your switch vendor’s best practices for performance and availability. Ports on a switch operate in
one of the following modes:

 Access mode – The port accepts only untagged packets and distributes the untagged packets to all VLANs on
that port. This is typically the default mode for all ports.
 Trunk mode – When this port receives a tagged packet, it passes the packet to the VLAN specified in the tag.
To configure the acceptance of untagged packets on a trunk port, you must first configure a single VLAN as a
“Native VLAN”. A “Native VLAN” is when you configure one VLAN to use as the VLAN for all untagged traffic.
 Tagged-access mode – The port accepts only tagged packets.

Network Traffic
Each VxRail node has either two 10GbE network ports or four 1GbE network ports. Each port must be connected to
a switch that supports IPv4 multicast and IPv6 multicast. To ensure vSphere vMotion traffic does not consume all
available bandwidth on the port, VxRail limits vMotion traffic to 4Gbps.

VxRail traffic on 10GbE NICs is separated as follows:

Traffic Type Requirements 1st 10GbE NIC 2nd 10GbE NIC

Management IPv6 multicast Standby Active

vSphere vMotion Standby Active

Virtual SAN IPv4 multicast Active Standby

Virtual Machines Standby Active

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 16

VxRail traffic on 1GbE NICs is separated as follows:

Traffic Type Requirements 1st 1GbE NIC 2nd 1GbE NIC 3rd 1GbE NIC 4th 1GbE NIC

Management IPv6 multicast Standby Active Unused Unused

vSphere vMotion Unused Unused Standby Active

Virtual SAN IPv4 multicast Unused Unused Active Standby

Virtual Machines Active Standby Unused Unused

Multicast Traffic
IPv4 multicast support is required for the Virtual SAN VLAN. IPv6 multicast is required for the VxRail
management VLAN. The network switch(es) that connect to VxRail must allow for pass-through of
multicast traffic on these two VLANs. Multicast is not required on your entire network, just on the ports
connected to VxRail.

Why multicast? VxRail has no backplane, so communication between its four nodes is facilitated via the network
switch. This communication between the four nodes uses VMware Loudmouth auto-discovery capabilities, based on
the RFC-recognized "Zero Network Configuration" protocol. New VxRail nodes advertise themselves on a network
using the VMware Loudmouth service, which uses IPv6 multicast. This IPv6 multicast communication is strictly
limited to the management VLAN that the nodes use for communication.

VxRail creates very little traffic via IPv6 multicast for autodiscovery and management. It is optional to limit traffic
further on your switch by enabling MLD Snooping and MLD Querier.

There are two options to handle Virtual SAN IPv4 multicast traffic. Either limit multicast traffic by enabling both
IGMP Snooping and IGMP Querier or disable both of these features. We recommend enabling both IGMP Snooping
and IGMP Querier, if your switch supports them.

IGMP Snooping software examines IGMP protocol messages within a VLAN to discover which interfaces are
connected to hosts or other devices interested in receiving this traffic. Using the interface information, IGMP
Snooping can reduce bandwidth consumption in a multi-access LAN environment to avoid flooding an entire VLAN.
IGMP Snooping tracks ports that are attached to multicast-capable routers to help manage IGMP membership
report forwarding. It also responds to topology change notifications. Disabling IGMP Snooping may lead to
additional multicast traffic on your network.

IGMP Querier sends out IGMP group membership queries on a timed interval, retrieves IGMP membership reports
from active members, and allows updates to group membership tables. By default, most switches enable IGMP
Snooping, but disable IGMP Querier.

Inter-switch Communication
In a multi-switch environment, configure the ports used for inter-switch communication to carry IPv6 multicast
traffic for the VxRail management VLAN. Likewise, carry IPv4 multicast traffic between switches for the Virtual SAN
VLAN. Consult your switch manufacturer’s documentation for how to do this.

Disable Link Aggregation

Do not use link aggregation, including protocols such as LACP and EtherChannel, on any ports directly connected to
VxRail. VxRail uses active/standby configuration (NIC teaming) for network redundancy, as discussed in the section
on Network Traffic.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 17

vSphere Security Recommendations
Security recommendations for vSphere are found here:

http://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-FA661AE0-
C0B5-4522-951D-A3790DBE70B4.html

In particular, ensure that physical switch ports are configured with Portfast if spanning tree is enabled. Because
VMware virtual switches do not support STP, physical switch ports connected to an ESXi host must have Portfast
configured if spanning tree is enabled to avoid loops within the physical switch network. If Portfast is not set,
potential performance and connectivity issues might arise.

Step 2B. Configure VLANs on your switch(es)

Now that you understand the switch requirements, it is time to configure your switch(es).

The VxRail network can be configured with or without VLANs. For performance and scalability, it is highly
recommended to configure VxRail with VLANs. As listed in the VxRail Setup Checklist, you will be configuring the
following VLANs:

 Management VLAN (default is untagged/native): make sure that IPv6 multicast is configured/enabled on the
management VLAN (regardless of whether tagged or native).
 Virtual SAN VLAN: make sure that IPv4 multicast is configured/enabled on the Virtual SAN VLAN (enabling
IGMP snooping and querier is highly recommended).
 vSphere vMotion VLAN
 VM Networks VLANs

Figure 4. VxRail VLAN configuration.

Using the VxRail Network Configuration Table configure each switch port that will be connected to a VxRail
node:

Configure the Management VLAN (Row 1) on the switch ports. If you entered “Native VLAN”, then set the ports
on the switch to accept untagged traffic and tag it to the custom management VLAN ID. Untagged management

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 18

traffic is the default management VLAN setting on VxRail. For example, on the EMC Connectrix VDX-6740 switch,
using “switchport trunk native VLAN vid” configures VxRail management traffic to travel on the customer’s
management VLAN.

Regardless of whether you are using an untagged Native VLAN or a tagged VLAN, you must set the management
VLAN to allow IPv6 multicast traffic to pass through. Depending on the type of switch you have, you may need to
turn on IPv6 and multicast directly on the port or on the VLAN. Be sure to review the previous section, Step 2A.
Understanding Switch Configuration, and consult the switch manufacturer for further instructions on how to
configure these settings.

Configure a vSphere vMotion VLAN (Row 33) on the switch ports.

Configure a Virtual SAN VLAN (Row 37) on the switch ports, set to allow IPv4 multicast traffic to pass through.

Configure the VLANs for your VM Networks (Rows 38-40) on the switch ports.

Step 2C. Confirm Your Configuration

Some network configuration errors cannot be recovered from and you will need VxRail support to reset to factory
defaults. When VxRail is reset to factory defaults, all data is lost. Please confirm your switch setting in this step.

Read your vendor instructions for your switch:

a. Confirm that IPv4 multicast and IPv6 multicast are enabled for the VLANs described in this document.
b. If you have two or more switches, confirm that IPv4 multicast and IPv6 multicast traffic is transported
between them.
c. Remember that management traffic will be untagged on the native VLAN on your switch, unless all ESXi
hosts have been customized for a specific management VLAN.

Network design and accessibility:

a. Confirm that you can ping or point to the VxRail initial IP address (Row 2).
b. Confirm that your DNS server(s) are reachable unless you are in an isolated environment (Row 5). The
DNS server must be reachable from the VxRail, vCenter Server, and ESXi network addresses. Then update
your DNS server with all VxRail hostnames and IP addresses.
c. Confirm that your management gateway IP address is accessible (Row 27). It is used for vSphere High
Availability (HA) to work correctly. You can use a corporate gateway on your VxRail network segment or
you may be able to configure your L3 switch as the gateway. When vSphere HA is not working, you will
see a “network isolation address” error. VxRail will continue to function, but it will not be protected by the
vSphere HA feature.
http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.avail.doc/GUID-5432CA24-14F1-
44E3-87FB-61D937831CF6.html
d. If you have configured NTP servers, proxy servers, or a third-party syslog server, confirm that you are
able to reach them from all of your configured VxRail IP addresses.

AFTER PLANNING AND SWITCH SETUP

Step 3: Cable & Power On
Rack and cable VxRail. After the nodes are cabled, power on all four initial nodes in your VxRail cluster.

Do not turn on any other VxRail nodes until you have completed the full configuration of the first four
nodes. See Adding Nodes to a VxRail Cluster.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 19

Management VLAN
If you did not configure your switch to tag the management traffic coming from VxRail for your management VLAN,
you can alternately tag it on each ESXi host. You will need to use the ESXi Command Line Interface (CLI) on each
ESXi host before using VxRail initial configuration to build VxRail.

To customize the management VLAN before VxRail is initially configured, changes are required for two different
portgroups on all ESXi hosts. The first portgroup is the ESXi “Management Network”, and the second portgroup is
the initial VxRail management network, called “VM Network”. During configuration the second portgroup is
renamed “vCenter Server Network”.

Login to each of the four ESXi hosts via the console interface, DCUI.

 Press <F2> to login with the username root and the password Passw0rd!
 Go to “Troubleshooting Options” and press <Enter> to select.
 Go to “Enable ESXi Shell” and press <Enter> to change.
 Press <ESC> to save.
 Press <ALT-F1> to get to the ESXi shell.
 Login to the shell with the username root and the password Passw0rd!
 Execute the following ESXi commands with the <VLAN_ID> from Row 1 in the VxRail Network
Configuration Table:
esxcli network vswitch standard portgroup set -p "Management Network" -v <VLAN_ID>
esxcli network vswitch standard portgroup set -p "VM Network" -v <VLAN_ID>
/etc/init.d/loudmouth restart
 To verify the VLAN ID was set correctly, run the following command:
esxcli network vswitch standard portgroup list
NOTE: If your management VLAN is customized on-site, your backup configBundle will not include the new VLAN.
If VxRail is ever reset, the management VLAN will have to be reconfigured.
Documentation for vSphere/ESXi command line interface is provided at http://pubs.vmware.com/vsphere-
60/index.jsp#com.vmware.vsphere.scripting.doc/GUID-7F7C5D15-9599-4423-821D-7B1FE87B3A96.html

Step 4: Connect & Configure

If you have successfully followed all of the previous steps, your network setup is complete and you are ready to
connect to VxRail from your workstation/laptop. VxRail is currently installed by EMC professional services
personnel. You will need to make arrangements for this step to be done for you.

Step 4A. Connect a workstation/laptop to access the VxRail initial IP address on your selected management
VLAN. It must be either plugged into the switch or able to logically reach the VxRail management
VLAN from elsewhere on your network.

If you cannot reach 192.168.10.200/24, you can change the initial IP address directly on
ESXi host #1, following the instructions in Appendix A.

Step 4B. Browse to the VxRail initial IP address. Configure and build VxRail as described in VxRail Initial
Configuration.

Step 4C. Configure your corporate DNS server for all VxRail hostnames and IP addresses unless you are in an
isolated environment.

Step 4D. Connect to VxRail Manager using either the VxRail Manager IP address (Row 25) or the fully-qualified
domain name (FQDN) (Row 24) that you configured on your DNS server (e.g.
https://vxrail.yourcompany.com).

When you add more nodes to a VxRail cluster, follow the steps in Adding Nodes to a VxRail Cluster.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 20

VXRAIL INITIAL CONFIGURATION
Use the information from your VxRail Network Configuration Table as you follow these steps in the VxRail
initial configuration user interface.

Step 1. Browse to the VxRail initial IP address (Row 2); for example, https://192.168.10.200. Ignore any
browser warnings about security (for example, by clicking “Advanced” and “Proceed”.) You will then see
the VxRail welcome splash page.
Step 2. Click Get Started. Then if you agree, accept the VxRail End-User License Agreement (EULA).
Step 3. Click Step-by-step to configure hostnames, IP addresses, VLAN IDs, and passwords to type in your
values from the VxRail Network Configuration Table.
Alternately, click Configuration File to upload a JSON-formatted configuration file that you have
created with your values. See Appendix C for the file format and valid values.

Step 4. Carefully enter your data or review each configuration field using the values in the rows of your VxRail
Network Configuration Table.
 System
Enter your time zone and your existing NTP and DNS server(s) from Rows 3-5. Enter the IP
address, port, username, and password for your proxy server (optional) from Rows 6-7.
 Management
Enter the ESXi host naming scheme and IP address range from Rows 8-13. Enter the internal or
external vCenter Server information from Rows 14-23. Enter the VxRail Manager hostname and IP
address, subnet mask, and gateway from Rows 24-27. Enter the ESXi hosts and vCenter
Server/VxRail passwords from Rows 28-29.
 vSphere vMotion
Enter the VLAN ID, IP addresses, and subnet mask for vSphere vMotion from Rows 30-33.
 Virtual SAN
Enter the VLAN ID, IP addresses, and subnet mask Virtual SAN from Rows 34-37.
 VM Networks
Enter the VLAN IDs and names for the VM Networks from Rows 38-40.
 Solutions
For logging, enter the IP address and hostname for vRealize Log Insight or for an existing third-
party syslog server (optional) in your network (Rows 41-43).
Step 5. Click the Review First or Validate button. VxRail verifies the configuration data, checking for conflicts.
Step 6. After validation is successful, click the Build VxRail button.
Step 7. The new IP address for VxRail will be displayed.
Click Start Configuration. Ignore any browser messages about security (for example, by clicking
“Advanced” and “Proceed”.)

NOTE: You may need to manually change the IP settings on your workstation/laptop to be on the same
subnet as the new VxRail IP address (Row 25).
NOTE: If your workstation/laptop cannot connect to the new IP address that you configured, you will get
a message to fix your network and try again. If you are unable to connect to the new IP address
after 20 minutes, VxRail will revert to its un-configured state and you will need to re-enter your
configuration at the initial IP address (Row 2).
NOTE: After the build process starts, if you close your browser, you will need to browse to the new IP
address (Row 25).

Step 8. Progress is shown as VxRail is built. VxRail implements services, creates the new ESXi hosts, sets up
vCenter Server, vMotion, and Virtual SAN.
When you see the Hooray! page, VxRail is built. Click the Manage VxRail button to continue to VxRail
management. You should also bookmark this IP address in your browser for future use.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 21

ADDING NODES TO A VXRAIL CLUSTER
VxRail can scale out to 64 ESXi hosts all on one Virtual SAN datastore, backed by a single vCenter Server and
VxRail instance. Deployment, configuration, and management are handled by VxRail, allowing the compute
capacity and the Virtual SAN datastore to grow automatically. New nodes are automatically discovered and easily
added to a VxRail cluster.

If you plan to scale out with additional nodes in this VxRail cluster over time, allocate extra IP addresses for each
of the ESXi, vMotion, and Virtual SAN IP pools when you initially configure VxRail (three extra IP addresses per
node). Then when you add nodes to a cluster, you will only need to enter the ESXi and VxRail / vCenter Server
passwords.

NOTE: If you have multiple independent VxRail clusters, we recommend using different VLAN IDs for Virtual SAN
traffic and for management across multiple VxRail clusters. Otherwise, all VxRail nodes on the same
network will see all multicast traffic.

Step 1: Plan Logical Network

Use the VxRail Setup Checklist to make sure you are ready for additional nodes. Work with your team to make
any decisions that were not made earlier.

 Enough switch ports for each VxRail node on your switch(es)

 IP addresses on the management VLAN for ESXi hosts for each node
 IP addresses on the Virtual SAN VLAN and the vSphere vMotion VLAN for each node
 Optional: Extra capacity and IP address for the out-of-band management port on each node
 Be sure you know the ESXi host and vCenter Server/VxRail root passwords

Step 2: Set Up Switch

Before you plug in a new VxRail node, configure each switch port just like you did for the initial VxRail nodes. On
each port connected to VxRail, configure the management VLAN, Virtual SAN VLAN, vMotion VLAN, and VM
Network VLANs, including IPv4 multicast and IPv6 multicast. Do not use link aggregation (LACP/EtherChannel) on
the VxRail ports on the switch. If you are using your switch to tag the traffic on the management VLAN, configure
that as well.

Step 3: Cable & On

Rack your new node and connect the ports on VxRail to the network switch(es). Power on one node – only one
node can be added at a time.

If you are tagging traffic for the management VLAN on each node, customize the management VLAN via the ESXi
Command Line Interface as you did when you first set up VxRail.

VxRail will not discover any ESXi hosts that are not on the same management VLAN. Login to the ESXi host
on the new node and follow the management VLAN instructions.

Step 4: Add VxRail Node

Go to VxRail Manager to see that the node was detected. Click the Add VxRail Node button. You will be prompted
to add IP addresses unless you have enough pre-allocated. Then just enter the ESXi and vCenter Server passwords
and VxRail will seamlessly configure all services on the new node and fully integrate it into the cluster.

Be sure to add any ESXi hostnames that were not previously entered in your corporate DNS, unless you
are in a totally isolated environment.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 22

VXRAIL NETWORK CONFIGURATION TABLE
Row Category Description Customer Values
1 VxRail Management Set tagging for your management VLAN on your
VLAN ID switch or on ESXi™ before you configure VxRail.
Default is untagged traffic on the Native VLAN.
2 VxRail initial IP If you cannot reach the default (192.168.10.200/24),
address set an alternate IP address
3 System Global settings Time zone
4 NTP server(s)
5 DNS server(s)
6 Proxy settings IP address and port
7 Username and password
8 Management ESXi ESXi hostname prefix
9 hostnames Separator
10 and IP Iterator
11 addresses Domain
12 ESXi starting address for IP pool
13 ESXi ending address for IP pool
14 vCenter Server vCenter Server hostname
15 (internal) vCenter Server IP address
16 Leave blank if Platform Services Controller hostname
17 external VC Platform Services Controller IP address
18 vCenter Server External Platform Services Controller (PSC)
(external) Hostname (FQDN)
Leave blank if Leave blank if PSC is internal
19 internal VC External vCenter Server hostname (FQDN)
20 Existing administrative username and password
21 New VxRail management username and password
22 Existing datacenter name
23 New cluster name
24 VxRail VxRail hostname
25 Manager VxRail IP address
26 Networking Subnet mask
27 Gateway
28 Passwords ESXI “root”
29 VxRail Manager and internal vCenter Server
“administrator@vsphere.local”
30 vMotion Starting address for IP pool
31 Ending address for IP pool
32 Subnet mask
33 VLAN ID
34 Virtual SAN Starting address for IP pool
35 Ending address for IP pool
36 Subnet mask
37 VLAN ID
38 VM VM Network name and VLAN ID
39 Networks … (unlimited number)
40 VM Network name and VLAN ID
41 Solutions Logging vRealize Log Insight™ hostname
42 vRealize Log Insight IP address
43 Syslog server (instead of Log Insight)

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 23

VXRAIL SETUP CHECKLIST

Physical Network

 VxRail cluster: Decide if you want to plan for additional nodes beyond the initial four-node cluster.
 Network switch: Two 10GbE ports (SFP+ or RJ-45) or four 1GbE ports for each VxRail node. VxRail initially
has four nodes. You can have up to 64 nodes in a VxRail cluster. Check cable requirements.
 Topology: Decide if you will have a single or multiple switch setup for redundancy.
 Workstation/laptop: Any operating system with a browser to access the VxRail user interface. The latest
versions of Firefox, Chrome, and Internet Explorer 10+ are all supported.
 Out-of-band Management (optional): One available port that supports 100Mbps for each VxRail node.

Logical Network

Step 1A  One management VLAN with IPv6 multicast for traffic from VxRail, vCenter Server, ESXi
Reserve VLANs (default is untagged/native).
 One VLAN with IPv4 multicast for Virtual SAN traffic.
 One VLAN for vSphere vMotion.
 One or more VLANs for your VM Network(s).

Step 1B  Time zone.

System  Hostname or IP address of the NTP server(s) on your network (recommended).
 IP address of the DNS server(s) on your network (required, except in isolated
environments).
 Optional: IP address, port, username, and password of your proxy server.

Step 1C  Decide on your ESXi host naming scheme.

Management  Reserve four or more contiguous IP addresses for ESXi hosts.
 Decide if you will use a vCenter Server that is external or internal to your new VXRail
cluster.
 Internal vCenter Server: Decide on hostnames for vCenter Server and PSC and reserve
two IP addresses.
 External vCenter Server: Determine PSC, hostname, administration user, and datacenter.
Create a new VxRail management user. Decide on a VxRail cluster name.
 Decide on a hostname and reserve one IP address for VxRail Manager.
 Determine IP address of the default gateway and subnet mask.
 Select a single root password for all ESXi hosts in the VxRail cluster.
 Select a single password for VxRail and vCenter Server.

Step 1D  Reserve four or more contiguous IP addresses and a subnet mask for vSphere vMotion.
vMotion and  Reserve four or more contiguous IP addresses and a subnet mask for Virtual SAN.
Virtual SAN

Step 1E  To use vRealize Log Insight: Reserve one IP address and decide on the hostname.
Solutions  To use an existing syslog server: Get the hostname or IP address of your third-party
syslog server.

Step 1F  Configure your workstation/laptop to reach the VxRail initial IP address.

Workstation  Make sure you also know how to configure it to reach the VxRail Manger IP address after
configuration.

Step 2  Configure your selected management VLAN (default is untagged/native). Confirm that
Set up Switch IPv6 multicast is configured/enabled on the management VLAN (regardless of whether
tagged or native).
 Configure your selected VLANs for Virtual SAN, vSphere vMotion, and VM Networks.
 In multi-switch environments, configure the management and Virtual SAN VLANs to carry
the multicast traffic between switches.
 Confirm configuration and network access.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 24

Step 3  Connect the VxRail ports to the switch ports as decided in the topology.
 Power on all four initial nodes. Do not turn on any other VxRail nodes until you have
Cable & On completed initial configuration.
 Optional: On each ESXi host, customize the management VLAN; otherwise the default is
your switch’s Native VLAN for untagged traffic.

Step 4  Connect a workstation/laptop to access the VxRail initial IP address on the management
Connect & VLAN.
Configure  Browse to the VxRail initial IP address (default https://192.168.10.200); configure & build
VxRail.
 Configure your corporate DNS server for all VxRail hostnames and IP addresses unless you
are in an isolated environment.
 Connect to the VxRail Manager IP address on the management VLAN.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 25

APPENDIX A: CUSTOMIZING THE VXRAIL INITIAL IP ADDRESS
To customize the VxRail initial IP address, follow these instructions to set the IP address, subnet mask, and
gateway for VxRail instead of the default initial address, 192.168.10.200/24.

You do not need to follow these instructions if you can reach the default VxRail initial IP address and merely wish to
change the post-configuration IP address to something else. Instead, use the VxRail initial configuration user
interface to enter the new IP address.

It will be easiest to select the IP settings that you want to use permanently for your VxRail cluster. Then all you
need to do is configure your workstation/laptop once. Otherwise, just follow the VxRail initial configuration user
interface.

Step 1. From your workstation/laptop, connect a VMware vSphere (C#) Client to the IP address of ESXi host #1
using the root user and the password specified during factory ESXi software installation, Passw0rd!
Step 2. Click the Virtual Machines tab and select “VxRail Manager”. The VM should already be powered on. If
not, click the green play button to power it and wait for it to boot.
Step 3. Open the Console and login as root with the default password Passw0rd!
Step 4. Stop vmware-marvin:
/etc/init.d/vmware-marvin stop

Step 5. Using the vami_set_network command, change the default IP address to a custom IP address, subnet
mask, and gateway using the syntax shown below (all arguments are required).
Use the VxRail Network Configuration Table, Row 2 for the <new_IP>, <new_netmask>, and
<new_gateway>.

/opt/vmware/share/vami/vami_set_network eth0 STATICV4 <new_IP> <new_netmask>

<new_gateway>
Step 6. Restart vmware-marvin and vmware-loudmouth on VxRail Manager:
/etc/init.d/vmware-marvin restart
/etc/init.d/vmware-loudmouth restart

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 26

APPENDIX B: NSX SUPPORT ON VXRAIL
VxRail supports VMware NSX software-defined networking (SDN) through vCenter Server. vCenter Server offers a
fully integrated option for SDN and network-layer abstraction with NSX. The NSX network-virtualization platform
delivers for networking what VMware delivers for compute and storage. In much the same way that server
virtualization allows operators to programmatically create, snapshot, delete, and restore software-based virtual
machines (VMs) on demand, NSX enables virtual networks to be created, saved, deleted, and restored on demand
without requiring reconfiguration of the physical network. The result fundamentally transforms the datacenter
network-operational model, reduces network-provisioning time from days or weeks to minutes, and dramatically
simplifies network operations. NSX is a non-disruptive solution that is deployed on any IP network, including
existing datacenter network designs or next-generation fabric architectures from any networking vendor.

With network virtualization, the functional equivalent of a “network hypervisor” reproduces the complete set of
Layer 2 to Layer 7 networking services (e.g., switching, routing, access control, firewalling, QoS, and load
balancing) in software. Just as VMs are independent of the underlying x86 hardware platform and allow IT to treat
physical hosts as a pool of compute capacity, virtual networks are independent of the underlying IP network
hardware and allow IT to treat the physical network as a pool of transport capacity that can be consumed and
repurposed on demand.

NSX coordinates ESXi’s vSwitches and the network services pushed to them for connected VMs to effectively
deliver a platform—or “network hypervisor”—for the creation of virtual networks. Similar to the way that a virtual
machine is a software container that presents logical compute services to an application, a virtual network is a
software container that presents logical network services—logical switches, logical routers, logical firewalls, logical
load balancers, logical VPNs and more—to connected workloads. These network and security services are delivered
in software and require only IP packet forwarding from the underlying physical network.

To connected workloads, a virtual network looks and operates like a traditional physical network. Workloads “see”
the same Layer 2, Layer 3, and Layers 4-7 network services that they would in a traditional physical configuration.
It’s just that these network services are now logical instances of distributed software modules running in the
hypervisor on the local host and applied at the vSwitch virtual interface.

The following NSX components are illustrated in Figure 4:

 NSX vSwitch operates in ESXi server hypervisors to form a software abstraction layer between servers and
the physical network.

 NSX Controller is an advanced, distributed state management system that controls virtual networks and
overlays transport tunnels. It is the central control point for all logical switches within a network and maintains
information of all virtual machines, hosts, logical switches, and VXLANs.

 NSX Edge provides network-edge security and gateway services to isolate a virtualized network. You can
install NSX Edge either as a logical (distributed) router or as a services gateway.

 NSX Manager is the centralized network management component of NSX, installed as a virtual appliance on
an ESXi host.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 27

Figure 5. NSX component information flow: NSX Manager, NSX
Controller, NSX Edge, NSX vSwitch
One NSX Manager maps to a single vCenter Server and to multiple NSX Edge, vShield Endpoint, and NSX Data
Security instances. Before you install NSX in your vCenter Server environment, consider your network
configuration and resources using the chart below.

NSX Resource Requirements:

Memory Disk Space vCPU

NSX Manager 12GB 60GB 4

NSX Edge:
 Compact 512MB 512MB 1
 Large 1GB 512MB 2
 Extra Large 8GB 4.5GB (with 4GB swap) 6
 Quad Large 1GB 512MB 4
vShield Endpoint 1GB 4GB 2
NSX Data Security 512MB 6GB per ESXi host 1

In a VxRail cluster, the key benefits of NSX are consistent, simplified network management and operations, plus
the ability to leverage connected workload mobility and placement. With NSX, connected workloads can freely
move across subnets and availability zones. Their placement is not dependent on the physical topology and
availability of physical network services in a given location. Everything a VM needs from a networking perspective
is provided by NSX, wherever it resides physically. It is no longer necessary to over-provision server capacity
within each application/network pod. Instead, organizations can take advantage of available resources wherever
they’re located, thereby allowing greater optimization and consolidation of resources. VxRail easily inserts into
existing NSX environments and provide NSX awareness so network administrators can leverage simplified network
administration. See the VMware NSX Design Guide for NSX best practices and design considerations.

For additional information related to NSX, refer to the following materials:

 VMware NSX Network Virtualization Platform Technical White Paper at

http://www.vmware.com/files/pdf/products/nsx/VMware-NSX-Network-Virtualization-Platform-WP.pdf

 Reference Design Guide: Vmware NSX for vSphere at https://www.vmware.com/files/pdf/products/nsx/vmw-

nsx-network-virtualization-design-guide.pdf

 Hyperconverged Transformation White Paper at http://www.vce.com/asset/documents/esg-whitepaper-

hyperconverged-transformation-sddc.pdf

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 28

APPENDIX C: JSON CONFIGURATION FILE
Before configuring VxRail, customers must read this setup guide and fill out the VxRail Network Configuration
Table. A JSON Configuration file can be uploaded in the VxRail initial configuration user interface.

Important Notes:

 The JSON file format may change throughout VxRail releases. Please get the sample JSON file that
corresponds to the software release that your VxRail nodes were built with at the factory; then edit the sample
file for your configuration.
 VxRail expects the data in the configuration file in a specific format. Any changes to the JSON format will result
in unexpected results and/or crashes.
 Use the VxRail Pre-Installation Site Checklist to automatically generate the JSON file.

Create a custom configuration file with the following steps:

Step 1. Obtain a sample json file for the VxRail release that you will be configuring.
Step 2. Edit your configuration file to insert the values from the VxRail Network Configuration Table.
Step 3. Make sure that the filename has a “.json” extension.
Step 4. Make sure that the file is in valid JSON format because VxRail will not validate the syntax (e.g., a
missing comma will cause the configuration file to fail). VxRail will validate the content of a correctly
formatted JSON file in the same manner that it validates manual entries, verifying data entry and
performing deep validation prior to building the cluster.
Step 5. Make this file accessible from your workstation/laptop.
Deploy VxRail as usual by configuring your switch, racking and cabling the nodes, and then powering on all four
initial VxRail nodes.

Step through the Initial Configuration User Interface section to upload your JSON configuration file.

JSON File Format and Valid Values

The JSON configuration file must be properly formatted and the values must be valid for VxRail and for your
network. The following list contains the fields and restrictions, both color-coded and in list format.

1. Variables in red can be replaced with custom names or IP addresses. All red fields are required.
o minIP, maxIP, ip, gateway, netmask: valid IP addresses and subnet mask in your network
o vlanId: valid numeric VLAN ID, configured on your switch
o name: alphanumeric string to identify a VM network segment. The number of VM Network segments is
not limited; please add more if necessary.
o prefix: alphanumeric string for the first part of an ESXi hostname
o tld: valid domain name in your network.
o evorail: alphanumeric string for the VxRail Manager hostname

2. Fields in purple contain multiple options. All purple fields are required.
separator: “” (no separator) or “-“ (dash)
o The general formula for the FQDN (fully qualified domain name) of an ESXi host is:
<hostname><separator><iterator>.<domain>
 When using “-“ as the separator, the FQDN of an ESXi host is:
<hostname>-<iterator>.<domain> (i.e. host-01.vsphere.local)
 When using “” as the separator, the FQDN of an ESXi host is:
<hostname><iterator>.<domain> (i.e. host01.vsphere.local)

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 29

 iterator: “NUMERIC_N” or “NUMERIC_NN” or “ALPHA”
o ALPHA means that the first host starts with A, the second host with B, etc
o NUMERIC_N means that the first host starts with “1”, the second host with “2”, etc
o NUMERIC_NN means that the first host starts with “01”, the second host with “02”, etc

timezone:
o Any value listed in http://en.m.wikipedia.org/wiki/List_of_tz_database_time_zones in the TZ column is
accepted as valid input.

logging: “LOGINSIGHT” or “SYSLOG” or “NONE”

o LOGINSIGHT means that Log Insight will be used as the log collection server. When this option is used,
loginsightServer and loginsightHostname must be filled out; otherwise, they must be left blank
(shown in yellow).
o SYSLOG means that an external log collection server will be used as the log collection server. When this
option is used, syslogServerCSV must be filled out; otherwise, it must be left blank (shown in brown). If
used, up to two IP addresses (or FQDNs) are supported in this field.
o NONE means that a log collection server will not be configured for VxRail. The loginsightServer,
loginsightHostname, and syslogServerCSV fields must be left blank.

3. Fields in blue are related to setting up and internal or external vCenter Server, as described in this list.
global.joinVC: “false” or “true”
o False - if you want to deploy a new vCenter Server. The externalVC* fields must be blank, and the
network.vcenter, network.psc, hostnames.vcenter, and hostnames.psc must be filled out.
o True - if you want to join an existing vCenter Server. The externalVC* fields must be filled out, and the
network.vcenter, network.psc, hostnames.vcenter, and hostnames.psc must be blank.

dnsServerCSV: a comma-separated list of IP address(es) for external DNS server(s). At least one DNS server
is required if joinVC is True. If you are using an internal vCenter Server, an external DNS server is required
except in isolated environments.

hostnames.vcenter, hostnames.psc: FQDN (alphanumeric strings) for internal vCenter Server and Platform
Services Controller (PSC) hostnames that must be filled out if joinVC is false; otherwise, they must be blank.

network.vcenter, network.psc: valid IP addresses for internal vCenter Server and PSC that must be filled
out if joinVC is false; otherwise, they must be blank.

externalVC.vcenter, externalVC.vcUsername, externalVC.managementUsername,

externalVC.datacenterName, externalVC.clusterName: alphanumeric strings that must be filled out when
joinVC is true; otherwise, they must be blank.

externalVC.nonEmbeddedMode: “false” or “true” (required when joinVC is True)

o False - if you want to join a vCenter Server with an embedded PSC. The externalVC.psc field must be
blank.
o True - if you want to join a vCenter Server with a non-embedded (external) PSC. The externalVC.psc
alphanumeric strings field must be filled out.
4. Fields in green are optional. If the field is not used, it should be left unfilled with just opening and closing
quotes, i.e. “”.

o proxyUsername: alphanumeric string

o proxyServer, proxyPort: IP address and port identifier

o ntpServerCSV: a comma-separated list of IP addresses or hostnames

5. Fields that contain passwords should only be filled out by a customer during VxRail initial configuration. They
should not be pre-filled in the clear-text JSON file for security reasons.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 30

6. In the event that you see a JSON file from a previous release of VxRail, the following fields are not used in
Release 3.5.0: network.hosts.management.vlandId, activeDirectory*, vendor.*, vmwSolutions.*. They have
been removed from the example shown below. They will be ignored by VxRail if they appear in your JSON file.

7. Do not modify the JSON syntax.

The following illustrates the JSON file in release 3.5.0:

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 31

{
"version": "3.5.0",
"network": {
"dhcp": false,
"hosts": {
"management": {
"pools": [
{
"minIp": "192.168.10.1",
"maxIp": "192.168.10.4"
}
],
"netmask": "255.255.255.0",
"gateway": "192.168.10.254"
},
"vsan": {
"pools": [
{
"minIp": "192.168.30.1",
"maxIp": "192.168.30.4"
}
],
"netmask": "255.255.255.0",
"vlanId": 212
},
"vmotion": {
"pools": [
{
"minIp": "192.168.20.1",
"maxIp": "192.168.20.4"
}
],
"netmask": "255.255.255.0",
"vlanId": 211
},
"vm": [
{
"name": "VM_Network_1",
"vlanId": 213
},
{
"name": "VM_Network_2",
"vlanId": 214
}
]
},
"vcenter": {
"ip": "required_if_joinVC_is_false_otherwise_it_must_be_blank"
},
"psc": {
"ip": "required_if_joinVC_is_false_otherwise_it_must_be_blank"
},
"evorail": {
"ip": "192.168.10.201"
}
},
"hostnames": {
"hosts": {
"prefix": "esxi-vxrail-12",
"separator": "-",
"iterator": "NUMERIC_NN"
},
"vcenter": "required_if_joinVC_is_false_otherwise_it_must_be_blank",
"evorail": "manager-vxrail-12",
"psc": "required_if_joinVC_is_false_otherwise_it_must_be_blank",
"tld": "my-company.com"
},
"passwords": {
"esxiPassword": "",
"esxiPasswordConfirm": "",
"vcPassword": "",

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 32

 "vcPasswordConfirm": "",
},
"global": {
"timezone": "UTC",
"ntpServerCSV": "optional_leave_blank_if_not_needed",
"dnsServerCSV": "required_if_joinVC_is_true",
"proxyServer": "optional_leave_blank_if_not_needed",
"proxyPort": "optional_leave_blank_if_not_needed",
"proxyUsername": "optional_leave_blank_if_not_needed",
"proxyPassword": "",
"loginsightHostname": "required_if_logging_is_LOGINSIGHT-otherwise_it_must_be_blank",
"loginsightServer": "required_if_logging_is_LOGINSIGHT-otherwise_it_must_be_blank",
"syslogServerCSV": "required_only_if_logging_is_SYSLOG-otherwise_it_must_be_blank",
"logging": "LOGINSIGHT",
"joinVC": false
},
"externalVC": {
"nonEmbeddedMode": false,
"psc": "required_if_joinVC_and_nonEmbeddedMode_are_true_otherwise_it_must_be_blank",
"vcenter": "required_if_joinVC_is_true_otherwise_it_must_be_blank",
"vcUsername": "required_if_joinVC_is_true_otherwise_it_must_be_blank",
"vcPassword": "required_if_joinVC_is_true_otherwise_it_must_be_blank",
"managementUsername": "required_if_joinVC_is_true_otherwise_it_must_be_blank",
"managementPassword": "required_if_joinVC_is_true_otherwise_it_must_be_blank",
"datacenterName": "required_if_joinVC_is_true_otherwise_it_must_be_blank",
"clusterName": "required_if_joinVC_is_true_otherwise_it_must_be_blank"
}
}

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 33

ABOUT VCE
VCE, an EMC Federation Company, is the world market leader in converged infrastructure and converged solutions. VCE
accelerates the adoption of converged infrastructure and cloud-based computing models that reduce IT costs while improving
time to market. VCE delivers the industry's only fully integrated and virtualized cloud infrastructure systems, allowing customers
to focus on business innovation instead of integrating, validating, and managing IT infrastructure. VCE solutions are available
through an extensive partner network, and cover horizontal applications, vertical industry offerings, and application development
environments, allowing customers to focus on business innovation instead of integrating, validating, and managing IT
infrastructure.
For more information, go to vce.com.

Copyright © 2010-2016 VCE Company, LLC. All rights reserved. VCE, VCE Vision, VCE Vscale, Vblock, VxBlock, VxRack, VxRail, and the VCE logo are registered
trademarks or trademarks of VCE Company LLC. All other trademarks used herein are the property of their respective owners.

© 2016 VCE COMPANY, LLC. ALL RIGHTS RESERVED 34