INF-NET2162

VXLAN Deep Dive

Arun Goel, VMware, Inc.

Kenneth Duda, Arista Networks

#vmworld (hashtag here)

Disclaimer

 This session may contain product features that are

currently under development.
 This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
 Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
 Technical feasibility and market demand will affect final delivery.
 Pricing and packaging for any new technologies or features
discussed or presented have not been determined.

2
Why Overlays (VXLAN)?

Web Tier needs to expand?

Web Capacity ?

IP Address ?

Network ?
App

DB

Compute Cluster A Compute Cluster B

3
Why Overlays (VXLAN)?

VXLAN Fabric

Web Web

App

DB

Compute Cluster A Compute Cluster B

VXLAN disassociates workloads from physical networks, allowing for

possible transition to cloud based providers
4
Use cases

 Increase compute utilization by pooling clusters

 Allow non contiguous cluster expansion

 Allow availability domains within a DC

 Leverage capacity across multiple L2 POD’s

 Overcome IP addressing challenges

 VLAN sprawl

 VLAN scale
5
 Physical Topology

6
 Deployment Model : Existing L2 or New L2 Access Layer

Optionally
Enable PIM
Aggregation Aggregation
vPC/MLAG
STP
Setup a single Subnet

Enable IGMP
Access Access

vSphere Distributed Switch Deploy VDS vSphere Distributed Switch

Edge Edge Edge Edge

Deploy VXLANs
VXLAN VXLAN VXLAN VXLAN

7
Deployment Model : Existing L3 or New L3 Access Layer

Aggregation

ECMP Enable PIM

Access

vSphere Distributed Switch Deploy VDS

Edge Edge Deploy VXLANs

VXLAN VXLAN

8
Deployment Model : Existing L3 or New L3 Access Layer

Aggregation

Deploying VXLANs is Very Easy

Enable PIM ECMP

Irrespective of Underlying Physical

Access
Network Topology
vSphere Distributed Switch Deploy VDS

Edge Edge Deploy VXLANs

VXLAN VXLAN

9
 Logical Topology

10
Use Case Enabler #1 : Logical Layer 2 across L2

VM VM
VM VM

VDS

Host VLAN 20 Host VLAN 20

Switch
L2 Pod

11
Use Case Enabler #2 : Logical Layer 2 across L3

VM VM
VM VM

VDS VDS

Physical Physical
Switch Switch

L2 Pod L2 Pod

DC
Router

12
 VXLAN Details

13
Virtual Extensible Local Area Network (VXLAN)

 Ethernet in IP overlay  Tunnel between ESX hosts

network • VMs do NOT see VXLAN ID
• Entire L2 frame encapsulated in  IP multicast used for L2
UDP broadcast/multicast, unknown
• 50 bytes of overhead unicast
 Include 24 bit VXLAN Identifier  Technology submitted to IETF
• 16 M logical networks for standardization
• With Cisco, Citrix, Red Hat, Broadcom,
 VXLAN can cross Layer 3 Arista, and Others

Outer Outer Inner Inner Optional Original

Outer Outer Outer Outer VXLAN ID
MAC MAC MAC MAC Inner Ethernet CRC
802.1Q IP DA IP SA UDP (24 bits)
DA SA DA SA 802.1Q Payload

VXLAN Encapsulation Original Ethernet Frame

14
VXLAN HOW – Initial Setup

VM1 VM2
MAC1 MAC2

ESX1 ESX 2
VTEP1 VTEP2

IGMP report for joining

IGMP report for joining
239.119.1.1
239.119.1.1

L2/L3 network
infra

VM1 and VM2 on VXLAN ID100, and VXLAN form the topology by joining mcast group 239.119.1.1

VTEP – Virtual Tunneling End Point

15
 VXLAN protocol – ARP Query

VM1 VM2
MAC1 MAC2
1 BCAST MAC1 ARP Req 5 BCAST MAC1 ARP
Req
4 Learn inner src MAC/ outer src IP
mapping
ESX1 NetID MAC IP
NetID MAC IP ESX 2
VTEP1 VTEP2
100 MAC1 IP1_vtep1

2 For original bcast, encap w/ ip 3 Encapped pkt comes in via mcast

mcast
MAC Hdr IP Hdr UDP Hdr VXLAN Hdr BCAS MAC1 ARP MAC Hdr IP Hdr UDP Hdr VXLAN Hdr BCAS MAC1 ARP
DA:239.119.1.1 VXLAN ID: T Req DA:239.119.1.1 VXLAN ID: T Req
SA:IP_vtep1 100 SA:IP_vtep1 100

L2/L3 network
infra

VM1 sends ARP request (BCAST) in VXLAN ID 100 to get VM2’s MAC
VTEP – Virtual Tunneling End Point

16
VXLAN Protocol – ARP Response

VM1 VM2
MAC1 MAC2
4 MAC1 MAC2 ARP 1 MAC1 MAC2 ARP
Resp Resp
3 Learn inner src MAC/ outer src IP
mapping
ESX1 NetID MAC IP
NetID MAC IP ESX 2
VTEP1 100 MAC1 IP_vtep1 VTEP2
100 MAC2 IP_vtep2

2 MAC1 is learnt, encap w/ IP unicast

MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 ARP MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 ARP
DA:IP_vtep1 VXLAN ID: Resp DA:IP_vtep1 VXLAN ID: Resp
SA:IP_vtep2 100 SA:IP_vtep2 100

L2/L3 network
infra

VM2 sends ARP reply to respond VM1’s ARP request

VTEP – Virtual Tunneling End Point

17
VXLAN Protocol – Gateway

Gateway MAC1 MAC2 Data VM2

MAC1 MAC2
MAC1 MAC2 Data

ESX1 ESX 2
VTEP1 VTEP2

MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 Data MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 Data
DA:IP_vtep1 VXLAN ID: DA:IP_vtep1 VXLAN ID:
SA:IP_vtep2 100 SA:IP_vtep2 100

L2/L3 network
infra

18
VXLAN Protocol – Gateway

Gateway VM2
MAC1 MAC2
MAC1 MAC2 Data

ESX1 ESX 2
VTEP1 VTEP2

IP Hdr SW MAC1 Data

DA: Yahoo L2/L3 network
MAC
SA: GW IP
infra

19
VXLAN Logical View – 2 VMs in same VXLAN

VM VM
VXLAN
192.168.1.10 192.168.1.11 192.168.1.0/24

192.168.1.1
Gateway

172.26.10.10

External Network
172.26.10.0/24

Internet

VM to VM communication
VM to Internet communication

20
VXLAN Physical View – 2 VMs in same VXLAN
192.168.1.10
Gateway

VM 192.168.1.1 VM
192.168.1.11

172.26.10.10
VTEP VTEP

VDS VDS

ESX Host ESX Host

VLAN 10 L3 Switch
VLAN 20

Router

Internet

VM to VM communication
VTEP – Virtual Tunneling End Point
VM to Internet communication

21
VXLAN Logical View – 2 VMs in different VXLAN

VM VM

192.168.1.10 192.168.2.10

VXLAN Blue VXLAN Purple

192.168.1.0/24 192.168.2.0/24
192.168.1.1 Gateway 192.168.2.1

172.26.10.10

External Network
172.26.10.0/24

Internet

VM to VM communication between 2 VXLANs

22
VXLAN Ecosystem: Performance, Visibility & Interoperability

VMware
Edge
Visbility

VXLAN Overlay
L2
Network Gateway
Service

Physical IP Network

Server
offloads

23
Summary

 VXLAN disassociates workloads from Physical Network, allowing

mobility across clusters or clouds in future

 VXLAN enables deploying networks and services in minutes

 VXLAN is built on a rich partner ecosystem

24
 The New Standard in Data Center Switching

The Role of the Physical Switch in a

VXLAN Environment
 Use Case: Hosting

10.10.10.3

10.10.11.4
128.218.12.3
128.218.12.4
128.218.12.5
128.218.12.6
128.218.12.7 128.218.12.8 128.218.12.9

Physical Hosting Virtual Hosting

 Use Case: Bridging Virtual to Physical

VNI MIT VNI CalBears

VNI Auburn

Any port, any workload, any service, any where...

 Automated Learning of Multicast/VNI State
vSphere 5

New VNI - CalBears

Multicast Group - 224.0.14.13
VNI ID - 650782

Interface Ethernet 24
VXLAN VTEP VNI CalBears

Interface Loopback0
VXLAN VTEP Gateway VNI Calbears
IP Address 204.181.40.1/24

<--Network

VM- Oski
VNI - CalBears
Incast SLA Assurance

Poor Service
Solution: VNI Segregation

1. Switch detects congestion (LANZ)

2. Switch samples traffic

3. Switch determines VNI RED is bad

4. Switch installs class-map mapping

RED to a separate queue
How do we
troubleshoot a
tunneled,
encapsulated,
multicast
environment???
VM Tracer for VXLAN
Full physical to
virtual visibility

Network audit to
ensure reachability

Automated provisioning

Workflow without
finger pointing

Other awesome
capabilities
 Visibility - so where is my VM now?

spine0
leaf1 leaf2

esx10 esx11
VNI ‘Test’: 224.0.0.12

Aubie WarEagle vshield vm-tiger

 Visibility - so where is my VM now?
spine0: show vmtracer interface vxlan Auburn

VTEP: ESX1 Role: vSwitch Switch/Port: ar16.foo.com/eth15

Name VNIC Status State IP Address
Aubie Network Interface 1 Up/Up vMotion 204.181.40.2
WarEagle Network Interface 2 Up/Up VM-FT-A 204.181.40.3
BooBama Network Interface 1 Up/Down -- 204.181.40.5

VTEP: ar24 Role: Router

spine0
Switch/Port: ar24.foo.com/loopback0
NAT/PAT leaf1 Status #ARPs IP Address leaf2
No Up/Up 45 204.181.40.1

VTEP: ar22 Role: Port-VTEP Switch/Port: ar22.foo.com/eth2

FQDN IP MAC VLAN Status
isilon16.foo.com 204.181.40.190 00-00-45-ab-12-fe 5 Up/Up

128.218.10.x 128.218.11.x

esx1 esx11
VNI ‘Test’: 224.0.0.12

Aubie WarEagle vshield vm-tiger

 Visibility - so where is my VM now?
spine0: show vmtracer interface vxlan Auburn

VTEP: ESX1 Role: vSwitch Switch/Port: ar16.foo.com/eth15

Name VNIC Status State IP Address
Aubie Network Interface 1 Up/Up vMotion 204.181.40.2
WarEagle Network Interface 2 Up/Up VM-FT-A 204.181.40.3
BooBama Network Interface 1 Up/Down -- 204.181.40.5

VTEP: ar24 Role: Router

spine0
Switch/Port: ar24.foo.com/loopback0
NAT/PAT leaf1 Status #ARPs IP Address leaf2
No Up/Up 45 204.181.40.1

VTEP: ar22 Role: Port-VTEP Switch/Port: ar22.foo.com/eth2

FQDN IP MAC VLAN Status
isilon16.foo.com 204.181.40.190 00-00-45-ab-12-fe 5 Up/Up

128.218.10.x 128.218.11.x

esx1 esx11
VNI ‘Test’: 224.0.0.12

Aubie WarEagle vshield vm-tiger

Summary
Arista simply provides the best network for
VMware

Native vSphere integration into the physical

network to automate, and operationalize the
infrastructure

Simplified systems architecture to reduce

complexity and improve TCO

Optimized power and efficiency to maximize

compute density and enable profitable public, or
cost-effective private cloud deployments

Co-authors and leading innovators of VXLAN to

enable stateful L3 vMotion
 Thank You!!

Q&A
arungoel@vmware.com
kduda@aristanetworks.com

37
Other Networking & Security Sessions to attend

 INF-NET1927: VMware vCloud® Networking: an Extensible and

Open Platform

 INF-NET1932: On Demand Virtual Networking with vShield Edge

 INF-NET2161: Vmware Networking 2012: Enabling the Software

Defined Network

 INF-NET2166: Leaders Series: “How I Build My SDN-based Cloud”

 INF-NET2965: Leaders Series: "I've Improved IT delivery with

vCloud Networking"

38
 FILL OUT
A SURVEY

EVERY COMPLETE SURVEY

IS ENTERED INTO
DRAWING FOR A
$25 VMWARE COMPANY
STORE GIFT CERTIFICATE
 INF-NET2162

VXLAN Deep Dive

Arun Goel, VMware, Inc.

Kenneth Duda, Arista Networks

#vmworld (hashtag here)