CNA2080BU

Deep Dive: How to Deploy

t i o n
and Operationalize i s tr ibu
or d
Kubernetes t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
Cornelia Davis, Pivotal V M
Nathan Ness
Technical Product Manager, CNABU
@nvpnathan

#VMworld #CNA2080BU
Disclaimer
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product. t i o n
tr ibu
r dis purchase orders, or
• Features are subject to change, and must not be included in contracts,
o
sales agreements of any kind.
a t i on
c u b li
• Technical feasibility and market demand will affect o p
r final delivery.
o t f
• n
Pricing and packaging for any new technologiest : N or features discussed or presented have not
ont e
been determined.
17 C
2 0
o r ld
VMw

#CNA2080BU CONFIDENTIAL 2
Agenda

1 What is the need?

t i o n
i s tr ibu
or d
2 Introducing the toolchain
t ion
bli c a
r p u
3
o t fo
Pivotal Container Service (PKS)
n t : N
4 onte1
PKSCDay
7
r ld 201
Mw5o PKS Day 2
V

#CNA2080BU CONFIDENTIAL 3
 t i o n
The Need for Operationalizing i s tr ibu
o r d
Kubernetes t io n
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Companies Have Many Ways to Package and Run Their
Workloads in the Cloud

t i o n
i s tr ibu
or d
t ion
CONTAINERS
bli c a
p u
BATCHES EVENT-DRIVEN
t f or FUNCTIONS
:N o
t e n t
C on
1 7
r ld 20
Mw o
V
MICROSERVICES MONOLITHIC
DATA SERVICES APPLICATIONS

#CNA2080BU CONFIDENTIAL 5
Workloads that Might Be Suitable for Kubernetes
Those:
• Requiring Persistence
t i o n
– MongoDB, CouchDB, Couchbase, Elastic Search, …
i s tr ibu
• Managed as a cluster or d
t ion
– nodes need to communicate with one another
bli c a
r p u
o t fo
– often with the help of service meshes such as Istio or Linkerd
– Spark, Elastic Search nt: N
o n te
• 1 7 C
Needing new architectural primitives
d 2 0
• w o rl ports, etc.
Misc things like multiple
VM

#CNA2080BU CONFIDENTIAL 6
Serving up Kubernetes Dial-tone

Kubernetes

t i o n
i s tr ibu
kubectl
o r
Routing d
Master n
Master
b licatio
rp u
t f o
:N o
t
Responsible for the

t e n K8s cluster(s)

on
Responsible for the
themselves
C
workloads running
in K8s
1 7
ld
etcd
r 20 Worker
Worker
Mw o etcd
etcd
Worker
V manage

#CNA2080BU CONFIDENTIAL 7
Operational Challenges with Any Platform
Day 1 - Build Day 2 - Operate
Multi-cloud Provide a reliable and smooth Patches Patching platform components with
experience for any cloud. tion feel normal.
thousands of apps runningushould
b
i s t ri
Open APIs Allow platform operations from rd
n o scale platform components to
i o
Scaling Seamlessly
a t changing demand.
different toolsets and the creation of CD pipelines.
ub l i c
accommodate
o r p
Notf
Consistency Provide a consistent setup
e n t : Upgrades How do you roll out new versions of the
C ont
experience, across different cloud environment platform with the lights on?
configurations.
d 2017
or l
w
VMdoes it take to setup a real Operating Effort Operating a platform should
Setup time How long
world working environment? Think hours, not require very few resources and minimum manual
weeks. intervention. Otherwise, is it really providing
operational benefits?

#CNA2080BU CONFIDENTIAL 8
Kubernetes - Especially Hard to Operationalize
High Availability. No out-of-the-box fault-tolerance
for the cluster components themselves (masters,
workers and etcd nodes).
i tion
b u
t r
Scaling. Kubernetes r dis
n oclusters handle scaling the
a i o
t the Nodes, but doesn’t provide a
pod/service
u b l i cwithin
p
o t for
mechanism to scale Workers, Masters & etcd VMs.
nt: N
o n te Health checks and healing. The Kubernetes cluster
17 C
2 0 only does routine health checks for the health of
w orld workloads running on Nodes.
V M
Upgrades. Rolling upgrades on a large fleet of
clusters is hard. Who manages the system it runs
on?

#CNA2080BU CONFIDENTIAL 9
 t i o n
i s tr ibu
or d
Introducing BOSH t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Powered by BOSH
Pivotal container service ops
Packaging w/ embedded OS
BOSH t
Server provisioning on any IaaS i o n
turib
rd i s
o across availability zones
ion
Software deployment
c a t
BOSH is an open source tool ub l i
o r p monitoring (server AND processes)
o t fHealth
for release engineering, n t : N
o n te Self-healing w/ Resurrector
deployment, lifecycle 17 C
2 0
o r ld Storage management
management,VM wand monitoring
of distributed systems. Rolling upgrades via canaries

Easy scaling of clusters

#CNA2080BU CONFIDENTIAL 11
Powered by BOSH
Pivotal container service ops
Kubernetes Packaging w/ embedded OS
Master
t
Server provisioning on any IaaS i o n
Master
turib
rd i s
o across availability zones
ion
Software deployment
c a t
ub l i
o r p monitoring (server AND processes)
etcd Worker
Worker o t fHealth
etcd
etcd Worker
nt: N
o n te Self-healing w/ Resurrector
17 C
2 0
w o rld Storage management
V M
BOSH
Rolling upgrades via canaries

Easy scaling of clusters

#CNA2080BU CONFIDENTIAL 12
Primary BOSH Entities
The definition of each of the nodes in the
cluster, including:
Kubernetes •TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
cluster, including: started on a node (jobs)
• The processes
Master •• The t i o n
ibu
Master bits installed on a node (packages)
Parameterized
s tr
• The processes started on a node (jobs)
i
or d BOSH release
t ion
bli c a
r p uA declaration of the desired state of the
etcd Worker
Worker o t fo cluster:
etcd
etcd Worker
nt: N • Assembly of the components from BOSH
C onte releases (relationships, dependencies)
1 7
r ld 20 • Parameter values

Mw o BOSH deployment
V BOSH
Relationship to the underlying infrastructure
BOSH cloud config

#CNA2080BU CONFIDENTIAL 13
The Workflow
The definition of each of the nodes in the
cluster, including:
Kubernetes •TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
cluster, including: started on a node (jobs)
• The processes
Master •• The t i o n
ibu
Master bits installed on a node (packages)
Parameterized
s tr
• The processes started on a node (jobs)
i
or d BOSH release
STEP a2:tio n and Manage Kubernetes
Install
u b l ic
Worker
f or p A declaration of the desired state of the
etcd Worker o t cluster:
etcd
etcd Worker
e n t:N • Assembly of the components from BOSH
on t
1 7 C releases (relationships, dependencies)

r ld 20 • Parameter values

Mw o BOSH deployment
V BOSH
Relationship to the underlying infrastructure
STEP 1: Install and configure BOSH
BOSH cloud config

#CNA2080BU CONFIDENTIAL 14
 t i o n
i s tr ibu
or d
Pivotal Container Service (PKS) a t i on
blic p u
fo r
N o t
n tent:
C o
0 17
rld 2
Mw o
V
Project Kubo

“Day 1” Build t i o n
Uniform way to instantiate, deploy, and t r
● Deploy Kubernetes i b ucluster via BOSH
is
r d
manage highly available Kubernetes on o
cia tOperate
l
“Day i 2”
clusters. On any cloud. r pu●b Self-healing VMs and monitoring via
o t fo
n t : N BOSH
ont e ● Elastic scaling for clusters
Launched by Pivotal & Google
1 7 CFeb
d 2 0 ● Rolling upgrades to latest Kubernetes
w o rl
2017, Donated to Cloud Foundry release
VM2017
Foundation June ● High-availability and multi-AZ support

#CNA2080BU CONFIDENTIAL 16
Kubo Provides Specification of K8S Components
Kubernetes
This forms the
Open Core
of Master
Pivotal Container Master
t i o n
Kubo Release Service
i s tr ibu
(PKS)
or d
t ion
Release
bli c a
templates
etcd r p u Worker
o t
etcdfo Worker
Worker
nt: N etcd

o n te
1 7 C
2 0
w o rld
V M bosh deploy
Manifest BOSH

17
#CNA2080BU CONFIDENTIAL
 Kubernetes Dial Tone:
• Health management
• Aggregated Metricsuand t i onLogging
Provides the control plane i s t r ib
• Autoscalingr d
for provisioning and o n o
i
u licat
• Persistence
b
interface
managing Kubo releases or p
o t f
e n t:N Control Plane:
on t • Provisioning Engine
1 7 C
Joint development 2 0
effort
o r ld • Self-service Clusters
VMw
between Pivotal, VMWare • Software Update Automation
and Google • Load balancing
• Networking
• Multi-tenancy
#CNA2080BU CONFIDENTIAL 18
PKS Leverages the Power of BOSH
Kubo Release

Release
templates
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
Manifest
nt: N
o n te
17 C
2 0
w orld
PKS

V M
BOSH

19
 t i o n
i s tr ibu
or d
Kubernetes Cluster – Day 1 t ion
bli c a
r p u
Deploy
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Starting with a BOSH Deployment...
The definition of each of the nodes in the
cluster, including:
Kubernetes
•TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
•cluster, including: started on a node (jobs)
Master
The processes
•• The t i o n
ibu
Master bits installed on a node (packages)
Parameterized
i s tr
• The processes started on a node (jobs)
or d BOSH release
t ion
bli c a
r p uA declaration of the desired state of the
etcd Worker
Worker o t fo
etcd
etcd Worker
nt: N cluster:
• Assembly of the components from BOSH
C onte releases (relationships, dependencies)
1 7
r ld 20 • Parameter values

Mw o BOSH deployment
V BOSH

#CNA2080BU CONFIDENTIAL 21
 Deploying a Kubernetes Cluster with Cloud Foundry BOSH

DB
t i o n
Deploy my
BOSH Director
i s tr ibuWorker VMs
K8s
or d
t ion
Blobs
bli c a
r p u
otfo etcd

n t : NMessage Bus
Deployment
ont e Master
Target VM
• Packages 1
Health Monitor7 C
d 2 0 Worker
Target VM
• Blobs
w o rl
• Source VM Target VM

• Jobs
• Manifest BOSH vSphere

#CNA2080BU CONFIDENTIAL 22
 t i o n
i s tr ibu
or d
Kubernetes Cluster – Day 2 t ion
bli c a
r p u
Operationalize
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Day 2: Operationalize

t i o n
1 Managing Health i s tr ibu
or d
t ion
bli c a
2 Scaling
r p u
o t fo
nt: N
3 Upgrade
o n te
17 C
2 0
w orld
V M
K8s Cluster Health: Processes are Monitored

AGENT

t i o n
Health Monitor
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
…
r ld 20
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 25
K8s Cluster Health: Processes are Monitored

AGENT

t i o n
Health Monitor
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
…
r ld 20
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 26
K8s Cluster Health: Processes are Monitored

AGENT

t i o n
Health Monitor
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
…
r ld 20
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 27
K8s Cluster Health: VMs are Monitored

Desired State Actual State AGENT

t i o n
Health Monitor BOSH Director
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
ressurector
r ld 20
…
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 28
K8s Cluster Health: VMs are Monitored

Desired State Actual State AGENT

t i o n
Health Monitor BOSH Director
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
ressurector
r ld 20
…
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 29
K8s Cluster Health: VMs are Monitored
CPI
Desired State AGENT
Actual State
t i o n
Health Monitor BOSH Director
i s tr ibu Master
or d
Responses: t ion
bli c a
pager
r p u AGENT

t f o
email
:N o
Message Bus
t e n t etcd
monitoring
C on
1 7
ressurector
r ld 20
…
Mw o AGENT

V
Worker

BOSH vSphere

#CNA2080BU CONFIDENTIAL 30
Day 2: Operationalize

t i o n
1 Managing Health i s tr ibu
or d
t ion
bli c a
2 Scaling
r p u
o t fo
nt: N
3 Upgrade
o n te
17 C
2 0
w orld
V M

#CNA2080BU CONFIDENTIAL 31
Primary BOSH Entities The definition of each of the nodes in the
cluster, including:
•TheThe
definition of eachonofathe
bits installed nodes
node in the
(packages)
Kubernetes cluster, including: started on a node (jobs)
• The processes
Master •• The bits installed on a node (packages)
Master Parameterized
• The processes started on a node (jobs)t i o n
i s tr ibu
BOSH release
or d
t ion
Worker bli c a
A declaration of the desired state of the
etcd Worker
r p u
cluster:
etcd
etcd Worker
o t fo • Assembly of the components from BOSH
nt: N
o n te releases (relationships, dependencies)
• Parameter values
17 C
2 0
w orld BOSH deployment
M
V BOSH
Relationship to the underlying infrastructure
BOSH cloud config

#CNA2080BU CONFIDENTIAL 32
instance_groups:
- name: etcd - name: master
instances: 2 - name: worker
instances: 3 instances: 3
networks: networks:
- name: *network-name networks:
- name: &network-name ((deployments_network)) - name: *network-name
azs: [z1] azs: [z1]
jobs: azs: [z1]
jobs: jobs:
- name: etcd - name: cloud-provider
release: kubo - name: docker
t i o n
ibu
release: kubo-etcd release: docker
properties: properties: {}
i s tr
etcd: - name: kubernetes-api properties:
or d
release: kubo t io
...n
require_ssl: false
bli c a
- name: kubeconfig
peer_require_ssl: false properties:
r p u release: kubo
stemcell: trusty
o fo
admin-username: admin
t properties:
vm_type: common
nt: N
admin-password: ((kubo-admin-password))
persistent_disk_type: 5120 o n te
... ...

17 C
- name: kubeconfig - name: kubelet
2 0 release: kubo
w o rld release: kubo
properties: properties:
Manifest
V M ... ...
... - name: kubernetes-proxy
stemcell: trusty release: kubo
vm_type: master properties:
...
stemcell: trusty
vm_type: worker
persistent_disk_type: 10240 33
instance_groups:
- name: etcd - name: master
instances: 2 - name: worker
instances: 3 instances: 3
networks: networks:
- name: *network-name networks:
- name: &network-name ((deployments_network)) - name: *network-name
azs: [z1] azs: [z1]
jobs: azs: [z1]
jobs: jobs:
- name: etcd - name: cloud-provider
release: kubo - name: docker
t i o n
ibu
release: kubo-etcd release: docker
properties: properties: {}
i s tr
etcd: - name: kubernetes-api properties:
or d
release: kubo t io
...n
require_ssl: false
bli c a
- name: kubeconfig
peer_require_ssl: false properties:
r p u release: kubo
stemcell: trusty
o fo
admin-username: admin
t properties:
vm_type: common
nt: N
admin-password: ((kubo-admin-password))
persistent_disk_type: 5120 o n te
... ...

17 C
- name: kubeconfig - name: kubelet
2 0 release: kubo
w o rld release: kubo
properties: properties:
V M ... ...
Scaling is a matter of changing...
the number of - name: kubernetes-proxy
stemcell: trusty release: kubo
instances and telling BOSH to
vm_type: master properties:
“make it so” ...
stemcell: trusty
vm_type: worker
persistent_disk_type: 10240 34
Day 2: Operationalize

t i o n
1 Managing Health i s tr ibu
or d
t ion
bli c a
2 Scaling
r p u
o t fo
Upgrade te nt: N
3
C on
2 017
orl d
VMw
K8s Cluster Upgrade: Canary Deployments

update: t i o n
t r i b u
canaries: 1 r d is
o n o
max_in_flight: 1
c a t i
u b l i
serial: true or p
o t f
n t : N
canary_watch_time: 10000-300000
t e
Con 10000-300000
update_watch_time:
1 7
2 0
w o rld
Manifest
V M

#CNA2080BU CONFIDENTIAL 36
 K8s Cluster Upgrade: Canary Deployments
EXAMPLE:

# OF CANARIES: 2

MAX IN FLIGHT: 2
t i o n
i s t ribu
or d
t ion
bli c a
r p u
o t fo
nt: N
te
CANARIES

C o n
0 1 7
rld 2
Mw o
V
V1.0 V1.1

#CNA2080BU CONFIDENTIAL 37
 K8s Cluster Upgrade: Canary Deployments
EXAMPLE:

# OF CANARIES: 2

MAX IN FLIGHT: 2
t i o n
i s t ribu
or d
t io n
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

V1.1 V1.2
Once failed, Canary VMs are kept
for troubleshooting purposes.

#CNA2080BU CONFIDENTIAL 38
 t i o n
i s tr ibu
or d
Operationalizing at Scale t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Supporting Kubernetes Needs at Scale

Kubo Release

Release
templates
t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
Manifest
o n te
17 C
2 0
w o rld
PKS Service Broker
VM
BOSH

40
 Supporting Kubernetes Needs at Scale
https://thenewstack.io/comcast-1500-developers-working-cloud-foundry

Kubo Release

create cluster Release

(with upgrade policy) templates
t i o n Ones

i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N manage

Thousands Manifest
o n te
17 C
2 0
w o rld
PKS Service Broker
VM
BOSH

41
 t i o n
i s tr ibu
or d
Let Us Show You… t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#CNA2080BU CONFIDENTIAL 43
 NSX-T Integration
NSX Container Plugin
Kubernetes
Adapter

t i o n
ibu
CloudFoundry
PaaS Control Plane Adapter NSX
i s tr
NCM
Infra
Manager
or d
API-Server
Mesos API Client
t io n
Adapter

bli c a
etcd
Libnetwork
r p u
Scheduler Adapter
o t fo
nt: N
te
NSX topology for K8s / CF
•
C o n
NSX Container Plugin (NCP) for integrating with Kubernetes
0 1 7 Proj: foo Proj: bar
• NSX Features for K8s PODs
rl d 2
• IP address per container / w o
VM
POD
• Container Network – Routed (BGP) & NATed mode
• Microsegmentation – via K8s Network Policy or native NSX APIs (mapping
K8s labels to NSX tags)
• Network & Security automation – created as part of app deployment
• Multi-tenant network topologies
vRealize Ops, vRealize Log Insight For Comprehensive Visibility
Launch in
VMware vRealize Context VMware vRealize
Operations Log Insight
Capacity, Performance and Log analytics, aggregation,

n
Configuration Management Events and search
t i o
i s tr ibu
Structured Data
or
Unstructured Data d
t ionMessages
Metrics Alerts Events Logs
bli c a
r p u
o t fo
nt: N
o n te
1 7 C
2 0 Virtual Applications

w orld
V M
46

vRealize Ops – Managing Kubernetes Clusters

K8S Summary –
Nodes, Pods, etc. i o n
ibu t
i s tr
or d
t ion
K8S Topology -
K8S Pods - pHealthbli c a
u
Health
t f or
: N o
t e n t
C on
0 1 7
l d 2
or
VMw
47

vRealize Ops – Kubernetes Integration Details

K8S Pod Relationship
to Components
K8S Alerts n
t i o
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
7 C
r 201
K8S Alerts
ld
Mw o
V
 Introducing Wavefront By VMware
SaaS-Based Metrics Monitoring and Analytics Platform
Iterate & Troubleshoot Trend & Alert Visualize Self-Service Metrics
Issues on Anomalies Metrics at Scale Analytics for All

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
UI and API Backend
o n te
17 C
0
Advanced Analytics Engine
2
w o rld
V M
Metrics Collection and Storage

Engineering &
Business

4
Wavefront – Container Monitoring Suite

t i o n
i s tr ibu
Amazon Docker
or d
ECS Swarm
t ion
bli c a
Docker Host Docker Host
r p u
o t fo
nt: N
o n te
17 C
2 0
rld
Container

Mw o
App Containers Metric Collector

V Docker Host

Docker
Cluster
Real-time insight into Docker containers and orchestration
systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS
50

Registry – Enterprise-grade Private Registry

user management & access control

role-based access control
t i o n
i b
AD/LDAP integration
t r u
r d is
n o
security
o
c at i
b l i vulnerability scanning
r pu
o t fo content trust - image signing
n t : N
ont e policy based image replication
1 7
Need Harbor Cscreenshot
0 d 2 audit and logs
or l
V Mw restful API
lightweight & easy deployment
open-source under Apache 2 license
51

Registry – Content Trust,

When Enabled Un-signed Images Can’t Be Pulled

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
52

Registry – Image Vulnerability Scanning Details

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
 VMware PKS
t i o n
Kubernetes on BOSH (Kubo)
i s t ribu
Analytics Automation
or d
a t i on GCP

lic
Service

pub
etcd master worker etcd master worker
Container Broker
Registry
fo r
N o t
e n t : BOSH
nt
Logging Monitoring

C o
0 1 7
l d 2 NSX
or
Operations Security
VMw
vSphere vSAN

Physical Infrastructure
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M