Beruflich Dokumente
Kultur Dokumente
Rafael Schloming
Co-Founder & Chief Architect
History
Datawire
Founded in 2014
Focused on microservices
Me
datawire.io 2
What is a microservice?
Wikipedia: ...no industry consensus
...implementation approach for SoA
...processes that communicate with each other to fulfill a goal
...Naturally enforces a modular structure...
Everything else:
Volumes of essays good, bad, and ugly...
datawire.io 3
Three aspects of Microservices
Process
People
Technology
datawire.io 4
From Three Sources
Experts Bootstrapping
Migrating
datawire.io 5
Starting Point
Technical:
Process:
???
People:
???
datawire.io 6
The Expert Source
Read just about every firsthand story out there
Went to conferences
And armed with a little bit of knowledge, we started filling in our picture
datawire.io 7
People Picture
Developer Happiness/Tooling/Platform Team
Service teams
datawire.io 8
Technical Picture
Control Plane Reference Architecture
Service Discovery
Logging + Metrics
Configuration
Smart Endpoints
Traffic Layer
HTTP
RPC
Messaging
datawire.io 9
First Picture
Technical:
Process:
???
People:
datawire.io 10
The Bootstrap Perspective
Five engineers building an out of the box control plane...
datawire.io 11
Ubiquitous Data Processing Pipeline
datawire.io 12
V1: Started with Discovery
Requirements:
highly available
low throughput
low latency
low operational complexity
able to survive a complete restart
capable of handling spikes
Initial Choices:
vert.x + hazelcast
websockets
smart clients
auth0 + python shim
Total Services: 2
datawire.io 13
V2: Added Tracing (PoC)
Requirements:
high throughput
highish latency ok
cannot impact application
Initial choices:
vert.x, hazelcast (only retained transient buffer of last 1000 log messages)
websockets
smart circular buffer minimized impact on application
Total Services: 3
datawire.io 14
V3: Added Persistence for Tracing
Requirements:
keep extended history
provide full text search
filtering, sorting, etc
Initial Choices:
elasticsearch for storage/search
query service
Total Services: 4
datawire.io 15
First hint of pain...
Rerouting data pathways:
touched multiple services
coupled changes
datawire.io 16
V4: Adding Persistence for Discovery
Requirements:
track errors associated with particular service nodes
store routing strategies
Initial Choices:
postgres (RDS) for persistence
datawire.io 17
Deployment Requirements
Stuff we had tried:
datawire.io 18
Deployment Redesign
Complete system definition in git
Contains all the information necessary to bootstrap the system from scratch in all of its operating
environments
System definition is well factored with respect to its environments
Abstract definition: my service needs postgres and redis
Development: service -> docker image, postgres -> docker image, redis -> docker image
Use minikube to run the whole system
Test: <same as dev for now>
Production: service -> docker image, postgres -> RDS, redis -> elasticache
Kubernetes cluster for stateless services
Tooling caters to the needs of each environment
Development: fast feedback cycle
Test: repeatable environments
Production: quick and safe updates/rollbacks
Tooling helps maintain environment parity
datawire.io 19
DevOps?
DevOps is presented as a solution to an organizational problem, but we all sat in the
same room
datawire.io 20
Process: Architecture vs Development (SoA vs SoD)
Systems (their shape in particular) are traditionally architected
Architecture
Development
datawire.io 21
Methodology for Developing Systems
Principles
small frequent changes
rapid feedback and good visibility
Applied to codebases:
Tooling for rapid feedback: compilers, incremental builds, test suites
Tooling for good visibility: printf, logging, debuggers, profilers
Applied to systems:
Key characteristics go beyond just logic and correctness
Performance within specified tolerance of the running system is a critical feature
datawire.io 22
Update the Dev Cycle
Tests assess impact on correctness...
datawire.io 23
Back to the Experts...
Canary Testing
Circuit Breakers
Dark Launching
Tracing
Metrics
Deployment
datawire.io 24
Second Picture
Technical:
Process:
People:
datawire.io 25
The Migration Perspective
Variety of stages...
datawire.io 26
Migration is about people
Starting point: team vs tech
Picking a tech stack for the entire eng org to adopt is slow
lots of organizational friction
Replatforming/refactoring an entire existing monolith is slow
lots of organizational and orchestrational friction
Creating a relatively autonomous team to tackle a particular problem in the form
of a service
datawire.io 27
The People Picture: Dividing up the Work
The work has two aspects:
build the features (dev)
keeping the system running (ops)
datawire.io 28
Third Picture
Technical:
Process:
People:
datawire.io 29
The Hard Way
datawire.io 30
The Easy Way
datawire.io 31
Microservices Cheat Sheet (What, Why & How)
People Process Technology
Microservices are a way to divide the work of Microservices are built from a process of Microservices are an application that is made up
building a cloud application frequent small changes with rapid feedback of a network of small services
and good visibility
This work falls into two categories: This is the application of the traditional dev This requires dev tooling to support quickly and
Keep the system running (ops) cycle to systems rather than codebases, and safely assessing system impact.
Build new features (dev). for it to work, key system properties must
become a first class features for developers. This requires fast deployment tooling and good
Dividing work along these categories creates visibility into key system level properties:
conflicting incentives between progress and Throughput
stability. New features from dev eventually Latency
become the biggest source of instability for Availability (error-rate)
ops.
Depending on your system, this may require
Unifying these roles (devops) allows you to tooling for:
minimize the tradeoff between progress and Fancy request routing (for canary
stability, but you now need to divide up the testing, dark launching)
work by dividing up the app. This results in a
network of services.
Give your dev teams operational responsibility! Extend the dev cycle to include a stage to Start with a fast deployment pipeline that
assess the impact on key system properties incorporates basic system level metrics and
Define service level objectives & agreements (SLOs) monitoring for each service.
for each service:
SLOs: throughput, latency, availability Build -> Test -> Deploy
SLAs: what happens when these arent met
Build -> Test -> Assess Impact -> Deploy
Commoditize common operational overhead.
datawire.io 32
Questions?
datawire.io 33
Microservices Cheat Sheet (What, Why & How)
People
Two aspects of work: keep it running (ops), build new features (dev)
Unifying roles (devops) to minimize tradeoff... divide work by dividing the app
datawire.io 34
Microservices Cheat Sheet (What, Why & How)
Process
Microservices are built from a process of frequent small changes with rapid
feedback and good visibility
This is the application of the traditional dev cycle to systems rather than
codebases, and for it to work, key system properties must become a first class
features for developers.
Extend the dev cycle to include a stage to assess the impact on key system
properties (SLOs)
datawire.io 35
Microservices Cheat Sheet (What, Why & How)
Technology
This requires dev tooling to support quickly and safely assessing system impact.
This requires fast deployment tooling and good visibility into key system level
properties:
Throughput
Latency
Availability (error-rate)
Start with a fast deployment pipeline that incorporates basic system level metrics
and monitoring for each service.
datawire.io 36
DevOps: you cant split the work (along these lines)
Dev
DevOps
Ops
User User
datawire.io 37
Features are the largest source of bugs
Dev
Ops
User Dev
Dev Dev
Ops
User
datawire.io 38
Microservices: Divide the work by dividing the app
Dev Infra
User
User
datawire.io 39
Dividing up Work
Dev Infra
User Dev
Dev Dev User
User
User
datawire.io 40
datawire.io 41