© 2019 SPLUNK INC.

4 Easy Steps for

Reclaiming Your Life
(Without Quitting
Your Job)
Bethany Abbott
TechOps Manager |

@ns1 ns1.com
 © 2019 SPLUNK INC.

Forward- During the course of this presentation, we may make forward‐looking statements
regarding future events or plans of the company. We caution you that such statements

Looking reflect our current expectations and estimates based on factors currently known to us
and that actual events or results may differ materially. The forward-looking statements

Statements made in the this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, it may not contain current or
accurate information. We do not assume any obligation to update
any forward‐looking statements made herein.

In addition, any information about our roadmap outlines our general product direction
and is subject to change at any time without notice. It is for informational purposes only,
and shall not be incorporated into any contract or other commitment. Splunk undertakes
no obligation either to develop the features or functionalities described or to include any
such feature or functionality in a future release.

Splunk, Splunk>, Turn Data Into Doing, The Engine for Machine Data, Splunk Cloud,
Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the
United States and other countries. All other brand names, product names, or
trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved.
 © 2019 SPLUNK INC.

@ns1 ns1.com

@ns1 ns1.com
 © 2019 SPLUNK INC.

My Background
It’s always DNS.

​NS1
•TechOps Manager
​= •Aug 2017 - Current
– First TechOps Hire
o Team of 8
o Across 5 time zones
o On call went from a week to one
day
– 25+ Global Pops

​600+ servers
@ns1 ns1.com
 © 2019 SPLUNK INC.

My Background
It’s always DNS.

​Dyn -> Oracle

•NOC Lead
•2012- 2017
– DDoS 10/21/16

@ns1 ns1.com
 © 2019 SPLUNK INC.

This is Fine

@ns1 ns1.com KC Green

 © 2019 SPLUNK INC.

No! This is
Not Fine.

@ns1 ns1.com
KC Green
 © 2019 SPLUNK INC.

Demo

Quick Exercise

@ns1 ns1.com
 © 2019 SPLUNK INC.

​“Currently 56.7% of tech workers

are experiencing burnout”
​Blind, anonymous employment app

@ns1 ns1.com
 © 2019 SPLUNK INC.

Burnout
Who has felt it?

​According to the World Health Organization, the official diagnosis for burnout includes:
• Feelings of energy depletion or exhaustion;
• Increased mental distance from one's job, or feelings of negativism or cynicism related to one's job; and
• Reduced professional efficacy

​Some other startling statistics from the Institute of Stress:

• 25% of workers have felt like screaming or shouting because of job stress
• Nearly 50% of workers say they need help learning how to manage stress
• More than a third of workers (35%) say they feel their jobs harm their physical or emotional health

@ns1 ns1.com
 © 2019 SPLUNK INC.

Four steps to
building
sustainable
oncall
processes
for happier
teams 1. Stop having a quarterback

@ns1 ns1.com
 © 2019 SPLUNK INC.

Stop Having a Quarterback

​Word: Quarterback

​Train

​Empower

​Everyone can run an incident

@ns1 ns1.com
 © 2019 SPLUNK INC.

Four steps to
building
sustainable
oncall
processes for
happier 1. Stop having a quarterback
teams
2. Build processes with other teams

@ns1 ns1.com
 © 2019 SPLUNK INC.

Build Processes With Other Teams

​Customer Success

​Developers

​Incident Management Process

@ns1 ns1.com
 © 2019 SPLUNK INC.

Customer Success Questions

Required for all customer impacting incidents

​What was customer impact?

​Why did it happen?

​What did we do to resolve it?

​Why won't it happen again?

@ns1 ns1.com
 © 2019 SPLUNK INC.

Four steps to
building
sustainable
oncall 1. Stop having a quarterback
processes
for happier 2. Build processes with other teams
teams
3. Actionable alerts through VictorOps that
raise awareness without causing alert
fatigue

@ns1 ns1.com
 © 2019 SPLUNK INC.

“​ Research shows that between

72% and 99% of all clinical alarms
are false which results in them
being completely ignored.”

​Nurse.org
@ns1 ns1.com
Actionable Alerts Through Victorops that © 2019 SPLUNK INC.

Raise Awareness Without Causing Alert

Fatigue

​Predictable on call schedules

​Easy to acknowledge

​Links to Runbooks

​Reporting

@ns1 ns1.com
 © 2019 SPLUNK INC.

Easy to Acknowledge
One click ack-ing

​Multiple ways to ack

• Slack channel
• Push notification
• Web
• SMS
• Call

@ns1 ns1.com
 © 2019 SPLUNK INC.

Links to Runbooks
Runbooks make oncall and onboarding easier

​All alerts should have a

runbook
• Rules Engine in VO
• Internal tools
• Runbook template

@ns1 ns1.com
 © 2019 SPLUNK INC.

Reporting
Actionable vs. unactionable alerts

​Using VO Tools
• Frequency
• Oncall workload

​Internal Audit
• Record actionable vs
un-actionable alerts

Take action on this data!

@ns1 ns1.com
 © 2019 SPLUNK INC.

Four steps to
building
sustainable 1. Stop having a quarterback
oncall
processes 2. Build processes with other teams
for happier
teams 3. Actionable alerts through VictorOps

4. Chaos Engineering and Wargames

@ns1 ns1.com
 © 2019 SPLUNK INC.

Chaos Engineering and Wargames

​DDoS Yourself

​Incident Drills

​Break things on purpose!

Toshifumi Kitamura/AFP/Getty Images

@ns1 ns1.com
Four
© 2019 SPLUNK INC.

Five steps to
building 1. Stop having a quarterback
sustainable
oncall 2. Build processes with other teams
processes for
happier 3. Actionable alerts through VictorOps
teams
4. Chaos Engineering and Wargames

5. Train for a half marathon. Teach kids to

code. Live your life.

@ns1 ns1.com
 © 2019 SPLUNK INC.

Live Your Life

Note: Work is not a hobby

▪ Train for a half marathon

Bethany Abbott Paul Bolcar

▪ Teach kids STEM principles

▪ Participate in Trivia Night
▪ Go Hiking
▪ Ride a bike extension.unh.edu/

▪ Motorbike across the country

▪ Binge some Netflix
▪ Build Lego sets Laura McMaster Laura McMaster

@ns1 ns1.com
Laura McMaster
 © 2019 SPLUNK INC.

1. Stop having a quarterback

Five steps
to building 2. Build processes with other teams
sustainable
oncall 3. Actionable alerts through VictorOps
processes 4. Chaos Engineering and Wargames
for happier
teams 5. Train for a half marathon. Teach kids to
code. Live your life.

@ns1 ns1.com
 © 2019 SPLUNK INC.

Q&A
Bethany Abbott | TechOps Manager

@ns1 ns1.com
 © 2019 SPLUNK INC.

Thank
You
!Go to the .conf19 mobile app to

RATE THIS SESSION

@ns1 ns1.com