Elasticsearch: When to

Host On-Prem and when

to turn to SaaS?

Liz Bennett - Data Platform Infrastructure Engineer

About Me
Liz Bennett
Currently @
Member of Compute Infrastructure Team

Previously @
Senior Engineer - Data Infrastructure

Previously @
Engineer - News Feed Team
 We are a personal styling company

Customers complete a “style profile”

Stitch Fix sends them clothes

Customers keep what they want, send back everything

else

Customers give us feedback for every item

Our 80+ Data Scientists own dozens of
verticals in the business

The Data Platform team builds

infrastructure and tooling to support all
things Data Science

We use ES as: an ELK stack + sundry

search use cases
Elasticsearch: When to
Host On-Prem and when
to turn to SaaS?
Today there’s a hosted solution for everything

And Elasticsearch is no different

Just because you can buy hosted ES, should

you?
 Before diving into specifics…

...my personal Elasticsearch journey

My ES Journey

1. LinkedIn: deployed an ELK stack on-premise

1. Loggly: maintained a massive, multi-clustered, on-premise

ES deployment

1. Stitch Fix: hosted ES with AWS

In June 2017, I wrote Some Things you Should Know Before
Using Amazon’s Elasticsearch Service on AWS
 I got to thinking…

...why do so many people use ES as a service?

...when should you and when should you not use a

hosted ES solution?
A tough decision regardless of technology

Elasticsearch is unique

Special considerations should be made when

deciding between on-prem vs. SaaS
 Main Considerations
1. Size of the dataset
2. Irregularity of the dataset
3. The skillsets of your team
4. Timeframe
5. Search’s impact to the business
6. Security Considerations
7. SLA requirements
Size of your dataset
Size of the Dataset

Small datasets (<50GB) vs. very large datasets (>5TB)

The size of a dataset matters because of these factors:

1. Cost in $$$
2. Operational Burden
3. Effectiveness of a hosted solution
Size of the Dataset - Cost $$$

Cost will scale linearly based off of the size of your dataset

The slope and intercept of the cost functions are different tho
Size of the Dataset - Cost $$$

Rough estimates of per node costs ( 1 node, 16GB Memory, 4 cores)

AWS Raw Instance cost (m4.xlarge) $0.2 / hour = $146 / month

AWS ES, per node cost $0.301 / hour = $220 / month

(m4.xlarge.elasticsearch)

Elastic Cloud (1 node cluster w/ 16GB mem) $0.5221 / hour = $382 / month

HP ProLiant DL360 Quad-Core, 16GB mem $0.08 / hour = $58 / month

($2200 amortized over 3 yrs)
Size of the Dataset - Operational Burden

! The larger your dataset, the more maintenance the cluster requires
! A small cluster can be operated more manually
! With bigger clusters, maintenance needs to be automated

! A very small cluster can be easily hosted on premise but is also cheap to buy
from a hosting service
! Other factors, like time-frame, will matter more if the dataset is small
Size of the Dataset - Effectiveness of a Hosted Solution

! Larger clusters may exceed the capabilities of a hosting service to provide a

stable deployment

! Particularly with ES, large deployments can require complex, custom

configurations

! Many hosted ES providers expose a limited set of configuration options, or

none at all
Irregularity of your dataset
Irregularity of the data set

An ES-specific factor that should not be underestimated

Regular datasets:
! Documents have a well defined set of fields

! Indexing volume is steady

! Data can be easily distributed across all nodes in the cluster

Irregular datasets:
! Documents can have any field, lots of dynamic fields

! Indexing volumes are sporadic and unpredictable

! Data is not evenly distributed across the cluster. Hot spots may occur
Irregularity of the data set

! Irregular datasets can cause a myriad of stability issues in ES

! Regardless of hosting on-premise or in the cloud

! Finding these issues, understanding, and fixing them requires an in-depth knowledge of ES
! You also need access to detailed dashboards and monitoring tools

! Regardless of on-premise vs. SaaS, you need to know how to model your data, how to structure your
indexes, ES best-practices, dos and don’ts, etc.

! If your data is irregular, you need an understanding of ES and its internals in order to ensure the stability
of your cluster

! Developing this knowledge is half the hurdle when it comes to operating and hosting ES yourself

! If your data is highly irregular, you could bite the bullet and host ES yourself
Irregularity of the data set

! Regular datasets will be far easier to manage

! The cluster will be more stable, easier to operate and to maintain

! Hosting yourself will be easier

! On the other hand, paying someone to host for you will offload the burden of having to know anything at
all about Elasticsearch
Skills of your team
Skills of your team

! Is your team mostly front end engineers? You probably want hosted ES

! Mostly backend engineers? Maybe on-premise

! Is your team very small and strapped for cycles? Go with hosted ES

! Your team is small, strapped for cycles, and strapped for cash? Maybe on-premise

DevOps Skills

! If you are considering using Elasticsearch, you already have a software stack
! Software built in-house, you already maintain, monitor, deploy, upgrade and operate
! You’ve maybe already invested in devops to some degree

! If you already have strong devops chops, the upfront cost of deploying your own infrastructure is much less
! For example, if your stack is mainly JVM with high SLAs, you likely already have the tools necessary to run
your own ES cluster
Timeframe
Timeframe

! Clicking a button on a UI is always faster than hosting something yourself

! If time is your biggest concern, the cloud is the way to go
! Hosted solutions are great for prototyping

! This is especially true with AWS, if you are already on AWS

! Buying a hosted solution may involve a contract negotiation process

Search’s Impact to the Business
Search’s Impact to the Business

! If search is central to your business, it could be worth investing in on-premise ES

! Developing in-house Elasticsearch expertise will ensure your cluster is:
○ More tailored to your use case
○ Cheaper to operate
○ More performant
○ More stable

! If search is central to your business, having a better ES deployment than your competitors gives you a
competitive advantage

! If you host ES on-prem, it is easier to get more involved with the open source community
! Easier to write and submit patches upstream
! If search is important to the business, it may be valuable to play an active role in open-source

! If you need search to power an internal tool, the ROI is less and a hosted solution could be better
Security Considerations
Security Considerations

! Because of security considerations, you may prefer on-premise software solutions.

! If you are storing sensitive data (such as PII) in your ES cluster, you may not want to send the data to a
third party.

! Some regulations can make it harder to work with hosted solutions, i.e. HIPAA, EU GDPR

! If the data in your ES cluster was breached, how bad would it be for your company?

! How much bureaucracy is involved when you decide to work with a hosted solution?
! Would it be faster and easier to just host it yourself?
SLA Requirements
SLA Requirements

! SLA requirements break down into

! Query performance
! Indexing performance
! Uptime

! Query performance and indexing performance can generally be improved by:

! Adding more nodes
! Optimizing sharding strategies
! Optimizing use of filters and caching
! Things you’d have to do regardless of on-prem vs. SaaS

! Uptime is a different story

SLA Requirements - Uptime

! When it comes to Elasticsearch, expect downtime.

! Many ES stability issues happen because “you shot yourself in the foot”.
! They are self-inflicted by your query/indexing patterns, sharding, document structure, etc.

! What does this have to do with on-prem vs. SaaS?

! With most SaaS infrastructure, you offload maintenance, monitoring & technical expertise
! With ES, it’s harder to offload the burden of technical expertise
! Also, monitoring is more complex and circumstantial. You have to do a lot of it yourself
! Even if you’re using ES as a service, don’t expect plug n’ play

! Here’s the kicker

! You won’t always be able to fix your cluster yourself, because administrative and monitoring APIs may be
gated by the hosting service
! You must work with support engineers
! Back and forth can take a long time, and periods of downtime can last hours

! If uptime is extremely important, hosting ES yourself may yield a more stable cluster
Some Examples
 When hosted ES makes sense
1. ELK stack for a small to moderately sized team
or department
2. To power search for a small and predictable
dataset, i.e. inventory in a warehouse, blog
posts, etc.
3. When prototyping a new search feature or
product
4. To power search for an internal tool
 When on-premise ES makes sense
1. You’re launching a search-centric business or
product line, i.e. Yelp
2. Company-wide ELK deployment for mid to large
sized company
3. You’re building an app that searches highly
sensitive data
 Why my team uses hosted ES

1. Our dataset is relatively small

2. We can tolerate downtime
3. The indexing and query patterns are somewhat
predictable
4. It is only used to power internal tools
5. Having a better ES deployment is not a
competitive advantage
Main Take-Aways
Main Take-Aways

! Hosted ES is faster to setup

! Hosted ES is more expensive

! Hosted ES will not offload the burden of technical expertise or monitoring

! Hosted ES does offload the burden of maintenance (upgrades, snapshots)

! On-prem ES will perform better and be more stable, especially for large clusters

! Cost of on-prem ES depends on the DevOps skills of your team

! Elasticsearch is not like other kinds of infrastructure. Do not expect plug & play
 Some armchair thoughts…

Many open-source companies (Elastic included) are increasingly relying on hosted solutions
for revenue

They are incentivized to convince you that hosting their software is onerous

They are incentivized to get you to upgrade and use more resources

They are incentivized to make it difficult for you to host their software

Always think critically, consider your options deeply and understand the ROI
 Thanks for coming!

Tweet me at @zzbennett

E-mail me at elizabeth.bennett@stitchfix.com

(Also we’re hiring!!!!!)

 ?
?
?
?
Questions?
?
? ?
?
? ?