Sie sind auf Seite 1von 32

Copyright

2014 Splunk Inc.

Islands of Splunk

MulJple Splunk as a Service


Architecture and ImplementaJon
Michael de Buin, Schuberg Philis mdebruin@schubergphilis.com
Gert Kremer, Schuberg Philis
gkremer@schubergphilis.com
Dani Flexer, Splunk

dexer@splunk.com

Disclaimer
During the course of this presentaJon, we may make forward looking statements regarding future events or the
expected performance of the company. We cauJon you that such statements reect our current expectaJons and
esJmates based on factors currently known to us and that actual events or results could dier materially. For
important factors that may cause actual results to dier from those contained in our forward-looking statements,
please review our lings with the SEC. The forward-looking statements made in the this presentaJon are being made as
of the Jme and date of its live presentaJon. If reviewed aTer its live presentaJon, this presentaJon may not contain
current or accurate informaJon. We do not assume any obligaJon to update any forward looking statements we may
make. In addiJon, any informaJon about our roadmap outlines our general product direcJon and is subject to change
at any Jme without noJce. It is for informaJonal purposes only and shall not, be incorporated into any contract or
other commitment. Splunk undertakes no obligaJon either to develop the features or funcJonality described or to
include any such feature or funcJonality in a future release.

Agenda
!
!
!

The MulJ-Splunk-as-a-Service (MSaaS) framework


MSaaS implementaJon @ Schuberg Philis
DemonstraJon

MSaaS Architecture

Background
!

Splunk administrators are increasingly required to provision Splunk


as a service oering to mulJple customers
Commonly requires provisioning a Splunk instance to each customer

MSaaS is a conceptual framework designed to help deliver such an


oering

Schuberg-Philis have implemented this framework using Chef on


Apache CloudStack

FuncJonal Requirements
!
!
!
!
!
!

AutomaJc deployment and update of mulJple Splunk instances


Packaged binaries, scripts and conguraJons
Modular
Each instance can scale from very small to as large as necessary
Each instance is customized as needed
System must funcJon independently without external resources

Except for authenJcaJon, a datacentre automaJon (DCA) tool and an


opJonal license manager to fulll its purpose

Archiving, backup and resilience requirements are dened per


customer
6

FuncJonal Requirements
!

Resilience

Disaster recovery (DR), no single point of failure, indexing Jer resilience,


storage resilience
!
!
!
!

Strict data segregaJon


Dierent network jurisdicJons isolated from each other with no
shared resources nor shared informaJon
Cross-jurisdicJon access possible when explicitly enabled
JurisdicJon hierarchy supported
A jurisdicJon can include other jurisdicJons

FuncJonal Requirements
!
!

Indexed data can be shared, subject to jurisdicJon, and indexes


copied between instances
MulJple licensing models supported
central license manager and license pools
per-Island license manager and mulJple keys assigned on deployment
global license

!
!

Roles dened independently for each instance


All credenJals maintained in the enterprise idenJty management
system and allocated at deployment

Architecture Concepts: Island


!
!
!
!
!

A complete Splunk deployment


No informaJon sharing between Islands
Own set of users and roles
Manages a set of Forwarders
Forwarders can send data to many
Islands but are managed by a single
Island

Architecture Concepts
!

Bridges

Islands without indexing capabiliJes that enable


search on mulJple Islands

Deployment unit

An independently deployed collecJon of Splunk


components

Customer

An independent user of Splunk a business unit or


customer

Island service agributes

ReplicaJon factor, search factor, DR requirements,


security, backup, storage Jer, performance,
retenJon plan, daily volume

AdministraJon Island monitors the other Islands


10

MSaaS Deployment Server


!

!
!

A centralized system responsible for


installing, conguring, and updaJng the
Islands
Maintains all binaries, applicaJons,
conguraJons, apps and rouJng
informaJon in a version control system
(VCS)
Updates the Islands binaries when
necessary
Each Splunk DS maintains its Islands
components

ConguraJon les supplied by the MSaaS


DS and propagated by Island DS

11

ApplicaJons

Each Island has any number of Splunk applicaJons a.k.a. apps


Apps are

Managed centrally and deployed with the Island


Versions are maintained in the VCS
Customized for the MSaaS as a whole or for a subset of the Islands

MSaaS administraJon apps reside on a dedicated Island

Monitors the other Islands for usage, security, charge-back, health and need
for maintenance

Standard apps deployed with each Island

S.o.S Splunk on Splunk


App for Unix or for Windows Infrastructure as appropriate
12

MSaaS
ImplementaJon
@SBP

My Company and I
Our customers:

Gert Kremer
Mission CriJcal Engineer since 2007

Engineering the MSaaS Architecture

Dual datacenter setup, no data backups (RF=SF=2)


Maximum 100 GB/day per Island
Centralized license server
AcJve-standby search heads (rsync-ed)

Not implemented:

!
!
!

Splunk Deployment Server


AdministraJon Island
Dedicated Job Servers
15

Ingredients
Cloud(Stack)
Requirements
M-SaaS
Github
Splunk
Chef
&
use
Meta
centerprise
ases
Architecture
DescripJon

16

Will it blend?
Congura)on

Indexing:
Avg KBPS

Indexing:
Avg EPS

Search: Avg Search: Avg


First Event Search (sec)
(sec)

HP DL380G7; CPU: 26 Xeon 2.67GHz;


Memory: 12GB; OS: Linux 64-bit, Fedora
14 (*)

22,400

79,057

2.48

20.18

Linux on EC2: c1.xlarge


800 pIOPS (*)

12,410

43,639

2.12

27.37

SBP: 4 CPU/16GB

12,449

43,865

2.82

18.24

SBP: 8 CPU/32GB

14,715

51,959

1.37

17.24

(*) hgp://blogs.splunk.com/2013/06/06/splunkit-v2-0-2-results-ec2-storage-comparisons/
17

Prior art on Splunk, Chef and Cloud


!

Best Buy Splunk cookbook

hgps://github.com/bestbuycom/splunk_cookbook

OpsCode Splunk cookbook

hgps://github.com/opscode-cookbooks/chef-splunk

Splunk Storm (Splunk as a Service)


hgp://www.getchef.com/customers/splunk/

18

Scope of AutomaJon
!

Island creaJon

Server instanJaJon

Splunk soTware installaJon

Cluster conguraJon

Data disk and indexes: creaJon and management

Data replicaJon between Search Heads

Security (rewall rules, SSL setup)

Monitoring (process, connecJvity, cluster health, Splunk alerts)

Single-sign-on (SSO)

Splunk applicaJons
19

Splunk Enterprise at Schuberg Philis


Datacenter 1

Datacenter 2

Cluster
master

Search head

Search head

Indexer

Indexer

License
server

Splunk Enterprise at Schuberg Philis


AuthenJcaJng
proxy

Bridge (SIEM)

Island Customer
Island Customer
Island Customer
Search head

Search head

Search head
Search head
Search head

SupporJng Systems

Search head
Search head

License
server

Search head
Cluster
Indexer
master
Cluster
Indexer
master
Cluster
Indexer
master

Indexer
Indexer

AcJve
Directory

Indexer

SupporJng
Cloud
Infrastructure

Island InstanJaJon
Proxy
Servers

Datacenter 1

Cluster Master

1. Congure island
2. Deploy island
3. Integrate island

Datacenter 2

Search Head

Search Head

Indexer

Indexer

License
Server

Island InstanJaJon
Proxy
Servers

Datacenter 1

Cluster Master

1. Congure island
2. Deploy island
3. Integrate island

Datacenter 2

Search Head

Search Head

Indexer

Indexer

License
Server

Returning the favor


!
!
!

Generalize available Chef Cookbooks


Splunk monitoring Nagios plugin available in Nagios Exchange
Splunk deployment best pracJces and tools

24

DemonstraJon
by Michael

THANK YOU
dexer@splunk.com,
gkremer@schubergphilis.com

Process Requirements
!

Service Request

When customers request the service, a process is triggered that results in a


deployed instance of Splunk that implements the customers use-cases and
the other agributes of the service requested.
!

Charge Back

The cost of the service to its operators can be charged back to its customers
based on the actual cost of provisioning the service.
!

Easy to Onboard

The process of incorporaJng data sources into the system is well dened
and simple
27

Data RouJng
!
!
!

Islands use-cases can overlap requiring them to share data and data-
sources
Data rouJng is maintained in a global rouJng table
On update, the rouJng table is converted into Splunk conguraJon
elements suitable for inserJon into the transforms.conf le that is
then propagated to the data collecJon Jer by the Master
Deployment Server and Island Deployment Servers
Heavy Forwarders are required to support data rouJng, otherwise a
Universal Forwarder is used

Island Service Agributes


!
!
!

Agributes are dened when a service is requested by a customer


The provisioning process implements the agributes and deploys the
Island
Agributes

Resilience
DR
License volume
Data segregaJon and availability
Storage volume
RetenJon requirements
Data sources and use cases

Customers
!
!
!

Each customer is an area of the enterprise or an external


organizaJon that requires Splunk
A customer consists of one or more use-cases
MulJple customers can be consolidated into a single MSaaS Island
segregaJon requirements permizng

Source Types and Data Sources


!

In addiJon to the default set of source types provided by Splunk, MSaaS


implements addiJonal source types that are used as needed by any Island

When an Island is implemented, new source types are dened as required


and the pre-dened MSaaS source types extended

Data sources indexed in a given Island are parJJoned into specic indexes
based on security, ownership, visibility constraints, retenJon requirements,
and resilience requirements.

These agributes can dier between Islands for common data sources,
depending on the requirements of the use-cases implemented by this
Island

Common Namespace and


InformaJon Model
!
!

Named Splunk enJJes are named in a common namespace so


dierent Islands can share informaJon, apps, indexes etc.
Requires MSaaS to maintain a centrally controlled global name
space