Architecture.

Design of the RWG Splunk

Deployment

Design of the RWG Splunk Deployment 1

Contents

1 Inhoudsopgave
2 Introduction 3
3 Deployment Proposal 1 4
4 Deployment Proposal 2 5
5 Description of both proposals 6
5.1 Search-Head-layer 6
5.2 Indexer-layer 6
5.3 Forwarder-layer 6
5.4 Scalability 7
5.5 Encryption 7
5.6 Authentication 7
6 Hardware 7
6.1 Search-Heads 7
6.1.1 Proposal 1 7
6.1.2 Proposal 2 7
6.2 Indexers 7
6.3 Cluster-Master 8
6.4 Deployment-Server 8
6.5 Syslog-Collector 8

Design of the RWG Splunk Deployment 2

2 Introduction
This design is based on the results of the recent meeting in which the requirements of the Splunk-
deployment were discussed and these are applied here. As we are not sure on what the requested
redundancy should include, we created two layouts, one with a single Search-Head, the other with a
Search-Head-Cluster that includes redundancy also for this layer.

Design of the RWG Splunk Deployment 3

3 Deployment Proposal 1

Search Head

Cluster Master,
License Manager,
Monitoring Console

Indexer Indexer

Deployment
Server
Syslog-Collectors Forwarders on
Forwarders Production-systems

Splunkd (8089)
RWG Splunk-Deployment
Data (9997)
Replication (9998) Proposal 1

Design of the RWG Splunk Deployment 4

4 Deployment Proposal 2

Load
Balancer

Search Head Search Head Search Head

Cluster Master,
License Manager,
Monitoring Console
Deployer

Indexer Indexer

Deployment
Server
Syslog-Collectors Forwarders on
Forwarders Production-systems

Splunkweb (8000)
Splunkd (8089) RWG Splunk-Deployment
Data (9997) Proposal 2
Replication (9998)

Design of the RWG Splunk Deployment 5

5 Description of both proposals
5.1 Search-Head-layer
Proposal 1 includes one Search-Head. This design does not include any kind of redundancy for this
layer. This means that in the event that the Search-Head is down, no user will be able to work with
Splunk. This situation does also effect the execution of scheduled searches including those that should
generate alerts and alert-actions like notifications. It is also important that this single Search-Head has
sufficient capacity to handle the activities of the 20 concurrent users.

In proposal 2 a Search-Head-Cluster is included. This Cluster acts as an additional redundancy-layer

and does add more performance. This means that an individual Search-Head can go down without the
loss of functionality. If all Search-Heads are available Splunk will utilize all available resources. This
means that there is no system doing nothing and just waiting for another system to fail. In order to
connect to the Search-Heads without having to bother which one is available and which one is less busy
it is advised to include a Load-Balancer into the access-path. As all members of the Search-Head-
Cluster are utilized as efficient as possible, the hardware of each system can have less capacity rather
than the one of proposal 1. Each Search-Head-Cluster requires a Deployer. This function can be
fulfilled by the Cluster-Master that is discussed later.

5.2 Indexer-layer
The Indexer-layer includes two Indexers being part of an Indexer-Cluster. Both are responsible for half
of the entire dataset each, this according to the Map-Reduce-Principle. As long as both are active they
will combine their resources. In addition the Indexers will replicate the indexed data to each other. This
means that Indexer1 will send copies of the indexed data to Indexer2 en vice versa. As a result there are
always two copies of all data available. In the event of one failing Indexer the remaining one will process
all data, searches and other activities. This means that one Indexer can fail without the loss of
functionality and data.

Another part of the Indexer-Cluster is the Cluster-Master. This instance directs the Search-Head(s) to
the available Indexers and takes care about the replication of the indexes. If an Indexer fails, the
Cluster-Master will make sure that the replicated data becomes available for searches. The Cluster-
Master itself does not introduce a “Single Point of Failure”. This is because it does only act if something
in the cluster is changed. This means that if the Cluster is stable, it can do without the Cluster-Master.
But, nevertheless, a failing Cluster-Master should be repaired or replaces a soon as possible. This can
be accomplished by having a standby system or backup in place.

5.3 Forwarder-layer
Both designs include two types of Forwarders. Both are Universal-Forwarders so that their “footprint” is
as small as possible. The first Forwarder-type is the one that should be installed on all production-
servers so that it acts as an agent. As such it will collect and send data to the Indexers. The other
Forwarder-type is supposed to be installed on a dedicated system that receives all syslog-data. It is
advised to assign a virtual Linux-system for this purpose that uses rsyslog or syslogng so that it act as a
syslog-daemon. This data will be stored in files and these are monitored with the local Forwarder. In
order to avoid the chance of data-loss we advise to configure two of these systems as an OS-cluster with
one virtual IP-address.

All Forwarders will load-balance the data so that both Indexers receive an equal amount of data.

This layer does also include a Deployment-Server that takes care about the distribution of the
configurations to all Forwarders.

Design of the RWG Splunk Deployment 6

5.4 Scalability
Both proposed designs are scalable. This means that systems easily can be added to all layers if an
extension is required, this without any downtime or reconfiguration of the existing systems.

5.5 Encryption
By default the only connections that are encrypted are the webservice-ports (splunkd). The
administrator can decide to change the Splunkweb-port to support https. Also the data-connections
between the Forwarders and the Indexers can be encrypted. It is recommended to only enable this if this
is really necessary as this has some impact on the resources of the production-servers. It is also needed
to take care about certificates. The by default self-signed certificates are not save. Therefore, one
should create his own certificates or use those from one of the commercial providers. Also these must
be maintained as certificates expire frequently.

5.6 Authentication
Although Splunk has a build-in authentication-system, it is recommended to use an external system for
this purpose. As Active-Directory is already utilized, users can simply being authenticated and groups
can be mapped with Splunk-roles, determining the capabilities and data-access.

6 Hardware
6.1 Search-Heads

6.1.1 Proposal 1
One hardware-system of:
o Intel 64-bit chip architecture
o 24 CPU-cores at 2 GHz or greater speed per core
o 64GB RAM
o 2 x 300GB, 10.000 RPM SAS harddisks, configured in RAID 1 (mirror)
o A 1Gb Ethernet NIC, with optional a second NIC for an isolated management network
o 64-bit Linux distribution (Red Hat, CentOS, etc.)

6.1.2 Proposal 2
Three hardware-systems of each:
o Intel 64-bit chip architecture
o 16 CPU-cores at 2 GHz or greater speed per core
o 12GB RAM
o 2 x 300GB, 10.000 RPM SAS harddisks, configured in RAID 1 (mirror)
o A 1Gb Ethernet NIC, with optional a second NIC for an isolated management network
o 64-bit Linux distribution (Red Hat, CentOS, etc.)

o One Load-Balancer (recommended)

6.2 Indexers
Two hardware-systems of each:
o Intel 64-bit chip architecture
o 24 CPU-cores at 2 GHz or greater speed per core
o 64GB RAM
o Disk subsystem capable of 800 average IOPS
o A 1Gb Ethernet NIC, with optional a second NIC for an isolated management network
o 64-bit Linux distribution (Red Hat, CentOS, etc.)

Design of the RWG Splunk Deployment 7

 With regards to the storage of the Indexer’s data we require an estimated amount of data being
indexes each day. We also need to know what the desired retention-time for each index is.

Assuming that 300GB per day is being indexed and the data is during 1 week high-speed accessible,
it would be sufficient to have 2,1TB per system that meets the recommended number of IOPS. The
older data can be stored in a slower and/or an external storage.

6.3 Cluster-Master
One virtual-system (recommended a similar system as backup)
o Intel 64-bit chip architecture
o 8 CPU-cores at 2 GHz or greater speed per core
o 12GB RAM
o 100GB storage
o A 1Gb Ethernet NIC, with optional a second NIC for an isolated management network
o 64-bit Linux distribution (Red Hat, CentOS, etc.)

6.4 Deployment-Server
One virtual-system
o Intel 64-bit chip architecture
o 4 CPU-cores at 2 GHz or greater speed per core
o 12GB RAM
o 50GB storage
o A 1Gb Ethernet NIC, with optional a second NIC for an isolated management network
o 64-bit Linux distribution (Red Hat, CentOS, etc.)

6.5 Syslog-Collector
One virtual-system (recommended two identical in an OS-cluster with VIP)
o Intel 64-bit chip architecture
o 4 CPU-cores at 2 GHz or greater speed per core
o 12GB RAM
o 50GB storage
o A 1Gb Ethernet NIC, with optional a second NIC for an isolated management network
o 64-bit Linux distribution (Red Hat, CentOS, etc.)
o rsyslog of syslogng

Design of the RWG Splunk Deployment 8