Splunk Book

Splunk Overview Training
Duration: 3 days
Skill Level: Introductory and beyond
Hands-On Format: This hands-on class is approximately 50% hands-on lab to 50% lecture ratio, combining engaging lecture, demos, group activities and
discussions with machine-based practical student labs and project work.
Course Overview
Are you in charge of creating Splunk knowledge objects for your organization? Then you will benefit from this course that walks you through the various
knowledge objects and how to create them. Working with Splunk is a comprehensive hands-on course that teaches students how to search, navigate, tag,
build alerts, create simple reports and dashboards in Splunk, and how to Splunk's Pivot interface.
Working in a hands-on learning environment, students will learn how to use Splunk Analytics to provide an efficient way to search large volumes of data.
Students will learn how to run Basic Searches, Save and Share Search Results, Create Tags and Event Types, Create Reports, Create Different Charts, Perform
Calculations and Format Search Data, and Enrich Data with Lookups. Examples will center around financial institution examples.
What Youll Learn: Course Objectives

After completion of this Splunk course, you will be able to:
Get insight into Splunk Search App
Learn to save and share Search Results
Understand the use of fields in searching
Learn Search Fundamentals using Splunk
Explore the available visualizations on the software
Create Reports and different Chart Types
Perform Data Analysis, Calculation and Formatting
Understand and execute various techniques of enriching data lookups
Recommended Audience & Pre-Requisites

This is a technical class for technical people, geared for Users, Administrators, Architects, Developers & Support Engineers who are new to Splunk. This course
is ideal for anyone in your organization who need to examine and use IT data.
Ideal attendees would include:
Beginners in Splunk who want to enhance their knowledge about this Software usage
System Administrators and Software Developers
Professionals who are eager to learn to search and analyze machine-generated data using a faster and agile software
Course Topics & Agenda

Course Modules 1-4 Day 1 - Morning
Module 1 - Basic Understanding of Architecture (Overview)
What are the components?
Discussion on Forwarders- UF/HF
Common ports for the set up
License Master/Slave relationship
Understanding of Deployment Server and Indexer
Module 2 - Introduction to Splunk's User Interface
Understand the uses of Splunk
Define Splunk Apps
Learn basic navigation in Splunk
Hands on Lab covering: Basic Navigation
End of Module Hands-on Quiz
Module 3 - Searching
Run basic searches
Set the time range of a search
Hands on Lab covering: Run basic searches,
Identify the contents of search results
Refine searches
Hands on Lab covering: Identify the contents of search results, Refine searches
Use the timeline
Work with events
Hands on Lab covering: Use the timeline, Work with events
Control a search job
Save search results
Hands on Lab covering: Control a search job, Save search results
Module 4 - Using Fields in Searches

Understand fields
Use fields in searches
Use the fields sidebar
Hands on Lab covering: Understand Fields, Use fields in searches, Use the fields sidebar
Course Modules 5-7 Day 1 - Afternoon

Module 5- Creating Reports and Visualizations
Save a search as a report
Edit reports
Create reports that include visualizations such as charts and tables
Hands on Lab covering: Save a search as a report, Edit Reports, Create reports that include visualizations such as charts and tables.
Add reports to a dashboard
Create an instant pivot from a search
Hands on Lab covering: Add reports to a dashboard, Create an instant pivot from a search.
End of Module Hands on Quiz
Module 6 - Working with Dashboards
Create a dashboard
Add a report to a dashboard
Hands on Lab covering: Create a dashboard, Add a report to a dashboard
Add a pivot report to a dashboard
Edit a dashboard
Hands on Lab covering: Add a pivot report to a dashboard, Edit a dashboard.
Module 7 - Search Fundamentals

Review basic search commands and general search practices
Examine the anatomy of a search
Use the following commands to perform searches:
Fields
Table
Rename
Rex
Multikv
Hands on Lab covering: Review basic search commands and general search practices, Examine the anatomy of a search, Use the following commands to
perform searches: Fields, Table, Rename, Rex, Multikv.
End of Module Hands on Quiz.
Course Modules 8-10 Day 2 Morning (Deep Dive Topics)
Module 8 - Reporting Commands, Part 1
Use the following commands and their functions:
Top
Rare
Hands on Lab covering: Top, Rare
Stats
Add coltotals
Hands on Lab covering: Stats, Add Coltotals
Explore the available visualizations
Create a basic chart
Split values into multiple series
Hands on Lab covering: Explore the available visualizations, Create a basic chart, Split values into multiple series
Omit null and other values from charts
Create a time chart

Chart multiple values on the same timeline
Hands on Lab covering: Omit null and other values from charts, Create a time chart, Chart multiple values on the same timeline
Format charts
Explain when to use each type of reporting command
Hands on Lab covering: Format Charts, Explain when to use each type of reporting command.
End of Module hands on Quiz
Module 10 - Analyzing, Calculating, and Formatting Results

Using the eval command
Perform calculations
Convert values
Hands on Lab covering: Using the eval command, Perform calculations, Convert values.
Round values
Format values
Hands on Lab covering: Round values, Format values
Use conditional statements
Further filter calculated results
Hands on Lab covering: Use conditional statements, Further filter calculated results
Course Modules 11-12 Day 2 Afternoon (Deep Dive Topics)
Module 11 - Creating Field Aliases and Calculated Fields
Define naming conventions
Create and use field aliases
Create and use calculated fields
Hands on Lab covering: Define naming conventions, Create and use field aliases, Create and use calculated fields.
Module 12 - Creating Field Extractions
Perform field extractions using Field Extractor
Hands on Lab covering: Perform field extractions using Field Extractor
Course Modules 13-15 Day 3 - Morning

Module 13 - Creating Tags and Event Types
Create and use tags
Describe event types and their uses
Create an event type
Hands on Lab covering: Create and use tags, Describe event types and their uses, create and event type.
Module 14 - Creating Workflow Actions
Describe the function of a workflow action
Create a GET workflow action
Hands on Lab covering: Describe the function of a workflow action, Create a GET workflow action
Create a POST workflow action
Create a Search workflow action
Hands on Lab covering: Create a POST workflow action, Create a SEARCH workflow action
Module 15 - Creating and Managing Alerts
Describe alerts
Create alerts
View fired alerts
Hands on Lab covering: Describe alerts, Create alerts, View fired alerts
Course Modules 16-17 Day 3 - Afternoon
Module 16 - Creating and Using Macros
Describe macros
Manage macros
Create and use a basic macro
Hands on Lab covering: Describe macros, Manage macros, Create and use a basic macro.
Define arguments and variables for a macro
Add and use arguments with a macro
Hands on Lab covering: Define arguments and variable for a macro, Add and use arguments with a macro.
Module 17 - Using Pivot

Describe Pivot
Understand the relationship between data models and pivot
Select a data model object
Hands on Lab covering: Describe Pivot, Understand the relationship between data models and pivot, Select a data model object.
Create a pivot report
Save pivot report as a dashboard
Hands on Lab covering: Create a pivot report, Save pivot report as a dashboard.
Post Course Final Quiz
At the end of class, each attendee will take a Post Course Quiz that will gauge the students retention of the skills and topics covered throughout the course.
The quiz will be distributed either on paper or online at the end of class and graded promptly.
Module 1 - Basic Understanding of Architecture (Overview)
What are the components?

Discussion on Forwarders- UF/HF
Common ports for the set up
License Master/Slave relationship
Understanding of Deployment Server and Indexer
Section 1-What are the components?

Splunk Enterprise performs three key functions as it moves data through the data pipeline. First, it consumes data from files, the
network, or elsewhere. Then it indexes the data. (Actually, it first parses and then indexes the data, but for purposes of this
discussion, we consider parsing to be part of the indexing process.) Finally, it runs interactive or scheduled searches on the indexed
data.
You can split this functionality across multiple specialized instances of Splunk Enterprise, ranging in number from just a few to
thousands, depending on the quantity of data you're dealing with and other variables in your environment. You might, for example,
create a deployment with many instances that only consume data, several other instances that index the data, and one or more
instances that handle search requests. These specialized instances are known collectively as components. There are several types of
components.
For a typical mid-size deployment, for example, you can deploy lightweight versions of Splunk Enterprise, called forwarders, on the
machines where the data originates. The forwarders consume data locally and then forward the data across the network to another
Splunk Enterprise component, called the indexer. The indexer does the heavy lifting; it indexes the data and runs searches. It should
reside on a machine by itself. The forwarders, on the other hand, can easily co-exist on the machines generating the data, because the
data-consuming function has minimal impact on machine performance.
This diagram shows several forwarders sending data to a single indexer:
As you scale up, you can add more forwarders and indexers. For a larger deployment, you might have hundreds of forwarders sending
data to a number of indexers. You can use load balancing on the forwarders, so that they distribute their data across some or all of the
indexers. Not only does load balancing help with scaling, but it also provides a fail-over capability if one of the indexers goes down.
The forwarders automatically switch to sending their data to any indexers that remain alive.
10
In this diagram, each forwarder load-balances its data across two indexers:
11
These are the fundamental components and features of a Splunk Enterprise distributed environment:
Indexers.
Forwarders.
Search heads.
Deployment server.
Indexer
A Splunk Enterprise instance that indexes data, transforming raw data into events and placing the results into an index. It also
searches the indexed data in response to search requests.
The indexer also frequently performs the other fundamental Splunk Enterprise functions: data input and search management. In
larger deployments, forwarders handle data input and forward the data to the indexer for indexing. Similarly, although indexers
always perform searches across their own data, in larger deployments, a specialized Splunk Enterprise instance, called a search head,
handles search management and coordinates searches across multiple indexers.
Forwarder
A Splunk Enterprise instance that forwards data to another Splunk Enterprise instance, such as an indexer or another forwarder, or to
a third-party system.
There are three types of forwarders:
12
A universal forwarder is a dedicated, streamlined version of Splunk Enterprise that contains only the essential components
needed to send data.
A heavy forwarder is a full Splunk Enterprise instance, with some features disabled to achieve a smaller footprint.
A light forwarder is a full Splunk Enterprise instance, with most features disabled to achieve a small footprint. The universal
forwarder supersedes the light forwarder for nearly all purposes. The light forwarder has been deprecated as of Splunk
Enterprise version 6.0.0.
The universal forwarder is the best tool for forwarding data to indexers. Its main limitation is that it forwards only unparsed data. To
send event-based data to indexers, you must use a heavy forwarder.
Search Heads
In a distributed search environment, a Splunk Enterprise instance that handles search management functions, directing search
requests to a set of search peers and then merging the results back to the user.
A Splunk Enterprise instance can function as both a search head and a search peer. A search head that performs only searching, and
not any indexing, is referred to as a dedicated search head.
Search head clusters are groups of search heads that coordinate their activities.
Deployment Server
A Splunk Enterprise instance that acts as a centralized configuration manager, grouping together and collectively managing any
number of Splunk Enterprise instances. Instances that are remotely configured by deployment servers are called deployment clients.
The deployment server downloads updated content, such as configuration files and apps, to deployment clients. Units of such content
are known as deployment apps.
13
Section 2-Discussion on Forwarders- UF/HF

The universal forwarder
The universal forwarder is Splunk's new lightweight forwarder. You use it to gather data from a variety of inputs and forward the data
to a Splunk Enterprise server for indexing and searching. You can also forward data to another forwarder, as an intermediate step
before sending the data onwards to an indexer.
The universal forwarder's sole purpose is to forward data. Unlike a full Splunk Enterprise instance, you cannot use the universal
forwarder to index or search data. To achieve higher performance and a lighter footprint, it has several limitations:
The universal forwarder has no searching, indexing, or alerting capability.

The universal forwarder does not parse data.
Heavy and light forwarders

While the universal forwarder is generally the preferred way to forward data, you might have reason (legacy-based or otherwise) to
use heavy forwarders as well. Unlike the universal forwarder, which is an entirely separate, streamlined executable, both heavy and
light forwarders are actually full Splunk Enterprise instances with certain features disabled.
A heavy forwarder (sometimes referred to as a "regular forwarder") has a smaller footprint than a Splunk Enterprise indexer but
retains most of the capability, except that it lacks the ability to perform distributed searches. Much of its default functionality, such as
Splunk Web, can be disabled, if necessary, to reduce the size of its footprint. A heavy forwarder parses data before forwarding it and
can route data based on criteria such as source or type of event.
14
This table summarizes the similarities and differences among the three types of forwarders:
Features and capabilities
Universal forwarder
Heavy forwarder
Type of Splunk Enterprise instance
Dedicated executable
Full Splunk Enterprise, with some

features disabled
Footprint (memory, CPU load)
Smallest
Medium-to-large (depending on enabled

features)
Bundles Python?
No
Yes
Handles data inputs?
All types (but scripted inputs might

require Python installation)
All types
Forwards to Splunk Enterprise?
Yes
Yes
Forwards to 3rd party systems?
Yes
Yes
Serves as intermediate forwarder?
Yes
Yes
Indexer acknowledgment (guaranteed

delivery)?
Optional
Optional (version 4.2+)
Load balancing?
Yes
Yes
Data cloning?
Yes
Yes
Per-event filtering?
No
Yes
Event routing?
No
Yes
Event parsing?
No
Yes
Local indexing?
No
Optional, by setting
indexAndForward attribute in
outputs.conf
Searching/alerting?
No
Optional
Splunk Web?
No
Optional
15
Section 3- Common ports for the set up
Splunk configures two ports at installation time:
The HTTP/HTTPS port. This port provides the socket for Splunk Web. It defaults to 8000.
The management port. This port is used to communicate with the splunkd daemon. Splunk Web talks to splunkd on this
port, as does the command line interface and any distributed connections from other servers. This port defaults to 8089.
Let's login to our lab environment

Please go to: http://www.uxcreate.com/guacamole
User name: admin
Password: admin
Your instructor will give you your machine number. Please remember your machine number throughout the training session.
Then please go to Start > All Programs > Splunk Enterprise > Splunk Enterprise
The Splunk web interface should come up.
The login details : username: admin
password: admin
16
Section 4-License Master/Slave relationship
Splunk Enterprise takes in data from sources you designate and processes it so that you can analyze it. We call this process indexing.
Splunk Enterprise licenses specify how much data you can index per calendar day (from midnight to midnight by the clock on the
license master).
Any host in your Splunk Enterprise infrastructure that performs indexing must be licensed to do so. You can either run a standalone
indexer with a license installed locally, or you can configure one of your Splunk Enterprise instances as a license master and set up a
license pool from which other indexers, configured as license slaves, can draw.
When a license master instance is configured, and license slaves are added to it, the license slaves communicate their usage to the
license master every minute. If the license master is unreachable for any reason, the license slave starts a 72 hour timer.
If the license slave cannot reach the license master for 72 hours, search is blocked on the license slave (although indexing continues).
Users cannot search data in the indexes on the license slave until that slave can reach the license master again.
17
Section 5-Understanding of Deployment Server and Indexer
The indexer is the Splunk Enterprise component that creates and manages indexes. The primary functions of an indexer are:
Indexing incoming data.

Searching the indexed data.
In single-machine deployments consisting of just one Splunk Enterprise instance, the indexer also handles the data input and search
management functions.
For larger-scale needs, indexing is split out from the data input function and sometimes from the search management function as well.
In these larger, distributed deployments, the indexer might reside on its own machine and handle only indexing, along with searching
of its indexed data. In those cases, other Splunk Enterprise components take over the non-indexing roles.
For instance, you might have a set of Windows and Linux machines generating events, which need to go to a central indexer for
consolidation. Usually the best way to do this is to install a lightweight instance of Splunk Enterprise, known as a forwarder, on each
of the event-generating machines. These forwarders handle data input and send the data across the network to the indexer residing on
its own machine.
Similarly, in cases where you have a large amount of indexed data and numerous concurrent users searching on it, it can make sense to
split off the search management function from indexing. In this type of scenario, known as distributed search, one or more search
heads distribute search requests across multiple indexers. The indexers still perform the actual searching of their own indexes, but the
search heads manage the overall search process across all the indexers and present the consolidated search results to the user.
18
Here's an example of a scaled-out deployment:
A deployment server uses server classes to determine what content to deploy to groups of deployment clients. The forwarder
management interface offers an easy way to create, edit, and manage server classes.
19
Module 2 - Introduction to Splunk's User Interface
Understand the uses of Splunk

Define Splunk Apps
Learn basic navigation in Splunk
20
Section 1-Understand the uses of Splunk

Splunk Enterprise makes it simple to collect, analyze and act upon the untapped value of the big data generated by your technology
infrastructure, security systems and business applicationsgiving you the insights to drive operational performance and business
results.
By monitoring and analyzing everything from customer clickstreams and transactions to security events and network activity, Splunk
Enterprise helps you gain valuable Operational Intelligence from your machine-generated data.
And with a full range of powerful search, visualization and pre-packaged content for use-cases, any user can quickly discover and
share insights. Just point your raw data at Splunk Enterprise and start analyzing your world.
Collects and indexes log and machine data from any source
Powerful search, analysis and visualization capabilities empower users of all types
Apps provide solutions for security, IT ops, business analysis and more
Enables visibility across on premise, cloud and hybrid environments
Delivers the scale, security and availability to suit any organization
Available as a software or SaaS ( Software as a Solution) solution
21
Section 2-Define Splunk Apps
A Splunk App is a prebuilt collection of dashboards, panels and UI elements powered by saved searches and packaged for a specific
technology or use case to make Splunk immediately useful and relevant to different roles.
As an alternative to using Splunk for searching and exploring, you can use Splunk Apps to gain the specific insights you need from
your machine data.
You can also apply user/role based permissions and access controls to Splunk Apps, thus providing a level of control when you are
deploying and sharing Apps across your organization.
Apps can be opened from the Splunk Enterprise Home Page, from the App menu, or from the Apps section of Settings.
22
Section 3-Learn basic navigation in Splunk

About SplunkHome
Splunk Home is your interactive portal to the data and apps accessible from this Splunk instance. The main parts of
Home include the Splunk Enterprise navigation bar, the Apps menu, the Explore Splunk Enterprise panel, and a
custom default dashboard (not shown here).
Apps
The Apps panel lists the apps that are installed on your Splunk instance that you have permission to view. Select the
app from the list to open it.
For an out-of-the-box Splunk Enterprise installation, you see one App in the workspace: Search & Reporting. When
you have more than one app, you can drag and drop the apps within the workspace to rearrange them.
You can do two actions on this panel:

Click the gear icon to view and manage the apps that are installed in your Splunk instance.
23
Click the plus icon to browse for more apps to install.
Explore Splunk Enterprise

The options in the Explore Splunk Enterprise panel help you to get started using Splunk Enterprise. Click on the icons to
open the Add Data view, browse for new apps, open the Splunk Enterprise Documentation, or open Splunk Answers.
About the Splunk bar
Use the Splunk bar to navigate your Splunk instance. It appears on every page in Splunk Enterprise. You can use it to
switch between apps, manage and edit your Splunk configuration, view system-level messages, and monitor the
progress of search jobs.
The following screenshot shows the Splunk bar in Splunk Home.
The Splunk bar in another view, such as the Search & Reporting app's Search view, also includes an App menu next
to the Splunk logo.
24
Return to Splunk Home

Click the Splunk logo on the navigation bar to return to Splunk Home from any other view in Splunk Web.
Settings menu
The Settings menu lists the configuration pages for Knowledge objects, Distributed environment settings, System and
licensing, Data, and Authentication settings. If you do not see some of these options, you do not have the permissions to
view or edit them.
25
User menu
The User menu here is called "Administrator" because that is the default user name for a new installation. You can
change this display name by selecting Edit account and changing the Full name. You can also edit the time zone
settings, select a default app for this account, and change the account's password. The User menu is also where you
Logout of this Splunk installation.
Messages menu
All system-level error messages are listed here. When there is a new message to review, a notification displays as a count
next to the Messages menu. Click the X to remove the message.
Activity menu
The Activity menu lists shortcuts to the Jobs, Triggered alerts, and System Activity views.
Click Jobs to open the search jobs manager window, where you can view and manage currently running
searches.
26
Click Triggered Alerts to view scheduled alerts that are triggered. This tutorial does not discuss saving and
scheduling alerts. See "About alerts" in the Alerting Manual.
Click System Activity to see Dashboards about user activity and status of the system.
Help
Click Help to see links to Video Tutorials, Splunk Answers, the Splunk Support Portal, and online Documentation.
Find
Use Find to search for objects within your Splunk Enterprise instance. Find performs non-case sensitive matches on the
ID, labels, and descriptions in saved objects. For example, if you type in "error", it returns the saved objects that contain
the term "error".
These saved objects include Reports, Dashboards, Alerts, and Data models. The results appear in the list
separated by the categories where they exist.
27
You can also run a search for error in the Search & Reporting app by clicking
Open error in search.

Take your time exploring the Splunk Web interface
28

Please refer to your virtual machine for test
29
Module 3 - Searching
Run basic searches

Hands on Lab covering: Run basic searches,
Identify the contents of search results
Refine searches
Hands on Lab covering: Identify the contents of search results, Refine searches
Use the timeline
Work with events
Hands on Lab covering: Use the timeline, Work with events
Control a search job
Save search results
Hands on Lab covering: Control a search job, Save search results
30
Run basic searches

Types of searches
Before delving into the language and syntax of search, you should ask what you are trying to accomplish. Generally, after getting data
into Splunk, you want to:
Investigate to learn more about the data you just indexed or to find the root cause of an issue.
Summarize your search results into a report, whether tabular or other visualization format.
Because of this, you might hear us refer to two types of searches: Raw event searches and Report-generating searches.
Raw event searches
Raw event searches are searches that just retrieve events from an index or indexes and are typically done when you want to analyze a
problem. Some examples of these searches include: checking error codes, correlating events, investigating security issues, and
analyzing failures. These searches do not usually include search commands (except search, itself), and the results are typically a list
of raw events.
Transforming searches
Transforming searches are searches that perform some type of statistical calculation against a set of results. These are searches
where you first retrieve events from an index and then pass them into one or more search commands. These searches will always
require fields and at least one of a set of statistical commands. Some examples include: getting a daily count of error events, counting
the number of times a specific user has logged in, or calculating the 95th percentile of field values.
31
Information density
Whether you're retrieving raw events or building a report, you should also consider whether you are running a search for sparse or
dense information:
Sparse searches are searches that look for single event or an event that occurs infrequently within a large
set of data. You've probably heard these referred to as 'needle in a haystack' or "rare term" searches. Some
examples of these searches include: searching for a specific and unique IP address or error code.
Dense searches are searches that scan through and report on many events. Some examples of these
searches include: counting the number of errors that occurred or finding all events from a specific host.
Search and knowledge

As you search, you may begin to recognize patterns and identify more information that could be useful as searchable fields. You can
configure Splunk to recognize these new fields as you index new data or you can create new fields as you search. Whatever you learn,
you can use, add, and edit this knowledge about fields, events, and transactions to your event data. This capturing of knowledge helps
you to construct more efficient searches and build more detailed reports.
32
The anatomy of a search

To better understand how search commands act on your data, it helps to visualize all your indexed data as a table. Each search
command redefines the shape of your table.
For example, let's take a look at the following search.
sourcetype=syslog ERROR | top user | fields - percent
The Disk represents all of your indexed data and it's a table of a certain size with columns represent fields and rows representing
events. The first intermediate results table shows fewer rows--representing the subset of events retrieved from the index that matched
the search terms "sourcetype=syslog ERROR". The second intermediate results table shows fewer columns, representing the results of
the top command, "top user", which summarizes the events into a list of the top 10 users and displays the user, count, and percentage.
Then, "fields - percent" removes the column that shows the percentage, so you are left with a smaller final results table.
33
Quotes and escaping characters

Generally, you need quotes around phrases and field values that include white spaces, commas, pipes, quotes, and/or brackets.
Quotes must be balanced, an opening quote must be followed by an unescaped closing quote. For example:
A search such as error | stats count will find the number of events containing the string error.
A search such as ... | search "error | stats count" would return the raw events containing error, a pipe,
stats, and count, in that order.
Additionally, you want to use quotes around keywords and phrases if you don't want to search for their default meaning, such as
Boolean operators and field/value pairs. For example:
A search for the keyword AND without meaning the Boolean operator: error "AND"
A search for this field/value phrase: error "startswith=foo"
The backslash character (\) is used to escape quotes, pipes, and itself. Backslash escape sequences are still expanded inside quotes.
For example:
The sequence \| as part of a search will send a pipe character to the command, instead of having the pipe
split between commands.
The sequence \" will send a literal quote to the command, for example for searching for a literal quotation
mark or inserting a literal quotation mark into a field using rex.
The \\ sequence will be available as a literal backslash in the command.
If Splunk does not recognize a backslash sequence, it will not alter it.
For example \s in a search string will be available as \s to the command, because \s is not a known escape
sequence.
However, in the search string \\s will be available as \s to the command, because \\ is a known escape
sequence that is converted to \.
34
Asterisks, *, can not be searched for using a backslash to escape the character. Splunk treats the asterisk character as a major
breaker. Because of this, it will never be in the index. If you want to search for the asterisk character, you will need to run a postfiltering regex search on your data:
index=_internal | regex ".*\*.*"
Examples
Example 1: myfield is created with the value of 6.
... | eval myfield="6"
Example 2: myfield is created with the value of ".

... | eval myfield="\""
Example 3: myfield is created with the value of \.

... | eval myfield="\\"
Example 4: This would produce an error because of unbalanced quotes.

... | eval myfield="\"
35
Time is crucial for determining what went wrong. You often know when something happened, if not exactly what happened. Looking
at events that happened around the same time can help correlate results and find the root cause.
Searches run with overly-broad time range wastes system resources and produces more results than you can handle.
Select time ranges to apply to your search
Use the time range picker to set time boundaries on your searches. You can restrict a search with preset time ranges, create custom
time ranges, specify time ranges based on date or date and time, or work with advanced features in the time range picker. These
options are described in the following sections.
Note: If you are located in a different timezone, time-based searches use the timestamp of the event from the instance that indexed the
data.
36
Select from a list of Preset time ranges
37
Define custom Relative time ranges

Use custom Relative time range options to specify a time range for your search that is relative to Now. You can select from the list of
time range units, "Seconds ago", "Minutes ago", and so on.
38
The labels for Earliest and Latest update to match your selection.
The preview boxes below the fields update to the time range as you set it.
39
Define custom Real-time time ranges

The custom Real-time option enables you to specify the start time for your real-time time range window.
40
Define custom Date ranges

Use the custom Date Range option to specify calendar dates in your search. You can choose among options to return events: Between
a beginning and end date, Before a date, and Since a date.
For these fields, you can type the date into the text box or select the date from a calendar:
41
42
Define custom Date & Time ranges

Use the custom Date & Time Range option to specify calendar dates and times for the beginning and ending of your search.
You can type the date into the text box or select the date from a calendar.
43
Use Advanced time range options

Use the Advanced option to specify the earliest and latest search times. You can write the times in Unix (epoch) time or relative time
notation. The epoch time value you enter is converted to local time. This timestamp is displayed under the text field so that you can
verify your entry.
44
Hands on Lab
Part 1 - Basic Concepts
There are a few concepts in the Splunk world that will be helpful for you to understand. Ill cover them in a few sentences,
so try to pay attention. If you want more details, see the Concepts section near the end of this document.
Processing at the time the data is processed: Splunk reads data from a source, such as a file or port, on a host (e.g. "my
machine"), classifies that source into a sourcetype (e.g., "syslog", "access_combined", "apache_error", ...), then extracts
timestamps, breaks up the source into individual events (e.g., log events, alerts, ), which can be a single-line or multiple
lines, and writes each event into an index on disk, for later retrieval with a search.
Processing at the time the data is searched: When a search starts, matching indexed events are retrieved from disk, fields
(e.g., code=404, user=david,...) are extracted from the event's text, and the event is classified by matched against eventtype
definitions (e.g., 'error', 'login', ...). The events returned from a search can then be powerfully transformed using Splunk's
search language to generate reports that live on dashboards.
45
Part 2 - Adding Data
Splunk can eat data from just about any source, including files, directories, ports, and scripts, keeping track of changes to
them as they happen. We're going to start simple and just tell Splunk to index a particular file and not monitor it for
updates:
1.
2.
3.
4.
5.
6.
7.
8.
Go to the Splunk Web interface (e.g. http://localhost:8000) and log in, if you havent already.
Click Settings in the upper right-hand corner of Splunk Web.
Under Settings, click Add Data.
Click Upload Data to upload file.
Click Select File.
Browse and find "websample.log" on your Desktop that we previously saved.
Accept all the default values and just click Submit.
Click Start Searching
Assuming all goes well, websample.log is now indexed, and all the events are timestamped and searchable.
46
Part 3 -Basic Searching
Splunk comes with several Apps, but the only relevant one now is the 'Search' app, which is the interface for generic
searching. (More apps can be downloaded and advanced users can built them themselves.) After logging into Splunk,
select the Search app and let's get started in searching. We'll start out simple and work our way up.
To begin your Splunk search, type in terms you might expect to find in your data. For example, if you want to find events
that might be HTTP 404 errors (i.e., webpage not found), type in the keywords:
http 404
You'll get back all the events that have both HTTP and 404 in their text.
47
Notice that search terms are implicitly AND'd together. The search was the same as "http AND 404". Let's make the search
narrower:
http 404 "like gecko"
48
Using quotes tells Splunk to search for a literal phrase like gecko, which returns more specific results than just searching
for like and gecko because they must be adjacent as a phrase.
Splunk supports the Boolean operators AND, OR, and NOT (must be capitalized), as well as parentheses to enforce
grouping. To get all HTTP error events (i.e., not 200 error code), not including 403 or 404, use this:
http NOT (200 OR 403 OR 404)
Again, the AND operator is implied; the previous search is the same as
http AND NOT (200 OR 403 OR 404)
Splunk supports the asterisk (*) wildcard for searching. For example, to retrieve events that has 40x and 50xx classes of
HTTP status codes, you could try:
http (40* OR 50*)
49
When you index data, Splunk automatically adds fields (i.e., attributes) to each of your events. It does this based on some
text patterns commonly found in IT data, and intermediate users can add their own extraction rules for pulling out additional
fields.
To narrow results with a search, just add attribute=value to your search:
sourcetype=access_combined status=404
This search shows a much more precise version of our first search (i.e., "http 404") because it will only return events that
come from access_combined sources (i.e., webserver events) and that have a status code of 404, which is different than just
having a 404 somewhere in the text. The 404 has to be found where a status code is expected on the event and not just
anywhere. In addition to <attribute>=<value>, you can also do != (not equals), and <, >, >=, and <= for numeric fields.
Part 4- Now heres your turn on your own:
1. Upload the file LoanStats3a.csv located on your desktop

2. Search for entries that contain the word " divorced"
3. Search for entries that are divorced and renting
50
Part 5 - Search App

Now click on Search on the Main toolbar
You will get the following screen:
51
Click on the Data Summary button, you will get:
Click on the Sources tab, you will get:
52
Now you can choose websample.log, you will get:
53
Part 6 - Lets upload another sample file:

1.
2.
3.
4.
Please upload sampledata.zip, whichh is located on the Desktop

Notice there is no preview.
Please take the defaults and start Searching
On the Sourcetypes panel, click access_combined_wcookie
54
You are a member of the Customer Support team for the online Flower & Gift shop. This is your first day on the job. You want to
learn some more about the shop. Some questions you want answered are:
What does the store sell?

How much does each item cost?
How many people visited the site?
How many bought something today?
What is the most popular item that is purchased each day?
It's your first day of work with the Customer Support team for the online Flower & Gift shop. You're just starting to dig into the Web
access logs for the shop, when you receive a call from a customer who complains about trouble buying a gift for
his girlfriend--he keeps hitting a server error when he tries to complete a purchase. He gives you his IP address, 10.2.1.44.
55
1. Type the customer's IP address into the search bar:

sourcetype="access_combined_wcookie" 10.2.1.44
As you type into the search bar, Splunk's search assistant opens.
Search assistant shows you typeahead, or contextual matches and completions for each keyword as
you type it into the search bar. These contextual matches are based on what's in your data. The entries
under matching terms update as you continue to type because the possible completions for your term
change as well.
56
Part 7 - Time Ranges

Try different time ranges like the previous week within the search toolbar
57
Identify the contents of search results and refine searches

Splunk supports the Boolean operators: AND, OR, and NOT. When you include Boolean expressions in your search,
the operators have to be capitalized.
Also you can mouse over results to refine searches
58
Hands on Lab
1.
Please choose the Data Source LoanStats3a.csv. Remember click on Search on the Toolbar and then click on
the Data Summary Button.
2.
Search for the word : Status
3.
Then click on the word Paid and add to the search
4.
Click on the word : RENT and exclude from search
BONUS LAB:
1.
Without the use of fields, find the status of Not Paid and Not Mortgage
59
Use the timeline
The timeline is a visual representation of the number of events returned by a search over a selected time range. The timeline is a type
of histogram, where the range is broken up into smaller time intervals (such as seconds, minutes, hours, or days), and the count of
events for each interval appears in column form.
When you use the timeline to display the results of real-time searches, the timeline represents the sliding time range window
covered by the real-time search.
Mouseover a bar to see the count of events. Click on a bar to drill-down to that time range. Drilling down in this way does not run a
new search, it just filters the results from the previous search. You can use the timeline to highlight patterns or clusters of events or
investigate peaks (spikes in activity) and lows (possible server downtime) in event activity.
Change the timeline format
The timeline is located in the Events tab above the events listing. It shows the count of events over the time range that the search was
run. Here, the timeline shows web access events over the Previous business week.
60
Format options are located in the Format Timeline menu:
You can hide the timeline (Hidden) and display a Compact or Full view of it. You can also toggle the timeline scale between linear
(Linear Scale) or logarithmic (Log Scale).
61
When Full is selected, the timeline is taller and displays the count on the y-axis and time on the x-axis.
Zoom in and zoom out to investigate events
Zoom and selection options are located above the timeline. At first, only the Zoom Out option is available.
The timeline legend is on the top right corner of the timeline. This indicates the scale of the timeline. For example, 1 minute per
column indicates that each column represents a count of events during that minute. Zooming in and out changes the time scale. For
example, if you click Zoom Out the legend will indicate that each column now represents an hour instead of a minute.
When you mouse over and select bars in the timeline, the Zoom to Selection or Deselect options become available.
62
Mouse over and click on the tallest bar or drag your mouse over a cluster of bars in the timeline. The events list updates to display
only the events that occurred in that selected time range. The time range picker also updates to the selected time range. You can cancel
this selection by clicking Deselect.
When you Zoom to Selection, you filter the results of your previous search for your selected time period. The timeline and events list
update to show the results of the new search.
63
You cannot Deselect after you zoomed into a selected time range. But, you can Zoom Out again.
64
Work with events

An event is a single piece of data in Splunk software, similar to a record in a log file or other data input. When data
is indexed, it is divided into individual events. Each event is given a timestamp, host, source, and source type.
Often, a single event corresponds to a single line in your inputs, but some inputs (for example, XML logs) have
multiline events, and some inputs have multiple events on a single line. When you run a successful search, you get
back events.
65
Hands on Lab
Back at the Flower & Gift shop, let's continue with the customer (10.2.1.44) you were assisting. He reported an error while
purchasing a gift for his girlfriend. You confirmed his error, and now you want to find the cause of it.
Continue with the last search, which showed you the customer's failed purchase attempts.
1. Type purchase into the search bar and run the search:
sourcetype="access_combined_wcookie" 10.2.1.44 purchase
When you search for keywords, your search is not case-sensitive and Splunk retrieves the events that contain those
keywords anywhere in the raw text of the event's data
Use Boolean operators
If you're familiar with Apache server logs, in this case the access_combined format, you'll notice that
most of these events have an HTTP status of 200, or Successful. These events are not interesting for
you right now, because the customer is reporting a problem.
Splunk supports the Boolean operators: AND, OR, and NOT. When you include
Boolean expressions in your search, the operators have to be capitalized.
2. Use the Boolean NOT operator to quickly remove all of these Successful page
requests. Type in:
66
sourcetype="access_combined_wcookie" 10.2.1.44 purchase NOT 200
The AND operator is always implied between search terms. So the search in Step 5 is
the same as:
sourcetype="access_combined_wcookie" AND 10.2.1.44 AND purchase NOT 200
You notice that the customer is getting HTTP server (503) and client (404) errors. But, he specifically
mentioned a server error, so let's quickly remove events that are irrelevant.
Another way to add Boolean clauses quickly and interactively to your search is to use your search
results. Splunk lets you highlight and select any segment from
67
Timeline Usage
Continue with the last search, which showed you the customer's failed purchase attempts.
1. Search for:
In the
last topic, you really just focused on the search results listed in the events viewer area of
this dashboard. Now, let's take a look at the timeline.
sourcetype="access_combined_wcookie" 10.2.1.44 purchase NOT 200 NOT 404
The location of each bar on the timeline corresponds to an instance when the events that match your
search occurred. If there are no bars at a time period, no events were found then.
2. Mouse over one of the bars.
A tooltip pops up and displays the number of events that Splunk found during the time span of that bar (1
bar = 1 hour).
68
The taller the bar, the more events occurred at that time. Often seeing spikes in the number of events
or no events is a good indication that something has happened.
3. Click one of the bars, for example the tallest bar.
This updates your search results to show you only the events at the time span. Splunk does not run the
search when you click on the bar. Instead, it gives you a preview of the results zoomed-in at the time
range. You can still select other bars at this point.
One hour is still a wide time period to search, so let's narrow the search down more.
4. Double-click on the same bar.
Splunk runs the search again and retrieves only events during that one hour span you selected.
69
You should see the same search results in the Event viewer, but, notice that the search overrides the
time range picker and it now shows "Custom time". (You'll see more of the time range picker later.) Also,
each bar now represents one minute of time (1 bar = 1 min).
5. Double-click another bar.
Once again, this updates your search to now retrieve events during that one minute span of time. Each
bar represents the number of events for one second of time.
Now, you want to expand your search to see everything else, if anything, that happened during this
second.
6. Without changing the time range, replace your previous search in the search bar with:
*
Splunk supports using the asterisk (*) wildcard to search for "all" or to retrieve events based on parts
70
of a keyword. Up to now, you've just searched for Web access logs. This search tells Splunk that you
want to see everything that occurred at this time range:
71
Control search job progress

After you launch a search, you can access and manage information about the search's job without leaving the Search page. Once your
search is running, paused, or finalized, click Job and choose from the available options there.
You can:
Edit the job settings. Select this to open the Job Settings dialog, where you can change the job read permissions, extend the
job lifetime, and get a URL for the job that you can use to share the job with others or put a link to the job in your browser's
bookmark bar.
Send the job to the background. Select this if the search job is slow to complete and you would like to run the job in the
background while you work on other Splunk activities (including running a new search job).
Inspect the job. Opens a separate window and display information and metrics for the search job using the Search Job
Inspector. You can select this action while the search is running or after it completes.
Delete the job. Use this to delete a job that is currently running, is paused, or which has finalized. After you have deleted the
job you can still save the search as a report.
72
Change the search mode

The Search mode controls the search experience. You can set it to speed up searches by cutting down on the event data it returns (Fast
mode), or you can set it to return as much event information as possible (Verbose mode). In Smart mode (the default setting) it
automatically toggles search behavior based on the type of search you're running.
73
Save the results

The Save as menu lists options for saving the results of a search as a Report, Dashboard Panel, Alert, and Event type.
Report: If you would like to make the search available for later use, you can save it as a report. You can run the report again
on an ad hoc basis by finding the report on the Reports listing page and clicking its name.
Dashboard Panel...: Click this if you'd like to generate a dashboard panel based on your search and add it to a new or existing
dashboard.
Alert Click to define an alert based on your search. Alerts run saved searches in the background (either on a schedule or in
real time). When the search returns results that meet a condition you have set in the alert definition, the alert is triggered.
Event Type Event types let you classify events that have common characteristics. If the search doesn't include a pipe operator
or a subsearch , you can use this to save it as an event type. Other search actions
Between the job progress controls and search mode selector are three buttons which enable you to Share, Export, and Print the
results of a search.
74
Click Share to share the job. When you select this, the job's lifetime is extended to 7 days and read permissions are set to
Everyone.
Click Export to export the results. You can select to output to CSV, raw events, XML, or JSON and specify the number of
results to export.
Click Print to send the results to a printer that has been configured.
75
Hands on Lab
1.
2.
Using your file LoanStats3a.csv, save your last search as an event type
Go to Settings, and click on event types to view your saved event type
76

77
Module 4 - Using Fields in Searches
Understand fields
Hands on Lab covering: Understand Fields, Use fields in searches, Use the fields sidebar
78
Understand fields
Fields exist in machine data in many forms. Often, a field is a value (with a fixed, delimited position on the line) or a name and value
pair, where there is a single value to each field name. A field can be multivalued, that is, it can appear more than once in an event and
has a different value for each appearance.
Some examples of fields are clientip for IP addresses accessing your Web server, _time for the timestamp of an event, and host for
domain name of a server. One of the more common examples of multivalue fields is email address fields. While the From field will
contain only a single email address, the To and Cc fields have one or more email addresses associated with them.
In Splunk Enterprise, fields are searchable name and value pairings that distinguish one event from another because not all events will
have the same fields and field values. Fields let you write more tailored searches to retrieve the specific events that you want.
79
Use the following syntax to search for a field: fieldname="fieldvalue" . Field names are case sensitive, but field values are not.
1. Go to the Search dashboard and type the following into the search bar:
sourcetype="access_*"
This indicates that you want to retrieve only events from your web access logs and nothing else.
sourcetype
is a field name and access_* is a wildcarded field value used to match any Apache web access event. Apache web
access logs are formatted as access_common, access_combined, or access_combined_wcookie.
2. In the Events tab, scroll through the list of events.
If you are familiar with the access_combined format of Apache logs, you recognize some of the information in each event, such as:
IP addresses for the users accessing the website.

URIs and URLs for the pages requested and referring pages.
HTTP status codes for each page request.
GET or POST page request methods.
80
To the left of the events list is the Fields sidebar. As Splunk Enterprise retrieves the events that match your search, the Fields sidebar
updates with Selected fields and Interesting fields. These are the fields that Splunk Enterprise extracted from your data.
Selected Fields are the fields that appear in your search results. The default fields host, source, and sourcetype are selected.
You can hide and show the fields sidebar by clicking Hide Fields and Show Fields, respectively.
3. Click All Fields.
The Select Fields dialog box opens, where you can edit the fields to show in the events list.
You see the default fields that Splunk defined. Some of these fields are based on each event's timestamp (everything beginning with
date_*), punctuation (punct), and location (index).
81
Other field names apply to the web access logs. For example, there are clientip, method, and status. These are not default fields.
They are extracted at search time.
This opens the field summary for the action field.
In this set of search results, Splunk Enterprise found five values for action, and that the action field appears in 49.9% of your search
results.
82
Hands on Lab
1. Go back to the Search dashboard and search for web access activity. Select
Other > Yesterday from the time range picker:
sourcetype="access_*"
You were actually using fields all along! Each time you searched for sourcetype=access_*, you told
Splunk to only retrieve events from your web access logs and nothing else.
To search for a particular field, specify the field name and value:
fieldname="fieldvalue"
is a field name and access_combined_wcookie is a field value. Here, the wildcarded value is
used to match all field values beginning with access_ (which would include access_common,
access_combined, and access_combined_wcookie) .
sourcetype
Note: Field names are case sensitive, but field values are not!
2. Scroll through the search results.
If you're familiar with the access_combined format of Apache logs, you will recognize some of the
information in each event, such as:
IP addresses for the users accessing the website.

URIs and URLs for the page request and referring page.
HTTP status codes for each page request.
Page request methods.
83
As Splunk retrieves these events, the Fields sidebar updates with selected fields
and interesting fields. These are the fields that Splunk extracted from your data.
Notice that default fields host, source, and sourcetype are selected fields and are displayed in your
search results:
3. Scroll through interesting fields to see what else Splunk extracted.

You should recognize the field names that apply to the Web access logs. For example, there's
clientip, method, and status. These are not default fields; they have (most likely) been extracted at
search time.
4. Click the Edit link in the fields sidebar.
The Fields dialogue opens and displays all the fields that Splunk extracted.
Available Fields are the fields that Splunk identified from the events in your
current search (some of these fields were listed under interesting fields).
Selected Fields are the fields you picked (from the available fields) to show in
84
your search results (by default, host, source, and sourcetype are selected).
5. Scroll through the list of Available Fields.

You're already familiar with the fields that Splunk extracted from the Web access logs based on your
search. You should also see other default fields that Splunk defined--some of these fields are based on
each event's timestamp (everything beginning with date_*), punctuation (punct), and location (index).
But, you should also notice other extracted fields that are related to the online store. For example,
there are action, category_id, and product_id. From conversations with your coworker, you may
know that these fields are:
Field name
action
Description
what a user does at the online shop.
85
category_id
the type of product a user is viewing or buying.
product_id
the catalog number of the product the user is viewing or buying.
6. From the Available fields list, select action, category_id, and product_id.
7. Click Save.
When you return to the Search view, the fields you selected will be included in your search results if they
exist in that particular event. Different events will have different fields.
86
The fields sidebar doesn't just show you what fields Splunk has captured from your data. It also displays
how many values exist for each of these fields. For the fields you just selected, there are 2 for action, 5
for category_id, and 9 for product_id. This doesn't mean that these are all the values that exist for each
of the fields--these are just the values that Splunk knows about from the results of your search.
What are some of these values?
8. Under selected fields, click action for the action field.
This opens the field summary for the action field.
This window tells you that, in this set of search results, Splunk found two values for action and they are
purchase and update. Also, it tells you that the action field appears in 71% of your search results. This
87
means that three-quarters of the Web access events are related to the purchase of an item or an update
(of the item quantity in the cart, perhaps).
9. Close this window and look at the other two fields you selected, category_id (what
types of products the shop sells) and product_id (specific catalog names for products).
Now you know a little bit more about the information in your data relating to the online Flower and Gift
shop. The online shop sells a selection of flowers, gifts, plants, candy, and balloons. Let's use these
fields, category_id and product_id, to see what people are buying.
Use fields to run more targeted searches
These next two examples compares the results when searching with and without fields.
Example 1
Return to the search you ran to check for errors in your data. Select Other > Yesterday from the
time range picker:
error OR failed OR severe OR (sourcetype=access_* (404 OR 500 OR 503))
88
Run this search again, but this time, use fields in your search.
The HTTP error codes are values of the status field. Now your search looks like this:
error OR failed OR severe OR (sourcetype=access_* (status=404 OR
status=500 OR status=503))
Notice the difference in the count of events between the two searches--because it's a more targeted
search, the second search returns fewer events.
When you run simple searches based on arbitrary keywords, Splunk matches the raw text of your data.
When you add fields to your search, Splunk looks for events that have those specific field/value pairs.
Example 2
Before you learned about the fields in your data, you might have run this search to see how many times
flowers were purchased from the online shop:
sourcetype=access_* purchase flower*
As you typed in "flower", search assistant shows you both "flower" and "flowers' in the typeahead. Since
89
you don't know which is the one you want, you use the wildcard to match both.
If you scroll through the (many) search results, you'll see that some of the events have
action=update and category_id that have a value other than flowers.
These are not events that you wanted!
Run this search instead. Select Other > Yesterday from the time range picker:
sourcetype=access_* action=purchase category_id=flower*
For the second search, even though you still used the wildcarded word "flower*", there is only one value
of category_id that it matches (FLOWERS).
90
Notice the difference in the number of events that Splunk retrieved for each search; the second search
returns significantly fewer events. Searches with fields are more targeted and retrieves more exact
matches against your data.
Now on your own:

1. Bring up the Loan data file
2. Using fields find entries that annual salary is less than 20,000 and they live in the state of
CA. Use addr_state for state
3. Refine the search for the field emp_title where it equals Walmart
91
End of Module Quiz

92
Module 5- Creating Reports and Visualizations

Edit reports
Hands on Lab covering: Save a search as a report, Edit Reports, Create reports that include visualizations such as charts
and tables.
Hands on Lab covering: Add reports to a dashboard, Create an instant pivot from a search.
93
To save your search as a report, click on the Report link. This opens the Save As Report dialog:
From here, you need to do the following:

1. Enter a Title (or name) for your report.
2. Enter an optional Description to remind users what your report does.
3. Indicate if you'd like to include the Splunk Time Range Picker as a part of your report.
94
Once you click Save, Splunk prompts you to either review Additional Settings for your newly created report (Permissions,
Schedule, Acceleration, and Embed), Add (the report) to Dashboard, View the report, or Continue Editing the search:
95
The additional settings that can be made to the report are given as follows:
Permissions: Allows you to set how the saved report is displayed: by owner, by app, or for all apps. In addition, you can make
the report read only or writeable (can be edited).
Schedule: Allows you to schedule the report (for Splunk to run/refresh it based upon your schedule). For example, an interval
like every week, on Monday at 6 AM, and for a particular time range.
Acceleration: Not all saved reports qualify for acceleration and not all users (not even admins) have the ability to accelerate
reports. Generally speaking, Splunk Enterprise will build a report acceleration summary for the report if it determines that the
report would benefit from summarization (acceleration).
Embed: Report embedding lets you bring the results of your reports to large numbers of report stakeholders. With report
embedding, you can embed scheduled reports in external (non-Splunk) websites, dashboards, and portals. Embedded reports
can display results in the form of event views, tables, charts, maps, single values, or any other visualization type. They use the
same formatting as the originating report. When you embed a saved report, you do this by copying a Splunk generated URL
into an HTML-based web page.
96
Edit reports
You can easily edit an existing report. You can edit a report's definition (its search string, pivot setup, or result formatting). You can
also edit its description, permissions, schedule, and acceleration settings.
To edit a report's definition
If you want to edit a report's definition, there are two ways to start, depending on whether you're on the Reports listing page or looking
at the report itself.
If you're on the Reports listing page, locate the report you want to edit, go to the Actions column, and click Open in Search
or Open in Pivot (you'll see one or the other depending on which tool you used to create the report).
If you've entered the report to review its results, click Edit and select Open in Search or Open in Pivot (you'll see one or the
other depending on which tool you used to create the report).
Edit the definition of a report opened in Search

After you open a report in search, you can change the search string, time range, or report formatting. After you rerun the report, a Save
button will be enabled towards the upper right of the report. Click this to save the report. You also have the option of saving your
edited search as a new report.
97
A visualization is a representation of data returned from a search. Most visualizations are graphical representations, however, a
visualization can also be non-graphical. In dashboards, a panel contains one or more visualizations. Visualizations available for
simple XML dashboards include:
chart
event listing
map
table
single value
A chart visualization has several types:
area
bar
bubble
column
filler gauge
line
marker gauge
pie
radial gauge
scatter
98
Hands on Lab covering

1. Click Search on the Toolbar, then click the Data Summary button:
2. Choose the SourceType Tab, and click on access_combined_wcookie:

99
3. Select under Interesting Fields, category_id . Then click under Reports, top values:
100
101
4. It should yield a report:
102
5. Now click on Statistics, notice the table of values:
6. Go back to the Visualization tab, under Format , then investigate all the different options
7. Under the Bar Chart drop, investigate all the different chart types as well
Bonus Lab:
Using the LoanStats3a.csv file, create a report from the data that top values across all the states
103

Once you have created your reports, you can easily add to the dashboard, by clicking Add to Dashboard button
104

From any search, simply select the Statistics tab and click on the Pivot Icon
Let's take a walkthrough:
1. Make sure to pick make interesting fields to be selected fields
105
2. Click the Statistics tab after you have the search you want:
3. Then click the Pivot Icon
106
4. Then you can choose the fields you have selected to Pivot, and click OK :
107
5. Then you can choose a field like annual_inc with a default of Sum to be part of your Pivot column values:
108
6. And then pick a field like addr_state to the row column
109
7. Finally pick a bar chart on the left side
110
Hands on Lab
1. Create a report out of LoanStats3a.csv source that looks into the annual income < 70000 and the addr_state of
CA ,FL, NY
2. Create an instant pivot out of the search from #1 above.
111
112
Module 6 - Working with Dashboards
Create a dashboard
Hands on Lab covering: Create a dashboard, Add a report to a dashboard
Edit a dashboard
Hands on Lab covering: Add a pivot report to a dashboard, Edit a dashboard.
113
Create a dashboard
You can create a dashboard from the search OR you can click on the Dashboard option on the Toolbar
OR
114

Click on Add to Dashboard from your report
115
Hands on Lab:
Let's use the flower shop transactions to create a dashboard and add a report to it
Before you learned about the fields in your data, you might have run this search to see how many times flowers were
purchased from the online shop:
sourcetype=access_* purchase flower*| top limit=20 category_id
1.
2.
3.
4.
Let's save the report of this search as Flowers Category

Click on the view button to view the report
Click Add to DashBoard to add report to Dashboard
Name the Dashboard, Flowers Dashboard
Bonus Lab:
The report out of LoanStats3a.csv source that looks into the annual income < 70000 and the addr_state of CA ,FL,
NY from the last module and create a dashboard
116

From your pivot , you can save as a dashboard panel
117
Edit a dashboard
From your dashboard, you can edit your dashboard from the menu
And then you could, for example edit Panels
118
Hands on Lab:
1. Create an instant pivot, like the one from the previous module out of LoanStats3a.csv source that looks into the
annual income < 70000 and the addr_state of CA ,FL, NY
2. Then add that pivot report to the dashboard
3. Create another report that looks at ALL the annual incomes in the states of CA,FL, NY
4. Add that report to the dashboard created in exercise #1
5. Edit the dashboard panels and add titles to your panels.
Bonus Lab:
1. Create another instant pivot or report and add to the existing dashboard
119
120
Module 7 - Search Fundamentals

Use the following commands to perform searches:
Fields
Table
Rename
Rex
Multikv
121
To successfully use Splunk, it is vital that you write effective searches. Using the index efficiently will make your initial discoveries
faster, and the reports you create will run faster for you and for others. In this chapter, we will cover the following topics:
How to write effective searches

How to search using fields
Understanding time
Saving and sharing searches
Using search terms effectively

The key to creating an effective search is to take advantage of the index. The Splunk index is effectively a huge word index, sliced by
time. The single most important factor for the performance of your searches is how many events are pulled from the disk. The
following few key points should be committed to memory:
Search terms are case insensitive: Searches for error, Error, ERROR, and ErRoR are all the same thing.
Search terms are additive: Given the search item, mary error, only events that contain both words will be found. There are
Boolean and grouping operators to change this behavior; we will discuss in this chapter under Boolean and grouping
operators.
Only the time frame specified is queried: This may seem obvious, but it's very different from a database, which would
always have a single index across all events in a table. Since each index is sliced into new buckets over time, only the buckets
that contain events for the time frame in question need to be queried.
Search terms are words, including parts of words: A search for foo will also match foobar.
122
With just these concepts, you can write fairly effective searches. Let's dig a little deeper, though:
A word is anything surrounded by whitespace or punctuation: For instance, given the log line 2012-0207T01:03:31.104-0600 INFO AuthClass Hello world. [user=Bobby, ip=1.2.3.3], the "words" indexed are
2012,02, 07T01, 03, 31, 104, 0600, INFO, AuthClass, Hello, world, user, Bobby, ip, 1, 2, 3, and 3. This
may seem strange, and possibly a bit wasteful, but this is what Splunk's index is really, really good atdealing with huge
numbers of words across a huge number of events.
Splunk is not grep with an interface: One of the most common questions is whether Splunk uses regular expressions for your
searches. Technically, the answer is no. Splunk does use regex internally to extract fields, including the auto generated fields,
but most of what you would do with regular expressions is available in other ways. Using the index as it is designed is the best
way to build fast searches. Regular expressions can then be used to further filter results or extract fields.
Numbers are not numbers until after they have been parsed at search time: This means that searching for foo>5 will not
use the index, as the value of foo is not known until it has been parsed out of the event at search time. There are different ways
to deal with this behavior, depending on the question you're trying to answer.
Field names are case sensitive: When searching for host=myhost, host must be lowercase. Likewise, any extracted or
configured fields have case sensitive field names, but the values are case insensitive.
Host=myhost will not work
host=myhost will work
host=MyHost will work
Fields do not have to be defined before indexing data: An indexed field is a field that is added to the metadata of an event at
index time. There are legitimate reasons to define indexed fields, but in the vast majority of cases it is unnecessary and is
actually wasteful.
123
Boolean and grouping operators

There are a few operators that you can use to refine your searches (note that these operators must be in uppercase to not be considered
search terms):
AND is implied between terms. For instance, error mary (two words separated by a space) is the same as
error AND mary.
OR allows you to specify multiple values. For instance, error OR mary means find any event that contains
either word.
NOT applies to the next term or group. For example, error NOT mary would find events that contain error but
do not contain mary.
The quote marks ("") identify a phrase. For example, "Out of this world" will find this exact sequence of
words. Out of this world would find any event that contains all of these words, but not necessarily in that
order.
Parentheses ( ( ) ) is used for grouping terms. Parentheses can help avoid confusion in logic. For instance,
these two statements are equivalent:
bob error OR warn NOT debug

bob AND (error OR warn)) AND NOT debug
The equal sign (=) is reserved for specifying fields. Searching for an equal sign can be accomplished by
wrapping it in quotes. You can also escape characters to search for them. \= is the same as "=".
Brackets ( [ ] ) are used to perform a subsearch.
124
You can use these operators in fairly complicated ways if you want to be very specific, or even to find multiple sets of events in a
single query. The following are a few examples:
error mary NOT jacky

error NOT (mary warn) NOT (jacky error)
index=myapplicationindex ( sourcetype=sourcetype1 AND ( (bob NOT error) OR (mary AND warn) ) ) OR (
sourcetype=sourcetype2 (jacky info) )
This can also be written with some whitespace for clarity:

index=myapplicationindex
(
sourcetype=security
AND
(
(bob NOT error)
OR
(mary AND warn)
)
)
OR
(
sourcetype=application
(jacky info)
)
125
Clicking to modify your search

Though you can probably figure it out by just clicking around, it is worth discussing the behavior of the GUI when moving your
mouse around and clicking.
Clicking on any word or field value will give you the option to Add to search or Exclude from search (the
existing search) or (create a) New search:
Clicking on a word or a field value that is already in the query will give you the option to remove it (from the
existing query) or, as above, (create a) new (search):
126
Event segmentation
In previous versions of Splunk, event segmentation was configurable through a setting in the Options dialog. In version 6.2, the
options dialog is not present although segmentation (discussed later in this chapter under field widgets section) is still an important
concept, it is not accessible through the web interface/options dialog in this version.
Field widgets
Clicking on values in the Select Fields dialog (the field picker), or in the field value widgets underneath an event, will again give us an
option to append (add to) or exclude (remove from) our search or, as before, to start a new search.
For instance, if source="C:\Test Data\TM1ProcessError_20140623213757_temp.log" appears under your event, clicking on that
value and selecting Add to search will append source="C:\\Test Data\\TM1ProcessError_20140623213757_temp.log" to your
search:
127
To use the field picker, you can click on the link All Fields (see the following image):
Expand the results window by clicking on > in the far-left column. Clicking on a result will append that item to the current search:
128
If a field value looks like key=value in the text of an event, you will want to use one of the field widgets instead of clicking on the
raw text of the event. Depending on your event segmentation setting, clicking on the word will either add the value or key=value. The
former will not take advantage of the field definition; instead, it will simply search for the word. The latter will work for events that
contain the exact quoted text, but not for other events that actually contain the same field value extracted in a different way.
129
Time
Clicking on the time next to an event will open the _time dialog (shown in the following image) allowing you to change the search to
select Events Before or After a particular time period, and will also have the following choices:
Before this time

After this time
At this time
In addition, you can select Nearby Events within plus, minus, or plus or minus, a number of seconds (the default), milliseconds,
minutes, hours, days, or weeks:
130
One search trick is to click on the time of an event, select At this time, and then use the Zoom out (above the timeline) until the
appropriate time frame is reached.
131
Fields command
Description
Keeps (+) or removes (-) fields from search results based on the field list criteria. If + is specified, only the fields that match one of the
fields in the list are kept. If - is specified, only the fields that match one of the fields in the list are removed. If neither is specified,
defaults to +.
Important: The leading underscore is reserved for all internal Splunk Enterprise field names, such as _raw and _time. By default,
internal fields _raw and _time are included in output. The fields command does not remove internal fields unless explicitly
specified with:
... | fields - _*
or more explicitly, with:

... | fields - _raw,_time
Note: Be cautious removing the _time field. Statistical commands, such as timechart and chart, cannot display date or time
information without the _time field.
Syntax
fields [+|-] <wc-field-list>
Required arguments
<wc-field-list>
Syntax: <string>, <string>, ...
Description: Comma-delimited list of fields to keep (+) or remove (-). You can use wild card characters in the
field names.
132
Examples
Example 1:
Remove the "host" and "ip" fields.
... | fields - host, ip
Example 2:
Keep only the host and ip fields. Remove all of the internal fields. The internal fields begin with an underscore character, for
example _time.
... | fields host, ip | fields - _*
Example 3:
Keep only the fields 'source', 'sourcetype', 'host', and all fields beginning with 'error'.
... | fields source, sourcetype, host, error*
133
Table command
Description
The table command is similar to the fields command in that it lets you specify the fields you want to keep in your results. Use
table command when you want to retain data in tabular format.
The table command can be used to build a scatter plot to show trends in the relationships between discrete values of your data.
Otherwise, you should not use it for charts (such as chart or timechart) because the UI requires the internal fields (which are the
fields beginning with an underscore, _*) to render the charts, and the table command strips these fields out of the results by default.
Instead, you should use the fields command because it always retains all the internal fields.
Syntax
table <wc-field-list>
Arguments
<wc-field-list>
Syntax: <wc-field> <wc-field> ...
Description: A list of field names. You can use wild card characters in the field names.
Usage
The table command returns a table formed by only the fields specified in the arguments. Columns are displayed in the same order
that fields are specified. Column headers are the field names. Rows are the field values. Each row represents an event.
Command type: The table command is a non-streaming command. If you are looking for a streaming command similar to the table
command, use the fields command.
Field renaming: The table command doesn't let you rename fields, only specify the fields that you want to show in your tabulated
results. If you're going to rename a field, do it before piping the results to table.
134
Rename command
Description
Use the rename command to rename a specified field or multiple fields. This command is useful for giving fields more meaningful
names, such as "Product ID" instead of "pid". If you want to rename multiple fields, you can use wildcards.
Use quotes to rename a field to a phrase:
... | rename SESSIONID AS sessionID
Use wildcards to rename multiple fields:

... | rename *ip AS *IPaddress
If both the source and destination fields are wildcard expressions with the same number of wildcards, the renaming will carry over the
wildcarded portions to the destination expression. See Example 2, below.
Note: You cannot rename one field with multiple names. For example if you had a field A, you cannot do "A as B, A as C" in one
string.
... | stats first(host) AS site, first(host) AS report
Note: You cannot use this command to merge multiple fields into one field because null, or non-present, fields are brought along with
the values. For example, if you had events with either product_id or pid fields, ... | rename pid AS product_id would not
merge the pid values into the product_id field. It overwrites product_id with Null values where pid does not exist for the event.
135
Syntax
rename <wc-field> AS <wc-field>...
Required arguments
wc-field
Syntax: <string>
Description: The name of a field and the name to replace it. You can use wild card characters in the field
names. Names with spaces must be enclosed in quotation marks.
136
Rex command
Description
Use this command to either extract fields using regular expression named groups, or replace or substitute characters in a field using
sed expressions.
The rex command matches the value of the specified field against the unanchored regular expression and extracts the named groups
into fields of the corresponding names. If a field is not specified, the regular expression is applied to the _raw field. Note: Running
rex against the _raw field might have a performance impact.
When mode=sed, the given sed expression used to replace or substitute characters is applied to the value of the chosen field. If a field
is not specified, the sed expression is applied to _raw. This sed-syntax is also used to mask sensitive data at index-time.
Use the rex command for search-time field extraction or string replacement and character substitution.
Syntax
rex [field=<field>] ( <regex-expression> [max_match=<int>] [offset_field=<string>] ) | (mode=sed <sed-expression>)
Required arguments
regex-expression
Syntax: "<string>"
Description: The PCRE regular expression that defines the information to match and extract from the
specified field. Quotation marks are required.
mode
Syntax: mode=sed
Description: Specify to indicate that you are using a sed (UNIX stream editor) expression.
sed-expression
Syntax: "<string>"
Description: When mode=sed, specify whether to replace strings (s) or substitute characters (y) in the
137
matching regular expression. No other sed commands are implemented. Quotation marks are required. Sed
mode supports the following flags: global (g) and Nth occurrence (N), where N is a number that is the
character location in the string.
Optional arguments
field
Syntax: field=<field>
Description: The field that you want to extract information from.
Default: _raw
max_match
Syntax: max_match=<int>
Description: Controls the number of times the regex is matched. If greater than 1, the resulting fields are
multivalued fields.
Default: 1, use 0 to mean unlimited.
offset_field
Syntax: offset_field=<string>
Description: If provided, a field is created with the name specified by <string>. This value of the field has
the endpoints of the match in terms of zero-offset characters into the matched field. For example, if the rex
expression is "(?<tenchars>.{10})", this matches the first ten characters of the field, and the offset_field
contents is "0-9".
Default: unset
Sed expression
When using the rex command in sed mode, you have two options: replace (s) or character substitution (y).
The syntax for using sed to replace (s) text in your data is: "s/<regex>/<replacement>/<flags>"
<regex> is a PCRE regular expression, which can include capturing groups.

<replacement> is a string to replace the regex match. Use \n for backreferences, where "n" is a single digit.
<flags> can be either: g to replace all matches, or a number to replace a specified match.
138
The syntax for using sed to substitute characters is: "y/<string1>/<string2>/"
This substitutes the characters that match <string1> with the characters in <string2>.
Usage
Splunk Enterprise uses perl-compatible regular expressions (PCRE).
When you use regular expressions in searches, you need to be aware of how characters such as pipe ( | ) and backslash ( \ ) are
handled.
Examples
Example 1:
Extract "from" and "to" fields using regular expressions. If a raw event contains "From: Susan To: Bob", then from=Susan and
to=Bob.
... | rex field=_raw "From: (?<from>.*) To: (?<to>.*)"
Example 2:
Extract "user", "app" and "SavedSearchName" from a field called "savedsearch_id" in scheduler.log events. If
savedsearch_id=bob;search;my_saved_search then user=bob , app=search and SavedSearchName=my_saved_search
... | rex field=savedsearch_id "(?<user>\w+);(?<app>\w+);(?<SavedSearchName>\w+)"
139
Example 3:
Use sed syntax to match the regex to a series of numbers and replace them with an anonymized string.
... | rex field=ccnumber mode=sed "s/(\d{4}-){3}/XXXX-XXXX-XXXX-/g"
Example 4:
Display IP address and ports of potential attackers.
sourcetype=linux_secure port "failed password" | rex "\s+(?<ports>port \d+)" | top src_ip ports showperc=0
This search used rex to extract the port field and values. Then, it displays a table of the top source IP addresses (src_ip) and ports the
returned with the search for potential attackers.
140
Multikv command
Description
Extracts field-values from table-formatted events, such as the results of top, netstat, ps, and so on. The multikv command creates a
new event for each table row and assigns field names from the title row of the table.
An example of the type of data multikv is designed to handle:
Name
Josh
Francine
Samantha
Age
42
35
22
Occupation
SoftwareEngineer
CEO
ProjectManager
The key properties here are:
Each line of text represents a conceptual record.

The columns are aligned.
The first line of text provides the names for the data in the columns.
multikv can transform this table from one event into three events with the relevant fields. It works more easily with the fixedalignment though can sometimes handle merely ordered fields.
The general strategy is to identify a header, offsets, and field counts, and then determine which components of subsequent lines should
be included into those field names. Multiple tables in a single event can be handled (if multitable=true), but may require ensuring that
the secondary tables have capitalized or ALLCAPS names in a header row.
Auto-detection of header rows favors rows that are text, and are ALLCAPS or Capitalized.
141
Syntax
multikv [conf=<stanza_name>] [<multikv-option>...]
Optional arguments
conf
Syntax: conf=<stanza_name>
Description: If you have a field extraction defined in multikv.conf, use this argument to reference the stanza
in your search.
<multikv-option>
Syntax: copyattrs=<bool> | fields <field-list> | filter <field-list> | forceheader=<int> | multitable=<bool> |
noheader=<bool> | rmorig=<bool>
Description: Options for extracting fields from tabular events.
Descriptions for multikv options
copyattrs
Syntax: copyattrs=<bool>
Description: When true, multikv copies all fields from the original event to the events generated from that
event. When false, no fields are copied from the original event. This means that the events will have no _time
field and the UI will not know how to display them.
Default: true
fields
Syntax: fields <field-list>
Description: Limit the fields set by the multikv extraction to this list. Ignores any fields in the table which
are not on this list.
filter
Syntax: filter <term-list>
Description: If specified, multikv skips over table rows that do not contain at least one of the strings in the
filter list. Quoted expressions are permitted, such as "multiple words" or "trailing_space ".
forceheader
142
Syntax: forceheader=<int>
Description: Forces the use of the given line number (1 based) as the table's header. Does not include empty
lines in the count.
Default: The multikv command attempts to determine the header line automatically.
multitable
Syntax: multitable=<bool>
Description: Controls whether or not there can be multiple tables in a single _raw in the original events.
Default: true
noheader
Syntax: noheader=<bool>
Description: Handle a table without header row identification. The size of the table will be inferred from the
first row, and fields will be named Column_1, Column_2, ... noheader=true implies multitable=false.
Default: false
rmorig
Syntax: rmorig=<bool>
Description: When true, the original events will not be included in the output results. When false, the
original events are retained in the output results, with each original emitted after the batch of generated
results from that original.
Default: true
Examples
Example 1: Extract the "COMMAND" field when it occurs in rows that contain "splunkd".
... | multikv fields COMMAND filter splunkd
Example 2: Extract the "pid" and "command" fields.

... | multikv fields pid command
143
Hands-on Lab
1. Use the source LoanStats3a.csv and only take a look at some fields out of the data
2. Use the source LoanStats3a.csv and the table command on the same fields in #1
3. Use the source LoanStats3a.csv and use the rename command to rename fields in #1
4. Use the source LoanStats3a.csv and use the rex command for:
a. source="LoanStats3a.csv" annual_inc=60000 | rex "Does not meet the credit policy.(?<all_util>.*)"
b. and then click on all_util field to demonstrate the rex results
144
End of Module Quiz
145

Use the following commands and their functions:
Top
Rare
Stats
Add coltotals
Hands on Lab covering: Stats, Add Coltotals
146
Top command
Description
Displays the most common values of a field.
Finds the most frequent tuple of values of all fields in the field list, along with a count and percentage. If the optional by-clause is
included, the command finds the most frequent values for each distinct tuple of values of the group-by fields.
Syntax
top [<N>] [<top-options>...] <field-list> [<by-clause>]
Required arguments
<field-list>
Syntax: <field>, <field>, ...
Description: Comma-delimited list of field names.
Optional arguments
<N>
Syntax: <int>
Description: The number of results to return.
<top-options>
Syntax: countfield=<string> | limit=<int> | otherstr=<string> | percentfield=<string> | showcount=<bool>
| showperc=<bool> | useother=<bool>
Description: Options for the top command. See Top options.
<by-clause>
147
Syntax: BY <field-list>
Description: The name of one or more fields to group by.
Top options
countfield
Syntax: countfield=<string>
Description: The name of a new field that the value of count is written to.
Default: "count"
limit
Syntax: limit=<int>
Description: Specifies how many tuples to return, "0" returns all values.
Default: "10"
otherstr
Syntax: otherstr=<string>
Description: If useother is true, specify the value that is written into the row representing all other values.
Default: "OTHER"
percentfield
Syntax: percentfield=<string>
Description: Name of a new field to write the value of percentage.
Default: "percent"
showcount
Syntax: showcount=<bool>
Description: Specify whether to create a field called "count" (see "countfield" option) with the count of that
tuple.
Default: true
showperc
Syntax: showperc=<bool>
Description: Specify whether to create a field called "percent" (see "percentfield" option) with the relative
prevalence of that tuple.
Default: true
148
useother
Syntax: useother=<bool>
Description: Specify whether or not to add a row that represents all values not included due to the limit
cutoff.
Default: false
Examples
Example 1:
Return the 20 most common values of the "referer" field.
sourcetype=access_* | top limit=20 referer
Example 2:
Return top "action" values for each "referer_domain".
149
sourcetype=access_* | top action by referer_domain
Because a limit is not specified, this returns all the combinations of values for "action" and "referer_domain" as well as the counts
and percentages
150
Example 3:
Return the top product purchased for each category. Do not show the percent field. Rename the count field to "total".
sourcetype=access_* status=200 action=purchase | top 1 productName by categoryId showperc=f
countfield=total
151
Rare command
Description
Displays the least common values of a field.
Finds the least frequent tuple of values of all fields in the field list. If the <by-clause> is specified, this command returns rare tuples of
values for each distinct tuple of values of the group-by fields.
This command operates identically to the top command, except that the rare command finds the least frequent instead of the most
frequent.
Syntax
rare [<top-options>...] <field-list> [<by-clause>]
Required arguments
<field-list>
Syntax: <string>,...
Description: Comma-delimited list of field names.
Optional arguments
<top-options>
Syntax: countfield=<string> | limit=<int> | percentfield=<string> | showcount=<bool> | showperc=<bool>
Description: Options that specify the type and number of values to display. These are the same <topoptions> used by the top command.
<by-clause>
Description: The name of one or more fields to group by.
152
Top options
countfield
Syntax: countfield=<string>
Description: The name of a new field to write the value of count into.
Default: "count"
limit
Syntax: limit=<int>
Description: Specifies how many tuples to return. If you specify >code>limit=0</code>, all values up to
maxresultrows are returned. See Limits section. Specifying a value larger than maxresultrows produces an
error.
Default: 10
percentfield
Syntax: percentfield=<string>
Description: Name of a new field to write the value of percentage.
Default: "percent"
showcount
Syntax: showcount=<bool>
Description: Specify whether to create a field called "count" (see "countfield" option) with the count of that
tuple.
Default: true
showperc
Syntax: showperc=<bool>
Description: Specify whether to create a field called "percent" (see "percentfield" option) with the relative
prevalence of that tuple.
Default: true
153
Limits
There is a limit on the number of results which rare returns. By default this limit is 10, but other values can be selected with the limit
option up to a further constraint expressed in limits.conf, in the [rare] stanza, maxresultrows. This ceiling is 50,000 by default, and
effectively keeps a ceiling on the memory that rare will use.
Examples
Example 1:
Return the least common values of the "url" field.
... | rare url
Example 2:
Find the least common "user" value for a "host".
... | rare user by host
154
1. Run
source="C:\\LoanStats3a.csv"| top limit=20 addr_state
Now , show the rare addr_state

2. Run another search on your own demonstrating your use of the top and rare functions
155
Stats command
Description
Calculates aggregate statistics over the results set, such as average, count, and sum. This is similar to SQL aggregation. If stats is
used without a by clause only one row is returned, which is the aggregation over the entire incoming result set. If you use a by clause
one row is returned for each distinct value specified in the by clause.
Syntax
Simple: stats (stats-function(field) [AS field])... [BY field-list]
Complete: stats [partitions=<num>] [allnum=<bool>] [delim=<string>] ( <stats-agg-term>... | <sparkline-agg-term>... ) [<byclause>]
Required arguments
stats-agg-term
Syntax: <stats-function>(<evaled-field> | <wc-field>) [AS <wc-field>]
Description: A statistical aggregation function. The function can be applied to an eval expression, or to a
field or set of fields. Use the AS clause to place the result into a new field with a name that you specify. You
can use wild card characters in field names. For more information on eval expressions, see Types of eval
expressions in the Search Manual.
sparkline-agg-term
Syntax: <sparkline-agg> [AS <wc-field>]
Description: A sparkline aggregation function. Use the AS clause to place the result into a new field with a
name that you specify. You can use wild card characters in the field name.
Optional arguments
allnum
syntax: allnum=<bool>
Description: If true, computes numerical statistics on each field if and only if all of the values of that field
156
are numerical.
Default: false
delim
Syntax: delim=<string>
Description: Specifies how the values in the list() or values() aggregation are delimited.
Default: a single space
by-clause
Description: The name of one or more fields to group by. You cannot use a wildcard character to specify
multiple fields with similar names. You must specify each field separately.
partitions
Syntax: partitions=<num>
Description: If specified, partitions the input data based on the split-by fields for multithreaded reduce.
Default: 1
Stats function options
stats-function
Syntax: avg() | c() | count() | dc() | distinct_count() | earliest() | estdc() | estdc_error() | exactperc<int>() |
first() | last() | latest() | list() | max() | median() | min() | mode() | p<in>() | perc<int>() | range() | stdev()
| stdevp() | sum() | sumsq() | upperperc<int>() | values() | var() | varp()
Description: Functions used with the stats command. Each time you invoke the stats command, you can use
more than one function. However, you can only use one by clause.
Usage
The stats command does not support wildcard characters in field values in BY clauses.
For example, you cannot specify | stats count BY source*.
157
Basic Examples
1. Return the average transfer rate for each host
sourcetype=access* | stats avg(kbps) by host
2. Search the access logs, and return the total number of hits from the top 100 values of "referer_domain"
Search the access logs, and return the total number of hits from the top 100 values of "referer_domain". The "top" command returns a
count and percent value for each "referer_domain".
sourcetype=access_combined | top limit=100 referer_domain | stats sum(count) AS total
3. Calculate the average time for each hour for similar fields using wildcard characters
Return the average, for each hour, of any unique field that ends with the string "lay". For example, delay, xdelay, relay, etc.
... | stats avg(*lay) BY date_hour
4. Remove duplicates in the result set and return the total count for the unique results
Remove duplicates of results with the same "host" value and return the total count of the remaining results.
... | stats dc(host)
158
Addcoltotals command
Description
The addcoltotals command appends a new result to the end of the search result set. The result contains the sum of each numeric
field or you can specify which fields to summarize. Results are displayed on the Statistics tab. If the labelfield argument is
specified, a column is added to the statistical results table with the name specified.
Syntax
addcoltotals [labelfield=<field>] [label=<string>] [<fieldlist>]
Optional arguments
<fieldlist>
Syntax: <field> ...
Description: A space delimited list of valid field names. The addcoltotals command calculates the sum only
for the fields in the list you specify. You can use the asterisk ( * ) as a wildcard in the field names.
Default: Calculates the sum for all of the fields.
labelfield
Syntax: labelfield=<fieldname>
Description: Specify a field name to add to the result set.
Default: none
label
Syntax: label=<string>
Description: Used with the labelfield argument to add a label in the summary event. If the labelfield
argument is absent, the label argument has no effect.
Default: Total
159
Examples
Example 1:
Compute the sums of all the fields, and put the sums in a summary event called "change_name".
... | addcoltotals labelfield=change_name label=ALL
Example 2:
Add a column total for two specific fields in a table.
sourcetype=access_* | table userId bytes avgTime duration | addcoltotals bytes duration
Example 3:
Filter fields for two name-patterns, and get totals for one of them.
... | fields user*, *size | addcoltotals *size
Example 4:
Augment a chart with a total of the values present.
index=_internal source=*metrics.log" group=pipeline |stats avg(cpu_seconds) by processor |addcoltotals
labelfield=processor
160
Hands on Lab
1.
Run a search query that uses the top, stats functions with Loan file to get the count:
source="C:\\LoanStats3a.csv"| top limit=20 addr_state | stats count

2. source="C:\\LoanStats3a.csv" addr_state=CA | stats count
3. Try running:
4.
sourcetype=access_* | table userId bytes avgTime duration | addcoltotals bytes duration

Come up with your own example for the Loans file
161

Please refer to your virtual machine for test details
162

Hands on Lab covering: Explore the available visualizations, Create a basic chart, Split values into multiple series
Create a time chart
Hands on Lab covering: Omit null and other values from charts, Create a time chart, Chart multiple values on the same
timeline
Format charts
Hands on Lab covering: Format Charts, Explain when to use each type of reporting command.
163

Accessing Splunk's visualization definition features
Splunk provides user interface tools to create and modify visualizations. You can access these tools from various places in Splunk
Web.
Search
Dashboards
Dashboard visual editor
Pivot
Reports
Visualizations from Splunk Search

You can modify how Splunk displays search results in the Search page. After running a search, select the Visualization tab, then select
the type of visualization to display and specify formatting options for the selected visualization. The search must be a reporting search
that returns results that can be formatted as a visualization.
Dashboard panel visualizations
When you base a new dashboard panel on search results, you can choose the visualization that best represents the data returned by the
search. You can then use the Visualization Editor to fine-tune the way the panel visualization displays.
To create a dashboard panel from search results, after you run the search click Save As > Dashboard Panel.
164
Events visualizations
Events visualizations are essentially raw lists of events.
You can get events visualizations from any search that does not include a transform operation, such as a search that uses reporting
commands like stats, chart, timechart, top, or rare.
Tables
You can pick table visualizations from just about any search, but the most interesting tables are generated by searches that include
transform operations, such as a search that uses reporting commands like stats, chart, timechart, top, or rare.
Charts
Splunk provides a variety of chart visualizations, such as column, line, area, scatter, and pie charts. These visualizations require
transforming searches (searches that use reporting commands) whose results involve one or more series.
Maps
Splunk provides a map visualization that lets you plot geographic coordinates as interactive markers on a world map. Searches for
map visualizations should use the geostats search command to plot markers on a map. The geostats command is similar to the stats
command, but provides options for zoom levels and cells for mapping. Events generated from the geostats command include latitude
and longitude coordinates for markers.
165
Charts
Splunk provides a variety of chart visualizations, such as column, line, area, scatter, and pie charts. These visualizations require
transforming searches (searches that use reporting commands) whose results involve one or more series.
A series is a sequence of related data points that can be plotted on a chart. For example, each line plotted on a line chart represents an
individual series. You can design transforming searches that produce a single series, or you can set them up so the results provide data
for multiple series.
It may help to think of the tables that can be generated by transforming searches. Every column in the table after the first one
represents a different series. A "single series" search would produce a table with only two columns, while a "multiple series" search
would produce a table with three or more columns.
All of the chart visualizations can handle single-series searches, though you'll find that bar, column, line, and pie chart visualizations
are usually best for such searches. In fact, pie charts can only display data from single series searches.
On the other hand, if your search produces multiple series, you'll want to go with a bar, column, line, area, or scatter chart
visualization.
Column and bar charts
Use a column chart or bar chart to compare the frequency of values of fields in your data. In a column chart, the x-axis values are
typically field values (or time, especially if your search uses the timechart reporting command) and the y-axis can be any other field
value, count of values, or statistical calculation of a field value. Bar charts are exactly the same, except that the x-axis and y-axis
values are reversed.
166
The following bar chart presents the results of this search, which uses internal Splunk metrics. It finds the total sum of CPU_seconds
by processor in the last 15 minutes, and then arranges the processors with the top ten sums in descending order:
index=_internal "group=pipeline" | stats sum(cpu_seconds) as totalCPUSeconds by processor | sort 10
totalCPUSeconds desc
Note that in this example, we've also demonstrated how you can roll over a single bar or column to get detail information about it.
When you define the properties of your bar and column charts, you can:
set the chart titles, as well as the titles of the x-axis and y-axis.
set the minimum y-axis values for the y-axis (for example, if all the y-axis values of your search are above 100 it may improve
clarity to have the chart start at 100).
167
set the unit scale to Log (logarithmic) to improve clarity of charts where you have a mix of very small and very large y-axis
values.
determine whether charts are stacked, 100% stacked, and unstacked. Bar and column charts are always unstacked by default.
See the following subsection for details on stacking bar and column charts.
If you are formatting bar or column charts in dashboards with the Visualization Editor you can additionally:
set the major unit for the y-axis (for example, you can arrange to have tick marks appear in units of 10, or 20,
or 45...whatever works best).
determine the position of the chart legend and the manner in which the legend labels are truncated.
turn their drilldown functionality on or off.
Stacked column and bar charts

When your base search involves more than one data series, you can use stacked column charts and stacked bar charts to compare the
frequency of field values in your data.
In an unstacked column chart, the columns for different series are placed alongside each other. This may be fine if your chart is
relatively simple--total counts of sales by month for two or three items in a store over the course of a year, for example--but when the
series count increases it can make for a cluttered, confusing chart.
In a column chart set to a Stack mode of Stacked, all of the series columns for a single datapoint (such as a specific month in the chart
described in the preceding paragraph) are stacked to become segments of a single column (one column per month, to reference that
example again). The total value of the column is the sum of the segments.
Note: You use a stacked column or bar chart to highlight the relative weight (importance) of the different types of data that make up a
specific dataset.
168
The following chart illustrates the customer views of pages in the website of MyFlowerShop, a hypothetical web-based flower store,
broken out by product category over a 7 day period:
Here's the search that built that stacked chart:

169
sourcetype=access_* method=GET | timechart count by categoryId | fields _time BOUQUETS FLOWERS GIFTS
SURPRISE TEDDY
Note the usage of the fields command; it ensures that the chart only displays counts of events with a product category ID; events
without one (categorized as null by Splunk) are excluded.
The third Stack mode option, Stacked 100%, enables you to compare data distributions within a column or bar by making it fit to
100% of the length or width of the chart and then presenting its segments in terms of their proportion of the total "100%" of the
column or bar. Stacked 100% can help you to better see data distributions between segments in a column or bar chart that contains a
mix of very small and very large stacks when Stack mode is just set to Stacked.
Line and area charts
Line and area charts are commonly used to show data trends over time, though the x-axis can be set to any field value. If your chart
includes more than one series, each series will be represented by a differently colored line or area.
This chart is based on a simple search that reports on internal Splunk metrics:
index=_internal | timechart count by sourcetype
170
The shaded areas in area charts can help to emphasize quantities. The following area chart is derived from this search, which also
makes use of internal Splunk metrics:
index=_internal source=*metrics.log group=search_concurrency "system total" NOT user=* | timechart
max(active_hist_searches) as "Historical Searches" max(active_realtime_searches) as "Real-time Searches"
171
When you define the properties of your line and area charts, you can:
determine what Splunk does with missing (null) y-axis values. You can have the system leave gaps for null datapoints, have
connect to zero datapoints, or just connect to the next positive datapoint. If you choose to leave gaps, Splunk will display
markers for datapoints that are disconnected because they are not adjacent to other positive datapoints.
set the minimum y-axis values (for example, if all the y-axis values of your search are above 100 it may improve clarity to
have the chart start at 100).
set the unit scale to Log (logarithmic) to improve clarity of charts where you have a mix of very small and very large y-axis
values.
determine whether charts are stacked, 100% stacked, and unstacked. Bar and column charts are always unstacked by default.
See the following subsection for details on stacking bar and column charts.
172
If you are formatting line or area charts in dashboards with the Visualization Editor you can additionally:
Stacked line and area charts

Stacked line and area charts operate along the same principles of stacked column and row charts (see above). Stacked line and area
charts can help readers when several series are involved; it makes it easier to see how each data series relates to the entire set of data
as a whole.
The following chart is another example of a chart that presents information from internal Splunk metrics. The search used to create it
is:
index=_internal per_sourcetype_thruput | timechart sum(kb) by series useother=f
173
Pie chart
Use a pie chart to show the relationship of parts of your data to the entire set of data as a whole. The size of a slice in a pie graph is
determined by the size of a value of part of your data as a percentage of the total of all values.
The following pie chart presents the views by referrer domain for a hypothetical online store for the previous day. Note that you can
get metrics for individual pie chart wedges by mousing over them.
174
When you define the properties of pie charts you can set the chart title. If you are formatting pie charts in dashboards with the
Visualization Editor you can additionally:
determine the position of the chart legend.

turn pie chart drilldown functionality on or off.
175
Scatter chart
Use a scatter chart ( or "scatter plot") to show trends in the relationships between discrete values of your data. Generally, a scatter plot
shows discrete values that do not occur at regular intervals or belong to a series. This is different from a line graph, which usually
plots a regular series of points.
Here's an example of a search that can be used to generate a scatter chart. It looks at USGS earthquake data (in this case a CSV file
that presents all magnitude 2.5+ quakes recorded over a given 7-day period, worldwide), pulls out just the Californian quakes, plots
out the quakes by magnitude and quake depth, and then color-codes them by region. As you can see the majority of quakes recorded
during this period were fairly shallow--10 or fewer meters in depth, with the exception of one quake that was around 27 meters deep.
None of the quakes exceeded a magnitude of 4.0.
176
To generate the chart for this example, we've used the table command, followed by three fields. The first field is what appears in the
legend (Region). The second field is the x-axis value (Magnitude), which leaves the third field (Depth) to be the y-axis value. Note
that when you use table the latter two fields must be numeric in nature.
source=usgs Region=*California | table Region Magnitude Depth | sort Region
You can download a current CSV file from the USGS Earthquake Feeds and add it as an input to Splunk, but the field names and
format will be slightly different from the example shown here.
When you define the properties of your scatter charts, you can:
set the minimum y-axis values for the y-axis (for example, if all the y-axis values of your search are above 100
it may improve clarity to have the chart start at 100).
set the unit scale to Log (logarithmic) to improve clarity of charts where you have a mix of very small and very
large y-axis values.
If you are formatting bar or column charts in dashboards with the Visualization Editor you can additionally:
177

Run for example:
sourcetype=access_* | timechart count(eval(method="GET")) AS GET, count(eval(method="POST")) AS POST
Then click the visualization tab to see the result of this having two series. Make sure to select Line Chart
178
Hands on Lab :
1. Upload a data file called: ImplementingSplunkDataGenerator.tgz located on the desktop
Run:
source="ImplementingSplunkDataGenerator.tgz:*" host="WIN-SQM8ERRKEIJ"| chart count over date_month
by date_wday
If you look back at the results from stats, the data is presented as one row per combination. Instead of a row per combination, chart
generates the intersection of the two fields. You can specify multiple functions, but you may only specify one field each for over and
by.
Switching the fields (by rearranging our search statement a bit) turns the data the other way.
By simply clicking on the Visualization tab (to the right of the Statistics tab), we can see these results in a chart:
179
This is an Area chart, with particular format options set. Within the chart area, you can click on Area to change the chart type (Line,
Area, Column, Bar, and so on) or Format to change the format options (Stack, Null Values, Multi-series Mode, and Drilldown).
Bonus Lab:
Create a chart from the Loan file csv on your desktop
180

Sometimes Splunk has extra null fields floating around.
If your records have a unique Id field, then the following snippet removes null fields:
| stats values(*) as * by Id
The reason is that "stats values won't show fields that don't have at least one non-null value".
If your records don't have a unique Id field, then you should create one first using streamstats:
| streamstats count as Id | stats values(*) as * by Id
181
Create a time chart

The timechart command
The timechart command generates a table of summary statistics which can then be formatted as a chart visualization where your data
is plotted against an x-axis that is always a time field. Use timechart to display statistical trends over time, with the option of splitting
the data with another field as a separate series in the chart. Timechart visualizations are usually line, area, or column charts.
Examples
Example 1: This report uses internal Splunk log data to visualize the average indexing thruput (indexing kbps) of Splunk processes
over time, broken out by processor:
index=_internal "group=thruput" | timechart avg(instantaneous_eps) by processor
182

Refer to lesson above for multiple series:
Run for example:
Then click the visualization tab to see the result of this having two series. Make sure to select Line Chart
183
Hands on Lab
Run
Create another example of a timechart with the Loan csv file
184
Format charts
Let's go ahead and take a look at the (chart) Format options. These options are grouped as:
General: Under general, you have the option to set the Stack Model (which indicates how Splunk will display your chart
columns for different series (alongside each other or as a single column), determine how to handle Null Values (you can leave
gaps for null data points, connect to zero data points, or just connect to the next positive data point), set the Multi-series mode
(Yes or No), and turn Drilldown (active or inactive) on or off.
X-Axis: Is mostly visual, you can set a custom title, allow truncation of label captions, and set the rotation of the text for your
chart labels.
Y-Axis: Here you can set not just a custom title, but also the scale (linear or log), the interval, and the min and max values.
Chart Overlay: Here you can set the following options:
Overlay: Select a field to show as an overlay.
View as Axis: Select On to map the overlay to a second Y-axis.
Title: Specify a title for the overlay.
Scale: Select Inherit, Linear, or Log. Inherit uses the scale for the base chart. Log provides a logarithmic scale,
useful for minimizing the display of large peak values.
Interval: Enter the units between tick marks in the axis.
Min Value: The minimum value to display. Values less than the Min Value do not appear on the chart.
Max Value: The maximum value to display. Values greater than the Max Value do not appear on the chart.
Legend: Finally, under Legend, you can set Position (where to place the legend (or to not include the legend) in the
visualization.) and Truncation (set how to represent names that are too long to display). Keep in mind that, depending on your
search results and the visualization options that you select, you may or may not get a useable result. Some experimentation
with the various options is recommended.
185
A reporting command primer

This subsection covers the major categories of reporting commands and provides examples of how they can be used in a search.
The primary reporting commands are:
chart:
used to create charts that can display any series of data that you want to plot. You can decide what
field is tracked on the x-axis of the chart.
timechart: used to create "trend over time" reports, which means that _time is always the x-axis.
top: generates charts that display the most common values of a field.
rare: creates charts that display the least common values of a field.
stats, eventstats, and streamstats: generate reports that display summary statistics.
associate, correlate, and diff: create reports that enable you to see associations, correlations, and differences
between fields in your data.
186
Note: As you'll see in the following examples, you always place your reporting commands after your search commands, linking them
with a pipe operator ("|").
chart, timechart, stats, eventstats,
and streamstats are all designed to work in conjunction with statistical functions. The list
of available statistical functions includes:
count, distinct count

mean, median, mode
min, max, range, percentiles
standard deviation, variance
sum
first occurrence, last occurrence
187
Hands on Lab
Please format your chart from the last lab exercise
188
189
Module 10 - Analyzing, Calculating, and Formatting Results
Using the eval command

Perform calculations
Convert values
Hands on Lab covering: Using the eval command, Perform calculations, Convert values.
Round values
Format values
Hands on Lab covering: Round values, Format values
Use conditional statements
Further filter calculated results
Hands on Lab covering: Use conditional statements, Further filter calculated results
190
Using the eval command and perform calculations

Use the eval command and functions
The eval command enables you to devise arbitrary expressions that use automatically extracted fields to create a new field that takes
the value that is the result of the expression's evaluation. The eval command is immensely versatile and useful. But while some eval
expressions are relatively simple, they often can be quite complex.
Types of eval expressions

An eval expression is a combination of literals, fields, operators, and functions that represent the value of your destination field. The
expression can involve a mathematical operation, a string concatenation, a comparison expression, a boolean expression, or a call to
one of the eval functions. Eval expressions require that the field's values are valid for the type of operation.
For example, with the exception of addition, arithmetic operations may not produce valid results if the values are not numerical. For
addition, eval can concatenate the two operands if they are both strings. When concatenating values with '.', eval treats both values as
strings, regardless of their actual type.
Example 1: Use eval to define a field that is the sum of the areas of two circles, A and B.
... | eval sum_of_areas = pi() * pow(radius_a, 2) + pi() * pow(radius_b, 2)
The area of circle is r^2, where r is the radius. For circles A and B, the radii are radius_a and radius_b, respectively. This eval
expression uses the pi and pow functions to calculate the area of each circle and then adds them together, and saves the result in a field
named, sum_of_areas.
191
Example 2: Use eval to define a location field using the city and state fields. For example, if the city=Philadelphia and state=PA,
location="Philadelphia, PA".
... | eval location=city.", ".state
This eval expression is a simple string concatenation.
192
Convert values
The convert command converts field values into numerical values. Unless you use the AS clause, the original values
are replaced by the new values.
Example 1
This example uses sendmail email server logs and refers to the logs with sourcetype=sendmail. The sendmail logs
have two duration fields, delay and xdelay.
The delay is the total amount of time a message took to deliver or bounce. The delay is expressed as "D+HH:MM:SS", which
indicates the time it took in hours (HH), minutes (MM), and seconds (SS) to handle delivery or rejection of the message. If the
delay exceeds 24 hours, the time expression is prefixed with the number of days and a plus character (D+).
The xdelay is the total amount of time the message took to be transmitted during final delivery, and its time is expressed as
"HH:MM:SS".
Change the sendmail duration format of delay and xdelay to seconds.
sourcetype=sendmail | convert dur2sec(delay) dur2sec(xdelay)
This search pipes all the sendmail events into the convert command and uses the dur2sec() function to convert the duration times of
the fields, delay and xdelay, into seconds.
193
Here is how your search results look after you use the fields sidebar to add the fields to your events:
You can compare the converted field values to the original field values in the events list.
Example 2
This example uses syslog data.
Convert a UNIX epoch time to a more readable time formatted to show hours, minutes, and seconds.
sourcetype=syslog | convert timeformat="%H:%M:%S" ctime(_time) AS c_time | table _time, c_time
The ctime() function converts the _time value of syslog (sourcetype=syslog) events to the format specified by the timeformat
argument. The timeformat="%H:%M:%S" arguments tells the search to format the _time value as HH:MM:SS.
Here, the table command is used to show the original _time value and the converted time, which is renamed c_time:
194
The ctime() function changes the timestamp to a non-numerical value. This is useful for display in a report or for readability in your
events list.
Example 3
This example uses syslog data.
Convert a time in MM:SS.SSS (minutes, seconds, and subseconds) to a number in seconds.
sourcetype=syslog | convert mstime(_time) AS ms_time | table _time, ms_time
The mstime() function converts the _time value of syslog (sourcetype=syslog) events from a minutes and seconds to just seconds.
195
Here, the table command is used to show the original _time value and the converted time, which is renamed ms_time:
The mstime() function changes the timestamp to a numerical value. This is useful if you want to use it for more calculations.
More examples
Example 1: Convert values of the "duration" field into number value by removing string values in the field value. For example, if
"duration="212 sec"", the resulting value is "duration="212"".
... | convert rmunit(duration)
Example 2: Change the sendmail syslog duration format (D+HH:MM:SS) to seconds. For example, if "delay="00:10:15"", the
resulting value is "delay="615"".
196
... | convert dur2sec(delay)
Example 3: Change all memory values in the "virt" field to Kilobytes.

... | convert memk(virt)
Example 4: Convert every field value to a number value except for values in the field "foo" Use the "none" argument to specify fields
to ignore.
... | convert auto(*) none(foo)
197
Hands on Lab
1.
Run:
source="ImplementingSplunkDataGenerator.tgz:*" host="WIN-SQM8ERRKEIJ" error

| stats count by logger user
| eventstats sum(count) as totalcount
| eval percent=count/totalcount*100
| sort -count
And explain the options
2.
Take the Loan csv file and develop some eval functions
198
Round and format values functions

Usage
All functions that accept strings can accept literal strings or any field.
All functions that accept numbers can accept literal numbers or any numeric field.
Comparison and Conditional functions

Function
case(X,"Y",...)
cidrmatch("X",Y)
Description
This function takes pairs of
arguments X and Y. The X
arguments are Boolean
expressions that will be
evaluated from first to last.
When the first X expression is
encountered that evaluates to
TRUE, the corresponding Y
argument will be returned. The
function defaults to NULL if
none are true.
This function returns true,
when IP address Y belongs to a
particular subnet X. The
function uses two string
arguments: the first is the CIDR
subnet; the second is the IP
address to match.
Example(s)
This example returns descriptions for the corresponding

http status code:
... | eval description=case(error == 404, "Not found",
error == 500, "Internal Server Error", error == 200,
"OK")
This example uses cidrmatch to set a field, isLocal, to

"local" if the field ip matches the subnet, or "not local" if it
does not:
... | eval isLocal=if(cidrmatch("123.132.32.0/25",ip),
"local", "not local")
199
This example uses cidrmatch as a filter:

... | where cidrmatch("123.132.32.0/25", ip)
coalesce(X,...)
This function takes an arbitrary

number of arguments and
returns the first value that is
not null.
Let's say you have a set of events where the IP address is

extracted to either clientip or ipaddress. This example
defines a new field called ip, that takes the value of either
clientip or ipaddress, depending on which is not NULL
(exists in that event):
... | eval ip=coalesce(clientip,ipaddress)
if(X,Y,Z)
This function takes three

This example looks at the values of error and returns
arguments. The first argument err=OK if error=200, otherwise returns err=Error:
X must be a Boolean expression.
... | eval err=if(error == 200, "OK", "Error")
If X evaluates to TRUE, the
result is the second argument Y.
If, X evaluates to FALSE, the
result evaluates to the third
argument Z.
like(TEXT,
PATTERN)
This function takes two

arguments, a string to match
TEXT and a match expression
string PATTERN. It returns
TRUE if and only if the first
argument is like the SQLite
pattern in Y. The pattern
language supports exact text
match, as well as % characters
This example returns islike=TRUE if the field value starts

with foo:
... | eval is_a_foo=if(like(field, "foo%"), "yes a
foo", "not a foo")
or
... | where like(field, "foo%")
200
for wildcards and _ characters

for a single character match.
match(SUBJECT,
"REGEX")
This function compares the

regex string REGEX to the
value of SUBJECT and returns
a Boolean value. It returns true
if the REGEX can find a match
against any substring of
SUBJECT.
This example returns true IF AND ONLY IF field matches

the basic pattern of an IP address. Note that the example
uses ^ and $ to perform a full match.
... | eval n=if(match(field,
"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"), 1, 0)
null()
This function takes no

arguments and returns NULL.
The evaluation engine uses
NULL to represent "no value";
setting a field to NULL clears
its value.
nullif(X,Y)
This function is used to compare

fields. The function takes two
... | eval n=nullif(fieldA,fieldB)
arguments, X and Y, and
returns NULL if X = Y.
Otherwise it returns X.
searchmatch(X)
This function takes one

argument X, which is a search
... | eval n=searchmatch("foo AND bar")
string. The function returns
true IF AND ONLY IF the
event matches the search string.
validate(X,Y,...)
This function takes pairs of
This example runs a simple check for valid ports:

201
arguments, Boolean expressions

... | eval n=validate(isint(port), "ERROR: Port is not
X and strings Y. The function
an integer", port >= 1 AND port <= 65535, "ERROR: Port
returns the string Y
is out of range")
corresponding to the first
expression X that evaluates to
False and defaults to NULL if
all are True.
Conversion functions
Function
tonumber(NUMSTR,BASE)
tonumber(NUMSTR)
tostring(X,Y)
Description
Examples
This function converts the input This example returns "164":

string NUMSTR to a number,
where BASE is optional and used ... | eval n=tonumber("0A4",16)
to define the base of the number
to convert to. BASE can be 2 to
36, and defaults to 10. If
tonumber cannot parse a field
value to a number, for example if
the value contains a leading and
trailing space, the function
returns NULL. Use the trim
function to remove leading or
trailing spaces. If tonumber
cannot parse a literal string to a
number, it returns an error.
This function converts the input
value to a string. If the input
value is a number, it reformats it
202
This example returns "True 0xF 12,345.68":
as a string. If the input value is a

... | eval n=tostring(1==1) + " " + tostring(15,
Boolean value, it returns the
"hex") + " " + tostring(12345.6789, "commas")
corresponding string value,
"True" or "False".
This example returns foo=615 and foo2=00:10:15:
... | eval foo=615 | eval foo2 = tostring(foo,
This function requires at least one

"duration")
argument X; if X is a number, the
second argument Y is optional and can This example formats the column totalSales to display values
be "hex" "commas" or "duration":
with a currency symbol and commas. You must use a period
between the currency value and the tostring function.
tostring(X,"hex") converts X to
...| fieldformat
hexadecimal.
totalSales="$".tostring(totalSales,"commas")
tostring(X,"commas")
formats X
with commas and, if the number
includes decimals, rounds to nearest
two decimal places.
tostring(X,"duration")
converts
seconds X to readable time format
HH:MM:SS.
Note: When used with the eval command, the values might
not sort as expected because the values are converted to ASCII.
Use the fieldformat command with the tostring function to
format the displayed values. The underlying values are not
changed with the fieldformat command.
Cryptographic functions
Function
Description
Example(s)
md5(X)
This function computes and returns the MD5 hash of a string value X.
... | eval
n=md5(field)
sha1(X)
This function computes and returns the secure hash of a string value X based
on the FIPS compliant SHA-1 hash function.
... | eval
n=sha1(field)
203
sha256(X)
... | eval
n=sha256(field)
sha512(X)
... | eval
n=sha512(field)
Date and Time functions

Function
Description
Example(s)
now()
This function takes no arguments and returns the time

that the search was started. The time is represented in
Unix time or in seconds since Epoch time.
relative_time(X,Y)
This function takes an epochtime time, X, as the first

argument and a relative time specifier, Y, as the second
argument and returns the epochtime value of Y applied
to X.
... | eval
n=relative_time(now(), "-1d@d")
strftime(X,Y)
This function takes an epochtime value, X, as the first

argument and renders it as a string using the format
specified by Y.
This example returns the hour

and minute from the _time field:
... | eval n=strftime(_time,
"%H:%M")
strptime(X,Y)
This function takes a time represented by a string, X, and If timeStr is in the form, "11:59",
parses it into a timestamp using the format specified by
this returns it as a timestamp:
Y.
... | eval n=strptime(timeStr,
"%H:%M")
time()
This function returns the wall-clock time with

microsecond resolution. The value of time() will be
204
different for each event based on when that event was

processed by the eval command.
Informational functions
Function
isbool(X)
Description
Example(s)
This function takes one argument X and returns

TRUE if X is Boolean.
... | eval n=if(isbool(field),"yes","no")
or
... | where isbool(field)
isint(X)

TRUE if X is an integer.
... | eval n=if(isint(field), "int", "not

int")
or
... | where isint(field)
isnotnull(X)

TRUE if X is not NULL. This is a useful check for
whether or not a field (X) contains a value.
... | eval n=if(isnotnull(field),"yes","no")
or
... | where isnotnull(field)
isnull(X)

TRUE if X is NULL.
... | eval n=if(isnull(field),"yes","no")
or
... | where isnull(field)
isnum(X)

TRUE if X is a number.
205
... | eval n=if(isnum(field),"yes","no")
or
... | where isnum(field)
isstr(X)

TRUE if X is a string.
... | eval n=if(isstr(field),"yes","no")
or
... | where isstr(field)
typeof(X)
This function takes one argument and returns a

string representation of its type.
This example returns

"NumberStringBoolInvalid":
... | eval n=typeof(12) + typeof("string") +
typeof(1==2) + typeof(badfield)
Mathematical functions
Function
abs(X)
Description
Examples
This function takes a number X and returns its

absolute value.
This example returns the absnum, whose

values are the absolute values of the
numeric field number:
... | eval absnum=abs(number)
ceil(X),
ceiling(X)
This function rounds a number X up to the next

highest integer.
exact(X)
This function renders the result of a numeric eval

calculation with a larger amount of precision in the
206
This example returns n=2:

... | eval n=ceil(1.9)
... | eval n=exact(3.14 * num)
formatted output.
exp(X)
floor(X)
ln(X)
This function takes a number X and returns the

exponential function eX.
The following example returns y=e3:
This function rounds a number X down to the nearest

whole integer.
This example returns 1:
This function takes a number X and returns its

natural log.
... | eval y=exp(3)
... | eval n=floor(1.9)
This example returns the natural log of the

values of bytes:
... | eval lnBytes=ln(bytes)
log(X,Y)
log(X)
This function takes either one or two numeric

arguments and returns the log of the first argument X
... | eval num=log(number,2)
using the second argument Y as the base. If the
second argument Y is omitted, this function evaluates
the log of number X with base 10.
pi()
This function takes no arguments and returns the

constant pi to 11 digits of precision.
... | eval area_circle=pi()*pow(radius,2)
pow(X,Y)
This function takes two numeric arguments X and Y

and returns XY.
... | eval area_circle=pi()*pow(radius,2)
round(X,Y)
This function takes one or two numeric arguments X

and Y, returning X rounded to the amount of decimal
places specified by Y. The default is to round to an
integer.
207
This example returns n=4:

... | eval n=round(3.5)
This example returns n=2.56:

... | eval n=round(2.555, 2)
This function takes one argument X, a number, and

rounds that number to the appropriate number of
significant figures.
sigfig(X)
1.00*1111 = 1111,
but
... | eval n=sigfig(1.00*1111)
returns n=1110.
This function takes one numeric argument X and
returns its square root.
sqrt(X)
This example returns 3:

... | eval n=sqrt(9)
Multivalue functions
Function
commands(X)
Description
This function takes a search string, or
field that contains a search string, X and
returns a multivalued field containing a
list of the commands used in X. (This is
generally not recommended for use
except for analysis of audit.log events.)
Example(s)
... | eval x=commands("search foo |
stats count | sort count")
returns a multivalued field X, that contains

'search', 'stats', and 'sort'.
mvappend(X,...)
This function takes an arbitrary number

of arguments and returns a multivalue
result of all the values. The arguments
can be strings, multivalue fields or
single value fields.
... | eval
fullName=mvappend(initial_values,
"middle value", last_values)
mvcount(MVFIELD)
This function takes a field MVFIELD.

The function returns the number of
... | eval n=mvcount(multifield)
208
values if it is a multivalue, 1 if it is a
single value field, and NULL otherwise.
mvdedup(X)
mvfilter(X)
This function takes a multivalue field X

and returns a multivalue field with its
duplicate values removed.
... | eval s=mvdedup(mvfield)
This function filters a multivalue field

based on an arbitrary Boolean
expression X. The Boolean expression X This example returns all of the values
can reference ONLY ONE field at a time. in field email that end in .net or .org:
Note:This function will return NULL values of ... | eval n=mvfilter(match(email,
the field x as well. If you don't want the NULL "\.net$") OR match(email, "\.org$"))
values, use the expression:
mvfilter(x!=NULL).
mvfind(MVFIELD,"REGEX")
mvindex(MVFIELD,STARTINDEX,
ENDINDEX)
mvindex(MVFIELD,STARTINDEX)
This function tries to find a value in

multivalue field X that matches the
regular expression REGEX. If a match
exists, the index of the first matching
value is returned (beginning with zero).
If no values match, NULL is returned.
This function takes two or three
arguments, field MVFIELD and
numbers STARTINDEX and
ENDINDEX, and returns a subset of the
multivalue field using the indexes
provided.
209
... | eval n=mvfind(mymvfield,

"err\d+")
Since indexes start at zero, this

example returns the third value in
"multifield", if it exists:
... | eval n=mvindex(multifield, 2)
For mvindex(mvfield, startindex,

[endindex]), endindex is inclusive and
optional. Both startindex and endindex can be
negative, where -1 is the last element. If
endindex is not specified, it returns only the
value at startindex. If the indexes are out of
range or invalid, the result is NULL.
mvjoin(MVFIELD,STR)
mvrange(X,Y,Z)
mvsort(X)
This function takes two arguments,

multivalue field MVFIELD and string
delimiter STR. The function
concatenates the individual values of
MVFIELD with copies of STR in
between as separators.
This function creates a multivalue field
for a range of numbers. This function
can contain up to three arguments: a
starting number X, an ending number Y
(exclusive), and an optional step
increment Z. If the increment is a
timespan such as '7'd, the starting and
ending numbers are treated as epoch
times.
This function uses a multivalue field X
and returns a multivalue field with the
values sorted lexicographically.
210
This example joins together the

individual values of "foo" using a
semicolon as the delimiter:
... | eval n=mvjoin(foo, ";")
This example returns a multivalue field

with the values 1, 3, 5, 7, 9.
... | eval mv=mvrange(1,11,2)
... | eval s=mvsort(mvfield)
mvzip(X,Y,"Z")
This function takes two multivalue

fields, X and Y, and combines them by
stitching together the first value of X
with the first value of field Y, then the
second with the second, and so on. The
third argument, Z, is optional and is
used to specify a delimiting character to
join the two values. The default
delimiter is a comma. This is similar to
Python's zip command.
... | eval nserver=mvzip(hosts,ports)
Statistical functions
In addition to these functions, a comprehensive set of statistical functions is available to use with the stats, chart, and related
commands.
Function
Description
Example(s)
This function takes an arbitrary number of numeric or string

max(X,...) arguments, and returns the max; strings are greater than
numbers.
This function takes an arbitrary number of numeric or string

min(X,...) arguments, and returns the min; strings are greater than
numbers.
random()
This function takes no arguments and returns a pseudo211
This example returns either "foo" or

field, depending on the value of field:
... | eval n=max(1, 3, 6, 7, "foo",
field)
This example returns either 1 or

field, depending on the value of field:
... | eval n=min(1, 3, 6, 7, "foo",
field)
random integer ranging from zero to 231-1, for example:

02147483647
Text functions
Function
Description
Examples
len(X)
This function returns the character

length of a string X.
lower(X)
This function takes one string argument

and returns the lowercase version. The
upper() function also exists for returning
the uppercase version.
ltrim(X,Y)
ltrim(X)
... | eval n=len(field)
rtrim(X)
... | eval username=lower(username)
This function takes one or two

arguments X and Y and returns X with
This example returns x="abcZZ":
the characters in Y trimmed from the left
side. If Y is not specified, spaces and tabs ... | eval x=ltrim(" ZZZZabcZZ ", " Z")
are removed.
This function returns a string formed by

substituting string Z for every occurrence
replace(X,Y,Z) of regex string Y in string X. The third
argument Z can also reference groups
that are matched in the regex.
rtrim(X,Y)
This example returns the value provided by the field

username in lowercase.

the characters in Y trimmed from the
right side. If Y is not specified, spaces
212
This example returns date with the month and day

numbers switched, so if the input was 1/14/2015 the
return value would be 14/1/2015:
... | eval n=replace(date,
"^(\d{1,2})/(\d{1,2})/", "\2/\1/")
This example returns n="ZZZZabc":

... | eval n=rtrim(" ZZZZabcZZ ", " Z")
and tabs are removed.
spath(X,Y)
split(X,"Y")
substr(X,Y,Z)
This function takes two arguments: an

input source field X and an spath
expression Y, that is the XML or JSON
formatted location path to the value that
you want to extract from X. If Y is a
literal string, it needs quotes,
spath(X,"Y"). If Y is a field name (with
values that are the location paths), it
doesn't need quotes. This may result in a
multivalued field. Read more about the
spath search command.
This function takes two arguments, field
X and delimiting character Y. It splits
the value(s) of X on the delimiter Y and
returns X as a multivalue field.
This example returns the values of locDesc elements:

... | eval locDesc=spath(_raw,
"vendorProductSet.product.desc.locDesc")
This example returns the hashtags from a twitter event:

index=twitter | eval output=spath(_raw,
"entities.hashtags")
... | eval n=split(foo, ";")
This function takes either two or three

arguments, where X is a string and Y
and Z are numeric. It returns a substring
of X, starting at the index specified by Y This example concatenates "str" and "ing" together,
with the number of characters specified
returning "string":
by Z. If Z is not given, it returns the rest
... | eval n=substr("string", 1, 3) +
of the string.
substr("string", -3)
The indexes follow SQLite semantics; they

start at 1. Negative indexes can be used to
indicate a start from the end of the string.
213

the characters in Y trimmed from both
sides. If Y is not specified, spaces and
tabs are removed.
trim(X,Y)
trim(X)
This function takes one string argument

and returns the uppercase version. The
lower() function also exists for returning
the lowercase version.
upper(X)
urldecode(X)
This function takes one URL string

argument X and returns the unescaped
or decoded URL string.
This example returns "abc":

... | eval n=trim(" ZZZZabcZZ ", " Z")
This example returns the value provided by the field

username in uppercase.
... | eval n=upper(username)
This example returns

"http://www.splunk.com/download?r=header":
... | eval
n=urldecode("http%3A%2F%2Fwww.splunk.com
%2Fdownload%3Fr%3Dheader")
Trigonometry and Hyperbolic functions

Function
Description
Examples
acos(X)
This function computes the arc cosine of X, in the interval [0,pi]

radians.
acosh(X)
This function computes the arc hyperbolic cosine of X, in radians.
asin(X)
This function computes the arc sine of X, in the interval [-pi/2,+pi/2]

radians.
214
... | eval n=acos(0)

... | eval
degrees=acos(0)*180/pi()
... | eval n=acosh(2)
... | eval n=asin(1)
... | eval
degrees=asin(1)*180/pi()
asinh(X)
This function computes the arc hyperbolic sine of X, in radians.
... | eval n=asinh(1)
atan(X)
This function computes the arc tangent of X, in the interval [pi/2,+pi/2] radians.
... | eval n=atan(0.50)
atan2(Y,
X)
This function computes the arc tangent of Y, X in the interval [-pi,+pi]

radians. Y is a value that represents the proportion of the ycoordinate. X is the value that represents the proportion of the xcoordinate.
.. | eval n=atan2(0.50,
0.75)
To compute the value, the function takes into account the sign of both arguments
to determine the quadrant.
atanh(X)
This function computes the arc hyperbolic tangent of X, in radians.
... | eval n=atanh(0.500)

... | eval n=cos(-1)
cos(X)
This function computes the cosine of an angle of X radians.

... | eval n=cos(pi())
cosh(X)
This function computes the hyperbolic cosine of X radians.
... | eval n=cosh(1)
This function computes the hypotenuse of a right-angled triangle

whose legs are X and Y.
hypot(X,Y)
... | eval n=hypot(3,4)
The function returns the square root of the sum of the squares of X and Y, as
described in the Pythagorean theorem.
... | eval n=sin(1)
sin(X)
This function computes the sine.
sinh(X)
This function computes the hyperbolic sine.
... | eval n=sin(90 *

pi()/180)
215
... | eval n=sinh(1)
tan(X)
This function computes the tangent.
... | eval n=tan(1)
tanh(X)
This function computes the hyperbolic tangent.
... | eval n=tanh(1)
216
Hands-on Lab
Please take a look at Loan csv file. Use that file and some of the functions in the table in the manual
Take a look at round and some other functions that are very popular
217
End of Module Quiz

Please refer to virtual machine for quiz
218
Module 11 - Creating Field Aliases and Calculated Fields

Hands on Lab covering: Define naming conventions, Create and use field aliases, Create and use calculated fields.
219

Example - Set up a naming convention for reports
You work in the systems engineering group of your company, and as the knowledge manager for your Splunk Enterprise
implementation, it's up to you to come up with a naming convention for the reports produced by your team.
In the end you develop a naming convention that pulls together:
Group: Corresponds to the working group(s) of the user saving the search.
Search type: Indicates the type of search (alert, report, summary-index-populating)
Platform: Corresponds to the platform subjected to the search
Category: Corresponds to the concern areas for the prevailing platforms.
Time interval: The interval over which the search runs (or on which the search runs, if it is a scheduled
search).
Description: A meaningful description of the context and intent of the search, limited to one or two words if
possible. Ensures the search name is unique.
Group Search type Platform

SEG
NEG
OPS
NOC
Alert
Report
Summary
Windows
iSeries
Network
Category
Time interval Description
Disk
<arbitrary>
Exchange
SQL
Event log
CPU
Jobs
Subsystems
220
<arbitrary>
Services
Security
Possible reports using this naming convention:
SEG_Alert_Windows_Eventlog_15m_Failures
SEG_Report_iSeries_Jobs_12hr_Failed_Batch
NOC_Summary_Network_Security_24hr_Top_src_ip
221

You can create multiple aliases for a field. The original field is not removed. This process enables you to search for the original field
using any of its aliases.
Important: Field aliasing is performed after key/value extraction but before field lookups. Therefore, you can specify a lookup table
based on a field alias. This can be helpful if there are one or more fields in the lookup table that are identical to fields in your data, but
have been named differentlyYou can define aliases for fields that are extracted at index time as well as those that are extracted at
search time.
You add your field aliases to props.conf, which you edit in $SPLUNK_HOME/etc/system/local/, or your own custom app directory
in $SPLUNK_HOME/etc/apps/. (We recommend using the latter directory if you want to make it easy to transfer your data
customizations to other index servers.)
Note: Splunk Enterprise's field aliasing functionality does not currently support multivalue fields.
To alias fields:
1. Add the following line to a stanza in props.conf:
FIELDALIAS-<class> = <orig_field_name> AS <new_field_name>
<orig_field_name> is the original name of the field.

<new_field_name> is the alias to assign to the field.
You can include multiple field alias renames in one stanza.
222
2. Restart Splunk Enterprise for your changes to take effect.

Example of field alias additions for a lookup
Say you're creating a lookup for an external static table CSV file where the field you've extracted at search time as "ip" is referred to
as "ipaddress." In the props.conf file where you've defined the extraction, you would add a line that defines "ipaddress" as an alias
for "ip," as follows:
[accesslog]
EXTRACT-extract_ip = (?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
FIELDALIAS-extract_ip = ip AS ipaddress
When you set up the lookup in props.conf, you can just use ipaddress where you'd otherwise have used ip:
[dns]
lookup_ip = dnsLookup ipaddress OUTPUT host
223
The eval command is immensely versatile and useful. But while some eval expressions are relatively simple, they often can be quite
complex. If you find that you need to use a particularly long and complex eval expression on a regular basis, you may find that
retyping the expression accurately in search after search is tedious business.
This is where calculated fields come to the rescue. Calculated fields enable you to define fields with eval expressions in props.conf .
Then, when you're writing out a search, you can cut out the eval expression entirely and reference the field like you would any other
extracted field. When you run the search, the fields will be extracted at search time and will be added to the events that include the
fields in the eval expressions.
For example, take this example search , which examines earthquake data and classifies quakes by their depth by creating a new
Description field:
source=eqs7day-M1.csv | eval Description=case(Depth<=70, "Shallow", Depth>70 AND Depth<=300, "Mid",
Depth>300 AND Depth<=700, "Deep") | table Datetime, Region, Depth, Description
Using calculated fields, you could define the eval expression for the Description field in props.conf and write the search as:
source=eqs7day-M1.csv | table Datetime, Region, Depth, Description
You can now search on Description as if it is any other extracted field. Splunk Enterprise will find the calculated field key in
props.conf and evaluate it for every event that contains a Depth field. You can also run searches like this:
source=eqs7day-M1.csv Description=Deep
Note: In the next section we show you how the Description calculated field would be set up in props.conf.
224
Hands on Lab:
To create a calculated field go to: Settings -> Fields -> Add new (under Calculated Fields sections)
Sourcetype : csv
Name of field : a_test
Eval function: annual_inc * 2
Save it
When you bring up the csv sourcetype in search , you will see the field a_test doubled the amount of annual_inc
Now you can try other calculated fields, if you like
225

Please refer to virtual machine for test
226
Module 12 - Creating Field Extractions

Hands on Lab covering: Perform field extractions using Field Extractor
227
As Splunk Enterprise processes events, it extracts fields from them. This process is called field extraction.
Splunk Enterprise automatically extracts some fields
Splunk Enterprise extracts some fields from your events without assistance. It automatically extracts host, source, and sourcetype
values, timestamps, and several other default fields when it indexes incoming events.
It also extracts fields that appear in your event data as key=value pairs. This process of recognizing and extracting k/v pairs is called
field discovery. You can disable field discovery to improve search performance.
When fields appear in events without their keys, Splunk Enterprise uses pattern-matching rules called regular expressions to extract
those fields as complete k/v pairs. With a properly configured regular expression, Splunk Enterprise can extract user_id=johnz from
the previous sample event. Splunk Enterprise comes with several field extraction configurations that use regular expressions to
identify and extract fields from event data.
To get all of the fields in your data, create custom field extractions
To use the power of Splunk Enterprise search, create additional field extractions. Custom field extractions allow you to capture and
track information that is important to your needs, but which is not automatically discovered and extracted by Splunk Enterprise. Any
field extraction configuration you provide must include a regular expression that tells Splunk Enterprise how to find the field that you
want to extract.
All field extractions, including custom field extractions, are tied to a specific source, sourcetype, or host value. For example, if you
create an ip field extraction, you might tie the extraction configuration for ip to sourcetype=access_combined.
228
Custom field extractions should take place at search time, but in certain rare circumstances you can arrange for some custom field
extractions to take place at index time.
Before you create custom field extractions, get to know your data
Before you begin to create field extractions, ensure that you are familiar with the formats and patterns of the event data associated
with the source, sourcetype, or host that you are working with. One way is to investigate the predominant event patterns in your
data with the Patterns tab..
Here are two events from the same source type, an apache server web access log.
131.253.24.135 - - [03/Jun/2014:20:49:53 -0700] "GET /wp-content/themes/aurora/style.css HTTP/1.1" 200 7464
"http://www.splunk.com/download" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0;
Trident/5.0)
10.1.10.14 - - [03/Jun/2014:20:49:33 -0700] "GET / HTTP/1.1" 200 75017 "-" "Mozilla/5.0 (compatible; Nmap
Scripting Engine; http://nmap.org/book/nse.html)"
While these events contain different strings and characters, they are formatted in a consistent manner. They both present values for
fields such as clientIP, status, bytes, method, and so on in a reliable order.
Reliable means that the method value is always followed by the URI value, the URI value is always followed by the status value, the
status value is always followed by the bytes value, and so on. When your events have consistent and reliable formats, you can
create a field extraction that accurately captures multiple field values from them.
For contrast, look at this set of Cisco ASA firewall log events:
Jul 15 20:10:27 10.11.36.31 %ASA-6-113003: AAA group policy for user AmorAubrey is being set to
1 Acme_techoutbound
Jul 15 20:12:42 10.11.36.11 %ASA-7-710006: IGMP request discarded from 10.11.36.36 to
2 outside:87.194.216.51
3 Jul 15 20:13:52 10.11.36.28 %ASA-6-302014: Teardown TCP connection 517934 for Outside:128.241.220.82/1561
229
to Inside:10.123.124.28/8443 duration 0:05:02 bytes 297 Tunnel has been torn down (AMOSORTILEGIO)
Apr 19 11:24:32 PROD-MFS-002 %ASA-4-106103: access-list fmVPN-1300 denied udp for user 'sdewilde7'
4 outside/12.130.60.4(137) -> inside1/10.157.200.154(137) hit-cnt 1 first hit [0x286364c7, 0x0] "
While these events contain field values that are always space-delimited, they do not share a reliable format like the preceding two
events. In order, these events represent:
1. A group policy change
2. An IGMP request
3. A TCP connection
4. A firewall access denial for a request from a specific IP
Because these events differ so widely, it is difficult to create a single field extraction that can apply to each of these event patterns and
extract relevant field values.
In situations like this, where a specific host, source type, or source contains multiple event patterns, you may want to define field
extractions that match each pattern, rather than designing a single extraction that can apply to all of the patterns. Inspect the events to
identify text that is common and reliable for each pattern.
Using required text in field extractions
In the last four events, the string of numbers that follows %ASA-#- have specific meanings. You can find their definitions in the Cisco
documentation. When you have unique event identifiers like these in your data, specify them as required text in your field extraction.
Required text strings limit the events that can match the regular expression in your field extraction.
Specifying required text is optional, but it offers multiple benefits. Because required text reduces the set of events that it scans, it
improves field extraction efficiency and decreases the number of false-positive field extractions.
230
The field extractor utility enables you to highlight text in a sample event and specify that it is required text.
Methods of custom field extraction in Splunk Enterprise
As a knowledge manager you oversee the set of custom field extractions created by users of your Splunk Enterprise implementation,
and you might define specialized groups of custom field extractions yourself. The ways that you can do this include:
The field extractor utility, which generates regular expressions for your field extractions.
Adding field extractions through pages in Settings. You must provide a regular expression.
Manual addition of field extraction configurations at the .conf file level. Provides the most flexibility for field
extraction.
The field extraction methods that are available to Splunk Enterprise users are described in the following sections. All of these methods
enable you to create search-time field extractions. To create an index-time field extraction, choose the third option: Configure field
extractions directly in configuration files.
Let the field extractor build extractions for you
The field extractor utility leads you step-by-step through the field extraction design process. It provides two methods of field
extraction: regular expressions and delimiter-based field extraction. The regular expression method is useful for extracting fields from
unstructured event data, where events may follow a variety of different event patterns. It is also helpful if you are unfamiliar with
regular expression syntax and usage, because it generates regular expressions and lets you validate them.
The delimiter-based field extraction method is suited to structured event data. Structured event data comes from sources like SQL
databases and CSV files, and produces events where all fields are separated by a common delimiter, such as commas, spaces, or pipe
characters. Regular expressions usually are not necessary for structured data events from a common source.
With the regular expression method of the field extractor you can:
Set up a field extraction by selecting a sample event and highlighting fields to extract from that event.
231
Create individual extractions that capture multiple fields.

Improve extraction accuracy by detecting and removing false positive matches.
Validate extraction results by using search filters to ensure specific values are being extracted.
Specify that fields only be extracted from events that have a specific string of required text.
Review stats tables of the field values discovered by your extraction.
Manually configure regular expression for the field expression yourself.
With the delimiter method of the field extractor you can:
Identify a delimiter to extract all of the fields in an event.

Rename specific fields as appropriate.
Validate extraction results.
The field extractor can only build search time field extractions that are associated with specific sources or source types in your data
(no hosts).
Define field extractions with the Field Extractions and Field Transformations pages
You can use the Field Extractions and Field Transformations pages in Settings to define and maintain complex extracted fields in
Splunk Web.
This method of field extraction creation lets you create a wider range of field extractions than you can generate with the field extractor
utility. It requires that you have the following knowledge.
Understand how to design regular expressions.

Have a basic understanding of how field extractions are configured in props.conf and transforms.conf.
If you create a custom field extraction that extracts its fields from _raw and does not require a field transform, use the field extractor
utility. The field extractor can generate regular expressions, and it can give you feedback about the accuracy of your field extractions
as you define them.
232
Use the Field Extractions page to create basic field extractions, or use it in conjunction with the Field Transformations page to define
field extraction configurations that can do the following things.
Reuse the same regular expression across multiple sources, source types, or hosts.
Apply multiple regular expressions to the same source, source type, or host.
Use a regular expression to extract fields from the values of another field.
The Field Extractions and Field Transformations pages define only search time field extractions.
233
Hands on Lab
Please refer to Lab on desktop
234

Please refer to quiz on Virtual Machine
235
Module 13 - Creating Tags and Event Types

Create and use tags
Hands on Lab covering: Create and use tags, Describe event types and their uses, create and event type.
236
Create and use tags
Settings Tags
List by tag name Click Add new
237
Event types are a categorization system to help you make sense of your data. Event types let you sift through huge amounts of data,
find similar patterns, and create alerts and reports.
Events versus event types

An event is a single record of activity within a log file. An event typically includes a timestamp and provides information about what
occurred on the system being monitored or logged.
An event type is a user-defined field that simplifies search by letting you categorize events. Event types let you classify events that
have common characteristics. When your search results come back, they're checked against known event types. An event type is
applied to an event at search time if that event matches the event type definition in eventtypes.conf. Tag or save event types after
indexing your data.
Event type classification

There are several ways to create your own event types. Define event types via Splunk Web or through configuration files, or you can
save any search as an event type.
238

You complete a search, then Click Save As Event Type
239
Hands on Lab covering

240

Please refer to quiz on virtual machine
241
Module 14 - Creating Workflow Actions

Hands on Lab covering: Describe the function of a workflow action, Create a GET workflow action
Hands on Lab covering: Create a POST workflow action, Create a SEARCH workflow action
242
Workflow actions have a wide variety of applications. For example, you can define workflow actions that enable you to:
Perform an external WHOIS lookup based on an IP address found in an event.

Use the field values in an HTTP error event to create a new entry in an external issue management system.
Launch secondary searches that use one or more field values from selected events.
Perform an external search (using Google or a similar web search application) on the value of a specific field
found in an event.
In addition, you can define workflow actions that:
Are targeted to events that contain a specific field or set of fields, or which belong to a particular event type.
Appear either in field menus or event menus in search results. You can also set them up to only appear in the
menus of specific fields, or in all field menus in a qualifying event.
When selected, open either in the current window or in a new one.
Define workflow actions using Splunk Web

You can set up all of the workflow actions described in the bulleted list at the top of this chapter and many more using Splunk Web.
To begin, navigate to Settings > Fields > Workflow actions. On the Workflow actions page you can review and update existing
workflow actions by clicking on their names. Or you can click Add new to create a new workflow action. Both methods take you to
the workflow action detail page, where you define individual workflow actions.
If you're creating a new workflow action, you need to give it a Name and identify its Destination app.
There are three kinds of workflow actions that you can set up:
243
GET workflow actions, which create typical HTML links to do things like perform Google searches on
specific values or run domain name queries against external WHOIS databases.
POST workflow actions, which generate an HTTP POST request to a specified URI. This action type
enables you to do things like create entries in external issue management systems using a set of relevant field
values.
Search workflow actions, which launch secondary searches that use specific field values from an event,
such as a search that looks for the occurrence of specific combinations of ipaddress and http_status' field
values in your index over a specific time range.
244
GET link workflow actions drop one or more values into an HTML link. Clicking that link performs an HTTP GET request in a
browser, allowing you to pass information to an external web resource, such as a search engine or IP lookup service.
To define a GET workflow action:
1. Navigate to Settings > Fields > Workflow Actions.
2. Click New to open up a new workflow action form.
3. Define a Label for the action.
The Label field enables you to define the text that is displayed in either the field or event workflow menu.
Labels can be static or include the value of relevant fields.
4. Determine whether the workflow action applies to specific fields or event types in your data.
Use Apply only to the following fields to identify one or more fields. When you identify fields, the
workflow action only appears for events that have those fields, either in their event menu or field menus. If
you leave it blank or enter an asterisk the action appears in menus for all fields.
Use Apply only to the following event types to identify one or more event types. If you identify an event
type, the workflow action only appears in the event menus for events that belong to the event type.
5. For Show action in determine whether you want the action to appear in the Event menu, the Fields menus, or Both.
6. Set Action type to link.
245
7. In URI provide a URI for the location of the external resource that you want to send your field values to.
Similar to the Label setting, when you declare the value of a field, you use the name of the field enclosed by
dollar signs.
Variables passed in GET actions via URIs are automatically URL encoded during transmission. This means
you can include values that have spaces between words or punctuation characters.
8. Under Open link in, determine whether the workflow action displays in the current window or if it opens the link in a new window.
9. Set the Link method to get.
10. Click Save to save your workflow action definition.
246
Hands-on Lab
247
Set up a POST workflow action

You set up POST workflow actions in a manner similar to that of GET link actions. However, POST requests are typically defined by
a form element in HTML along with some inputs that are converted into POST arguments. This means that you have to identify POST
arguments to send to the identified URI.
1. Navigate to Settings > Fields > Workflow Actions.
2. Click New to open up a new workflow action form.
3. Define a Label for the action.
The Label field enables you to define the text that is displayed in either the field or event workflow menu.
Labels can be static or include the value of relevant fields.
4. Determine whether the workflow action applies to specific fields or event types in your data.
Use Apply only to the following fields to identify one or more fields. When you identify fields, the
workflow action only appears events that have those fields, either in their event menu or field menus. If you
leave it blank or enter an asterisk the action appears in menus for all fields.
Use Apply only to the following event types to identify one or more event types. If you identify an event
type, the workflow action only appears in the event menus for events that belong to the event type.
5. For Show action in determine whether you want the action to appear in the Event menu, the Fields menus, or Both.
6. Set Action type to Link.
248
7. Under URI provide the URI for a web resource that responds to POST requests.
8. Under Open link in, determine whether the workflow action displays in the current window or if it opens the link in a new window.
9. Set Link method to Post.
10. Under Post arguments define arguments that should be sent to web resource at the identified URI.
These arguments are key and value combinations. On both the key and value sides of the argument, you can
use field names enclosed in dollar signs to identify the field value from your events that should be sent over to
the resource. You can define multiple key/value arguments in one POST workflow action.
Enter the key in the first field, and the value in the second field. Click Add another field to create an
additional POST argument.
11. Click Save to save your workflow action definition.
Splunk software automatically HTTP-form encodes variables that it passes in POST link actions via URIs. This means you can
include values that have spaces between words or punctuation characters.
249
To set up workflow actions that launch dynamically populated secondary searches, you start by setting Action type to search on the
Workflow actions detail page. This reveals a set of Search configuration fields that you use to define the specifics of the secondary
search.
In Search string enter a search string that includes one or more placeholders for field values, bounded by dollar signs. For example, if
you're setting up a workflow action that searches on client IP values that turn up in events, you might simply enter
clientip=$clientip$ in that field.
Identify the app that the search runs in. If you want it to run in a view other than the current one, select that view. And as with all
workflow actions, you can determine whether it opens in the current window or a new one.
Be sure to set a time range for the search (or identify whether it should use the same time range as the search that created the field
listing) by entering relative time modifiers in the in the Earliest time and Latest time fields. If these fields are left blank the search runs
over all time by default.
Finally, as with other workflow action types, you can restrict the search workflow action to events containing specific sets of fields
and/or which belong to particular event types.
250
Hands-on Lab
251
End of Module Quiz

Please refer to questions on virtual machine
252
Module 15 - Creating and Managing Alerts
Describe alerts
Create alerts
View fired alerts
Hands on Lab covering: Describe alerts, Create alerts, View fired alerts
253
Describe alerts
An alert is an action that a saved search triggers based on the results of the search. When creating an alert, you specify a condition that
triggers the alert. Typically the action is an email based on the results of the search. But you can also choose to run a script or to list
the alert as a triggered alert in Settings. When you create an alert you are creating a saved search with trigger conditions for the alert.
To avoid sending out alerts too frequently, specify a throttle condition for an alert.
The following list describes the types of alerts:
Per result alert. Based on a real-time search. The trigger condition is whenever the search returns a result.
Scheduled alert. Runs a search according to a schedule that you specify when creating the alert. You specify results of the
search that trigger the alert.
Rolling-window alert. Based on a real-time search. The trigger condition is a combination of specified results of the search
within a specified time window.
254
Create alerts
A scheduled alert runs periodically at a scheduled time, responding to a condition that triggers the alert.
This example uses a search to track when there are too many errors in a Splunk Enterprise instance during the last 24 hours. When the
number of errors exceeds 5, the alert sends an email with information about the conditions that triggered the alert. The alert sends an
email every day at 10:00AM when the number of errors exceed the threshold.
1. From the Search Page, create the following search
index=_internal " error " NOT debug source=*splunkd.log* earliest=-24h latest=now
2. Click Save As > Alert.

3. Specify the following values for the fields in the Save As Alert dialog box:
Title: Errors in the last 24 hours
Alert type: Scheduled
Time Range: Run every day
Schedule: At 10:00
Trigger condition: Number of Results
Trigger if number of results: is Greater than 5.
255
4. Click Next.
5. Click Send Email.
6. Set the following email settings, using tokens in the Subject and Message fields:
To: email recipient
Priority: Normal
Subject: Too many errors alert: $name$
Message: There were $job.resultCount$ errors reported on $trigger_date$.
Include: Link to Alert and Link to Results
Accept defaults for all other options.
256
7. Click Save.
After you create the alert you can view and edit the alert in the Alerts Page.
When the alert triggers, it sends the following email:
257
258
View fired alerts

Simply go to the Alerts Page on the top Toolbar
259
Hands-on Lab
260
End of Module Quiz
Please refer to questions on virtual machine
261
Module 16 - Creating and Using Macros
Describe macros
Manage macros
Create and use a basic macro
Hands on Lab covering: Describe macros, Manage macros, Create and use a basic macro.
Define arguments and variables for a macro
Add and use arguments with a macro
Hands on Lab covering: Define arguments and variable for a macro, Add and use arguments with a macro.
262
Describe Macros
Search macros are chunks of a search that you can reuse in multiple places, including saved and ad hoc searches.
Search macros can be any part of a search, such as an eval statement or search term, and do not need to be a
complete command. You can also specify whether or not the macro field takes any arguments.
263
Manage and create macros

In Settings > Advanced Search > Search macros, click "New" to create a new search macro.
Define the search macro and its arguments
Your search macro can be any chunk of your search string or search command pipeline that you want to re-use as part of another
search.
Destination app is the name of the app you want to restrict your search macro to; by default, your search macros are restricted to the
Search app.
Name is the name of your search macro, such as mymacro. If your search macro takes an argument, you need to indicate this by
appending the number of arguments to the name; for example, if mymacro required two arguments, it should be named mymacro(2).
You can create multiple search macros that have the same name but require different numbers of arguments: foo, foo(1), foo(2),
etc.
Definition is the string that your search macro expands to when referenced in another search. If the search macro requires the user to
input arguments, they are tokenized and indicated by wrapping dollar signs around the arguments; for example, $arg1$. The
arguments values are then specified when the search macro is invoked.
If Eval Generated Definition? is checked, then the 'Definition' is expected to be an eval expression that
returns a string that represents the expansion of this macro.
If a macro definition includes a leading pipe character ("|"), you may not use it as the first term in
searches from the UI. Example: "| metadata type=sources". The UI does not do the macro expansion and
cannot correctly identify the initial pipe to differentiate it from a regular search term. The UI constructs the
search as if the macro name were a search term, which after expansion would cause the metadata command
to be incorrectly formed and therefore invalid.
264
Arguments are a comma-delimited string of argument names. Argument names may only contain the characters: alphanumeric 'a-Z,
A-Z, 0-9'; underscore '_'; and dash '-'. This list should not contain any repeated elements.
If a macro argument includes quotes, you need to escape the quotes when you call the macro in
your search. For example, if you wanted to pass a quoted string as your macro's argument, you would use:
`my-macro("He said \"hello!\"")`.
Validate your argument values

You can verify that the argument values used to invoke the search macro are acceptable. How to invoke search macros are discussed
in the following section, "Apply macros to saved and ad hoc searches".
Validation Expression is a string that is an 'eval' expression that evaluates to a boolean or a string.
If the validation expression is a boolean expression, validation succeeds when it returns true. If it returns
false or is null, validation fails, and the Validation Error Message is returned.
If the validation expression is not a boolean expression, it is expected to return a string or NULL. If it returns null, validation is
considered a success. Otherwise, the string returned is rendered as the error string.
Apply macros to saved and ad hoc searches
To include a search macro in your saved or ad hoc searches, use the left quote (also known as a grave accent) character; on most
English-language keyboards, this character is located on the same key as the tilde (~). You can also reference a search macro within
other search macros using this same syntax.
Note: Do NOT use the straight quote character that appears in the same key as the double quote (").
265
Hands-on Lab
266
End of Module Quiz
Please refer to virtual machine for quiz
267
Module 17 - Using Pivot

Describe Pivot
Select a data model object
Hands on Lab covering: Describe Pivot, Understand the relationship between data models and pivot, Select a data model object.
Create a pivot report
Save pivot report as a dashboard
Hands on Lab covering: Create a pivot report, Save pivot report as a dashboard.
268
Describe Pivot
The Pivot tool lets you report on a specific data set without the Splunk Enterprise Search Processing Language (SPL). First, identify
a dataset that you want to report on, and then use a drag-and-drop interface to design and generate pivots that present different aspects
of that data in the form of tables, charts, and other visualizations.
How does Pivot work? It uses data models to define the broad category of event data that you're working with, and then uses
hierarchically arranged collections of data model objects to further subdivide the original dataset and define the attributes that you
want Pivot to return results on. Data models and their objects are designed by the knowledge managers in your organization. They do
a lot of hard work for you to enable you to quickly focus on a specific subset of event data.
269
Data models drive the Pivot tool. They enable users of Pivot to create compelling reports and dashboards without designing the
searches that generate them. Data models can have other uses, especially for Splunk Enterprise app developers.
Splunk Enterprise knowledge managers design and maintain data models. These knowledge managers understand the format and
semantics of their indexed data and are familiar with the Splunk Enterprise search language. In building a typical data model,
knowledge managers use knowledge object types such as lookups, transactions, search-time field extractions, and calculated fields.
What is a data model?

A data model is a hierarchically structured search-time mapping of semantic knowledge about one or more datasets. It encodes the
domain knowledge necessary to build a variety of specialized searches of those datasets. These specialized searches are used by
Splunk Enterprise to generate reports for Pivot users.
When a Pivot user designs a pivot report, she selects the data model that represents the category of event data that she wants to work
with, such as Web Intelligence or Email Logs. Then she selects an object within that data model that represents the specific dataset on
which she wants to report. Data models are composed of objects, which can be arranged in hierarchical structures of parent and child
objects. Each child object represents a subset of the dataset covered by its parent object.
If you are familiar with relational database design, think of data models as analogs to database schemas. When you plug them into the
Pivot Editor, they let you generate statistical tables, charts, and visualizations based on column and row configurations that you select.
To create an effective data model, you must understand your data sources and your data semantics. This information can affect your
data model architecture--the manner in which the objects that make up the data model are organized.
270
Data Model Object
Data models are composed of one or more objects. Here are some basic facts about data model objects:
An object is a specification for a dataset. Each data model object corresponds in some manner to a set of data in an index.
You can apply data models to different indexes and get different datasets.
Objects break down into four types. These types are: Event objects, search objects, transaction objects, and child objects.
Objects are hierarchical. Objects in data models can be arranged hierarchically in parent/child relationships. The top-level
event, search, and transaction objects in data models are collectively referred to as "root objects."
Child objects have inheritance. Data model objects are defined by characteristics that mostly break down into constraints and
attributes. Child objects inherit constraints and attributes from their parent objects and have additional constraints and
attributes of their own.
271
Hands-on Lab
272
End of Module Quiz
Please refer to virtual machine for Quiz
273
End of Course Quiz

Please refer to virtual machine for Quiz
274

Splunk Book

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Splunk Book

Hochgeladen von

Copyright:

Verfügbare Formate

Splunk Overview Training

What Youll Learn: Course Objectives

Recommended Audience & Pre-Requisites

Course Topics & Agenda

Module 4 - Using Fields in Searches

Course Modules 5-7 Day 1 - Afternoon

Module 7 - Search Fundamentals

Create a time chart

Module 10 - Analyzing, Calculating, and Formatting Results

Course Modules 13-15 Day 3 - Morning

End of Module Hands on Quiz

Module 17 - Using Pivot

Module 1 - Basic Understanding of Architecture (Overview)

What are the components?

Section 1-What are the components?

This diagram shows several forwarders sending data to a single indexer:

Section 2-Discussion on Forwarders- UF/HF

The universal forwarder has no searching, indexing, or alerting capability.

Heavy and light forwarders

Type of Splunk Enterprise instance

Full Splunk Enterprise, with some

Footprint (memory, CPU load)

Medium-to-large (depending on enabled

Handles data inputs?

All types (but scripted inputs might

Forwards to Splunk Enterprise?

Forwards to 3rd party systems?

Serves as intermediate forwarder?

Indexer acknowledgment (guaranteed

Optional (version 4.2+)

Section 3- Common ports for the set up

Splunk configures two ports at installation time:

Let's login to our lab environment

Section 4-License Master/Slave relationship

Section 5-Understanding of Deployment Server and Indexer

Indexing incoming data.

Here's an example of a scaled-out deployment:

Module 2 - Introduction to Splunk's User Interface

Understand the uses of Splunk

Section 1-Understand the uses of Splunk

Section 2-Define Splunk Apps

Section 3-Learn basic navigation in Splunk

You can do two actions on this panel:

Click the plus icon to browse for more apps to install.

Explore Splunk Enterprise

The following screenshot shows the Splunk bar in Splunk Home.

Return to Splunk Home

Hands on Lab covering: Basic Navigation

End of Module Hands-on Quiz

Run basic searches

Run basic searches

Search and knowledge

The anatomy of a search

Quotes and escaping characters

Example 2: myfield is created with the value of ".

Example 3: myfield is created with the value of \.

Example 4: This would produce an error because of unbalanced quotes.

Set the time range of a search

Select from a list of Preset time ranges

Define custom Relative time ranges

Define custom Real-time time ranges

Define custom Date ranges

Define custom Date & Time ranges

Use Advanced time range options