McAfee Enterprise Security Manager

Writing Custom Parsing Rules in McAfee ESM

September 20, 2018

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 1 of 20

Table of Contents
1 Introduction...................................................................................................................................................................... 2
2 Prerequisites..................................................................................................................................................................... 2
3 Creating a Custom ASP Rule ............................................................................................................................................. 3
3.1 Create a New Data Source......................................................................................................................................... 3
3.2 Create a Custom Parsing Rule for ASP ....................................................................................................................... 3
3.3 Enable the New ASP Rule in Policy .......................................................................................................................... 14
4 Creating a New Custom Type ......................................................................................................................................... 15
4.1 Create a Custom Type ............................................................................................................................................. 15
5 Testing the New ASP Rule .............................................................................................................................................. 17
5.1 Importing a Log Sample ........................................................................................................................................... 17
6 Best Practices ................................................................................................................................................................. 18
6.1 Know Your Data ....................................................................................................................................................... 19
6.2 Regular Expressions ................................................................................................................................................. 19
6.3 Content .................................................................................................................................................................... 19

1 Introduction
This guide details how to create and deploy a custom advanced syslog parser (ASP) rule in the
McAfee ESM. A custom ASP rule could be necessary if there is an unknown/unparsed log in the McAfee ESM.
This guide also includes some “Best Practices” to follow when writing ASP rules.

2 Prerequisites
The following will be required, or useful, in creating custom parsing rules:

• A sample of the log to parse.

• Account privileges to add data sources. In some cases, administrator privileges, such as the ability to
define custom types may also be needed.

• Vendor documentation can be helpful to determine the meaning of each field in the logs desired to
be parsed.

• A working knowledge of PCRE (Perl Compatible Regular Expressions).

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 2 of 20

3 Creating a Custom ASP Rule
Follow the steps below to build a new SYSLOG data source and create an ASP parsing rule.

3.1 Create a New Data Source

This step is optional if a data source already exists and is collecting the logs needing to be parsed.

1. Select the Receiver, and then click on the “+” icon to add a data source.

2. The Add Data Source window will open, set the Data Source Vendor as Generic and the Data Source
Model as Advanced Syslog Parser. Give the new data source a name, add the proper IP address,
and ensure that the Support Generic Syslogs field is left as Do nothing.

3. Select OK, and then apply the new data source settings by confirming in the next
window. This will create a new data source with an empty rule policy.
3.2 Create a Custom Parsing Rule for ASP
1. From the McAfee ESM console, select the data source and open the Policy Editor.
The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 3 of 20

 2. The Policy Editor window should appear. In the menu on the left, expand Receiver by selecting the
“+” symbol, and select Advanced Syslog Parser. Initially this list will be blank while looking at the
policy for the new data source.

3. Select New > Advanced Syslog Parser Rule.

4. The Advanced Syslog Parser Rule window will appear. Within the General tab, assign information
as displayed below:

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 4 of 20

 Name – Assign a name, making it as descriptive and unique as possible. This is the event message
that will show in the ESM views when the rule matches a log, unless the message is mapped
directly from the log text within the rule.
Tags – Assign tags to the rule. It is best practice to assign one or more tags to which this rule
belongs. This helps in finding and grouping sets of rules created for a given device or application
within the policy editor. Any tags added to a rule, will cause ESM to automatically include the rule
in any policy that has enabled the given tagged rule set.
Default Normalized ID – The normalization or category is a key step as many views, correlation
rules, and reports use this field as a filter. It is very important to select the most relevant value in
the taxonomy for the parsing rule in order to get the greatest value from within the McAfee ESM.
Default Severity – If the log message does not contain a value for the severity, then the event will
be assigned the value that is set here (the default is 25, with a scale of 1-100, 1 being the lowest
and 100 being the highest).
Rule Assignment Type – Rules can be grouped together; this pull-down menu provides a list of
supported products to group the parsing rules by, separating the events from other data sources.
This allows the event to be reported for a specific product.
Description – Give the new rule a clear human-readable description. It should be a clear and
complete description of the rule with complementary information to what was given in the rule’s
name.
5. Click on the next tab, Parsing, assign information as displayed below:

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 5 of 20

 Process Name – This is an optional filter that acts similar to the content string, but only applies to
the process name found within the SYSLOG header. Due to various syslog header formats used, and
no guarantee that a syslog header will be present in the log, it is recommended to use content
strings instead of the process name match.
Content String – If a fixed string is always going to be found in the log then add it as a content
string. The content string(s) of an ASP rule should uniquely identify each log if possible. Content
strings are very important as they will not only make matches more accurate, but will speed up the
execution of the rules in the running policy. It is highly recommended to have one or more content
strings per ASP rule. This is a pre-filter for optimization – only logs that match the given content
strings will be considered for matching and parsing by the regular expressions. The log must contain
all given content strings, otherwise the logs will not be parsed using the regular expressions.
Regular Expression – The first regular expression determines if the ASP rule will match the log. Any
additional expressions will be used to capture values from the log that may or may not be found in
all logs.
Named Captures – Named captures can be used in regular expressions to more easily identify
capture groups. The label used for the named capture can consist of alphanumeric and
underscore characters but cannot begin with a number or include a space. The regular expression
syntax for a named capture is: (?P<NAME>regular expression capture). Here is an
example of a named capture where hostname is the name assigned to the capture group:
Host\x3d(?P<hostname>\S+). When using named captures the policy editor will display the
capture name instead of the capture number, in the right-hand side of the Parsing tab as shown
below.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 6 of 20

 Targeted Regular Expressions – Targeted regular expressions allow for applying an additional
regular expression to the resulting value from another capture. For example, a regular expression
could be written to pull a user’s full name from a log. Referencing that capture as the target, two
targeted regular expressions could then be written to pull out the user’s first and last name. This
data can then be mapped to different database fields. Follow the steps below to create a
targeted regular expression:

1. On the Parsing tab, add a regular expression.

2. Add another regular expression and select the capture (name or number) from the Target
key drop-down list that references the first expression. The second regular expression
should contain an expression to capture additional information from the captured value of
the first regular expression.

In the image below, the named capture group full_name has a value of “Sam Johnson”. This
named capture is being used as the Target Key for the first_name and last_name targeted
regular expressions. The regular expression for first_name and last_name will be evaluated
against the value of the full_name named capture. The result shows the key of first_name:1 with
a value of “Sam” and last_name:1 with a value of “Johnson”. This specific example could be done
within a single regular expression, and is a simplified example to illustrate the purpose of the
Target Key field used with regular expressions. In general, it is recommended to run as few
regular expressions as possible, but that may vary with specific use cases.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 7 of 20

 Sample Log Data – In this field a sample log can be added and will be saved with the rule. The
parts of the log matching defined regular expressions will be highlighted blue.
Format – ASP can pre-process certain logging formats to simplify the mapping of data. The
following formats are available:
Generic – This is the default and should be used if the log does not match the other
available formats.
CEF (Common Event Format) – This eliminates the need to create a regular expression
for each capture, and will allow the data to be mapped using the CEF key names found
in the log.
JSON – Similar to CEF, this eliminates the need to create a regular expression for each
capture, and will allow data to be mapped using the JSON key names found in the log.
XML - Basic, Simple, or Positional – This will allow ASP to parse logs that are in XML
format and assign parsed data. The XML format choice will depend upon the type of XML
that is within the logs.
XML – Basic expects XML without any repeated elements.
XML – Simple expects XML with either a single node with attributes, or a single
set of non-repeated elements without nesting.
XML – Positional expects XML that can have multiple nodes with attributes and
multiple repeated elements with nesting.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 8 of 20

 Parsed Values – The Key/Value fields on the right displays what is being parsed from the log
sample(s) by the regular expression(s). The Key will display two numbers, divided by a colon. The
first number indicates the regular expression being used, and the second number indicates the
capture group used within that regular expression. If a captured value is the 4th capture in the 3rd
regular expression defined, the key would display “3:4”.

6. Add applicable Content Strings by selecting Edit. These are very important and can drastically
affect performance of the created rule. Any content strings added will need to be a string that is
guaranteed to be within the log, as these act as a type of “prefilter”. Multiple unique content
strings can be added to increase the speed and efficiency of the rule.

7. Select applicable regular expression options by adding checks within the boxes add one or more
Regular Expressions by selecting the “+” symbol.

Only use regular expressions for parsing purposes – Checking this box will remove the
requirement for the first regular expression to match the log. The parser will rely upon the
Content String for matching purposes, and will only use regular expressions to parse data when it
finds a matching pattern.
Case Insensitive – If the log may contain either upper or lower case letters in some fields, it may
be simpler to write the expression in the same case and then use this option. This enables the
case insensitivity option for all regular expressions defined in the parsing rule.
Trigger when data doesn’t match – This option will make the rule trigger when the regular
expression does not match the log.

a. The Edit Regular Expression window will appear. Add the expression and select OK.

8. With the expression added, the parsed values should be highlighted in blue, with the Group and
Value shown to the right. Verify that each value is parsed out correctly and that all of the log line
samples match. Make sure to know the meaning (from the vendor device documentation or own
knowledge) of all the fields within the logs and capture the values that are useful.
9. After all the regular expressions are written to extract out the needed values from the log, select
the Field Assignment tab at the top of the window.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 9 of 20

 10. Available fields will be displayed, along with the Group and Value section as shown in the
previous Parsing tab.

11. Drag and drop the Values from the right hand side to the Expression column next to the Field on
the left.

a. For more complicated log file formats, multiple regular expressions may be used to
capture the same data. In such cases, a field such as First Time might come from one of
several potential several expressions. McAfee ESM allows the ability to specify the
regular expression values in order of preference, so that if one does not match, it will
use the second, or third, or so on. An example of this is shown below:

12. If the field is not displayed that is needed, click “+” above the Sample Value column, to display all
custom type fields.

13. The Custom Types window will now be displayed. Select the desired field and then select OK.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 10 of 20

 14. If a custom type does not exist for the type of data being mapped, the McAfee ESM allows for
user-defined custom types to be added. Once a new custom type has been created, it will be
listed within the Custom Types window and be available for use to map the values to. Please see
Section 4 for details on creating a new custom type.

15. Click on the final tab, Mapping. This section will allow parsing the date format, mapping the
action, and mapping the severity found within the log message.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 11 of 20

 Time Format – The date/timestamp of a log message can be parsed using the variables defined
in these fields. Many standard date/timestamps are recognized automatically by the McAfee
ESM, however there may be a format that is either not recognized or that would be preferred to
display differently. This section will allow for formatting the time to show up in the proper
format when parsed.
Action Mapping – Use this option if there is an action found within the log to be mapped to an
available McAfee ESM action.
Severity Mapping – The severity mapping allows for a value within the log to be mapped to a
severity from 1-100. For example, a vendor might define their severity as either “Low”,
“Medium”, or “High” in their logs. With the Severity Map section the severity value can map
“Low” as 25, “Medium” as 50, and “High” as 75.

16. To add a Time Format, select the “+” symbol in the upper-right of the time format section.

a. The Add Time Format window will appear.

b. Enter a custom time format. As the values are entered, a small window will appear to
assist in creating the custom time. The values can either be typed or selected from the
given list.

c. Select the fields that this custom format will apply to. It is best practice to apply it to
both First Time and Last Time.

d. Select OK to save.

17. To add an Action Map, find an applicable Action Value by scrolling through the list and then click
inside the Action Key column. Enter the value that will be parsed from the device logs. If this
value is parsed and assigned to “Action”, then it will map to the assigned Action Value.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 12 of 20

 Below the Action Mapping table, the option will be given to use a default action when one is not
specified. This is generally a good option to set in order to insure that all logs receive an action
type. Depending on the device, the default action should be different.
To add the default action map, place a check next to Use the following action for the default if
one is not specified, and then select a default action from the drop down menu.

18. To add a Severity Mapping, select the “+” symbol in the upper-right of the Severity Mapping
section.

a. The Add Severity windows will appear.

b. Within the Severity Mapping field, enter the value that will be parsed from the device
logs. If this value is parsed and assigned to “Severity”, then it will map to the assigned
Severity Value.

c. Move the sliding scale to the desired Severity Value.

d. Select OK to save.

19. Below the Severity Mapping section, the option to add a default severity will be given. This is
generally a good option to set in order to insure that all logs receive a severity value. Depending
on the device, the default severity should be different. To add the default severity, place a check
next to Use the following severity for the default if one is not specified, and then enter a default
value.

20. With the previous steps complete, the new ASP rule should be finished. Select Finish, located in
the bottom-right hand corner to finish and save.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 13 of 20

 3.3 Enable the New ASP Rule in Policy
Once the new rule has been created, it will need to be enabled and added to the policy in order for the
receiver to begin parsing the associated logs.

1. Open the Policy Editor for the data source that the new ASP rule was created for, and beneath
Receiver, select Advanced Syslog Parser. The newly created ASP rule(s) will be listed with the
Action column displaying “disabled”. Click on disabled and then change the value to enabled.

2. With the new rule set as enabled, click on the Rollout icon displayed in the upper-right hand
corner of the Policy Editor window.

3. If the rule didn’t already save previously, it will prompt to save. Select Yes to save.

4. Once saved, the Rollout window will appear and will display a list of devices to roll policy out to.
Ensure that the applicable data source is set to “Roll this policy out now.” and select OK to do
so.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 14 of 20

4 Creating a New Custom Type
4.1 Create a Custom Type
To create a custom type, login with a user that has system administrator privileges and then select the
System Properties icon.

1. Select the Custom Types option from the menu on the left.

2. Select Add from the options shown on the right.

Note: Select the Edit option to edit a user defined pre-existing custom type if needed.

3. The Add Custom Type window will appear. Setup the new custom type as desired and select OK.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 15 of 20

 Name – This will be the name for the new Custom Type that will be displayed within McAfee
ESM.
Data Type – This will be the type of data that the McAfee ESM will expect to be populated into
this field. Selection of the correct data type is important. Commonly used data types include:
Random String, IP Address, Decimal, Integer, and GUID.
Events Field – These fields are for Events. There are a specific number of fields that can be used
to have data parsed into. If two different custom types occupy the same field, only one can be
used. When creating custom types, make sure to place it within a field that is not already being
used by another important field that will be used.
Flows Field – These fields are for Flows. There are a specific number of fields that can be used to
have data parsed into. If two different custom types occupy the same field, only one can be used.
When creating custom types, make sure to place it within a field that is not already being used
by another important field that will be used.
Index Data – This option will determine on whether or not the data that is parsed is indexed
within the database. Indexed data is searchable within the McAfee ESM.
Description – This will be a simple description to explain what type of data the newly created
custom type contains.

4. With the desired custom type setting entered, select OK.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 16 of 20

5 Testing the New ASP Rule
5.1 Importing a Log Sample
Once the new ASP rule has been created and added to the current policy, it is a good idea to test it to
ensure that it is parsing as expected.

1. Obtain sample logs from the device that the new rule was created for and save these to a text
file.

2. Select the data source to be tested and click Properties.

3. The Data Source Properties window will appear, select the Upload button.

4. Navigate to the log sample file and select it to upload. Select the Upload button and then a
confirmation window will be displayed showing the file was successfully uploaded. Select Close
to close out the window.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 17 of 20

 5. To initiate the log and flow gathering process manually, click on the Get Events and Flows
button.

6. The Get Events and Flows window will appear. Select Events and then click the Start button to
collect the events.

7. Once the events are collected, find the events in the dashboard and verify the newly created ASP
rule is parsing as expected. If any adjustments need to be made, make the changes and then
send the sample logs through again to verify that the rule is parsing as expected.

6 Best Practices
The following are given as best practices that can help with creating ASP rules that work well within the
McAfee ESM.
Following these recommendations will help ensure reporting is accurate, not overly complex, and will
help parsers and Receivers maintain intended performance and efficiency.

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 18 of 20

 6.1 Know Your Data
It is important to understand the logs that are being sent to the SIEM, and whether or not they have value
in your organizations reporting needs. It is recommended to only parse logs that meet your organization’s
reporting goals.

• Ensure that captured values mapped to ESM database fields align with the intended use of the
specific custom type fields.

• Avoid indexing fields that contain unique and random or high cardinality data such as URLs.

• Ensure rules mapping event messages directly from the log are not mapping unique and random
or high cardinality strings as messages. The ESM creates a data source rule for each unique event
message, and ESM performance can be negatively impacted if there are a high rate of unique
strings being mapped to message.

• Categorize events by adding a normalized category to the rule. Data source rules, generated by
parsing rules, inherit the normalization assigned to the main parsing rule. This means that if the
main parsing rule is left normalized to “Uncategorized”, then the parsed events will also be
normalized as “Uncategorized”, making a search for “Uncategorized” events to find unparsed
events in accurate.

6.2 Regular Expressions

By default, the first regular expression determines if the ASP rule will match the log and will need to
match a pattern that is always present in the log. Additional regular expressions can be written to capture
values from the log and mapped to custom types within the McAfee ESM. Subsequent regular expressions
do not determine the rule match, and are used for parsing only.
While it is possible to test regular expression results on a few log lines within the McAfee ESM console
itself, we recommend using a graphical tool. There are many free web based tools that can be used in
addition to standalone installable tools. Optionally, another useful tool would be a text editor that
supports regular expression searches. Any tools used to test regular expressions need to support pcre
expressions.
Make sure that regular expressions are efficiently written because poorly written expressions can take
more system resources to run, and can affect the parsing performance of a specific data source or even
the entire Receiver.

6.3 Content
Ensure there is at least one value in the content field section. Content strings should be at least 3
characters in length and should be as unique as possible for the specific event. It is advised to include
enough content matches to uniquely identify the log. Using one or more content fields within the ASP rule
will significantly speed up the matching and parsing process on the Receiver.

The table below shows an example of how to use the content field to uniquely identify a log sample.
Syslog Sample Content Fields

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 19 of 20

 <180>Jan 1 00:00:00 testhost ftpd[4325]: FTP “ftpd”
LOGIN FROM test.org [192.168.1.1], anonymous “FTP LOGIN FROM”

<180>Jan 1 00:00:00 testhost ftpd[4325]: FTP “ftpd”

LOGIN REFUSED (ftp not in /etc/passwd) FROM “FTP LOGIN REFUSED”
192.168.1.1 [192.168.1.1], anonymous

The information contained in this document is confidential and proprietary. Please do not
redistribute without permission.

Writing Custom Parsing Rules in McAfee ESM Page 20 of 20