Week 4 Part 5 - Module 20 Log Analysis

EC-Council Network Security Administrator
Log Analysis
Exam 312-38
MODULE 20
Log Analysis
Module Objectives
This module will familiarize you with the following concepts:
Introduction to log analysis
Categorizing the log files
Understanding the concept of web server log analysis
Identifying Syslog statistics and analysis
Understanding the concept of logging
Introduction to monitoring and security events
Understanding log analysis tool
Various log parsing tools
Introduction to log file rotation tools
Understanding log security
Module XX Page | 999
Network Security Administrator Copyright by EC-Council

All Rights Reserved. Reproduction is Strictly Prohibited.

Log Analysis
Exam 312-38
Introduction
This module covers the following EC-Councils Network Security Administrator certification objectives:
1.
Topic A:
Introduction to Log Analysis: Discusses the log analysis, audit events, and log types.
2. Topic B:
Categorizing Log Files: Discusses log files, Access_log, Agent_log, Error_log, refer_log, and
TCPDump Logs.
3. Topic C:
Understanding the Concept of Web Server Log Analysis: Discusses web server log analyses
and its tools, Apache log, IIS log, and the limitations of log file analysis.
4. Topic D:
5.
Identifying Syslog Statistics and Analysis: Discusses syslog, and statistics and analysis of
syslog.
Topic E:
Understanding the Concept of Logging: Overview of logging, setting up remote logging,

windows logging, and application logging, and firewall logging.
6. Topic F:
7.
Introduction to Monitoring and Security Events: Discusses monitoring and security events,
the importance of time synchronization, and passive detection methods and scripting.
Topic G:
Understanding Log Analysis Tool: Discusses log analysis tools like Userlock and WStool and
auditing tools like ADSIC, Tenshi, SpoofMAC, and Gentle MAC Pro.
8. Topic H:
Various Log Parsing Tools: Discusses various log parsing tools like LogSentry, SL2, Flog,
SLCT, xlogmaster, Geek Tool, Dump.exe, and Watchlog and Logdog.
9. Topic I:
Introduction to Log File Rotation Tools: Discusses various log rotation tools like
Logcontroller, Newsyslog, Spinlogs, Trimlog, SLRS, and Bzip2.
10. Topic J:
Understanding Log Security: Discusses secure logs and features.


Log Analysis
Exam 312-38
TOPIC A
Introduction to Log Analysis
Log Analysis
Log analysis is the process of detecting attacks on a specific system, network, or applications. Log analysis
uses logs of firewall, web server, system, IDS events, or Windows event. Log analysis is an important
aspect to be considered for operational computer security. Log analysis usually works with different
aspects like system logs, application logs, database logs, and so on. Log analysis allows users to notify the
actual changes to the database. Log analysis helps users in recovering or restoring lost or corrupted data.
It is more flexible and has more expertise in restoring data. Restoring data can be done through DB2 logs.
Audit Events
Logs usually note what security breaches or malfunctions may have occurred and ensuresmooth
functioning of the system. Descriptions of notable events are contained in audit logs, such as crashes of
system programs, system resource exhaustion, failed login attempts, etc. These types of events are crucial
for investigation after an attack has occurred. In such incidents, where it is suspected that someone has
attacked the system in some manner, having audit data is very important. Audit data provides
information such as establishment of the network address of the system that was used to initiate an
attack, incident occurrence time, attack type, whether the attack was a success or failure, and possibly
even provide information about an attacker.
The first target of a skilled attacker will be the audit log system. The motive of an attacker includes wiping
out traces of the compromise, evading detection as well as to keep the method of attack a secret to prevent
the security holes that are exploited during intrusion from detection by the system administrator(s).
To make the audit log secure, it is necessary to prevent the attacker from altering the audit log data. To
serve the purpose, auditing comes into the picture. As per definition, auditing is the process of user
activities and operating system activities that are termed as events on a computer. A security log is
maintained that keeps the record of valid and invalid login attempts and events related to creating,
opening, or deleting files or other objects.
An audit policy defines the types of security events that can be recorded in the security log. An audit policy
fulfills two purposes:
Tracks the success and failure events
Eliminates or minimizes the risk of unauthorized access
While planning an audit policy one should determine what to audit on the computer as auditing is turned
off by default. Once it is determined, the next step is implementing the audit policies. The configuration
settings indicate success and failure attempts for each event.
The following are the events that can be audited:
Account logon events User logs onto the domain.
Account management Creation of accounts, modification, renaming, or deletion .
Directory service access An active directory object was contacted. The active directory object
should have audit mode on.
Logon events A user logs on to or off a computer.
Object access An object accessed should have auditing on.
Policy change A user right, security policy, or other policy was altered.
Privilege use A user right, other than access to a computer or log on locally, was exploited.


Log Analysis
Exam 312-38
Process tracking A process initiated.
System events Any event such as shut down, restarting, or security that has occurred.
Log Types
Logs can be classified into three different categories as below:
Content
Source
Format
Content
Content mainly describes information, alerts, and warnings, or fatal errors related to logs. The best
examples of information logs are access logs in Apache and IIS. Alerts, warnings, and fatal errors are
combined together into a single error log or may be further divided into specific types of errors or
sources. In some cases, all log information is combined together into a single file, and its file content helps
you to describe what a particular entry is referring to.
Source
Log source is used as a method of classification. Logs can vary from applications and the system to drivers
and libraries. System logs are generated and handled by the operating system; application logs may be
with the application of system logs that is stored either in a central location or in a temporary location.
Format
Logs can be written either in text or binary format. Logs are usually written in text format because text
format is very east to work with and can be easily understood by both developers and readers. Binary
format is quite a difficult format to read as it uses some form of processing. Binary log information can be
formatted by using specific and structured data types for elements like dates, time, and classification.
When this format is known, it is easy to search the content.


Log Analysis
Exam 312-38
ACTIVITY 20-1
1. What is a log analysis?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. What is an audit event?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. List the events that can be audited.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What are the three different types of log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. What is log content?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. What is a log source?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
7. What is a log format?


Log Analysis
Exam 312-38
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC B
Categorizing Log Files
Log Files
Log files are text files that usually vary in size from 1 KB to 100 MB. Log files are classified into four
different types that are automatically created and dynamically updated by a web server. They are as
follows:
1.
Access_log
2. Agent_log
3. Error_log and
4. Refer_log
Access_log
The Access-log contains a list of individual files like HTML and they are images that users have requested
from a web site. Access logs are used to analyze:
Total number of visitors to a home page
Origin of users with associated servers domain name
Total number of requests for each home page that can be displayed with most requests listed first
Details of accessing home page in terms of time of day, day of week, and week of month
Below is an example line of text from an access_log:

Smx-ca8-50.ix.netcom.com- - [30/Sep/1996:02:57:07 -0400] GET /Proj/main.html
Access_log uses three variables for analysis mentioned below:
1.
Domain name
2. Data and time

3. Item accessed
Domain Name or Internet Protocol (IP) Number

Domain name for the user computer in the above example is smx-ca8-50.ix.netcom.com. In the United
States, an organization type can be specified with the help of six suffixes as shown below:
1.
.com U.S. commercial business (e.g., google.com, aol.com)
2.
.gov U.S. governmental agencies (e.g., whitehouse.gov, epa.gov)
3.
.edu U.S. Educational (colleges, universities) (e.g., syr.edu, bu.edu)
4.
.net U.S. network providers (e.g., internic.net, internet.net)
5.
.org U..S non profit organizations (e.g., madd.org, greenpeace.org)
6.
.mil U.S. military (e.g., army.mil, navy.mil)
Date and Time

Includes the data and time a page was accessed by a user. In the above example, date and time accessed is
on September 30 1996 at 2:57 AM and 07 seconds. The default time will be based on 24 hours.


Log Analysis
Exam 312-38
Item Accessed
Item here refers to an image, movie, sound, or html file. In the above example, main.html is the item
accessed. It is important to note that the full path name (from document root) is given in order to avoid
confusion, (e.g., there can be different main.html on a server).
Analysis of Access_log
Most of the programs like freewareand shareware are available for statistical analysis that concentrates on
analyzing the access_log. Below are examples of statistics being tracked by the access_log.
How many numbers of users are accessing the site from a specific domain type (e.g., com, edu,
net, mil, gov)? This can be broken down into sub-domain. For example, how many hits have
microsoft.com or ibm.com or us.mil or google.com made? It is also important to know how many
hits the server is getting from countries outside the United States.
How many hits the server is getting during peak hours. These statistics will help network
administrators know the optimal time to perform maintenance and/or upgrades.
Determine how many hits every html page is receiving within a site. This statistic can show a
webmeister about the most and least successful parts of a site.
Threading of Users
An understanding of how a user navigates (thread) through a site reveals multiple users.
1.
Entrance
What percentage of users visit the homepage of a site when compared with users visiting other
pages on the same site? For example, if a webmeister (web master) was to put an announcement
only on the homepage, then users who bypass the homepage cannot see the announcement.
2. Exit
Which page(s) has the user visited before exiting the site? Many times this information shows that
a number of users tend to exit the site on a specific page.
3. Clock Analysis
It analyzes the amount of time (minutes or seconds) that a user spends on a specific page. Most
webmeisters would be thrilled if a page within a site that contains a full page of text receives an
average of 800 accesses a week. But, what if the average time a user looks at that page is four
seconds? Did they read the content, or did they use the page as a pathway to another page?
4. Download Time
Downloading time is the amount of time (either in minutes or seconds) that users are spending to
download a page, including graphics. Users access sites with different bandwidth constraints (e.g.,
9.6, 14.4, 28.8, ISDN, Ethernet). A page that takes six seconds to download for one user, may take
six minutes for another.
Agent_Log
Analysis of Agent_log
The three main aspects of agent log analysis are:
1. Browser
What type of browser does a user typically use to access sites? There are over 20 different web
browsers available on the market today which are categorized into two parts, graphical browsers
and textual browsers. Most of the users use graphical browsers like Netscapes Navigator,
Microsoft Internet Explorer, Mozilla Fox, etc. Some users use textual browsers like Lynx, Mosaic,
and so on. Knowing the type of browsers users use can help webmeisters determine how to design
a site.


Log Analysis
Exam 312-38
2. Version
This aspect determines what type of browser version the user is using. Just because a user has a
Netscape browser does not mean that they can view all websites. Some users are still using
Netscape version .96 , which is the original Netscape browser.
3. Operating System
Most of the websites have different users who are using different computers with different
operating systems like Windows, Macintosh, Linux, PowerPC, SunOS etc. To design a website, it
is important to know what type of computer and operating system is being used.
Error_log
This error log determines the type of error encountered at the time of browsing a web page.
Analysis of Error_log
The major error usually found during the web surfing is a File Not Found error. Given below are some
examples of statistics being tracked by the error_log:
1.
Error 404
Bookmarks help users to access their favorite links without searching them again and again on the
website. However, most of the websites URLs are changing day by day. That may lead to a change
in the content, but the URL that is bookmarked by the user has been stored under a different URL
on the server. Thus, the above example will lead the user to get an Error 404. The error_log tells
the webmaster the time, domain name, and page (see descriptions under access_log) that a user
got the error.
2. Stopped Transmission
This is another type of error that occurs during the transmission of a file. Suppose a user is
downloading a file from a web page and that file takes a long time to respond, then the user stops
the downloading. Error_log gives details of occurrence of error like time, domain name and pages
to webmeister. With that information, webmeister can redesign the pages within that website.
3. Cross Reference
How many users stop surfing the website after receiving an error? Most of the time, the
percentage of users receiving an error can be greater than 60%. This percentage can be found by
using the domain name and time of the user who received an error from the error_log and then
look into access log to see whether the domain name shows after the time of the error.
Refer_log
Analysis of Refer_log
The refer_log informs a webmeister about who on the web is linking to their site. Using a simple html tag
(href) any web page developer can link their web page to any other URL on the web. A webmeister should
be able to identify what sites are linking to them and in what context that link is made.
Referral
If a user is on a site (e.g., aps.nri.edu), clicks on a link to another site (e.g., www.sun.com), then sun.com
will receive an entry in their refer_log. The log will show that the above user came to the sun site via
aps.nri.edu (the referral). Webmeister should maintain a database of all referrals for the following
reasons:
Missing link
As stated above (Error 404), many sites whose URLs are available today may be changed
tomorrow. When the URL of a page within www.sun.com changes, the webmeister of sun.com
should notify all referrals (e.g., aps.nri.edu) of the change. If the user clicks the link www.sun.com
from ericir.syr.edu then they will receive a 404 error File Not Found error.


Log Analysis
Exam 312-38
TCPDump Log
TCPDump is a program that is used for monitoring network traffic. The program produces several huge
text files, each having a detailed log of every network packet that is passed. It allows viewing the entire
data portion of an Ethernet frame or other link layer protocol and can optionally print the frame header as
well.
Tcpdump gives the clear picture of specific part of the network that is very useful when something is not
working properly. One can get the information about the nature of an attack using this tool as it helps in
examining the contents of the traffic.
Tcpdump can also help restore functionality after denial of service attacks. It allows you to view the source
address, destination address, and type of traffic involved if a network is flooded and all other attempts to
determine the source or destination of the traffic fail.
Figure 20-1: A TCPDump log.


Log Analysis
Exam 312-38
ACTIVITY 20-2
1. What is a log file?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. What is an Access_log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. What is an Agent_log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What is an Error_log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. What is Refer_log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. What is a TCPDUMP log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC C
Understanding the Concept of Web Server Log
Analysis
Web Server Log Analysis
Web server log analysis plays an important role in the Internet. The web server log is involved in
processing log files on the website, either manually or by using software products. The following are the
statistics about the website visitor that is provided by the log analysis:
Type of browser used by the users to view the website
Details of service provided by the Internet to connect an user, including IP address
Details of website location like their country and region
Number of pages viewed by the user and the amount of time spent on that website
Sample paths or the order of viewing web pages
The following are the web traffic analysis software products that vary from the free to the high cost, high
functionality products for large enterprises:
Analog
Mach5 FastStats Analyzer
WebTrends
Happy Log
Net Merit
ClickTracks
WordTrackers
Analog
Analog is a simple and freely downloaded software product. It is quite difficult to set up and use. Analog
will analyze your web server log files and create HTML, text, or even email reports of your website traffic.
The main objective of Analog is to develop a report on web traffic. Analog does a good job on all the basic
reports, but does not cover more complex reports such as navigation paths, length of time pages viewed,
etc. Analog can be run from a web page or from a command line. This analog program can be directly
installed on virtual servers. However for a small site, which is only getting a minimal number of visitors a
day, it is perfectly adequate.

Similar to Analog, FastStats log analysis software is capable of generating reports on website traffic.
FastStats Analyzer is one of the fastest analysis software that runs at incredible speeds of over 200
megabytes a minute (up to 100 times faster than other log file analysis tools like Web Trends). FastStats
Analyzer is scalable to any website, and produces reports that contain all the detailed information you
need to make your website successful.


Log Analysis
Exam 312-38
Figure 20-2: Mach5 Faststats Analyzer
WebTrends
WebTrends is a simple and easy website traffic tool. It is a de facto standard for medium size sites in
which more than 150 visitors are using this tool per day. It provides a full and comprehensive set of
reports that are even emailable. This tool is also capable of generating reports on website traffic. This
product is designed to run in operating systems like Windows 9x/NT/2000.
Happy Log
Happy Log is a log file analyzer. This tool will examine the log files from your website based on which it
generates a report as lists or graphs, and this report can be viewed as HTML, Word, or Excel files. This
tool will provide details of statistics related to hits and visitors and which browser and operating system
are used. A sample snap shot of this Happy Log is shown below:


Log Analysis
Exam 312-38
Figure 20-3: Happy Log
Net Merit
Net Merit provides hosted services by combining website visitor analytics and also monitors the online
website traffic.
ClickTracks
ClickTracks is a web server log analyzer that displays behavior of the website visitor directly on the pages
of your website. It has a different approach to log file analysis, which targets to make it easier to
understand how visitors are using website navigation. This tool also generates reports in an HTML format
that contain details about the visitors of a website, type of OS, and the browser used and so on.
WordTrackers
WordTracker is a tool to find out the current popular words that are being used on search engines. This is
also uncovers which words are unreliable, as existing sites will have already optimized the more popular
ones. WordTracker may help you find the right words to make your unique selling proposition more
visible if you are in a very rich market.


Log Analysis
Exam 312-38
Figure 20-4: Wordtracker
Apache Logs
Apache log web server will provide log information on user activities and errors. Apache logs store all
non-error messages and error messages in two separate files. The level of messages can be varied; it can
be increased or decreased. These messages can be split into multiple log files using the multiple log files
option. This option is useful when the server has to handle multiple domains.
Default Logs
By default, Apache usually logs general messages to /var/log/httpd/access_log, and it logs error
messages to error_log in the same directory. These are usually set by two directives in the httpd.conf
configuration file, typically located in /etc/httpd/conf directory: TransferLog and ErrorLog. To
change the file path or the name of the log files, just search for these directives and modify them:
TransferLog /data/apache/logs/gen_msg
ErrorLog /data/apache/logs/error_msg
Log Levels
There are several levels to store error messages for web server activities. The level to which Apache logs
are set will determine the type of messages and how much data is logged. The important log levels are:
emerg, alert, crit, crit, error, warn, notice, info, and debug.
Log Format
Through log format, entries are made to Apache logs. These logs are set by the log format directive in the
htttpd.conf file. By default, access log entries are set as shown below:
12.127.17.72 - russell [21/Jun/2004:11:50:20 -0500]
"GET /cgi-bin/admin/index.cgi HTTP/1.1" 200 5528


Log Analysis
Exam 312-38
Rotating Logs
Apache logs usually change. By default, logs are rotated weekly: access_log is copied to
access_log.1, access_log.1 to access_log.2, and so forth, but executed in reverse, of course.
If you create your own logs, then you will need to set up a log rotation. This can be done in different ways.
The key, though, is to do this without loss of data while rotating logs and without shutting down Apache
for long. A moderately clean scenario is to write a simple shell script for cron to execute regularly:
#!/bin/bash
mv -f gen_msg.2 gen_msg.3
mv -f gen_msg.1 gen_msg.2
mv -f gen_msg gen_msg.1
apachectl graceful
The first line in the above command tells cron that it is a bash script. The next three lines are used to
rename or move the old log files to their new file names. The fifth line states that Apache successfully
restarts so that a new log file (gen_msg) is created without any current connections being lost or
unrecorded.
IIS Logs
A critical function to the secure administration of a web server is to capture and maintain log files.
Generally, the log does not capture an intrusion until the request has been processed; however, a diligent
administrator might couple logging with tools such as a URLScan that can supplement and strengthen
security via the logging process. Here some of the best practices that can be followed are discussed . The
best way to emphasize the value and importance of IIS log files is to remember that they can be used as
evidence in a crime scene. In fact, IIS logs must be treated as if they are the evidence already, in case
they are needed for the future prosecution of, say, a hacker. Combining IIS logs with other monitoring
records like firewall logs, IDS logs, and even TCPDump can provide more reliability if the log is needed for
evidence. All visits to a web server are recorded in the in log files, which are located at:
<%systemroot%>\logfiles
If proxies are not used, then IP can be logged.
This command lists the log files:
http://victim.com/scripts/..%c0%af../..%c0%af../..%c0%af../..%c0%af../..%c0%a
f../..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+C:\Winnt\s
ystem32\Logfiles\W3SVC1
The following rules need to be followed regarding IIS logs:
The first rule: Configure the IIS logs to record each and every available field. Gathering
information about web visitors may establish a source of an attack through a system or a user.
The more information collected, the better the chance of tracing the intruder.
The second rule: Store events with a proper time stamp as IIS records logs use UTC time. The
accuracy of the UTC time can be ensured only if the local time zone setting is correct.
Third rule: Ensure continuity in the logs. An IIS log registers a log entry only if the server gets a
hit in a 24-hour period. This makes an empty log file unclear: whether the server was offline, was
deleted, or had no hits in a particular day. The easiest workaround is to use the Task Scheduler
and schedule hits. In general, scheduled requests can indicate that the logging mechanism is
functioning properly. Therefore, if a log file is missing, it is probably because the file was
intentionally deleted.
The fourth rule: Ensure that logs are not modified in any way after they have been originally
recorded. Once a log file is created, it is important to prevent the file from being accessed in order
to audit any authorized and/or unauthorized access. This can be achieved by moving IIS logs


Log Analysis
Exam 312-38
away from the web server. File signatures are useful because if a single file is corrupted, then it
does not invalidate the rest of the logs. Also, when doing any log file analysis, copies of the files
must be used because the original files must be preserved in their original state. After a log is
closed, no one should have permission to modify its content.
IISLogger
IISLogger is an addition to the standard Internet Information Server log. It produces additional log data
and forwards it over Syslog. It even logs the data when the web request was aborted and not completely
processed by IIS.
IISLogger is an ISAPI filter. It is a Dynamic Link Library (.dll) embedded in the IIS environment. Even if
the IIS calls an ISAPI filter notification, IISLogger prepares header information and logs this information
to Syslog in a certain format. This occurs each time, for each notification IISLogger is configured for.
Being an ISAPI filter, IISLogger starts automatically with IIS.
The following are the features of IISLogger:
Generates additional log information from IIS
Recognizes hacker attacks
Forwards IIS log data to Syslog
Uses graphical user interface to configure IISLogger
Figure 20-5: IIS Logger
Limitations of Log File Analysis

Log file analysis does not provide a clear idea and reliable information about the number of users of a
website, either their geographic distribution or sector representation.
In general, log file analysis provides an overestimation of the number of users, because of the
difficulty of finding search engines and other intelligent agents that index your website.
As web pages are cached on intermediary servers, you cannot know the exact number of users
accessing the website. The log file analysis provides only the number of users accessing the
website directly.
Most traffic (over 65 percent) may not be identified by your Internet service provider due to the
increase in firewalls and other security measures. Due to this, you cannot come to a conclusion on
which host is accessing your site, say, by type (.com, .edu, .org, etc.) or by geographical domain.


Log Analysis
Exam 312-38
Site statistics should be carefully used for corporate promotional purposes, and always with the caution
that it is difficult to report levels of traffic, geographic, and sectoral reach with any definitive degree of
accuracy.


Log Analysis
Exam 312-38
ACTIVITY 20-3
1. What is a web server log analysis?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. What is an analog?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. What is a Faststats Analyzer?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What is a WebTrends?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. What is a Happy Log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. Happy Log is a log file Analyzer (True/False) _______?
_____________________________________________________________________________
7. What is a ClickTracks?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
8. What is a WordTracker?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
9. What is an Apache Log?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
10. What is an IIS Logger?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
11. What are the limitations of log file analysis?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC D
Identifying Syslog Statistics and Analysis
System Log Aggregation, Statistics and Analysis
Introduction to Syslog
Syslog is a combined audit mechanism employed for LINUX operating systems and application messages.
Syslog allows you to log significant system information to a remote server. The mechanism is designed to
abridge integration with logging for system analysts and applications developers. It permits the system
administrators to collect and distribute audit data with a single point of management. Syslog permits both
local and remote log collection. Syslog is controlled on a per machine basis by the file
/etc/syslog.conf. The syslog behavior includes how data from different applications and subsystems
on the host operating system, and of different levels of severity is stored or distributed.
This configuration file consists of multiple lines like the following:
mail.info <Tab><Tab> /var/log/maillog
The format of configuration lines is:
facility.level <Tab><Tab> action
The <Tab> key is used to define white space between the selector on the left side of the line and the action
on the right side.
The facility is the operating system component or application that generates a log message and the level is
the severity of the message that has been generated. The action gives the definition of what is done with
the message that matches the facility and level. The system administrator can customize messages based
on which part of the system is generating data and severity of the data using the facility and level
combination.
The big advantage of syslog is that all the reported messages are collected in the message file. To log all
messages to a file, selector and action fields need to be replaced by a wildcard character *.
*.* /var/log/syslog
Logging priorities can be enabled by configuring /var/log/syslog. All authorized messages can be
logged with a priority such as emerg (highest), alert, crit, err, warning, notice, info, or debug (lowest).
Events such as bad login attempts and the users last login date are also recorded. If an attacker logs into a
Linux server as root using the secure shell service and guessed a password, the attackers login
information will be saved in a syslog file.
But the attacker could delete or modify the /var/log/syslog message file, wiping out the evidence. To
avoid this problem, set up remote logging.
Building a Central Loghost

During initial setup of a new machine, syslog messages are locally stored. Automatically routing your log
files to a centralized location as your enterprise grows can provide the following benefits:
Easier to analyze what may have happened (normal behavior versus curious event).
Difficult for infiltrator to corrupt or alter relocated logs. Less likely that a successful infiltrator
could corrupt or alter relocated logs.
Simplifies the archival of collected logs off-line to removable media, or even a line printer.
A logging host is also known as a loghost. Loghost is a machine with more disk storage dedicated to a
single purpose of receiving these log messages. A logging host should ideally be hardened with all external


Log Analysis
Exam 312-38
services disabled except syslog and be accessed only directly from the console for administration. Normal
users should not have accounts on this machine.
Parsing and Normalizing

Log Parsing
Log parsing is a process of gathering information from a log so that the parsed value can be used as an
input for another logging process. Example: reading a text-based log file that contains say, 10 commaseparated values per line and extracting the 10 values from each line. Parsing is done as part of many
other logging functions like log conversion and log viewing.
Log Normalizing
In log normalization, each log data field is modified to a particular data representation and constantly
categorized. The most common use of normalization is storing dates and times in a single format. For
example, one log generator might store the event time in a twelve-hour format (2:34:56 P.M. EDT)
categorized as Timestamp, while another log generator might store it in twenty-four (14:34) format
categorized as Event Time, with the time zone stored in a different notation (-0400) in a different field
categorized as Time Zone. Normalizing the data makes analysis and reporting much easier when using
multiple log formats. However, normalization can be very resource-intensive, especially for complex log
entries (e.g., typical intrusion detection logs).
Bayesian Spam Filters for Logging

Bayesian spam filters calculate the occurrence of a message being spam based on its contents. These
filters are different from simple content-based filters.
Bayesian spam filtering is an efficient approach to distinguish spam from good mail. Bayesian spam filters
are very robust in nature. It is one of the best approaches to finding spam and returns less false positives.
The percentage of finding spam mail with good mail is around zero.
Storage and Rotation

Log Storage
System-level administrators should be able to identify how each log source stores its data. This should be
motivated by the organizational policies related to log storage and requirements to provide entries to log
management. Once these requirements are met, administrators can change other log storage settings. The
following are the storage options for log entries:
Not Stored
Entries that are identified as small or having no value to the organization, debugging messages
that can only be understood by the software vendor, or error messages that do not log any details
of the activity, generally do not need to be stored.
System Level Only

These entries might have some value or interest to a system-level administrator, but are not
important enough to be sent to log management infrastructure, and are to be stored on the
system. For example, if an incident occurs then additional system-level log entries provide more
information on the series of events related to that incident.
System-level administrators might also find it helpful to review these entries to develop baselines
of typical activity and identify long-term trends.
Both System Level and Infrastructure Level

Entries judged to be of particular interest should be saved on the system and also transmitted to
the log management infrastructure. Reasons for having the logs in both locations include the
following:


Log Analysis
Exam 312-38
If either the system or infrastructure logging should fail, the other should still have the log
data. For example, if a log server fails or a network failure prevents logging hosts from
contacting it, logging to the system helps to ensure that the log data is not lost.
During the progress of an incident, there is a chance that the system logs can be altered by the
attacker. Generally, the infrastructure logs cannot be altered by the attacker. The incident
response team can check the infrastructure and system logs to find if they were altered or
deleted, which may reveal the data that the attacker attempted to hide.
System or security administrators for a particular system are often responsible for analyzing
its logs, but not for analyzing its log data on infrastructure log servers. Accordingly, the
system logs need to contain all data of interest to the system-level administrators.
Infrastructure Level Only

If logs are stored on infrastructure servers it is also preferable to store at the system level. This
may not always be possible because systems with less capacity for logs or log sources may not be
capable of storing logs. For example: applications that can only log to remote logging server.
Log Rotation
Local log rotation is one of the critical steps in configuring log sources. Log sources are to be configured to
perform log rotation when certain conditions are met such as at regular time intervals, when a log size
limit exceeds. If the log files are not capable of rotating logs, the system-level, and infrastructure
administrators have to deploy a third-party log rotation tool or utility. In certain cases where the thirdparty tools are not supported by the log sources, the administrator can choose from the following:
Stop Logging
This is an unacceptable option because logging permits operations to continue without allowing
monitoring of security events.
Overwrite the Oldest Entries

This is an acceptable entry for lower-priority log sources, when the significant log entries have
already been transmitted to a log server or archived to offline storage. This is one of the best
methods for logs that are very difficult to rotate.
Stop the Log Generator

When logging is critical, it may be necessary to configure the OS, security software, or application
generating the logs to shut down when there is no space left for more log entries. On such
systems, administrators should take reasonable measures to ensure that log generators have
adequate space for their logs and that log usage is monitored closely.
Databases and Logs

Databases are used to write severe error and warning conditions to the local error log (syslog). Typically,
databases also write errors to a database log file, such as the db2diag.log file on DB2.
You can open the db2diag.log file in a text editor.
The example below shows a typical entry from the db2diag.log file:
Jun 18 15:02:53
sqlobeep(2)
bluj
DB2[46827]:
DB2(db2inst1.000(1))oper_system_services
reports:(3)
Jun 18 15:02:53 (4) bluj (5) DB2[46827(6)]: extra symptom string provided:(7)
RIDS/sqlesysc_
Jun 18 15:02:53 bluj DB2[46827]:
78616d70
data: (8)
54686973
20697320
616e2065


Log Analysis
Exam 312-38
Jun 18 15:02:53 bluj DB2[46827]:

20646174
Jun 18 15:02:53 bluj DB2[46827]:
data:
data:
6c65206f
66206c6f
67676564
61
Jun 18 15:02:53 bluj DB2[46827]: 2 piece(s)

file(9) /u/db2inst1/
of dump data provided... to
Jun 18 15:02:53 bluj DB2[46827]:

(10)
1. 'DUMP EXAMPLE #1' has been dumped
Jun 18 15:02:53 bluj DB2[46827]:
2. 'DUMP EXAMPLE #2' has been dumped
The bold numbers in the example show various options in the log file:
1.
The instance name and node number
2. The reporting component and function

3. The probe ID and error and alert numbers
4. A time stamp for when the event occurred
5.
The host name
6. The process ID of the reporting process. Uses ps command to view information about the process
ID of the reporting process. For example, enter the following command to get information about
the reporting process:
ps -fu 46827
7.
A symptom string that contains additional information about where and why the problem
occurred
8. A hexadecimal dump of data that includes return codes and other information that can be
interpreted by your IBM Support Center
9. Information about additional dump files. Larger structures and other binary data might be
discarded to additional files. The name of the file is identified in the syslog file
10. An entry to identify a piece of dump data


Log Analysis
Exam 312-38
ACTIVITY 20-4
1. What is a syslog?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. What is a log host?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. What is log parsing?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What is log normalization?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. How do you to store logs?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. What is log rotation?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC E
Understanding Concept of Logging
Overview of Logging
Secure Audit Logs
There are three identical types of properties of a secure audit log that are designed to prevent and detect
tampering, control data and search access. They are:
1.
Tamper Resistance
2. Verifiability
3. Data Access Control and Search ability
Tamper Resistance
The secure audit logs must be tamper resistant. The creator of the log can only create the valid entries and
once entries are created it cannot be altered. An attacke can be prevented by deleting log entries that are
not copied or transferred to another system. The objective of a secure audit log for such conditions is to
make sure that the attacker cannot modify existing logs; if any attempts are made to delete the existing
entries then it can be detected. There is more possibility that one would like to detect the attempts that are
made to delete or modify entries created up to the time a host is compromised. There are some
applications for which it is enough to have a logging host checkpoint that states periodically for copying
the log data, or some functions of its log data to another host to ensure that the entries that are made up
to the checkpoint are not deleted or modified.
Verifiability
A secure audit log must be verifiable as it is necessary to check whether entries in the log are present and
are not modified. Audit logs can be:
Public verifiable
Trusted verifier
Public verifiable
Audit logs can be verified by anyone who is holding appropriate authentic public information like logging
system public key or an authenticated hash of all the existing audit entries.
Trusted Verifier
These are audit logs that can be verified by the designated party holding one or more secrets, like a MAC
key. Selection of approach is application dependent.
Systems that sign the log entry digitally just after creating make the storage of audit logs easy on untrusted systems. The trusted verifier system, like Schneier and Kelsey scheme, allows a greater degree of
forward security in relation to audit logs. This makes it possible to detect the attempts made to delete the
log entries made before the system is compromised, without the communication of those entries to the
other systems.
Data Access Control and Search Ability

Data in the audit log must be encrypted as it is sensitive. An authenticated search access to a subset of all
audit log entries can be allowed. Delegation of capabilities is to be made as it allows an investigator to
search and view the entries of the narrow escape. For example, if Alice Smith wants to inquire about and
investigate all the entries related to the audit escrow which matches the word smith and nothing else.


Log Analysis
Exam 312-38
The alternative to this is to have the master secret holder do the search as it is unnecessary to expose a
highly trusted component of the system.
Setting up Remote Logging

Remote Logging in Windows
The log data can be written to a remote share in the network by using the full Universal Naming
Convention (UNC) path for the centralized log file storage and backup. The remote login feature can
negatively affect the performance because IIS writes the log file and the data of the log file over the
network.
Before configuring remote logging, Internet Protocol Security (IPSec) should be enabled between the
server running IIS and the remote server. If IPSec is not enabled then the web server running IIS and the
remote server, the data packets containing log data, can be intercepted by malicious individuals and wire
sniffing applications when the data packets travel in the network.
As long as the remote computer allows the IIS to store the log files on a remote share, IIS creates a log file
and writes the data to the remote share. With the help of the following procedure, a remote machine can
be configured so that IIS can create a log file and write the data to a remote share.
Procedures to configure permissions for remote logging:
Step No.
Procedure
Step 1
On the remote computer, navigate to systemroot\System32, right-click the

LogFiles folder, and then click Sharing and Security.
Step 2
On the Sharing tab, click Share this folder, and then click Permissions.
Step 3
Click Add.
Step 4
Click Object Types.
Step 5
Select the Computers check box, and click OK. If you like, you can deselect all
other options.
Step 6
In the Enter the object name to select box, type the object name in the form
Domain\WebServer, and then click OK.
Step 7
In the Group or user names list, select the Domain\WebServer object, and in
the Permissions section, select the Allow check box next to Full Control.
Step 8
In the Group or user names list, select Everyone.
Step 9
In the Permissions section, clear all permissions and click OK. The remote
computer now has the appropriate access permissions.
Step 10
To set the appropriate file permissions, click the Security tab.
Step 11
Select the Domain\WebServer object and in the Permissions section, select the
Allow check box next to Full Control.
Step 12
Click Apply, and then click OK.
Linux Process Tracking

Process tracking is the audit mechanism for Linux operating systems. It tracks the process execution and
logon/off events. It tracks each command that users execute. The process tracking file can be found at
/var/adm, /var/log or /usr/adm. The process accounting mechanism can be enabled by accton command.
Process accounting during audit track logs all the messages in its own binary format to the
/var/log/psacct. The tracked files can be viewed using the lastcomm command. The lastcomm gives
information about previously executed commands.
The following lines show the output after the execution of the lastcomm command:


Log Analysis
Exam 312-38
[root@server log]# lastcomm

clear
man
sh
sh
less
root
stdout
0.01 secs Thu Nov 14 07:20
root
stdout
0.00 secs Thu Nov 14 07:19
root
stdout
0.01 secs Thu Nov 14 07:19
root
stdout
0.00 secs Thu Nov 14 07:19
root
stdout
0.00 secs Thu Nov 14 07:19
crond
root
0.00 secs Thu Nov 14 07:20
mrtg
root
1.02 secs Thu Nov 14 07:20
crond
root
??
0.00 secs Thu Nov 14 07:20
sadc
root
??
0.02 secs Thu Nov 14 07:20
In this output, the first row stands for the processes executed each followed by a flag. The S flag stands
for the superuser (root) and the F flag stands for the forked process. Each process contains the following
information:
How the process was executed.
Who executed the process.
When the process was ended.
Which terminal type was used.
Limitation of process tracking:
It audits the information after the execution of the process.
It audits only the execution of commands.
Windows Logging
Logging is important when attempts to decipher information has taken place on server. Logging is mostly
used to rectify the events that are committed and that are not preempted. It is a complex job to monitor
and view every log on a vast network. A filter can be applied to take only the pertinent information that is
mature enough to reveal what is happening on network. The Event Log service starts automatically at
Windows startup. Application and system logs can be viewed by all users. Security logs access is available
only to administrators. The security logs are turned off by default. To make the security log available, it
should be turned on by the administrator.
There are many logs in Windows that can be monitored. The security log is important as it tracks and
records all the events in the network. The different log types in Windows logging are:
Application logs are the events logged by the applications.
Security logs contain the record of logon attempts and the events related to the use of resources
like creating, opening, and deleting files and objects.
System logs contains system component events like the hardware and driver failure issues.
File replication service logs the windows file replication service events, like Sysvol changes are
logged in the file replication log.
DNS machines stores DNS events in logs.
Logging in Windows Loghosts

NTsyslog


Log Analysis
Exam 312-38
This program runs as a service under Windows NT-based operating systems. It formats all system,
security, and application events into a single line and sends them to a syslog(3) host.
Synopsis:
NTsyslog [ -install] [ -remove]
Options
-install: installs the service
-remove: removes the service
How to install:
Install the service by executing the following command:
NTsyslog install
The service control manager automatically starts this service during system startup. A user can start and
stop the service manually from the Services Control Panel.
By default the service runs under the Local System account. It can be configured to run as a local user with
the following rights:
Log on as a service
Manage auditing and security log
NTSyslogCtrl is used to configure the types of messages that are to be monitored and the priority to use
for each type. Event Log types priority controls the service and facility that the Syslog message is sent to.
The priority for each event log type controls the service and facility that the syslog message is sent to. Each
log type has a seperate priority. Default is 9, user.alert when the priority does not exist, or at the time of
upgrade, or using an old NTSyslogCtrl app.
Syslog refers to a "facility" and "severity" referres to a single value called priority.
To calculate the priorities from normal facility and severity codes:
Take the numeric value for the facility, multiply by 8, and add the numeric value for the severity.
Standard facility and severity values are:
Facility:
(0) kernel
(12) ntp
(1) user
(13) log audit
(2) mail
(14) log alert
(3) system
(15) clock 2
(4) security/auth 1
(16) local 0
(5) syslog
(17) local 1
(6) line printer
(18) local 2
(7) news
(19) local 3
(8) uucp
(20) local 4
(9) clock 1
(21) local 5
(10) security/auth 2
(22) local 6
(11) ftp
(23) local 7
Severity:
(0) emergency
(4) warning

Log Analysis
Exam 312-38
(1) alert
(5) notice
(2) critical
(6) information
(3) error
(7) debug
Application Logging
The Enterprise Library Logging Application Block simplifies the implementation of common logging
functions. Developers can use the Logging Block to write information to a variety of locations:
The event log
An e-mail message
A database
A message queue
A text file
A WMI event
Custom locations using application block extension points
A consistent interface is provided by the application block for logging the information to any destination.
Information about destination is not specified in the application code. The configuration setting
determines that the application block will write the logging information and location for that information.
Even operator and developer can alter the logging behavior of an application without changing the
application code.
The logging application block helps with application development in different ways:
Consistent logging practices are maintained within an application and across the enterprise.
An implementation which can be used to solve common application logging task.
The code written by the developer traces the flow of control in application with the help of components
involved in the execution of application. Applications need to write information, either locally or over the
network. Events can also be collated from multiple sources into a centralized location.
Application development is simplified by the logging application block andby providing a small set of
ease-to-use classes and methods that encapsulate most of the logging scenarios. These scenarios include:
Populating and logging event information.
Including context information in the event.
Tracing application activities.
Extended Logging
Firewall Logging
The event on firewall falls into three broad categories:
1.
Critical system issues
2. Significant authorized administrative events

3. Network connection logs
Generally following events are captured:
Host operating system logs
Changes to network interface
Changes to firewall policies


Log Analysis
Exam 312-38
Adds/deletes/changes of administrative accounts
System compromises
Network connection logs
Reviewing Firewall Logs with grep command

grep allows you to search one or more files looking for a specific pattern
Syntax of grep command is:
grep
[search string][filename]
example:- grep share firewall.log

To match all the case variation of the string share, use the switch -i
Grep i share firewall.log
To match all the lines that do not contain the string share, use -v switch
grep v sharefirewall.log
Organizing Firewall logs

Traffic is processed in two passes, first by creating a file that contains the data that matches the pattern,
and second the data not matching pattern. If the IP address 12.2.21.10 is being used by webserver
Grep12.2.21.10/80 firewall.log>web-server1.txt
Grep v 12.2.21.10/80 firewall.log > temp-file1.txt
The first command will grab all the traffic going from port 80 on web server and records it in file web server1.txt,
In the second command, all log entries are matched and matched log entries are written to the temporary
file called temp-file1.txt. The next thing to track is SMTP traffic to the mail server. To bin this traffic, use
a similar format:
grep 12.2.21.11/25' temp-file1.txt > inbound-smtp.txt
grep -v '12.2.21.11/25' temp-file1.txt > temp-file2.txt


Log Analysis
Exam 312-38
ACTIVITY 20-5
1. UNC stand for _____________?
a. Universal Network Communication
b. Universal Naming Convention
c. Universal Network Control
d. Universal Network Channel
______________________________________________________________________________
2. What does 'S' stand for in Linux process tracking?
a. Server
b. Super user
c. Standard
d. System
_____________________________________________________________________________
3. Syntax of grep command is:
a. grep [Filename 1][filename2]
b. grep [search string][string]
c. grep [search string][filename]
d. grep [inpt file][output file]
_____________________________________________________________________________
4. What are the different types of logs in Window logging?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. What is verifiability?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. Write the steps to configure permissions for remote logging.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC F
Introduction to Monitoring and Security Events
Monitoring for Intrusion and Security Event
Both passive and active tasks are included in monitoring for intrusion and security events. The log files
contains all the events, many of the intrusions are identified after the attack has taken place by viewing
the log file. The post-attack detection is referred to as the passive intrusion detection. Log file is the only
thing that can be reviewed to get the information of the strategy of attack.
Other intrusion attempts can be detected as the attack is fired. This methodology is known as Active
Intrusion detection, which looks for known attack patterns or commands and blocks the execution of
commands.
Importance of Time Synchronization

When intrusion monitoring is done and security events between multiple computers have been activated,
clocks should be synchronized. An administrator can easily rectify what has happened during the attack if
the clocks are synchronized. In the absence of synchronization of time, it is difficult to identify what
events have taken place and how those events are related. All the servers should use the same time source.
The Windows 2000 W32Time service provides time synchronization for Windows 2000 and Windows
XPbased computers running in an Active Directory domain.
The W32Time service ensures that the client clocks of Windows 2000based computers are synchronized
with the domain controllers in a domain. The Kerberos version 5 authentication protocols require this
type of synchronization. Time synchronization also assists in event log analysis.
Passive Detection Methods

Event logs and application logs are reviewed manually in a passive intrusion detection system. Analysis
and detection of attack strategy are done on event log data at the time of inspection. For reviewing the
logs, several tools and utilities are available.
EventCombMT
EventCombMT is a multithreaded tool that will parse event logs from many servers at the same time,
spawning a separate thread of execution for each server that is included in the search criteria.
EventCombMT is included with Microsoft Windows Server 2003 Resource Kit Tools.
The tool allows you to:
Define either a single event ID, or multiple event IDs to search for. You can include a single event
ID, or multiple event IDs separated by spaces.
Define a range of event IDs to search for. The endpoints are inclusive. For example, if you want to
search for all events between and including event ID 528 and event ID 540, you would define the
range as 528 > ID < 540.
Limit the search to specific event logs. You can choose to search the system, application, and
security logs. If executed locally at a domain controller, you can also choose to search FRS, DNS,
and Active Directory logs.
Limit the search to specific event message types. You can choose to limit the search to error,
informational, warning, success audit, failure audit, or success events.
Limit the search to specific event sources. You can choose to limit the search to events from a
specific event source.


Log Analysis
Exam 312-38
Search for specific text within an event description. With each event, you can search for specific
text. This capability is useful if you are trying to track specific users or groups.
You cannot include search logic, such as AND, OR, or NOT in the specific text. In addition, do not
delimit text with quotes.
Define specific time intervals to scan back from the current date and time. This approach allows
you to limit your search to events in the past week, day, or month.
Event Collection
The actions or strategies followed by the attack on the network are the main goal of auditing. An attacker
can attempt to compromise multiple computers and network devices. This is essential to understanding
the extent of an attack. If the time is synchronized in all the computers, sorting on time fields, and
tracking of events that are based on time intervals are easy.
Scripting
Scripts are written that can collect the event log information from the remote location and store it in the
central location. Execution of the scripts can be scheduled and corresponding actions are to be taken after
the event log is copied to the central location.
An example in which a batch file is created that uses Dumpel.exe from the Windows 2000 Server
Resource Kit. Launch that batch at regular intervals by using the scheduled tasks in the Control Panel.
The Windows 2000 Resource Kit, Supplement One includes Eventquery.pl. This file is a Perl script
that displays events from the event viewer logs on local and remote computers running Windows 2000
and offers a wide range of filters to help you find specific events.


Log Analysis
Exam 312-38
ACTIVITY 20-6
1. What is the importance of time synchronization in networks?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. Explain the tool available for parsing event logs from many servers.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. Explain passive detection methods.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What is an EventCombMT?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. What is an event collection?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. Explain Scripting.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC G
Understanding Log Analysis Tool
Log Analysis Tool
Userlock
The Userlock tool is used to secure the access to Windows networks by limiting and restricting
simultaneous sessions.
This restriction is imposed by limiting user access to the network and by providing administrators with
remote session control, alert options, and advanced reporting for session analysis. The features of
Userlock are:
Simultaneous session prevention and restriction
Workstation restriction
Alerts and notifications
Remote session management
Connectivity surveillance and monitoring
Analysis and reporting
Flexibility and ease of use and security
Centralized administration
Management via web interface
Pin-pointed protection
Figure 20-5: Administrative console of user lock



Log Analysis
Exam 312-38
WSTool
WSTOOL is an OS-independent web vulnerable scanner. The WSTool includes:
SQL injection For SQL server
XSS Cross Site Scripting
404/500 server error
Admin/manage folder search
Web-based or command-line scanner by PHP
Check up collate with HTML FORM and URL-Link
Auditing Tool
ASDIC
ASDIC is a system for advanced traffic analysis. It helps to determine the type of traffic in the network.
The functions of ASDIC are as follows:
Collects traffic data from various sources like routers, switches, real time flow monitors and
firewalls.
Converts different kinds of data into a comparable and consistent format.
Stores network sessions in a searchable central database.
Analyses network flows.
Creates all significant aggregate flows.
Detects anomalies and deviations and reports events of interest.
Figure 20-6: Main window of ASDIC
Tenshi
Tenshi is a specially designed program to watch multiple log files to match the user-defined regular
expression and report it. Regular expressions are used for the alert interval and a list of mail recipients.
Queues can be configured to send notification for the log line assigned or to send periodic reports. The
program reads a configuration file and then forks a deamon for monitoring the specified log files.
Additionally, uninteresting fields in the log lines (such as PID numbers) can be masked with the standard
regular expression grouping operators ( ). This allows for a cleaner and more readable report.


Log Analysis
Exam 312-38
Consider the setting in tenshi.cnf:

...
set hidepid on
set queue mail
tenshi@localhost sysadmin@localhost [0 */12 * * *]
set queue misc
tenshi@localhost sysadmin@localhost [0 */24 * * *]
set queue critical tenshi@localhost sysadmin@localhost [now]

group îpop3d:
mail îpop3d: Login user=(.+)
mail îpop3d: Logout user=(.+)
mail îpop3d: pop3s SSL service init from (.+)
mail îpop3d: pop3 service init from (.+)
mail îpop3d: Command stream end of file, while reading.+
mail îpop3d: Command stream end of file while reading.+
critical îpop3d: Login failed.+
trash îpop3d:.+
group_end
critical ^sudo: (.+) : TTY=(.+) ; PWD=(.+) ; USER=root ; COMMAND=(.+)
misc .*
SpoofMAC
SMAC is a powerful, easy to use, and intuitive MAC address changing (spoofing) tool for Windows system.
It Includes Network Adapter Restart, IPConfig, and Reporting.
Features of SpoofMAC are as follows:
Automatically restarts network adapter
Changes MAC addresses quickly and easily
Removes spoof MAC addresses completely
Generates new random MAC addresses
Generates new MAC addresses by network adapter vendor
IPConfig tool to view network adapter information
Views all network adapters or active network adapters easily
Remembers most recently used (MRU) MAC addresses
Automatic MAC address change logging (optional in Professional Edition)
Manages MAC address lists for easy MAC address selection (Professional Edition)
Generates comprehensive network adapter detail reports (Professional Edition)
Displays detailed configuration information of:

o
Device ID
NIC description
NIC manufacturer


Log Analysis
Network connection
Hardware ID
Configuration ID
Active status
Spoofed status
IP address
Subnet mask
Default gateway
Current MAC address
Spoof MAC address
Exam 312-38
Network adapter vender lookup and selection
Figure 20-7: SpoofMAC main window.


Log Analysis
Exam 312-38
Gentle MAC PRO

Gentle MAC pro is a network utility. By using the features of MAC pro you can change the MAC address
and IP simultaneously. The Gentle MAC pro lets you manage the network automatically and manually by
changing the IP and MAC address.
Features of Gentle MAC pro include:
It changes MAC address and IP address within two to three seconds. The program assumes
control of changing MAC addresses and restarting network devices.
Capable to change MAC address of any network adapter including WiFi-cards.
Changes MAC address without reboot.
You can change IP and MAC addresses separately or simultaneously.
You can manage your devices via command line and scripts.
It assumes tcontrol over set tasks, making work with network devices most transparent, helping
you to be more confident and effective.
Figure 20-8: Gentle MAC Pro


Log Analysis
Exam 312-38
ACTIVITY 20-7
1. Mention some of the features of userlock.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. What is the use of WStool?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. What are the functions of ASDIC?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What is display detailed configuration information of SpoofMAC?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. List some of the features of Gentle MAC pro.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC H
Various Log Parsing Tool
Generic Log Parsing Tool
LogSentry
LogSentry (formerly Logcheck) is designed to automatically run and check system log files for security
violations and unusual activities.
It uses a program called logtail that remembers the last position it read from in a log file and uses this
position on subsequent runs to process the new information.
SL2
SL2 tool scans log file for anomalies. This script will scan the directory where your log files reside and
report everything that it finds with the exception of those expressions found in the ignore file,
scanlog.ignore. For more information go to this link: http://www.ip-solutions.net/syslog-ng/sl2
Flog
Flog is a simple ftpd log file analysis tool. It generates basic statistics about traffic and server utilization,
which it outputs to a file. For more information go to this link: http://www.securityfocus.com/tools/400
Simple Log Clustering Tool (SLCT)

SLCT is a tool that was designed to find clusters in logfile(s), so that each cluster corresponds to a certain
line pattern that occurs frequently enough. Here are some examples of the clusters that SLCT is able to
detect:
Dec 18 * myhost.mydomain sshd[*]: log: Connection from * port *
Dec 18 * myhost.mydomain sshd[*]: log: Password authentication for * accepted.
Some of the options used with SLCT are:
-b <byte offset>
-c <clustertable size>
-d <regexp>
-f <regexp>
-g <slice size>
-i <seed>
-o <outliers file>
-r
-t <template>
Xlogmaster
XLogmaster is a program for overall monitoring of the system. It aids in reading log files, status
information, and translation of data, applying filters, highlighting, and hiding or taking prescribed actions
on user-defined events.
GeekTool (mac O.S)



Log Analysis
Exam 312-38
GeekTool is a preference pane to show the desired log files on the desktop background. It can also be used
to show output of shell commands.
Dumpel
Dumpel is a command-line tool that dumps event logs for local or remote systems in a tab-separated text
file. It can be used to filter for or filter out certain event types.
Syntax of Dumpel:
dumpel -f file [-s \\server] [-l log [-m source]] [-e n1 n2 n3...] [-r] [-t]
[-d x]
Where:
-f file: Specifies the file name for the output file. There is no default for -f, so you must specify the
file.
-s server: Specifies the server for which you want to dump the event log. Leading backslashes on the
server name are optional.
-l log: Specifies which log (system, application, security) to dump. If an invalid logname is specified,
the application log is dumped.
-m source: Specifies in which source (such as rdr, serial, and so on) to dump records. Only one source
can be supplied. If this switch is not used, all the events are dumped. If a source is used that is not
registered in the registry, the application log is searched for records of this type.
-e n1 n2 n3: Filters for event id nn (up to 10 can be specified). If the -r switch is not used, only
records of these types are dumped; if -r is used, all records except records of these types are dumped. If
this switch is not used, all events from the specified sourcename are selected. You cannot use this switch
without the -m switch.
-r: Specifies whether to filter for specific sources or records, or to filter them out.
-t: Specifies that individual strings are separated by tabs. If -t is not used, strings are separated by
spaces.
-d x: Dumps events for the past x days.
Figure 20-9: Execution of dumpel command

To dump the local application log to a file named Event.out, and get all events except ones from the
Garbase source, use:


Log Analysis
Exam 312-38
dumpel -f event1.out -l application -m garbase -r
Figure 20-10: Log captured by dumpel command.
Watchlog
Watchlog is a Perl program designed to give you a better real-time view of your web traffic. Simply doing a
'tail -f' on the server log file often yields confusing results as you can be bombarded with scrolling with a
single hit. Watchlog presents the same information in a formatted way, a real-time view of the events on
the website by tracing logfile and showcasing only the relevant data.
LogDog
LogDog is used to monitor the messages that pass through syslogd and trigger an action based on
keywords and phrases. A configuration file in LogDog gives specification for alerts on keywords or
phrases. It also specifies a list of commands that are triggered when these words are encountered.
Features of LogDog include:
Monitors syslogd messages for key words and phrases and runs system commands based on the
content.
Logs all activity to a file.
HUP signal is intercepted correctly and causes Logdog to reload its configuration and refresh all
filehandles.
Reads data from syslogd via a FIFO for efficiency and low latency alerts.
Script returns error status to the system when an error occurs.
Verbose and descriptive error messages if something goes wrong.
Multiple debugging levels.


Log Analysis
Exam 312-38
Figure 20-11: Logdog options with command line.


Log Analysis
Exam 312-38
ACTIVITY 20-8
1. LogSentry uses a program______ for remembering last position in log file.
a. Logbuffer
b. Logtail
c.
Log reader
d. Log manager
_____________________________________________________________________________
2. _____is a simple ftpd log file analysis tool.
a. flog
b. Wflog
c.
SL2
d. LogSentry
_____________________________________________________________________________
3. Logdog is a tool that monitors messages passing through______.
a. Firewall
b. Logfile
c.
Syslogd
d. Watchdog
____________________________________________________________________________
4. Explain the tool that was designed to find clusters in logfile.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. Explain Dumpel with options.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC I
Introduction to Log File Rotation Tools
Log File Rotation Tools
Log Controller
LogController allows you to restrict the size of log files that become too large. Config files allow specifying
the files to monitor and specification of size. If any file goes beyond the size, they are automatically
truncated to a new user defined size and truncated area is erased and stored.
Newsyslog
newsyslog is a highly configurable program for managing and archiving log files.
Portable (using GNU Autoconf) and it can be compiled and installed on most modern Unix or
Unix-like systems.
Supports fixed time-of-day daily archiving with a command-line option to identify the daily rollover invocation (which may be at midnight or at any other regular daily time).
Supports the FreeBSD feature that allows specification of the log roll-over time as a daily, weekly,
or monthly interval.
Supports optional PID files so that non-standard daemons can be told to re-open their logfiles
after archiving has taken place.
It can send a signal other than SIGHUP to the daemon associated with a given log file.
It can leave the most recently archived log file uncompressed, which is necessary for daemons like
httpd and smail because they continue to write to the current log file until their current jobs have
been completed.
It supports the FreeBSD feature of being able to restrict processing to just those log files specified
on the command line.
Unlike the NetBSD version, it first parses the config file before taking any action, meaning that if
any errors are encountered it will report them and quit without doing anything.
Unlike the FreeBSD version, it will roll a log file if either the interval or size limits have been
reached.
It uses an advisory lock on the current configuration file to prevent multiple invocations from
tripping over each other.
Spinlogs
Spinlogs is a shell script for rotating system logs. It can be configured through the text file that is similar
in feature and format to a new Syslog program in FreeBSD. Unix systems running ksh can use it. Many
options are available that provide how and when log files are rotated, and the config file is very
straightforward. Rotating log files; using this tool is easy.
Trimlog
Trimlog is used to trim system log files beyond the limit to keep them from growing without restriction.
When invoked, it reads commands from the file that tell it which files to trim, how to trim them, and by
how much they should be trimmed.
System Log Rotation Service (SLRS)

The System Log Rotation Service (SLRS) is a tool that automates rotation of collected Unix system log
files. All included systems and their managed log files are centralized, which helps when troubleshooting
obscure system or network problems. This utility also provides the following capabilities on a per log file
basis:


Log Analysis
Exam 312-38
A unique definable size threshold
If enabled, automatic compression of the rotated log file
Self-purging of the rotated log file when the age of the rotated file exceeds a definable limit
Daemon tickling/ notification when the log file is rotated
An ability to bypass the rotation on defined hosts
Definable protections on the rotated log files
Definable destinations for the rotated log files
Bzip2
Bzip2 compresses the files for 10 to 15 percent of the best techniques (the PPM family of statistical
compressors), whilst being around twice as fast at compression and six times faster at decompression. It
compresses well, so it packs more stuff into your overfull disk drives, distribution CDs, backup tapes, and
Zip disks, etc. And it reduces your phone bills, customer download times, and long-distance network
traffic.


Log Analysis
Exam 312-38
ACTIVITY 20-9
1. What is a log controller?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. What is a newsyslog?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. Explain Spinlogs.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. What is a trimlog?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. Expand SLRS________
a. Service Log Rotation System
b. Server Log Rotation System
c.
System log Rotation Service
d. Server Log Rotation Service

_____________________________________________________________________________
6. Explain Bzip2.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
TOPIC J
Understanding Log Security
How to Secure Logs (Log Security)
The administrators of infrastructure and system level must protect the integrity and availability of log
data and confidentiality. The security considerations for the log on the systems, in storage, and in transit
include the following:
Limit access to log files
Avoid recording unneeded sensitive data
Protect archived log files
Secure the process that generates log entries
Configure each log source to behave appropriately when logging errors occur
Implement secure mechanisms for transporting log data from the system to the centralized log
management servers
Limit Access to Log Files

To access the log files there should be some level of access for the users for creating log entries. If there is,
then the users should have append-only privileges and no read access. Operations like renaming, deleting,
and other file operations on the log files cannot be performed by the users.
Avoid Recording Unneeded Sensitive Data

There are some logs that record sensitive data like passwords that are not required to be logged. The
information that is not necessary should not be recorded as it presents a substantial risk if it is accessed
by unauthorized persons.
Protect Archived Log Files

It includes the creation and securing of message digest for the file, encrypting log files, and providing
adequate physical protection for the archival media.
Secure the Process that Generates Log Entries

Access and manipulation of the log source process should not be done by unauthorized parties. Execution
of files, configuration of files, and other components of log sources should be protected as it makes an
impact on logging.
Configure Each Log Source to Behave Appropriately when Logging Errors

Occur
Logging can be so important for a particular source that it can be configured to stop its functionality
totally when the logging fails.
Implement Secure Mechanisms for Transporting Log Data from the System
to the Centralized Log Management Servers
The transportation of log data from various systems to the centralized log management server is
important; if the secure mechanism is not needed and not provided automatically by the log management
infrastructure then the transport protocols like HTTP and FTP cannot provide protection. The up
gradation of the logging software should be done to provide extra security features. There is one more


Log Analysis
Exam 312-38
alternative to encrypt the logging communication through the separate protocols like Internet protocol
security (IPSec) or SSL.


Log Analysis
Exam 312-38
ACTIVITY 20-10
1. Explain log security.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. List the features of log security.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. How do you protect archived log files?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. Explain how to access log files.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________


Log Analysis
Exam 312-38
Lesson FlowUp
1. Explain in brief about log analysis.
2. Explain different types of log files.
3. Explain in detail about logging and different types of logging.
4. Explain different log analysis tools.


Log Analysis
Exam 312-38
Reference
Log Types:
http://www.serverwatch.com/tutorials/article.php/10825_3366531_1
Log Files:
http://www.ii.fsu.edu/~cmcclure/logs.html
Access Log:
http://searchsecurity.techtarget.com/sDefinition/0,,sid14_gci212498,00.html
http://digitalarchive.oclc.org/da/ViewObjectMain.jsp;jsessionid=84ae0c5f824035f3f3fce7d243bc80d4
ed923c344444?fileid=0000003456:000000088016&reqid=9939
Web Server Log Analysis:
http://www.wynsoft.co.uk/logs/
http://www.motive.co.nz/glossary/weblogs.php
Analog
http://www.bluereef.net/support/extensions/stats/analog.html
http://www.mach5.com/download/downloadsignup.php?submit1.x=61&submit1.y=24&Name=&Comp
anyName=&Email=&Telephone=&Timezone=&Info=&package=fsanalyzer.exe&RedirectURL=http%3A
%2F%2Fmach5.fileburst.com%2F&Product=ANALYZER
Happy Log
http://www.sharewareplaza.com/Happy-Log-download_16697.html
Click Tracks
http://www.clicktracks.com/?a=18235
http://www.softpedia.com/progDownload/ClickTracks-Analyzer-Download-3737.html
Apache Logs
http://www.unixreview.com/documents/s=9233/ur0407i/
IIS Logs
http://www.iislogger.com/en/
Source taken from CEH v5 module 11
IISLoggers
Source taken from CEH v5 Module 27
Limitations of log files Analysis
http://www.sdcn.org/webworks/monitoring/
System Log Aggregation, Statistics And Analysis
Syslog
Source taken from CHFI v3 Mod 27.
Building a central loghost
http://www.giac.org/certified_professionals/practicals/gsec/0355.php


Log Analysis
Exam 312-38
Parsing and normalizing

http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf
Bayesian spam filters for logging
http://email.about.com/cs/bayesianfilters/a/bayesian_filter.htm
Storage and rotation
http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf
Databases and logs
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r0m0/index.jsp?topic=/com.ibm.etools.mft.doc
/an04110_.htm
Secure Audit Logging
www.isoc.org/isoc/conferences/ndss/04/proceedings/Papers/Waters.pdf
Setting Up Remote Logging
http://technet2.microsoft.com/windowsserver/en/library/3668062a-64b7-44d4-b9103ab84b32cae41033.mspx?mfr=true
http://technet2.microsoft.com/windowsserver/en/library/6a61f682-5dc7-496c-bd4bac58da5ff6161033.mspx?mfr=true
Windows Logging
http://www.windowsecurity.com/articles/Understanding_Windows_Logging.html
NTsyslog
http://ntsyslog.sourceforge.net/
Application Logging
http://msdn2.microsoft.com/en-us/library/aa480464.aspx
Firewall Logging
http://www.loganalysis.org/sections/parsing/application-specific/firewall-logging.html
Monitoring for Intrusion and Security Event
http://www.microsoft.com/technet/security/prodtech/windows2000/secwin2k/swin2k09.mspx
Importance of Time Synchronization
Passive Detection Methods
EventCombMT
Event Collection
Scripting
Log analysis Tool


Log Analysis
Exam 312-38
UserLock
http://www.isdecisions.com/en/software/userlock/screenshots.cfm#20
Auditing Tools
ASDIC
http://info.ping.se/storage/users/3/3/images/23/exempel.png
Tenshi
SpoofMAC
http://www.socketsoft.net/products.asp?p=smac
Gentle MAC PRO
http://www.solarst.com/productindex.html
Generic Log Parsing Tools
LogSentry
http://sourceforge.net/projects/sentrytools
SL2
http://www.ip-solutions.net/syslog-ng/sl2
Flog
http://www.securityfocus.com/tools/400
Simple Log Clustering Tool(SLCT)
http://kodu.neti.ee/%7Eristo/slct/slct.html
http://kodu.neti.ee/%7Eristo/slct/
xlogmaster
http://www.gnu.org/software/xlogmaster/
GeekTool (mac O.S)
https://sourceforge.net/projects/geektool
Dumpel.exe (Windows O.S)
http://support.microsoft.com/kb/927229
Watchlog
http://www.bshell.com/projects/WatchLog/
http://sourceforge.net/projects/watchlog
LogDog
http://caspian.dotconf.net/menu/Software/LogDog/
http://caspian.dotconf.net/menu/Software/LogDog/v1.0-old/
Log File Rotation Tools
LogController
Newsyslog
http://www.weird.com/~woods/projects/newsyslog.html


Log Analysis
Exam 312-38
Spinlogs
http://freshmeat.net/projects/spinlogs/
Trimlog
System Log Rotation Service (SLRS)
http://www.openchannelfoundation.org/projects/SLRS
Bzip2
http://www.bzip.org/
How to Secure Logs (Log Security)
www.csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf


Week 4 Part 5 - Module 20 Log Analysis

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Week 4 Part 5 - Module 20 Log Analysis

Hochgeladen von

Copyright:

Verfügbare Formate

EC-Council Network Security Administrator

Introduction to log analysis

Categorizing the log files

Understanding the concept of web server log analysis

Identifying Syslog statistics and analysis

Understanding the concept of logging

Introduction to monitoring and security events

Understanding log analysis tool

Various log parsing tools

Introduction to log file rotation tools

Understanding log security

Module XX Page | 999

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Understanding the Concept of Logging: Overview of logging, setting up remote logging,

Understanding Log Security: Discusses secure logs and features.

Module XX Page | 1000

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Tracks the success and failure events

Eliminates or minimizes the risk of unauthorized access

Account logon events User logs onto the domain.

Account management Creation of accounts, modification, renaming, or deletion .

Logon events A user logs on to or off a computer.

Object access An object accessed should have auditing on.

Module XX Page | 1001

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Process tracking A process initiated.

Module XX Page | 1002

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Module XX Page | 1003

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Module XX Page | 1004

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Total number of visitors to a home page

Origin of users with associated servers domain name

Below is an example line of text from an access_log:

2. Data and time

Domain Name or Internet Protocol (IP) Number

.com U.S. commercial business (e.g., google.com, aol.com)

.gov U.S. governmental agencies (e.g., whitehouse.gov, epa.gov)

.edu U.S. Educational (colleges, universities) (e.g., syr.edu, bu.edu)

.net U.S. network providers (e.g., internic.net, internet.net)

.org U..S non profit organizations (e.g., madd.org, greenpeace.org)

.mil U.S. military (e.g., army.mil, navy.mil)

Date and Time

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Module XX Page | 1007

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Figure 20-1: A TCPDump log.

Module XX Page | 1008

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator

Module XX Page | 1009

Network Security Administrator Copyright by EC-Council

EC-Council Network Security Administrator