The Third International Conference on Availability, Reliability and Security

Intrusion Detection with Data Correlation Relation Graph

Amin Hassanzadeh, Babak Sadeghian

Data Security Research La (DSRL),
Computer Engineering and IT Department,
Amirkabir University of Technology
{Hassanzadeh , Basadegh}@aut.ac.ir

Abstract higher-level descriptions of attacks or a more

Intrusion Detection Systems are designed based on
the assumption that the behavior of an intruder is
different from a normal user of a system. We show that
intrusion detection can be done based on the
assumption that the correlation of system events and
parameters is changed during an attack to the system.
In this paper, we propose a new method in correlating
data and events for "Network Based Intrusion
Detection Systems". When an attack occurs, the
correlation of security parameters is changed. We
propose to use the state of correlation between
parameters to detect an attack. First we show how to
select effective security parameters for our detection
engine with statistical correlation methods. Then, we
propose how to build Correlation Relation Graphs
(CRG) for the parameters showing higher correlation.
Finally we show how the attack may be detected with
comparing the CRG parameter pairs for each session
with the deviation from the regression line of them. We
present our results for detecting a SynFlood attack
with this method. We give also the corresponding
detection rate and false alarm rate.
1. Introduction
Evidence of attacks against a network and its
resources can be scattered over several hosts. Intrusion
detection systems have to collect and relate alert
information from different sources to spot complete
attack scenarios. The process of collecting and relating
alert information is called alert correlation. Recently,
alert correlation gained momentum and a number of
academic and commercial correlation approaches have
been suggested. However, there is no consensus on
what this process is or how it should be implemented
or evaluated [1]. Some systems use different IDSs and
then correlate the analysis results and the
corresponding alarms. This approach aims at achieving
condensed view of the security issues highlighted
during the analysis without losing security-relevant
information[2] [3].
Alarm correlation based IDSs only determine1 the
relation and correlation between alarms, generated by
IDS’ sensors, but there are some other systems that
focus on alert correlation. Some of these systems,
introduced in [4], do not use independent IDSs. For
example Ning et al. in [5] and [6] construct attack
scenarios by correlating alerts in an individual IDS on
the basis of prerequisites and consequences of
intrusions. In [1] some correlated alert create a new
Meta-Alert to achieve higher-level descriptions of
attacks. In [5], Ning suggests that several alert
correlation methods have been proposed for intrusion
detection systems and categorized them into three
classes. In the first class, the alerts are correlated based
on the similarities between alert attributes (for example
[7]). In the second class, the alert correlations are based
on scenarios which are specified by human users or
learned through training datasets, and are restricted to
known attack scenarios. In the third class, the alert
correlations are the third class is based on the
preconditions and consequences of individual attacks
(for example see [5] and [6]).
In this paper, our approach is data correlation. Data
correlation means associating sets of events recognized
through various means and applying knowledge to
determine whether they are related, and if so, in what
manner and to what degree. As the amount of
correlation between two parameters is increased, the
relation between parameters and their behavioral
similarity is more justifiable. In [8], it is suggested that
this kind of correlation requires comparing
observations on different parameters, such as source
and destination IP address, an identifiable network
1 This research is partially supported by Iran Telecommunication
Research Center.

0-7695-3102-4/08 $25.00 © 2008 IEEE 982

DOI 10.1109/ARES.2008.119
route, commands entered by a suspected attacker, and handshake (see Figure 1). The first step in the process
the time when activity began or ended. is a SYN packet that is sent from Source to
Each behavior is constructed with some parameters. Destination. The second message, from Destination to
For example in every TCP sessions the "protocol" field Source, has both the SYN and ACK flags set indicating
of IP packet is equal to "6", or for another example, all that Destination acknowledges the SYN and is
segments of each message in IP layer get a same value continuing the handshake. The third message, from
for identification field. In network attacks we have Source to Destination has its ACK bit set, and is an
same situation. In a TCP Flood attack, we have a lot of indication to Destination that both hosts agree that a
TCP packets with "SYN=1" and other specific connection has been established. The third message
features. But an ICMP Flood attack never has these may contain user payload data.
specifications. In fact, each attack has its own
properties that are different from other attack
properties and even with normal behavior. We use this
feature to compare each session with normal or a
specific behavior.
In this paper, we calculate the correlation between
parameters in the attack traffic with statistical methods.
If this value is greater than a defined threshold value,
the correlated parameter pairs are considered to be
included in a correlation relation graph. This threshold
is defined by an expert. We consider these final
parameters to present such an attack. This method
reduces processing load of detection engine and the set Figure 1. Three-way Handshake
of parameters that are needed for intrusion detection.
Our approach is a misuse detection approach. We also Synflood is a DoS attack to which every TCP/IP
show how to detect a Synflood attack, which is one implementation to some degree is vulnerable. Each
type of DoS attacks. We select correlated parameters of half-open TCP connection made to a machine cause
Synflood attack and calculate their regression relation. the ‘tcpd’ server to add a record to the data structure
Finally, we compare the regression relations of that stores information describing all pending
correlated parameters of each network session with connections (see Figure 2). This data structure is of
regression relations of our graph. Our approach is finite size, and it can be made to overflow by
misuse detection and we calculate the correlation of intentionally creating too many partially-open
parameters based on new observation of attack connections. The half-open connections data structure
sessions. In [9], we have shown that our method can be on the victim server system will eventually fill and the
used for anomaly detection as well. system will be unable to accept any new incoming
This paper is organized as follows. In Section 2 the connections until the table is emptied out.
structure of Neptune (Synflood) attack will be
described. In Section 3 we will describe data
correlation methods. In Section 4 the construction
algorithm of correlation relation graph will be given
and in Section 5 we will compare behavior of some
parameters in attack traffic with their behavior in
normal traffic. In Section 6 our data correlation based
detection engine and its performance for Synflood
attack detection will be presented. Finally we will give
some conclusion for the results obtained in this paper
in Section 7.

2. Synflood Attack Figure 2. Attacking to Victim with half-open

connections
Before data can be transmitted between a source
host and a destination host, TCP needs to establish a Normally there is a time out associated with a
connection between them. The connection pending connection, so the half-open connections will
establishment process is called the three-way eventually expire and the victim server system will
recover. However, the attacking system can simply

983
continue sending IP-spoofed packets requesting new
connections faster than the victim system can expire
the pending connections. Ref [10] believes that typical
Synflood attacks can vary several parameters: the
number of SYN packets per source address sent in a
batch, the delay between successive batches, and the
mode of source address allocation. Ref [11] says that in
some cases, the system may exhaust memory, crash, or
be rendered otherwise inoperative.
3. Data Correlation Methods
All the devices, whether aimed at prevention or
detection, generate huge volumes of audit data.
Firewalls and other devices logging network
connection information are especially guilty of
producing vast oceans of data. Many diverse data
formats and representations are used for those log files
and audit trails. Also, a percentage of events generated
by network IDS and IPS are false alarms and do not
map to real threats. Further confusing issue is that the
different devices might report on the same things
happening on the network, but in a different way, with
no apparent way of figureuring the truth of their
relationship. There is a definite need for a consistent
analysis framework to identify various threats,
prioritize them and learn their impact on the target
system.
Correlation is defined as relationships between
entities, however, a good Intrusion Detection-specific
definition is lacking. From security point of view,
event correlation may be defined to improve the threat
identification and the assessment process by looking
not only at individual events, but also at their sets.
Chuvakin in [12] asserts that security-specific
correlation can be loosely categorized as rule-based or
statistical (algorithmic).
A rule-based correlation engine has some pre-
existing knowledge of the attack (the rule), and it is
able to define what is actually detected in precise
terms, base on that. Such attack knowledge is used to
relate events and analyze them together in a common
context. Statistical correlation does not employ any
pre-existing knowledge of the malicious activity, but
instead relies upon the knowledge (and recognition) of
normal activities, which has been accumulated over
time. Ongoing events are then rated by a built-in
algorithm and may also be compared to the
accumulated activity patterns, to distinguish normal
from abnormal (suspicious) behavior. This distinction
among correlation types is somewhat similar to
signature vs. anomaly IDS.
In this paper, we propose another data correlation
method that calculates the correlation value between
984
security parameters with statistical analysis of an attack
behavior. In fact, we combine both statistical and rule-
based methods. This method of data correlation
processes the gathered statistical samples of security
parameters from attack sessions during train phase. In
addition to pre-existing knowledge, this method uses
statistical correlation to obtain some correlation values
for security parameters in attack behavior.
We call this method as Statistical Rule-based
Correlation (SRC), which calculates correlation value
between parameters in attack sessions using statistical
methods. A security parameter may vary in different
observations. We can find correlation value between
two parameters with different pairs of observation.
Calculation of correlation value between such
parameters gives us the magnitude of relation between
them.
In random samples of statistical population, n
observation of X and Y variables are represented by
( X i , Yi ) pairs, for i=1, 2,… n. These pairs have equal
bi-variable distribution and different pairs are
independent of each other. A simple relation between
X and Y creates some points around the straight
regression line. We use Pearson correlation coefficient
to determine the value of correlation between two
parameters:
n
∑ (X
i =1
i − X )(Yi − Y )
(1)
r=
 n
2
n
2
∑ ( X i − X )  ∑ (Yi − Y ) 
 i =1   i =1 
In which a ( X i , Yi ) pair is an observations of X and
Y random variables. X ,Y are mean values for X and Y
respectively. r has a value in [-1, 1]. While r=1 means
that all points of ( X i , Yi ) pairs are on a straight line with
a positive slope. r=-1 means that all points of
( X i , Yi ) pairs are on a straight line with a negative
slope. When r approaches from these two values to
zero, the degree of correlation decreases; such that
there is no correlation in zero point.
3.1. Correlation Hypothesis Test
The required solution for a problem might be
simplified into a choice between two competing
hypothesis between which we have a choice, i.e., the
null hypothesis, denoted by H0, against the alternative
hypothesis, denoted by H1. The null hypothesis, H0, is
a statement of a theory that has been put forward,
either because it is believed to be true or because it is
to be used as a basis for argument, but has not been
proved. The alternative hypothesis, H1, is a statement
of what a statistical hypothesis test is set up to
establish.
We considered H0 as the null hypothesis about the
correlation coefficient r between all samples. Note that
r is a random variable. We considered our H0 and H1
as:
H 0 : ρ = ρ0 = 0 (2)
H1 : ρ ≠ ρ0(3)
Where ρ is correlation coefficient of entire
population and calculated by:
Cov ( X , Y )
ρ = Corr ( X , Y ) = (4)
σ XσY
The Rejection of the null hypothesis, suggests that
the alternative hypothesis may be true and r is valid.
We can use a Confidence Value (CV) for rejecting H0.
If P-value2 < (1 - CV), H0 is rejected with the
confidence value of CV.
4. Correlation Relation Graph (CRG)
4.1. Correlation Relation
Selecting the optimum security parameters from
several available parameters is one of the major
problems of intrusion detection systems. An expert that
selects these parameters for intrusion detection engine
and gives them to the system analyzer faces some
challenges. One of them is determining the parameters
which present more effective statistical information to
a system analyzer.
We examined security parameters that are
correlated such that the value of their correlation
coefficient is more than threshold level introduced by
an expert. Hence, the parameters that haven’t this
condition are not considered for examination.
Ref [14] introduces a set of DoS attack traffic that is
used for Intrusion Detection Evaluation. We select
only ten percent of all Neptune sessions of KDD
Cup99 dataset for our statistical analysis. Each session
of KDD Cup99 dataset, either attack or normal, affects
on 41 introduced security parameters. We calculated
the correlation coefficient between these parameters,
and finally introduced some of them as optimum
parameters for intrusion detection.
Parameters that have required correlation value and
participate in CRG are effective security parameters in
our intrusion detection system. The first step for
constructing CRG is to create correlation matrix of
security parameters. Each entry of this matrix is the
2 P-value is a statistic. For more details See [13].
985
correlation coefficient between two parameters that are
calculated using (1), and its validity is examined by the
hypothesis test. For example, CRG i× j represents the
correlation value between i and j that the former
parameter is in the ith row and the latter one is in the
jth column.
We selected the entries that are greater than our
defined threshold. Since the correlation coefficient is a
value in [-1, 1] and 0 ≤ r ≤ 1 , we intuitively
considered 0.5 as the appropriate threshold. It should
be noted that the sign of r only shows the correlation
direction.
4.2. CRG Construction Algorithm
CRG is a graph for modeling a set of parameters
that make an Equivalence Class under the correlation
relation. In fact, this graph is just a way to illustrate the
correlated parameters and analyzing the correlation
relations. There are some other ways to select
correlated parameters such as factor analysis and
statistical clustering methods. CRG just helps the
analyzer to get a graphical view of correlated
parameters and their relations. In this section we
explain CRG construction algorithm to select
correlated parameters of each behavior. The related
parameters have three properties in an equivalence set,
i.e.:
• Reflexive
• Symmetric
• Transitive
The CRG construction algorithm is defined as
follows:
1. Each member is a CRG.
2. To add a new member to CRG two bellow
conditions should be satisfied:
a) Correlation coefficient of new member
with all of the previous equivalence
set members should be greater than the
threshold value.
b) The null hypothesis ( H 0 : ρ = 0 )
should be rejected for all of the
correlation coefficient of CRG.
When a new parameter is added to an equivalence
class, it satisfies all of the above mentioned properties.
Note that, each variable is correlated with itself; hence
all of the security parameters satisfy the reflexive
property. It is clear by the definition that correlation
relation is a symmetric relation between two
parameters. It also remains to show that CRG has the
third property to be an equivalence class. As each new
member should be correlated with all of the previous
members of an equivalence class, the third property is
also satisfied.
Note that there is no necessity for the “b” condition
in values that are greater than threshold, as H0
hypothesis is rejected for high values. In practice, we
used this condition for members that have suitable
correlation with the other parameters, except one of
them. In this case if H0 was rejected but there was a
justification from an expert analyzer’s point of view, it
could be added to CRG. If a new member has a weak
correlation at least with one of CRG members, we
can’t join it to CRG because of violating equivalence
class.

4.3. CRG for Neptune (Synflood) Attack

As an example, we analyzed Neptune sessions of

KDD dataset and calculated the correlation matrix for
its 41 parameters. Each parameter is known by an
index that simplifies our demonstration of the analysis.
Nodes of CRG are the indices of security parameters
defined in KDD and the correlation value of two
parameters has shown beside the corresponding edge
between them. If a CRG has more nodes it will be
more useful for Intrusion Detection Engine, as it
checks further parameters and increases the percentage
of detection. For Neptune attack 11 final nodes in two
independent CRG graphs are formed (see Figure 3).
Table 1 gives the correlated parameters and
corresponding indices for Neptune attack sessions.
Note that all of these parameters can be categorized in
3 groups, i.e., parameters of Reject Errors, parameters
of Syn Errors and parameters of Number of
Connections with their type of services. Parameters of
each group have positive correlation, while parameters
in the first group (parameters of Reject Errors) and the
second group (parameters of Syn Error) have also
negative correlations. As these types of attacks
increase Syn packets, and all of half-open connections
would be rejected, then the obtained correlation is
justified.
Table 1. Final Parameters of Neptune Attack
24 Srv_Count
25 Serror_Rate
26 Srv_Serror_Rate
27 Rerror_Rate
28 Srv_Rerror_Rate
33 Dst_Host_Srv_Count
34 Dst_Host_Same_Srv_Rate
38 Dst_Host_Serror_Rate
39 Dst_Host_Srv_Serror_Rate
40 Dst_Host_Rerror_Rate
41 Dst_Host_Srv_Rerror_Rate
Figure 3. Correlation Relation Graph for
Neptune Attack
5. Behavior of Neptune Detection
Parameters
We observe that only eleven security parameters out of
41 parameters are needed for detection of Neptune
attack. Behavior of these parameters changes in the
two sessions of attack and normal. Hence, the
correlation between the parameters for normal traffic
will be different from those for attack traffic. Let’s call
these 11 parameters as Neptune detection parameters.
We show that these 11 parameters create other graphs
in normal sessions and their correlation matrix is
different.
We analyzed the behavior of these parameters in
normal traffic and calculate the correlation coefficient
between them. Table 2 shows the decrease of
correlation values. A Comparison between these values
and former values shows that in normal sessions
members of second group have weak correlations and
the parameter number 24 leaves its equivalence set.
These 11 parameters do not show previous correlation
relations and their equivalence sets is different. Fig 4
demonstrates this difference. The correlation between
parameters number 33 and 34 decreased a few. The
decrease is too large for the 8-node equivalence set.
The number of its nodes is reduced to 4 nodes.
Parameters number 25, 26, 38 and 39 are removed
from the equivalence relation, and other parameters
also show less correlation.

986
 another CRG there are 28 regression equation and
confidence interval.
We examine KDD sessions to evaluate the
operation of this method. We used 107201 sessions for
statistical analysis that resulted to two CRG such as
CRGs represented in previous sections. First, we
examined 494021 KDD sessions (10 percent of all)
with only 3-node CRG. Then these sessions evaluated
Figure 4. CRG of 11 Parameters in Normal with 8-node CRG and we used both CRGs to detect
Sessions SynFlood attacks finally. Table III shows the results of
this approach on these 494021 sessions.
We also examine completed KDD file that it has
Table. 2. Correlation Matrix of Parameters in 4898431 sessions. Table IV shows the results of
Normal Traffic evaluating all of sessions with three sets: 3N.CRG,
8N.CRG and both of them.

Table 3. Detection Results for Test on 494021

Sessions

Detected Percent False Percent

Sessions Alarm
3N.CRG 85231 79.5 34961 32.61
8N.CRG 106856 99.67 234 0.21
Both CRGs 104323 97.31 119 0.11
6. Data Correlation Based Detection
Engine
Table 4. Detection Results for Test on
In the past sections, we described our method to 4898431 Sessions
select efficient parameters that were more useful for
intrusion detection in our designed system. We
selected only 11 parameters out of 41 security Detected Percent False Percent
parameters. Now we’ll introduce our detection engine, Sessions Alarm
and the results of SynFlood attack detection will be 3N.CRG 798351 74.47 496811 46.34
8N.CRG 1060056 98.88 5953 0.55
represented.
Both CRGs 1023825 95.5 4176 0.38
Neptune detection parameters created two CRG
graphs such that one of them had 3 nodes (parameters)
and the other had 8 nodes. Our detection system looks 7. Conclusion
at correlated parameters. A detection engine that
employs these parameters should be based on In this paper, we introduced a new method for
correlation relation and the deviation from them. In the intrusion detection. For the detection of intrusions, any
section III, we pointed out the effect of correlation IDS requires some data related to the security
relation on regression line of two statistical parameters. parameters that report the security conditions of the
As the correlation value between two parameters monitored system. It is clear that increasing the number
increases, scattering of points around the regression of parameters may be useful in increasing the system
line of them will be decreased. For correlation values analyzer’s information about the monitored system,
near 1 or -1, we can say that they are over regression while gathering more information increases the amount
line completely. of information and their analysis. Some of these
We also consider a confidence interval for each security parameters are more useful than others and
regression line. This interval determines the allowed some of them are not. The analyzer should be able to
deviation for each regression equation and every pair select effective parameters from the ineffective and
that has greater intercept couldn’t justify this relation. decrease the load of processing. One of the methods
There are three regression equation and confidence for this selection is data correlation.
interval for three nodes CRG that determine the A recent trend in intrusion detection is using
relation between the parameters 24, 33, and 34. For different intrusion detection systems, and then
correlating the analysis results and the corresponding

987
alerts [2][3]. These systems only determine the relation
and correlation between alerts generated by IDS’s
sensors, and need to be monitored by an expert. In
these systems a new method of attack can not be
detected because it depends on alerts generated by
IDS’s. We introduced a method that depends on
statistical reports and any change in attack method is
detected. We used correlation coefficient to select
effective parameters that have suitable correlation and
can create CRGs. We intuitively considered 0.5 as the
appropriate threshold. For lower values we’ll have
CRGs with more nodes and weaker links that can
increase false alarms.
Then we calculated regression equation for each
link. Each regression equation has a confidence
distance that formulates based on errors of each pairs
with their regression equation. Finally we detected
Synflood attack sessions with calculating the deviation
of each pairs from their regression equation. Note that
if attacker changes his way of attack, these CRGs will
change and our IDS works with new CRGs.
The results show that all of CRGs are not useful for
detection engine. The first CRG, that have only 3
nodes and its links are not heavy, gives more false
alarms because its regression equations have greater
confidence intervals, and some normal sessions fall on
its region. The second CRG shows better results
because it has more nodes and they have full
correlation. If we use both CRGs, however the
detection rate decreases a little but false alarms also
decrease.
Our IDS only looks at correlated parameters and
never examine non-correlated parameters. For
example, the Synflood attacks use TCP protocol and
one of the 41 parameters of KDD dataset is “protocol”.
If this parameter existed in the CRGs formed for
Synflood attack, it would be examined. If an ICMP
session creates one of the CRGs like these, it might
report an attack session, which increases the false
alarms. But we could check its “protocol” parameter
and ignore it. We can reduce false positives by looking
at specific parameters of each alarm.
In fact, our IDS does not need an expert to select
security parameters, although expert defined
parameters maybe useful for detection. Thus we have
proposed an unsupervised method to select effective
parameters. This system can work without any expert
network analyzer, so it can be modified for any new
attack behavior. For each new unknown attack, we
only need to select correlated parameters and their
regression relation.
There are two main stages in using our data
correlation based IDSs. The first stage requires relative
long time to analyze a behavior to find useful
parameters and their correlation relations. This stage
988
can be performed every day or week (depending on
attacks and their modification rates). The second stage
requires a short time to compare each event and its
parameters with regression relations. This is the main
operation in detection engine. In fact, the former
operation that requires a long time operation never
decreases the detection speed, as the analyzer can run it
in an off-line mode. If the CRGs of a behavior changed
in a time between two former type operations, the
analyzer would just updates the regression relations.
Please also notice that we have not claimed about
the speed up of this method comparing with other
methods. But we have proposed an approach that helps
in reducing the number of parameters involved in
intrusion detection and help in automating the selection
of effective parameters.
8. References
[1] Christopher Kruegel, Fredrik Valeur, Govanni Vigna,
Intrusion Detection and Correlation: Challenges and
Solutions, Springer (2005)
[2] T. Chyssler, S. Nadjm-Tehrani, “Alarm Reduction and
Correlation in Defence of IP Networks”, Proceedings of the
13th International Workshops on Enabling Technologies
(WETICE04), IEEE Computer Society, June (2004), pp. 229-
234
[3] T. Chyssler, S. Burschka, “Alarm Reduction and
Correlation in Intrusion Detection Systems”, Proceedings of
Detection of Intrusions and Malware & Vulnarability
Assessment workshop (DIMVA), Gesellschaft f r Informatik,
June (2004), pp. 9-24
[4] U. Zurutuza and R. Uribeetxeberria, “Intrusion
Detection Alarm Correlation: A Survey”, Proceedings of the
IADAT International Conference on Telecommunications
and Computer Networks, 1-3 December, 2004
[5] P.Ning, Y.Cui, and D.S.Reeves. “Constructing attack
scenarios through correlation of intrusion alerts”, Proceeding
of the 9th ACM Conference on Computer & Communications
Security, pages 245–254, Nov. 2002.
[6] D. Xu, P. Ning, “Alert Correlation through Triggering
Events and Common Resources”, 20th Annual Computer
Security Applications Conference, December 2004
[7] Valdés, A., and Skinner, K., “Probabilistic Alert
Correlation”, Proceedings of the 4th International
Symposium on Recent Advances in Intrusion detection
(RAID), Springer Verlang, California, USA, 2001, pp. 54-68
[8] Carl Endorf, Eugene Schult, Jim Mellander, Intrusion
Detection & Prevention, McGraw-Hill, (2004)
[9] Amin Hassanzadeh, Babak Sadeghian, “Anomaly
Intrusion Detection with Data Correlation Relation Graph”, [12] Anton Chuvakin, “Event Correlation in Security”,
Proceeding of the 11th International CSI Computer whitepaper, www.securitydocs.com, May (2004)
Conference (CSICC' 2006), Tehran, Iran, Jan. 24-26, 2006
[In Persian] [13] T. T.Soong, Fundamentals of Probability and Statistics
[10] Christopher L. Schuba, Ivan V. Krsul,et al., “Analysis of for Engineers, John Wiley and Sons Ltd, (2004)
a denial of service attack on TCP”, IEEE Computer Society
Press, May (2000) [14] Http: // kdd.ics.uci.edu / databases / kddcup99 /
task.html
[11] J.W.Haines, R.P.Lippmann, et al., “1999 DARPA
Intrusion Detection Evaluation: Design and Procedures”,
Technical report 1062, Lincoln Laboratory, MIT university,
26 February (2001)

989