A Statistical Method for Structure Learning of Bayesian Networks from Data

Soojung Ha*, Seyun Kim*, Minkook Suh*, Hyunwoo Seong*, Kwang Mo Jeong† , Sung-Ho Kim‡
*
Korea Science Academy, † Pusan National University,
‡
Korea Advanced Institute of Science and Technology
sjhakorea@hanmail.net, whataud@naver.com, minkook789@hanmail.net,
hwseong@hotmail.com, kmjung@pusan.ac.kr, sung-ho.kim@kaist.edu

Abstract is the number of nodes) is large even for a small

The Bayesian network, a powerful tool for
predicting and diagnosing uncertain phenomena, is
used in various fields including artificial intelligence,
business administration, and medical science. We use
a statistical approach, and present a simple algorithm
for learning Bayesian network structure from data.
First we obtain from data the original correlation
graph and the correlation graphs when one or two
variables are fixed. Then we construct a Bayesian
network that would produce the most similar
correlation graphs. Simulation results are given to
demonstrate that the algorithm determines the network
structure with a high accuracy.
1. Introduction
Bayesian networks (BN) are graphs that represent
the probabilistic relationships among a large number of
variables [1]. They are used in many areas that involve
probabilistic inference, such as artificial intelligence,
business administration, medical science, biology, and
social science. Basically, Bayesian networks are
directed acyclic graphs with joint probability
distributions. The nodes denote the random variables,
and the directed edges represent the causal relationship
between the random variables.
When the number of variables is small and the
causal relationships between the variables are known,
BN structure can be constructed directly without the
help of computers. However, when the number of
variables is large or when the causal relationships are
not fully known, the BN structure has to be induced
from data. This procedure is often very difficult
because the number of possible structures (2n, where n
number of variables, and if we were to investigate all
the possible cases one by one, the computing time
would be enormously long. Moreover, the data we can
obtain are only part of the population. When the
sample data do not represent the population well, the
computation for the BN structure becomes more prone
to error.
Recently, many researches have been conducted to
develop efficient methods for learning Bayesian
networks from data. There are two main approaches to
this problem. One is to define a scoring function of the
network and search for the network which has a
maximum score. Various scoring functions [2] [3] and
searching techniques [4] [5] have been developed. The
other approach is to determine the presence or absence
of an edge by examining conditional independencies
obtained from data. Some other researches use both
approaches, to form hybrid algorithms.
In this paper we present a simple and time-
efficient algorithm based on the second approach.
Then we give simulation results to show that the
algorithm produces BN structure with a high accuracy.
2. The correlation graph
The correlation graph is a key concept in our
algorithm. A correlation graph of a BN represents the
correlation (or dependency) between the random
variables. There is an edge between two variables if
and only if the two variables are dependent.
Fixing variable A to a means that when
considering the joint probability function between two
variables X and Y, instead of P(x, y), we consider P(x, y
| A=a). We can also think of correlation graphs with
some variables fixed. These correlation graphs will be
different from the correlation graph with no variables
fixed.
When there is a BN, we can deduce a correlation
graph from it. Also, when there is data, we can derive a
correlation graph from them. Deducing a correlation
graph from a BN is simple. Two variables A and B are
connected in the correlation graph if 1) either of them
is an ancestor of the other or 2) they share a common
ancestor. If they satisfy neither condition, no edge
exists between A-B. The following figure is an
example of deducing a correlation graph from a BN.
Figure 1. Deducing a correlation graph from a known
BN
When some variables are fixed, the correlation
graph can change in accordance with the conditional
distribution of the remaining variables of the BN. Two
originally dependent / independent variables can
become independent / dependent. This happens in two
cases: 1) if a common descendent of two originally
independent variables is fixed, the two variables and
all their descendents become dependent and 2) if in all
the paths connecting two originally dependent
variables, at least one variable is fixed, the two
variables become independent. This is illustrated in the
following figure.
a) c)
b) d)
Figure 2. a) BN b) fixing A c) correlation graph
d) correlation graph when A is fixed
To obtain a correlation graph from data, we must
consider errors in the data. Even if two variables A and
B are independent, P(A|B)=P(A) cannot be fulfilled
precisely unless the sample size is infinite. Instead we
set a correlation function for measuring dependency
(or independency) and when the function value is over
a certain value, determine that the two variables are
dependent. From the data, we compute the correlation
function of two variables and if they are dependent,
add an edge between them in the correlation graph.
To calculate the correlation graph when some
variables are fixed, we sort out the data before
computing the correlation function. For example,
suppose A is a discrete variable that can have values 0
and 1. From the database, we first select only the data
in which A=0, and compute the correlation function.
Next we select only the data in which A=1, and
compute the correlation function. In this respect, fixed
variables may be regarded as conditioning variables.
Now we average the two correlation values,
giving more weight to the case with larger data size.
Determining dependency of all the pairs of variables,
we obtain a new correlation graph where the
conditioning variables are isolated from the other
variables since the correlation coefficient of a random
variable with a constant is zero. We may regard a
conditioning variable a constant since its value is fixed
when computing for their correlation graph.
If the data size is large enough, the correlation
graphs obtained from a BN and data will most likely
be identical. Our research is based on this observation.
In our algorithm, we first obtain the correlation graphs
from data, and then try to construct a Bayesian
network that would produce the same, or the most
similar outcome.
3. The proposed algorithm
Our algorithm consists of two main parts. First we
find the skeleton of the BN, i.e., the undirected version
of the BN, and then we determine orientation of the
edges.
3.1. Construction of skeleton
[Initialization]
1. Initial skeleton has all the nodes, and no edges.
[Adding edges]
2. Obtain a correlation graph from data. Add all the
edges to the skeleton to obtain the correlation.
[Deleting edges]
3. If when A is fixed, the correlation coefficient between
B and C is close to zero, delete the edge between B and
C.
4. If when A and B are fixed, the correlation coefficient
between C and D is close to zero, delete the edge
between C and D.
Figure 3. Part 1 of the proposed algorithm
 If two variables are thought to have a direct causal
relationship, we connect them in the skeleton.
In step 1, we start with an initially empty skeleton.
In step 2, we add all the possible ‘candidates’ for edges
that represent direct causal relationship. If two
variables have a direct causal relationship, they will be
correlated to each other. Therefore, an edge between
the variables will be added to the skeleton after this
step. However, two variables can be correlated without
a direct causal relationship. So some edges must be
deleted after this step.
In steps 3 and 4, we consider the cases explained
above. This can happen when one variable is an
ancestor of the other or when the two variables share a
common ancestor. In either case, if we block every
path connecting the two variables by fixing certain
variables on the paths, their correlation will disappear
and we delete the edge between them.
As we only fix up to two variables at a time in this
paper, if the structure is so complex that we cannot
block all the paths between certain two variables, we
may not be able to delete all the unnecessary edges.
However, we can think of three problems arise from
fixing many variables at a time: lengthy execution time,
decrease of sample size, and higher risk of deleting
edges that do represent a causal relationship. Suppose
that the number of nodes is n, and that it takes
approximately t time to sort out the data, obtain a
correlation graph, and delete edges accordingly. If up
to n-2 variables could be fixed, (2n-n-2)t time would be
required. This can be a problem when n is large.
Decrease of sample size is an equally serious problem.
When computing for correlation graphs with some
variables fixed, the sample size decreases dramatically
in the data sorting process. Also, by adding more steps
to delete edges, we may increase the possibility of
deleting edges that should not be deleted. In some
cases the correlation between a parent and a child is
not very strong, and the edge can be deleted
accidentally. Since omitting edges that represent direct
relationships is a more serious problem than adding
edges that do not represent direct relationships, it is
preferable to include only a few steps in edge deletion.
3.2. Edge Orientation
(If orienting an edge according to the algorithm creates a
cycle, leave the edge undirected)
[Convergence of edges]
5. If edges A-C and B-C exist in the skeleton but no
edge exists between A and B in the correlation graph,
and the correlation between A and B is stronger when C
is fixed, A→C, B→C
6. If edges A-C and B-C exist in the skeleton but no
edge exists between A and B in the correlation graph
when another variable D is fixed, and the correlation
between A and B becomes stronger when C is fixed,
A→C, B→C
7. If edges A-C and B-C exist in the skeleton but no
edge exists between A and B in the correlation graph
when D and E are fixed, and the correlation between A
and B becomes stronger when C is fixed, A→C, B→C
[No convergence of edge direction]
8. If an edge A-B exists in the correlation graph (but
not in the skeleton), and there is only one path that
connects A-B in the skeleton, direct the edges so that no
convergence of edge direction (i.e. X→Y and Z→Y when
X, Y, Z are adjacent in the path) happens in the path.
[Preventing cycles]
9. If A is an ancestor of B and there is an edge
between A and B in the skeleton, A→B.
Figure 4. Part 2 of the proposed algorithm
Steps 5 to 7 find three variables A, B, C that A and
B are independent, but both are connected to C. In this
case, A and B are both parents of C [6]. This is because
of the four possible cases
A→C→B, A←C←B, A←C→B, A→C←B.
where there will be no correlation relationship between
A and B only in the last case. For verification, we
check the correlation between A and B when C is
fixed: if the correlation gets weaker we do not set the
arrow directions.
Sometimes A and B which are not adjacent to each
other might be both parents of C but not independent.
This is when they share a common ancestor or if one is
an ancestor of the other. Steps 6 and 7 are designed to
solve this problem. If we fix some variables along the
paths connecting A and B (other than A-C-B), we might
detect that the dependency of A and B is not
contributable to C. Then we can set the arrow
directions as in step 5.
Step 8 works in the other way. For two variables
to be dependent while not having a direct causal
relationship, they must share a common ancestor or
one must be an ancestor of the other. If there is only
one path connecting the two variables, one variable
must be an ancestor of the other, or a variable along
the path must be a common ancestor of both. This is
equivalent to ‘no convergence’ of edge direction in the
path. Step 9 directs all the edges from ancestor to
descendent. If directed otherwise, the edge direction
creates a cycle.
At the end of the algorithm, some edges are left
with no direction assigned. This is because for a
certain set of correlation graphs, there are many
Bayesian networks that would produce the same set. A
simple example is two BNs A→B and A←B. They are
indistinguishable in regard to data. Another example is
when the network is a tree. Regardless of whichever
leaf node we choose as the root, we can obtain a BN
that is a faithful representation of data.
Among the steps in the algorithm, step 7 has the
most computation time. With two fixed nodes, we
should consider three other nodes, yielding the time
complexity of the algorithm O(n5) in worst case.
similar correlation graphs. The skeleton is constructed
by first adding edges from the original correlation
graph, and then deleting edges considering the
correlation graphs when one or two variables are fixed.
The orientations of the edges are determined in three
successive steps, respectively related to convergence
or no convergence of edge direction, and preventing
cycles. The experimental results demonstrate the
algorithm’s ability to learn BN structure with a high
accuracy.

4. Experimental Results
We implemented the algorithm using C++. The
program accepts data as input and produces BN as
output. To test the accuracy of the proposed algorithm,
we created four Bayesian networks and generated four
sets of test data from them. The sample size was 10000.
Then we determined the network structure for each set
of data with our program. Figure 5. Original and deduced graphs (BN3)
The table below summarizes the results. It shows
that our algorithm has a high accuracy. We found from 6. References
the experiments that there was no missed edge, which
means that recall rate is 98.85% (86/87*100%). There [1] R. E. Neopolitan, Learning Bayesian Networks, Prentice
were only a few edges(9) added incorrectly. We got Hall, Chicago, Illinois, 2004.
precision rate 90.53%(86/(86+9)*100%). It should be
also noted that there was no edge with wrong direction. [2] N. Friedman and Z. Yakhini, “On the sample complexity
Figure 5 illustrates one of the simulation results: BN3, of learning Bayesian networks,” Proceedings of the 12th
conference on Uncertainty in Artificial Intelligence, Morgan
where only two edges are added incorrectly, and no
Kaufmann, 1996.
edge in the original graph is missed.
[3] Wallace, C.S., and K. Korb, “Learning Linear Causal
Table 1. Simulation results Models by MML Sampling”, in Gammerman, A. (Ed.):
Random Correct Wrong edges
Total Causal Models and Intelligent Data Mining, Springer-Verlag,
Variable edges (No Wrong
Edges Missed Added New York, 1999.
s direction) direction
BN1 10 16 15 (6) 1 0 0
[4] P. Larranaga, M. Poza, Y. Yurramendi, R.H. Murga,
BN2 13 21 21 (18) 0 3 0 C.M.H. Kuijpers, “Structure learning of Bayesian networks
BN3 16 15 15 (9) 0 2 0 by genetic algorithms: a performance analysis of control
parameters”, IEEE Transactions on Pattern Analysis and
BN4 20 35 35 (17) 0 4 0
Machine Intelligence, Vol. 18, No. 9, 1996, pp. 912-926.
Total 59 87 86 (50) 1 9 0
[5] L.M. de Campos, J.M. Fernandez-Luna, J.A. Gamez, J.M.
5. Conclusion Puerta, “Ant colony optimization for learning Bayesian
networks”, International Journal of Approximate Reasoning,
Vol. 31, No. 3, 2002, pp. 291-311.
In this paper, we presented a simple and efficient
BN learning algorithm based on a statistical approach. [6] G. Rebane and J. Pearl, “The recovery of causal poly-
First we obtain from data the original correlation graph trees from statistical data”, in Proceedings of the Third
and the correlation graphs with some variables fixed. Conference on Uncertainty in Artificial Intelligence, Seattle,
Then we construct a BN that would produce the most Washington, July 1987, pp. 222-228.