Optimizing Directed Acyclic Graph
Support Vector Machines
Fumitake Takahashi Shigeo Abe
Graduate School of Science and Technology, Kobe University
Kobe Japan
abe@eedept.kobe-u.ac.jp
Abstract
To resolve unclassifiable regions for pairwise support
vector machines, decision acyclic graph support vector
machines have been proposed. But by this architecture,
the generalization ability depends on the tree structure.
In this paper, to improve the generalization ability, we
propose to optimize the structure so that the class pairs
with higher generalization abilities are put in the upper
nodes of the tree. We show the effectiveness of our method
for some benchmark data sets.
1. Introduction
Support vector machines are originally formulated for
two-class classification problems [1]. But since decision
functions of two-class support vector machines are di-
rectly determined to maximize the generalization ability,
an extension to multiclass problems is not unique. There
are roughly three ways to solve this problem: one-against-
all, pairwise, and all-at-once classifications.
Original formulation by Vapnik [1] is one-against-all
classification, in which one class is separated from the re-
maining classes. By this formulation, however, unclassifi-
able regions exist. Instead of discrete decision functions,
Vapnik [2, p. 438] proposed to use continuous decision
functions. Namely, we classify a datum into the class
with the maximum value of the decision functions. Inoue
and Abe [3] proposed fuzzy support vector machines, in
which membership functions are defined using the deci-
sion functions. Abe [4] showed that support vector ma-
chines with continuous decision functions and fuzzy sup-
port vector machines are equivalent.
In pairwise classification, the n-class problem is con-
verted into n(n − 1)/2 two-class problems. Kreßel
[5] showed that by this formulation, unclassifiable re-
gions reduce, but still they remain. To resolve unclas-
sifiable regions for pairwise classification, Platt, Cristian-
ini, and Shawe-Taylor [6] proposed decision-tree-based
pairwise classification called Decision Directed Acyclic
Graph (DDAG). Pontil and Verri [7] proposed to use
rules of a tennis tournament to resolve unclassified re-
gions. Not knowing their work, Kijsirikul and Ussi-
vakul [8] proposed the same method and called it Adap-
tive Directed Acyclic Graph (ADAG). The problem with
DDAGs and ADAGs is that the generalization regions de-
pend on the tree structure [4]. Abe and Inoue [9] extended
one-against-all fuzzy support vector machines to pairwise
classification.
In all-at-once formulation we need to determine all the
decision functions at once [10, 11], [2, pp. 437–440].
But this results in simultaneously solving a problem with
a larger number of variables than the above mentioned
methods.
In this paper, we propose to determine the optimal
structure of a decision tree. By decision trees, the un-
classifiable regions are assigned to the classes associated
with leaf nodes. Thus if class pairs with low general-
ization ability are assigned to the leaf nodes, associated
decision functions are used for classification. Thus the
classes that are difficult to be separated are classified us-
ing the decision functions determined for these classes.
As a measure to estimate the generalization ability, we
can use any measure that is developed for estimating the
generalization ability for two-class problems, because in
DDAGs and ADAGs, decision functions for all the class
pairs are needed to be determined in advance.
In Section 2, we explain two-class support vector ma-
chines, and in Section 3 we discuss pairwise SVMs. In
Section 4, we explain DDAGs and ADAGs and in Section
5, we propose to optimize DDAGs and ADAGs. In Sec-
tion 6, we demonstrate the effectiveness of our method by
computer experiments.
2 Two-class Support Vector Machines
Let m-dimensional inputs xi (i = 1, . . . , M ) belong
to Class 1 or 2 and the associated labels be yi = 1 for
Class 1 and −1 for Class 2. Let the decision function be
D(x) = wt x + b, (1)
where w is an m-dimensional vector, b is a scalar, and
yi D(xi ) ≥ 1 − ξi for i = 1, . . . , M. (2)
Here ξi are nonnegative slack variables.
x2
The distance between the separating hyperplane
D(x) = 0 and the training datum, with ξi = 0, near-
D12(x) = 0
est to the hyperplane is called margin. The hyperplane D13(x) = 0 Class 1

D(x) = 0 with the maximum margin is called optimal

separating hyperplane.
To determine the optimal separating hyperplane, we
minimize Class 2
Class 3

1
M
2
w + C ξi (3) D23(x) = 0
2 i=1

subject to the constraints:

0 x1
yi (w xi + b) ≥ 1 − ξi
t
for i = 1, . . . , M, (4)
Figure 1. Unclassifiable regions by the pair-
where C is the margin parameter that determines the wise formulation
tradeoff between the maximization of the margin and
minimization of the classification error. The data that sat-
isfy the equality in (4) are called support vectors.
Let the decision function for class i against class j,
To enhance separability, the input space is mapped
with the maximum margin, be
into the high-dimensional dot-product space called fea-
ture space. Let the mapping function be g(x). If the dot Dij (x) = wij
t
x + bij , (8)
product in the feature space is expressed by H(x, x
) =
g(x)t g(x), H(x, x
) is called kernel function, and we do where wij is an m-dimensional vector, bij is a scalar, and
not need to explicitly treat the feature space. The kernel Dij (x) = −Dji (x).
functions used in this study are as follows: The regions
1. Dot product kernels Ri = {x | Dij (x) > 0, j = 1, . . . , n, j = i} (9)



H(x, x ) = x x .
t
(5) do not overlap and if x is in Ri , we classify x into class i.
If x is not in Ri (i = 1, . . . , n), we classify x by voting.
2. Polynomial kernels Namely, for the input vector x we calculate
H(x, x
) = (xt x
+ 1)d , (6)

n
Di (x) = sign(Dij (x)), (10)
where d is an integer. j=1,j=i

3. RBF kernels where 

2
H(x, x ) = exp(−γ x − x  ),
where γ is a positive parameter for slope control.
To simplify notations, in the following we discuss sup-
port vector machines with the dot product kernel. The
extension to the feature space is straightforward.
3 Pairwise Support Vector Machines
In pairwise support vector machines, we determine the
decision functions for all the combinations of class pairs.
In determining a decision function for a class pair, we use
the training data for the corresponding two classes. Thus,
in each training the number of training data is reduced
considerably compared to one-against-all support vector
machines, which use all the training data. But the number
of decision functions is n(n − 1)/2, compared to n for
one-against-all support vector machines, where n is the
number of classes.
1 for x ≥ 0,
sign(x) = (11)
(7) −1 for x < 0
and classify x into the class
arg max Di (x). (12)
i=1,...,n
If x ∈ Ri , Di (x) = n − 1 and Dk (x) < n − 1.
Thus x is classified into class i. But if any of Di (x) is
not n, (12) may be satisfied for plural i’s. In this case,
x is unclassifiable. If the decision functions for a three-
class problem are as shown in Fig. 1, the shaded region is
unclassifiable since Di (x) = 1 (i = 1, 2, and 3).
4 Decision-tree Based Support Vector Ma-
chines
4.1 Decision Directed Acyclic Graphs
Fig. 2 shows the decision tree for the three classes
shown in Fig. 1. In the figure, i shows that x does not
 1 Training of a DDAG is the same with conventional
2 D12(x)
− determine n(n − 1)/2 decision functions for an n-class
− 3
2 1
1 2
D13(x) D32(x)
3 3
− − − −
3 1 2 3
Class 1 Class 3 Class 2
Figure 2. Decision-tree-based pairwise clas-
sification
x2
D12(x) = 0
D13(x) = 0 class 1
class 2
class 3
D23(x) = 0
0 x1
Figure 3. Generalization region by decision-
tree-based pairwise classification
belong to class i. As the top-level classification, we can
choose any pair of classes. And except for the leaf node if
Dij (x) > 0, we consider that x does not belong to class
j, and if Dij (x) < 0 not class i. Then if D12 (x) > 0,
x does not belong to Class 2. Thus it belongs to either
Class 1 or 3 and the next classification pair is Classes 1
and 3. The generalization regions become as shown in
Fig. 3. Unclassifiable regions are resolved but clearly the
generalization regions depends on the tree structure.
Classification by a DDAG is executed by list process-
ing. Namely, first we generate a list with class numbers
as elements. Then, we calculate the decision function,
for the input x, corresponding to the first and the last ele-
ments. Let these classes be i and j and Dij (x) > 0. We
delete the element j from the list. We repeat the above
procedure until one element is left. Then we classify x
into the class that corresponds to the element number. For
Fig. 2, we generate the list {1, 3, 2}. If D12 (x) > 0, we
delete element 2 from the list; we obtain {1, 3}. Then if
D13 (x) > 0, we delete element 3 from the list. Since
only 1 is left in the list, we classify x into Class 1.
pairwise support vector machines. Namely, we need to
problem. The advantage of DDAGs is that classification
is faster than conventional pairwise support vector ma-
chines or pairwise fuzzy support vector machines. In a
DDAG, classification can be done by calculating (n − 1)
decision functions.
4.2 Adaptive Directed Acyclic Graphs
An ADAG is based on the rules of tennis tournament.
For three-class problems, an ADAG has an equivalent
DDAG. Reconsider the example shown in Fig. 1. Let
the first round matches be {Class 1, Class 2} and {Class
3}. Then for an input x, in the first match, x is classified
into Class 1 or Class 2, and in the second match x is clas-
sified into Class 3. Then the second round match is either
{Class 1, Class 3} or {Class 2, Class 3} according to the
outcome of the first round match. The resulting general-
ization region for each class is the same as those shown in
Fig. 3. Thus for a three-class problem there are three dif-
ferent ADAGs, each having an equivalent DDAG. When
the number of classes is more than three, we can show that
the set of ADAGs is included in the set of DDAGs [4].
According to the computer simulations [12, 8], classifica-
tion performance of the two methods are almost identical.
5 Optimizing Decision Trees
Classification by DDAGs or ADAGs is faster than by
pairwise fuzzy SVMs. But the problem is that the gener-
alization ability depends on the structure of decision trees.
In DDAGs, the unclassifiable regions are assigned to the
classes associated with the leaf nodes. For example, by
the DDAG for the three-class problem shown in Fig. 2,
the unclassifiable region is assigned to Class 3, which is
associated with the leaf node D32 (x) as shown in Fig. 3.
Since any ADAG is converted into a DDAG, the above
discussions hold for ADAGs. Namely, the unclassifiable
regions are assigned to the classes associated with the leaf
nodes of the equivalent DDAG.
Thus, if we put the class pairs that are easily separated
in the upper nodes, unclassifiable regions are assigned to
the classes that are difficult to be separated. This means
that the class pairs that are difficult to be separated are
classified by the decision boundaries that are determined
by these pairs.
In forming a DDAG or an ADAG, we need to train
SVMs for all pairs of classes. Thus, in determine the op-
timal structure, we can use any of the measures that are
developed for estimating the generalization ability. In the
computer experiment we use the following estimate of the
generalization error Eij for classes i and j:
SVij
Eij = , (13)
Mij
where SVij is the number of support vectors for classes i
and j and Mij is the number of training data for classes i
and j [1].
Therefore, the algorithm to determine the DDAG
structure for an n-class problem is as follows:
1. Generate the initial list: {1, . . . , n}.
2. If there are no generated lists, terminate the algo-
rithm. Otherwise, select a list and select the class
pair (i, j) with the highest generalization ability
from the list.
3. If the list selected at Step 2 has more than two ele-
ments, generate two lists deleting i or j from the list.
Go to Step 2.
Figure 4 shows an example of a four-class problem. At
first we generate the list {1, 2, 3, 4}. Then at the top level
we select a pair of classes that has the highest generaliza-
tion ability from Classes 1 to 4. Let them be Classes 1 and
2. Then we generate the two lists: {2, 3, 4} and {1, 3, 4}.
We iterate the above procedure for the two lists.
(1, 2, 3, 4)
1 vs 2
1 2
(1, 3, 4) 3 vs 4 2 vs 3 (2, 3, 4)
Figure 4. Determination of DDAG for a four-
class problem
The above procedure determines the structure off-line.
We can determine the structure while classifying x as fol-
lows (An extension to ADAGs is straightforward.):
1. Generate the initial list: {1, . . . , n}.
2. Select the class pair (i, j) with the highest general-
ization ability from the list. Let x be on the Class i
side of the decision function. Delete j from the list.
Otherwise, delete i.
3. If the list selected at Step 2 has more than one ele-
ment, go to Step 2. Otherwise, classify x into the
class associated with the element and terminate the
algorithm.
6 Performance Evaluation
Since the generalization abilities of the ADAGs and
DDAGs did not differ very much, in the following we
show the results for DDAGs. We evaluated performance
Table 1. Benchmark data specification
Data Inputs Classes Train. Test
Thyroid 21 3 3772 3428
Blood cell 13 12 3097 3100
Hiragana-50 50 39 4610 4610
Hiragana-105 105 38 8375 8356
Hiragana-13 13 38 8375 8356
MNIST 784 10 60000 10000
of the proposed method using the thyroid data,1 blood cell
data, hiragana data [13], and the MNIST data2 listed in
Table 1. We used the polynomial kernel with degree 3
and RBF kernel with γ = 1. The range of input was nor-
malized into [0, 1]. We trained support vector machines
by the primal-dual interior point method combined with
the decomposition technique. For the thyroid and MNIST
data we set C = 10000 and for other data sets C = 1000.
We used an Athron MP 2000 personal computer.
Table 2 shows the recognition rates of the test data for
the conventional pairwise support vector machine (SVM),
pairwise fuzzy support vector machine (FSVM) [9], and
DDAGs. Column “OPT” lists the recognition rate for the
optimum DDAG. To compare the recognition rate of the
DDAGs, the maximum, minimum, and average recogni-
tion rates for the DDAGs are also listed. Poly3 denotes
the polynomial kernel with degree 3 and RBF1 denotes
the RBF kernels with γ = 1.
The number of DDAGs for a three-class problem is 3,
but the number of classes explodes as n increases. Thus,
except for the iris and thyroid data, we randomly gen-
erate 10000 DDAGs and calculate the maximum, mini-
mum, and average recognition rates of the test data.
From the table, for the thyroid data, the recognition
rate of the optimum DDAG is the minimum recognition
rate of the DDAG among 3 DDAGs. But in general, the
recognition rate of the optimum DDAG is comparable to
the average recognition rate of DDAGs.
The recognition rates of FSVMs are equal to or better
than the average recognition rates of DDAGs and compa-
rable to the maximum recognition rates of DDAGs.
For pairwise classification, training time is the same
for fuzzy SVMs and DDAGs. But the advantage of
DDAGs and ADAGs is short classification time. Table
3 lists classification time of the blood cell and hiragna-50
test data. Classification by DDAGs is much faster than by
SVMs and FSVMs.
7 Conclusions
In this paper we proposed to optimize the structure
of a decision acyclic graph support vector machine by
putting the class pairs with higher generalization ability
at the upper nodes of the decision tree. According to the
1 ftp://ics.uci.edu: pub/machine-learning-databases
2 http://yann.lecun.com/exdb/mnist/
 Table 2. Recognition rates of pairwise SVMs (%)
Data Kernel SVM FSVM DDAG
Max. Min. Ave OPT
Thyroid Poly3 97.81 97.87 97.89 97.81 97.86 97.81
RBF1 97.26 97.37 97.40 97.26 97.34 97.26
Blood Poly3 92.07 92.77 93.00 92.32 92.47 92.65
Cell RBF1 91.93 92.23 92.41 91.84 92.08 92.00
Hiragana-50 Poly3 98.57 98.89 99.37 98.52 98.75 98.61
RBF1 98.37 98.85 99.22 98.29 98.56 98.50
Hiragana-105 Poly3 100 100 100 100 100 100
RBF1 99.98 100 100 99.98 99.99 99.99
Hiragana-13 Poly3 99.63 99.66 99.72 99.58 99.64 99.66
RBF1 99.61 99.65 99.70 99.55 99.60 99.63
MNIST Poly3 97.85 98.01 97.99 97.91 97.96 97.91
RBF1 97.18 97.78 97.49 97.31 97.41 97.54

Advances in Neural Information Processing Systems

Table 3. Classification time comparison (s)
Data Parm DDAG SVM FSVM
Blood cell Poly3 0.9 1.2 2.4
RBF1 0.7 2.6 5.2
Hiragana-50 Poly3 15 66 114
RBF1 17 76 159
computer experiments for several benchmark data sets,
the optimized DDAGs show the average recognition rates
of the DDAGs.
References
[1] V. N. Vapnik. The Nature of Statistical Learning
Theory. Springer-Verlag, London, UK, 1995.
[2] V. N. Vapnik. Statistical Learning Theory. John
Wiley & Sons, New York, NY, 1998.
[3] T. Inoue and S. Abe. Fuzzy support vector machines
for pattern classification. In Proceedings of Interna-
tional Joint Conference on Neural Networks (IJCNN
’01), volume 2, pages 1449–1454, July 2001.
[4] S. Abe. Analysis of multiclass support vector ma-
chines. In Proceedings of International Conference
on Computational Intelligence for Modelling Con-
trol and Automation (CIMCA’2003), pages 385–
396, Vienna, Austria, February 2003.
[5] U. H.-G. Kreßel. Pairwise classification and support
vector machines. In B. Schölkopf, C. J. C. Burges,
and A. J. Smola, editors, Advances in Kernel Meth-
ods: Support Vector Learning, pages 255–268. The
MIT Press, Cambridge, MA, 1999.
[6] J. C. Platt, N. Cristianini, and J. Shawe-Taylor.
Large margin DAGs for multiclass classification. In
S. A. Solla, T. K. Leen, and K.-R. Müller, editors,
12, pages 547–553. The MIT Press, Cambridge,
MA, 2000.
[7] M. Pontil and A. Verri. Support vector machines for
3-d object recognition. IEEE Transactions on Pat-
tern Analysis and Machine Intelligence, 20(6):637–
646, 1998.
[8] B. Kijsirikul and N. Ussivakul. Multiclass sup-
port vector machines using adaptive directed acyclic
graph. In Proceedings of International Joint Con-
ference on Neural Networks (IJCNN 2002), pages
980–985, 2002.
[9] S. Abe and T. Inoue. Fuzzy support vector machines
for multiclass problems. In Proceedings of the Tenth
European Symposium on Artificial Neural Networks
(ESANN”2002), pages 116–118, Bruges, Belgium,
April 2002.
[10] K. P. Bennett. Combining support vector and mathe-
matical programming methods for classification. In
B. Schölkopf, C. J. C. Burges, and A. J. Smola, ed-
itors, Advances in Kernel Methods: Support Vec-
tor Learning, pages 307–326. The MIT Press, Cam-
bridge, MA, 1999.
[11] J. Weston and C. Watkins. Support vector machines
for multi-class pattern recognition. In Proceed-
ings of the Seventh European Symposium on Artifi-
cial Neural Networks (ESANN’99), pages 219–224,
1999.
[12] C. Nakajima, M. Pontil, and T. Poggio. People
recognition and pose estimation in image sequences.
In Proceedings of International Joint Conference on
Neural Networks (IJCNN 2000), volume IV, pages
189–194, 2000.
[13] S. Abe. Pattern Classification: Neuro-fuzzy Meth-
ods and Their Comparison. Springer-Verlag, Lon-
don, UK, 2001.