Using Game Theory for Image Segmentation

Elizabeth Cassell Sumanth Kolar Alex Yakushev

21st March 2007

1 Introduction
The goal of image segmentation, is to distinguish objects from background.
Robust segmentation of structures is indeed required for a variety of image
analysis applications, including robot vision and biomedical image processing.
The digital images that we discuss in this paper are represented as matrix y of
intensity levels. We will assume that the intensity level is a fraction between
zero and one, and the intensity of a pixel at (i, j) is given by yi,j . Our goal
in segmentation, is to produce a segmentation matrix x, where xi,j can take
on only a xed number of dierent values between zero and one, corresponding
to the number of segmentation levels. For example, in the case of two levels,
xi,j ∈ {0, 1}.

2 Related Work
There are a number of approaches for image segmentation, the common ones
include: threshold techniques, boundary nding methods, region-based tech-
niques, and connectivity-preserving relaxation methods. We look at integrating
two of these approaches, boundary nding based and region-based techniques
using game theory.

2.1 Region-based method

A region-based method usually proceeds as follows: the image is partitioned
into connected regions by grouping neighboring pixels of similar intensity levels.
Adjacent regions are then merged under some criterion involving homogeneity or
sharpness of region boundaries. A common approach is to minimize an objective
function of the form
 
X 2
XX 2
E= (yi,j − xi,j ) + λ  (xi,j − xis ,js )  (1)
i,j i,j is ,js

Where is and js are indices in the neighborhood of pixel xi,j . The justication
for this objective function is simple. The rst summation is trying to minimize

1
Figure 1: An example of simple image thresholding on a noisy image. Original
image is on the left (SNR 0.67), the thresholded image is on the right.

the dierence between the classication and the pixel intensity. Thus, in the
two-level case, we would want the pixels where yi,j is small (i.e. the pixel is
dark) to correspond to xi,j = 0 (black). Ignoring the second term (setting
λ = 0) would correspond to a simple threshold method. (See gure 1)
The second part of equation 1, seeks to minimze the dierence between
classications of neighboring pixels, essentially to minimize the region boundary.
As can be seen from gure 1, in only considering the intensity value we have
misclassied a huge number of noisy pixels. Enforcing the smoothness constraint
would force, for example, a single black pixel surrounded by white pixels to
become white.
There is a number of implementations of region-based methods. We cannot
do an exhaustive search, since even in the two-level case there are 2M N pos-
sibilities (where M and N are the image width and height.) One approach is
called pyramid-linking, initially suggested by Brut, referenced from Jäne [2] and
Morel et. al. [4]. The idea is rst to construct a pyramid from the image in the
following manner: each level of the pyramid has half the pixels of the previous
level, and each higher-level pixel has the average intensity of a 4x4 pixel block
of the previous pixel. Each pixel could be related to any one of 16 pixels on the
level below. We link a pixel on a higher level in such a way that minimizes our
energy function from (1). We will end up with a tree, that we could now easily
split according to the number of segmentation levels required.
Another approach is called region growing. The idea is to start from a seed,
which could be a single pixel, placed inside the region of interest. We then take
all the pixels on the boundary of the seed, and compute the value of E if each
of those pixels was in the region. We will accept all pixels for which the value
E falls below a certain threshold, and repeat the process until no pixels can be
accepted. You can see the output of this method in gure 2.

2
Figure 2: An example of using region segmentation only

3
Figure 3: Example of the Canny edge-detection lter on an image of a canine
heart. (a)original image. (b) Detected edges

2.2 Edge-based methods

A common edge-based approach is to nd the edges of the object of interest
using image gradient information. One way is to simply look at the image
gradients, and nd places where the gradient has a peak. This procedure is
usually implemented as a simple lter. Figure 3 shows an example of one such
lter  the Canny lter. The gure shows that since intensity information is
lost, it is very dicult to nd the region of interest in these images. Many
edges are broken due to obstruction, blurring, or noise, while shadows introduce
additional, unwanted edges.
A much more advanced approach involves deformable contours. The idea is
to start with a large contour, approximately surrounding the area of interest.
Then we use the gradient information to deform the contour in such a way, that
it lies along points with largest gradient, while maintain smooth curvature. An
excellent implementation of this method was done by Li et. al. [3]. Their results
are shown in gure 4. Though they produce a very good t, their implementa-
tion is well suited to only very small images, the example shown in gure 4, for
example, is of a 64x80 image. The Matlab code they provided took 10ms per
iteration on the image in gure 4. When given a larger 300x300 image, it took
160ms per iteration. Note that it takes around 800 iterations to determine the
contour completely.
Additionally, many of these approaches suer from problems of poor localiza-
tion and oversegmentation. Overstringent criteria create fragmentation; lenient
ones overlook blurred boundaries and merge unrelated regions. Region based
segmentation is not likely to give us precise information regarding the shape and
location, boundary nding may not be feasible if some of the structures are not
well located. Clearly, neither boundary nding nor region-based segmentation
alone is not likely to provide us with all the necessary answers.

4
Figure 4: An example of an implementation of deformable contours by Li et.
al. [3]

Figure 5: Flow Diagram for game theoretic image segmentation

5
2.3 Game-theoretic integration
Our solution comes on the basis of the work done by Chakraborty and Duncan
[1]. We adapt a parallel game-theoretic decision-making procedure as shown
in the gure to image segmentation problem. The game is being played out
by a set of decision makers (or players) which in our case, will correspond to
the two segmentation modules that we want to integrate. The segmentation
problem discussed here can be formulated as a two player game. If p1 is the set
of strategies of player 1, and p2 is the set of strategies of player 2, then each
player is trying to minimize their payo function F i (p1 , p2 ). The goal is then
to nd the Nash Equilibrium of the system (p1 , p2 ), such that

F 1 (p1 , p2 ) ≤ F 1 (p1 , p2 ); F 2 (p1 , p2 ) ≤ F 2 (p1 , p2 )

We will try to move toward the Nash Equilibrium iteratively, taking k as

time index we can view the game as [1]

p1k+1 = arg min

1 1
F 1 (p1 , p2k ); p2k+1 = arg min F 2 (p1k , p2 )
p ∈P

Chakraborty and Duncan[1] give a proof of a theorem, which shows that

there always exists a NE solution if F 1 and F 2 are of the following form:

F 1 (p1 , p2 ) = f1 (p1 ) + αf21 (p1 , p2 )

F 2 (p1 , p2 ) = f2 (p2 ) + βf12 (p1 , p2 )

where α and β are some scaling constants, F 1 is bounded in p1 ∈ P 1 , F 1 is

continuously second-order dierentiable in p1 ∈ P 1 , and ∃ a closed neighborhood
u1 ⊆ P 1 such that F 1 is strongly convex in u1 . Given the above assumptions,
Chakraborty and Duncan[1] provide this theorem on the existence of a Nash
Equilibrium solution:
Theorem: For the above-given structure of F 1 (p1 , p2 ) and F 2 (p1 , p2 ), there
exists a locally stable Nash Equilibrium solution, i.e., for any p1 ∈ U 1 ⊆ P 1
and p2 ∈ U 2 ⊆ P 2 , the sequence of rational choices generated by the parallel
decision-making process converges and the limit point is a Nash equilibrium
solution if α and β satisfy the following condition:
°"µ ¶−1 µ ¶#
° ∂2 ∂2 ∂2
° −1 1 1 2 1 2
° α f1 (p ) + 1 1 f21 (p , p ) f21 (p , p )
° ∂p1 ∂p1 ∂p ∂p ∂p1 ∂p2
"µ ¶−1 µ ¶#°
∂2 ∂2 ∂2 °
−1 2 1 2 1 2 °
∗ β f2 (p ) + 2 2 f12 (p , p ) f12 (p , p ) ° < 1
2
∂p ∂p 2 ∂p ∂p 2
∂p ∂p 1 °
(2)

The proof is given in Appendix A of Chakraborty and Duncan[1], which

proves a Nash Equilibrium exists under the given conditions.
In our case, the objective functions are as follows:

6
 For the region-based module (player 1),
  
X X X
F 1 (p1 , p2 ) = min  [yi,j − xi,j ]2 + λ2  (xi,j − xi−1,j )2 + (xi,j − xi,j−1 )2 (3)

x
i,j i,j i,j
 
X X
+α  (xi,j − u)2 + (xi,j − v)2  (4)
(i,j)∈Ap̃ (i,j)∈Āp̃

where y is the intensity of the original image, x is the segmented image given
by p1 , u is the intensity of the image inside the contour given by p2 , and v is the
intensity outside the contour given by p2 . The rst term is trying to minimize
the dierence between the pixel intensity values and the found region, as well
as enforce continuity. The second term is trying to match the region and the
contour. The objective function for player 2 (the boundary nding module) is
as follows:

F 2 (p1 , p2 ) = arg max

→
−
[Mgradient (Ig , p~) + βMregion (Ir , p~)] (5)
p

where p~ is the parameterization of the contour given by p2 , Ig is the gradient

image, and Ir is the region segmented image obtained from x and p1 . Mgradient
is a measure of the match between the gradient image, Ig , and the contour given
by p~. Mregion is a measure of the match between the region-segmented image
Ir and the contour, p~. Chakraborty and Duncan show that equations 4 and 5
comply with the convexity requirements of the theorem.

3 Implementation
Our implementation somewhat simplies Chakraborty and Duncan's approach,
however, having made this simplication relieves us of the requirement of the
prior shape information. The implementation is detailed in the sections below.

3.1 Region-Based Module

We used the region growing approach discussed in section 2.1, gure 6 illustrates
the algorithm. Though it requires an initial seed, the seed only needs to be a
single pixel within the region of interest. At each iteration we look at the
neighboring pixels and compute the value of E from equation 1. The pixels
for which this value is less than a threshold, will be accepted into the region.
The choice of the threshold depends on the characteristics of the image and the
region of interest. While automatic threshold selection could be possible, it is
still highly dependent on the image type [5]. Since we worked with a variety of
images from dierent domains, we usually chose the threshold manually.

7
Figure 6: Illustration of the region growing algorithm. (a) Original image. (b)
Seed Image. (c) Pixels queried at the rst iteration are marked with a question
mark. (d) Pixels accepted at the rst iteration are checked. (e) Pixels queried
at the second iteration. (f) Pixels accepted at the second iteration are checked,
pixels marked with an x are rejected.

8
Figure 7: Illustration of the boundary nding module - Closing operation on a
binary image

3.2 Boundary Finding Module

The main work of the boundary nding module is done via a closing operation
A • B = (A ⊕ B) ª B , which is a combination of dilation followed by erosion.
While either set A or B can be thought of as an "image", A is usually considered
as the image and B is called a structuring element. The structuring element is to
mathematical morphology what the convolution kernel is to linear lter theory.
Dilation A ⊕ B is the set of all points generated by obtaining the reection of
B about its origin and then shifting this reection by x. Dilation, in general,
causes objects to dilate or grow in size; erosion, however which performs the
morhological dual of dilation, causes objects to shrink. The amount and the
way that they grow or shrink depend upon the choice of the structuring element.
The closing operation uses the image gradient to identify pixels regions in the
region segmentation, where holes/valleys need to be lled. This morphological
operation identies regions of smoothness and lls them in case a hole ( black
pixels instead of white ones) is present in those regions. However, if there are
sharp pixel dierences over a span of pixels, the holes are not lled. This results
is a smoother region as illustrated in Figure 7. Because our implementation
does not use any prior shape information, no initialization is needed for this
module.

4 Results
Our rst experiment was on a synthetic image with added noise. We compared
the output generated using our game theoretic integration method with the
output obtained using our region based segmentation module alone. We also
tested our method with real images from a variety of domains. As you can
see from the gures, our integrated game-theoretic image segmentation method

9
Figure 8: Synthetic image example. (a) Original image with noise added. (b)
Output of region-based segmentation using game-theoretic integration. (c) Out-
put of region-based segmentation module alone. (d) Seed image.

produces a contour that ts the original image signicantly better than the
output produced from the region-based method alone.

5 Conclusions
We have presented a successful game-theoretic approach to the image segmenta-
tion problem. Our approach produces better results than each of the individual
modules, and it signicantly more robust to noise. Future work in this area
could include improving the region based and boundary nding algorithms, for
example by creating a region based algorithm that learns the seed instead of
having to provide it.

10
Figure 9: Example using an image of a split papaya. (a) Original image. (b)
Boundary of the fruit interior found using the region-based segmentation mod-
ule alone, without using game theoretic integration. (c) Output using game-
theoretic integration.

Figure 10: Example using an image of a mushroom. (a) Original image. (b)
Output using game-theoretic integration.

11
Figure 11: Example using an MR image of the brain. (a) Original image.
(b) Output using game-theoretic integration. The black contour outlines the
boundary of the corpus callosum.

References
[1] Amit Chakraborty and James S. Duncan. Game-theoretic integration for
image segmentation. In Pattern Analysis and Machine Intelligence, IEEE
Transaction on., pages 1230. IEEE, 1999.
[2] Bernd Jähne. Digital Image processing. Springer, Berlin, 6th edition, 2005.

[3] Chunming Li, Chenyang Xu, Changfeng Gui, and Martin D. Fox. Level
set evolution without re-initialization: A new variational formulation. In
Pattern Analysis and Machine Intelligence, IEEE Transaction on. IEEE,
2005.

[4] Jean-Michel Morel and Sergio Solimini. Variational Methods in Image Seg-
mentation. Birkhäuser, Boston, 1995.
[5] Memet Sezgin and Bülent Sankur. Survey over image thresholding tech-
niques and quantitative performance evaluation. Journal of Electronic Imag-
ing, 13(1):146168, 2004.

12