Scarpa, G.; Gaetano, R.; Haindl, M. & Zerubia, J.

;
IEEE Trans. on Im. Proc., 2009, 18:8, pp. 1830-1843

Introduction Hierarchical Texture Modeling

The problem of texture segmentation The Hierarchical Multiple Markov Chain (HMMC) Model

Texture Fragmentation and Reconstruction (TFR)

Overall Scheme and Initialization Context-based analysis via spatial clustering Bottom-up texture formation via region merging

Early results
The Prague Segmentation Benchmark Application to the Berkley dataset

Land Classification of VHR remote sensing images

Results on IKONOS satellite images

According to the different application goals, an image segmentation process can use visual information of many kinds
Intensities, colors, shapes, etc

An heterogeneous (and of course challenging) case is the one of texture segmentation:

Except for trivial cases, texture can have a deep heterogeneous visual appearence. An example from the remote sensing domain:

San Diego, CA, USA

1m resolution, pansharpened

IKONOS / Planetek ITALIA

4

Except for trivial cases, texture can have a deep heterogeneous visual appearence An example from the remote sensing domain:

1. 2. 3.

Dense Urban area

Residential area
Vegetation/non-urban area

San Diego, CA, USA

1m resolution, pansharpened

IKONOS / Planetek ITALIA

5

The texture of a surface represent its local visual appearance, where this locality has to be meant at multiple scales:

Fine scale: 3 textures Homogeneous in color, microtextured

The texture of a surface represent its local visual appearance, where this locality has to be meant at multiple scales:

Zoom out (coarser scale): 1 texture Regular pattern, 3 colors

The texture of a surface represent its local visual appearance, where this locality has to be meant at multiple scales:

Zoom out (coarser scale): 6 texture Structured/non-structured patterns

The texture of a surface represent its local visual appearance, where this locality has to be meant at multiple scales:

Zoom out (coarser scale): 1 texture Many colors, heterogeneous structures

The texture of a surface represent its local visual appearance, where this locality has to be meant at multiple scales:

Key points
Description of visual appearance Focus on scales

Classical Approaches
Pixel-based analysis of low order neighbourhood Fixed scales
10

Let us consider the following image patch:

Three color states

Transitions in the space

Multiple Markov Chains Transition probability in the direction j:

= Pr(+1 = | = , = )

j-th MC: MMC:

(|)

11

Highly interacting states can be aggregated

The red and black color states present a strong interaction in some directions

12

Highly interacting states can be aggregated

Merging the red and black provides a 2-state representation. A hierarchy tree keeps track of the merging.

13

Highly interacting states can be aggregated

The last (trivial) merge completes the hierarchy definition.

14

A whole image can be seen as a combination of textures

Two main issues for building the H-MMC model Retrieve elementary states (leaves) Study spatial interactions over different scales
15

H-MMC Model Specification

= (n) , , =

(n) is the set of state at scale n =1 () = pj =1

n

P = P (n)

( |)

,,

is a suitable tree structure

H-MMC Short Specification

L ,

, :

(L) , () ,

(L1) , (1) ,

(1) , (1) ,

16

A hierarchical image segmentation algorithm:

Provides a set of nested segmentation maps

Pixel level processing

Color Based Classification

Region level processing

Spatial Context Based Clustering

Top-down Image fragmentation

Region Merging

Bottom-up Image reconstruction

17

Color map

Pixel level processing

Color Based Color Based Classification Classification

Spatial Context Based Clustering

Region level processing

Region Merging

CBC Block

Provides a partition in regions with homogeneous color / micro-textures

18

Color map (partial view)

Pixel level processing

Color Based Color Based Classification Classification

Spatial Context Based Clustering

Region level processing

Region Merging

CBC Block

Focus is now on connected component and spatial interaction among them

19

SBC map (partial view)

Pixel level processing

Color Based Classification

Region level processing

Spatial Context Spatial Context Based Clustering Based Clustering

Region Merging

SBC Block

Connected components are grouped according to their spatial context

20

Hierarchical map

Pixel level processing

Color Based Classification Spatial Context Based Clustering

Region level processing

Region Merging Region Merging

Region Merging Block

Regions are sequentially merged to form the hierarchical structure and map
21

Hierarchical map

Pixel level processing

Color Based Classification Spatial Context Based Clustering

Region level processing

Region Merging Region Merging

Region Merging Block

Regions are sequentially merged to form the hierarchical structure and map
22

Hierarchical map

Pixel level processing

Color Based Classification Spatial Context Based Clustering

Region level processing

Region Merging Region Merging

Region Merging Block

Regions are sequentially merged to form the hierarchical structure and map
23

Hierarchical map

Pixel level processing

Color Based Classification Spatial Context Based Clustering

Region level processing

Region Merging Region Merging

Region Merging Block

Regions are sequentially merged to form the hierarchical structure and map
24

Initial color segmentation is obtained using TS-MRFs:

Robustness to noise Control over the spatial scale

Model based on assuming data as tree structured:

Local MRFs are associated to the inner nodes of the tree Sequential approximation is performed from the root to the leaves

25

The color segmentation provides color classes and uniform connected components (fragments) In the general case, fragments of the same color can belong to different textures:
At any scale, spatial interactions among different colors change For each color, fragments with similar contexts are clustered

Fragment Features

=1

, =

k
N k=1

[L 8]

k Pr(k )
26

The feature matrices are usually too big (but redundant) To ensure reliability to clustering, PCA reduction is used:
Weighted average over spatial directions 75% of energy preserved along the state dimension

Different clustering techniques have been used with similar result:

K-means algorithm is used (reduced computational load but tuning needed) To limit human interaction, Mean-shift based clustering can be used

27

Source Image

Red mask

Reduced Features

Clustering Result

28

In the last phase, regions are sequentially merged pairwise from the elementary states (leaves of the hierarchy tree) up to the root. The merging process is driven by a metric called Texture Score (TS) The Texture Score

1 1 = = |) (|) max ( |) max (

()

(|)

At each step of the merging procedure, the state with lowest TS is merged to its dominant neighbor.
29

In the last phase, regions are sequentially merged pairwise from the elementary states (leaves of the hierarchy tree) up to the root.

The merging process is driven by a metric called Texture Score (TS)

The Texture Score

log

= min log

() ( |)

log

To reinforce the contextual term, spatial closeness is enhanced using Kulback-Leibler divergence.

30

http://mosaic.utia.cas.cz
31

http://mosaic.utia.cas.cz
32

http://mosaic.utia.cas.cz
33

http://mosaic.utia.cas.cz
34

http://mosaic.utia.cas.cz
35

http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/

36

Panchromatic segmentation

Pixel level processing

PAN-MS Fusion
Region level processing

Spectral Clustering

To preserve fine details, only the panchromatic image is processed at pixel level and segmented. The resulting connected components are featured with a spectral signature and clustered to obtain the CBC map.
37

38

39

40

H-MMC is a new family of hierarchical image models aimed at describing the textural content, based on color and contextual properties. TFR is the corresponding hierarchical image segmentation algorithm, providing a set of nested segmentation maps.

Qualifying Points
Hierarchical segmentation Quick processing Unsupervised Broad-domain applications

Future Works
Morphological fragmentation at pixel level Geometrical descriptors for shape featuring Graph-based descriptors for spatial interactions New metrics for the texture merging phase

41