MRI Images in RA

Longitudinal analysis of MRI images in
rheumatoid arthritis
Kelvin Ka-fai Leung
A dissertation submitted in partial fulfillment

of the requirements for the degree of
Doctor of Philosophy
of the
University of London.
Centre for Medical Image Computing

Department of Computer Science and Department of Medical Physics
University College London
November 16, 2007

2
I, Kelvin Ka-fai Leung, confirm that the work presented in this thesis is my own. Where information
has been derived from other sources, I confirm that this has been indicated in the thesis.
3
To my parents
Abstract
Rheumatoid arthritis (RA) is a chronic, systemic, autoimmune, and inflammatory joint disease. Longi-
tudinal imaging using MRI has been proposed as an effective tool for patient management and clinical
studies, but the analysis is based on visual inspection or interactive analysis. This thesis investigates
image registration and analysis algorithms to automatically quantify changes in a bone in in-vivo lon-
gitudinal MR images of an ankle in an experimental rat model of RA, and to improve the ease of use,
reproducibility, and analysis time of visual scoring systems in the hands and wrists of human patients.
In the first part of this thesis, two different methods were used to automatically quantify changes in
a bone in the experimental model of RA. The first method used segmentation propagation to delineate a
bone from the longitudinal MR images giving a global measure of temporal changes in bone volume. The
second method used rigid body registration to determine intensity change within a bone using an intensity
thresholding algorithm, and then mapped these into a common space using nonrigid registration. This
gave a local measure of temporal changes in bone lesion volume. The global bone volume was found
to be fluctuating over time. However, significant temporal changes were detected in local bone lesion
volume in 4 out of 8 identified candidate bone lesion regions, and significant difference in local bone
lesion volume between male and female subjects was detected in 3 out of 8 candidate bone lesion regions.
Some of the results were confirmed by comparison with the histology of the subjects.
The second part of this thesis described an integrated spatio-temporal segmentation algorithm,
which built on the results of the first part that bone lesions are localised in space, time, and intensity
of the difference images. A 5-dimensional feature space (3 spatial dimensions, 1 intensity dimension,
and 1 temporal dimension) was built from the longitudinal MR images after rigid body registration to
integrate the time-domain information across all the time points. The feature space was then delineated
by the mean shift algorithm to give high-intensity bone lesions as 4D segmentations. This technique
was quantitatively compared with the previous method using simulated and real MR images, and was
qualitatively compared with the histology of the subjects.
In the third part of this thesis, image registration was used to align each bone in the MR images of
the hands and wrists of human patients to a common reference coordinate system, based on a reference
atlas such as the EULAR-OMERACT rheumatoid arthritis MRI reference image atlas. This method may
be used to increase the sensitivity of visual scoring to subtle changes by reducing the impact of variation
in position, and could provide a pre-processing step for automatic quantification of erosion or synovitis.
Medical research and drug discovery rely increasingly on comparisons between MR images from
Abstract 5
large numbers of subjects, often with multiple time points for each subject. In the final part of this
thesis, the computational grid (the Grid), which provides an infrastructure for researchers in different
organisations to share their work and facilities (e.g. image data and computing resources), was used to
execute the algorithms remotely and in parallel. The algorithms in the first three parts of this thesis were
expressed in a workflow language provided by the Virtual Data System so that their execution was fully
automated. The system was also designed to keep track of data provenance that provides information
about the origin of a piece of data and the process by which it was derived. This information is crucial
for the understanding and validation of scientific results, and can be used to automatically generate the
audit trail required in the handling of electronic records in a pharmaceutical company.
Acknowledgements
I would like to express my sincere thanks to my first supervisor Prof. Derek Hill. for his guidance and
encouragement throughout my three years at KCL and UCL. My thanks are due to Dr. Mark Holden
for his guidance and discussions during the early stages of this project. I am grateful for my industrial
supervisor Dr. Nadeem Saeed for discussions and his care during my visits to GlaxoSmithKline (GSK).
I also thank other colleagues in GSK for the preparation of the experimental model and the image acqui-
sition. Thanks to Dr. Kate McLeish for the image acquisition of RA patients, and Dr. Stephen Oakley
and Dr. Bruce Kirkam for the patient recruitment, clinical assessment and various work in the RA study.
I am grateful to my second supervisor Prof. David Hawkes for his comments on my thesis. Thanks also
to Prof. Jo Hajnal and Prof. Daniel Ruckert for many interesting discussions and comments on my work
during the IXI meetings. Thanks to my English tutor Dr. Simon Williams for helping me to write better
English. I must also thank other members of the department in KCL and UCL for discussions and help.
Many thanks to my family and friends for their support during this project. In particular, I would like to
thank my wife Annie and my son Bryan for their patience and support during this project.
Contents
1 Introduction 17
1.1 Background and motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2 Image registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.3 Thesis aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.3.2 Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4 Thesis organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.5 Software and other materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.6 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.6.1 Overall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.6.2 Individual chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2 Literature review 24
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2 Imaging of joints in RA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.1 Conventional radiography (CR) . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.2 Microfocal x-ray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.3 Computed Tomography (CT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.4 Bone scintigraphy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.5 Magnetic resonance imaging (MRI) . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.6 Ultrasound (US) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3 Measuring changes in longitudinal images . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.1 Visual examination and scoring . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.2 Change in volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3.3 Subtraction after registration and the boundary shift integral . . . . . . . . . . . 28
2.4 Image registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.1 The feature-based image registration algorithm . . . . . . . . . . . . . . . . . . 29
2.4.2 The intensity-based image registration algorithm . . . . . . . . . . . . . . . . . 30
2.5 Supervised and unsupervised classifications . . . . . . . . . . . . . . . . . . . . . . . . 37
2.5.1 Unsupervised classification (or data clustering) . . . . . . . . . . . . . . . . . . 37
Contents 8
2.6 Animal Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.7 Histological analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3 Automatic Quantification of Changes in Longitudinal MR Images of Joints I 46

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2 Aims and contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3 Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4.1 Segmentation propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4.2 Otsus thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.4.3 Mathematical morphology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.5 Overviews of methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.5.2 Assumptions and observations . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.5.3 Overview of the measurement of changes in bone volume (Method 1) . . . . . . 53
3.5.4 Overview of the measurement of changes in bone lesion volume (Method 2) . . . 53
3.6 Materials and methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.6.1 Image data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.6.2 Atlas construction (used in Methods 1 and 2) . . . . . . . . . . . . . . . . . . . 54
3.6.3 Approximate delineation of the talus bone (used in Methods 1 and 2) . . . . . . 54
3.6.4 Accurate delineation of the talus bone (used in Methods 1 and 2) . . . . . . . . . 56
3.6.5 Generation of difference images (used in Method 2) . . . . . . . . . . . . . . . 57
3.6.6 Identification of potential lesion voxels (used in Method 2) . . . . . . . . . . . . 57
3.6.7 Identification of candidate bone lesion regions (used in Method 2) . . . . . . . . 57
3.6.8 Calculation of bone lesion volume (used in Method 2) . . . . . . . . . . . . . . 58
3.7 Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.7.1 Method 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.7.2 Method 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.8 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.8.1 Method 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.8.2 Method 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.9 Discussion and conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4 Automatic Quantification of Changes in Longitudinal MR Images of Joints II 77

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.3 Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.4 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.1 Mean shift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Contents 9

4.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.5.2 Image data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.5.3 Atlas construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.5.4 Determination of ROI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5.5 Generation of difference images . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5.6 Spatio-temporal segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5.7 Identification of candidate bone lesion regions . . . . . . . . . . . . . . . . . . 82
4.5.8 Calculation of bone lesion volume . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.5.9 Speed improvement of the mean shift segmentation by subsampling . . . . . . . 83
4.5.10 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.5.11 Comparison with manual segmentation of the bone lesions . . . . . . . . . . . . 84
4.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.6.1 Experiments on simulated data . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.6.2 Experiments on real data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.6.3 Comparison with histology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.6.4 Comparison with manual bone lesion volume . . . . . . . . . . . . . . . . . . . 93
4.6.5 Running time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5 Application to Clinical Data 96

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.3 Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4.1 Image data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.4.2 The standard reference image . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.4.3 Automatic bone localisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.4.5 Automatic rigid bone alignment to the standard reference image . . . . . . . . . 102
5.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.5.1 Image data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.5.2 Automatic bone localisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.5.4 Automatic rigid bone alignment to the standard reference image . . . . . . . . . 109
Contents 10
6 Use of the Grid in Automated Medical Image Analysis 121

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.3 Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.4 Grid Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.4.1 The Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.4.2 Condor and Condor MW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.4.3 The Virtual Data Grid (VDG) . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.5.1 The web-based image analysis workbench . . . . . . . . . . . . . . . . . . . . . 128
6.5.2 VDL wrapping of the image analysis algorithms . . . . . . . . . . . . . . . . . 129
6.5.3 The parallel implementation of the spatio-temporal segmentation algorithm . . . 132
6.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.6.1 The web-based image analysis workbench . . . . . . . . . . . . . . . . . . . . . 132
6.6.2 The parallel implementation of the spatio-temporal segmentation algorithm . . . 135
6.7 The evaluation of the workbench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7 Conclusions 145
7.1 Summary of findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.1.1 Automatic quantification of changes in longitudinal MR images of joints I . . . . 145
7.1.2 Automatic quantification of changes in longitudinal MR images of joints II . . . 146
7.1.3 Application to clinical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.1.4 Use of the Grid in automated medical image analysis . . . . . . . . . . . . . . . 147
7.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.2.1 Improvements to the quantification of changes in the experimental model of RA . 148
7.2.2 Other applications of the spatio-temporal segmentation . . . . . . . . . . . . . . 148
7.2.3 Automatic window width determination in the spatio-temporal segmentation . . 149
7.2.4 Improvements to automatic bone localisation and alignment . . . . . . . . . . . 149
7.2.5 Dynamic contrast enhanced (DCE) imaging of synovitis in RA patients . . . . . 152
7.2.6 Improvements to the integration of medical image analysis and the Grid . . . . . 152
7.2.7 Different ways of building the feature space in the spatio-temporal segmentation 152
7.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Appendices 153
A Appendix of Chapter 3 154

A.1 Binary opening and dilation operators . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
A.2 Average bone lesion volumes in various candidate bone lesion regions . . . . . . . . . . 154
Contents 11
B Appendix of Chapter 4 160

B.1 Average bone lesion volumes in various candidate bone lesion regions . . . . . . . . . . 160
Bibliography 160
Publications 183
List of Figures
1.1 Joint affected by rheumatoid arthritis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1 Intensity-based image registration algorithm . . . . . . . . . . . . . . . . . . . . . . . . 30

2.2 Hierarchical clustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.3 k-means clustering of the pattern in Figure 2.2(a) for k = 2. . . . . . . . . . . . . . . . 39
2.4 Mean joint scores in SCW model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.5 Histologic appearance of joint tissue in SCW model . . . . . . . . . . . . . . . . . . . . 43
2.6 Surface rendered impressions of the plantar surface of MR and X-ray CT . . . . . . . . 44
2.7 Synovial histology in RA. The intimal lining is the outer layer of the cells pointed by the
solid black arrow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.1 Principle of segmentation propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

3.2 Examples of some common structuring elements . . . . . . . . . . . . . . . . . . . . . 51
3.3 The effects of the basic morphological operations. . . . . . . . . . . . . . . . . . . . . . 52
3.4 Flow diagram to describe all the steps of Methods 1 and 2. . . . . . . . . . . . . . . . . 55
3.5 Mathematical morphological operators applied after each aggregation stage. . . . . . . . 58
3.6 Kernels in 2D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.7 Graph of volume of automatic segmentation of the talus bone of all the subjects against
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.8 Graph of volume of manual segmentation of the talus bone of subjects 1, 2 and 3 against
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.9 Baseline images and transformed follow-up images of 6 subjects . . . . . . . . . . . . . 63
3.10 Difference images of subject 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.11 Candidate bone lesion regions (in green) of the talus bone at day -4 before morphological
operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.12 Candidate bone lesion regions (in green) at day -4 of the talus bone after morphological
operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.13 Sum of the candidate bone lesion regions (in green) of the 5 time points. . . . . . . . . . 65
3.14 Candidate bone lesion regions in the talus bone. . . . . . . . . . . . . . . . . . . . . . . 65
3.15 Surface rendering of the candidate bone lesion regions . . . . . . . . . . . . . . . . . . 66
List of Figures 13
3.16 The average bone lesion volume in the talus bone in the left ankle in all the candidate
bone lesion region. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.17 Graph of average bone lesion volume of male and female subjects in region 24 against
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.18 Comparison between histology section and MR image of subject 2. . . . . . . . . . . . . 69
3.21 Relationship between the volume of manual bone lesion segmentation and that of the
automatic bone lesion segmentation in candidate bone lesion region 24. . . . . . . . . . 72
3.22 The graph of the average scaling factors recovered using the talus bone against time . . . 74
3.23 An example of mapping potential lesion voxels to the atlas using affine registration. . . . 76
4.1 Comparison between the histological section, real image, and the simulated image of
subject 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2 Graph of similarity index against noise standard deviation for 5 different lesion volumes. 86
4.3 Effect of window size in different dimensions. . . . . . . . . . . . . . . . . . . . . . . . 86
4.4 Effect of very small window width in the temporal dimension. . . . . . . . . . . . . . . 88
4.5 Comparison of the bone lesions segmented by OT and ST at different time points of the
talus bone of subject 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.6 Graph of average bone lesion volume of male and female subjects in region 3 obtained
by ST against time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.8 Relationship between the volume of manual bone lesion segmentation and that of the
automatic bone lesion segmentation in candidate bone lesion region 3. . . . . . . . . . . 93
5.1 Example bone localisation scores from 0 to 5. . . . . . . . . . . . . . . . . . . . . . . . 100

5.2 Diagrams illustrating the hierarchical approach and multiple starting estimates. . . . . . 101
5.3 Points used to determine the orientation of the 2nd metacarpal bone in the standard ref-
erence image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.4 Example baseline images of a patient acquired by the three different MR sequences . . . 104
5.5 The results of automatic bone localisation. . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.6 The results of automatic bone localisation. . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.7 The results of spatio-temporal segmentation using clinical data. . . . . . . . . . . . . . . 110
5.8 The graph of the bone lesion volume in the 3rd metacarpal bone in logarithmic scale
against time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.9 The graph of the bone lesion volume in the capitate in logarithmic scale against time. . . 111
5.10 The graph of the bone lesion volume in hamate in logarithmic scale against time. . . . . 112
5.11 The results of automatic bone alignment of the 5th metacarpal bone. . . . . . . . . . . . 113
5.12 The results of automatic bone alignment of the 3rd metacarpal bone. . . . . . . . . . . . 114
List of Figures 14
5.13 The failed results of automatic bone localisation - patient image I11. . . . . . . . . . . . 116
5.14 Corrected FOV for the automatic bone localisation - patient image I11. . . . . . . . . . . 117
5.15 The failed results of automatic bone localisation - lunate. . . . . . . . . . . . . . . . . . 117
5.16 The capitate bone of the patient ppt02. . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.1 The architecture of the web-based image analysis workbench. . . . . . . . . . . . . . . 129

6.2 Example compound workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.3 The RLS Upload web page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.4 The VDS Catalogue web page - transformations. . . . . . . . . . . . . . . . . . . . . . 134
6.5 The VDS Catalogue web page - the parameters of a transformation (rigid image regis-
tration). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
6.6 The VDS Catalogue web page - derivations. . . . . . . . . . . . . . . . . . . . . . . . . 136
6.7 The VDS Catalogue web page - data provenance. . . . . . . . . . . . . . . . . . . . . . 136
6.8 The VDS Catalogue web page - data provenance. . . . . . . . . . . . . . . . . . . . . . 137
6.9 The Job Status web page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.10 The RLS Query web page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.11 The load of the computer cluster at Rochester Institute of Technology over a year
(http://cluster.rit.edu/ganglia/). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
6.12 The load of the computer cluster at Worcester Polytechnic Institute over a year
(http://wpi-master.wpi.edu/ganglia/). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.1 Results of rigid registration of longitudinal MR brain images of MS patients. . . . . . . 150

7.2 Results of the spatio-temporal segmentation of longitudinal MR brain images of MS
patients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.3 Deformation vector fields of the first follow-up image with and without MS lesions. . . . 151
A.1 Example demonstrating the effects of the morphological operations . . . . . . . . . . . . 155

A.2 Graph of average bone lesion volume of male and female subjects in region 1 against time.155
A.3 Graph of average bone lesion volume of male and female subjects in region 7 against time.156
A.4 Graph of average bone lesion volume of male and female subjects in region 24 against
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
List of Figures 15
time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
B.1 Graph of average bone lesion volume of male and female subjects in region 1 obtained
List of Tables
3.1 Mean similarity index and mean % difference between the volume of automatic and
manual segmentations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2 Mean similarity index and mean % difference between the volume of repeated manual
segmentations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.3 Results of the comparison of the bone lesion volume between different time points and
between male and female subjects using ANOVA. . . . . . . . . . . . . . . . . . . . . . 68
3.4 Results of pair comparisons that are significant at P < 0.05 between different time points
in regions 1, 24, 37, 93 and 121. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.5 Comparison between histology and the results. . . . . . . . . . . . . . . . . . . . . . . 68
3.6 Average scaling factors recovered using the 9-dof registration of the talus and calcaneous 74
4.1 Effect of varying window width around the window width (0.24mm, 0.24mm, 0.24mm,
400, 1.0) at noise standard deviation 914. . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2 Results of the comparison of the bone lesion volume between different time points, and
between male and female subjects using ANOVA. . . . . . . . . . . . . . . . . . . . . . 89
4.3 Results of pair comparisons that are significant at P < 0.05 between different time points
in regions 1, 3, 7, 8, 9, and 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.4 Comparisons between the approximations and the exact solutions of the mean shift seg-
mentations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.5 Comparison between histology and the results. . . . . . . . . . . . . . . . . . . . . . . 91
5.1 The bone hierarchy and multiple starting estimates for the automatic bone localisation. . 105
5.2 The visual scores of the automatic localisation of the individual bones of the 21 images
from 6 patients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.3 The mapping between the image names (I1,...I21) and the time points of the patients
(ppt01,...,ppt06). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.4 The bone hierarchy for the automatic rigid bone alignment to the standard reference image.112
5.5 The accuracy of the automatic alignment of individual metacarpal bones. . . . . . . . . . 115
Chapter 1
Introduction
1.1 Background and motivation

Rheumatoid arthritis (RA) is a common, chronic, systemic, autoimmune inflammatory disease that
causes disability, chronic ill-health, and premature mortality [AA95, Fel02]. It targets synovial joints,
in which there is a massive accumulation of blood-borne cells such as T cells and macrophages. Blood
vessels are formed to support this new tissue and the whole mass is called a pannus (see Figure 1.1).
Progressive erosion to cartilage and bone leads to disability in patients. RA affects about 0.5% 1%
of the population (ratio of female to male patients is about 3:1). 75% of all patients are disabled by the
third years after the start of the disease. The life expectancy of a patient is reduced by 4 10 years.
The average cost of the disease ranges from e6 815 e48 429 per patient per annum depending on
factors such as the type of drug used and the age of the patient (Loss of employment is a major indirect
cost in young patients.) [RB04]. In the UK, there were around 386 600 people with adult-onset RA in
2000 [STW+ 02]. Although the aetiology of the disease is unknown, better understanding of the roles
of inflammatory mediators has led to innovative therapies [Fel02]. In addition, sensitive biomarkers of
disease progression are being developed, so that the performance of candidate disease modifying drugs
can be quantified in a short period of time.
Figure 1.1: Joint affected by rheumatoid arthritis. Excerpted from http://www.cchs.net/health/health-

info/docs/0700/0739.asp?indexi=4924.
1.1. Background and motivation 18
The development and use of imaging biomarkers are becoming more important and critical in drug
discovery. The US Food and Drug administration (FDA) report Innovation/Stagnation: Challenge and
Opportunity on the Critical Path to New Medical Products addresses the recent slowdown in innova-
tive medical therapies submitted to the FDA for approval, and describes the urgent need to modernise
the medical product development process - the Critical Path - to make product development more pre-
dictable and efficient. The result of this report is the Critical Path Initiative, which has identified that
Imaging is a key technology for assessing, accelerating the development of, and guiding the use of
new therapeutic options1 . In addition, the use of imaging biomarkers can greatly facilitate all phases
of drug discovery2 , e.g. during pre-clinical studies, imaging biomarkers can help to confirm drug ac-
tivities in-vivo and explore the drug exposure-response relationship. However, there are some potential
problems regarding the precision of imaging biomarkers3 : (a) in order to minimise the variation in
images in multi-centre clinical trials, imaging techniques must be standardised across multi-centres, and
cross-consistency must be established within and across imaging systems; (b) in order to minimise the
variability in the results of clinical trials due to human interaction, manual image processing techniques
should be avoided, and automatic algorithms should be developed. Therefore, it is important to develop
automatic image analysis algorithms for quantifying changes in imaging biomarkers.
Longitudinal imaging, in which the same subject is imaged repeatedly over time, allows the moni-
toring of the disease progression and the quantification of changes between different time points. How-
ever, existing techniques for analysing longitudinal images often compare each image in the temporal
sequence to the baseline image in order to measure small changes between different time points. This
pair-wise analysis of images does not make full use of the time-domain information available in the tem-
poral sequence. Therefore, the results are likely to be less consistent and more prone to artifacts present
in one follow-up image (e.g. geometric distortions in MR images) [SD04]. It is thus desirable to make
use of time-domain information in the analysis, so that the results could be more consistent, sensitive to
small changes, and robust to errors such as image noise.
Medical research and drug discovery rely increasingly on making comparisons between images
from large numbers of subjects, often with multiple time points. New challenges emerge when trying to
analyse such large numbers of images. For example, (a) image analysis algorithms, such as nonrigid im-
age registration algorithms [RSH+ 99] and the ones described in this thesis, can be very computationally
demanding. When they are used to analyse large cohorts, hundreds of computing tasks can take weeks
to complete on a single desktop computer; (b) the size of the image data and the results is huge (often
many terabytes). Both the image data and the results need to be stored safely, and retrieved easily when
required. The computational grid (the Grid), which provides an infrastructure for the sharing of human
and computer resources, has the potential to address some of these challenges.
This thesis focuses on using image registration and image analysis algorithms that incorporate time-
1 TheCritical Path to New Medical Products (http://www.fda.gov/oc/initiatives/criticalpath/initiative.html) March 2006

2 Imaging as a Biomarker for Development of Human Therapeutics, Jeffrey L. Evelhoch, ISMRM plenary, May 2006
3 Regulatory Opportunities & Challenges of Imaging as a Drug Development Tool (www.fda.gov/cder/regulatory/medImaging
/imagingKeyNote.ppt) G. Mills, May 2005

1.2. Image registration 19
domain information to automatically and sensitively measure small changes in longitudinal MR images
in RA in order to quantify the disease progression. It also investigates the potential of Grid technologies
to address various challenges in the automated analysis of large volumes of image data. Although the
disease in question is RA, the methods developed in this thesis have potential use for the quantification
of the progression of other diseases, such as multiple sclerosis.
1.2 Image registration

In this thesis, image registration refers to the determination of the spatial alignment between images of
the same or different subjects, acquired with the same or different modalities [HBHH01]. The spatial
alignment can be described as a geometric mapping or transformation T that transforms a position x
from one image to another.
There are several transformations that can be used to describe the spatial alignment, depending
on the nature and motion of the object in the image, and the nature of the imaging system. The rigid
body transformation has six degrees of freedom that include three translations and three rotations. It is
widely used in medical applications where the structures of interest are rigid, such as bone. The affine
transformation has twelve degrees of freedom that include the rigid body transformation, three scaling,
and three skewing. It can be used to compensate for the skewing and scaling errors introduced by the
miscalibration in some imaging systems such as CT and MRI. However, many more degrees of freedom
are needed to describe the tissue deformation in most organs in the body like the brain atrophy occurring
in patients with Alzheimers disease. This transformation is called nonrigid or warping transformation
and is also used to describe the shape and size variations in the organs in different subjects.
Image registration algorithms determine the geometric transformation T based on image feature,
image intensity, or a combination of them. Feature-based registration algorithms extract image features
like points and surfaces from images either interactively or automatically. The features are then used to
align the images to give T . On the other hand, intensity-based registration algorithms calculate similarity
measures directly from the voxel values of images to evaluate their similarity or dis-similarity. T is
determined iteratively by optimising the similarity measures of the images. Finally, feature-based and
intensity-based registration algorithms can be combined by first creating derived images from the image
features extracted from the original images, and then registering the derived images using intensity-based
registration.
Further information about image registration and its applications can be found in [HBHH01,
HHH01, MV98, PMV03, CHH04].
1.3 Thesis aims

1.3.1 Overview
The main aims of this thesis are (a) to investigate whether small changes in longitudinal MR images of
the joint in RA can be automatically and sensitively measured using image registration and other image
analysis algorithms by incorporating time-domain information; (b) to demonstrate the potential of the
1.4. Thesis organisation 20
Grid in addressing various challenges in the automated analysis of large volumes of image data.
In Chapter 3, two methods are used to automatically quantify small changes in longitudinal MR
images of the joint in an experimental model of RA. The first method uses nonrigid registration to
accurately delineate a bone, and measures the changes in the bone volume. The results are validated
by comparing them with manual segmentations of the bone. The second method analyses longitudinal
intensity changes within a bone in a reference coordinate system, and measures the changes in the bone
lesion volume. The results are validated by comparing them to the disease model, histological data and
the manual segmentations of bone lesions. In order to improve the sensitivity to small changes and
robustness to errors, Chapter 4 investigates the incorporation of time-domain information in the analysis
of longitudinal intensity changes within a bone. Building on the results of Chapter 3, a spatio-temporal
segmentation method is developed to delineate bone lesions from a 5D feature space that is built by
using the (x, y, z, intensity, time) value of each voxel. The results are validated by comparing them to
the disease model, histological data and the manual segmentations of bone lesions.
In Chapter 5, the method developed in Chapter 4 is applied to longitudinal MR images of human
RA patients. Since there are 5 meta-carpal bones and 8 carpal bones in the hand and the wrist, a method
is developed to automatically localise and roughly delineate the meta-carpal and carpal bones in order
to define regions of interest around the bones for further processing, such as accurate bone delineation.
As part of the validation of the disease progression, the OMERACT RAMRIS scoring system is used
to measure joint damage in the MR images. A method is developed to automatically register the meta-
carpal and carpal bones to a reference image in order to improve the ease of use, and the intra- and
inter-rater variability of the scoring system.
In Chapter 6, image analysis algorithms are integrated with Grid technologies to develop a web-
based workbench. Using the image analysis algorithms described in chapters 3 and 4 as exemplars,
the workbench demonstrates the potential of the Grid in addressing various challenges in the automated
analysis of large volumes of image data, such as the demand for computational and storage resources, by
allowing the sharing of computer and human resources between different institutions, the management
of workflows, and the tracking of data provenance.
1.3.2 Aims
To develop a methodology to incorporate time-domain information in the analysis of longitudinal
images in order to improve the sensitivity to small changes and robustness to errors.
To demonstrate the potential of the Grid in addressing various challenges in the automated analysis
of large volumes of image data.
1.4 Thesis organisation

This chapter describes the background, motivations, aims and contribution of this thesis. Chapter 2
provides a general review of related literature that complements specific reviews and methods in the
individual chapters. Chapter 3 applies image registration and other image analysis algorithms to auto-
matically quantify small changes in the longitudinal MR images of the joint in an experimental model
1.5. Software and other materials 21
of RA. Chapter 4 provides a method to incorporate the time-domain information in the same cohort of
longitudinal MR images in order to obtain more sensitive and robust results. Chapter 5 describes the
preliminary results of applying the method developed in Chapter 4 to longitudinal MR images of human
RA patients, and investigates the use of image registration to improve the OMERACT RAMRIS scoring
method. Chapter 6 demonstrates the potential of the Grid in addressing various challenges in the auto-
mated analysis of large numbers of images by allowing the sharing of medical image data, computational
resources, storage resources and image analysis software between different institutions, the management
of workflows, and the tracking of the data provenance.
1.5 Software and other materials

Chapter 3 uses Holdens segmentation propagation [HSH02], previously applied to brain images, for the
automatic delineation of a bone, and the mathematical morphological operators and connected compo-
nent analysis in the Visualisation Toolkit [vtk03] and Insight Toolkit [itk] for the intensity analysis. The
Grid softwares used in Chapter 6 are the Globus Toolkit [glo], the Condor [conb], the Condor MW [cona]
and the Virtual Data System [FVWZ02]. In addition, Chapter 6 uses the Java Servlet Technology [jst] for
the web portal. The VTK CISG Registration Toolkit [HRSH02] is used to view images and to perform
affine and nonrigid image registration in this thesis. The manual segmentations of the bones in this thesis
are done using Analyze [Bio03].
I added the simulated annealing optimisation algorithm and the ability to use multiple starting esti-
mates to the affine registration in the VTK CISG Registration Toolkit. I also implemented the intensity
analysis algorithm in Chapter 3 and the spatio-temporal segmentation in Chapter 4 in C++, the web
portal in Chapter 6 in Java, and various shell scripts used in this thesis.
1.6 Contribution
1.6.1 Overall
The main contributions of this thesis are described in chapters 3, 4, 5 and 6. Chapter 2 provides an
overview of the literature, which complements the specific review and methods in each chapter. The
majority of the work in chapters 3, 4 and 6 has been published in conference proceedings. In addition,
the work in Chapter 3 was submitted as a journal article. They are listed at the end of the references.
1.6.2 Individual chapters

Chapter 3 applies two methods to automatically quantify small changes in a bone in longitudinal MR
images of joints in an experimental model of RA. The first method accurately measures global bone vol-
ume by delineating the bone using segmentation propagation and nonrigid registration. The results are
validated by comparing them with the manual segmentations of the bone. The second method measures
local bone lesion volume by analysing longitudinal intensity changes in a common reference coordi-
nate system using nonrigid registration, optimal image thresholding and mathematical morphology. The
results are validated by comparing them with the disease model, histology data and the manual segmen-
tations of bone lesions. In addition, there are statistically significant differences in the local bone lesion
1.6. Contribution 22
volume between male and female subjects, and statistically significant changes in the local bone lesion
volume over time, which suggests that the results of the second method are likely to sensitive to small
disease-related changes. Specific contributions are
Method 1
Automatic, accurate and quantitative measurement of global bone volume using nonrigid
registration and segmentation propagation.
Comparison of the automatic bone delineations and the manual bone segmentations.
Method 2
A new method to automatically quantify local bone lesion volume by analysing longitudi-
nal intensity changes in a common reference coordinate system using nonrigid registration,
image thresholding and mathematical morphology.
Demonstration that the results are likely to sensitive to small disease-related changes by the
fact that there are statistically significant differences in the local bone lesion volume between
male and female subjects, and statistically significant changes in the local bone lesion volume
over time.
Comparison of the local bone lesion volume with known characteristics of the disease model.
Comparison of candidate bone lesion regions with 2D histological sections of the subjects.
Comparison of the local bone lesion volume with the volume of the manual bone lesion
segmentations in one of the candidate bone lesion regions.
Chapter 4 describes an integrated spatio-temporal image analysis method that incorporates the space
and time-domain information from temporal image sequences, and applies it to quantify small changes
in longitudinal MR images of the joint in an experimental model of RA. The method is used to obtain
4D segmentations of bone lesions in a 5D feature space that is built from a temporal image sequence.
Nonrigid registration allows the results to be analysed in a common reference coordinate system. Com-
parisons with Method 2 described in Chapter 3 using real and simulated image data show that the spatio-
temporal method is more sensitive to small changes and robust to image noise. The results of the spatio-
temporal method are validated using the disease model, histology data and the manual segmentations of
bone lesions. Specific contributions are
A new method to automatically quantify local bone lesion volume by integrating the space and
time-domain information from temporal image sequences.
Comparison of the spatio-temporal method with Method 2 in Chapter 3 using real and simulated
image data.
Comparison of local bone lesion volume with the known characteristics of the disease model.
Comparison of candidate bone lesion regions with 2D histological sections of the subjects.
1.6. Contribution 23
Comparison of local bone lesion volume with the volume of the manual bone lesion segmentations
in one of the candidate bone lesion regions.
Chapter 5 describes a method that combines a hierarchical approach and multiple starting estimates
to localise the bones in MR images of the hand and the wrist. In addition, a bone hierarchy and multiple
starting estimates are obtained, based on simple heuristics that are developed to maximise the robustness
and minimise the running time of the registration process. Initial validation using 21 images (252 bones
to be localised) shows that 220 bones are successfully localised. Like the automatic bone localisation,
a method that uses a hierarchical approach is developed to rigidly align the bones in MR images of the
hand and the wrist to a standard reference image. Initial validation using 21 images (105 bones to be
registered) shows that 92 metacarpal bones are successfully registered to the standard reference image.
Specific contributions are
A new method that combines a hierarchical approach and multiple starting estimates to automati-
cally localise the bones in MR images of the hand and the wrist.
Demonstration that segmentation propagation and spatio-temporal segmentation have potential in

automatically quantifying small changes in longitudinal MR images of the hand and the wrist of
RA patients.
A method to automatically align the bones in MR images of the hand and the wrist to a standard
reference image in order to improve the ease of use and the inter-rater variability of the OMERACT
scoring method.
Chapter 6 demonstrates the potential of the Grid by integrating existing Grid technologies and image
analysis algorithms to develop a web-based image analysis workbench. It addresses various challenges
in the automated analysis of large volumes of image data by allowing the sharing of medical image data,
computational resources, storage resources and image analysis software between different institutions,
the management of workflows, and the tracking of data provenance. In addition, the parallel version of
the spatio-temporal segmentation described in Chapter 4 is implemented. Specific contributions are
The integration of existing Grid technologies and image analysis algorithms to allow the sharing
of
medical image data
computational resources
storage resources
image analysis software
between different institutions, the management of workflows, and the tracking of data provenance.
The implementation of the parallel version of the spatio-temporal segmentation described in Chap-
ter 4.
Chapter 2
Literature review
2.1 Introduction
This chapter provides a general review of the literature, which complements the specific review and
methods in each chapter. Section 2.2 reviews various imaging modalities that are used to image joints
in RA. The key issue is to understand the strengths and weaknesses of various imaging modalities in
the longitudinal imaging of joints in RA. Section 2.3 reviews existing image analysis techniques that are
used to quantify changes in longitudinal images and their limitations. Section 2.4 includes a review of
image registration algorithms, which are used throughout this thesis. Section 2.5 reviews classification
algorithms. This provides the basis for selecting a classification algorithm for the spatio-temporal seg-
mentation of bone lesions in Chapter 4. Section 2.6 reviews existing animal models in RA. It describes
the importance of animal models in the understanding of the human disease. Section 2.7 reviews histo-
logical analysis and its limitations. Histological sections of the animal subjects are used to validate the
results of the work in Chapter 3 and Chapter 4.
2.2 Imaging of joints in RA

A number of different imaging modalities are used to obtain images of joints. They include conventional
radiography (CR), microfocal x-ray, computed tomography (CT), bone scintigraphy, ultrasonography
(US), and magnetic resonance imaging (MRI).
2.2.1 Conventional radiography (CR)

CR (or plain film x-ray) is commonly used for initial clinical evaluation and the monitoring of disease
progression in RA. The symptoms and damage of RA, including joint space narrowing (indirect sign of
cartilage thinning), marginal erosions, soft tissue swellings, and malalignment of bones, can be seen in
the radiographs of hands, wrists, and feet [Res95, Ory03].
There are many clear advantages of CR. It is cheap, fast and easy to produce, widely available,
and reproducible [Ory03]. There are scoring methods in CR, such as the Larsen and Sharp methods, to
allow semi-quantitative measurement of severity of the disease. Furthermore, there are vast amounts of
reported data on the radiological progression of RA in the literature. And it has been shown that there
is a relatively strong and probably causal relationship between joint damage seen in CR and subsequent
2.2. Imaging of joints in RA 25
disability, which is most marked in late RA [BTF+ 94].

Although CR is used as the gold standard for the assessment of disease progression and the
effectiveness of drug treatment [BTF+ 94, vdH00, vdHSS+ 02, Ory03], its weaknesses are 2D projection
of 3D pathology, lack of sensitivity to soft tissue changes, the use of ionising radiation, and relative
insensitivity to bone erosion when compared to MRI [OS03, OHS+ 03]. Therefore, a relatively long
period is required to observe and evaluate the effects of drug treatment in follow-up radiographs [OS03,
MCO+ 99].
2.2.2 Microfocal x-ray

High-resolution (7-20m) and high-magnification (10 or more) plain film radiographs can be ac-
quired by placing the specimen close to a small x-ray source and placing the film at some distance
away [BW84]. It can therefore detect small bony changes in RA earlier than plain film radiographs.
Quantitative information about the disease activity can be obtained by measuring joint space widths, and
the number and areas of erosions in macroradiographs of hands and wrists [BWCW86, BWCCG93].
Like CR, microfocal x-ray is a 2D projection of 3D pathology, and therefore, the careful and consistent
positioning of the hand is very important when inferring 3D information from 2D projection images.
Furthermore, microfocal x-ray shows a lack of sensitivity to soft tissue changes.
2.2.3 Computed Tomography (CT)

CT provides additional diagnostic information on complex joint anatomy where CR has difficulties in
showing it [Res95]. Examples are the hip, spine, midfoot and hindfoot, and wrist. CT has better sensitiv-
ity to soft tissues than the CR, but much less sensitivity when compared to MRI. Another disadvantage
is the higher x-ray dose than CR. Although advances have been made to reduce the x-ray dose, CT ex-
aminations should not be repeated on the same patient when other procedures, such as MRI and US, can
provide equal or greater diagnostic information [KMT+ 04, RB00, RBKea00].
2.2.4 Bone scintigraphy

Bone scintigraphy (or bone scan) is potentially useful in screening patients with early RA [BKS+ 99].
It is performed following the IV injection of radioactive technetium(Tc)-99m methylene diphosphonate
(MDP), which decays yielding a gamma ray and may be chemisorbed (chemical and physical processes
leading to absorption) at kink and dislocation sites on the surface of hydroxyapatite [Res95]. Sites of
rapid bone turnover, such as growth centres and reactive bone lesions, are associated with a large mineral
surface that is available for exchange and chemisorption by the complex. Therefore, bone scans are very
sensitive to pathological changes in joints. However, disadvantages of bone scans include: (a) since the
update of tracers like Tc-99m-MDP depends on a number of factors such as the regional blood flow
(vascularity), the bone remodelling, and the stress and weight-bearing of the bone, the specificity of the
bone scan is poor [Ell03, p.84]; (b) the tradeoff between the radiation dose and spatial resolution results
in poor spatial resolution (commonly in mm).
There is evidence suggesting that bone scintigraphy may be used to differentiate active and inactive
phases of RA. In some cases, bone scans do not show increased activity when degenerative changes in
2.2. Imaging of joints in RA 26
the spine of RA patients are seen in radiographs [Res95]. This probably suggests the inactivity of the
disease at the time of the scan. Thus, it would appear that bone scintigraphy might offer a means of
differentiating inflammatory from degenerative joint alterations, and active from burned-out disease.
2.2.5 Magnetic resonance imaging (MRI)

MRI is said to make a whole-organ assessment possible because it allows assessment of all the struc-
tures (bone, synovial membrane, cartilage, ligament, bone marrow, and fluid collections) involved in
RA [OS03]. It has many advantages over CR: no ionising radiation, 3D imaging, and more sensitivity
to inflammatory and erosive changes [OS03]. MRI detects more bone erosions than CR and it also de-
tects bone erosions earlier than CR [OHS+ 03, BKS+ 99]. The volume of erosion can be estimated by
manually outlining the erosions [BLSE03].
Intravenous injection of gadolinium (Gd) contrast agents allows the differentiation of synovial
membrane from surrounding tissues and fluid in MRI [BKS+ 99, KSW90, OHSL96, CIL+ 03, OHS+ 99].
Gd compounds shorten T1 relaxation time, resulting in increased signal intensity on T1-weighted images
compared with unenhanced images. Synovial membranes can be outlined manually with the help of the
subtraction image of pre- and post-contrast images [OHSL96]. The synovial volume was then found to
be higher in wrists with clinical signs of synovitis than in clinically inactive wrists and was closely re-
lated to the rate of progressive joint destruction [OHSL96, OHS+ 99]. Dynamic MRI after administration
of a contrast agent is able to differentiate between active destructive pannus and inactive fibrous pannus
or oedema [BKS+ 99, CIL+ 03]. A pure oedema or inactive pannus shows a flat signal increase in the
signal-time curve, whereas hypervascularised pannus tissue is characterised by a steep signal increase
followed by an early plateau [BKS+ 99]. The rate of early synovial enhancement (REEsyn ) was calcu-
lated over a region of interest from the difference image of pre- and post-contrast images [OHSL96]. The
REEsyn was shown to be able to distinguish between active and inactive RA [CIL+ 03]. Active pannus
can be segmented by using intensity thresholding after manually selecting regions of interest containing
both pannus with enhancement and surrounding tissues without enhancement [KSW90].
MRI allows direct visualisation of cartilage [BGHB99, BBG00, BG03, HGL+ 99, HGB+ 03,
RPAG03, VPNG02]. Although new steady state sequences can delineate cartilage and surrounding
tissues quite well, all studies are almost exclusively in large joints like the knee [HGB+ 03, OS03].
Currently, cartilage is not included in recommendations for RA joint assessment until systematic studies
of small joint cartilage are available [OS03, MLE+ 03].
There are several disadvantages of MRI: higher costs and lower availability than CR, longer ex-
amination times resulting in patient discomfort, image artefacts due to patients motion or the field
inhomogeneity resulting in the decrease in image quality or the need to re-scan the patients, and less data
on standardisation, reproducibility and prognostic value than CR [OS03, DPSA+ 02, TGSG02].
2.2.6 Ultrasound (US)

US can visualise inflammatory changes of soft tissues and erosive changes of cartilages and
bones [Res95, OS03, WGC+ 00, GC98, SCS+ 01, Tay03]. Other advantages include no ionising ra-
diation, low running costs, fast to image (hence, ability to image more joints), multiplanar imaging
2.3. Measuring changes in longitudinal images 27
capability [OS03, GC98]. US also detects more bone erosions than CR and is comparable to MR [GC98].
Modern high frequency transducers produce high-resolution images of small joints, such as finger
joints, to help clinicians to diagnose RA and monitor its progression. Under conventional grey scale
(B-mode) US, joint effusion appears as an anechoic area. Inflamed synovial membrane typically appears
as irregular clusters of soft echoes, and cartilage appears as a well-defined hypoechoic layer because of
high water content [GC98]. Power Doppler US (PDUS) may provide new insight into RA pathogenesis
by allowing the visualisation of blood vessels of diameter less than 1mm in synovial inflammation as
angiogenesis is considered to play an important role in early RA [GC98, SCS+ 01, Tay02].
US is not without disadvantages, which include the requirement of a trained person to acquire and
interpret images, potential problems with reproducibility, less data on standardisation and prognostic
value than MRI and CR, limited field of view because of its low penetrating power, and the need of
an acoustic window, which means that transducers need to be positioned perpendicular to the area of
interest to receive reflected sound [OS03, GC98].
2.3 Measuring changes in longitudinal images

2.3.1 Visual examination and scoring
Traditionally, changes in longitudinal images at different time points are often assessed visually by expe-
rienced radiologists. In order to quantify the changes, scoring methods have been developed to evaluate
RA induced damage in plain film radiographs and MR images. The Larsen and the Sharp methods,
for example, are extensively used to evaluate changes in radiographs of hands and feet in clinical trials
that are obtained in posterior-anterior view [vdH00, vdHSS+ 02, Ory03]. The Larsen method and its
modifications are based on assessment of bone erosions and involve manual scoring by comparing with
standard reference films. The Sharp method and its modifications manually score joint space narrowing
and bone erosions separately. A computer program has recently been developed to assist the scoring in
the Larsen method [WPB+ 03].
OMERACT (www.omeract.org) stands for Outcome Measures in Rheumatoid Arthritis Clinical
Trials, which strives to improve outcome measurement through a data driven, iterative consensus process.
The OMERACT rheumatoid arthritis magnetic resonance image scoring system (RAMRIS) has been
suggested for the evaluation of inflammatory and destructive changes in RA hands and wrists [MLE+ 03].
The core set of MRI sequences include at least the following: Imaging in 2 planes, with T1-weighted
images before and after intravenous (IV) gadolinium-contrast; plus a T2-weighted fat saturated sequence
or, if the latter is not available, a STIR sequence. The use of IV gadolinium contrast is not essential
if destructive changes alone are considered important. As a result, important RA joint pathologies,
including synovitis, bone erosion, and bone oedema, can be observed and given a score manually.
Visual scoring methods suffer from the following problems: (a) they are labour-intensive; (b) train-
ing and expert knowledge are required in order to reduce the intra-observer and inter-observer variabil-
ities; (c) small bone lesions or erosions may not satisfy the definition of the bone lesion or erosion laid
down by the scoring methods. This leads to the floor effect in the scores; (d) the bone destruction can
still continue after the maximum score. This leads to the ceiling effect in the score.
2.3.2 Change in volume

Volume change in the region of interest can be used to quantify change in longitudinal images. In
the analysis of longitudinal neuroimages, the volume changes in various parts of the brain have been
used to try to quantify and understand the progression of various disease, such as multiple sclerosis
(MS) [MG03, WGR+ 01, FJL+ 00], schizophrenia [DeL99], and Alzheimers Disease (AD) [FWF+ 96a,
CSF01, THdZ+ 03].
However, in osteoarthritis (OA), although the volume of the cartilage can be accurately quantified
from MR images of the knee in longitudinal studies, recent studies show that there is no significant loss
of total cartilage volume in 11 patients with knee OA over a 3-year period and suggest that focal changes
in the cartilage should be studied [GDK+ 02]. Using the active appearance model and the rigid image
registration, the cartilages in the MR images of the knee at different time points are aligned so that the
average intensity changes in the cartilages can be calculated [NVB+ 04]. The results show that there is a
significant annual percentage change in the signal intensity of 1.02 0.29(P < 0.01) in the femoral
compartment. Manual segmentation of the cartilage at the first time point is required for each temporal
sequence.
2.3.3 Subtraction after registration and the boundary shift integral

Subtle changes in the brain can be observed in the difference images (or subtraction images) produced
from the registered longitudinal images [HSO+ 95, FWF96b]. A further study shows that radiologists can
detect very small changes (> 200m mean voxel shift) in serial MR images of the brain by studying the
registered difference images [DHC+ 00]. In addition, the same study shows that when unregistered lon-
gitudinal MR images are used to assess anatomical change in the brains of the growth-hormone patients
and normal volunteers, no significant difference between the two groups are detected and the agreement
between radiologists are low. However, when registered difference images are used, significant differ-
ence between the two groups are detected (P < 0.01) and the agreement between radiologists is high
( 0.085).
To quantify the subtle changes observed in the difference images, the boundary shift integral (BSI)
is proposed to measure the cerebral volume changes using registered difference images [FF97]. It cal-
culates the volume changes by integrating the intensity difference at the regions around the segmented
cortical surfaces of the brain, and detects large differences in the loss in brain tissues in AD patients
(mean = 34.7cc) and normal controls (mean = 1.8cc). Since BSI is applied to the regions around the
segmented cortical surfaces of the brain, its accuracy is affected by factors like the accuracy of the seg-
mentation of the cortical surfaces and the existence of high-intensity lesions in the cortical surfaces (e.g.
due to multiple sclerosis).
2.4 Image registration

In this thesis, image registration plays an important role. In particular, the following image registrations
are carried out throughout this thesis.
Intra-subject rigid body registration: it is commonly used to compensate for the motion of the
same subject in repeated scans in order to detect very small changes due to disease progression or
treatment [DHC+ 00, HHD+ 00, HSO+ 95, FWF96b, LWM+ 98].
Inter-subject nonrigid body registration: it is often used to compensate for the variability between
different subjects so that their images can be mapped to a common coordinate system. This allows
the image data to be compared across different subjects (e.g. normal subjects and patients) for
the determination of abnormalities or for medical research [HHR+ 02, AF99]. Another important
application is in the automatic segmentation or labelling [CR00, CE97, CSF01], which is described
in more details in Section 3.4.1.
Both feature-based and intensity-based image registration algorithms have been used to performed the
above registrations.
2.4.1 The feature-based image registration algorithm

The feature-based image registration algorithm determines the spatial alignment between images by
aligning corresponding image features, such as points (landmark-based) and surfaces (surface-based),
which are extracted from the images either interactively or automatically. Since the set of the image
features contains less data than that of the original images, an advantage of the feature-based image
registration algorithm is the relatively fast optimisation procedure [MV98]. Therefore, it is often used
to align pre-operative and intra-operative images during image guided surgeries [FWM98, HDM+ 98,
Haw98, MV98].
The landmark-based registration algorithm

The landmark-based registration algorithm is mostly used for rigid or affine registration [MV98]. Al-
though the rigid or affine alignment between two sets of landmarks from two images can be easily
determined by minimising the average distance between each landmark and its counterpart [HBHH01],
the main difficulty lies in the identification of corresponding landmarks in the images. There are various
kinds of landmarks:
Extrinsic landmarks are markers that are physical attached to subjects, such as the fiducial markers
screwed into the bone. They are undesirable for patients because of the invasiveness.
Intrinsic landmarks are anatomical or geometrical landmarks that can be accurately identifiable.
They are undesirable when the underlying anatomy changes due to disease progression or other
factors.
The surface-based registration algorithm

The surface-based registration algorithm can determine the spatial alignment between two surfaces using
the head-and-hat algorithm or the iterative closest point (ICP) algorithm [HBHH01]. Like the landmark-
based registration algorithm, the main difficulty lies in the accurate segmentation of corresponding sur-
faces: the segmentations of the surfaces are not always automated, and the registration accuracy is limited
by the accuracy of the surface segmentations [MV98].
Figure 2.1: Intensity-based image registration algorithm. A is the source image, B is the target image,
A is the transformed source image, and T is the transformation. The figure shows the three iterative
steps of the intensity-based image registration algorithm - calculate similarity measure, transform
and search for new transformation, which are described in details in Section 2.4.
2.4.2 The intensity-based image registration algorithm

The intensity-based image registration algorithm is often used to carry out the image registrations men-
tioned at the beginning of this section due to its accuracy, robustness and ease of use. The algorithm
determines the spatial alignment between images by iteratively optimising the similarity measure that
is calculated directly from the intensity values in the images. It has been widely used and validated in
aligning images of brain, breast, lung, liver, heart and other parts of the body. Most of the intensity-based
image registration algorithms consist of the following three iterative steps when registering the source
(or moving) image A to the target (or fixed) image B (Figure 2.1):
The transform step: transforms the source image A into the intermediate image A by the trans-
formation T using some interpolation scheme (e.g. trilinear interpolation). The transformation
can range from a simple rigid body transformation with three degrees of freedom to a nonrigid
body transformation with many more degrees of freedom.
The calculate similarity measure step: measures the similarity of the two images by calculating
the similarity measure.
The search for new transformation step: searches for a new transformation T based on some
optimisation technique.
The algorithm stops when no transformation is found to give a better value of the similarity measure
between the two images, to a preset tolerance. More details about each step are described below.
Similarity measure
The similar measure is one of the most important parameters in the intensity-based image registration
algorithm because it measures the similarity (and hence, the goodness of the spatial alignment) between
images. Let A(x) be the intensity of the voxel at position x in the source image A with a field of view
FA , B(x) be the intensity of the voxel at position x in the target image B with a field of view FB . In
general, the two images do not have the same field of view, and it is likely that, at some position xB in
the image B, A(xB ) is not defined. Therefore, similarity measures are calculated from the intensities of
the voxels inside the overlapping region of the two images. The overlapping region of the source image
A and the target image B is given by the positions in field of view of the target image B that are also in
the field of view of the source image A, and is defined as
FBA = {xB FB xB FA }. (2.1)
The following three similarity measures are commonly used in the registration of serial images
acquired from the same modality.
Sum of squared differences (SSD) is the sum of intensity differences between two images, and is the
optimal similarity measure if the two images only differ by Gaussian noise. For N voxels in the
overlapping region FBA ,
1 X
SSD = kA(x) B(x)k2 . (2.2)
N
xFBA
However, the optimal condition cannot be met in many common scenarios, like the registration
of images from different modalities, and the registration of serial MR images because the noise
distribution in MR images is rician. SSD is also very sensitive to a small number of voxels that
have very large intensity differences between images A and B. As a result, the sum of absolute
differences (SAD) is sometimes used instead, and is given by
1 X
SAD = kA(x) B(x)k. (2.3)
N
xFBA
Correlation coefficient (CC) comes from cross correlation, and is defined as

P
xFBA (A(x) A)(B(x) B)
CC = P , (2.4)
{ (A(x) A)2 P
(B(x) B) 2 } 12
xFBA xFBA
where A and B
are the mean intensities of the voxels within FBA in images A and B respectively.
CC is an optimal similarity measure if the two images differ by Gaussian noise and their intensities
have a linear relationship, which is a more relaxed optimal condition compared to that in SSD.
Ratio image uniformity (RIU) measures the uniformity of a ratio image that is calculated by dividing
the intensity of the target image B by that of the source image A on a voxel-by-voxel basis. Given
N voxels in the overlapping region FBA , the ratio image R and the RIU are given by
B(x)
R(x) = , x FBA , (2.5)
A(x)
q
1
P 2
R)
N xFBA (R(x)
RIU = 1
P . (2.6)
N xFBA R(x)
For images from different modalities and subjects, there is in general no simple relationship between
the intensities of images A and B. Since this violates the basic assumptions of the above similarity
measures, the following similar measures are often used instead.
Partitioned intensity uniformity (PIU) is a modified version of RIU and is developed with the aim to
register PET (Positron Emission Tomography) and MR images. The algorithm assumes that all
voxels with a particular intensity value in the MR image represent the same tissue type so that
intensity values of corresponding voxels in the PET image should be similar to each other. It
therefore partitions the MR images into separate sub-images based on the intensity value of the
MR voxels, and try to maximise the uniformity of the the PET voxel values within each sub-image.
In terms of images A and B, if the target image B is partitioned, the voxels of the sub-image in
the intensity bin b is given by
Fb = {x FBA B(x) b}. (2.7)
Furthermore, P IUA is the sum of normalised standard deviation of intensities in image A for each
intensity bin b in image B, and is defined as
X nb A (b)
P IUA = , (2.8)
N A (b)
b
where
X
nb = 1, (2.9)
xFb
1 X
A (b) = A(x), (2.10)
nb
xFb
2 1 X
A (b) = (A(x) A (b))2 (2.11)
nb
xFb
are the number of voxels in the intensity bin b, and A and A are the mean and standard deviation
values of the voxels in image A within Fb respectively.
Joint entropy (JE) , mutual information and normalised mutual information are all based on the com-
munication theory developed by Shannon [Sha48]. Consider the following cases: when two im-
ages are correctly aligned, the corresponding structures will overlap; whereas when two images
are misaligned, duplicated versions of the corresponding structures will appear in the combined
image. Therefore, the amount of information or uncertainty is minimised when two images are
in alignment. Using the communication theory, the information supplied by an image can be
calculated from the intensity histogram, and is given by the Shannon-Wiener entropy
X
H= pi log pi , (2.12)
i
where p1 , p2 , p3 , ..., pn are the probabilities of the intensity values derived from the intensity
histogram. Finally, JE is the entropy of the combined image of the source image A and the target
image B, and is calculated from the joint histogram1 of the two images
XX
JE = H(A, B) = pAB (a, b) log pAB (a, b). (2.13)
a b
where pAB (a, b) is the probability of having the intensity value a in image A and intensity value b
in image B at the same voxel.
However, low JE does not always lead to correct alignment of two images because it is too depen-
dent on the overlapping region FBA . For example, if the overlap of the air regions (with intensity
b0 ) increases, pAB (b0 , b0 ) will increase, thus reducing JE. By minimising JE, the algorithm max-
imises the amount of air in the overlapping region.
Mutual information (MI) considers not only the information in the combined image (the joint en-
tropy), but also the information contributed to the overlapping region by each image. It is defined
as
XX pAB (a, b)
I(A, B) = H(A) + H(B) H(A, B) = pAB (a, b) log , (2.14)
a
pA (a)pB (b)
b
where
X
H(A) = pA (a) log pA (a) A(x) = a|x FBA , (2.15)
a
X
H(B) = pB (b) log pB (b) B(x) = b|x FBA , (2.16)
b
are the marginal entropies of image A and B respectively. The aim of the image registration
is to maximise MI, so that JE is minimised and the marginal entropies (H(A) and H(B)) are
maximised. In other words, the correct alignment is achieved by minimising the information in
the combined image, and maximising the information contributed to the overlapping region by
each image. As a result, MI is more robust to the overlapping region than JE.
Normalised mutual information (NMI) is a normalised version of MI, and is devised to further im-
prove the robustness to the overlapping region. By normalising the mutual information with re-
spect to the joint entropy, NMI is given by
H(A) + H(B) I(A, B)

NMI = = . (2.17)
H(A, B) H(A, B)
Image interpolation and transformation

Due to the discrete nature of medical images, the transformation step that transforms the source image A
to the intermediate image A involves the interpolation of intensity values. Since the step is performed
in each iteration, high accuracy interpolation techniques, such as the windowed-sinc interpolation of a
1 The joint histogram is built by plotting the intensities of the voxels in image A against the intensities of the corresponding
voxels in image B in the overlapping region.
bandlimited signal or image, are not normally used because of their high computational costs. Two com-
monly used low cost interpolations are the nearest neighbour interpolation and the trilinear interpolation.
Nearest neighbour interpolation determines the intensity value from the closest voxel to the transformed
position. Trilinear interpolation determines the intensity value by taking a weighted average of the near-
est eight neighbours. The weightings, which sum up to one, are inversely proportional to the distance of
each neighbour to the transformed position.
Various transformations are used to describe the spatial alignment between images, depending on
the nature and motion of the object in the image. The affine transformation consists of three translations,
three rotations, three scalings, and three skewing for a whole image, and is used to describe the motion of
non-deformable objects. However, for deformable objects, the affine transformation alone is insufficient
because it can only account for the gross or global differences in translation, rotation, scaling and skewing
between the deformable objects. The nonrigid body transformation, which allows localised deformations
in an image, is therefore needed to describe the remaining differences in the deformable objects after the
affine transformation.
There are many models of nonrigid body transformation. First, by extending the linear polynomial
terms in the affine transformation, nonrigid deformations can be modelled by higher order nonlinear
polynomial terms (e.g. for second order: x2 , y 2 , z 2 , xy, xz and yz) [WGW+ 98]. However, a change in
one coefficient in the polynomial affects a whole image. The method therefore requires the removal of
some structures in the image that are likely to vary in size and shape in ways that are distinct to the struc-
tures of interest to maintain the accuracy. Second, nonrigid deformations can be modelled by various
physical processes. They have the advantage of being able to constrain the underlying deformation in a
plausible manner. Elastic models treat images as linear, elastic solids, which are deformed by external
forces derived from a similarity measure [BK89]. However, the linear elasticity assumption is invalid
for large deformations. Viscous fluid models [CRM96] based on the flow of viscous fluid can be used to
model the large deformations instead. In some cases, the viscous fluid model is regarded as too flexible
because it may allow anatomically incorrect alignment of different but adjacent structures through the
same mechanism that allows large deformation matches [RWG+ 01]. For example, one gyrus may flow
from the source brain image to match two or more different gyri in the target brain image, which may or
may not be desirable. Finite element models allow more control of nonrigid deformations by assigning
physical properties (e.g. Youngs modulus and shear modulus) of anatomical structures to the corre-
sponding structures in the image. They have been applied to model the intraoperative brain deformations
caused by brain shift or tumour resection [FNM+ 02, HRS+ 99], where high accuracy is required. The
drawbacks of the finite element model include the need to pre-segment various structures, the determi-
nation of physical properties of anatomical structures, and the long computational time. The final model
is based on the free form deformation [SP86], which deforms an image by manipulating an underlying
mesh of control points. The deformation at any point in the image is calculated by interpolating the
displacements of the control points from their original positions using splines, which are piecewise poly-
nomials jointed together by the control points, or other basis functions. The thin-plate spline [Boo89]
has been applied to describe the deformations in the brain [MBK+ 97, DKB+ 99]. but the high computa-
tional cost limits its ability to model complex and localised deformations because each control point
has a global influence on the deformations. In contrast, cubic B-splines have local support, which
means that moving one control point only affects the deformations in its local neighbourhood [RSH+ 99],
and therefore can efficiently model local deformations. The free form deformation based on cubic B-
spline has been applied to model deformations in various parts of the body (e.g. the brain [RFS03], the
heart [CMR04b], the chest [MHV+ 03], the liver [BKP+ 01] and the breast [RSH+ 99]). Other models of
nonrigid body transformation can be found in [HBHH01, HHH01, ZF03, CHH04].
In this thesis, the free form deformation model based on the cubic B-spline is used to describe the
spatial alignment between different subjects, and between different time points of the same subject. Let
(x, y, z) be a point in the image volume, and denote a nx ny nz mesh of control points i,j,k with
uniform spacing , the free form deformation TF F D is defined by the 3D tensor product of the 1D cubic
B-splines
3 X
X 3 X
3
TF F D (x, y, z) = Bl (u)Bm (v)Bn (w)i+l,j+m,k+n , (2.18)
l=0 m=0 n=0
where i = bx/c 1, j = by/c 1, k = bz/c 1, u = x/ bx/c, v = y/ by/c,

w = z/ bz/c, and Bl are the lth basis function (or weight) of the B-spline:
B0 (u) = (1 u)3 /6 (2.19)
B1 (u) = (3u3 6u2 + 4)/6 (2.20)
B2 (u) = (3u3 + 3u2 + 3u + 1)/6 (2.21)
B3 (u) = u3 /6. (2.22)
The spacing of the control points controls the resolution of the the mesh , which in turn controls the
degree of localised deformations that can be modelled. A large spacing allows the modelling of global
nonrigid deformations, whereas a small spacing allows the modelling of highly localised deformations.
However, the smaller the spacing of the control points, the more the degrees of freedom and the higher
the computational cost. Therefore, in practice, a hierarchical approach that starts at a large spacing of
control points and uses successively smaller values to model the deformations is used to minimise the
computational cost and model highly localised deformations.
Optimisation
Optimisation is another very important part of the image registration algorithm. During each iteration
of the optimisation in Fig 2.1, the optimisation algorithm searches for a new transformation, applies it
to the source image A, and evaluates the value of the similarity measure. The iteration stops when no
transformation can be found to give a better value of the similarity measure, to a preset tolerance. How-
ever, this does not guarantee that the final transformation is a correct solution because the optimisation
algorithm may have found a local optimum.
To understand the difficulty of the optimisation problem in image registration, one needs to look at
the number of parameters in the transformation that affects the value of the similarity measure. There are
six degrees of freedom in the rigid body registration, twelve degrees of freedom in the affine registration,
and many more (often thousands) degrees of freedom in the nonrigid body registration. The optimisation
algorithm has to search through the values in each degree of freedom in order to locate the optimal
value of the similarity measure. This is a multidimensional optimisation problem with a vast number of
dimensions.
There are various textbook algorithms for the multidimensional optimisation problem, see
[PTVF92]. Among the many optimisation algorithms, the multistep method, the steepest descent
method, and the simulated annealing method are introduced because they are used by the image regis-
tration algorithms in this thesis [SHH97, HHD+ 00, RSH+ 99].
The multistep method evaluates the similarity measure within a local neighbourhood of the current
transformation, and tests for convergence. The neighbourhood is the set of transformations that are
calculated by adding and subtracting the current transformation by a step size t in each degree of
freedom. The values of the similar measure in the neighbourhood and at the current transformation
are evaluated to find a better transformation. When no better transformation can be found, the step
size t is reduced, and the process starts again in a smaller local neighbourhood. The optimisation
stops when the current transformation gives the best value of the similarity measure in the local
neighbourhood for the smallest step size.
The steepest descent method is a simple optimisation method that uses the gradient (vector of first
partial derivatives) to find the optimal value of a function. To minimise a function f (t), the method
chooses the next point ti+1 by minimising along the direction f (ti ), where f (t) decreases most
rapidly at the current point ti . The process is repeated until the function converges to a minimum,
subject to a tolerance. When the exact form of the function is not known, like the function of
the similarity measure, the gradient can be calculated numerically. The problem with the steepest
descent method is that it is slower than other gradient methods, such as the conjugate gradient
method.
The simulated annealing method is based on the manner in which the energy of a system is minimised
in the process of annealing. In an annealing process, a molten metal, initially at high temperature
and disordered, is slowly cooled so that the system at any time is approximately in thermodynamic
equilibrium. While cooling, the energy distribution of the system is governed by the Boltzmann
probability distribution,
E
prob(E) e kT , (2.23)
where E is the energy, T is the temperature, and k is the Boltzmanns constant. Therefore, even at
low temperature, the system has a finite chance of being in a high energy state. In other words, in
the process of slow cooling, the energy of the system sometimes goes up as well as down, but the
lower the temperature is, the less likely the energy will go up.
The principles of this thermodynamic system are incorporated into the minimisation of a function
by allowing the function to occasionally have a higher value than the current value. However, the
2.5. Supervised and unsupervised classifications 37
more iterations the algorithm goes through, the less likely the function is allowed to take a higher
value. The probability of taking a new higher value is
(E2 E1 )
prob(E1 E2 ) = e T , (2.24)
where E1 and E2 are the current value and the new higher value respectively, and T is the control
parameter that depends on the number of iterations so far. Therefore, instead of greedily going
for the quick and nearby solution like many optimisation algorithms (e.g. the steepest descent
method), which leads to a local optimal, the simulated annealing method has a better chance of
finding a better, and more global optimal.
As a refinement to the optimisation strategy, a multiresolution approach can be used: the images
are first registered at low resolution, then the result obtained is used as the starting estimate for the
registration at a higher resolution, and so on. This approach can (a) reduce the incidents of local optimum
found by the optimisation algorithm; (b) reduce the number of iterations needed in the optimisation
algorithm to find the optimal value; (c) increase the capture range of the optimisation algorithm [TU00,
MCV+ 97]. The low resolution image can be formed by resampling the current high resolution image by
a factor of 2, and simple trilinear interpolation can be used in order to save computational cost. When
compared to the more computation expensive Gaussian resampling scheme, where images of different
resolutions are generated by blurring using different sizes of Gaussian kernels, the optimisation results
are very similar [SHH97].
2.5 Supervised and unsupervised classifications

In supervised classification, given a collection of labelled patterns, the problem is to label a newly en-
countered pattern. In unsupervised classification, the problem is to group a collection of unlabelled
patterns into meaningful clusters. Unsupervised classification (more specifically, the mean shift proce-
dure) is used in this thesis because it is more generic and flexible. Since the images from animal subjects
and human subjects are very different, the algorithm developed to analyse the animal images does not
need to be re-trained in order to analyse the human images.
2.5.1 Unsupervised classification (or data clustering)

Data clustering is the organisation of a collection of patterns (usually represented as vectors of mea-
surements or points in a multidimensional space) into clusters based on similarity [JMF99]. It has
many applications in medical image analysis, such as image segmentation [PXP00], the analysis of
fMRI images [DBW+ 04] and the classification of abnormal tissues[SPK06, AVvO+ 04]. Some of the
commonly used clustering algorithms are briefly reviewed in this section. Details about other algo-
rithms, such as fuzzy clustering, evolutionary approaches and search-based approaches, can be found
in [JMF99, DBW+ 04].
Hierarchical clustering yields a dendrogram (Figure 2.2) representing the nested grouping of patterns
based on the similarities of the patterns. The basic algorithm is:
(a) The collection of patterns. (b) The dendrogram as a result of hierarchical

clustering
Figure 2.2: Hierarchical clustering.
1. Each pattern is placed in its own cluster.
2. The similarities between any two clusters are evaluated. A popular metric to measure the
similarity between two clusters is the Euclidean distance.
3. The two most similar clusters are joined together.
4. Repeat 2 and 3 until a single cluster is formed.
The hierarchical algorithm is versatile because it does not make any assumption about the shapes
of the underlying clusters. However, the major disadvantage is the relative large time complexity
O(n2 log n), where n is the number of patterns, when compared to other clustering algorithms.
k-Means clustering and the rest of the following clustering algorithms yield a single partition of the
patterns (Figure 2.3), instead of a dendrogram. The basic algorithm is:
1. k cluster centres are randomly chosen.
2. Each pattern is assigned to the closest cluster center to generate a partition.
3. The center of each cluster is re-calculated.
4. Repeat 2 and 3 until no (or minimal) reassignment of patterns to new clusters or minimal
decrease in the squared error e2
nj
k X
(j)
X
e2 = |~xi ~cj |2 , (2.25)
j=1 i=1
(j)
where ~xi is the ith pattern belonging to the j th cluster and ~cj is the center of the j th cluster.
The k-means clustering algorithm is relatively fast, with time complexity O(nkl), where l is the
number of iterations (usually fixed in advance). The number of clusters k must be specified and
is fixed. Although no explicit assumption is made about the shapes of the underlying clusters, the
algorithm has a tendency to a partition of hyperspherical clusters due to the stopping criteria.
Figure 2.3: k-means clustering of the pattern in Figure 2.2(a) for k = 2.
Artificial neural networks (ANN) are based on biological neural networks. Similar patterns are
grouped by the network and represented by a single unit (neuron). Well-known examples include
self-organising map (SOM) [Koh89] and neural gas [MBS93]. The idea behind these ANN is
to change the weights of the units iteratively, which is called learning, until a termination criterion
is satisfied. In general, the performance of ANN is better than the k-means clustering. ANN is
relatively fast, with time complexity O(nkt), where t is the number of iterations (or the learning
time). However, all ANNs have a fixed number of output nodes, i.e. fixed k and are sensitive to
the selection of various learning and control parameters. Furthermore, single-layered ANNs are
suitable for detecting only hyperspherical clusters. Two- or more layered networks are required to
detect arbitrary shaped clusters.
Mean shift is a density estimation-based nonparametric clustering technique [CM02]. It regards a fea-
ture space as the empirical probability density function (p.d.f.) of the represented parameters.
Dense regions (i.e. regions with dense points) in the feature space correspond to local maxima of
the p.d.f., which are the modes of the unknown density. Mean shift locates these local maxima or
modes so that clusters associated with them can be delineated.
Given n data points ~xi , i = 1, ..., n, and a radially symmetric kernel K(~x) with single window
width2 h, the multivariate kernel density estimator (also known as the Parzen window technique)
fh,K (~x) in d-dimensional space is given by
n
1 X ~x x~i
fh,K (~x) = K . (2.26)
nhd i=1 h
Since the kernel K(~x) is radially symmetric, it can be represented by a one-dimensional function
k(x) satisfying
K(~x) = ck,d k(k~xk2 ), (2.27)
where k(x) is called the profile of the kernel K(~x), and ck,d is the normalisation constant which
ensures that K(~x) integrates to 1. Using this profile notation, (2.26) becomes
n !
x xi 2

ck,d X
fh,K (~x) = k . (2.28)
nhd i=1 h
2 Window width is also called bandwidth, or smoothing parameter.
2.6. Animal Model 40
Since the local maxima are located in the zeros of the gradient, from (2.28), we have
" n !#
~x ~xi 2

2ck,d X
fh,K = g m
~ h,G (~x), (2.29)
nhd+2 i=1 h
where g(x) = k 0 (x) is the profile of the kernel G(~x), m ~ h,G (~x) is the mean shift and is given by
Pn ~x~x 2
i=1 ~
x i g
h
i
m
~ h,G (~x) = P ~x. (2.30)
~x~xi 2

n
i=1 g h
An interesting property noted by Comaniciu and Meer [CM02] is that the summation
~x~xi 2 in (2.29) is equal to the density estimation at ~x with the kernel G(~x), i.e.
Pn
i=1 g h
n !
~x ~xi 2
X
fh,G(~x) = g
, (2.31)
i=1
h
m
~ h,G can then be written as
h,K (~x)
f
~ h,G (~x) = h2 c
m . (2.32)
fh,G (~x)
Therefore, the direction of the mean shift vector is the same as the direction of maximum increase
in the density, and the magnitude of the vector is proportional to the density gradient estimate
obtained with the kernel K, normalised by the density estimate computed with the kernel G. By
following the path of the mean shift vector, a local maximum of the estimated density, where the
gradient is zero, will be reached. In other words, the mean shift procedure can determine the local
maximum of the estimated density near a point ~x by iteratively
calculating the mean shift vector m

~ h,G (~x) using (2.30),
translating ~x by the mean shift vector m

~ h,G (~x),
until ~x converges. In addition, the size of the translation (or step) in each iteration is given by the
magnitude of the mean shift vector, which is normalised by the density estimate computed with
the kernel G. This adaptive nature is very desirable because the procedure chooses large step sizes
in regions of low density values, where features are of no interest, and chooses small step sizes
in regions of high density values, where features are more important and the feature space can be
examined in greater detail. As a result, the mean shift procedure is an adaptive gradient ascent
method in which the size of each mean shift step is adjusted by the density estimate computed
with the kernel G.
Mean shift has advantages over other clustering techniques by not requiring a prior knowledge of
the number of clusters, and by not making any assumption about the shapes of the clusters in the
feature space [CM02]. However, the time complexity can be quite large O(nlc ), where lc is the
number of iterations required for the mean shift procedure to converge.
2.6 Animal Model

Animal models of RA have proven very useful in the understanding of some of the disease mecha-
nisms and in the identification of new strategies for treatment [ZWL05, KOK05]. For example, studies
2.6. Animal Model 41
have consistently found tumour necrosis factor- (TNF) and other key proinflammatory cytokines and
chemokines expressed in a variety of models including collagen induced arthritis and bacterial cell-wall
induced arthritis. Further research into the roles of certain cytokines leads to the development of anti-
cytokine therapies, such as the anti-TNF treatment.
Although none of the animal models is exactly the same as the human RA, they are useful and
important because: (a) the models allow the evaluation of various aspects of the disease under tightly
controlled experimental conditions, so that multiple variables can be compared; (b) tissues can be easily
sampled and analysed in the molecular, cellular and tissue levels; (c) the relationship between drug
dose and biological response can be quantified (especially for pharmaceutical companies); (d) better
understanding of the models may provide insight into human RA.
There are many animal models of RA including the adjuvant-induced arthritis (AA), the collagen-
induced arthritis (CIA), the streptococcal cell wall-induced arthritis (SCW), the antigen-induced arthritis
(AIA), the spontaneous arthritis, and the transgenic model [HEP05]. Some of the widely used models
are briefly described below:
Adjuvant-induced arthritis (AA) in rats can be induced by an intradermal injection of heat-killed

bacteria (Mycobacterium butyricum, M. phlei or M. tuberculosis) suspended in mineral oil or
liquid paraffin [ZWL05, Bil05]. Symptoms (such as swelling) in the joints appear around day 10
12. During the following 2 months, pannus tissue forms in the synovium leading to the destruction
of the cartilage and the bone, in a similar manner to that seen in human RA. Intense bone remod-
elling (bone removal and reformation) happens at sites of ligament and muscle insertion. AA in
rats has been commonly used to evaluate the non-steroid anti-inflammatory drugs.
Collagen-induced arthritis (CIA) in mice can be induced by the intradermal injection of native type
II collagen in water-in-oil emulsion [ZWL05, TTK77]. The average peak onset of the disease
occurs at 20 days after the injection. Pannus is formed in the synovium leading to the erosion of
the cartilage and subchondral bone. One advantage of CIA over AA is that CIA is an autoimmune
response to a connective tissue component rather than to a bacterial antigen. Therefore, CIA can
provide insight into the understanding of the autoimmune response of human RA, and is used for
evaluating new anti-rheumatic therapies [TDT05].
Streptococcal cell wall-induced arthritis (SCW) in rats is usually elicited by an intraperitoneal injec-
tion of a suspension of the group A streptococcal cell wall fragments [ZWL05, Sch05, WAW+ 83].
An initial acute inflammation of the ankles, wrists and small joints of the feet reaches a peak at
35 days and then subsides. This is followed by recurrence of inflammation at 24 weeks with
synovitis, pannus formation, and cartilage and bone erosion. SCW resembles human RA in that
the model involves the peripheral joints and usually spares the axial skeleton [Sch05], and in the
histological and radiological features [CCA+ 79].
The disease model used in chapters 3 and 4 of this thesis is a variation of the SCW model - the
monoarticular reactivation model, in which the contra-lateral ankle is injected with saline and used
2.7. Histological analysis 42
as a control [ESCS85]. An initial joint inflammation is induced by an intra-articular injection of

group A streptococcal cell wall fragments. The inflammation reaches a peak within 24 hours, and
then subsides so that the joints appeared grossly normal or minimally inflamed by 23 weeks. A
systemic injection of a small dose of the cell-wall fragments reactivates the previously inflamed
joint, whereas the control joint injected with saline remains normal. Figure 2.4 shows an example
of the progression of the inflammation in Sprague-Dawley rats, which is scored grossly on a scale
of 0 (no apparent inflammation) to 4 (severe inflammation) based on erythema (abnormal redness
of the skin caused by capillary congestion) and edema of the periarticular tissues and enlargement
of the joints. Histology of the inflamed joint shows pannus formation, increased vascularity, and
erosions of articular cartilage and bone at 1020 days after reinjection. Figure 2.5 shows an exam-
ple of the histologic appearance of the joint tissue in Sprague-Dawley rats, which are scored (from
0 to ++++) based on the following changes in the tissues of the ankle and adjacent tarsal joints:
presence of an acute exudative reaction with edema, infiltration of inflammatory cells, and
extravasation of fibrin and red blood cells;
proliferation of synovial lining cells, and increased numbers of fibroblasts and vascularity in
the synovial stroma;
formation of pannus and replacement of bone and articular cartilage by pannus; and
presence of a chronic inflammatory reaction with diffuse and focal collections of lympho-
cytes and macrophages, fibrosis of the synovial stroma, and hyperplasia of the synovial lining
cells.
Furthermore, an example of the locations of the erosions in the talus bone is shown in Figure 2.6.
2.7 Histological analysis

Histology is the study of tissues of the body and of how these tissues are arranged to constitute or-
gans [JC05]. It allows the visualisation of cell, tissue and organ in a microscopic level so that abnormal
tissue structure and its relationship with function defects can be studied. In RA, for example, the his-
tology of synovium reveals the infiltration of T cells and macrophages, which causes the hyperplasia of
the intimal lining (Figure 2.7) [Fir03]. Locally expressed degradative enzymes digest the extracellular
matrix and destroy the articular structures.
The most common procedure of preparing tissues for examination under a light microscope in-
volves:
Fixation: this step is to avoid tissue digestion by bacteria or by enzymes within the cells, and to
preserve the cell structure and molecular composition. As soon as the tissue is removed from the
body, it is immersed into a fixative (e.g. 4% formaldehyde) for a period of time (minimum 24
hours) to allow the fixative to diffuse into the tissue.
Decalcification [additional step for the bone]: this step is to facilitate the sectioning of the bone by
immersing it into 10% formic acid to remove the calcium.
Figure 2.4: Mean joint scores in SCW model (excerpted from [ESCS85]). Mean joint scores of the right
ankle of 8 Sprague-Dawley rats that were injected in the right ankle with 5g of Streptococcus pyogenes
peptidoglycan-polysacchaharide (PG-PG) and, 25 days later, were reinjected intravenously (IV) with
300g of the same PG-PS preparation. Joint inflammation was scored grossly, and mean scores of all
right joints including those of nonresponders are shown. Control joints initially injected with phosphate
buffered saline did not develop inflammation following either injection.
Figure 2.5: Histologic appearance of joint tissue of 8 Sprague-Dawley rats in SCW model prior to and at
various times following reinjection of a normally subarthropathic dose of peptidoglycan-polysaccharide
fragments isolated from Streptococcus pyogenes (excerpted from [ESCS85]). Mean joint score as
determined by gross observation of the joints at the time of killing.
Figure 2.6: Surface rendered impressions of the plantar surface of MR (left) and X-ray CT (right) images
of the talus bones of PGPS sensitised rats before (Day 0, top) and after (Day 14, bottom) reactivation.
Excerpted from [BCB+ 03].
Figure 2.7: Synovial histology in RA (excerpted from [Fir03])

Embedding: this step is to give a rigid consistency to the tissue for sectioning. Firstly, the tissue
is dehydrated by bathing through a graded series of mixtures of ethanol and water (usually in the
order of 70%, 80%, 90%, 95% and 100%). Secondly, the ethanol is removed by bathing through
a series of ethanol/xylene mixtures (usaully in the order of 75%/25%, 50%/50% and 0%/100%).
Thirdly, the tissue is impregnated in melted paraffin wax, typically at 60 C so that the heat causes
the xylene to evaporate. Finally, the paraffin wax is left to cool down and harden.
Sectioning: the embedded tissue is sectioned into 1-10m-thick slices using a microtome.
Staining: since most of the cell structures are colourless, this step is to stain different cell struc-
tures with different colours for examination under a light microscope. The most commonly used
combination of dyes is haematoxylin and eosin (H&E). Haematoxylin stains the cell nucleus and
other acidic structures blue, while eosin stains the cytoplasm and collagen pink. Another useful
dye is the toluidine blue, which stains cartilage blue due to its proteoglycan content.
The whole procedure is labour-intensive and may take 12 hours to 2 days depending on the size of
the tissue, the fixative, and the embedding medium.
There are problems and limitations in the use of histology [JC05]. The first problem is that the
histological sections are usually chosen randomly and contain only a fraction of the organ. Only the
study of serial sections and their reconstruction into a three-dimensional volume make it possible to
understand a complex organ. The second problem is that the sections have various artefacts caused by
the tissue preparation procedure. Typical artefacts include (a) differential shrinkage that is caused by
the fixative, by the ethanol, or by the heat during the embedding; (b) loss of molecules that are not
properly kept by the fixative and removed by the preparation steps during embedding; (c) wrinkles of
the sections; (d) precipitates of stain. The third problem is that it is impossible to differentially stain all
tissue components on one slide. Two or more slides with different stains are usually required to examine
different tissue components. The final problem is that the subject is often killed in order to obtain the
tissue for the analysis.
Chapter 3
Automatic Quantification of Changes in

Longitudinal MR Images of Joints I
3.1 Introduction
Recent innovations in drug therapies have made it highly desirable to obtain sensitive biomarkers of
disease progression in RA that can be used to quantify the performance of candidate disease modify-
ing drugs. MRI allows visualisation and quantification of bone erosion, bone oedema, and synovial
inflammation, which are important features and image biomarkers in RA [OHS+ 03, MBP+ 03, BLSE03,
OHSL96]. Although longitudinal imaging using MRI has been proposed as an effective tool for quantify-
ing disease progression in clinical studies, the analysis is based on visual inspection [MBP+ 03, CLO+ 03]
or interactive analysis [OHSL96, BLSE03]. In this chapter, I apply two different methods to automati-
cally quantify small changes in a bone in in-vivo longitudinal MR images of joints in an experimental rat
model of RA in order to the quantify disease progression. Both methods make use of rigid and nonrigid
registration to establish correspondences between time points and across subjects. The first method uses
segmentation propagation [CR00] to enable automatic delineation of a small bone from MR volume
images, and gives a global measure of temporal changes in bone volume. The results are validated by
comparing them with the manual segmentations of the bone. The second method analyses longitudinal
intensity changes within the bone to obtain a local and sensitive measure of temporal change in bone
lesion volume. The results are validated by comparing them with the disease model, the histology of the
subjects and the manual segmentations of bone lesions.
3.2 Aims and contribution

Aims
To use segmentation propagation to enable automatic and accurate delineation of a small bone
from MR volume images, and obtain a global measure of temporal changes in bone volume.
To compare the automatic delineations of the bones from segmentation propagation with manual
segmentations.
To analyse longitudinal intensity changes within the bone to obtain a local and sensitive measure
3.3. Review 47
of temporal changes in bone lesion volume.
To compare bone lesion volumes between male and female subjects, and measure bone lesion
volume changes over time in all the candidate bone lesion regions.
To compare the candidate bone lesion regions with the histology of the subjects.
To compare the volume of automatic segmentations of bone lesions with that of the manual seg-
mentations of bone lesions in one of the candidate bone lesion regions.
Contribution
This chapter applies two methods to automatically quantify small changes in a bone in longitudinal MR
images of joints in an experimental model of RA. The first method accurately measures the global bone
volume by delineating the bone using nonrigid registration and segmentation propagation. The results
are validated by comparing with the manual segmentations of the bone. However, the global bone
volume is fluctuating over time. The second method measures the local bone lesion volume by analysing
longitudinal intensity changes in a common reference coordinate system using nonrigid registration,
image thresholding and mathematical morphology. The results are validated by comparing with the
disease model, histology data and the manual segmentations of bone lesions. In addition, there are
statistically significant differences in the local bone lesion volume between male and female subjects,
and statistically significant changes in the local bone lesion volume over time. This suggests that the
results of the second method are likely to be sensitive to the disease progression.
Contribution of others to this work

GSK prepared the experimental model and acquired the longitudinal MR images. The VTK CISG image
registration toolkit, which was developed by Thomas Hartkens, was used to perform the image regis-
tration and visualisation. The open-source visualisation toolkit (VTK) and insight toolkit (ITK) were
used to perform the mathematical morphology, connected component analysis and surface rendering.
All other image and data analysis was my own work.
3.3 Review
Existing methods to quantify disease progression in RA often involve considerable human interaction
and expert knowledge. The OMERACT (Outcome Measures in Rheumatoid Arthritis Clinical Tri-
als) rheumatoid arthritis magnetic resonance image scoring system (RAMRIS) was suggested for the
evaluation of inflammatory and destructive changes in RA hands and wrists [MBP+ 03]. The core set
of MRI methods includes at least the following: imaging in 2 planes, with T1-weighted images be-
fore and after intravenous (IV) gadolinium-contrast; plus a T2-weighted fat saturated sequence or, if
the latter is not available, a STIR sequence. The use of IV gadolinium contrast is not essential if de-
structive changes alone are considered important. RA joint pathologies, including synovitis, bone ero-
sion and bone oedema, can be observed and given a score manually according to their intensities and
sizes in the MR image. Other manual scoring methods were also proposed and used in different stud-
ies [MSC+ 98, BKS+ 99, FJLS03]. Although a semi-automatic computerised method of measuring the
3.3. Review 48
erosion and synovial volume was proposed in [BLSE03], it involved manually outlining the erosion and
synovium in the MR images. Other methods include active contours [LZZ+ 00, SEM+ 99] and active
shape models [SHWT97], which were used to segment cartilage in order to access its volume, thickness
or integrity in osteoarthritis. Active contours require the manual initialisation of points or contours (ex-
pert knowledge) in each image as starting points for the contours to evolve, while active shape models
require the collection of example shapes (shapes of pathological bones, in this case) to build the statistical
shape model.
Recently, an automatic method for quantifying disease progression in RA was suggested by Carano
et al.. They used multispectral images to classify tissues in the metacarpophalangeal (MCP) joints of the
hands into 3 tissue classes (Cortical bone and tendon, soft tissue, and bone marrow) [CLR+ 04]. After
registering the follow-up image to the baseline image, the change in voxel class between time points was
used to quantify changes in bone lesion volume. The results showed that there were strong correlations
with the changes in bone lesion volume calculated from the manual segmentations of bone lesions. In
addition, the best correlation was obtained by using the k-means classifier (Pearson correlation coeffi-
cient = 0.94), compared to using the multivariate Gaussian classifier (Pearson correlation coefficient
= 0.84) and the k-nearest neighbour classifier (Pearson correlation coefficient = 0.86). Since only T1-
weighted images are required by the methods presented in this chapter, the imaging time is less than
the time taken to acquire multispectral images because (1) it typically takes 5 10 minutes to acquire a
T1- or T2-weighted image, and (b) the amount of re-scanning due to poor quality images is reduced if
only one image is acquired. Therefore, potential advantages are that: (a) when imaging hundreds of time
points in large-scale studies, a large amount of time can be saved; (b) it is generally painful for human
RA patients to stay in the scanning positions while their hands or wrists are imaged. The comfort of the
patients can be improved if they spend less time in the scanner.
Histology is the arbiter of diagnosis in many pathological conditions, and has often been used
to validate studies involving MR images [CVC+ 95, JWSZ+ 99, PDN+ 04, VdWHB+ 04, CJHS04].
In [PDN+ 04, VdWHB+ 04, CJHS04, LAYC+ 04], specimens were sliced in planes corresponding to
the MR images, and histological sections were manually aligned to, and compared with, the MR image
slices.
Automatic registration of histological sections to MRI data is a very challenging problem. Firstly,
the histology is high resolution in one plane, but frequently has very poor sampling in 3D (often to the
extent that only one slice of histology is obtained). Secondly, the sample preparation stages can introduce
substantial distortions, with, for example, differential shrinkage of different tissues within the sample,
and in some cases, parts of the sample tearing or being destroyed. Post-mortem optical data can be much
easier to register (most famously, the visible human dataset [vhp]) because the cut face of the sample is
repeatedly imaged using the same camera geometry. However, the optical data is much lower resolution,
and does not provide the same opportunities for staining.
Techniques for the automatic registration of histological sections or post mortem optical data with
MR images are being actively researched. These approaches involve reconstructing 3D image data from
3.4. Theory 49
multiple 2D histological sections or optical data before registration [JWSZ+ 99, BCR+ 01, KBH+ 03,
BDM+ 02, SvMDZ01, OBD+ 01]. In [JWSZ+ 99], in-vivo multislice MR images of rat brains were reg-
istered to the 3D histological volumes using rigid (surface-based) and nonrigid (surface and landmark-
based) registration. The non-rigid registration was required to compensate for distortion in the histolog-
ical sections. In [BCR+ 01, KBH+ 03], 3D post mortem optical data were registered to the MR images
of a formalin-fixed post mortem human brain using surface-based rigid registration and intensity-based
affine registration. Then, each 2D optical image was allowed to move within a constraint to refine the
result of the registration. The authors of [BCR+ 01] stated that work was in progress to register the 2D
optical data to 2D histological sections.
The slicing of multiple histological sections from multiple specimens is a labour intensive task, and
reliably combining these slices into consistent 3D volumes remains an unsolved problem. In animal
studies, it is possible to use fiducial markers to assist in the alignment, e.g. [LLB+ 03], where needles
were inserted into the subject before MR image acquisition. The markers were used as landmarks in
the rigid registration of tissue section photographs and MR images of rabbit thigh. But such approaches
do not correct for the distortions in the histology. In the work described in this thesis, only a single
histological section is available with each stain, so the only feasible approach for comparing this with
MRI is by means of manual alignment. The aim is not, therefore, accurate volumetric comparison, but
to ascertain whether lesions visible in the histological sections are also visible in the processed MRI.
3.4 Theory
3.4.1 Segmentation propagation
Segmentation propagation is an automatic segmentation technique that makes use of the deformation
field generated by image registration [CR00]. One requirement of the technique is to construct an atlas
that is a reference image plus the segmentations of the structures of interest. The same structures of inter-
ests in a target image can then be delineated using the deformation field calculated from the registration
of the atlas and the target image. The principle of segmentation propagation is summarised in Figure 3.1.
In [HSH02], a multi-stage registration process was used to calculate the deformation field. The
first stage involved using affine registration to compensate for the global motion and gross differences
between source and target images. The second and later stages involved using cubic B-spline based
nonrigid registration to compensate for local deformation. The B-spline nonrigid registration algorithm
was developed for the registration of dynamic contrast enhanced MR breast images [RSH+ 99]. It uses
cubic B-spline based free form deformation (FFD), which models local deformation as translations of
a regularly spaced grid of control points. It followed that the control point spacing was an important
parameter in FFD - a large spacing of control points allows the modelling of large scale nonrigid
deformation and a small spacing of control points allows the modelling of highly local deformation.
Successively smaller values of control point spacing were used in the second and later stages to model
more localised deformations.
3.4. Theory 50
Figure 3.1: Principle of segmentation propagation. IA represents the atlas (source) image with a seg-
mented structure, RA , defined by a connected set of boundary points at voxel locations, shown as dots
and lines. I1 and I2 represent the baseline and follow-up images of a subject. Nonrigid registration of
IA to I1 and I2 produces the transformations TA1 and TA2 . Image IA is transformed by TA1 and TA2
into the space of I1 and I2 , which results in propagated structures R1 and R2 . Because a transformation
results generally in translations of boundary points by a non-integer number of voxels, the transformed
set of boundary points does not, in general, coincide with the voxel locations of I1 and I2 . Excerpted
from [HSH02].
3.4.2 Otsus thresholding

Grey-scale thresholding is a very simple segmentation technique. Optimal thresholding algorithms
model the histogram of an image using a weighted sum of two or more probability densities with nor-
mal distribution. And the optimal thresholding value is chosen by the minimum probability between the
maxima of two or more normal distributions [Gla93].
One well-known and commonly used thresholding technique is Otsus thresholding, which selects
the optimum threshold value based on maximising the between-class variance of two classes. For an
image with N pixels and grey-levels [1, 2, ..., L], let the number of pixels at grey-level i be ni . The
histogram of the image can be regarded as a probability distribution:
L
ni X
pi = , pi >= 0, pi = 1. (3.1)
N i=1
If the pixels are divided into two classes by the threshold value k, the between-class variance of the two
classes is then given by
2
B = 0 1 (1 0 )2 (3.2)
where
k
X
0 = pi , (3.3)
i=1
L
X
1 = pi , (3.4)
i=k+1
3.4. Theory 51
Figure 3.2: Examples of some common structuring elements
k
X ipi
0 = , (3.5)
i=1
0
L
X ipi
1 = (3.6)
1
i=k+1
are the probabilities of class occurrence and the class mean levels. The optimal threshold value k is
2
determined by maximising B .
3.4.3 Mathematical morphology

Mathematical morphology is a theory based on set theory, integral geometry, and lattice algebra for
analysing spatial structures [Soi03]. It is also a very powerful technique for performing image filtering,
image segmentation, and image measurements. In order to extract relevant structures or remove irrele-
vant structures from an image, a structuring element (SE) that is a small set or image is used to probe the
image. The shape of SE is usually chosen according to the structure of the interest. And the origin of
SE must be defined to allow the positioning of SE at a given point or pixel. Examples of some common
structuring elements are shown in Figure 3.2.
Some basic morphological operations, including erosion, dilation, opening and closing, are de-
scribed below and their effects are shown in Fig 3.3.
The erosion of a set X by a structuring element B is denoted by B (X) and is defined as the locus
of points X such that B is included in X when its origin is placed at X:
B (X) = {x|Bx X}. (3.7)
Essentially, the erosion of an image removes all structures that is smaller than the structuring
elements, and shrinks all the other ones.
The dilation of a set X by a structuring element B is denoted by B and is defined as the locus of
points x such that B hits X when its origin coincides with x:
B (X) = {x|Bx X 6= 0}. (3.8)
Essentially, the dilation of an image connects all the structures that are closer than the size of the
structuring element, and enlarges them.
The opening of a set X by a structuring element B is denoted by B and is defined as the erosion
which is given by B
of X by B, followed by dilation with the reflected SE B, = {b|b B}.
Essentially, the opening of an image is to dilate the eroded image in order to recover the original
image as much as possible.
3.5. Overviews of methods 52
(a) Original image. (b) The structuring ele- (c) The erosion.
ment.
(d) The dilation. (e) The opening. (f) The closing.
Figure 3.3: The effects of the basic morphological operations.
The closing of a set X by a structuring element B is denoted by B and is defined as the dilation

of X, followed by erosion with the reflected SE B.
Essentially, the closing of an image is to erode the dilated image in order to recover the original
image as much as possible.
It can be seen that the opening and the closing are constructed from the erosion and the dilation. In
fact, morphological operations can be built by combining the basic morphological operations in various
ways. The interested reader can refer to the excellent book by Soille [Soi03].
3.5 Overviews of methods

In this section, I define terms, state assumptions and observations, and describe brief overviews of the
two image analysis methods. More details of the methods are given in Section 3.6.
3.5.1 Definitions
Bone lesion is defined as any abnormal change in the bone. A typical bone lesion appears as a high-
intensity region in T1-weighted images, as shown in Figure 3.18 (page 69). A candidate bone lesion
region is defined as a region in the bone where bone lesions are likely to occur.
3.5.2 Assumptions and observations

The bone lesions caused by the disease arise in similar locations in all subjects. This is supported by: (a)
juxtaarticular cartilage and bone erosion at the edge of the articular cartilage and adjacent subchondral
3.5. Overviews of methods 53
bone is a characteristic pattern of human RA [RYS97, Fre95]; (b) pannus formation and erosion of
the margins of the articular cartilage have been identified in previous histological assessment of the
animal model that we are using [ESCS85]. This is consistent with our histological analysis as shown
in Figure 3.18, where the bone destruction in the two boxes are at the edge of the cartilage; (c) the
experiment conditions are carefully kept to be very similar for all the subjects (see Section 3.6.1).
3.5.3 Overview of the measurement of changes in bone volume (Method 1)

The aim of the first method is to delineate the boundaries of the bones from all the time points in each
subject, so that changes in the bone volume can be measured. Each of the following steps is explained
in more details in sections 3.6.2 to 3.6.4 respectively.
1. Atlas construction: an atlas is constructed from a single subject for use in bone delineation by
segmentation propagation.
2. Approximate delineation of the talus bone: the regions of interest are roughly delineated from all
the time points using rigid registration to the atlas. The results of the registrations are used as the
starting estimates for the accurate delineation.
3. Accurate delineation of the talus bone: the regions of interest (the talus bones) in all the time
points are delineated using segmentation propagation after inter-subject nonrigid registration of
each subject to the atlas.
This technique was applied to a single bone (the talus bone) in the animal model. It is validated by
comparing with manual segmentations of the talus bone in Section 3.7.1.1.
3.5.4 Overview of the measurement of changes in bone lesion volume (Method 2)

The aim of the second method is to locate and quantify intensity changes within the bones delineated by
Method 1 in a common reference coordinate system. Each of the following steps is explained in more
detail in sections 3.6.5 to 3.6.8 respectively.
1. Generation of difference images: difference images are generated by subtracting the baseline im-
age of each subject from the registered follow-up images using the output of Step (2) of Method 1.
2. Identification of potential lesion voxels: histogram thresholding is used to identify high intensity
voxels in the difference images that are potential lesion voxels. The potential lesion voxels from
all the subjects and all the time points are then mapped to a reference coordinate system (the atlas)
using the results of nonrigid inter-subject registration from Step (3) of Method 1.
3. Identification of candidate bone lesion regions: based on the assumptions in Section 3.5.2, the
candidate bone lesion regions are identified by aggregating the potential lesion voxels from all the
subjects and all the time points in the reference coordinate system.
4. Calculation of bone lesion volume: the candidate bone lesion regions are mapped back to the
coordinate system of the baseline image of each individual subject. The number of potential
lesion voxels within each region is determined.
3.6. Materials and methods 54
The technique was applied to the same bone as in method 1 (the talus bone), and is validated by
comparing with the disease model (Section 3.7.2.1), the histology of the subjects (Section 3.7.2.2) and
manual segmentation of the bone lesions (Section 3.7.2.3).
3.6 Materials and methods

All the steps in Methods 1 and 2 are summarised in in Figure 3.4.
3.6.1 Image data

The data set consisted of a group of 12 Lewis rats (subjects 1 - 12; 6 male, 6 female; mean age: 60 days at
the time of first scan). Animals were housed, maintained, and experiments conducted, in accordance with
the Home Office Animals (Scientific Procedures) Act 1986, UK. An RA inducing agent (peptidoglycan
polysaccharide (PG-PS) from Streptococcus pyogenes) was injected into the right rear ankle of the rats
at day -14 (see Section 2.6 for more details about the disease model). Reactivation at day 0 to produce
joint inflammation was carried out by injecting PG-PS intravenously into the tail vein [ESCS85]. MR
scans of the right ankle were taken at six time points: day -12, day -4, day +3, day +10, day +14 and day
+21. MR scans of the left ankle were taken at day -12 and day -4 to provide a control. The T1-weighted
images were acquired on a 7T 20cm bore (Bruker BiospecTM ) system and a Birdcage coil using a 3D
gradient echo sequence with the following parameters: TE = 3ms, TR = 14ms, flip angle = 30 , FOV =
15 40 15(mm)3 , matrix = 256 256 256. This gave voxel size of 58.6 156 58.6(m)3 . Only
data from subjects 1 - 11 were analysed because subject 12 (female) died after the first MR scan.
Animal imaging was performed by trained animal handlers who had personal licences issued by
the Home Office. To avoid the movement of small animals during imaging, they were anaesthetised
using isoflurane and remain anaesthetised for the duration of imaging. A tube of flowing warm water
was attached to the anaesthetised animal to keep it warm. The temperature and the breath cycle of the
animal had to be closely monitored to assess the depth of anaesthesia. After the imaging, the animals
were transfered back to the housing area, and left to wake up.
3.6.2 Atlas construction (used in Methods 1 and 2)

The baseline image of subject 1 was randomly chosen to be the reference image. An expert manually
segmented the talus bone in the baseline image, which took about 3 hours. The baseline image plus the
manually segmented bone formed the atlas.
3.6.3 Approximate delineation of the talus bone (used in Methods 1 and 2)

The joint is composed of multiple bones, each of which is rigid, but which move with respect to one
another in a nonrigid fashion. For the work described in this chapter, only the talus bone was considered.
Instead of registering the whole image at each time point to the baseline (target), a region of interest (ROI)
around the talus bone was identified in the baseline image, and only voxels in this region were used in
the registration. This improved the registration accuracy, as within the ROI the rigid body assumption
was valid. The determination of ROI will be described in the next section (section 3.6.3.1).
Due to different orientations of the joints in different time points, the rigid registration was ini-
Figure 3.4: Flow diagram to describe all the steps of Methods 1 and 2. See Section 3.6.2 to Section 3.6.8
for more details.
tialised by interactively aligning the talus bones in follow-up images and baseline images so that they
were within approximately 1mm and 5 of rigid alignment. This usually took less than 1 minute. Rigid
registration was performed over the ROI using the starting estimate. The correlation coefficient (CC)
was used as the similarity measure and for a set of n data points (xi , yi ),
P
(xi x
)(yi y)
CC = P P . (3.9)
(xi x) (yi y)
Simulated annealing was used to optimise the similarity measure, and the running time for registering a
follow-up image to an ROI in a baseline image was less than 10 minutes.
3.6.3.1 Determination of ROI

All ROIs were derived from the manual segmentations of the talus bone in the reference image (baseline
image of subject 1).
The ROI for the talus bone in the reference image was defined by the boundary that was obtained
by dilating the manual segmentation twice in Analyze (Mayo Foundation, Rochester, USA) using the
kernel size of x = 1 voxel, y = 3 voxels, z = 1 voxel. The size of the kernel was chosen such that there
were about 12 extra voxels surrounding the manual segmentation according to visual inspection. A
symmetrical kernel (x = 3 voxels, y = 3 voxels, z = 3 voxels) would dilate the manual segmentation
more, so that the resultant image contained more voxels of nearby bones. Since the nearby bones might
be in a different position with respect to the talus bone in the target image, this could lead to greater error
in the rigid registration.
Segmentation propagation was then used to obtain the ROIs in the baseline images of the other 10
subjects. This requires a starting estimate that was chosen interactively by aligning the talus bones in the
reference image and in the baseline image of another subject so that the two images were approximately
within 1mm and 5 of rigid alignment. Affine registration was carried out to register the two images
using the starting estimate. The resulting transformation matrices were inverted and used to propagate
the ROI in the reference image to obtain the ROIs in the baseline images of the other 10 subjects. The
correlation coefficient was used as the similarity measure because it was found to have a larger capture
range and produced similar results when normalised mutual information was used. The running time for
registering a baseline image to an ROI in the reference image was less than 10 minutes.
3.6.4 Accurate delineation of the talus bone (used in Methods 1 and 2)

In addition to using intra-subject rigid registration to generate difference images, inter-subject nonrigid
registration [RSH+ 99] was used to automatically delineate the talus bones from all the time points. This
was a 4-stage registration process, which used the output of section 3.6.3 as a starting estimate. The
source image was the baseline image of subject 1 (the reference image). The target images were all time
points of the 11 subjects. Each time point was treated independently.
The first stage was an affine registration with 12 degrees of freedom. The second, third and fourth
stages were cubic B-spline nonrigid registrations with control point spacing of 1.172mm, 0.586mm and
0.293mm respectively. These control point spacings corresponded to 20 pixels (1.172mm), 10 pixels
(0.586mm), and 5 pixels (0.293mm) in the high resolution plane. Each registration stage used the result
of the previous one as the starting estimate. The number of dilations for defining the ROI in the target
image depended on the amount of misregistration before the nonrigid registration. Although the interpo-
lation of B-splines depended on a support region provided by the dilation, the rigid property of the bone
implied that the soft tissue outside the bone had minimal effect on the shape of the bone. The processing
time is approximately proportional to the number of voxels in the ROI. It was desirable to choose a small
ROI to shorten the processing time. A dilation of 4 voxels was used in all stages. The final deformation
field was used to propagate the manual segmentation in the reference image to obtain a boundary of the
corresponding bone in the target image. The correlation coefficient and normalised mutual information
were used as the similarity measures in the affine registration and nonrigid registration respectively. The
running time of segmentation propagation for a bone was 24 hours.
3.6.5 Generation of difference images (used in Method 2)

Follow-up images were resliced using the results of the rigid registration of the baseline image of
each subject and the follow-up images from Section 3.6.3. The interpolation was performed using a
windowed-sinc function (Hanning window of size (diameter) = 7 voxels). Difference images were then
generated by subtracting the baseline image from the resliced follow-up images.
3.6.6 Identification of potential lesion voxels (used in Method 2)

Otsus thresholding [Ots79] was applied to all the difference images to identify the high-intensity voxels
in the talus bones. These are referred to as potential lesion voxels. Then, the potential lesion voxels were
mapped to the reference coordinate system of the atlas using the results of the inter-subject nonrigid
registration from Section 3.6.4.
3.6.7 Identification of candidate bone lesion regions (used in Method 2)

We used a two-stage process to aggregate the potential lesion voxels in the reference coordinate system:
(a) the potential lesion voxels from the same time points of all the subjects were summed to generate
candidate bone lesion regions for each time point; (b) those candidate bone lesion regions were summed
across all time points to generate a single candidate bone lesion region map for the entire subject cohort.
Each stage is described in more details below.
In the first stage, for each time point, an image comprising the union of all potential bone voxels
from all the subjects in the reference coordinate system was generated. Mathematical morphology was
applied to the unioned images, as shown in Figure 3.5. Binary opening operators were applied to remove
isolated potential lesion voxels which could have resulted from either registration errors or false positives
from the thresholding. The kernels shown in Figure 3.6 were constructed to avoid removing excessive
voxels in the y-direction due to the anisotropic voxel size (described in more details in A.1). A binary
dilation operator of kernel size 3 3 3 was then used to generate remaining candidate bone lesion
region masks that are slightly larger than those generated by the opening operator. The results were a
time series of five images containing candidate bone lesion regions of all the subjects.
In the second stage, the time series of five images were summed. Bone lesions that were found in
only one time point were removed by thresholding (using the value 2) to ensure that the final candidate
3.7. Validation 58
Figure 3.5: Mathematical morphological operators applied after each aggregation stage. See sec-
tion 3.6.7 for more details.
Figure 3.6: Kernels in 2D: (a) A 2D upper half kernel; (b) A 2D lower half kernel.
bone regions consisted of bone lesions that appeared in two or more time points. Isolated candidate bone
lesion regions were removed by the same mathematical morphology used in the first stage.
Finally, candidate bone lesion regions, which were defined in the image space of the atlas, were
uniquely identified by the connected component analysis.
3.6.8 Calculation of bone lesion volume (used in Method 2)

The candidate bone lesion regions in the image space of the atlas were transformed to the image space
of the baseline image of each subject by applying the nonrigid deformation fields from the automatic
delineation. Misregistration or interpolation artefacts could result in high signal in the difference image
at the bone boundary. To reduce these edge effects on the result, the automatic segmentations were
eroded by a kernel of size 3 3 3 so that voxels around the edge of the automatic segmentation were
ignored. The eroded segmentations were used as masks to apply to the transformed candidate bone
lesion regions. Then, the number of voxels in each masked candidate bone lesion region in the Otsus
thresholded difference images produced in Section 3.6.6 was counted to give the bone lesion volume.
3.7 Validation
3.7.1 Method 1
3.7.1.1 Comparison with manual segmentation
A comparison of the automatic segmentations with manual segmentation was carried out for three sub-
jects. The talus bones in all the time points of subject 1, subject 2 and subject 3 were segmented
3.7. Validation 59
manually and were compared to the results of the automatic segmentation using the similarity index
[DHT+ 99, HSH02]. If V1 and V2 are two segmentations to be compared, the similarity index, SI, was
given by
T
2 volume(V1 V2 )
SI = . (3.10)
volume(V1 ) + volume(V2 )
The talus bones in the baseline images of subject 2 and subject 3 were segmented by the same
observer (observer A) 3 times to measure the intra-observer variability. The talus bones in the baseline
images of subject 2 and subject 3 were segmented by a different observer (observer B) once. The manual
segmentations by observer B were compared to the manual segmentations by observer A to measure the
inter-observer variability using similarity index.
3.7.2 Method 2
3.7.2.1 Assessments of gender differences and disease progression

As a partial validation of this method, the results were used to quantify gender differences and disease
progression in the animal model. Gender differences and disease progression are known from previ-
ous histological analysis [WCG+ 82, ESCS85], but these have not previously been reported in imaging
studies. The average bone lesion volumes in male and female subjects were then calculated in all the
candidate bone lesion regions.
3.7.2.2 Histological analysis

All animals were killed after imaging at day +21 for histological analysis. The left and right legs were
dissected from the animals and fixed in formalin. The legs were immersed in 10% formic acid to remove
calcium from the bones. The legs were then dehydrated and embedded in paraffin wax. Two 5m thick
sagittal sections were cut from each block, in very close proximity to each other. One section was stained
with haematoxylin and eosin (H&E), and the other one was stained with toluidine blue for visualisation
of cartilage. These are widely used histological stains.
An expert histologist examined all the histological sections and recorded any changes in the syn-
ovium, cartilage, bone, and soft tissue. Typical bone changes in the disease model are shown in the
white box in Figure 3.18(a). It shows destruction along the shafts by inflammatory tissues extending
from synovium and adjacent soft tissues.
Since only two histology sections were obtained from each leg, it was not possible to correlate and
align all the MR-derived candidate bone lesion regions with the histology data. An observer visually
identified the closest slice (in sagittal, axial or coronal orientation) in MRI to the histology section for
each subject. For each lesion visible on the histological section, the corresponding location in the MRI
difference images was visually determined, and the presence or absence of a MR-visible lesion was
recorded. Where the MR-visible lesion was part of a candidate bone lesion region, this was noted. For
all subjects, only a small part of the bone volume was visible in histology, because it is a single slice
technique, so it was not possible to compare each MR lesion with histology.
3.8. Results 60
Figure 3.7: Graph of volume of automatic segmentation of the talus bone of all the subjects against time.
3.7.2.3 Comparison with manual bone lesion volume

Erosion volume measured by the manual delineation of bone erosion in MR image was shown to be
highly correlated to OMERACT RAMRIS erosion score and Larson score [BLSE03]. To support the
results of the automatic measurement of changes in bone lesion volume, bone lesions in a randomly
selected candidate bone lesion region were manually delineated from all the time points and was used to
compare with the results.
3.7.2.4 Error estimation

According to [ESCS85], the ankle without the injection of PG-PS at day -14 should be unaffected by the
reactivation at day 0. Therefore, the MR scans of the left ankles were used to estimate the error in the
calculation of bone lesion volume assuming that the bone lesion volume was zero.
The methods used to calculate the bone lesion volume in the talus bones of the left ankles were
the same as described in section 3.6.8. The average bone lesion volume in all the candidate bone lesion
regions determined in the right ankles was calculated.
3.8 Results
3.8.1 Method 1
3.8.1.1 Automatic and manual delineation of talus bone
The volume of automatic segmentation of talus bone of all the subjects is plotted against time in Fig-
ure 3.7. The volume of the manual segmentation of the talus bone of subject 1, 2, and 3 is plotted against
time in Figure 3.8 for comparison with the graph of volume of the automatic segmentation in Figure 3.7.
3.8. Results 61
Figure 3.8: Graph of volume of manual segmentation of the talus bone of subjects 1, 2 and 3 against
time.
Table 3.1: Mean similarity index and mean % difference between the volume of automatic and manual
segmentations of talus bone of subject 1, 2, and 3 at all the time points.
Subject Mean similarity Index Mean % difference between the volume of auto-
matic and manual segmentations
Subject 1 0.954 2.09
Subject 2 0.942 2.92
Subject 3 0.946 3.78
3.8.1.2 Comparison with manual segmentation of the talus bone

The similarity index and the mean percentage difference between the volume of automatic and manual
segmentations of talus bone of subject 1, 2 and 3 at all the time points are summarised in Table 3.1.
An estimate of intra-observer variability was given by the similarity index between the repeated
manual segmentations of the talus bones in the baseline images of subject 2 and subject 3 by the same ob-
server (observer A). An estimate of inter-observer variability was given by the similarity index between
the repeated manual segmentations of the talus bones in the baseline images of subject 2 and subject 3
by two different observers (observer A and observer B). The results are summarised in Table 3.2.
3.8.2 Method 2
3.8.2.1 Visual inspection of bone lesions
The baseline images and the transformed follow-up images were placed next to each other for visual
assessment. Bone lesions were found in similar locations in the talus bones of 6 subjects, as shown in
Figure 3.9.
3.8. Results 62
Table 3.2: Mean similarity index and mean % difference between the volume of repeated manual seg-
mentations of talus bone in the baseline image of subject 2 and subject 3 as measures of intra- and
inter-observer variability.
Variability Mean similarity Index Mean % difference between the volume of repeated
manual segmentations
Intra-observer 0.9786 1.02
Inter-observer 0.9437 1.29
3.8.2.2 Generation of difference images

An example of difference images for longitudinal images is shown in Figure 3.10. They were generated
by subtracting the baseline image from the transformed follow-up images of subject 2.
3.8.2.3 Identification of bone lesion regions

Some intermediate results of Method 2 are described in the following figures. Figure 3.11 shows can-
didate bone lesion regions in the talus bone at day -4 before morphological operations were applied.
Figure 3.12 shows candidate bone lesion regions in the talus bone at day -4 after morphological oper-
ations were applied. Figure 3.13 shows the sum of the candidate bone lesion regions in 5 time points
(day -4, day +3, day +10, day +14 and day +21). After morphological operations were performed on the
image in Figure 3.13, Figure 3.14 shows candidate bone lesion regions in the talus bone. The surface
rendering of candidate bone lesion regions are shown in Figure 3.15. In total, eight candidate bone lesion
regions (region 93, region 41, region 37, region 24, region 121, region 105, region 7 and region 1) were
found. Region 24 was the candidate bone lesion region observed in all the subjects in Figure 3.9.
3.8.2.4 Assessment of error estimates, gender differences and disease progression

Figure 3.16 shows the graph of the average bone lesion volume in the talus bone in the normal left ankle
against candidate bone lesion regions. Assuming that the disease did not affect the left ankle, the error of
the bone lesion volume in each region was equal to the average bone lesion volume in the graph. Region
1 and region 41 had the largest errors. The error estimates were smaller than twice the standard deviation
of the average bone lesion volume.
The average bone lesion volume in male and female subjects in candidate bone lesion region 24 is
shown in Figure 3.17. The error bars in the graphs indicate the standard deviation of the average bone
lesion volume at each time point. Increase in bone lesion volume in region 24 over time can be observed
in Figure 3.17.
Statistical analysis: when examining the influence of time and gender on the bone lesion volume
in this longitudinal study, we used a repeated measure analysis of variance. The bone lesion volume
in all the candidate regions at all the time points were analysed in an analysis of variance using SPSS
(SPSS Inc., Chicago, Illinois, USA) with time (day -4 vs day +3 vs day +10 vs day +14 vs day +21) as
a within-subject factor and gender (male vs female) as a between-subject factor. The results are shown
in Table 3.3. The ANOVA yielded significant time main effect in regions 1, 24, 37, 93 and 121, and
3.8. Results 63
(a) Subject 1 (b) Subject 2 (c) Subject 3
(d) Subject 4 (e) Subject 5 (f) Subject 6
Figure 3.9: Baseline images and transformed follow-up images of the right ankles of 6 subjects after
rigid registration. Erosion was observed around a particular region (white arrow) in the talus bones of 6
subjects. In each subfigure, the top row shows the baseline image (day -12) and the bottom row shows
the transformed follow-up image (day +21). Each row shows the transaxial, coronal and sagittal view
(from left to right) from the same image.
3.8. Results 64
Figure 3.10: Difference images (transformed follow-up image minus baseline image) for the serial im-
ages of the right ankle of subject 2. Each row shows the transaxial, coronal and sagittal view (from left
to right) from the same time point. The talus bone is within the white rectangle. From top to bottom, the
images are from follow-up images at day -4, day +3, day +10, day +14 and day +21. The black arrows
indicates a bone lesion appearing at day -4.
Figure 3.11: Candidate bone lesion regions (in green) of the talus bone at day -4 before morphological
operations. The transaxial, coronal and sagittal views of the same image are shown from left to right.
The brighter the green voxel, the more subjects in which a bone lesion was found in that voxel.
3.8. Results 65
Figure 3.12: Candidate bone lesion regions (in green) at day -4 of the talus bone after morphological
operations. The transaxial, coronal and sagittal views of the same image are shown from left to right.
The brighter the green voxel, the more subjects in which a bone lesion was found in that voxel.
Figure 3.13: Sum of the candidate bone lesion regions (in green) of the 5 time points (day-4, day +3, day
+10, day +14 and day +21). The transaxial, coronal and sagittal views are shown from left to right.
Figure 3.14: Candidate bone lesion regions in the talus bone. Each region is coloured differently by the
image viewer.
3.8. Results 66
Figure 3.15: Surface rendering of candidate bone lesion regions overlaid with the semi-transparent talus
bone in the atlas. Each candidate bone lesion region is coloured in a different colour as specified by the
legend on the bottom left corner. (Note that the colours assigned to the regions by the image viewer are
different to those in Figure 3.14.)
Figure 3.16: The average bone lesion volume in the talus bone in the left ankle (control ankle) in all
the candidate bone lesion region. This gives error estimates in various regions. The error bars show the
standard deviations of average bone lesion volumes.
3.8. Results 67
Figure 3.17: Graph of average bone lesion volume of male and female subjects in region 24 against time.
significant gender between-subject effect in regions 24, 93 and 121.
Pair comparisons between bone lesion volume at different time points in regions 1, 24, 37, 93 and
121 (regions where the ANOVA yielded significant time main effect) were performed using Bonferroni
adjustment. Only pair comparisons that are significant at P < 0.05 are shown in Table 3.4 .
3.8.2.5 Comparison with histology
Histology sections of the 11 subjects were examined microscopically. Only 5 subjects were observed to
show RA-like pathology, such as bone destruction and lesion, in the histology section taken through the
talus bone. Because only a single slice of histology was available, it was not possible to judge whether
RA-like pathology might have been visible elsewhere in the talus bone. Bone lesions in the histology
sections were found to be near to candidate bone lesion region 24 and 105.
Figure 3.18 to Figure 3.20 show some results of comparison between the histology and MR images.
Each figure contains (a) a histological section of the right leg with RA-like pathology enclosed in white
boxes; (b) the registered follow-up image (at day +21) and the baseline image reformatted in the sagittal
and axial plane, centred on the location in the MR image that was judged visually closest to the histology
lesion. MRI bone lesions that are near to the pathology in the histological section in (a) are enclosed in
white boxes; (c) the difference image generated by subtracting the baseline image from the registered
follow-up image (at day +21) in (b). The MRI bone lesion corresponds to the bright voxels enclosed
in white boxes. Bone lesions in white boxes are found to be close to one of the candidate bone lesions
regions that were calculated by the method, but the bone lesions in black dotted boxes are not. Other
details are described in the caption of individual figure. Table 3.5 summarises the comparison between
histology and the results:
3.8. Results 68
Table 3.3: Results of the comparison of the bone lesion volume between different time points and be-
tween male and female subjects using ANOVA. P values that are significant at P < 0.05 are marked with
*.
Region Within-subject factor (time) Between-subject factor (gender)

Region 1 P < 0.009* P < 0.274
Region 7 P < 0.127 P < 0.201
Region 24 P < 0.0005* P < 0.024*
Region 37 P < 0.015* P < 0.459
Region 41 P < 0.114 P < 0.913
Region 93 P < 0.0005* P < 0.025*
Region 105 P < 0.154 P < 0.343
Region 121 P < 0.0005* P < 0.022*
Table 3.4: Results of pair comparisons that are significant at P < 0.05 between different time points in
regions 1, 24, 37, 93 and 121. Va and Vb are the bone lesion volume at time point a and time point b
respectively.
Region Pair (time point a and Significance Mean difference (= Va - Vb ) 95% CI for the
time point b) mean difference
Region 1 Day +3 and day +10 P < 0.032 -16.58 (=8.62-25.20) -31.94 to -1.22
Region 24 Day +3 and day +14 P < 0.010 -28.00(=7.68-35.68) -49.62 to -6.38
Day +3 and day +21 P < 0.019 -57.53(=7.68-65.22) -106.6 to -8.46
Day +10 and day +14 P < 0.026 -15.05(=20.63-35.68) -28.49 to -1.61
Day +10 and day +21 P < 0.019 -44.58(=20.63-65.22) -82.65 to -6.51
Region 37 No significant pair
Region 93 Day -4 and day +3 P < 0.005 -11.58(=1.80-13.38) -19.66 to -3.50
Day -4 and day +10 P < 0.008 -13.83(=1.80-15.63) -24.16 to -3.51
Day +3 and day +14 P < 0.022 9.27(=13.38-4.12) 1.21 to 17.32
Day +10 and day +14 P < 0.024 11.52(=15.63-4.12) 1.32 to 21.71
Region 121 Day -4 and day +3 P < 0.015 -4.02(=0.10-4.12) -7.30 to -0.74
Table 3.5: Comparison between histology and the results.
Subject Candidate bone lesion region that is Bone lesion volume calculated by
close to bone lesions observed in the Method 2 in the candidate bone le-
histological section sion region
Subject 2 Region 105 21 voxels
3.8. Results 69
(a) Histological section of the talus (curved dash line) from the right
leg of subject 2 (H&E, magnification 2). The irregular and jagged
bone edge in the white box (compared with the smooth normal bone edge
pointed by the black arrows) reveals bone destruction. Bone destruction
in the black dotted box is not in any of the candidate bone lesion regions.
(b) Registered follow-up image at day +21(left) and baseline image

(right) of subject 2. The top and bottom rows show the transaxial and
coronal planes from the same image. Bone lesions were observed in
the white boxes in the follow-up image.
(c) Difference image of the registered follow-up (day +21) and base-
line image. From left to right, it shows the transaxial, coronal and
sagittal views of the same image. Bright bone lesions were observed
in the white boxes that correspond to candidate bone lesion region 105.
Figure 3.18: Comparison between histology section and MR image of subject 2.

3.8. Results 70
(a) Histological section of the talus from the right leg of subject 8 (H&E,
magnification 2). The dark arrow points to normal bone marrow (fat)
cells. Slight replacement of bone marrow with inflammatory tissue was
observed in the white box, as indicated by the replacement of normal fat
cells by smaller cells.

coronal planes from the same image. Bone lesions were observed in the
white boxes in the follow-up image.
(c) Difference image of the registered follow-up (day +21) and baseline
image. From left to right, it shows the transaxial, coronal and sagittal
views of the same image. Bright bone lesions were observed in the white
boxes that correspond to candidate bone lesion region 24.

3.8. Results 71
(a) Histological section of the right leg of subject 8 (H&E, magnification

2). Slight replacement of bone marrow with inflammatory tissue was
observed in the black dotted box. Bright bone lesions were observed
in the black dotted boxes that do not correspond to any candidate bone
lesion region.

coronal planes from the same image. Bone lesions were observed in the
black dotted boxes in the follow-up image.
(c) Difference image of the registered follow-up (day +21) and baseline
image. From left to right, it shows the transaxial, coronal and sagittal
views of the same image.

3.9. Discussion and conclusions 72
Figure 3.21: Relationship between the volume of manual bone lesion segmentation and that of the auto-
matic bone lesion segmentation in candidate bone lesion region 24. The slope and intercept of the linear
regression are 0.578 and 3.99.
3.8.2.6 Comparison with manual bone lesion volume

Candidate bone lesion region 24 was randomly chosen and bone lesions within this region were manually
delineated from all the time points. There were strong positive correlations between the volume of
manual bone lesion segmentation and that of the automatic bone lesion segmentation (Pearson correlation
= 0.867), as shown in Figure 3.21.
3.9 Discussion and conclusions

Two different methods were applied to automatically quantify changes in the talus bones of 11 subjects
over 6 time points. Method 1, based on segmentation propagation, was used to accurately delineate the
bones in order to measure changes in the bone volume. A total of 65 segmentations were performed
(discounting the one bone at one time point used for the atlas). The global bone volumes were plotted
against time, and the automatic segmentations were compared with those obtained by labour-intensive
manual segmentation for subjects 13. The agreement, assessed using the similarity index, between
the automatic segmentations and manual segmentations is very close to the inter-observer agreement
in the manual segmentation. It is expected from the disease model that the bone volume will decrease
monotonically over time as the disease develops and erosions progress. However, the results in Figure 3.7
show considerable temporal variability in volume. This could be due to errors in the segmentation
technique, or could result from geometrical instability in the MR scanner, which are discussed in more
detail below. Method 2, based on the intensity analysis of the bones delineated by Method 1, was used to
automatically identify candidate bone lesion regions within the talus bones and hence quantify disease
progression in each subject from this intensity information. It has advantages over the first method
because it is not dependent on accurate boundary delineation and can identify the locations of lesions
in the bone rather than just giving a single measure for the bone. A total of eight candidate bone lesion
regions were identified and were shown using surface rendering to allow visualisation in 3D. Region 24
and 105 were found to be near the RA-like pathology in the histology sections. In regions 24, 93 and
121, bone lesion volume in male and female subjects were significantly different. Significant changes
between some time points were also detected in the same regions. Therefore, the bone lesions volumes
in the candidate bone lesion regions calculated by Method 2 are likely to be sensitive to the disease
progression, and have the potential to be used as the basis of bone erosion and oedema biomarkers in
order to quantify the disease progression in the experimental model of RA.
Possible reasons for the fluctuation in the bone volume in Figure 3.7 are that (a) the bone volume is
a global measure and include changes due to other factors such as growth and bone remodelling; (b) the
precision of the automatic and manual segmentations are too low to detect the volume changes involved
(the average intra-observer variability - assessed by similarity index - of the manual segmentation was
0.9786); (c) the amount of change in the bone volume may be comparable to the geometric distortion
caused by field inhomogeneity, gradient miscalibration, and other errors relating to the stability of the
MR scanner. Therefore, it is expected that the segmentation propagation would be more useful in as-
sessing the long-term effect of RA in the later stage when bone erosions are more prominent. In the
experimental subjects studied here, the intensity analysis of Method 2 was found more effective as it
was able to identify gender differences in the disease model, identified lesions that were consistent with
histology, and provided good agreement with manual lesion delineation for one selected region (Region
24).
It might be possible to improve the performance of Method 1 by correcting for geometric insta-
bility in the scanner. Since no phantom experiment was carried out to correct for the geometric dis-
tortion in the images, we tried to recover the linear changes in the voxel size of the images using the
nine degrees of freedom (translation, rotation and scaling) registration method (or the 9-dof registration
method) [WSL+ 04] that has previously been used in longitudinal brain imaging. After registering the
talus bone in each follow-up image to that in the baseline image using the 9-dof registration, the lin-
ear changes in the voxel size in each follow-up image was given by the scaling factors in the results.
However, when another bone (the calcaneous bone) was used instead of the talus bone in the 9-dof reg-
istration, different scaling factors were obtained (see Table 3.6). This suggests that the 9-dof registration
method, which was devised to correct for linear changes in voxel size in serial brain MRI studies, may
not be applicable to MR images of joints, where disease related changes are likely to be correlated with
scaling changes. Another possibility is that the geometric distortion is non-linear, rather than linear.
However, the scaling factor recovered using the talus bone may partly explain the apparent consistency
of the progression of global bone volume because the largest scaling (shrinking) factor is at day +3 (see
Figure 3.22), which is coincident with the systemic drop in bone volume at day +3 in Figure 3.7. Since
the method based on the intensity analysis detected increase in bone lesion volume over time in region
24, it seems that the method is less sensitive to these geometric instabilities in the scanner.
Table 3.6: Average scaling factors recovered using the 9-dof registration of the talus and calcaneous
Bone Time point Mean (Standard deviation) 95% confidence interval

Talus 4 0.9969 (0.0141) 0.98861.0052
+3 0.9637 (0.0234) 0.94990.9775
+10 0.9569 (0.0322) 0.93790.9760
+14 0.9775 (0.0171) 0.96740.9876
+21 0.9780 (0.0208) 0.96570.9903
Calcaneous 4 1.0153 (0.0157) 1.00591.0246
+3 1.0088 (0.0274) 0.99261.0250
+10 1.0143 (0.0347) 0.99391.0348
+14 1.0427 (0.0237) 1.02871.0567
+21 1.0518 (0.0329) 1.03241.0713
Figure 3.22: The graph of the average scaling factors recovered using the talus bone against time
The results of Method 2 were compared with the histology of the subjects. Comparison of in-vivo
imaging and histology remains a great challenge in many aspects of medical imaging. The work in
this chapter is not a full validation, but the histology does support the findings of Method 2. Candidate
bone lesion region 24 and region 105 were close to regions with RA-like pathology in the histology
sections of 4 subjects (true positives). Since only 1 histological section per stain was taken from each
leg, no information was provided in other parts of the leg and therefore it is impossible to correlate all the
candidate bone lesion regions with the histology. This prevents assessing whether Method 2 generates
false positives compared to histology. In future studies, more histology sections could be taken to try
to correlate other candidate bone lesion regions in the talus bone, but this is a highly labour intensive
procedure, and in any case, the preparation of the histology sections is likely to make it difficult to detect
bone marrow oedema (see Section 2.7). The histology sections also contained bone lesions that were
not close to any candidate bone lesion regions found from the MRI (false negatives). This is probably
because those bone lesions are small (black dotted boxes in Figure 3.18) in all the subjects. Chapter 4
addresses this issue by incorporating time-domain information across all the time points.
The results of Method 2 were also supported by strong correlations with the volume of the manual
segmentation of bone lesions (Pearson correlation = 0.867). A major reason for the discrepancies be-
tween our results and the volume of manual segmentation was the difficulties in estimating normal bone
border in the manual segmentation as suggested in [BJL+ 05].
Method 2 could be applied to measure similar image features that satisfy the assumptions and
observations stated in Section 3.5.2. Other possible applications include the quantification of disease
progression in human RA or osteoarthritis. Most parameters in the method would remain the same
except the control point spacing used in the inter-subject nonrigid registration. The kernel used in the
mathematical morphology can be simplified to used a normal 3 3 3 structuring element if the images
have isotropic voxel size.
Errors in the calculation of bone lesion volume was estimated by using images of the normal left
ankles, assuming that the disease does not affect the left ankles. The errors were found to be small
(maximum 10 voxels in regions 1 and 21), and less than twice of the standard deviation of the bone
lesion volumes in the diseased right ankles. This suggests that the variability in the subjects has a larger
effect on the results of Method 2, and have little effect on the results of the statistical analysis.
The change in the bone lesion volume in all bone lesion regions was very small, less than 1% of the
volume of the talus bone. It would be very hard to achieve this level of accuracy in manual segmentations.
A lot of labour was saved in automatically identifying candidate bone lesion regions in the images. It
would have been extremely difficult to make these comparisons without the automatic mapping of all
bones to a reference coordinate frame.
The correct mapping of the Otsus thresholded difference images into a reference coordinate system
(i.e. atlas) was very important. The mapping into a reference coordinate system was initially carried out
using affine registration. However, it was found that bone lesions were mapped outside the talus bone in
the atlas as shown in Figure 3.23. Subsequent use of nonrigid alignment greatly improved the results.
Figure 3.23: An example of mapping potential lesion voxels to the atlas using affine registration. The
figure shows the potential lesion voxels (in green) in the talus bone at day +3 after mapping to the atlas
using affine registration. The figure shows the transaxial, coronal and sagittal view (from left to right)
from the same image.
In the calculation of the bone lesion volume, the automatic segmentations of the talus bones were
eroded to act as a mask to apply to the bone lesion regions in order to decrease the sensitivity to misreg-
istration. This also decreased the sensitivity to any bone lesion around the edge of the bone, especially
in the y direction. Decreasing the voxel size in the y direction would also help to increase the sensitivity
to bone lesion around the edge of the bone.
Few studies focus on discussing the area or region of the talus that are involved in the disease model.
Chapter 2 mentioned a study comparing the erosions seen in MRI and micro-CT (see Figure 2.6). By
comparing it with Figure 3.15, it can be seen that regions 1, 24 and 37 are close to the erosion seen in
the micro-CT images.
Other candidate bone lesion regions show a different trend of lesion progression than region 24. The
size of the lesion generally peaks at day +3 or day +10. This may be a reflection of the acute exudative
inflammation in the bone, which peaks at 3 days post-reinjection as described in histology of the disease
model (see Figure 2.5).
This chapter describes two different methods that use image registration to monitor longitudinal
imaging changes in an animal model of RA. Such tools are of great value in drug discovery and devel-
opment, where image-based biomarkers have the potential to provide sensitive measures of the effect of
drugs on disease progression. The two approaches were compared on a cohort of 11 subjects. I used both
segmentation propagation to automatically quantify volume changes over time, and a spatial-temporal
analysis of intensity features on the cohort within a coordinate frame defined by an atlas image. Com-
parison of the result of the analysis with known characteristics (sex predilection and lesion progression)
of this animal model, histology from these subjects, and manual segmentation of bone lesions demon-
strated that the intensity analysis approach was likely to have greater sensitivity to changes caused by
the disease. Unlike the currently used histological technique, the methods in this chapter provide a true
3D assessment of pathology.
Chapter 4
Automatic Quantification of Changes in

Longitudinal MR Images of Joints II
4.1 Introduction
Existing techniques for analysing longitudinal images often compare each image in the temporal se-
quence to the baseline image, and extract image features to study the progression of the disease over
time [FWF+ 96a, OHS+ 99, FRS+ 02, CLR+ 04]. This approach has been used to quantify the progression
of many diseases, such as RA (Chapter 3), Alzheimers disease (AD) [FWF+ 96a, THdZ+ 03], multiple
sclerosis (MS) [SYS+ 05] and osteoarthritis (OA) [GPZS05, BLD+ 04]. However, this pair-wise analysis
of images does not make full use of the time-domain information available in the temporal sequence.
Based on the conclusions of Chapter 3 that bone lesion changes are localised in space and time, a
more integrated spatio-temporal analysis technique is proposed in this chapter. The main idea is to build
a 5D feature space by mapping the (x, y, z, intensity, time) value of each voxel of difference images to a
point in the feature space. The 5D feature space can then be segmented according to the density of the
points using the mean shift algorithm [FH75, CM02]. The results are 4D segmentations of bone lesions
that extend across multiple time points. Inter-subject nonrigid registration then enables the mapping of
bone lesions of different subjects to a common reference coordinate system defined by an atlas bone.
This allows the comparison of localised bone lesion volume changes in anatomically corresponding
locations between different subjects or groups. Using simulated and real MR images, this new approach
is compared with the methods used in Chapter 3
The potential advantages of this technique are that (a) localised temporal changes are delineated as
4D segmentations so that changes identified in one follow-up image are directly related to the changes
identified in another follow-up image; (b) results are more consistent and more robust to errors (due to
registration) or image artifacts present in one follow-up image.
4.2. Aims and contribution 78

Aims
To devise a technique that combines the space and time-domain information of a temporal se-
quence for the quantification of the disease progression in an experimental model of RA.
To compare the accuracy of this technique with the methods used in Chapter 3.
To validate the results of this technique by comparing them against the disease model, the histology
data and the manual segmentations of bone lesions.
Contribution
This chapter describes an integrated spatio-temporal image analysis technique that incorporates the space
and time-domain information from temporal image sequences, and applies it to study the progression of
bone lesions in an experimental model of RA. After building a 5D feature space from the temporal
sequence, 4D segmentations of bone lesions are obtained using the mean shift algorithm, and changes
in the bone lesion volumes are quantified. Comparisons with the methods described in Chapter 3 using
real and simulated data show that the spatio-temporal technique are more sensitive to small changes and
robust to image noise. Finally, the results of spatio-temporal technique are validated using the disease
model, histology data and the manual segmentations of bone lesions.

GSK prepared the experimental model and acquired the longitudinal images. The VTK CISG image reg-
istration toolkit, which was developed by Thomas Hartkens, was used to perform the image registration
and visualisation. All other image and data analysis was my own work.
4.3 Review
This is the first work to incorporate information from all the serial images (more than 2) to directly
extract 4D image features in order to quantify temporal changes in bone diseases. Existing methods use
either a single image [BLSE03, OHSL96] or 2 images [CLR+ 04] to quantify the disease progression.
Although many spatio-temporal analysis techniques have been proposed to analyse fMRI data, they
are not directly applicable to the problem of identifying bone lesions in RA because (a) they focus on
studying features that change in intensity, but bone lesions are image features that change in both size
and intensity; (b) there is a limitation on the number of time points in longitudinal structural imaging
studies such as in RA, which generally have fewer time points than fMRI studies; (c) there is no reference
pattern in RA against which the intensity variation can be matched.
Other related work mostly comes from image analysis of MR brain images of diseases such as
AD and MS. In [FF97], time-domain information was incorporated by calculating the difference images
using baseline and follow-up images to analyse the progression of AD. In [MG03], a general time-
series analysis technique was proposed to analyse serial MR images of MS brains. The time-domain
information was integrated in the analysis by creating ratio or difference images after spatial and intensity
4.4. Theory 79
normalisation. In [WGR+ 01], a 4D MS lesion model was built by applying principal component analysis
(PCA) to analyse the spatio-temporal evolution (which includes radius, time, and intensity) of a database
of 23 manually segmented active MS lesions over 6 time points. The model was used to segment lesions
from other 4D MR images. The algorithm described in this chapter does not require segmentation
of each time point prior to the spatio-temporal analysis, but uses the spatio-temporal analysis for the
segmentation task.
Davatzikos and colleagues [XSD05, fan, SD04] proposed a temporally-consistent and spatially-
adaptive longitudinal MR brain image segmentation algorithm. Time-domain information was directly
integrated into the calculation of the objective function by the addition of a temporally-adaptive smooth-
ness constraint. Experiments with simulated and real data showed that the volume of the segmented
tissues over time were smoother and more consistent than the results from FANTASM [fan], in which
time-domain information was not used.
Time-domain information has been directly included in image registration to improve temporal
consistency and robustness. In [SD04], the attribute vector was modified to include voxels from adjacent
time points. In [CMR04a], the 3D B-spline in the free-form deformation was extended to a 4D B-
spline that added control points in the temporal dimension. The deformation was thus affected by the
displacement of control points in adjacent time points.
4.4 Theory
4.4.1 Mean shift
The 5D feature space is built by mapping the (x, y, z, intensity, time) value of each voxel in an image to a
point in it, and is analysed by mean shift [FH75], which is reviewed in Chapter 2. Mean shift is a robust
nonparametric feature space analysis technique that has been applied to the segmentation of grey level
and colour images [CM02], and the spatio-temporal segmentation of video sequences [DeM02]. The
method has subsequently been applied to medical images. For example, Jimenez et al. combined mean
shift with a robust, edge-oriented region fusion technique to delineate structures in brain MRI [JMY03].
Mean shift has advantages over other clustering techniques by not requiring a prior knowledge of the
number of clusters, and by not making any assumption about the shape of the clusters in the feature
space [CM02].
For a temporal sequence of 3-dimensional grey-level images, a 5-dimensional feature space is
formed by mapping the spatial coordinates (x, y, and z), intensity, and time value of each voxel to a
point in the feature space. Let ~xi , i = 1, ..., n be the d-dimensional input image voxels in the feature
space. The mean shift segmentation is as follows:
1. The mean shift filtering: for each point ~xi in the feature space
(a) Initialise j = 1 and ~yi,1 = ~xi .

y~ ~x 2
xi g i,jh i
Pn
~
i=1
(b) Compute ~yi,j+1 = Pn

y

xi 2
~i,j ~
until convergence at ~yi,c , where g is the profile of
i=1 g h
a radially symmetric kernel with single window width h (see (2.27) in Chapter 2).
2. The grouping of similar modes: delineate the feature space by grouping all ~yi,c that are within the
window width of the kernel.
Other reasons for using mean shift are that (a) the method is simple to use because the result only
depends on the window width of a given kernel in each dimension. And there is the potential for an
autonomous image segmentation algorithm by combining a window width selection technique with prior
knowledge of the images [CM02]; (b) the feature space used for the 3D grey level segmentation can be
easily extended to include the dimension in time by giving each voxel a time value depending on which
follow-up image it belongs to. Furthermore, the feature space can also be extended to include data
from multispectral images, which has been used to classify tissues in the MCP joints of the hands in
RA [CLR+ 04].

4.5.1 Overview
The results in Chapter 3 show that bone lesions are localised in the space, time, and intensity of the
difference images because (a) spatially and temporally localised candidate bone lesion regions are found;
(b) the intensity thresholding technique works reasonably well to delineate the bone lesions. This is the
major motivation for delineating the bone lesions directly from the space-time-intensity space.
The methods described in this chapter provide a more integrated spatio-temporal analysis technique
for analysing longitudinal images, which can be applied to an arbitrary number of bones in the hand and
foot - each of which is treated separately in the analysis. Like Chapter 3, the analysis is performed in a
common coordinate system. Therefore, it shares some of the analysis steps described in Chapter 3, and
they are included in this chapter for ease of reading and completeness. The main steps are:
1. Atlas construction: this is used to delineate ROI from the serial images, and define a common
coordinate system for analysis. This is the same Atlas construction step in Chapter 3.
2. Determination of ROI: using nonrigid registration [RSH+ 99] and segmentation propaga-
tion [CR00], the ROI, a bone of interest in the problem, is automatically delineated from the
longitudinal images using a pre-segmented atlas image from a reference subject. This is the
same Determination of ROI step in Chapter 3.
3. Generation of difference images: based on the observation that bone lesion changes are associated
with intensity changes in difference images, difference images are generated by subtracting the
baseline image from the registered follow-up images for later analysis.
4. Spatio-temporal segmentation: the difference images of the same subject within the ROI are used
to build a 5D feature space, which is segmented by using mean shift to identify the bone lesions
as 4D segmentations. A way to speed up the segmentation is also discussed.
5. Identification of candidate bone lesion regions: based on the observation that bone lesions arise
in similar locations in all the subjects, all the subjects and all the time points are combined in a
common reference coordinate system to identify these regions.
6. Calculation of bone lesion volume: the candidate bone lesion regions are mapped back to the
coordinate system of each image. The volume of bone lesion belonging to each region is counted.
After describing the image data, each step will be explained in detail. Next, the methods are tested on
simulated and real data. The purposes of testing on the simulated data are to select the window width pa-
rameters, and to quantitatively compare this technique with the methods described in Chapter 3. Finally,
the results obtained from the real data are compared to those of the methods described in Chapter 3,
single-sliced histology data and the manual segmentations of bone lesions.
4.5.2 Image data

The same data set described in Section 3.6.1 of Chapter 3 were used. It consisted of a group of 12 Lewis
rats. T1-weighted MR images of the right ankle were taken at six time points: day -12, day -4, day
+3, day +10, day +14 and day +21. Only data from subjects 1 - 11 were analysed because subject 12
(female) died after the first MR scan.
4.5.2.1 Simulated image data

Due to the lack of a gold standard in clinical data, simulated image data were used to compare the perfor-
mance of the method in Chapter 3 and the spatio-temporal segmentation method proposed this chapter.
Longitudinal images were generated by adding simulated bone lesions and noise to a randomly selected
baseline image from the real data. Bone lesions were simulated using the shape of an ellipsoid, which
had a similar size, intensity, and location to a typical real bone lesion from an MR image (confirmed by
histology). Hence, the mean intensity of the lesion was found to be 7487, and the sizes of the lesions
were chosen to be 0, 42, 22, 52, 90, and 163 voxels from the first to the last time point. To simulate the
partial volume effect around the edge of the lesion, a Gaussian kernel of standard deviation (1.0, 1.0, 1.0)
voxel was chosen to approximate the point spread function of the MRI and was used to blur the edges of
the lesions.
Noise was added to the simulated images by assuming that the noise distribution was Gaus-
sian in both the real and imaginary component. Since only modulus images were available, let f (x)
be the modulus of an MR image at voxel location x, the image with simulated noise was given by
p
(f (x) + n1 (x))2 + n2 (x)2 ), where n1 (x), n2 (x) were Gaussian distributions with mean zero and
standard deviation . The value of , which was called noise standard deviation, in a modulus image
can be estimated by dividing the standard deviation of a region of interest (ROI) containing only air by
0.655 [Hen85].
4.5.3 Atlas construction

The baseline image of subject 1 was selected to be the reference image, which was the same reference
image used in Chapter 3. An expert manually segmented the talus bone in the baseline image. The
baseline image plus the manually segmented bone formed the atlas.
4.5.4 Determination of ROI

The same ROIs obtained in Section 3.6.3.1 of Chapter 3 were used in the chapter. Segmentation prop-
agation based on multi-stage registrations [HSH02] was used to automatically delineate the ROI, the
talus bone, from each time point in the longitudinal images. The source image was the reference image
in the atlas. The target images were all time points of the 11 subjects. Each time point was treated
independently. The first stage was an affine registration with 12 degrees of freedom (DOF). The second,
third, and fourth (final) stages were cubic B-spline nonrigid registrations with control point spacing of
1.172mm, 0.586mm and 0.293mm respectively. The final deformation field was used to propagate the
manual segmentation in the atlas to obtain a boundary of the corresponding bone in the target image.
Correlation coefficient (CC) (for its larger capture range) and normalised mutual information (NMI) (for
handling intensity differences due to inter-subject variability) were used as the similarity measures in the
affine registration and nonrigid registration respectively.
4.5.5 Generation of difference images

Rigid registration was used to register the talus bone in the baseline image and the follow-up images.
The target image was the baseline image, and the source images were the follow-up images of the
same subject. CC was used as the similarity measure. Follow-up images were transformed using the
resulting transformations by sinc interpolation (using Hanning window of kernel size (diameter) = 7
voxels (0.352 0.936 0.352(mm)3 )). Difference images were generated by subtracting the baseline
image from the transformed follow-up images. The intensity of the ROI in the serial images of the same
subject was normalised by scaling the mean intensity of each ROI to the mean intensity of all the ROIs.
4.5.6 Spatio-temporal segmentation

The aim of this step was to segment the high-intensity bone lesions from the difference images. For
each subject, 5 difference images were generated from the 6 time points. Since the amount and rate of
pathological changes between two adjacent time points were unknown, they were assumed to be similar.
Therefore, the difference images were assigned time values from 0 to 4 according to the chronological
order so that the time value between 2 adjacent time points was the same. A 5D feature space was built
by mapping the (x, y, z, intensity, time) value of each voxel of the difference images to a point in it.
The mean shift segmentation was applied to segment the feature spaces using a Gaussian kernel, which
generated better results according to [CM02]. Since it was observed that most bone tissues were normal,
the largest segmentation belonged to normal bone tissues. High-intensity bone lesions were given by the
4D segmentations whose mean intensity was higher than that of the largest segmentation. The window
width was chosen to be 0.24mm in the spatial dimensions, 400 in the intensity dimension, and 1.0 in
the temporal dimension. These parameters were chosen through a combination of prior knowledge and
experimental results from simulated data, and will be discussed in more details in Section 4.5.10.
4.5.7 Identification of candidate bone lesion regions

The segmented bone lesions in all the time points were mapped to the common coordinate system of the
atlas by applying the inverse of the nonrigid deformation fields obtained from the determination of ROI
in Section 4.5.4. They were thresholded to binary images so that the value 1 in a voxel represented bone
lesion.
After summing the binary images, bone lesions that were found in only one time point were removed
by thresholding (using the value 2) to ensure that the final candidate bone regions consisted of bone
lesions that appeared in two or more time points. Candidate bone lesion regions, which were defined in
the coordinate system of the atlas, in the talus bones, were defined by the connected components in the
image.
4.5.8 Calculation of bone lesion volume

The candidate bone lesion regions in the image space of the atlas were transformed to the image space
of the baseline image of each subject by applying the nonrigid deformation fields obtained from the
determination of ROI. Since the 4D segmentations of bone lesions might not completely overlap with
the candidate bone lesion regions, the 4D segmentations were assigned to the regions in the following
way: (a) If a bone lesion voxel was inside a region, it was assigned to that region. This was performed
for all the voxels; (b) If a bone lesion voxel was outside any region, it was assigned to the region that
contained the highest number of voxels from the same 4D segmentations of that voxel. Otherwise, the
voxel was discarded.
4.5.9 Speed improvement of the mean shift segmentation by subsampling

The aim of this section is to investigate a possible way to speed up the mean shift segmentation. Since the
computational complexity of the mean shift segmentation depends on the number of points in the feature
space, the speed can be improved by removing some points from the feature space that are known to
be mapped from the voxels of normal bone tissue. However, this will confound the density estimation
because less sample data are now available [Sil86]. Instead, all the points in the feature space are kept,
but not all the points have to be processed by the mean shift segmentation. For example, consider two
points in the feature space that are mapped from two nearby voxels of normal bone tissue in the difference
image. This means that the two voxels will probably have similar spatial coordinates, similar intensity
values and the same time value. After the mean shift filtering, it is therefore highly likely that they will
converge to the same location or mode in the feature space. Since the next procedure (the grouping
of similar modes) only depends on the modes found in the mean shift filtering, one of the mean shift
filterings is redundant. As a result, the mean shift segmentation can be speeded up by not processing all
the voxels of normal bone tissue that converge to the same mode after the mean shift filtering.
However, it is non-trivial to identify the voxels that will converge to the same mode without actually
performing the mean shift filtering. Instead, the voxels that converge to similar modes after the mean shift
filtering are considered. Although only an approximation of the mean shift segmentation is obtained by
excluding those voxels in the mean shift filtering, the important point is that those voxels can be identified
quickly and easily. As in Chapter 3, most of the voxels with similar intensity values from the normal
bone tissue in the difference image have intensity values that are less than the mean intensity seq plus
the standard deviation seq of the intensity values of the temporal sequence. The locations of the voxels
are given by
{(x, y, z, t) : v(x, y, z, t) < seq + seq }, (4.1)
where v(x, y, z, t) is the intensity of the voxel at (x, y, z, t). Next, the above voxels are spatially sub-
sampled in order to exclude the voxels that have similar spatial coordinates and similar intensity values.
From (4.1), the mean shift segmentation can be speeded up by only processing the following voxels to
give an approximate solution:
{(x, y, z, t) : (x ksx ) (y lsy ) (z msz ) (v(x, y, z, t) < seq + seq )}, (4.2)
where sx , sy and sz are the voxel sizes in x, y and z directions respectively, and k, l and m are non-zero
integers and represent the sampling factors along the x, y and z directions respectively.
Experiments will be described in Section 4.5.10 to compare the approximations of the mean shift
segmentation with the exact solutions using various sampling factors.
4.5.10 Experiments
4.5.10.1 Simulated data
As described in Section 4.5.2.1, a set of simulated longitudinal images with 6 time points could be
generated by adding simulated bone lesions and noise to a randomly selected baseline image. A total
of 5 sets with noise standard deviations 0.6r , 0.8r , r , 1.2r , and 1.4r were generated and used in
the experiments, where r was the noise standard deviation in the randomly selected baseline image and
was calculated to be 914 (see Section 4.5.2.1).
Spatio-temporal segmentation using different window widths were applied to delineate bone lesions
from the 5 sets of data. All combinations of the following window widths were tested: (a) 0.16mm,
0.20mm, 0.24mm, 0.28mm, and 0.32mm in the spatial dimensions; (b) 200, 300, 400, 500, 600, 700,
and 800 in the intensity dimension; (c) 0.01, 0.6, 0.8, 1.0, 1.2, and 1.4 in the temporal dimension.
Otsus thresholding was also applied to delineate the bone lesions from the 5 sets of data. The
performance of the methods was compared using the similarity index [DHT+ 99] of the segmentations
and the reference bone lesions.
4.5.10.2 Real data

The spatio-temporal analysis technique (section 4.5.3 to section 4.5.8) was used to process the real data.
The results were qualitatively compared with those obtained by a previous method.
The following sampling factors (x, y, z) were applied to the real data to investigate the speed im-
provement of the mean shift segmentation: (2, 2, 2), (2, 1, 2), (3, 1, 3) and (4, 1, 4). The results were
compared to the exact solutions of the mean shift segmentation using the similarity index.
4.5.11 Comparison with manual segmentation of the bone lesions

As in Chapter 3, to support the results of the automatic measurement of changes in bone lesion volume,
bone lesions in a candidate bone lesion region were manually delineated from all the time points and
were used to compare with the results.
4.6. Results 85
(a) Histolog- (b) Real image (c) Simulated

ical section at day +21 image at day
+21
Figure 4.1: Comparison between the histological section, real image, and the simulated image of subject
9. (a) Histological section of the right leg of subject 9 (H&E, magnification 2). Moderate bone
destruction was observed in the white box. (b) Real image at day +21. (c) Simulated image with noise
standard deviation 914 at day +21 generated from the real baseline image (day -12). The volume of
simulated bone lesion was 163 voxels.
4.6 Results
4.6.1 Experiments on simulated data
The baseline image of subject 9 was randomly selected as the basis of the simulated data. The histo-
logical section, the real image, and the simulated image with noise standard deviation 914 are shown in
Figure 4.1.
ST is used to refer to the method described in this chapter, and OT is used to refer to the method
described in Chapter 3. In general, ST performed better than OT. Figure 4.2 shows the graph of similarity
index against noise standard deviation using kernel window width of 0.2mm in the spatial dimensions
600 in the intensity dimension, and 1.0 in the temporal dimension. The performance of OT was close
to the performance of ST when the size of bone lesion was large and the noise standard deviation was
small.
4.6.1.1 Effect and choice of window width

The mean similarity index as a function of window width in each dimension was calculated by averaging
the similarity indices from all the experiments that used that window width. The effect of window
width in different dimensions is summarised in Figure 4.3. It was clear that suitable window widths
were around 400 in the intensity dimension, and around 0.24mm in the spatial dimension. A range of
window widths in the temporal dimension were found to produce similar results. The window width in
the temporal dimension was chosen to be 1.0. Furthermore, the effect of varying window widths around
the chosen window width at noise standard deviation 914 (estimated noise standard deviation in real
images) are summarised in table 4.1, which indicated that good results (mean similarity index > 0.83)
with small standard deviation (< 0.041) were obtained around the chosen window width.
4.6. Results 86
Figure 4.2: Graph of similarity index against noise standard deviation for 5 different lesion volumes.
Similar trends were observed in all lesions.
Figure 4.3: Effect of window size in different dimensions. Error bar indicates the standard deviation of
the mean similarity index.
4.6. Results 87
Table 4.1: Effect of varying window width around the window width (0.24mm, 0.24mm, 0.24mm, 400,
1.0) at noise standard deviation 914.
Varying window width Fixed window width Mean similarity index

(standard deviation)
intensity = 300, 400, 500, spatial = 0.24mm, time = 1.0 0.831 (0.0377)
600, 700, 800
time = 0.6, 0.8, 1.0, 1.2, 1.4 spatial = 0.24mm, intensity = 400 0.834 (0.0401)
spatial = 0.16mm, 0.2mm, intensity = 400, time = 1.0 0.839 (0.0295)
0.24mm, 0.28mm,
0.32mm
4.6.1.2 Effect of time-domain information

To investigate the effect of time-domain information on the segmentation of bone lesions, the informa-
tion was removed by setting the window width in the temporal dimension to be very small (0.01), which
meant that the spatio-temporal segmentation was effectively performed on each difference image sepa-
rately. The effect is shown in Figure 4.4. The larger window width in the temporal dimension produced
more consistent and smoother results in different sizes of bone lesions across the time points. The small
window width in the temporal dimension had a larger effect on the results with small window widths in
the intensity dimension and small lesion volume.
4.6.2 Experiments on real data

Experiments using real data produced results that were consistent with those using simulated data. The
lesion segmentations of subject 9 from OT and ST were more similar at the time point day +21 than at
the time point day 4 as shown in Figure 4.5. Furthermore, the mean similarity indices of the lesion
segmentations of all the subjects generated by OT and ST at the time point day +21 and day 4 were
0.756 (0.061) and 0.498 (0.217) respectively. Since the volume of bone lesion at day +21 was larger
than that in day 4 according to the disease model, the results showed that the performance of OT was
close to the performance of ST when the volume of bone lesion was large. This was consistent with the
results of the experiments using simulated data.
Seven candidate bone lesion regions were found. Their locations were similar to ones found using
OT, and the mappings to the previous regions are shown in Table 4.2. A new candidate bone lesion region
was also found, and the number of lesions belonging to each candidate bone lesion region was different.
The average bone lesion volume of region 3 is shown in Figure 4.6, and the average bone lesion volumes
of other regions are shown in Appendix B.
When examining the influence of time and gender on the bone lesion volume in this longitudinal
study, we used a repeated measure analysis of variance. The bone lesion volumes in all the candidate
regions at all the time points were analysed in an analysis of variance using SPSS (SPSS Inc., Chicago,
Illinois, USA) with time (day -4 vs day +3 vs day +10 vs day +14 vs day +21) as a within-subject
factor and gender (male vs female) as a between-subject factor. The results are shown in Table 4.2. The
4.6. Results 88
Figure 4.4: Effect of very small window width in the temporal dimension. The graph of similarity index
against bone lesion volume in simulated images using temporal window width 0.01 (top). The graph
of similarity index against bone lesion volume in simulated images using temporal window width 1.0
(bottom). In both cases, the noise standard deviation in the simulated images is 914, the spatial window
width is 0.2mm, and the intensity window widths are shown inside the brackets in the legend.
4.6. Results 89
(a) Day -4 (b) Day +21
Figure 4.5: Comparison of the bone lesion segmented by OT and ST at different time points of the talus
bone of subject 9. The images are shown in coronal views with segmentations highlighted in white. In
each sub-figure, the segmentation on the left was obtained by OT, and the segmentation on the right was
obtained by ST.
Table 4.2: Results of the comparison of the bone lesion volume between different time points, and
between male and female subjects using ANOVA. P values that are significant at P < 0.05 are marked
with *.
Region Within-subject factor Between-subject factor Region from previous

(time) (gender) method
Region 1 P < 0.0005* P < 0.836 Region 1
Region 2 P < 0.094 P < 0.712 Region 7
Region 3 P < 0.0005* P < 0.050* Region 24
Region 4 P < 0.0005* P < 0.619 Region 37, 121, and 105
Region 5 P < 0.217 P < 0.535 Region 41
Region 6 P < 0.003* P < 0.086 n/a
Region 7 P < 0.0005* P < 0.165 Region 93
ANOVA yielded significant time effect in regions 1, 3, 4, 6, and 7, and significant gender between-subject
effect in region 3 only.
Pair comparisons between bone lesion volume at different time points in regions 1, 3, 4, 6, and
7 (regions where the ANOVA yielded significant time main effect) were performed using Bonferroni
adjustment. Only pair comparisons that are significant at P < 0.05 are shown in Table 4.3.
4.6.2.1 Speed improvement of the mean shift segmentation

After applying the mean shift segmentation with various sampling factors to the real data, the results
were compared to the exact solutions using the similarity index. The mean similarity index for each
sampling factor was calculated by averaging the similarity indices obtained from all the time points.
Table 4.4 summarises the comparisons between the approximations and the exact solutions of the mean
shift segmentation. By subsampling the voxels of the normal bone tissue by a factor of (3, 1, 3), the
4.6. Results 90
Figure 4.6: Graph of average bone lesion volume of male and female subjects in region 3 obtained by
ST against time. Error bars denote the standard deviations of the average bone volume.
Table 4.3: Results of pair comparisons that are significant at P < 0.05 between different time points in
regions 1, 3, 7, 8, 9, and 10.
Region Pair (time point a and time point b) Significance

Region 1 Day -4 and day +14 P < 0.021
Day +3 and day +14 P < 0.012
Day +3 and day +10 P < 0.012
Day +3 and day +14 P < 0.004
Day +3 and day +21 P < 0.046
Day +10 and day +14 P < 0.045
Day -4 and day +10 P < 0.003
Day -4 and day +10 P < 0.002
Day -4 and day +21 P < 0.032
Day +10 and day +14 P < 0.040
Day +10 and day +21 P < 0.032
4.6. Results 91
Table 4.4: Comparisons between the approximations and the exact solutions of the mean shift segmen-
tations.
Sampling factor Mean similarity index Speed improvement

(2, 2, 2) 0.9615 4.325
(2, 1, 2) 0.9979 2.810
(3, 1, 3) 0.9979 4.189
(4, 1, 4) 0.9622 4.945
Table 4.5: Comparison between histology and the results.
Subject Candidate bone lesion region that is Bone lesion volume calculated by the spatio-
close to bone lesion observed in the temporal analysis technique in the candidate
histological section bone lesion region
running time of the mean shift segmentation was reduced by a factor of 4.2 with little loss in the accuracy
of the segmentations when compared to the exact solutions (similarity index = 0.9979.)
4.6.3 Comparison with histology

Histology sections of the 11 subjects were examined microscopically. Only 5 subjects were observed to
show RA-like pathology, such as bone destruction and lesion, in the histology section taken through the
talus bone. However, because only a single slice of histology was available, it was not possible to judge
whether RA-like pathology might have been visible elsewhere in the talus bone. Bone lesions in the
histology sections were found to be near to candidate bone lesion region 3 (in subject 4 and subject 8),
region 4 (in subject 2 and subject 3), and region 6 (in subject 2). Table 4.5 summarises the bone lesion
volumes calculated in those candidate bone lesion regions.
Figure 4.7 shows an example of the comparison between the histology and MR images. The figure
contains (a) a histological section of the right leg with RA-like pathology enclosed in white boxes; (b)
the follow-up image (at day +21), with candidate bone lesion regions highlighted in white, reformatted
in the transaxial plane, centred on the location in the MR image that was judged visually closest to
the histology lesion; (c) the follow-up image (at day +21), the baseline image and the corresponding
difference image reformatted in the same way as (b). MRI bone lesions that are near to the pathology
in the histological section are enclosed in white boxes. The histology bone lesions on the left, which
corresponds to the candidate bone lesion region 6, does not correspond to any of the candidate bone
lesion regions found in a previous method.
4.6. Results 92
(a) Histological section (b) Follow-up image at

day +21 with candi-
date bone lesion regions
highlighted in white
(c) Follow-up image at day +21(left), baseline image (middle) and the corresponding
difference image (right)
Figure 4.7: Comparison between histology section and MR image of subject 2. (a) Histological section
of the right leg of subject 2 (H&E, magnification 2). Moderate bone destruction was observed in the
white boxes. (b) Follow-up image at day +21 of subject 2 in transaxial plane with bone lesion regions
5 and 7 highlighted in white. (c) Follow-up image at day +21 (left), baseline image (middle) and the
corresponding difference image (right) of subject 2 in the transaxial planes. Bone lesions were observed
in the white boxes in the follow-up image. Bright bone lesions were observed in the white boxes that
correspond to candidate bone lesion regions 4 and 6.
Figure 4.8: Relationship between the volume of manual bone lesion segmentation and that of the auto-
matic bone lesion segmentation in candidate bone lesion region 3. The slope and intercept of the linear
regression are 1.24 and 26 respectively.
4.6.4 Comparison with manual bone lesion volume

In Chapter 3, candidate bone lesion region 3 was randomly chosen and bone lesions within this region
were manually delineated from all the time points. There were strong positive correlations between the
volume of manual bone lesion segmentation and that of the automatic bone lesion segmentation (Pearson
correlation = 0.782) (Figure 4.8).
4.6.5 Running time

The running time of the method was dominated by the spatio-temporal segmentation (approximately 12
hours). All the analysis was run on AMD Athlon 1.0GHz to 1.5GHz machines with 1.5Gb of RAM,
running Linux version 2.4.

An integrated spatio-temporal analysis technique is proposed in this chapter to identify candidate bone
lesion regions and calculate bone lesion volume in longitudinal MR images of joints. The technique is
applied to analyse the talus bones of simulated data with various bone lesion volumes and noise standard
deviations, and the real MR images of 11 subjects, which are scanned at 6 time points. The results
suggest that the spatio-temporal technique is more sensitive and robust at detecting small bone lesions in
both simulated and real data.
The results of the simulation show that (a) the spatio-temporal analysis technique is more sensitive
to small lesion changes in simulated data, as shown in Figure 4.2; (b) the inclusion of time-domain
information produces more sensitive and more consistent results, as shown in Figure 4.4; (c) the results
are not critically sensitive to the choice of window widths, as shown in Figure 4.3 and table 4.1. The
window widths are subsequently chosen to be 0.24mm in the spatial dimensions, 400 in the intensity
dimension, and 1.0 in the temporal dimension to analyse the real images.
The results are compared with the histology of the subjects. Accurate image registration and com-
parison of in-vivo imaging and histology remains a great challenge in many aspects of medical imaging.
The work in this chapter is not a full validation, but the histology does support the findings of the spatio-
temporal analysis technique. It is confirmed that candidate bone lesion regions 3, 4 ,and 6 are close to
regions with RA-like pathology in the histology sections of 4 subjects (true positives). And region 6 is
not detected by the method in Chapter 3, suggesting that the spatio-temporal segmentation method has
higher sensitivity to the disease (see Figure 4.7). Since only 1 histological section per stain is taken from
each leg, no information is provided on other parts of the leg, and therefore it is impossible to correlate
all the candidate bone lesion regions with the histology. This prevents an assessment of whether the
technique generates false positives compared to histology.
A difference between this method and the method from Chapter 3 is that no mathematical morphol-
ogy was required in this method. This helps preserve small candidate bone lesion regions such as region
6, which was not detected in a previous method.
The progressions of the lesions in the all candidate bone lesion regions are similar to the ones calcu-
lated in Chapter 3. The sizes of the lesions calculated in this chapter are larger because no mathematical
morphology is used. The fact that similar results are obtained using different methods suggests that the
measurements are less likely due to artifacts in the images or in the analysis.
The window widths in the mean shift algorithm are chosen based on the prior knowledge of the
size of bone lesion, and results from simulated data. The choices of window widths are not very critical
because experiments on simulated data show that a range of window widths in each dimension produces
similar results, as shown in Figure 4.3 and table 4.1. The chosen window widths have small standard
deviation of the similarity index, which means that results will be robust to different noise standard
deviations and changes to window widths. Several window widths (1.0, 1.2, and 1.4) in the temporal
dimension produce very similar results, and the value 1.0 is chosen because a smaller window width is
preferred to maximise the sensitivity to small details in the feature space. It should also be noted that
window widths are chosen using simulated data, and care should be taken not to overly rely on these
results. [CRM01] suggests an automatic method to determine the window width in an adaptive version
of mean shift. However, initial experiments do not produce satisfactory results in the simulated and real
data.
Although the results from simulated data are consistent with those from real data, the simulated
images are not a true representation of real data for the following reasons: (a) the shape and the intensity
distribution of real bone lesions are not necessarily elliptical and Gaussian; (b) image artifacts, such as
shading from B1 field inhomogeneity or motion ghosting, are present in real images and are different
in different time points and these are not included in the simulation; (c) there are other small normal
biological changes apart from the changes in bone lesions between time points in real data, such as
changes in the bone marrow or bone growth; and (d) there are errors due to image registration in real
data.
The running time of the spatio-temporal segmentation is approximately 12 hours using one com-
puter. By processing only the voxels from the normal bone tissue using a simple subsampling scheme,
the running time of the mean shift segmentation is reduced by a factor of 4.189 without sacrificing the
accuracy of the results. Furthermore, since the processing of voxels in the mean shift algorithm is inde-
pendent of each other, the processing can be executed in parallel. A parallel implementation using the
Condor Master-Worker [cona] will be discussed in Chapter 6, which shows that the processing can be
shortened to 3 hours by running it on 4 machines simultaneously.
The novel method is applied to simulated data and MR images from a cohort of 11 subjects. The
comparisons of the results with simulated data, and histology of 11 subjects demonstrate that the new
approach has greater sensitivity to small changes caused by the disease than the method in Chapter 3 that
do not fully take account of the time-domain information.
Chapter 5
Application to Clinical Data
5.1 Introduction
In chapters 3 and 4, longitudinal MR images from an experimental model of RA were analysed, and
the results showed that the methods were able to measure small changes in order to quantify disease
progression. This chapter describes the applications of the spatio-temporal segmentation and image
registration to facilitate the analysis of longitudinal MR hand and wrist images from human RA patients.
In collaboration with the rheumatology clinics at Guys and St Thomass Hospitals, 16 patients
that met the 1987 American College of Rheumatism criteria for rheumatoid arthritis [AEB+ 88], were
on stable treatment, and had evidence of active disease as indicated by the Disease Activity Score
(DAS) [vdHvtHvR+ 90] of greater than 3.0 were recruited. The study was designed to acquire MR
images of their dominant hands and wrists at five time points - 0, 3, 6, 9 and 12 months.
Since there are 5 metacarpal bones and 8 carpal bones in the hand and wrist, the first aim of this
chapter is to automatically localise and roughly delineate the metacarpal and carpal bones in patients
images to allow the identification of ROI around each bone for further analysis, such as accurate bone
delineation in the spatio-temporal segmentation and the automatic rigid bone alignment that will be
described later.
The second aim of this chapter is to demonstrate with the longitudinal clinical image data the po-
tential of the spatio-temporal segmentation to quantify changes in the MR images by measuring changes
in bone lesion volumes.
As part of the validation process, the OMERACT (Outcome Measures in Rheumatology Clinical
Trials) MRI scoring method (RAMRIS) [OPC+ 03] is used to measure the joint damage in the longi-
tudinal MR images, and the scores are compared with the results of the spatio-temporal segmentation.
Although the reproducibility of the RAMRIS scoring method is found to be generally high, the experi-
ence of a rater affects the consistency of scores [HOE+ 05]. To improve the performance and generalis-
ability of the RAMRIS scoring method, a set of standard reference images (atlas) is made available for
use as a guide to score the RA joint damage [OEM+ 05]. The rater is advised to compare the relevant
images with the images in the atlas, similar to the Larsen method of scoring plain film radiographs. The
third aim of this chapter is to use image registration to further improve the ease of use, reproducibility,
and scoring time of the RAMRIS scoring method. By rigidly aligning each bone in the MR images
to a common reference coordinate system, such as the EULAR-OMERACT RA MRI reference image
atlas [EML+ 05, CBE+ 05], the variations in the positions of the bones are reduced, and this allows the
raters to concentrate on subtle disease changes.

Aims
To localise or roughly delineate the bones in MR images of the hand and the wrist to allow the
identification of ROI for further analysis.
To apply the spatio-temporal segmentation to quantify the changes in the MR images of the hands
and wrists of RA patients.
To rigidly and automatically align each bone in MR images of the hand and the wrist to a common
reference coordinate system in order to improve the ease of use, reproducibility and scoring time
of the OMERACT RAMRIS scoring method.
Contribution
A method that combines a hierarchical approach and multiple starting estimates was developed to localise
the bones in MR images of the hand and the wrist. In addition, a bone hierarchy and multiple starting
estimates were obtained, based on simple heuristics that were developed to maximise the robustness and
minimise the running time of the registration process. Initial validation using 21 images from 6 patients
(252 bones to be localised) showed that 220 bones were successfully localised.
Preliminary results using spatio-temporal segmentation were presented. The potential of the spatio-
temporal segmentation was demonstrated by automatically measuring the changes in bone lesion volume
over time.
Like the automatic bone localisation, a method that uses a hierarchical approach was developed to
rigidly align the bones in MR images of the hand and the wrist to a standard reference image. Initial vali-
dation using 21 images from 6 patients showed that 92 out of the 105 metacarpal bones were successfully
registered to the standard reference image.

Kate McLeish designed the MR sequences and acquired the hand and wrist images of RA patients. Bruce
Kirkham and Steven Oakley performed all the clinical work and recruited the RA patients. All the image
and data analysis was my own work.
5.3 Review
Existing techniques for the quantification of disease progression in RA were reviewed in chapters 2,
3 and 4.
This is the first technique that has been proposed to improve the ease of use, reproducibility and
scoring time of the OMERACT RAMRIS scoring method by aligning each bone in MR images to a
standard reference coordinate system. This alignment is analogous to the accurate positioning of the hand
(commonly in the posterior-anterior view) in plain film radiography, which is essential for the analysis
of longitudinal radiographs and for the proper exposure of the required joints for accurate scoring using
methods like the Sharp and Larsen [vdH00]. Although bone alignments are performed in the study of
joint kinematics [MLC06, NMC00], they often involve aligning the bones in different images from the
same subject (intra-subject registration). In [MLC06], an accurate registration algorithm was developed
to register the positions and orientations of metacarpal and carpal bones across multiple CT images
from the same subject. Geometric models of the bones were extracted from a reference image of the
same subject. A distance field that specified, at each point, the signed distance from the point to the
bone boundary was then derived from the target CT image using an unsupervised tissue classifier. The
geometric models of various bones were registered to the distance field using a bone hierarchy that was
determined by using 30 images as training data. However, it is different from the work described in this
chapter, which requires inter-subject registration in order to align patients images to a standard reference
image.

5.4.1 Image data
16 patients that met the 1987 American College of Rheumatism criteria for rheumatoid arthri-
tis [AEB+ 88], were on stable treatment, and had evidence of active disease as indicated by the Disease
Activity Score (DAS) [vdHvtHvR+ 90] greater than 3.0, were recruited from the rheumatology clinics at
Guys and St Thomas Hospital. The study was designed to acquire MR images of their dominant hands
and wrists at five time points (0, 3, 6, 9 and 12 months after the start of the study). However, not all the
time points were acquired for all the patients because 3 patients dropped out of the study after the first or
second scan and some patients missed their appointments. At the end, among the 16 patients, 5 of them
completed only 1 or 2 time points, another 5 of them completed only 3 time points, a further 5 of them
completed 4 time points, and 1 of them completed all 5 time points. This chapter analysed the images
from the 6 patients who had a more complete set of time points, i.e. were scanned at 4 or more time
points.
Clinical assessment including blood tests (to check the level of inflammatory markers - erythrocyte
sedimentation rate (ESR) and C reactive protein), Health Assessment Questionnaire and rheumatologic
examination (tender and swollen joint counts by an experienced assessor) were performed at each time
point. Plain film radiographs of both hands and feet were taken at 0 and 12 months after the start of the
study and were graded using the Sharp scores.
The MR images were acquired using a Philips 1.5T Intera and three different sequences:
3D T1/T2-weighted Steady State Free Precession (bFFE) sequence - TR = 5.5ms, TE = 2.8ms,

flip angle = 60 , FOV = 140 98 mm2 , matrix = 256 179, number of slices = 55 (thickness =
0.8mm). The voxel sizes of the reconstructed image were 0.547 0.547 0.8. The scanning time
was 1 minute 16 seconds.
3D T1-weighted Gradient Echo (FFE) sequence - TR = 23ms, TE = 4.6ms, flip angle =35 ,
FOV = 140 98 mm2 , matrix = 256 179, number of slices = 55 (thickness = 0.8mm). The
voxel sizes of the reconstructed image were 0.547 0.547 0.8. The scanning time was 5 minute
20 seconds.
3D T2-weighted fat-suppressed Gradient Echo (fsFFE) sequence - TR = 22ms, TE =7.2ms, flip

angle = 45 , FOV = 140 98 mm2 , matrix = 256 179, number of slices = 55 (thickness =
0.8mm). The voxel sizes of the reconstructed image were 0.547 0.547 0.8. The scanning time
was 5 minute 06 seconds.
5.4.2 The standard reference image

The standard reference images were acquired from a volunteer using the three protocols described in
Section 5.4.1. Metacarpal and carpal bones from the reference images were manually segmented by an
expert observer.
5.4.3 Automatic bone localisation

This section describes the automatic localisations of the metacarpal and carpal bones in the patients
images. The aim of bone localisation was to roughly delineate a bone, so that a region of interest around
the bone was defined for further analysis, such as accurate bone delineation. Although only rough de-
lineation was required, it was important to define a quality metric to assess the quality of the delineation
in order to define successful bone localisation and assess the performance of the algorithm. Like the
assessment of bone erosion in the OMERACT RAMRIS scoring method, the rough bone delineation
was scored 0 10, according to the proportion of the bone outside the boundary of the bone delineation:
0:0%, 1:110%, 2:1120%, ..., 10:91100%. Only bone localisation with a score 0 or 1 was regarded as
successful. Examples of bone locailisation with scores from 0 to 5 are shown in Figure 5.1.
Based on the principle of segmentation propagation, after the bones in the patients images had
been registered to the corresponding bones in the standard reference image, they were delineated by
propagating the boundaries of the bones in the standard reference image to the patients images, using
the results of image registration. Since the aim was to localise and roughly delineate the bones,
the nine degree-of-freedom (9-dof) registration that included translation, rotation and scaling in
the transformation was used to compensate for the gross differences in the positions and sizes of
the bones between the patients image and the standard reference image;
the bones in the patients image were delineated by propagating the boundaries of the dilated bones
in the standard reference image. The kernel size of the dilation operator was 4 voxels.
The hand and wrist are composed of multiple bones, each of which is rigid, but which move with
respect to one another in a nonrigid fashion. Therefore, the registration of the whole hand and wrist was
performed first. Next, the registration of each bone was performed.
Whole hand and wrist registration: 9-dof registration was used to align the whole hands and wrists in
the patients image and the standard reference image. In addition, to maximise the capture range,
(a) Bone localisation with a score 0 (b) Bone localisation with a score 1
(c) Bone localisation with a score 2 (d) Bone localisation with a score 3
(e) Bone localisation with a score 4 (f) Bone localisation with a score 5
Figure 5.1: Example bone localisation scores from 0 to 5.

(a) The tree representation of the hierarchical (b) The acyclic graph (tree-like) representa-
approach tion of the hierarchical approach with multi-
ple starting estimates
Figure 5.2: Diagrams illustrating the hierarchical approach and multiple starting estimates.
the multi-resolution approach of three levels was used, and the cross correlation was used as the
similarity measure. The result was used as the starting estimate in the bone registration.
Bone registration: each bone in the patients image was aligned to the corresponding bone segmenta-
tion in the standard reference image using 9-dof registration. In addition, the registrations of all
the bones was performed using the combination of a hierarchical approach and multiple starting
estimates in order to increase the robustness of the registration process.
A hierarchical approach: the whole registration process was represented using a tree, as shown
in Figure 5.2(a), in which each node corresponded to the registration of a bone, except that
the root node corresponded to the whole hand and wrist registration. The result of a parent
node was used as the starting estimate for a child node.
Multiple starting estimates: it was found that the hierarchical approach was not robust enough
to correctly register all the bones in some of the patients images. Therefore, the current node
with depth d was allowed to use the results of any nodes with depth less than d as starting
estimates. In other words, a node with depth d in the tree was allowed to have more than one
parent whose depth1 was less than d, as shown in Figure 5.2(b).
5.4.3.1 The determination of a bone hierarchy and multiple starting estimates

The following heuristics were used to develop a procedure in order to determine a bone hierarchy and
multiple starting estimates for the automatic bone localisation.
The depth of the bone hierarchy should be as small as possible to minimise the dependency of the
leaf nodes. This helped to maximise the robustness of the registration process.
The number of estimates included in multiple starting estimates should be as small as possible.
This helped to minimise the running time of the registration process.
1 The definition of the depth of a node needs to be modified to be 1 plus the maximum depth of its parent nodes.
Using 10 randomly chosen patients images as training data, the following procedure was used
to determine a bone hierarchy and multiple starting estimates for the automatic bone localisation by
systematically trying different combinations of bone hierarchies and starting estimates on the 10 images.
1. To determine the bones with depth 1, the bones in the 10 images were registered to the bone seg-
mentations in the standard reference image using the result of the whole hand and wrist registration
(the root node) as the starting estimate. The depths of the bones that were successfully localised
in all the 10 images were set to 1.
2. The depth d was set to 2.
3. To determine the bones with depth d, the remaining bones in the 10 images were registered to
the bone segmentations in the standard reference image using combinations of the results of the
nodes with depth less than d as starting estimates. The depths of the bones that were successfully
registered in all the 10 images were set to d, and the corresponding starting estimates were assigned
as the parents of the bones.
4. d was incremented by 1. Step (3) was repeated to determine the depths and the parents of the
remaining bones until all the bones were assigned a depth value.
As an initial validation, the automatic bone localisation was performed on 21 additional images
using the resulting bone hierarchy and multiple starting estimates.

The spatio-temporal segmentation was applied to analyse the changes in three bones (3rd metacarpal
bone, capitate and hamate) in the temporal sequences from 6 patients. Based on initial experiments and
prior knowledge about the sizes and intensities of bone lesions from the MR images, the window widths
were chosen to be (4.0, 4.0, 4.0, 4.0, 1.0) in the x, y, z, intensity and time dimensions respectively.
5.4.5 Automatic rigid bone alignment to the standard reference image

This section describes the automatic rigid bone alignment to the standard reference image to reduce the
variations in the positions of the bones in patients images for visual scoring. Like automatic bone lo-
calisation, a quality metric was required to define successful bone alignment and assess the accuracy
of bone alignment. Since inter-subject registration was performed, the different shapes and sizes of the
bones from different subjects and erosion in patients bones prevented the use of standard techniques
for validating the registration, such as the measurement of target registration errors based on landmarks.
Therefore, for metacarpal bones, the accuracy of bone alignment was assessed by comparing the orienta-
tions of the long axes of two bones. In addition, the orientation of the long axis of a bone was determined
by manually choosing two points near the centres of both ends of cortical bone (the end of cortical bone
was chosen because it was not eroded in the early and middle stages of RA), as shown in Figure 5.3. The
accuracy of bone alignment was then assessed by the angle between the long axes of two bones, and
bone alignment was regarded as successful if is less than 5 . For carpal bones, the accuracy of bone
alignment was only assessed visually.
5.5. Results 103
(a) The point at the upper end of the cortical bone (b) The point at the lower end of the cortical bone
Figure 5.3: Points used to determine the orientation of the 2nd metacarpal bone in the standard reference
image.
Using the results of the automatic bone localisation, the rough bone delineations in the patients
images were rigidly aligned to the corresponding bone segmentations in the standard reference images. A
simpler hierarchical approach, without the use of multiple starting estimates (compared to Section 5.4.3),
was used for the rigid registration. This was because the registration problem in this section was simpler
- there was only one bone segmentation in each patients image and the standard reference image.
5.4.5.1 The determination of a bone hierarchy

Using the same training data in Section 5.4.3.1, the following procedure was used to determine a bone
hierarchy by systematically trying different bone hierarchies on the 10 images.
1. To determine the bones with depth 1, the rough bone delineations in the 10 images were rigidly
registered to the bone segmentations in the standard reference image using the result of the whole
and wrist registration (the root node) as the starting estimate. The depths of the bones that were
successfully registered in all the 10 images were set to 1.
2. The depth d was set to 2.
3. To determine the bones with depth d, the remaining rough bone delineations in the 10 images were
registered to the bone segmentations in the standard reference image using the results of the nodes
with depth d 1. The depths of the bones that were successfully registered in all the 10 images
were set to d, and the corresponding starting estimate was assigned as the parent of the bone.
4. d was incremented by 1. Step (3) was repeated to determine the depths and the parents of the
remaining bones until all the bones were assigned a depth value.
As an initial validation, automatic rigid bone alignment was performed on 21 additional images
using the resulting bone hierarchy.
5.5 Results
5.5.1 Image data
Example baseline images of a patient acquired by the three different MR sequences are shown in Fig-
ure 5.4. Bone erosion was present in the third metacarpal bone within the white rectangles.
5.5. Results 104
(a) 3D T1/T2-weighted Steady State Free Precession (bFFE) sequence
(b) 3D T1-weighted Gradient Echo (FFE) sequence
(c) 3D T2-weighted fat-suppressed Gradient Echo (fsFFE) sequence
Figure 5.4: Example baseline images of a patient acquired by the three different MR sequences. The
bone erosions in the third metacarpal bone are enclosed in white rectangles. Each sub-figure shows the
coronal, transaxial and sagittal views of the same image from left to right.
5.5. Results 105
Table 5.1: The bone hierarchy and multiple starting estimates for the automatic bone localisation. The
multiple starting estimates column lists the bones of which the results of the registrations are used as
starting estimates.
Depth Bone Multiple starting estimates

1 capitate whole hand and wrist
hamate whole hand and wrist
scaphoid whole hand and wrist
trapezium whole hand and wrist
lunate whole hand and wrist
2 triquetrum scaphoid
trapezoid trapezium
2nd metacarpal bone whole hand and wrist
capitate
3 1st metacarpal bone whole hand and wrist
2nd metacarpal bone
trapezium
3rd metacarpal bone whole hand and wrist
2nd metacarpal bone
4 4th metacarpal bone whole hand and wrist
3nd metacarpal bone
5 5th metacarpal bone whole hand and wrist
4nd metacarpal bone
5.5.2 Automatic bone localisation

The bone hierarchy and multiple starting estimates for the automatic bone localisation is shown in Ta-
ble 5.1. When applying this combination to the 21 additional images from 6 patients2 as an initial
validation, 220 out of 252 bones were successfully localised (the visual score was either 0 or 1). The
relationship between images (I1,...,I21) and the time points of the patients (ppt01,...,ppt06) is shown
in Table 5.3. In addition, the automatic bone localisation of a patients image is shown in Figure 5.5.
The mean visual score of all the bones was 1.29 with standard deviation 1.76. The visual scores of the
individual bones are summarised in Table 5.2.

The results of spatio-temporal segmentation in the 3rd metacarpal bone of the patient ppt01 are shown
in Figure 5.7. The graphs of the volume of the bone lesion detected in the 3rd metacarpal bone, capitate
and hamate are shown in Figure 5.8, Figure 5.9 and Figure 5.8 respectively. Logarithmic scale was used
because of the large numerical range of the bone lesion volume.
2 Notice that 4 images from the 6 patients were included in the training data, and not used in the validation experiment.
5.5. Results 106
Table 5.2: The visual scores of the automatic localisation of the individual bones of the 21 images
(I1,...,I21) from 6 patients. The abbreviations are 1st metacarpal bone (mc1), 2nd metacarpal bone
(mc2),3rd metacarpal bone (mc3), 4th metacarpal bone (mc4), 5th metacarpal bone (mc5), capitate (cap),
hamate (hamate), scaphoid (scp), lunate (lun), triquetrum (tri), trapezoid (tpz) and trapezium (tpm).
Bone mc1 mc2 mc3 mc4 mc5 cap ham scp lun tri tpz tpm
I1 1 0 1 1 0 0 0 1 0 0 0 0
I2 0 1 1 1 0 0 0 1 10 1 0 0
I3 1 0 0 0 0 0 0 1 0 0 0 0
I4 10 0 0 1 1 0 0 0 1 0 0 0
I5 1 1 0 1 0 0 0 0 0 0 0 0
I6 1 0 0 0 0 0 0 0 0 0 0 0
I7 0 1 1 1 5 0 0 1 10 0 0 0
I8 0 0 1 1 0 0 0 1 10 10 0 0
I9 0 0 0 0 0 0 0 0 0 0 0 0
I10 0 0 0 0 0 1 0 0 0 0 0 0
I11 10 0 0 7 10 10 10 10 10 10 10 10
I12 1 0 0 0 0 0 0 0 0 0 0 1
I13 0 0 1 0 5 0 0 0 10 0 0 1
I14 8 0 1 0 0 0 0 1 0 5 0 1
I15 0 0 1 0 3 0 0 1 0 1 1 1
I16 1 0 0 0 0 1 6 1 1 1 0 1
I17 4 0 0 0 0 1 1 10 0 8 0 1
I18 1 0 1 0 0 1 1 1 1 5 1 0
I19 0 0 0 5 1 0 1 0 0 1 0 0
I20 1 0 7 7 7 1 1 0 0 1 0 0
I21 0 0 1 7 10 0 1 0 0 1 0 1
average 1.90 0.14 0.76 1.5 2 0.71 1 1.38 2.52 2.10 0.57 0.81
5.5. Results 107
Table 5.3: The mapping between the image names (I1,...I21) and the time points of the patients
(ppt01,...,ppt06).
Image name Patient name Time point

I1 ppt06 0 months
I2 ppt01 0 months
I3 ppt04 0 months
I4 ppt03 0 months
I5 ppt02 0 months
I6 ppt02 3 months
I7 ppt01 9 months
I8 ppt01 12 months
I9 ppt02 6 months
I10 ppt02 12 months
I11 ppt03 9 months
I12 ppt03 12 months
I13 ppt04 3 months
I14 ppt04 6 months
I15 ppt04 9 months
I16 ppt05 3 months
I17 ppt05 6 months
I18 ppt05 9 months
I19 ppt06 3 months
I20 ppt06 6 months
I21 ppt06 12 months
5.5. Results 108
(a) capitate (b) hamate
(c) scaphoid (d) trapezium
(e) lunate (f) triquetrum
(g) trapezoid (h) 1st metacarpal bone
Figure 5.5: The results of automatic bone localisation. The bone localisations of the image I3 are
highlighted by the white boundaries. Each sub-figure shows the coronal, transaxial and sagittal views of
the same image from left to right.
(a) 2nd metacarpal bone (b) 3rd metacarpal bone
(c) 4th metacarpal bone (d) 5th metacarpal bone
Figure 5.6: The results of automatic bone localisation. The bone localisations of the image I3 are
highlighted by the white boundaries. Each sub-figure shows the coronal, transaxial and sagittal views of
the same image from left to right.
5.5.4 Automatic rigid bone alignment to the standard reference image

The bone hierarchy for the automatic rigid bone alignment to the standard reference image is shown
in Table 5.4. When applying this hierarchy to the 21 additional images from 6 patients3 as an initial
validation, 92 out of the 105 metacarpal bones were successfully aligned to the standard reference im-
age (the angle between the long axes of the bones was less than 5 ). In addition, the automatic rigid
bone alignments of a patients image are shown in Figure 5.11 (5th metacarpal bone) and Figure 5.12
(3rd metacarpal bone). The mean accuracy of bone alignment of all the metacarpal bones was 4.10
with standard deviation 7.75 . The accuracy of bone alignment of individual metacarpal bones are sum-
marised in Table 5.5.

This chapter applies image registration and the spatio-temporal segmentation to facilitate the analysis
of longitudinal MR hand and wrist images from human RA patients. Firstly, a method was developed
to automatically localise metacarpal and carpal bones in MR images to allow the identification of ROI
around the bones for further analysis. Initial validation showed that 220 out of 252 bones in 21 images
were successfully localised. Secondly, the spatio-temporal segmentation was applied to the longitudinal
images of RA patients. Thirdly, a method was developed to rigidly align metacarpal and carpal bones
automatically to a standard reference image to improve the ease of use, reproducibility and scoring time
of the OMERACT RAMRIS scoring method. Initial validation showed that 92 out of 105 metacarpal
3 Notice that 4 images from the 6 patients were included in the training data, and not used in the validation experiment.
(a) Baseline image
(b) Registered time point 1 (3 months) (c) No segmented lesions on difference image of time
point 1
(d) Registered time point 2 (6 months) (e) Segmented lesions overlaid on difference image of
time point 2
(f) Registered time point 3 (9 months) (g) Segmented lesions overlaid on different image of time
point 3
(h) Registered time point 4 (12 months) (i) Segmented lesions overlaid on different image of time
point 3
Figure 5.7: The results of spatio-temporal segmentation of the images from the patient ppt01. The left
column shows the registered images of the 3rd metacarpal bone of a patient acquired at baseline and 3,
6, 9 and 12 months after the baseline. The right column shows the highlighted bone lesions delineated
by spatio-temporal segmentation overlaid on top of the difference images. Each sub-figure shows the
coronal, transaxial and sagittal views of the same image from left to right.
Figure 5.8: The graph of the bone lesion volume in the 3rd metacarpal bone in logarithmic scale against
time. Since the logarithmic scale was used, the error/noise due to the data or method in the smallest
volume appears large in the graph.
Figure 5.9: The graph of the bone lesion volume in the capitate in logarithmic scale against time. It
shows that an apparent increase in the bone lesion volume in the patient ppt05 during the period 69
months, and an apparent decrease in the bone lesion volume in the patient ppt04 decreased in the period
36 months.
Figure 5.10: The graph of the bone lesion volume in hamate in logarithmic scale against time. It shows
an apparent increase in the bone lesion volume in the patient ppt05 in the periods 36 months and 69
months, and an apparent decrease in the bone lesion volume in the patient ppt04 in the period 3-6 months.
Table 5.4: The bone hierarchy and the starting estimates for the automatic rigid bone alignment to the
standard reference image. The starting estimate column lists the bones for which the results of the
registrations are used as starting estimates.
Depth Bone Starting estimate

1 capitate whole hand and wrist (rigid component of the result
of the 9-dof registration from Section 5.4.3)
hamate same as above
scaphoid same as above
trapezium same as above
lunate same as above
2nd metacarpal bone same as above
2 triquetrum hamate
trapezoid hamate
3 1st metacarpal bone trapezium
3rd metacarpal bone 2nd metacarpal bone
4 4th metacarpal bone 3rd metacarpal bone
5 5th metacarpal bone 4rd metacarpal bone
(a) The 5th metacarpal bone of the standard reference

image
(b) The image I1 - unaligned (c) The image I1 - aligned
(d) The image I3 - unaligned (e) The image I3 - aligned
(f) The image I5 - unaligned (g) The image I5 - aligned
Figure 5.11: The results of automatic bone alignment of the 5th metacarpal bone. The 5th metacarpal
bones are highlighted by the white rectangles. Each sub-figure shows the coronal, transaxial and sagittal
views of the same image from left to right.
(a) The 3rd metacarpal bone of the standard reference

image
(b) The image I1 - unaligned (c) The image I1 - aligned
(d) The image I3 - unaligned (e) The image I3 - aligned
(f) The image I5 - unaligned (g) The image I5 - aligned
Figure 5.12: The results of automatic bone alignment of the 3rd metacarpal bone. The 3rd metacarpal
bones are highlighted by the white rectangles. Each sub-figure shows the coronal, transaxial and sagittal
views of the same image from left to right.
Table 5.5: The accuracy of the automatic alignment of individual metacarpal bones using images from
6 patients (see Table5.3 for the mapping between the image name and the patient time point). The
abbreviations are 1st metacarpal bone (mc1), 2nd metacarpal bone (mc2),3rd metacarpal bone (mc3),
4th metacarpal bone (mc4) and 5th metacarpal bone (mc5).
Bone mc1 mc2 mc3 mc4 mc5

I1 0.73 1.39 0.71 0.95 1.02
I2 2.30 0.89 1.60 1.78 0.94
I3 2.32 1.87 0.71 2.32 1.25
I4 2.47 1.82 1.10 1.99 1.08
I5 1.39 1.30 1.17 1.52 2.43
I6 2.82 1.16 0.23 3.13 1.05
I7 0.30 0.46 1.47 3.56 11.71
I8 2.17 1.97 1.68 2.49 2.48
I9 4.60 1.30 2.16 1.12 1.21
I10 4.44 1.44 2.77 3.83 2.85
I11 33.20 2.64 1.86 16.03 1.38
I12 53.11 3.76 2.21 2.66 0.75
I13 0.63 1.72 2.48 2.46 7.08
I14 23.72 0.89 1.46 0.11 1.08
I15 1.79 1.15 2.67 1.33 10.89
I16 3.07 0.64 1.26 1.63 1.96
I17 25.51 1.66 3.59 1.93 3.80
I18 3.21 2.94 1.60 0.83 3.40
I19 1.43 2.24 0.45 8.95 1.20
I20 2.15 3.80 18.51 6.66 5.59
I21 3.02 3.11 0.08 2.28 37.42
average 8.30 1.82 2.37 3.22 4.79

Figure 5.13: The different sizes, positions and fields of view (FOV) of the hand and wrist of the reference
image (left) and the patient image I11 (right). The corresponding radius bones are enclosed in the white
boxes for comparison.
bones in 21 images were successfully registered to the standard reference image.

The bone hierarchy and multiple starting estimates in the automatic bone localisation and bone
alignment are just the first ones found using the heuristics described in Section 5.4.3 and Section 5.4.5.
Other bone hierarchy and multiple starting estimates that give similar or better results may exist. Al-
though a unique bone hierarchy is found in [MLC06], the problem in the paper is basically a intra-subject
rigid registration problem, which is very different from the inter-subject 9-dof registration problem de-
scribed in this chapter.
From Table 5.2, the bone hierarchy and multiple starting estimates in the automatic bone localisation
work better for the 2nd metacarpal bone, 3rd metacarpal bone, capitate, hamate, trapezium and trapezoid,
with average visual score less or equal to 1. In the future, the method should make use of these bones
as starting points to improve the bone hierarchy and multiple starting estimate, in order to achieve better
bone localisation of other bones.
Patient image I11 contributed to about one-third of failed cases in the bone localisation. The prob-
lem is the failure of the whole hand and wrist registration at the beginning of the method. This is
because the patient image and the reference image have quite different sizes, positions and fields of view
(FOV) of the hand and wrist, as seen in Figure 5.13. After reducing the FOV (see Figure 5.14) to make
it more comparable to the reference image, all the bones were successfully localised. In future studies,
the radiographers can be asked to acquire a standardised FOV of the whole hand and wrist, so that the
bone localisation errors may be reduced.
Further failed cases of the automatic bone localisation are shown in Figure 5.15. A possible reason
for the failure of the registration is that both bones, the lunate in particular, have large erosions caused by
the disease. This suggests that the method may have greater difficulties localising bones that have large
erosions at the later stage of the disease. Further work can be done to fine-tune the bone hierarchy and
multiple starting estimates to improve the registration process in these cases.
Figure 5.14: The FOV of the patient image I11 (right) is reduced to make it more comparable to the
reference image (left). The automatic bone localisation method successfully localised all the bones in
the corrected image.
(a) The lunate of image I2
(b) The 1st metacarpal bone of image I4
Figure 5.15: The two bones that cannot be automatically localised are within the white rectangles. Each
sub-figure shows the coronal, transaxial and sagittal views of the same image from left to right.
From Table 5.5, the bone alignment method performed better to align the 2nd, 3rd and 4th
metacarpal bones. The results are similar to those of the bone localisation method because the bone
alignment method depends on the results of the bone localisation method. For the carpal bones, only
simple visual assessment was used to check the results of the bone alignment, and 130 out of the 147
carpal bones were successfully aligned to the reference image.
After studying the results of the bone localisation and alignment methods and exploring the reasons
for failures, I have identified areas where the success rates of the methods may be substantially increased
by using a combination of improved radiographic technique and a quality control step prior to the start
of the analysis:
If the FOV is correct in the image I11 (see images in Figure 5.14), the success rate will be increased
to 91%. This may be accomplished by
better positioning of the subject by the radiographer;
a quick quality control step to make sure the FOV is correct before the start of the analysis.
In the quality control step, the user will compare the FOV in the newly acquired image to
a reference image (see Figure 5.13). If the FOV is not correct, the user can correct it by
choosing a FOV with a mouse in the newly acquired image that is similar to the FOV in the
reference image.
Furthermore, if the bone localisation method in the 1st metacarpal bone, 5th metacarpal bone and
triquetrum (the 3 worst success rates in Table 5.2 apart from the lunate) are successful, the success
rate will be increased to 96%. This may be accomplished by
better and more consistent positioning of the hand and wrist by the radiographer, e.g. by ask-
ing the patient to hold an object that fits in the palm of the hand. The aim is to minimise the
variabilities in the positions of the 1st metacarpal bone, 5th metacarpal bone and triquetrum
relative to the other metacarpal and carpal bones, so that the starting estimates given by the
bone hierarchy (see Table 5.1) will be good enough for all the images;
Careful training of radiographers and rapid quality control on acquired image data are standard practice
in pharmaceutical company clinical trials, so an approach like that described above is applicable. In
addition to the above improvements in process, the bone localisation and alignment methods may be
refined to increase the success rate by
improving the bone hierarchy (see Table 5.1) by using more training data (e.g. 20 images instead
of 10 images), in order to include the variability in relative bone positions in other images;
using the spatial transformation of an adjacent bone to transform the bone that cannot be easily
localised (e.g. due to severe bone erosion - see Figure 5.15), because the adjacent bone may
provide a rough alignment to the bone in the reference image.
If the success rates of the bone localisation and alignment methods can be increased to closer to 100%,
they will have great potential in helping OMERACT RAMRIS by both speeding up the scoring process
and reducing the intra- and inter-observer variabilities.
When applying spatio-temporal segmentation to the clinical data, Figure 5.7 shows that bone lesions
were automatically segmented in the 3rd metacarpal bone of the patient ppt01. Furthermore, the bone
lesion volume showed progressive increase, as seen in Figure 5.8.
The bone lesion volume detected by the spatio-temporal segmentation varied dramatically in dif-
ferent bones in different patients. For example, the changes in the volume of bone lesion volume in the
capitate in patients ppt02 and ppt03 in the 0 3 month period were 932mm3 and 621mm3 (see Fig-
ure 5.9), and there were almost no change in the bone lesion volume in later periods. However, the 0
and 3 months images of the patient ppt02 are shown in Figure 5.16. Although the images at 0 and 3
months look similar in figures 5.16(a) and 5.16(b), the subtraction image shown in Figure 5.16(c) shows
that there may be structural changes within the bone. Further work should investigate the causes of the
intensity change in details.
Another problem in applying the spatio-temporal segmentation to the clinical images is the se-
lection of bandwidth. Various bandwidth had been tried, and the results were visually assessed. The
bandwidth that produced the best visual results was (4.0, 4.0, 4.0, 4.0, 1.0) in the x, y, z, intensity and
time dimensions respectively. Further work should investigate the automatic selection of bandwidth.
Various image artifacts are observed in MR images acquired in this study. The most common
artifacts are motion, chemical shift and field inhomogeneity (shading), all of which may cause problems
in the quantitative analysis and in the OMERACT RAMRIS scoring method [MsP+ 05]. To minimise the
motion artifact, patients must not move in the scanner, and this causes great discomfort in the patients.
If the quantitative analysis of disease progression using the images acquired using the Steady State Free
Precession (bFFE) sequence is successful, the short scanning time (about 1 minute 16 seconds compared
to 5 minutes 20 seconds when using the Gradient Echo (FFE) sequence) can greatly benefit the patients.
(a) The capitate of the patient ppt01 at 0 month
(b) The capitate of the patient ppt01 at 3 month
(c) The subtraction image of the capitate of the patient ppt01 between 0 and 3 months
Figure 5.16: The capitate bone of the patient ppt02.

Chapter 6
Use of the Grid in Automated Medical Image

Analysis
6.1 Introduction
All the chapters so far have focused on the methodology of automated medical image analysis. However,
when researchers in hospitals or pharmaceutical companies try to apply these methods to thousands of
medical images in medical research, patient management or clinical trials on a daily basis, they are faced
with the following challenges:
The huge volume of image data results not only from the images acquired from imaging equipment
but also from the automated image analysis. In industrialised countries, a hospital typically pro-
duces several terabytes of medical image data per year. However, the size of the results of the
automated image analysis can be a few times more than the acquired image data. For example, in
Chapter 5, twelve registered images are generated from one image of the hand after registering it
to each carpal and metacarpal bone in the reference image. In addition, the difficulty of locating
a particular image or result of the analysis in the huge volume of image data cannot be underes-
timated. There is a need for a scalable storage system that allows easy retrieval of image data by
their attributes.
Computationally demanding algorithms are frequently employed in automated image analysis. In

chapters 3 and 4, the running times of the B-spline nonrigid registration algorithm and the mean
shift segmentation are 2 - 4 hours and 12 hours respectively. As a result, the analysis of thousands
of medical images can take months if a single desktop computer is used. There is a need for a
scalable computational system.
Multiple processing steps (or Workflows) are often involved in automated image analysis. Examples
are the multiple steps used in the analysis in chapters 3 and 4, and in the analysis of neuroim-
ages [ZFE02]. Although the multiple processing steps can be encapsulated in a monolithic com-
puter program, this approach lacks the flexibility to customise existing workflows or construct new
workflows that are best suited to the problem [RMT03]. In other cases, scripts are used to link to-
gether multiple processing steps. However, many of these scripts are written for a single purpose,
6.1. Introduction 122
and they are often modified over time, leading to an inability to generate the old solutions. There is
a need for a scalable and flexible system where complicated workflows can be easily constructed,
reused, modified, archived, and applied to thousands of medical images.
The data provenance is the description of the origins of a piece of data and the process by which it
is derived [BKT01]. For medical and legal reasons, the detailed and reliable documentation of
the data provenance of automated image analysis is very important. For example, the U.S. Food
and Drug Administration (FDA) requires pharmaceutical companies to maintain an audit trail
to protect the authenticity and integrity of electronic records as they are created, modified, or
deleted [fda]. However, the detailed and reliable documentation of the data provenance is not a
trivial task. The data provenance of the automated image analysis in Chapter 3 includes the size of
the control point spacing used in the B-spline nonrigid registration. In a study, five or six different
sizes of the control point spacing may be applied to the image data to find out the optimal value.
For thousands of medical images, it is not easy to automatically and properly maintain the data
provenance of the analysis. There is a need for a scalable data provenance tracking system.
The collaboration of different institutions is often required in clinical trials or studies, which adds an
extra layer of complexity to the above issues. For example, in multi-centre clinical trials, not only
are the image data acquired in geographically different sites, but the automated image analysis
may be performed in geographically different sites. Although the image data and the results can
be transferred electronically between different sites, the data logistics of terabytes of image data is
complex, tedious and error-prone. There is a need for a scalable system that allows researchers in
different institutions to work together more effectively.
This is where the Grid comes in. The Grid is born from the need to share human and computer
resources within scientific collaborations [SWDC97, FKNT], such as the visualisation of large dataset
and distributed computing for computationally demanding data analysis. The main idea behind the Grid
is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organisa-
tions [FKT01]. It enables the sharing of computers, software, data, storage space and other resources
between different institutions in a transparent, highly controlled and secure way. The concept of the vir-
tual organisation, which consists of institutions that obey the same resource sharing rules, facilitates the
collaboration of different institutions. Therefore, the Grid offers a unique environment to deal with the
challenges in automated image analysis.
However, this chapter does not attempt to fully address all the challenges in automated image anal-
ysis. Instead, by integrating existing image analysis techniques and Grid technologies, the web-based
image analysis workbench is developed to demonstrate the potential of the Grid to address these chal-
lenges. Using the automated image analysis described in chapters 3 and 4 as exemplars, the workbench
allows all the analysis to be performed in a single web site to demonstrate the sharing of medical im-
age data, computational resources, storage resources and image analysis softwares between different
institutions, the management of workflows, and the tracking of the data provenance. Finally, the paral-
lel implementation of the mean shift segmentation, which takes advantage of distributed computational
resources, is described.

Aims
To develop the web-based image analysis workbench by integrating existing image analysis tech-
niques and Grid technologies in order to demonstrate the potential of the Grid to address the
challenges in automated image analysis. In particular, using the image analyse described in chap-
ters 3 and 4 as exemplars, the workbench demonstrates the sharing of
medical image data
computational resources
storage resources
image analysis software
between different institutions, the management of workflows, and the tracking of the data prove-
nance.
To implement a parallel version of the mean shift segmentation.
Contribution
By integrating existing Grid technologies and image analysis techniques, I developed a web-based image
analysis workbench to demonstrate the sharing of medical image data, computational resources, storage
resources and image analysis software between different institutions, the management of workflows, and
the tracking of the data provenance. In addition, I also implemented the parallel version of the mean
shift segmentation, so that its running time is reduced by a factor of four if four computers are available
to perform the segmentation.

The following technologies were developed by other institutions: the Globus Toolkit, the Condor, the
Condor MW, the Virtual Data System, XML and the Java Servlet Technology. All other computer
programs were my own work.
6.3 Review
The purpose of this section is to review the use of Grid and related computer technologies in facili-
tating automated medical image analysis, but not to review general-purpose Grid and related computer
technologies. Basic Grid and computer technologies used in this chapter will be described in Section 6.4.
Many medical image storage systems have been proposed to handle the huge volume of image
data acquired from imaging equipment. The most well-known is the Picture Archiving and Commu-
nication System (PACS), which allows the transfer of medical images from imaging equipment, and
the archiving, retrieval, communication, display and management of medical images and their related
6.3. Review 124
data [OVdBVdM96]. Although it is a scalable storage system that allows the retrieval of image data
across different institutions, it is not designed for the tracking of data provenance and the automated
image analysis that involves workflows and computationally demanding algorithms. Based on Grid
softwares such as the Globus Toolkit [glo] and the EGEE-gLite software [gli06], similar scalable med-
ical image storage systems have been developed. Examples are the MammoGrid virtual organisation
(MGVO) [EMR05], the diagnostic mammography national database (eDiaMoND) project, and the na-
tional digital mammography archive (NDMA) project [ndm06], all of which allow the sharing and coor-
dinated access of mammography data and computational resources across different institutions. Again,
they are not designed for the tracking of the data provenance of the image analysis. Nonetheless, by
using the data grid to share PACSs among different institutions, the fault tolerance levels of the PACSs
across the different institutions are increased [HZL+ 05].
The following systems focus on the issue of workflows. The LONI pipeline processing environment
tackles the issue by providing the ability to link together individual image analysis programs or modules
into a processing tree (or workflow), and storing the workflows in a repository [RMT03]. A graphi-
cal user interface (GUI) is provided for the creation and modification of workflows. The weaknesses
of the environment are that (a) it can only allow workflows to be executed on remote servers using a
client/server model; and (b) it does not store the data provenance of the analysis. Interactive steps, such
as checking intermediate results or performing semi-automatic segmentation, can be added to workflows
in the event-driven workflow management system [LMPT+ 04]. Again, the system does not store the
data provenance of the analysis.
The Baxgrid software package demonstrates the computational power of the Grid by providing
real-time functional MRI analysis [BMT+ 05]. The testbed consists of five computer clusters at four
geographically separated sites. Each cluster has 2 to 32 Pentium III computers running at about 1GHz.
The functional MRI analysis includes image preprocessing (image registration and spatial smoothing),
incremental general linear modelling (GLM), and statistical analysis. By distributing the analysis across
multiple computers, the system takes 2.65s to process a function MR image of 64 64 30 voxels
(240kbytes data size) for remote display. This allows a temporal resolution of 3s with the given spatial
resolution in order to achieve fully real-time analysis. The total processing time using a single desktop
computer is approximately 9s. However, the system does not facilitate the processing of the huge volume
of image data, the management of workflows and the tracking of the data provenance.
The need to have a system that can address all the challenges in automated image analysis drives
researchers from different disciplines to work together. The results are the development of several auto-
mated image analysis systems.
The Biomedical Informatics Research Network (BIRN) project aims to build a database and net-
work infrastructure to allow researchers to post, query, and analyse raw and processed data [SG03].
The storage of the image data is provided by the Storage Resource Broker (SRB) [RW02], which
can handle petabytes of data [RWM+ 02]. The Meta-Catalog (MCAT) stores the meta-data of the
image data such as patient identifiers, study names or any other information to allow easy retrieval.
6.4. Grid Technologies 125
Workflow management is provided by other systems such as the LONI pipeline processing envi-
ronment, the Kepler system [LAB+ 05] and the DataCutter [HKL+ 03]. The workflows are stored
in SRB as the data provenance of the analysis.
One of the aims of the European DataGrid (EDG) Information Society Technology (IST) project
is to identify and address the requirements of performing medical image processing on the
Grid [MBBC+ 04]. The Distributed Medical Data Manager (DM2 ) is used to manage the im-
age data on distributed Digital Image and Communication in Medicine (DICOM) servers. Since
data security is identified as an important requirement because of the distributed architecture of
the Grid, the key management system is designed to provide access control to decryption keys of
underlying encrypted storage systems. Workflows can be described as Directed Acyclic Graphs
(DAG) and executed using the DAG job submission service. For data provenance, the preaccess
program execution interface, which is part of the access control system, is used to provide infor-
mation about how a piece of data is derived.
The Information eXtraction from Images (IXI) project aims to demonstrate how Grid technolo-
gies can be used to scale automated image analysis to larger numbers of images and more compu-
tationally intensive algorithms [HHL+ 03, RBH+ 04]. In order to explore different approaches to
integrate Grid technologies and automated image analysis, work in different part of the project is
deliberately overlapped. Such overlapping happens in many Grid projects in the US and Europe.
As described in [RBH+ 04], image data are stored on a DICOM-based image database system,
which allows them to be searched using meta-data such as age, gender or MR scanning protocol.
Workflows are constructed using the Medical Imaging Component Language (MICL), which en-
sures strict typing of all inputs and outputs [BRR+ 04], and are stored in the workflow database
for later retrieval. The workflow engine submits the workflows to remote computers for parallel
execution. On the completion of a workflow, the results and the data provenance are inserted to
the DICOM-based image database system and the workflow database respectively.
6.4 Grid Technologies

The section reviews the basic Grid technologies used in this chapter.
6.4.1 The Grid

To support coordinated resource sharing and problem solving in dynamic, multi-institutional virtual
organisations, the Grid needs to provide mechanisms for the sharing of multiple resources and con-
nectivities, and the discovery of available resources provided by the virtual organisations [FKT01]. In
addition, the Globus Toolkit has been widely used to implement the functionalities of the Grid [glo]. It
has the following components:
The security component, which is provided by the Grid Security Infrastructure (GSI), allows au-
thentication (the process of verifying ones identity), communication protection (the processing of
encrypting transmissions) and authorisation (the process of finding out if one is allowed to have
the resource) between different components of the Grid.
The monitoring and discovery component, which is provided by the Grid Monitoring and Indexing
Services, allows the discovery of the existence and properties of resources, and the monitoring of
the status of resources (e.g. for failure and overload).
The computation component, which is provided by the Grid Resource Access and Management
(GRAM) protocol, allows the execution of programs, and the monitoring and controlling of the
running programs. Apart from the default job scheduler, which executes the program on the server
machine, other job schedulers are available, e.g. Condor [conb] and the Sun Grid Engine [sge].
The data management component consists of two sub-components - the data transfer compo-
nent, which is provided by the Grid File Transfer Protocol (GridFTP), and the data management
component, which is provided by the Replica Location Service (RLS). GridFTP allows high-
performance, secure, encrypted and reliable data transfer on high-bandwidth wide-area networks.
RLS allows the mappings of logical names (which do not include specific pathnames nor storage
system information) onto physical names (which do include storage system addresses and specific
pathnames).
6.4.2 Condor and Condor MW

Condor is a specialised workload management system for computationally-intensive jobs [conb]. It
provides a computing environment that delivers large amounts of computational power over a long period
of time. Condor can be used to manage a cluster of desktop workstations or dedicated compute nodes.
When the desktop workstations are idle (e.g. no keyboard or mouse activities), Condor can use the
wasted CPU power to perform useful computations. When the machine is no longer available, Condor
can produce a checkpoint and migrate a job to a different machine which would otherwise be idle. The
checkpointing mechanism is also the key to long-running computations. If a machine crashes or has to
be rebooted for an administrative task, a checkpoint preserves computation already completed.
Condor MW (master-worker) provides a C++ framework that allows the implementation of paral-
lel algorithms in the distributed, opportunistic environment of Condor [GLY00]. Based on the master-
worker paradigm, the master process manages a set of user-defined tasks and a pool of workers (available
computers). The framework handles workers joining and leaving the computation, assigns tasks to ap-
propriate workers, and rematches running tasks when workers are lost. On the other hand, the worker
processes perform user-specified tasks that are given by the master process. Although the framework
does not automatically produce checkpoints, the class structure encourages the definition of a user-
defined checkpoint function. Finally, the framework provides a range of message passing mechanisms
including Parallel Virtual Machine (PVM) [pvm], TCP sockets and Condor remote I/O in order of de-
creasing communication performance.
6.4.3 The Virtual Data Grid (VDG)

Apart from the challenges stated at the beginning of this chapter, consider the following problems faced
by a scientific researcher:
I want to segment the white matter from this brain image. If a program that performs this seg-
mentation exists, I do not have to write one from scratch.
I have found an error in my segmentation and want to know what results have to be recomputed.
I want to nonrigidly register thousands of brain images to a standard atlas. If the registrations
have already been done by a nonrigid registration program and the results are stored, I can save
weeks of computation..
To further explore the potential of the Grid, the concept of the Virtual Data Grid (VDG) is pro-
posed to address these problems by integrating data, computational procedures and the data provenance
involved in the analysis [FVWZ03]. The result is the development of the Virtual Data System (VDS),
which allows the documentation of the data provenance, the discovery of available methods and on-
demand data generation (so-called virtual data) [FVWZ02]. It has been applied successfully to two
physics data analysis computations at the European Organization for Nuclear Research (CERN) and in
the Sloan Digital Sky Survey (SDSS). In addition, none of the systems reviewed in Section 6.3 explicitly
address these issues.
VDS consists of the virtual data catalogue and the virtual data language interpreter.
The virtual data catalogue is based on the relational virtual data schema, which provides a repre-
sentation of computational procedures and their invocations. All the data objects (input and output
files) are referenced by logical file names (LFN), which are mapped to physical file names using
the Replica Location Service (RLS). Computation procedures and their invocations are described
or wrapped using the virtual data language (VDL). An executable program is described by the
transformation (TR) statement (like a function in C programming language), and an invocation of
a transformation is described by the derivation (DV) statement (like a function call in C program-
ming language). Workflows are defined in two ways:
Explicit workflows are defined by constructing compound transformations that include mul-
tiple transformations.
Implicit workflows are defined by data dependency chains among derivations. For example,
given that the output of a derivation DV1 is the input of another derivation DV2, when the
output of DV2 is requested, a workflow that contains DV1 and DV2 is constructed. This
mechanism allows on-demand invocation of derivations and, hence, on-demand data gener-
ation.
One disadvantage in the workflows is the lack of type checking in the parameters of the transfor-
mations and derivations.
The virtual data language interpreter handles requests for constructing and querying the database
entries. Queries can be issued to search for transformations and derivations by name, version
number, input and/or output filenames. When a request for a data product (e.g. an output file) is
received by VDS, it generates the workflow in a form of an abstract directed acyclic graph (DAG),
which describes how to construct the data product. The abstract DAG refers only to LFNs and is
therefore not bound to any Grid site.
The Planning for Execution in Grids (Pegasus) system [DBG+ 04] is used to execute the abstract
DAG on the Grid. Firstly, it removes any unnecessary derivations by checking if the results have already
existed in RLS. Only derivations that have not been executed are retained in the DAG. Secondly, it
resolves the LFNs to physical file names. Finally, it submits the DAG to Condor or other schedulers.
As a result, the problems at the beginning of this section are addressed in VDS by storing all the
transformations and derivations in a searchable virtual data catalogue to allow the discovery of available
methods and data provenance, and on-demand data generation. In addition, Pegasus simplifies work-
flows, so that only computations which have not been executed are submitted to remote schedulers.
The Chiron project provides a generic web portal to VDS [ZWF+ 06], which is very similar to the
work described in this chapter. Both systems are started without knowledge of each other, and developed
independently. However, the Chiron project is not an open-source project. Nonetheless, it allows the
annotations of transformations and derivations, and has the ability to search through the annotations.

6.5.1 The web-based image analysis workbench
I developed the web-based image analysis workbench using the Java Servlet Technology [jst], the Apache
Tomcat Engine [jak] and Grid softwares including the Globus Toolkit, Condor and the Virtual Data
System (VDS). The architecture of the workbench is shown in Figure 6.1.
The DataGridService consisted of the Replica Location Service and multiple storage elements, on
which GridFTP server was installed. It was responsible for the transfer and storage of image data
and results between different components. Multiple storage elements allow image data to be stored
at geographically different sites.
The ComputeGridService consisted of multiple execution elements, on which Condor, Globus GRAM
server and GridFTP server were installed. It was responsible for executing tasks that were sent
from the WorkBenchServer. Multiple execution elements allow image analysis to be performed at
geographically different sites.
The WorkBenchClient consisted of the web browser that allows users to access the web pages on the
WorkBenchServer.
The WorkBenchServer consisted of VDS, Condor, the Globus GRAM client, the GridFTP client and
the Tomcat Servlet Engine. Four web pages were provided:
Figure 6.1: The architecture of the web-based image analysis workbench.
The VDS Catalogue web page allowed the user to query VDS by the names of transfor-
mations and derivations. The web page directed all the queries to VDS and displayed the
eXtensible Markup Language (XML) [xml] search results on the web page using eXtensible
Style Language Transformation (XSLT) [xsl].
When transformations were displayed as a result of the query, the user could select
his/her desired transformation. A web form was then generated from the XML descrip-
tion of the transformation, presenting the user with the parameters of the transformation.
After the workbench directed the form request to VDS, a concrete DAG was generated
and submitted to the ComputeGridService.
When derivations were displayed as a result of a query, data objects used in the deriva-
tion were hyperlinked to the original derivations that were used to generate the data
objects, or hyperlinked to the derivations that consumed the data objects as inputs. This
allowed the user to browse the data provenance of the analysis.
The RLS Upload web page allowed the user to upload image data to the DataGridService. It
transferred the file to remote storage elements using GridFTP and registered the file to RLS.
The RLS Query web page allowed the user to query and download image data and results
from the DataGridService.
The Job Status web page allowed the user to query the status of Condor jobs.
6.5.2 VDL wrapping of the image analysis algorithms

Image analysis algorithms must be wrapped in VDL before they can be stored in VDS. There are
two types of transformation (TR) statements: simple and compound. Single executable programs are
wrapped by using simple TR statements, which allow the user to specify the namespaces, names, ver-
sion numbers and arguments of the executable programs. For example, the executable program vtk-
param, which creates parameter files for the affine registration program, is wrapped by the following
TR statement:
TR kkl::vtkparam:1.0(output param, none res, none bins, none iterations,
none steps, none length, none similarity, none lambda)
{
argument = " -r "${none:res};
argument = " -b "${none:bins};
argument = " -i "${none:iterations};
argument = " -s "${none:steps};
argument = " -e "${none:length};
argument = " -m "${none:similarity};
argument = " -l "${none:lambda};
argument = " "${output:param};
}
The first line assigns kkl (my initials) as the namespace for differentiating transformations provided
by different institutions or users, vtkparam as the name for use by derivations, and 1.0 as the version
number for keeping proper provenance information. It creates an output file called param, and takes
text arguments res, bins, iterations, steps, length, similarity and lambda. To map the
transformation to the physical executable program, VDS allows the user to specify the mappings to the
physical executable programs at different sites in a text file as below:
ixi kkl__vtkparam_1.0 bond.cs.ucl.ac.uk /opt/vtkCISG/bin/vtkparam.condor
ral kkl__vtkparam_1.0 grid-data.rl.ac.uk/jobmanager-pbs /home/ngs0213/vtkparam.condor
wrg kkl__vtkparam_1.0 grid-compute.leeds.ac.uk/jobmanager-pbs /home/data01_a/ngs0213/vtkparam.condor
man kkl__vtkparam_1.0 grid-data.man.ac.uk/jobmanager-pbs /home/ngs0213/vtkparam.condor
As another example, the rigid registration program is wrapped in the simple TR statement, as shown
below. The TR statement takes target as the input target image, source as the input source image,
param as the input parameter file, and dof as the output result of the registration.
TR kkl::areg:1.0(output dof, input target, input source, input param)
{
argument = " "${input:target}" "${input:source};
argument = " -parameter "${input:param};
argument = " -dofout "${output:dof};
argument = " -rigid";
argument = " -v 2";
}
The wrapping of workflows is done using compound TR statements, which are designed to describe
the orchestrated execution of multiple programs and the passing of files among them. The following
example shows the wrapping of the automatic bone delineation algorithm used in Chapter 3, which
consists of the affine registration, nonrigid registration and the segmentation propagation.
TR kkl::vtkFindBones:1.0(output talus, input talus_seg, input reference_image,
input dofin, input target, io talus_param, io aregdof, output tal_dof,
input bone_areg_param, input bone_nreg_param, none ds, none dilate)
{
call kkl::areg:1.0,(dof=${output:aregdof}, target=${input:target}, source=${input:reference_image},
param=${input:bone_areg_param}, dofin=${input:dofin});
Figure 6.2: Example compound workflow. The example shows the workflow for the automatic bone
delineation algorithm used in Chapter 3, which consists of the affine registration, nonrigid registration
and the segmentation propagation.
call kkl::vtksegpropparam:1.0,(param=${output:talus_param}, param_in=${input:bone_nreg_param},

ds=${ds}, dilate=${dilate});
call kkl::vtkSegProp:1.0,(param=${input:talus_param}, dofin=${input:aregdof},

seg_source=${input:talus_seg}, target=${input:target},
source=${input:reference_image}, seg=${output:talus}, dofout=${tal_dof});
}
Like the simple TR statement, the first line gives the namespace, name and version number of the trans-
formation. The workflow consists of the following three programs:
areg aligns the reference image and the target image using affine registration.
vtksegpropparam creates a parameter file for the segmentation propagation program vtkSeg-
Prop.
vtkSegProp nonrigidly registers the reference image and the target image, and propagates the
boundary of the talus bone in the reference image to delineate the talus bone in the target image.
The workflow is graphically shown in Figure 6.2. Firstly, the compound transformation calls areg to
align the reference image and the target image. Secondly, the transformation calls vtksegpropparam to
generate the parameter file for the segmentation propagation program. Finally, the transformation calls
the segmentation propagation taking the output files of areg and vtksegpropparam.
Various parameters in the transformation are described next. The output files are talus and
tal dof, which are the segmentation of the talus bone and the result of the nonrigid registration re-
6.6. Results 132
spectively. For the input files: (a) reference image and talus seg are the reference image and the
segmentation of the talus bone respectively. They constitute the atlas in the segmentation propagation; (b)
dofin is the initial estimate for the registration; (c) target is the target image which the talus bone is
delineated; (d) bone areg param is the parameter file for the affine registration; (e) bone nreg param
is the parameter file for the nonrigid registration. ds and dilate are text parameters that specify the
size of control point spacing in the nonrigid registration and the amount of dilation used in the segmenta-
tion propagation program. The keyword io specifies files that are passed between the programs within
the workflow. aregdof is the results of the affine registration passed from areg to vtkSegProp.
talus param is the parameter file passed from the program vtksegpropparam to vtkSegProp.
6.5.3 The parallel implementation of the spatio-temporal segmentation algo-

rithm
The section describes the parallel implementation of the spatio-temporal segmentation algorithm in Sec-
tion 4.4.1 (page 79). Based on the observation that the mean shift filtering of each point in the feature
space is independent of each other, the mean shift filtering can be applied to all the points in the feature
space in parallel. Using Condor MW, the master process divided the points in the feature space into
batches and distributed them to the available workers, so that the mean shift filtering of the batches of
points were executed in parallel. Checkpointing was implemented by storing the results of completed
mean shift filtering in a text file.
6.6 Results
6.6.1 The web-based image analysis workbench
The web-based image analysis workbench was successfully developed. Initially, the user was required
to upload his/her data to the storage elements, so that the data could be accessed by VDS. Image data
were uploaded to the storage elements using the RLS Upload web page. An example of the uploading
of the file rarfe3 1 iA1 3 1.hdr is shown in Figure 6.3. The parameters are explained below:
RLS server is the name of the site where the Replica Location Service in the DataGridService is
installed.
Pool is the name of the site where the storage element is located. Possible site names are ixi,
ral, wrg and man.
ixi refers to the local storage element located at UCL.
ral refers to the CCLRC-RAL Data Cluster, which is part of the National Grid Ser-
vices [ngs].
wrg refers to the White Rose Grid, which is part of the National Grid Services.
man refers to the Manchester Grid Data Node, which is part of the National Grid Services.
LFN is the unique logical file name given to the file by the user. This name is used by RLS and
VDS.
6.6. Results 133
Figure 6.3: The RLS Upload web page. It shows the uploading of the local file
/a/violet2/cs/research/medic/gskjoint1/paramedic/gskjoint-1/gsk/rarfe3 1/rarfe3 1 iA1 3 1.hdr to the
storage element ral using GridFTP.
PFN is the physical file name given to the file by the user. It specifies where the file is physically
stored.
File to upload is the name of the local file to be uploaded.
Next, algorithms such as rigid and nonrigid image registration algorithms and the spatio-temporal
segmentation were wrapped using VDL. Some of the available transformations are shown in Figure 6.4.
For example, kkl:vtkparam is a transformation that generates parameter files for the affine image reg-
istration program, and kkl:areg is a transformation that performs the rigid image registration. The user
invoked one of the transformations, say, kkl:vtkareg, by clicking on it. The web form was presented
to allow the user to fill in the required parameters, as shown in Fig 6.5. The parameters are explained
below:
Derivation Name is the name of the derivation required by VDS.
Source and Destination Pool are the site names of storage elements for the input files and output
files respectively. Possible site names are the same as those described in the RLS Upload web
page.
Run Pool is the site name of the execution element. Possible site names are the same as those
described in the RLS Upload web page.
6.6. Results 134
Figure 6.4: The VDS Catalogue web page - transformations. It shows some of the available transforma-
tions in the virtual data catalogue.
dof(output) is the name of the output of the rigid image registration.
param(input) is the name of the input parameter file.
source(input) is the name of the input source image.
target(input) is the name of the input target image.
Add to VDS only controls whether the derivation will be submitted to the ComputeGridService for
execution not. When it is checked, the derivation will not be submitted to the ComputeGridService
for execution. The virtual data that corresponds to the derivation will be created only when it
is required by another derivation (on-demand data generation). When Add to VDS only is not
checked, the derivation will be submitted to the ComputeGridService for execution.
Some complicated derivations that performed the affine and nonrigid registration, and segmentation
propagation in Chapter 3 are shown in Figure 6.6. To explore the data provenance of the results, the user
can select either a hyperlinked input file or a hyperlinked output file. For example, in Figure 6.6,
The input file bone areg param.txt inside the white box can be selected to find out how it
was generated. Figure 6.7 shows the derivation and the parameters that are used to generate
bone areg param.txt.
The output file vtkFindBone08 rarfe3 2 iw1 tal.dof inside the black box can be selected to find
out what derivations have used it. Figure 6.8 shows all the derivations that have used the file
6.6. Results 135
Figure 6.5: The VDS Catalogue web page - the parameters of a transformation (rigid image registration).
It shows the web form that needs to be filled in by the user.
vtkFindBone08 rarfe3 2 iw1 tal.dof as inputs. This information can be used to determine what
results have to be recomputed if there is an error in the file vtkFindBone08 rarfe3 2 iw1 tal.dof.
After submitting the job, its status can be queried using the Job Status web page. An example is
shown in Figure 6.9.
Finally, results and image data in the storage elements could be searched and downloaded using the
RLS query web page. An example query using the logical file name ms rarfe3 11 fixed* is shown
in Figure 6.10. Files were downloaded to the local computer by selecting the hyperlinked file names.
Unwanted data could be deleted by selecting remove on the web page.
In total, using the workbench, more than 2000 derivations and more than 8500 images were created
and stored in VDS. Separate tasks, such as the automatic delineations of the talus bones at different time
points, were distributed to different sites on the Grid. In addition, the parallel version of the mean shift
algorithm provided speed improvement by dividing the tasks into smaller batches and offloading them
to other worker computers on the same site.
6.6.2 The parallel implementation of the spatio-temporal segmentation algo-

rithm
The parallel version of the spatio-temporal segmentation algorithm was successfully implemented using
Condor MW. The running time of the algorithm depended on the number of workers available on the
Condor cluster, which was shared by the jobs submitted by other users. The following statistics were
6.6. Results 136
Figure 6.6: The VDS Catalogue web page - derivations. It shows some of the derivations describing the
affine and nonrigid registration, and segmentation propagation used in Chapter 3.
Figure 6.7: The VDS Catalogue web page - data provenance. the derivation and the parameters that are
used to generate bone areg param.txt
6.6. Results 137
Figure 6.8: The VDS Catalogue web page - data provenance. It shows derivations that use the file
vtkFindBone08 rarfe3 2 iw1 tal.dof
Figure 6.9: The Job Status web page. It shows that the job vtkFixedMeanShift is running.
6.6. Results 138
Figure 6.10: The RLS Query web page. It shows the results of querying the logical file name
ms rarfe3 11 fixed*.
outputted from the Condor MW:

...
14:28:12 **** Statistics ****
14:28:12 Dumping raw stats:
14:28:12 Vid CpuUsage
14:28:12 0 3165.5400
14:28:12 1 4192.0100
14:28:12 2 4056.4800
14:28:12 3 2693.1600
14:28:12 4 2080.5500
14:28:12 5 1401.5300
14:28:12 6 1356.4100
14:28:12 7 1351.4800
14:28:12 8 530.8000
14:28:12 9 685.4500
14:28:12 10 653.8600
14:28:12 11 283.5700
14:28:12
14:28:12 Number of (different) workers: 12
14:28:12 Wall clock time for this job: 9762.4808
14:28:12 Total wall clock time of workers: 22468.5820
14:28:12 Total up time of workers: 35705.3874
...
Vid is an identifier given to each worker, CpuUsage is the actual CPU time spent on the calculation
by a worker, and the total up time of workers is the total time (including any overhead) spent by all
the workers. Depending on the availabilities of the workers, some workers spent more time processing
6.7. The evaluation of the workbench 139
the job than others. The speed-up factor was calculated by the total wall clock time of workers divided
by the wall clock time for the job, and was equal to 2.30(= 22468.58/9762.48). The overhead of this
job was given by the difference between total up time of workers and the total wall clock time of
workers, and was equal to 3.68 hours (= (35705 22468)/3600). Notice that there were other users
jobs running on the cluster. Therefore, the computers were not available to my job all the times.
6.7 The evaluation of the workbench

Based on the results and experiences of using the web-based image analysis workbench, this section
evaluates the potential of the Grid in addressing the problems in medical image analysis mentioned in
the introduction of this chapter.
The huge volume of image data: The RLS upload (Figure 6.3) and query (Figure 6.10) web
pages allow images to be stored and retrieved at the four storage elements (ixi, ral, wrg
and man). The original MR images (i.e. images of the 12 rats scanned at 6 time points) of size
about 2 gigabytes (GB) were stored at the ixi storage element. When an image analysis task (e.g.
the automatic bone delineation) was submitted to a remote site (ral, wrg and man), images
were automatically transferred from the ixi storage element to the remote site. The intermediate
and final results were normally stored at the remote site after the analysis. In total, about 25GB of
data (more than 10 times of the size of the original data) were generated from the analysis, and they
were distributed across the storage elements. Furthermore, the RLS query web page allows images
to be seamlessly retrieved using the logical filenames. Although the size of data is not huge in term
of the capacity of modern storage device1 , the ability to store and retrieve data seamlessly across
multiple remote sites shows that the Grid has the potential to address the storage requirements of
large studies, such as the still on-going Osteoarthritis Initiative project (http://www.oai.ucsf.edu/),
in which about three terabytes of image data have already been acquired from more than 2600
subjects.
However, the problem of searching the RLS using the attributes of the images was not addressed.
Although the RLS allows extra attributes to be attached to the images, this was not explored.
Further work should investigate what information should be stored as attributes (e.g. scanning
parameters and/or data provenance), how the search can be carried out (e.g. simple keyword
search), how effective the search is, and how to perform combined search of the attributes and data
provenance.
The network bandwidth can also be a problem when transferring images between local and remote
sites. For example, the user may want to collect many images from the remote sites to the local site
for simple local processing, such as checking the results in an image viewer. Furthermore, image
analysis tasks with short run-time (from seconds to a few minutes) are not suitable for running on
the Grid since the time to transfer the data to the remote site may be longer than the run-time of
the task.
1 500GB hard disks are common at the time of writing this thesis.
6.7. The evaluation of the workbench 140
Computationally demanding algorithm: The VDS Catalogue web page (Figure 6.5) allows a work-
flow to be submitted to one of the four compute elements (ixi, ral, wrg and man). Each
compute element is or was connected to a computer cluster:
ixi was connected to a cluster of 40 single-CPU computers in Kings College London

(KCL) before moving to University College London (UCL). It is not connected to a computer
cluster at UCL;
ral is connected to a cluster of 20 dual-CPU computers (CCLRC-RAL Data Cluster in

Rutherford Appleton Laboratory);
wrg is connected to a cluster of 64 dual-CPU computers (White Rose Grid in the University
of Leeds);
man is connected to a cluster of 20 dual-CPU computers (Manchester Grid Data Node in

the University of Manchester).
The analysis tasks in chapters 3 and 4 were submitted to the four compute elements to speed up
the analysis. When the clusters were empty, the overheads in the data transfer and job scheduling
were small (less than 5 minutes). A test run of 20 automatic delineation tasks on the empty ixi
cluster gave a speed-up factor of 14.5. Since existing user agreements at the ral, wrg and
man do not allow me to exclusively use all the CPUs in the clusters all the time, the average
total number of concurrent jobs running on the compute elements were about 10, which should
gave a speed-up factor between 5 to 10. This shows that the Grid has a scalable architecture for
the computationally demanding algorithms in medical image analysis.
The fact that I did not get a higher speed-up factor was not due to any limitation in the underlying
architecture. The reason was that I had to equally share the CPUs with other users of the National
Grid Services because of the user agreements. Since the Grid provides a coordinated sharing of
resources, a user may negotiate or request a better user agreement to have a higher priority or
share of the CPUs, in order to get a higher speed-up factor.
Multiple processing steps: The medical image analysis algorithms described in chapters 3 and 4
were wrapped in VDL and displayed as available transformations, as shown in Figure 6.4. Work-
flows were found to be easily created and modified in VDL. The medical image analysis algorithms
were wrapped as compound transformations in VDL using simple transformations such as rigid
registration, nonrigid registration and spatio-temporal segmentation. Further work should imple-
ment a graphical user interface for the visual editing of the VDL transformation for the novice
users, e.g. clinical researchers.
In the current implementation, the user has to specific the name of the compute element in or-
der to submit a transformation for execution. This is inconvenient and does not allow automatic
load balancing among the compute elements. Further work should use the monitoring and discov-
ery component in the Globus Toolkit (Section 6.4) to discover the status of the computers in the
compute elements, so that the system can evenly distribute the tasks among the compute elements.
The data provenance: All the analysis described in chapters 3 and 4 were performed in VDS,
and the executed workflows were automatically stored as derivations, as shown in figures 6.6 and
6.7. More than 2000 derivations were stored in VDS. The data provenance allows me to find out
exactly what parameters I have used in each workflow. This have saved me time from
running workflows that have already been executed;
going through scripts or lab books to find out what parameters I have used in a certain work-
flow.
The data provenance also allows the automatic creation of an electronic audit trail of the image
analysis. This is particularly important for the pharmaceutical companies when submitting clin-
ical trial results to FDA. However, the VDS and Globus Toolkit are designed for the sharing of
computer resources. Data and their attributes in the system can be accessed and modified by many
users. Before the system can be used in a regulatory environment, each component in VDS and
the Globus Toolkit should be examined carefully to make sure that full audit trails, which include
the automatic analysis audit trails and manual user modification audit trails, are maintained.
The collaboration of different institutions: The web interface allows researchers in different insti-
tutions to upload image data, perform pre-defined image analysis and download the results. Since
the only main user is me, the workbench should be tested by users in different institutions in future.

A web-based image analysis workbench was developed by integrating existing medical image analysis
algorithms and Grid technologies, including the Globus Toolkit, Condor and the Virtual Data System.
The workbench was used to perform automatic delineation of multiple bones and the spatio-temporal
segmentation of bone lesions in an experimental model of RA. The web pages provided by the work-
bench allow users from different institutions to share
medical image data by using the RLS Upload and Query web pages.
computational resources by submitting to various Run Pools on the VDS Catalogue web page.
storage resources by using the the RLS Upload and Query web pages, and the Source Pools and
the Destination Pools on the VDS Catalogue web page.
image analysis software by using various transformations on the VDS catalogue web page.
At the moment, resources from four different sites, namely ixi, ral, wrg and man, are available
to the user, and new resources can easily be added to the system by entering simple details such as the
name of the site and port numbers. Workflows can be explicitly or implicitly constructed and maintained
using VDS. The data provenance of all the analyses is stored automatically, and can be retrieved and
browsed easily using the VDS Catalogue web pages. In addition, the concept of virtual data and on-
demand data generation, which no other system has incorporated, avoids the recomputation of image
Figure 6.11: The load of the computer cluster at Rochester Institute of Technology over a year
(http://cluster.rit.edu/ganglia/).
analysis programs when they have already been performed and the results are stored. For example, the
VDS Catalogue web pages allow the searching of nonrigid registrations in VDS, so that the user or
other users can see if certain nonrigid registration has already existed. Finally, the parallel version of the
spatio-temporal segmentation is implemented using Condor MW. The speed-up depends on the number
of available computers on the cluster.
The benefit provided by the computational resources of the Grid is two-fold, as demonstrated by
the computations described in this chapter. The first speed improvement is gained by distributing sep-
arate tasks to different sites on the Grid. The second speed improvement is gained by parallelising the
algorithm and executing it on several computers within a site on the Grid.
A web-based image analysis workbench should scale well with larger studies, as partly discussed
in Section 6.7. The Grid provides a scalable infrastructure for data storage and computational resources.
The current database system used in VDS is PostgreSQL (http://www.postgresql.org/), which can handle
thousand and millions of transformation and derivation records. The web interface should be re-designed
to provide a better search capability.
From the results of this study, other computationally and image intensive studies should consider
using the Grid to perform the image analysis and store the analysis results. The computer resources in
the Grid can help to speed up the analysis in the long term (months and years). However, the amount of
speed-up depends on the user agreement and the load of the Grid. It is interesting to look at the load of
some computer clusters over a year, as shown in Figures 6.11 and 6.12. There are periods of low CPU
load level (enclosed in black boxes). Personal experience and informal discussion with colleagues show
that users need time to understand the results of the analysis, and the clusters are not always 100% busy.
Therefore, it can be seen that many computational resources are likely to be available in the Grid for
users and researchers of computational intensive image analysis studies to use. Furthermore, the benefits
of storing the data provenance in VDS apply to all kinds of scientific studies.
The workbench can be further improved in a number of ways. Firstly, RLS should store more in-
formation about the image data, such as scanning parameters, for searching. This can easily be done by
adding the required information to RLS, or by integrating with other systems such as SRB used in the
BIRN project. Secondly, there is no type checking in the workflows constructed using VDL, as present
Figure 6.12: The load of the computer cluster at Worcester Polytechnic Institute over a year (http://wpi-
master.wpi.edu/ganglia/).
in the Medical Imaging Component Language (MICL). This means that if the output and the input of two
programs are not compatible, and the output of a program is used as the input of the other program in a
workflow, errors occur only at runtime. VDL has to be modified in order to incorporate type checking.
Thirdly, access control can be built in to the data provenance because some details should be hidden
from some users. An example is that an observer whose task is to compare the results of a segmentation
algorithm should not know the parameters (e.g. threshold values) used by the algorithm, so that he/she
will not be biased. Again, VDL has to be modified to incorporate security attributes to the parameters
of the transformations and derivations. Finally, none of the systems mentioned so far have demonstrated
the ability to search data, workflows and algorithms based on the semantics (or underlying meaning) that
are provided by the my Grid project [SRG03], in which services and data are annotated using ontolo-
gies2 to produce semantically rich data and services to help bioinformaticians to construct workflows
or distributed queries efficiently and effectively. The workbench needs to incorporate ontologies in the
medical image data and analysis in order to express their semantics. However, developing ontologies
for medical image data and analysis is very difficult due to the size and diversity of the field of medical
science.
The large overhead in the Condor MW is mainly due to the time spent in the merging step of
the spatio-temporal segmentation. The merging step was not parallelised and was performed on one
computer. However, the Condor MW did not release the workers until the whole job had been finished.
Most of the workers were therefore idle during the merging step. Further implementation should separate
the parallel filtering step and the non-parallel merging step, in order not to waste CPU time.
Grid technologies are changing very rapidly. Although an earlier version of the Globus Toolkit
(version 2.4), which is not based on Web Services, is used, the same principles and architectures de-
scribed in this chapter still apply when using a different version of the Globus Toolkit. The migration to
a later version will help to solve potential technical difficulties in deploying version 2.4. One difficulty is
that version 2.4 requires that some ports in the firewall are open to incoming connections, which creates
potential security risks in institutions or companies. This problem is automatically solved if the latest
2 An ontology is a data model that represents a domain and is used to reason about the objects in that domain and the relations
between them (taken from Wikipedia (http://en.wikipedia.org/wiki/Ontology (computer science)).
version of the Globus Toolkit is used because it is based on Web Services, which does not require open
ports in the firewall (firewall-friendly).
All the softwares were found to be quite stable. There is a minor problem in RLS, which sometimes
requires an updating command to be issued manually to update the database. More feedback is required
from users to evaluate the value of the system in helping them to perform automated medical image
analysis. The feedback will also help to remove any inadequacy in the system and suggest requirements
for future development.
Chapter 7
Conclusions
7.1 Summary of findings

The FDA Critical Path Initiative regards imaging as a key technology for assessing, accelerating the de-
velopment of, and guiding the use of new therapeutic options. Imaging biomarkers, such as bone erosion
and oedema in MR images of RA patients, have the potential to be surrogate endpoints in clinical trials.
In addition, it is important to develop automatic image algorithms for the quantification of changes in
image biomarkers in order to minimise variability in the results of clinical trials due to human interac-
tion. In this thesis, I devised image analysis algorithms that were based on image registration for the
automatic quantification of small disease-related changes in longitudinal MR images of joints, and for
assisted visual scoring of RA. I also implemented a scalable infrastructure that was based on the Grid to
enable automatic image analysis algorithms to be applied to large cohorts in the late phase trial, as well
as small numbers of subjects in early phase trials and pre-clinical studies.
The main novel work in this thesis is described in chapters 3, 4, 5 and 6. In this section, the key
findings of these chapters are summarised.
7.1.1 Automatic quantification of changes in longitudinal MR images of joints I

Chapter 3 described two different methods that automatically quantified small changes in the talus bones
of 11 subjects over 6 time points.
Method 1, based on segmentation propagation, was used to accurately delineate the bones in order
to measure changes in global bone volume. A total of 65 segmentations were performed (discounting
the one bone at one time point used for the atlas). For subjects 1-3, the automatic segmentations were
compared with the manual segmentations. Agreement between the automatic segmentations and manual
segmentations, assessed using the similarity index, was very close to the inter-observer agreement in
the manual segmentation. It is expected from the disease model that the bone volume will decrease
monotonically over time as the disease develops and erosions progress. However, the results in Figure 3.7
(page 60) show considerable temporal variability in volume. This could be due to geometrical instability
in the MR scanner, or errors in the segmentation technique.
Method 2, based on the intensity analysis of the bones delineated by Method 1, was used to identify
candidate bone lesion regions within the talus bones and hence quantify disease progression in each
7.1. Summary of findings 146
subject from this intensity information. It has advantages over the first method because it is not dependent
on accurate boundary delineation and can identify the locations of lesions in the bone rather than just
giving a single measure for the bone. A total of eight candidate bone lesion regions were identified and
shown using surface rendering to allow visualisation in 3D. The results of Method 2 were validated by
comparison with the known characteristics of the disease model. In regions 24, 93 and 121, the
bone lesion volumes in male and female subjects are significantly different (P < 0.05). Significant
changes (P < 0.05) in the bone lesion volume between some time points are also detected in the
same regions. Therefore, this shows that the results are likely sensitive to small disease-related
changes. Although there are limitations on the statistical significance due to the small sample size,
the bone lesions volumes in the candidate bone lesion regions have the potential to be used as the
bases of erosion and oedema biomarkers in the experimental model of RA;
comparison with the histology of the subjects. Candidate bone lesion region 24 and region 105
were found to be close to regions with RA-like pathology in the histology sections of 4 subjects;
comparison with manual segmentation of bone lesions. The automatic bone lesion volumes have
strong correlations with the volumes of the manual bone lesion segmentations in candidate bone
lesion region 24 (Pearson correlation = 0.867).
7.1.2 Automatic quantification of changes in longitudinal MR images of joints II

An integrated spatio-temporal analysis method was proposed in Chapter 4 to automatically and sensi-
tively quantify small changes in longitudinal images of joints in RA. The method was used to identify
candidate bone lesion regions and calculate bone lesion volumes in the talus bones of simulated temporal
image sequences and the real longitudinal MR images of 11 subjects over 6 time points. Seven candidate
bone lesion regions were found in the real MR data. When compared to Method 2 in Chapter 3, the
results show that the spatio-temporal method (a) is more sensitive to small lesion changes in simulated
data and in real MR data; and (b) is more robust to image noise. Furthermore, the results of the spatio-
temporal method are not critically sensitive to the choice of the parameters of the algorithm (the window
widths). This suggests that the spatio-temporal method is better at detecting small bone lesions in both
simulated and real data than Method 2 in Chapter 3. The results were validated using the same methods
as in Chapter 3 by
comparison with the known characteristics of the disease model. In region 3, the bone lesion
volumes in male and female subjects are significantly different (P < 0.05). Significant changes
(P < 0.05) in the bone lesion volume between some time points were detected in regions 1, 3, 4,
6 and 7. Although there are limitations in the statistical significance due to the small sample size,
the bone lesions volumes in the candidate bone lesion regions have the potential to be used as the
bases of erosion and oedema biomarkers of disease in the experimental model of RA;
comparison with the histology of the subjects. Candidate bone lesion regions 3, 4 and 6 are close
to regions with RA-like pathology in the histology sections of 4 subjects. In addition, region 4 is
7.1. Summary of findings 147
not detected by Method 2 in Chapter 3;
comparison with manual segmentation of bone lesions. The automatic bone lesion volumes have
strong correlations with the volumes of the manual bone lesion segmentations in candidate bone
lesion region 3 (Pearson correlation = 0.782).
7.1.3 Application to clinical data

Chapter 5 applied image registration and spatio-temporal segmentation to facilitate the analysis of lon-
gitudinal MR hand and wrist images from human RA patients. Firstly, a method was developed to
automatically localise metacarpal and carpal bones in MR images to allow the identification of ROI
around the bones for further analysis, such as the quantification of bone erosions or the quantification of
synovitis from dynamic contrast enhanced MRI. Initial validation showed that 220 out of 252 bones in
21 images were successfully localised. Secondly, the spatio-temporal segmentation was applied to the
longitudinal MR images of RA patients. Thirdly, a method was developed to rigidly align metacarpal
and carpal bones automatically to a standard reference image to improve the ease of use, reproducibil-
ity and the scoring time of the OMERACT RAMIS scoring method. Initial validation showed that 92
out of 105 metacarpal bones in 21 images were successfully registered to the standard reference image.
The spatio-temporal segmentation detected changes in the bone lesion volume over time, which were
confirmed by visual assessment.
7.1.4 Use of the Grid in automated medical image analysis

In Chapter 6, a web-based image analysis workbench was developed by integrating existing medical
image analysis algorithms and Grid technologies, including the Globus Toolkit, Condor and the Virtual
Data System. The workbench was used to perform automatic delineation of multiple bones and the
spatio-temporal segmentation of bone lesions in an experimental model of RA. The web pages provided
by the workbench allow users from different institutions to share
medical image data by using the RLS Upload and Query web pages,
computational resources by submitting to various Run Pools on the VDS Catalogue web page,
storage resources by using the the RLS Upload and Query web pages, and the Source Pools and
the Destination Pools on the VDS Catalogue web page,
image analysis software by using various transformations on the VDS catalogue web page.
At the moment, resources from four different sites, namely ixi, ral, wrg and man, are available
to the user, and new resources can easily be added to the system by entering simple details such as the
name of the site and port numbers. Workflows can be explicitly or implicitly constructed and maintained
using VDS. The data provenance of all the analyses are stored automatically, and can be retrieved and
browsed easily using the VDS Catalogue web pages. In addition, the concept of virtual data and on-
demand data generation avoids the recomputation of image analysis programs when they have already
been performed and the results stored. For example, the VDS Catalogue web pages allow the searching
7.2. Future work 148
of nonrigid registrations in VDS, so that the users can see if certain nonrigid registration has already
been calculated. Finally, the parallel version of the spatio-temporal segmentation was implemented
using Condor MW. The speed-up depends on the number of available computers on the cluster.
7.2 Future work

7.2.1 Improvements to the quantification of changes in the experimental model
of RA
Since only one histological section per stain is available, it is impossible to validate all the candidate
bone lesion regions. In the future, it may be possible to obtain more histological sections to allow more
complete validation. However, tissue distortion inherent to the histological sample preparation process
will mean that accurate spatial comparison of candidate bone lesion regions from MRI and regions iden-
tified in the histology will remain highly challenging. It is likely that there are non-linear geometric
distortions in the MR images from the results of the 9-dof registration. It is desirable to conduct phan-
tom experiments to measure geometric distortions in different time points and to try to correct them, and
to minimise positioning differences when undertaking repeat scans so that the bone positions are as con-
stant as possible with respect to gradient induced non-linearities. There are limitations on the statistical
significances in the preliminary study of the experimental model of RA in chapters 3 and 4 due to the
small number of subjects (n = 11). A larger number of subjects is therefore recommended in future
studies.
Future experiments may use the method to quantify the efficacies of existing disease modifying
therapies, such as TNF alpha inhibitors, and other novel biological compounds. A key challenge in this
area is to quantify erosion healing or repair, which is the opposite of bone erosion. Therefore, by
looking at high-intensity (bone lesion) and low intensity (bone healing) regions in the difference images,
the method may be able to quantify bone erosion and bone healing at the same time.
7.2.2 Other applications of the spatio-temporal segmentation

The spatio-temporal segmentation could be applied to quantify changes in longitudinal images of other
diseases, such as osteoarthritis (OA), multiple sclerosis (MS) and Alzheimers disease (AD). To demon-
strate its potential, I used it to quantify changes in longitudinal MR brain images of MS patients
Current methods for quantifying changes in longitudinal MR images of MS patients include (a)
measuring the brain atrophy, and (b) assessing the status of lesions by counting their total number and
volume. Therefore, image registration and spatial-temporal segmentation are used to automate the two
analyses.
To calculate the volume of the lesions, the follow-up images are rigidly registered to the brain of
the baseline image. Difference images are generated by subtracting the baseline image from the
transformed follow-up images, which are sinc-interpolated using a Hanning window with a kernel
diameter of 7 voxels. The lesions in the longitudinal difference images are delineated using the
spatial-temporal segmentation algorithm, which uses information provided by adjacent time points
to track the lesions.
To measure brain atrophy, one way is to nonrigidly register the follow-up images and the brain
of the baseline image and then calculate the atrophy from the Jacobian of the deformation field.
But, the change in size of the lesions would affect the nonrigid registration algorithm. An example
is that the growth of a lesion is probably due to more scarred or inflamed brain tissue instead of
due to the expansion (increase in volume) of the underlying brain tissue in the lesion, which is the
result given by nonrigid registration. Therefore, two ways are proposed to solve this problem by
using the delineations of lesions calculated in the last paragraph:
1. When calculating the atrophy using the Jacobian, the values inside the lesions are ignored or
set to zero.
2. When performing the nonrigid registration, the control points inside the lesions are set to
be passive so that no deformation is allowed, and the calculation of the similarity measure
ignores the voxel values within the lesions.
The above method is applied to longitudinal MR brain images of MS patients provided by the
Dementia Research Centre, Institute of Neurology, UCL, UK. The results of the rigid registration of a
baseline and a follow-up image are shown in Figure 7.1. A MS lesion is contained in the white boxes.
The results of the spatio-temporal segmentation are shown in Figure 7.2. The results of the nonrigid
registration are shown in Figure 7.3. It should be noticed that the deformation vector field generated by
the nonrigid registration without the MS lesions in the images is much smaller than that with the MS
lesions. Experiments are still ongoing to obtain more results.
7.2.3 Automatic window width determination in the spatio-temporal segmenta-

tion
The window widths are the most important parameters in the spatio-temporal segmentation. In Chapter 4,
they are determined using simulated image data that model the intensity and progression of a typical
bone lesion in RA. This is not ideal when trying to apply the algorithm to other image data or to quantify
changes in longitudinal images of other diseases, such as multiple sclerosis. Existing automatic window
width selection algorithms (e.g. [CRM01] and [Sil86]) have proven successful in a number of areas,
such as the segmentation of 2D colour video image sequences. Although initial experiments using the
algorithms to automatically select window widths in Chapter 4 do not produce good results, they should
be further investigated in order to give an automatic window selection algorithm for the spatio-temporal
segmentation.
7.2.4 Improvements to automatic bone localisation and alignment

In Chapter 5, only 6 images are used to validate the combination of bone hierarchy and multiple starting
estimates. The combination should be tested on more images when they are available. The automatic
bone localisation and alignment failed on two out of 72 cases. Future work should investigate how to
improve the image registrations in the two failed cases. For example, (a) a better atlas that matches
Figure 7.1: Results of rigid registration. The top, middle, and bottom rows show the baseline image,
transformed follow-up image, and difference image (transformed follow-up image minus baseline image)
respectively. A MS lesion is contained in the white boxes. The coronal, transaxial, and sagittal view are
shown from left to right.
Figure 7.2: Results of the spatio-temporal segmentation. The lesions delineated by the algorithm are
highlighted in white in the baseline image. The coronal, transaxial, and sagittal view are shown from left
to right.
Figure 7.3: Deformation vector fields of the follow-up image with and without MS lesions. The top,
middle, and bottom rows show the baseline image, the vector field generated by the nonrigid registration
with the MS lesions in the images, and the vector field generated by the nonrigid registration without the
MS lesions in the images respectively. The coronal, transaxial, and sagittal view are shown from left to
right.
7.3. Summary 152
the gender, age and ethnicity of the patient could be used; (b) the combination of bone hierarchy and
multiple starting estimates could be improved by using different images in determining the combination.
7.2.5 Dynamic contrast enhanced (DCE) imaging of synovitis in RA patients

Synovitis is one of the important features of RA, as reviewed in Chapter 2. The DCE images allow the
delineation of synovial membranes and the calculation of the rate of early synovial enhancement. The
synovial volume was found to be closely related to the rate of progressive joint destruction [OHSL96,
OHS+ 99], and it was shown that the rate of early synovial enhancement could distinguish between active
and inactive RA [CIL+ 03]. Future work could investigate whether the synovial volume and the rate of
early synovial enhancement can automatically be determined from the DCE images.
7.2.6 Improvements to the integration of medical image analysis and the Grid
The usability of the web portal developed in Chapter 6 should be evaluated in the future. It is desirable to
get the feedback of end users in order to improve the user interface of the web portal. In the future, after
the ontologies in the medical image data and analysis have been developed, they could be incorporated
into the workbench, so that the discovery and construction of data, workflows and algorithms can be
based on their semantics (or underlying meaning).
7.2.7 Different ways of building the feature space in the spatio-temporal segmen-
tation
Since multispectral analysis is used to classify the tissues in the joint [CLR+ 04], information from
MR images with different contrasts can be added to the existing feature space by introducing extra
dimensions. However, the curse of dimensionality will confound the calculations in a high-dimensional
space (say, more than 7 dimensions [Sil86]).
7.3 Summary
This thesis is concerned with the automatic and sensitive quantification of small changes in longitudinal
MR images in RA in order to quantify the disease progression, and the potential of Grid technologies to
address various challenges in the automated analysis of large volume of image data. Existing methods
for quantifying changes in longitudinal MR images often involve labour-intensive manual segmentation
and visual scoring. Chapter 3 applied two different methods to automatically quantify small changes in
the longitudinal MR images of joints in an experimental model of RA. The first method used nonrigid
registration to accurately delineate a bone, and measured the changes in the bone volume. The second
method analysed longitudinal intensity changes within a bone in a reference coordinate system, and
measured the changes in the bone lesion volume, which were likely to sensitive to small disease-related
changes. Chapter 4 proposed a novel method that incorporated the time-domain information in the tem-
poral image sequences to further increase the sensitivity and robustness of the quantification of small
changes in longitudinal MR images in the experimental model. Chapter 5 applied the method developed
in Chapter 4 to delineate bone lesions from longitudinal MR images of human RA patients for the quan-
tification of disease progression. Changes in bone lesion volume were detected and confirmed by visual
7.3. Summary 153
assessment. It also proposed two methods, which were based on image registration, to automatically
and roughly delineate the carpal and meta-carpal bones in the MR images for further processing (such
as accurate bone delineation), and to automatically align the bones to a reference image for better vi-
sual scoring. Chapter 6 integrated image analysis algorithms with Grid technologies to demonstrate the
potential of the Grid in addressing the challenges of applying these computationally demanding image
analysis algorithms to large cohorts collected from many geographically different imaging centres.
Appendix A
Appendix of Chapter 3
A.1 Binary opening and dilation operators

This section describes the binary opening and dilation operations in the algorithm. The binary opening
would remove excessive bone lesions if a normal 3 3 3-structuring element was used. This was due
to the anisotropic voxel size (58.6 156 58.6(m)3 ), giving lower resolution in the y-direction of
the MR images. Although the images can be resampled to have isotropic voxel size, the interpolation
causes the images to become slightly blurred and have larger data size. Since mathematical morphology
was chosen to remove the isolated bone lesions, the upper half kernel and the lower half kernel were
designed to address the problem of anisotropic voxel size by only removing voxels either above or below
the kernel in the erosion step. An upper half kernel was a 3 3 3-structuring element with 0 at all
the positions where y = -1, while a lower half kernel was a 3 3 3-structuring element with 0 at all
the positions where y = +1. A 2D version of the kernels are shown in Figure 3.6.
Figure A.1 shows an example demonstrating the effect of the binary opening and dilation operations
in the algorithm. It should be noted that (a) the anisotropic pixel size in the example resembles the
anisotropic voxel size in the MR images; (b) the four pixels at the top of the image simulate errors due to
misregistration and thresholding around the top edge of the bone; (c) the union operation compensates
for any asymmetry introduced by one of the kernels. If a normal 3 3-structuring element was used in
a binary opening operation, all the bone lesions would be removed from the image.
A.2 Average bone lesion volumes in various candidate bone lesion

regions
The average bone lesion volumes in various candidate bone lesion regions are shown below.
A.2. Average bone lesion volumes in various candidate bone lesion regions 155
Figure A.1: Example demonstrating the effects of the morphological operations used in Figure 3.5.
Figure A.2: Graph of average bone lesion volume of male and female subjects in region 1 against time.
Error bars denote the standard deviations of the average bone volume.
Figure A.8: Graph of average bone lesion volume of male and female subjects in region 105 against
time. Error bars denote the standard deviations of the average bone volume.
Figure A.9: Graph of average bone lesion volume of male and female subjects in region 121 against
time. Error bars denote the standard deviations of the average bone volume.
Appendix B
Appendix of Chapter 4
B.1 Average bone lesion volumes in various candidate bone lesion

regions
The average bone lesion volumes in various candidate bone lesion regions are shown below.
Figure B.1: Graph of average bone lesion volume of male and female subjects in region 1 obtained by
B.1. Average bone lesion volumes in various candidate bone lesion regions 161
Bibliography
[AA95] M. Akil and R. S. Amos. ABC of Rheumatology: RHEUMATOID ARTHRITISI:

CLINICAL FEATURES AND DIAGNOSIS. BMJ, 310(6979):587, 1995.
[AEB+ 88] F. C. Arnett, S. M. Edworthy, D. A. Bloch, D. J. McShane, J. F. Fries, N. S. Cooper,

L. A. Healey, S. R. Kaplan, M. H. Liang, and H. S. Luthra. The American Rheumatism
Association 1987 revised criteria for the classification of rheumatoid arthritis. Arthritis
Rheum, 31(3):315324, Mar 1988.
[AF99] J. Ashburner and K. J. Fiston. Nonlinear spatial normalization using basis functions.
Human Brain Mapping, 7(4):254266, 1999.
[AVvO+ 04] Petronella Anbeek, Koen L Vincken, Matthias J P van Osch, Robertus H C Bisschops,
and Jeroen van der Grond. Automatic segmentation of different-sized white matter
lesions by voxel probability estimation. Med Image Anal, 8(3):205215, Sep 2004.
[BBG00] D. Burstein, A. Bashir, and M. L. Gray. MRI techniques in early stages of cartilage
disease. Investigative Radiology., 35(10):622, 2000.
[BCB+ 03] K. J. Brooks, K. K. Changani, J. B. Buckton, A. E. White, D. G. Reid, A. L. Busza,

B. P. Hayes, and P. F. Life. PG-PS (proteoglycan-polysaccharide)-induced rheumatoid
arthritis in the rat ankle: First characterisation by MRI and X-ray CT ex vivo. In
Proceedings of the International Society of Magnetic Resonance in Medicine, Toronto,
Ontario, Canada, 2003.
[BCR+ 01] E. Bardinet, Alan C. F. Colchester, A. Roche, Y. Zhu, Y. He, S. Ourselin, B. Nailon,
S. A. Hojjat, J. Ironside, S. Al-Sarraj, N. Ayache, and J. Wardlaw. Registration of
reconstructed post mortem optical data with mr scans of the same patient. In MICCAI
2001: Proceedings of the 4th International Conference on Medical Image Computing
and Computer-Assisted Intervention, page 957. Springer-Verlag, 2001.
[BDM+ 02] E. Bardinet, D. Dormont, G. Malandain, D. Tande, K. Parain, N. Ayache, and J. Yelnik.
Co-registration of histological, optical and MR data of the human brain. In MICCAI
and Computer-Assisted Intervention, volume 2488, page 548, 2002.
Bibliography 165
[BG03] D. Burstein and M. Gray. New MRI techniques for imaging cartilage. Investigative
Radiology., 85-A Suppl 2:70, 2003.
[BGHB99] A. Bashir, M. L. Gray, J. Hartke, and D. Burstein. Nondestructive imaging of hu-

man cartilage glycosaminoglycan concentration by MRI. Magnetic Resonance in
Medicine., 41(5):857, 1999.
[Bil05] M.E.J. Billingham. Mechanisms and Models in Rheumatoid Arthritis, chapter 20,
pages 389410. Academic Press, 2005.
[Bio03] Biomedical Imaging Resource, Mayo Foundation. Analyze 6.0, 6.0 edition, 2003.
[BJL+ 05] Paul Bird, Fredrick Joshua, Marissa Lassere, Ron Shnier, and John Edmonds. Train-
ing and calibration improve inter-reader reliability of joint damage assessment using
magnetic resonance image scoring and computerized erosion volume measurement. J
Rheumatol, 32(8):14528, Aug 2005.
[BK89] Ruzena Bajcsy and Stane Kovacic. Multiresolution elastic matching. Comput. Vision
Graph. Image Process., 46(1):121, 1989.
[BKP+ 01] Jane M. Blackall, Andrew P. King, Graeme P. Penney, A. Adam, and David J. Hawkes.
A statistical model of respiratory motion and deformation of the liver. In MICCAI
and Computer-Assisted Intervention, pages 13381340, London, UK, 2001. Springer-
Verlag.
[BKS+ 99] M. Backhaus, T. Kamradt, D. Sandrock, D. Loreck, J. Fritz, K. J. Wolf, H. Raber,

B. Hamm, G. R. Burmester, and M. Bollow. Arthritis of the finger joints: a compre-
hensive approach comparing conventional radiography, scintigraphy, ultrasound, and
contrast-enhanced magnetic resonance imaging. Arthritis Rheum., 42(6):1232, 1999.
[BKT01] P. Buneman, S. Khanna, and W. C. Tan. Why and where: A characterization of data
provenance. Lecture notes in computer science., (1973):316, 2001.
[BLD+ 04] Gabrielle Blumenkrantz, Colleen T. Lindsey, Timothy C. Dunn, Hua Jin, Michael D.
Ries, Thomas M. Link, Lynne S. Steinbach, and Sharmila Majumdar. A pilot, two-
year longitudinal study of the interrelationship between trabecular bone and articular
cartilage in the osteoarthritic knee. Osteoarthritis and Cartilage, 12(12):9971005,
December 2004.
[BLSE03] P. Bird, M. Lassere, R. Shnier, and J. Edmonds. Computerized measurement of mag-

netic resonance imaging erosion volumes in patients with rheumatoid arthritis: a com-
parison with existing magnetic resonance imaging scoring systems and standard clini-
cal outcome measures. Arthritis Rheum., 48(3):614, 2003.
Bibliography 166
[BMT+ 05] E. Bagarinao, K. Matsuo, Y. Tanaka, L. F G Sarmenta, and T. Nakai. Enabling on-
demand real-time functional MRI analysis using grid technology. Methods Inf Med,
44(5):665673, 2005.
[Boo89] F. L. Bookstein. Principal warps: Thin-plate splines and the decomposition of defor-
mations. IEEE Trans. Pattern Anal. Mach. Intell., 11(6):567585, 1989.
[BRR+ 04] M Burns, AL Rowland, D Rueckert, JV Hajnal, K Leung, DLG Hill, and J Vickers.
Information eXtraction from images (IXI): Grid services for medical imaging. In Dis-
tributed Databases and processing in Medical Image Computing ( DiDaMIC), 2004.
[BTF+ 94] M. Boers, P. Tugwell, D. T. Felson, P. L. van Riel, J. R. Kirwan, J. P. Edmonds, J. S.

Smolen, N. Khaltaev, and K. D. Muirden. World Health Organization and International
League of Associations for Rheumatology core endpoints for symptom modifying an-
tirheumatic drugs in rheumatoid arthritis clinical trials. J Rheumatol.Suppl, 41:86,
1994.
[BW84] J. C. Buckland-Wright. Microfocal radiographic examination of erosions in the wrist

and hand of patients with rheumatoid arthritis. Ann Rheum Dis, 43(2):16071, Apr
1984.
[BWCCG93] J. C. Buckland-Wright, G. S. Clarke, I. C. Chikanza, and R. Grahame. Quantitative mi-

crofocal radiography detects changes in erosion area in patients with early rheumatoid
arthritis treated with myocrisine. J Rheumatol, 20(2):2437, Feb 1993.
[BWCW86] J. C. Buckland-Wright, I. Carmichael, and S. R. Walker. Quantitative microfocal ra-

diography accurately detects joint changes in rheumatoid arthritis. Ann Rheum Dis,
45(5):37983, May 1986.
[CBE+ 05] P. Conaghan, P. Bird, B. Ejbjerg, P. OConnor, C. Peterfy, F. McQueen, M. Lassere,

P. Emery, R. Shnier, J. Edmonds, and M. Ostergaard. The EULAR-OMERACT
rheumatoid arthritis MRI reference image atlas: the metacarpophalangeal joints. Ann
Rheum Dis, 64 Suppl 1:i1121, Feb 2005.
[CCA+ 79] R. L. Clark, J. T. Cuttino, S. K. Anderle, W. J. Cromartie, and J. H. Schwab. Radiologic

analysis of arthritis in rats after systemic injection of streptococcal cell walls. Arthritis
Rheum, 22(1):2535, Jan 1979.
[CE97] D. L. Collins and A. C. Evans. Animal: Validation and applications of nonlinear

registration-based segmentation. International Journal of Pattern Recognition and
Artificial Intelligence, 11:12711294, 1997.
[CHH04] W. R. Crum, T. Hartkens, and D. L G Hill. Non-rigid image registration: theory and
practice. Br J Radiol, 77 Spec No 2:S14053, 2004.
Bibliography 167
[CIL+ 03] M. A. Cimmino, S. Innocenti, F. Livrone, F. Magnaguagno, E. Silvestri, and G. Gar-

laschi. Dynamic gadolinium-enhanced magnetic resonance imaging of the wrist in pa-
tients with rheumatoid arthritis can discriminate active from inactive disease. Arthritis
Rheum., 48(5):1207, 2003.
[CJHS04] S. A. Connolly, D. Jaramillo, J. K. Hong, and F. Shapiro. Skeletal development in

fetal pig specimens: Mr imaging of femur with histologic comparison. Radiology,
233(2):505, 2004.
[CLO+ 03] P. Conaghan, M. Lassere, M. Ostergaard, C. Peterfy, F. McQueen, P. OConnor,

P. Bird, B. Ejbjerg, M. Klarlund, R. Shnier, H. Genant, P. Emery, and J. Edmonds.
OMERACT rheumatoid arthritis magnetic resonance imaging studies. exercise 4: an
international multicenter longitudinal study using the RA-MRI score. J Rheumatol.,
30(6):1376, 2003.
[CLR+ 04] R. A. Carano, J. A. Lynch, J. Redei, S. Ostrowitzki, Y. Miaux, S. Zaim, D. L. White,

C. G. Peterfy, and H. K. Genant. Multispectral analysis of bone lesions in the hands
of patients with rheumatoid arthritis. Magn Reson.Imaging, 22(4):505, 2004.
[CM02] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space
analysis. IEEE Trans. Pattern Anal. Mach. Intell., 24(5):603619, 2002.
[CMR04a] R. Chandrashekara, R. H. Mohiaddin, and D. Rueckert. Cardiac motion tracking in

tagged MR images using a 4D B-spline motion model and nonrigid image registration.
IEEE International Symposium on Biomedical Imaging (ISBI 2004), page 468, 2004.
[CMR04b] Raghavendra Chandrashekara, Raad H Mohiaddin, and Daniel Rueckert. Analysis of

3-D myocardial motion in tagged MR images using nonrigid image registration. IEEE
Trans Med Imaging, 23(10):124550, Oct 2004.
[cona] The Condor Master-Worker Project (http://www.cs.wisc.edu/condor/mw/), 2005.
[conb] The Condor Project (http://www.cs.wisc.edu/condor/), 2003.
[CR00] G. Calmon and N. Roberts. Automatic measurement of changes in brain volume

on consecutive 3d mr images by segmentation propagation. Magn Reson Imaging,
18(4):439453, May 2000.
[CRM96] GE Christensen, RD Rabbitt, and MI Miller. Deformable template using large defor-
mation kinematics. IEEE Transactions on Image Processing, 5:14351447, 1996.
[CRM01] D. Comaniciu, V. Ramesh, and P. Meer. The variable bandwidth mean shift and data-
driven scale selection. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth
IEEE International Conference on, volume 1, pages 438445 vol.1, 2001.
Bibliography 168
[CSF01] William R. Crum, Rachael I. Scahill, and Nick C. Fox. Automated hippocampal seg-
mentation by regional fluid registration of serial MRI: Validation and application in
alzheimers disease. NeuroImage, 13(5):847855, May 2001.
[CVC+ 95] L. P. Clarke, R. P. Velthuizen, M. A. Camacho, J. J. Heine, M. Vaidyanathan, L. O.

Hall, R. W. Thatcher, and M. L. Silbiger. MRI segmentation: methods and applica-
tions. Magn Reson.Imaging, 13(3):343, 1995.
[DBG+ 04] Ewa Deelman, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Sonal
Patil, Mei-Hui Su, Karan Vahi, and Miron Livny. Pegasus: Mapping scientific work-
flows onto the Grid. In AxGrids 2004, volume 3165, pages 1120, January 2004.
[DBW+ 04] Evgenia Dimitriadou, Markus Barth, Christian Windischberger, Kurt Hornik, and
Ewald Moser. A quantitative comparison of functional mri cluster analysis. Artifi-
cial Intelligence in Medicine, 31(1):5771, May 2004.
[DeL99] L. E. DeLisi. Regional brain volume change over the life-time course of schizophrenia.
J Psychiatr Res, 33(6):535541, 1999.
[DeM02] D. DeMenthon. Spatio-temporal segmentation of video by hierarchical mean shift

analysis. In Statistical Methods in Video Processing Workshop, Copenhagen, Den-
mark, 2002.
[DHC+ 00] E. R. Denton, M. Holden, E. Christ, J. M. Jarosz, D. Russell-Jones, J. Goodey, T. C.

Cox, and D. L. Hill. The identification of cerebral volume changes in treated growth
hormone-deficient adults using serial 3D MR image processing. Journal of Computer
Assisted Tomography., 24(1):139, 2000.
[DHT+ 99] B. M. Dawant, S. L. Hartmann, J. P. Thirion, F. Maes, D. Vandermeulen, and P. De-

maerel. Automatic 3-D segmentation of internal structures of the head in MR images
using a combination of similarity and free-form transformations: Part I, methodology
and validation on normal subjects. IEEE Trans.Med Imaging, 18(10):909, 1999.
[DKB+ 99] J. R. DeQuardo, M. S. Keshavan, F. L. Bookstein, W. W. Bagwell, W. D. Green, J. A.

Sweeney, G. L. Haas, R. Tandon, N. R. Schooler, and J. W. Pettegrew. Landmark-based
morphometric analysis of first-episode schizophrenia. Biol Psychiatry, 45(10):1321
1328, May 1999.
[DPSA+ 02] V. Devauchelle-Pensec, A. Saraux, S. Alapetite, D. Colin, and P. Le Goff. Diagnostic

value of radiographs of the hands and feet in early rheumatoid arthritis. Joint, Bone,
Spine: Revue du Rhumatisme., 69(5):434, 2002.
[Ell03] P.J. Ell. Imaging in Rheumatology. Oxford University Press, 2003.

Bibliography 169
[EML+ 05] B. Ejbjerg, F. McQueen, M. Lassere, E. Haavardsholm, P. Conaghan, P. OConnor,

P. Bird, C. Peterfy, J. Edmonds, M. Szkudlarek, H. Genant, P. Emery, and M. Oster-
gaard. The EULAR-OMERACT rheumatoid arthritis MRI reference image atlas: the
wrist joint. Ann Rheum Dis, 64 Suppl 1:i2347, Feb 2005.
[EMR05] Florida Estrella, Richard McClatchey, and Dmitry Rogulin. The MammoGrid Virtual
Organisation - Federating Distributed Mammograms. Stud Health Technol Inform,
116:935940, 2005.
[ESCS85] R. E. Esser, S. A. Stimpson, W. J. Cromartie, and J. H. Schwab. Reactivation of

streptococcal cell wall-induced arthritis by homologous and heterologous cell wall
polymers. Arthritis Rheum., 28(12):1402, 1985.
[fan] FANTASM: Fuzzy and Noise Tolerant Adaptive Segmentation Method

(http://www.iacl.ece.jhu.edu/projects/fantasm/), 2005.
[fda] Guidance for industry on part 11, electronic records; electronic signaturesscope and
application (http://www.fda.gov/ohrms/dockets/98fr/5667fnl.pdf), 2004.
[Fel02] M. Feldmann. Development of anti-TNF therapy for rheumatoid arthritis. Nature

Reviews.Immunology., 2(5):364, 2002.
[FF97] P. A. Freeborough and N. C. Fox. The boundary shift integral: an accurate and robust
measure of cerebral volume changes from registered repeat MRI. IEEE Trans Med
Imaging, 16(5):6239, Oct 1997.
[FH75] K. Fukunaga and L. Hostetler. The estimation of the gradient of a density function,
with applications in pattern recognition. Information Theory, IEEE Transactions on,
21(1):32, 1975.
[Fir03] Gary S. Firestein. Evolving concepts of rheumatoid arthritis. Nature, 423(6937):356

361, May 2003.
[FJL+ 00] N. C. Fox, R. Jenkins, S. M. Leary, V. L. Stevenson, N. A. Losseff, W. R. Crum, R. J.

Harvey, M. N. Rossor, D. H. Miller, and A. J. Thompson. Progressive cerebral atrophy
in MS: a serial study using registered, volumetric MRI. Neurology, 54(4):807812,
Feb 2000.
[FJLS03] K. Forslind, A. Johanson, E. M. Larsson, and B. Svensson. Magnetic resonance imag-

ing of the fifth metatarsophalangeal joint compared with conventional radiography
in patients with early rheumatoid arthritis. Scandinavian Journal of Rheumatology.,
32(3):131, 2003.
[FKNT] Ian Foster, Carl Kesselman, Jeffery M. Nick, and Steven Tueche. The physiology of
the Grid, 2002/06/22/.
Bibliography 170
[FKT01] I. Foster, C. Kesselman, and S. Tuecke. The anatomy of the Grid: Enabling scalable
virtual organizations. International J.Supercomputer Applications, 15(3), 2001.
[FNM+ 02] Matthieu Ferrant, Arya Nabavi, Benot Macq, P. M. Black, Ferenc A Jolesz, Ron Kiki-
nis, and Simon K Warfield. Serial registration of intraoperative MR images of the
brain. Med Image Anal, 6(4):337359, Dec 2002.
[Fre95] A.J. Freemont. Mechanisms and Models in Rheumatoid Arthritis. Number p.83-p.113.
Academic Press Inc, 1995.
[FRS+ 02] E. Fisher, R. A. Rudick, J. H. Simon, G. Cutter, M. Baier, J. C. Lee, D. Miller,

B. Weinstock-Guttman, M. K. Mass, D. S. Dougherty, and N. A. Simonian. Eight-year
follow-up study of brain atrophy in patients with MS. Neurology, 59(9):14121420,
2002.
[FVWZ02] I. Foster, J. Vockler, M. Wilde, and Y. Zhao. Chimera: A virtual data system for
representing, querying, and automating data derivation. In Proceedings of 14th Inter-
national Conference on Scientific and Statistical Database Management, Proceedings
of 14th International Conference on Scientific and Statistical Database Management,
2002.
[FVWZ03] I. Foster, J. Vockler, M. Wilde, and Y. Zhao. The virtual data grid: A new model and
architecture for data-intensive collaboration. In Conference on Innovative Data System
Research (CIDR 2003 ), 2003.
[FWF+ 96a] N. C. Fox, E. K. Warrington, P. A. Freeborough, P. Hartikainen, A. M. Kennedy, J. M.

Stevens, and M. N. Rossor. Presymptomatic hippocampal atrophy in alzheimers dis-
ease. a longitudinal MRI study. Brain, 119 ( Pt 6):20017, Dec 1996.
[FWF96b] P. A. Freeborough, R. P. Woods, and N. C. Fox. Accurate registration of serial 3D MR

brain images and its application to visualizing change in neurodegenerative disorders.
Journal of Computer Assisted Tomography., 20(6):1012, 1996.
[FWM98] J. M. Fitzpatrick, J. B. West, and C. R. Maurer. Predicting error in rigid-body point-

based registration. IEEE Trans Med Imaging, 17(5):694702, Oct 1998.
[GC98] W. Grassi and C. Cervini. Ultrasonography in rheumatology: an evolving technique.

Ann.Rheum.Dis., 57(5):268, 1998.
[GDK+ 02] S. J. Gandy, P. A. Dieppe, M. C. Keen, R. A. Maciewicz, I. Watt, and J. C. Waterton.

No loss of cartilage volume over three years in patients with knee osteoarthritis as
assessed by magnetic resonance imaging. Osteoarthritis and Cartilage, 10(12):929
937, December 2002.
Bibliography 171
[Gla93] C. A. Glasbey. An analysis of histogram-based thresholding algorithms. CVGIP:

Graph. Models Image Process., 55(6):532537, 1993.
[gli06] EGEE lightweight middleware for grid computing (http://glite.web.cern.ch/glite/),

2006.
[glo] The Globus Toolkit (http://www.globus.org/toolkit/), 2003.
[GLY00] Jean-Pierre Goux, Jeff Linderot, and Michael Yoder. Metacomputing and the master-
worker paradigm. Technical report, Mathematics and Computer Science Division,
Argonne National Laboratory, 2000.
[GPZS05] P. Garnero, C. Peterfy, S. Zaim, and M. Schoenharting. Bone marrow abnormalities on

magnetic resonance imaging are associated with type II collagen degradation in knee
osteoarthritis: A three-month longitudinal study. Arthritis & Rheumatism, 52(9):2822
2829, 2005.
[Haw98] D. J. Hawkes. Algorithms for radiological image registration and their clinical appli-
cation. J Anat, 193 ( Pt 3):347361, Oct 1998.
[HBHH01] D. L. Hill, P. G. Batchelor, M. Holden, and D. J. Hawkes. Medical image registration.

Physics in Medicine & Biology., 46(3):R1, 2001.
[HDM+ 98] J. L. Herring, B. M. Dawant, C. R. Maurer, D. M. Muratore, R. L. Galloway, and J. M.

Fitzpatrick. Surface-based registration of CT images to physical space for image-
guided surgery of the spine: a sensitivity study. IEEE Trans Med Imaging, 17(5):743
752, Oct 1998.
[Hen85] R. Mark Henkelman. Measurement of signal intensities in the presence of noise in MR

images. Medical Physics, 12(2):232233, 1985.
[HEP05] B. Henderson, J.C.W. Edwards, and E.R. Pettipher, editors. Mechanisms and Models
in Rheumatoid Arthritis. Academic Press, 2005.
[HGB+ 03] B. A. Hargreaves, G. E. Gold, C. F. Beaulieu, S. S. Vasanawala, D. G. Nishimura,

and J. M. Pauly. Comparison of new sequences for high-resolution cartilage imaging.
Magn Reson.Med, 49(4):700, 2003.
[HGL+ 99] B. A. Hargreaves, G. E. Gold, P. K. Lang, S. M. Conolly, J. M. Pauly, G. Bergman,

J. Vandevenne, and D. G. Nishimura. MR imaging of articular cartilage using driven
equilibrium. Magnetic Resonance in Medicine., 42(4):695, 1999.
[HHD+ 00] M. Holden, D. L. Hill, E. R. Denton, J. M. Jarosz, T. C. Cox, T. Rohlfing, J. Goodey,

and D. J. Hawkes. Voxel similarity measures for 3-D serial MR brain image registra-
tion. IEEE Transactions on Medical Imaging., 19(2):94, 2000.
Bibliography 172
[HHH01] J. Hajnal, D. J. Hawkes, and D. L. G. Hill, editors. Medical Image Registration. CRC
Press, 2001.
[HHL+ 03] Rolf A. Heckemann, Thomas Hartkens, Kelvin K. Leung, Yalin Zheng, Derek L. G.
Hill, Joseph V. Hajnal, and Daniel Rueckert. Information extraction from medical
images: Developing an e-science application based on the globus toolkit. In UK e-
Science All Hands Meeting, 2003.
[HHR+ 02] D.L.G. Hill, J.V. Hajnal, D. Rueckert, S.M. Smith, T. Hartkens, and K. McLeish. A
Dynamic Brain Atlas, volume 2488. January 2002.
[HKL+ 03] Shannon Hastings, Tahsin Kurc, Stephen Langella, Umit Catalyurek, Tony Pan, and
Joel Saltz. Image processing or the Grid: A toolkit or building Grid-enabled image
processing applications. In CCGRID 03: Proceedings of the 3st International Sym-
posium on Cluster Computing and the Grid, page 36, Washington, DC, USA, 2003.
IEEE Computer Society.
[HOE+ 05] Espen A Haavardsholm, Mikkel Ostergaard, Bo J Ejbjerg, Nils P Kvan, Till A Uh-
lig, Finn G Lilles, and Tore K Kvien. Reliability and sensitivity to change of the
OMERACT rheumatoid arthritis magnetic resonance imaging score in a multireader,
longitudinal setting. Arthritis Rheum, 52(12):38607, Dec 2005.
[HRS+ 99] A. Hagemann, K. Rohr, H. S. Stiehl, U. Spetzger, and J. M. Gilsbach. Biomechanical

modeling of the human head for physically based, nonrigid image registration. IEEE
Trans Med Imaging, 18(10):875884, Oct 1999.
[HRSH02] T. Hartkens, D. Rueckert, J.A. Schnabel, and D.J. Hawkesand D.L.G. Hill. Vtk cisg
registration toolkit: An open source software package for affine and non-rigid regis-
tration of single- and multimodal 3d images. In Proc. Bildverarbeitung in der Medizin
2002, LNCS, pages 409412, Leipzig, March 2002. Springer Verlag.
[HSH02] M. Holden, J. A. Schnabel, and D. L. Hill. Quantification of small cerebral ventricular

volume changes in treated growth hormone patients using nonrigid registration. IEEE
Transactions on Medical Imaging., 21(10):1292, 2002.
[HSO+ 95] J. V. Hajnal, N. Saeed, A. Oatridge, E. J. Williams, I. R. Young, and G. M. Bydder.

Detection of subtle brain changes using subvoxel registration and subtraction of serial
MR images. Journal of Computer Assisted Tomography., 19(5):677, 1995.
[HZL+ 05] H. K. Huang, Aifeng Zhang, Brent Liu, Zheng Zhou, Jorge Documet, Nelson King,
and L. W. C. Chan. Data grid for large-scale medical image archive and analysis. In
MULTIMEDIA 05: Proceedings of the 13th annual ACM international conference on
Multimedia, pages 10051013, New York, NY, USA, 2005. ACM Press.
Bibliography 173
[itk] The Insight Segmentation and Registration ToolKit (http://www.itk.org/), 2003.
[jak] The Jakarta Site - Apache Tomcat (http://jakarta.apache.org/tomcat/), 2004/06/16/.
[JC05] Luiz Carlos Junqueira and Jose Carneiro. Basic Histology: text & atlas. McGraw-Hill,
11 edition, 2005.
[JMF99] A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: a review. ACM Comput.
Surv., 31(3):264323, 1999.
[JMY03] J.R. Jimenez, V. Medina, and O. Yanez. Nonparametric MRI segmentation using mean
shift and edge confidence maps. In Medical Imaging 2003: Image Processing, volume
5032, pages 14331441, 2003.
[jst] Java Servlet Technology (http://java.sun.com/products/servlet/), 2006.
[JWSZ+ 99] M. A. Jacobs, J. P. Windham, H. Soltanian-Zadeh, D. J. Peck, and R. A. Knight. Regis-

tration and warping of magnetic resonance images to histological sections. Med.Phys.,
26(8):1568, 1999.
[KBH+ 03] C. Kenwright, E. Bardinet, S. A. Hojjat, G. Malandain, N. Ayache, and Alan C. F.

Colchester. 2-D to 3-D refinement of post mortem optical and MRI co-registration. In
Medical Image Computing and Computer-Assisted Intervention - MICCAI 2003, page
957. Springer-Verlag, 2003.
[KMT+ 04] Mannudeep K. Kalra, Michael M. Maher, Thomas L. Toth, Leena M. Hamberg,
Michael A. Blake, Jo-Anne Shepard, and Sanjay Saini. Strategies for CT radiation
dose optimization. Radiology, 230(3):619628, March 2004.
[Koh89] T Kohonen. Self-organization and associative memory. Springer-Verlag New York,

Inc. New York, NY, USA, 1989.
[KOK05] Krishnaswamy Kannan, Robert A Ortmann, and Donald Kimpel. Animal mod-
els of rheumatoid arthritis and their relevance to human disease. Pathophysiology,
12(3):167181, Oct 2005.
[KSW90] Hermann Knig, Juergen Sieper, and Karl-Juergen Wolf. Rheumatoid arthritis: Evalu-
ation of hypervascular and fibrous pannus with dynamic MR imaging enhanced with
Gd-DTPA. Radiology, 176:473, 1990.
[LAB+ 05] Bertram Ludascher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew
Jones, Edward A. Lee, Jing Tao, and Yang. Zhao. Scientific workflow management
and the Kepler system. Concurrency and Computation: Practice and Experience,
9999(9999):n/a, 2005.
Bibliography 174
[LAYC+ 04] P. Lam, K. M. Au-Yeung, P. W. Cheng, W. I. Wei, A. P. Yuen, N. Trendell-Smith, J. H.

Li, and R. Li. Correlating MRI and histologic tumor thickness in the assessment of
oral tongue cancer. AJR Am.J.Roentgenol., 182(3):803, 2004.
[LLB+ 03] R. S. Lazebnik, T. L. Lancaster, M. S. Breen, J. S. Lewin, and D. L. Wilson. Volume

registration using needle paths and point landmarks for evaluation of interventional
MRI treatments. IEEE Trans.Med.Imaging, 22(5):653, 2003.
[LMPT+ 04] Lifeng Liu, Dominik Meier, Mariann Polgar-Turcsanyi, Pawel Karkocha, Rohit Baksh,
and Charles R. G. Guttmann. Event-driven workflow management for medical image
processing and analysis in a large image database. DiDaMIC Workshop-MICCAI,
2004.
[LWM+ 98] L. Lemieux, U. C. Wieshmann, N. F. Moran, D. R. Fish, and S. D. Shorvon. The

detection and significance of subtle changes in mixed-signal brain lesions by serial
MRI scan matching and spatial normalization. Medical Image Analysis., 2(3):227,
1998.
[LZZ+ 00] J. A. Lynch, S. Zaim, J. Zhao, C. G. Peterfy, and H. K. Genant. Cartilage segmenta-
tion of 3D MRI scans of the osteoarthritic knee combining user knowledge and active
countours. Proc SPIE, 3979:925, 2000.
[MBBC+ 04] J. Montagnat, F. Bellet, H. Benoit-Cattin, V. Breton, L. Brunie, H. Duque, Y. Legr, I. E.

Magnin, L. Maigne, S. Miguet, J. M. Pierson, L. Seitz, and T. Tweed. Medical images
simulation, storage, and processing on the European DataGrid Testbed. Journal of
Grid Computing, 2(4):387400, December 2004.
[MBK+ 97] C. R. Meyer, J. L. Boes, B. Kim, P. H. Bland, K. R. Zasadny, P. V. Kison, K. Koral,

K. A. Frey, and R. L. Wahl. Demonstration of accuracy and clinical versatility of
mutual information for automatic multimodality image fusion using affine and thin-
plate spline warped geometric deformations. Med Image Anal, 1(3):195206, Apr
1997.
[MBP+ 03] F. M. McQueen, N. Benton, D. Perry, J. Crabbe, E. Robinson, S. Yeoman, L. McLean,

and N. Stewart. Bone edema scored on magnetic resonance imaging scans of the
dominant carpus at presentation predicts radiographic joint damage of the hands and
feet six years later in patients with rheumatoid arthritis. Arthritis and Rheumatism.,
48(7):1814, 2003.
[MBS93] Thomas M. Martinetz, Stanislav G. Berkovich, and Klaus J. Schulten. neural-gas

network for vector quantization and its application to time-series prediction. IEEE
Transactions on Neural Networks, 4(4):558569, July 1993.
Bibliography 175
[MCO+ 99] D. McGonagle, P. G. Conaghan, P. OConnor, W. Gibbon, M. Green, R. Wakefield,

J. Ridgway, and P. Emery. The relationship between synovitis and bone changes in
early untreated rheumatoid arthritis: a controlled magnetic resonance imaging study.
Arthritis Rheum., 42(8):1706, 1999.
[MCV+ 97] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens. Multimodality

image registration by maximization of mutual information. IEEE Trans Med Imaging,
16(2):187198, Apr 1997.
[MG03] Dominik S. Meier and Charles R. G. Guttmann. Time-series analysis of MRI intensity
patterns in multiple sclerosis. NeuroImage, 20(2):11931209, October 2003.
[MHV+ 03] David Mattes, David R Haynor, Hubert Vesselle, Thomas K Lewellen, and William
Eubank. PET-CT image registration in the chest using free-form deformations. IEEE
Trans Med Imaging, 22(1):120128, Jan 2003.
[MLC06] G. Elisabeta Marai, David H Laidlaw, and Joseph J Crisco. Super-resolution registra-
tion using tissue-classified distance fields. IEEE Trans Med Imaging, 25(2):177187,
Feb 2006.
[MLE+ 03] F. McQueen, M. Lassere, J. Edmonds, P. Conaghan, C. Peterfy, P. Bird, P. OConnor,

B. Ejbjerg, M. Klarlund, N. Stewart, P. Emery, R. Shnier, H. Genant, and M. Oster-
gaard. OMERACT rheumatoid arthritis magnetic resonance imaging studies. summary
of OMERACT 6 MR imaging module. J Rheumatol., 30(6):1387, 2003.
[MSC+ 98] F. M. McQueen, N. Stewart, J. Crabbe, E. Robinson, S. Yeoman, P. L. Tan, and

L. McLean. Magnetic resonance imaging of the wrist in early rheumatoid arthritis
reveals a high prevalence of erosions at four months after symptom onset. Annals of
the Rheumatic Diseases, 57(6):3506, 1998.
[MsP+ 05] F. McQueen, M. stergaard, C. Peterfy, M. Lassere, B. Ejbjerg, P. Bird, P. OConnor,

H. Genant, R. Shnier, P. Emery, J. Edmonds, and P. Conaghan. Pitfalls in scoring MR
images of rheumatoid arthritis wrist and metacarpophalangeal joints. Ann Rheum Dis,
64 Suppl 1:i48i55, Feb 2005.
[MV98] J. B. Maintz and M. A. Viergever. A survey of medical image registration. Med Image
Anal, 2(1):136, Mar 1998.
[ndm06] University of Pennsylvania Consortium and National Digital Mammography Archive

Grid (http://www-306.ibm.com/e-business/ondemand/us/innovation/univofpa.shtml),
2006.
[ngs] National Grid Services (http://www.ngs.ac.uk/), 2006.

Bibliography 176
[NMC00] C. P. Neu, R. D. McGovern, and J. J. Crisco. Kinematic accuracy of three surface regis-
tration methods in a three-dimensional wrist bone study. J Biomech Eng, 122(5):528
533, Oct 2000.
[NVB+ 04] Josephine H. Naish, Graham Vincent, Mike Bowes, Manish Kothari, David White,
John C. Waterton, and Chris J. Taylor. A method to monitor local changes in mr
signal intensity in articular cartilage: A potential marker for cartilage degeneration
in osteoarthritis. In Medical Image Computing and Computer-Assisted Intervention
MICCAI 2004: 7th International Conference, volume 3217, pages 959966, January
2004.
[OBD+ 01] S. Ourselin, E. Bardinet, D. Dormont, G. Malandain, A. Roche, N. Ayache, D. Tand,

K. Parain, and J. Yelnik. Fusion of histological sections and MR images: Towards the
construction of an atlas of the human basal ganglia. MICCAI, 2001.
[OEM+ 05] M Ostergaard, J Edmonds, F McQueen, C Peterfy, M Lassere, B Ejbjerg, P Bird,

P Emery, H Genant, and P Conaghan. An introduction to the EULAR-OMERACT
rheumatoid arthritis MRI reference image atlas. Ann Rheum Dis, 64(suppl 1):i37,
2005.
[OHS+ 99] M. Ostergaard, M. Hansen, M. Stoltenberg, P. Gideon, M. Klarlund, K. E. Jensen, and

I. Lorenzen. Magnetic resonance imaging-determined synovial membrane volume as a
marker of disease activity and a predictor of progressive joint destruction in the wrists
of patients with rheumatoid arthritis. Arthritis Rheum., 42(5):918, 1999.
[OHS+ 03] M. Ostergaard, M. Hansen, M. Stoltenberg, K. E. Jensen, M. Szkudlarek, B. Pedersen-

Zbinden, and I. Lorenzen. New radiographic bone erosions in the wrists of patients
with rheumatoid arthritis are detectable with magnetic resonance imaging a median of
two years earlier. Arthritis and Rheumatism., 48(8):2128, 2003.
[OHSL96] M. Ostergaard, M. Hansen, M. Stoltenberg, and I. Lorenzen. Quantitative assessment

of the synovial membrane in the rheumatoid wrist: an easily obtained MRI score re-
flects the synovial volume. British Journal of Rheumatology., 35(10):965, 1996.
[OPC+ 03] Mikkel Ostergaard, Charles Peterfy, Philip Conaghan, Fiona McQueen, Paul Bird,
Bo Ejbjerg, Ron Shnier, Philip OConnor, Mette Klarlund, Paul Emery, Harry Genant,
Marissa Lassere, and John Edmonds. OMERACT Rheumatoid Arthritis Magnetic
Resonance Imaging Studies. Core set of MRI acquisitions, joint pathology definitions,
and the OMERACT RA-MRI scoring system. J Rheumatol, 30(6):13856, Jun 2003.
[Ory03] P. A. Ory. Interpreting radiographic data in rheumatoid arthritis. Annals of the

Rheumatic Diseases., 62(7):597, 2003.
Bibliography 177
[OS03] M. Ostergaard and M. Szkudlarek. Imaging in rheumatoid arthritiswhy MRI and

ultrasonography can no longer be ignored. Scandinavian Journal of Rheumatology.,
32(2):63, 2003.
[Ots79] N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions
on Systems, Man, and Cybernetics, 9(1):62, 1979.
[OVdBVdM96] M. Osteaux, R. Van den Broeck, F. Verhelle, and J. de Mey. Picture archiving and
communication system (PACS): A progressive approach with small systems. European
Journal of Radiology, 22(3):166174, June 1996.
[PDN+ 04] M. Pilavaki, A. Drevelegas, H. Nenopoulou, A. Scordalaki, D. Chourmouzi, I. Sofro-

niadis, and A. S. Dimitriadis. Foci of decreased signal on T2-weighted MR images
in leiomyosarcomas of soft tissue: correlation between mr and histological findings.
Eur.J.Radiol., 51(3):279, 2004.
[PMV03] Josien P W Pluim, J. B Antoine Maintz, and Max A Viergever. Mutual-information-

based registration of medical images: a survey. IEEE Trans Med Imaging, 22(8):986
1004, Aug 2003.
[PTVF92] WH Press, SA Teukolsky, WT Vetterling, and BP Flannery. Numerical Recipes in C:

The Art of Scientific Computation, chapter 10, pages 408425. Cambridge U. Press,
Cambridge, Mass, 1992.
[pvm] PVM: Parallel Virtual Machine (http://www.csm.ornl.gov/pvm/), 2006.
[PXP00] Dzung L. Pham, Chenyang Xu, and Jerry L. Prince. Current methods in medical image
segmentation. Annual Review of Biomedical Engineering, 2(1):315337, 2000.
[RB00] M. M. Rehani and M. Berry. Radiation doses in computed tomography. The increasing
doses of radiation need to be controlled. BMJ, 320(7235):5934, Mar 2000.
[RB04] Anne-Christine Rat and Marie-Christophe Boissier. Rheumatoid arthritis: direct and
indirect costs. Joint Bone Spine, 71(6):518524, November 2004.
[RBH+ 04] AL Rowland, M Burns, T Hartkens, JV Hajnal, and DLG Hill. Information eXtraction
from images (IXI): Image processing workflows using a grid enabled image database.
DiDaMIC Workshop-MICCAI, 2004.
[RBKea00] M.M. Rehani, G. Bongartz, W. Kalender, and et al. Managing X-ray dose in Computed
Tomography. Technical report, ICRP Special Task Force Report, 2000.
[Res95] D. Resnick. Diagnosis of bone and joint disorders. WB Saunders, Philadelphia, 1995.
[RFS03] Daniel Rueckert, Alejandro F Frangi, and Julia A Schnabel. Automatic construction
of 3-D statistical deformation models of the brain using nonrigid registration. IEEE
Trans Med Imaging, 22(8):10141025, Aug 2003.
Bibliography 178
[RMT03] David E Rex, Jeffrey Q Ma, and Arthur W Toga. The LONI Pipeline Processing
Environment. Neuroimage, 19(3):10331048, Jul 2003.
[RPAG03] S. B. Reeder, N. J. Pelc, M. T. Alley, and G. E. Gold. Rapid MR imaging of articular

cartilage with steady-state free precession and multipoint fat-water separation. AJR
Am.J Roentgenol., 180(2):357, 2003.
[RSH+ 99] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. G. Hill, M. O. Leach, and D. J. Hawkes.

Nonrigid registration using free-form deformations: application to breast MR images.
Medical Imaging, IEEE Transactions on, 18(8):712, 1999.
[RW02] A Rajasekar and M Wan. SRB & SRBRack-Components of a Virtual Data

Grid Architecture. In Advanced Simulation Technologies Conference (ASTC02).
metabirn.nbirn.net, 2002.
[RWG+ 01] Jan Rexilius, Simon K. Warfield, Charles R. G. Guttmann, X. Wei, R. Benson, L. Wolf-
son, Martha Elizabeth Shenton, Heinz Handels, and Ron Kikinis. A novel nonrigid
registration algorithm and applications. In MICCAI 01: Proceedings of the 4th Inter-
national Conference on Medical Image Computing and Computer-Assisted Interven-
tion, pages 923931, London, UK, 2001. Springer-Verlag.
[RWM+ 02] Arcot Rajasekar, Michael Wan, Reagan Moore, Arun Jagatheesan, and George
Kremenek. Real-life Experiences with Data Grids: Case Studies in using the SRB.
The 6th International Conferenceon High Performance Computing (HPCAsia-2002),
2002.
[RYS97] D. Resnick, J.S. Yu, and D. Sartoris. Textbook of Rheumatology. Number p.626-86.
WB Saunders, Philadelphia, 1997.
[Sch05] J.H. Schwab. Mechanisms and Models in Rheumatoid Arthritis, chapter 22, pages
431446. Academic Press, 2005.
[SCS+ 01] M. Szkudlarek, Payen Court, C. Strandberg, M. Klarlund, T. Klausen, and M. Oster-
gaard. Power Doppler ultrasonography for assessment of synovitis in the metacar-
pophalangeal joints of patients with rheumatoid arthritis: a comparison with dynamic
magnetic resonance imaging. Arthritis Rheum., 44(9):2018, 2001.
[SD04] Dinggang Shen and Christos Davatzikos. Measuring temporal morphological changes
robustly in brain MR images via 4-dimensional template warping. Neuroimage,
21(4):150817, Apr 2004.
[SEM+ 99] T. Stammberger, F. Eckstein, M. Michaelis, K. H. Englmeier, and M. Reiser. Interob-

server reproducibility of quantitative cartilage measurements: comparison of B-spline
snakes and manual segmentation. Magn Reson.Imaging, 17(7):1033, 1999.
Bibliography 179
[SG03] Simone Santini and Amarnath Gupta. Role of internet images in the biomedical infor-
matics research network. volume 5018, pages 99110. SPIE, 2003.
[sge] The Grid Engine Project (http://gridengine.sunsource.net/), 2006.
[Sha48] C. E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J, 1948.
[SHH97] C. Studholme, D. L. Hill, and D. J. Hawkes. Automated three-dimensional registration

of magnetic resonance and positron emission tomography brain images by multireso-
lution optimization of voxel similarity measures. Med Phys, 24(1):2535, Jan 1997.
[SHWT97] S. Solloway, C. E. Hutchinson, J. C. Waterton, and C. J. Taylor. The use of active shape
models for making thickness measurements of articular cartilage from MR images.
Magn Reson.Med, 37(6):943, 1997.
[Sil86] B. W. Silverman. Density Estimation for Statistics and Data Analysis, chapter 4, pages
7594. Chapman & Hall/CRC, 1986.
[Soi03] P. Soille. Morphological Image Analysis. Springer-Verlag Berlin Heidelberg New

York, 2003.
[SP86] Thomas W. Sederberg and Scott R. Parry. Free-form deformation of solid geometric
models. In SIGGRAPH 86: Proceedings of the 13th annual conference on Computer
graphics and interactive techniques, pages 151160, New York, NY, USA, 1986. ACM
Press.
[SPK06] E. Sakka, A. Prentza, and D. Koutsouris. Classification algorithms for microcalcifica-

tions in mammograms. Oncol Rep, 15 Spec no.:10491055, 2006.
[SRG03] Robert D Stevens, Alan J Robinson, and Carole A Goble. myGrid: personalised bioin-
formatics on the information grid. Bioinformatics, 19 Suppl 1:i302i304, 2003.
[STW+ 02] D. Symmons, G. Turner, R. Webb, P. Asten, E. Barrett, M. Lunt, D. Scott, and A. Sil-
man. The prevalence of rheumatoid arthritis in the united kingdom: new estimates for
a new century. Rheumatology, 41(7):793, 2002.
[SvMDZ01] T. Schormann, M. von Matthey, A. Dabringhaus, and K. Zilles. Alignment of 3-D

brain data sets originating from MR and histology. Bioimaging, 1(2):119, 2001.
[SWDC97] Rick Stevens, Paul Woodward, Tom DeFanti, and Charlie Catlett. From the I-WAY to
the National Technology Grid. Commun. ACM, 40(11):5060, 1997.
[SYS+ 05] S. Sriram, S. Y. Yao, C. Stratton, H. Moses, P. A. Narayana, and J. S. Wolinsky. Pilot
study to examine the effect of antibiotic therapy on MRI outcomes in RRMS. J Neurol
Sci, 234(1-2):8791, Jul 2005.
Bibliography 180
[Tay02] P. C. Taylor. VEGF and imaging of vessels in rheumatoid arthritis. Arthritis Res., 4
Suppl 3:S99, 2002.
[Tay03] Peter C Taylor. The value of sensitive imaging modalities in rheumatoid arthritis.
Arthritis Res Ther, 5(5):210213, 2003.
[TDT05] D.E. Trentham and R. Dynesius-Trentham. Mechanisms and Models in Rheumatoid

Arthritis, chapter 23, pages 447456. Academic Press, 2005.
[TGSG02] B. Taouli, A. Guermazi, K. E. Sack, and H. K. Genant. Imaging of the hand and wrist
in RA. Ann.Rheum.Dis., 61(10):867, 2002.
[THdZ+ 03] Paul M Thompson, Kiralee M Hayashi, Greig de Zubicaray, Andrew L Janke,
Stephen E Rose, James Semple, David Herman, Michael S Hong, Stephanie S Dittmer,
David M Doddrell, and Arthur W Toga. Dynamics of gray matter loss in alzheimers
disease. J Neurosci, 23(3):9941005, Feb 2003.
[TTK77] DE Trentham, AS Townes, and AH Kang. Autoimmunity to type II collagen an exper-

imental model of arthritis. J. Exp. Med., 146(3):857868, September 1977.
[TU00] P. Thevenaz and M. Unser. Optimization of mutual information for multiresolution

image registration. Image Processing, IEEE Transactions on, 9(12):20832099, 2000.
[vdH00] D. M. van der Heijde. Radiographic imaging: the gold standard for assessment of
disease progression in rheumatoid arthritis. Rheumatology (Oxford), 39 Suppl 1:9,
2000.
[vdHSS+ 02] D. van der Heijde, L. Simon, J. Smolen, V. Strand, J. Sharp, M. Boers, F. Breedveld,
M. Weisman, M. Weinblatt, R. Rau, and P. Lipsky. How to report radiographic data
in randomized clinical trials in rheumatoid arthritis: guidelines from a roundtable dis-
cussion. Arthritis Rheum., 47(2):215, 2002.
[vdHvtHvR+ 90] D. M. van der Heijde, M. A. van t Hof, P. L. van Riel, L. A. Theunisse, E. W. Lubberts,
M. A. van Leeuwen, M. H. van Rijswijk, and L. B. van de Putte. Judging disease
activity in clinical practice in rheumatoid arthritis: first step in the development of a
disease activity score. Ann Rheum Dis, 49(11):916920, Nov 1990.
[VdWHB+ 04] H. J. Van der Woude, H. M. Hazelbag, J. L. Bloem, A. H. Taminiau, and P. C. Hogen-
doorn. MRI of adamantinoma of long bones in correlation with histopathology. AJR
Am.J.Roentgenol., 183(6):1737, 2004.
[vhp] The Visible Human Project (http://www.nlm.nih.gov/research/visible/visible human.html).
[VPNG02] S. S. Vasnawala, J. M. Pauly, D. G. Nishimura, and G. E. Gold. MR imaging of knee

cartilage with FEMR. Skeletal Radiology., 31(10):574, 2002.
Bibliography 181
[vtk03] The Visualization ToolKit (http://public.kitware.com/VTK/), 2003.
[WAW+ 83] R. L. Wilder, J. B. Allen, L. M. Wahl, G. B. Calandra, and S. M. Wahl. The patho-
genesis of group A streptococcal cell wall-induced polyarthritis in the rat. Compara-
tive studies in arthritis resistant and susceptible inbred rat strains. Arthritis Rheum,
26(12):14421451, Dec 1983.
[WCG+ 82] R. L. Wilder, G. B. Calandra, A. J. Garvin, K. D. Wright, and C. T. Hansen. Strain and
sex variation in the susceptibility to streptococcal cell wall-induced polyarthritis in the
rat. Arthritis Rheum., 25(9):1064, 1982.
[WGC+ 00] R. J. Wakefield, W. W. Gibbon, P. G. Conaghan, P. OConnor, D. McGonagle,

C. Pease, M. J. Green, D. J. Veale, J. D. Isaacs, and P. Emery. The value of sonography
in the detection of bone erosions in patients with rheumatoid arthritis: a comparison
with conventional radiography. Arthritis Rheum., 43(12):2762, 2000.
[WGR+ 01] Daniel Welti, Guido Gerig, Ernst-Wilhelm Radu, Ludwig Kappos, and Gabor Szekely.
Spatio-temporal segmentation of active multiple sclerosis lesions in serial MRI data.
IPMI, pages 438445, 2001.
[WGW+ 98] R. P. Woods, S. T. Grafton, J. D. Watson, N. L. Sicotte, and J. C. Mazziotta. Automated

image registration: II. Intersubject validation of linear and nonlinear models. J Comput
Assist Tomogr, 22(1):153165, 1998.
[WPB+ 03] M. Wick, P. Peloschek, K. Bogl, W. Graninger, J. S. Smolen, and F. Kainberger. The
X-Ray RheumaCoach software: a novel tool for enhancing the efficacy and accel-
erating radiological quantification in rheumatoid arthritis. Annals of the Rheumatic
Diseases., 62(6):579, 2003.
[WSL+ 04] Jennifer L. Whitwell, Jonathan M. Schott, Emma B. Lewis, David G. MacManus,
and Nick C. Fox. Using nine degrees-of-freedom registration to correct for changes
in voxel size in serial MRI studies. Magnetic Resonance Imaging, 22(7):993999,
September 2004.
[xml] Extensible Markup Language (XML) (http://www.w3.org/XML/), 2006.
[XSD05] Zhong Xue, Dinggang Shen, and Christos Davatzikos. CLASSIC: Consistent Longi-
tudinal Alignment and Segmentation for Serial Image Computing. Lecture Notes in
Computer Science, 3565:101113, July 2005.
[xsl] XSL Transformations (XSLT) (http://www.w3.org/TR/xslt), 2006.
[ZF03] B Zitova and J Flusser. Image registration methods: a survey. Image and Vision
Computing, 21:9771000, 2003.
Bibliography 182
[ZFE02] Alex P Zijdenbos, Reza Forghani, and Alan C Evans. Automatic pipeline analysis
of 3-D MRI data for clinical trials: application to multiple sclerosis. IEEE Trans Med
Imaging, 21(10):12801291, Oct 2002.
[ZWF+ 06] Yong Zhao, Michael Wilde, Ian Foster, Jens Voeckler, James Dobson, Eric Gilbert,
Thomas Jordan, and Elizabeth Quigg. Virtual data Grid middleware services for
data-intensive science. Concurrency and Computation: Practice and Experience,
18(6):595608, 2006.
[ZWL05] J. Zhang, Barry M. Weichman, and Alan J. Lewis. Mechanisms and Models in
Rheumatoid Arthritis, chapter 18, pages 363371. Academic Press, 2005.
Publications
Journal articles
1. K.K. Leung, M. Holden, N. Saeed, K. J. Brooks, J.B. Buckton, A. Williams, S.P. Campbell, K.
Changani, D.G. Reid, Y. Zhao, M. Wilde, D. Rueckert, J.V. Hajnal, and D.L. Hill. Automatic
quantification of changes in bone in serial MR images of joints, Medical Imaging, IEEE Transac-
tions on, 25, pages 161726, 2006.
Conference proceedings
1. R.A. Heckemann, T. Hartkens, K. Leung, D.L.G. Hill, J.V. Hajnal, D. Rueckert. Information
Extraction from Medical Images (IXI): Developing an e-Science Application Based on the Globus
Toolkit, Proceedings of the UK e-Science All Hands Meeting 2003, Nottingham, UK, 2003.
2. K.K Leung, M. Holden, R.A. Heckmann, N. Saeed, K.J. Brooks,J.B. Buckton, K. Changani, D.G.
Reid, D. .Rueckert, J.V. Hajnal, and D.L. Hill. Analysis of serial MR images of joints, Analysis of
serial MR images of joints IEEE International Symposium on Biomedical Imaging (ISBI 2004), 1,
pages 221224, 2004.
3. K.K Leung, M. Holden, R.A. Heckmann, N. Saeed, K.J. Brooks,J.B. Buckton, K. Changani, D.G.
Reid, D. .Rueckert, J.V. Hajnal, and D.L. Hill. Use of Data Provenance and the Grid in Medical
Image Analysis and Drug Discovery - An IXI Exemplar, Proceedings of the UK e-Science All
Hands Meeting 2004, Nottingham, UK, 2004.
4. M. Burns, A.L. Rowland, D. Rueckert, J.V. Hajnal, K. Leung, D.L.G. Hill, and J. Vickers. Infor-
mation eXtraction from Images (IXI): Grid Services for Medical Imaging, DiDaMIC Workshop -
MICCAI 2004, Rennes, France, 2004.
5. K. Leung, M. Holden, N. Saeed, K. Brooks, J. Buckton, A. Williams, S.P. Campbell, K. Changani,

D. Reid, D. Rueckert, J. V. Hajnal, and D.L.G. Hill. Automated quantitative analysis of rheuma-
toid arthritis lesions in MRI, Proceedings of the International Society of Magnetic Resonance in
Medicine, Miami, US, 2005.
6. K. Leung, N. Saeed, S.P. Campbell, K. Changani, and D.L.G. Hill. Spatio-temporal segmentation
of rheumatoid arthritis lesions in serial MR images of joints, MMBIA 2006: IEEE Computer
Society Workshop on Mathematical Methods in Biomedical Image Analysis, New York, US, 2006.

MRI Images in RA

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

MRI Images in RA

Hochgeladen von

Copyright:

Verfügbare Formate

Longitudinal analysis of MRI images in

Kelvin Ka-fai Leung

A dissertation submitted in partial fulfillment

Centre for Medical Image Computing

November 16, 2007

2.6 Animal Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3 Automatic Quantification of Changes in Longitudinal MR Images of Joints I 46

4 Automatic Quantification of Changes in Longitudinal MR Images of Joints II 77

4.5 Materials and methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5 Application to Clinical Data 96

6 Use of the Grid in Automated Medical Image Analysis 121

A Appendix of Chapter 3 154

B Appendix of Chapter 4 160

1.1 Joint affected by rheumatoid arthritis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.1 Intensity-based image registration algorithm . . . . . . . . . . . . . . . . . . . . . . . . 30

3.1 Principle of segmentation propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.1 Example bone localisation scores from 0 to 5. . . . . . . . . . . . . . . . . . . . . . . . 100

6.1 The architecture of the web-based image analysis workbench. . . . . . . . . . . . . . . 129

7.1 Results of rigid registration of longitudinal MR brain images of MS patients. . . . . . . 150

A.1 Example demonstrating the effects of the morphological operations . . . . . . . . . . . . 155

1.1 Background and motivation

Figure 1.1: Joint affected by rheumatoid arthritis. Excerpted from http://www.cchs.net/health/health-

1 TheCritical Path to New Medical Products (http://www.fda.gov/oc/initiatives/criticalpath/initiative.html) March 2006

/imagingKeyNote.ppt) G. Mills, May 2005

1.2 Image registration

1.3 Thesis aims

1.4 Thesis organisation

1.5 Software and other materials

1.6.2 Individual chapters

Demonstration that segmentation propagation and spatio-temporal segmentation have potential in

medical image data

image analysis software

2.2 Imaging of joints in RA

2.2.1 Conventional radiography (CR)

disability, which is most marked in late RA [BTF+ 94].

2.2.2 Microfocal x-ray

2.2.3 Computed Tomography (CT)

2.2.4 Bone scintigraphy

2.2.5 Magnetic resonance imaging (MRI)

2.2.6 Ultrasound (US)

2.3 Measuring changes in longitudinal images

2.3.2 Change in volume

2.3.3 Subtraction after registration and the boundary shift integral

2.4 Image registration

2.4.1 The feature-based image registration algorithm

The landmark-based registration algorithm

The surface-based registration algorithm

2.4.2 The intensity-based image registration algorithm

FBA = {xB FB xB FA }. (2.1)

Correlation coefficient (CC) comes from cross correlation, and is defined as

Fb = {x FBA B(x) b}. (2.7)

H(A) + H(B) I(A, B)

Image interpolation and transformation

where i = bx/c 1, j = by/c 1, k = bz/c 1, u = x/ bx/c, v = y/ by/c,

B0 (u) = (1 u)3 /6 (2.19)

B1 (u) = (3u3 6u2 + 4)/6 (2.20)

B2 (u) = (3u3 + 3u2 + 3u + 1)/6 (2.21)

B3 (u) = u3 /6. (2.22)

2.5 Supervised and unsupervised classifications

2.5.1 Unsupervised classification (or data clustering)

(a) The collection of patterns. (b) The dendrogram as a result of hierarchical

Figure 2.2: Hierarchical clustering.

1. Each pattern is placed in its own cluster.

3. The two most similar clusters are joined together.

B (X) = {x|Bx X}. (3.7)