BAG OF SEGMENTS FOR MOTION TRAJECTORY ANALYSIS Yue Zhou and Thomas S.

Huang University of Illinois at Urbana-Champaign, Urbana, IL, 61801 USA

ABSTRACT


We propose a novel method to represent motion trajectory bag of segments. Motivated by the bag of words approach in text mining and more recently in image categorization, we represent each trajectory as a collection of segments, each assigned a membership to a codeword of a dictionary. The trajectory segments are represented by their shape and motion information while the inter-segment relationship is ignored. In the bag of trajectory segments representation, the trajectories are transformed to a vector in the codeword space. The codebook is generated by clustering a large amount of trajectory segments using Expectation Maximization (EM) algorithm and each codeword is a weighted regression model. Given the codebook, trajectories are segmented and associated with a soft-frequency vector in codeword space as an intermediate representation. With the bag of segments representation, trajectory analysis can be carried out in vector space as traditional data. Experiments show the bag of segments method is very effective in trajectory similarity search and classication. Index Terms Motion analysis, Pattern classication, Information retrieval 1. INTRODUCTION With the rapid advances of object detection and visual tracking technologies [1] [2] [3], it has become possible to trace the trajectories of moving objects in the videos. Due to the accumulation of spatial-temporal tracking data, it is increasingly important to perform data analysis tasks over trajectories obtained from visual tracking algorithms. A key consideration of it is a robust and accurate method to represent trajectories. Previous research [4] [5] mostly relies on raw representation in analyzing the motion trajectory. The problem with it is that the raw representation is a time series and normal machine learning algorithm in vector space can not be directly applied to it. In this paper we propose a novel method to represent motion trajectory - Bag of Segments. Similar models have been successfully used in the text community for analyzing documents and are known as bag-of-words models [6], since each document is represented by a distribution over xed vocabulary.

Fig. 1. The codebook formation by clustering. Our key observation is that representing a trajectory as a whole is not appropriate for detecting similar patterns of the trajectory. Each trajectory may have different lengths and complicated patterns. Therefore, even though some key portion of the trajectory share similar patterns, the whole trajectories may be different. Our approach is to partition the trajectories into a set of trajectory segments and group the segments into clusters. Each cluster represents a codeword in the codebook of the trajectory segments. The codebook can be then used to represent any new trajectories to give a density distribution with respect to each codeword. 2. FORMING THE CODEBOOK OF TRAJECTORY SEGMENTS In this section, we discuss the segment-and-cluster method to form the codebook. Figure 1 shows the skeleton of the codebook formation. It consists two phases: 1. In the segmentation phase, each trajectory is partitioned to a set of trajectory segments. These segments are provided to the next phase. 2. In the clustering phase, a clustering algorithm is performed to form the codewords.

978-1-4244-1764-3/08/$25.00 2008 IEEE

757

ICIP 2008

2.1. Trajectory parameterization and segmentation In this section, we present the algorithm to parameterize and segment trajectories. A trajectory is a sequence of multi-dimensional points. The ith trajectory is denoted as T Ri = y1 y2 y3 ...yj ...yni . Here, yj (1 j ni ) is a d-dimensional point. The length leni of a trajectory can be different from those of other trajectories. A trajectory pc1 , pc2 ...pck (1 c1 < c2 < ... < ck ni ) is called a trajectory segment of T Ri . We assume that each observed trajectory T R in y space is a smooth function of a known variable X . For instance, X could be the time index when the measurement is performed. It is a model-based approach where each trajectory is modelled as a function with some variability associated with it. Formally, we dene the regression relationship between X and T R as: T Ri = g (X ) + e (1)

the j th measurement of SEGi as yi (j ) taken at time xi (j ), Xi = (xi (1), xi (2), ..., xi (ni )) and X = (X1 , X2 , ..., XM ) We assume the observed trajectory segments are generated by the following mixture of regression model: 1. A regression model is selected with probability wk K from a pool of K regression functions. k=1 wk = 1. 2. An observed trajectory segment SEGi is generated with probability pk (SEGi |Xi , k ). k is the parameters of regression model k . Given the observed dataset {SEGi }, and a particular functional form of pk (SEGi |Xi , k ), we use Expectation Maximization (EM) [7] algorithm to nd the maximum likelihood estimate of the parameters of the mixture model. The log-likelihood of the trajectory segment set is:
K

log p(S |X, ) =

i

log(
k=1

wk pk (SEGi |Xi , k ))

(3)

where e is zero-mean Gaussian with standard deviation 2 and g (X ) is a deterministic function of X . Because we assume the noise is a Gaussian, we have that the likelihood probability p(T Ri |X, ) has mean g (X ) and standard deviation 2 . denotes the model parameters of g (X ) and . Formally, we dene the log-likelihood of the trajectory as: log p(T Ri |X, ) = log p(y1 y2 y3 ...yni |x1 x2 x3 ...xni , ) 1 log N (yj |g (xj ), ) (2) = ni
1j ni

In Equation 3, there is a hidden variable that corresponds to the unknown membership of each trajectory segments with respect to the regression models. The estimation problem will be simple if the value of the hidden variables are known. Let C be the matrix of membership variables such that ci,k = 1 for one 1 k K and ci,t = 0 for any t = k . If ci,k = 1, the ith trajectory segment is generated from the k th regression function and vice versa. With the new hidden variable, the log-likelihood becomes:
K

log p(S, C |X, ) =

i K

(
k=1

ci,k log wk + (4)

where N (yj |g (xj ), ) is the Gaussian function with mean g (xi ) and standard deviation . Here we normalize the loglikelihood with the length of the trajectory ni so that the loglikelihood of trajectories with different lengths are comparable with each other. We use a greedy partition algorithm which start from the rst measurement of the trajectory and sequentially nd the segmenting points of the trajectory. The criterion to split one segment is that the log-likelihood of regression function given the observed segment is smaller than a pre-set threshold. minLogLikelihood and minLength are two parameters that represent the minimum log-likelihood of one segment and the minimum segment length respectively. 2.2. Clustering the trajectory segments by the mixture of regression Let the trajectory segments dataset S consist M trajectory segments SEGi where 1 i M . We denote the length of SEGi as ni . In this work, we assume the trajectory segment is a smooth polynomial function of time index starting from 1 although it is not strictly necessary. We denote

ci,k log pk (SEGi |Xi , k ))

k=1

The EM algorithm consists two steps: E-step and M-step as follows: 1. E-step The expected value of the log-likelihood function in Equation 4 with respect to p(C |S, X, t1 ) - the conditional probability of C given the previous estimation of the model parameters. Here t1 is the parameters estimated at time t 1. 2. M-step The expectation is maximized with respect to the model parameter to compute the new estimation t . The expectation of Equation 4 can be computed as:
K

E [log p(S, C |X, )] =

i K

(
k=1

hi,k log wk +

hi,k log pk (SEGi |Xi , k )) (5)

k=1

758

where hi,k = E [ci,k ] = p(ci,k = 1|SEGi , Xi , t1 )

t1 wk p(SEGi |Xi , k )

(6)

In this work, we assume the regression function is a pth order polynomial function of X . The equation for the k th regression function is dened as follows:
p k + ek SEGi = Xi

(7) Fig. 2. The top 12 clusters.

where SEGi = (yi (1), ..., yi (ni )) and 1 xi (1) xi (1)2 ... xi (1)p 1 xi (2) xi (2)2 ... xi (2)p p . . ... . Xi = . . . . ... . 1 xi (ni ) xi (ni )2 ... xi (ni )p

2 ek is a zero-mean Gaussian with variance k . k = (k (1), k (2), ..., k (p)) is the linear regression coefcients. 2 ) are updated in the MThe parameters k = (wk , k , k step as follows: pT Hk X p )1 X pT Hk S k = (X T p (S X p k ) Hk (S X k ) M i=1 M

(8) (9) (10)

k =

hi,k h( i, k)

w k =

1 M

i=1 hi,k ni

(a) Query

(b) Top 4 similar trajectory

1,k , h 2,k , ..., h M,k ) and h i,k = In where Hk = diag (h i Ini is a identity matrix of size ni . S = (SEG1 , ..., SEGM ) p p p and X p = (X1 , X2 , ..., XM ). After the EM clustering, we further prune out the clusters with too small weights to nalize the codebook. Figure 2 shows the top 12 clusters obtained from the EM clustering algorithm. 3. BAG OF SEGMENTS APPROACH FOR TRAJECTORY ANALYSIS 3.1. Representing trajectories by bag of segments With the codebook at hand, we are ready to represent trajectory as a bag of segments. Each trajectory is regarded as a collection of trajectory segments. The segments are described by their shape and velocity while their inter-relationship are largely ignored. The ownership p(SEG|Xk , k ) is computed by the mixture of regression model as:
2 , wk ) p(SEG|Xk , k ) = p(SEG|Xk , k , k

Fig. 3. Trajectory similarity search by trajectory example

(a) Query

(b) Top 4 similar trajectory

2 ) p(SEG|Xk , k , k

wk

(11)

Fig. 4. Trajectory similarity search by partial trajectory example

759

Walking Running Mixed

Walking 86.4% 8.3% 7.1%

Running 13.6% 83.3% 0%

Mixed 0% 8.3% 92.9%

4. CONCLUSION We present the bag of segments representation for motion trajectory. It has three advantages over the traditional raw data measurements representation: rst, it represents trajectories with different lengths under a unied framework in codeword space probabilistically. Any standard machine learning algorithms for vector space can be applied to trajectory using this representation. Second, experiments show that bag of segments representation for trajectories is very compact and expressive in the task of similarity search and classication. Third, trajectories with the bag of segments representation are very easy to compare. Therefore it can be used for fast indexing and searching in trajectory database. Future work on bag of segments representation for trajectory may include a model for high-level context information of trajectory, and other algorithms to form the codebook. 5. ACKNOWLEDGEMENTS This research was funded by the U. S. Government VACE program. 6. REFERENCES [1] Dorin Comaniciu, Visvanathan Ramesh, and Peter Meer, Kernel-based object tracking, IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 5, pp. 564575, 2003. [2] M. Isard and A. Blake, Condensation : conditional density propagation for visual tracking, International Journal of Computer Vision, vol. 29, no. 1, pp. 528, 1998. [3] Paul Viola and Michael J. Jones, Robust real-time face detection, Int. J. Comput. Vision, vol. 57, no. 2, pp. 137 154, 2004. [4] Chris Stauffer and W. Eric L. Grimson, Learning patterns of activity using real-time tracking, IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 747757, 2000. [5] Namrata Vaswani, Amit Roy Chowdhury, and Rama Chellappa, Activity recognition using the dynamics of the conguration of interacting objects, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 02, pp. 633, 2003. [6] Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 5057, 1999. [7] A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society series B, vol. 39, pp. 138, 1977.

Table 1. The classication result using bag of segments representation

3.2. Trajectory similarity search We apply the bag of segments method for fast trajectory retrieval. The database we used for the experiments was collected at the indoor environment with a still camera. The camera (Sony 908E, 30f/s) was installed in the second oor of a building, and captures the hall area in the rst oor from the top-view. The database consists 51 video sequences with 116 trajectories. To estimate the trajectories of the two moving objects in the videos, we use the mean-shift tracking algorithm [1], and the tracker is automatically initialized from the Gaussian Mixture Models (GMM) based background subtraction algorithm. We use 1 2 of the data to form the codebook of trajectory segments. The codebook includes 30 codewords which corresponds to the top 30 clusters out of 45. The rest 1 2 data is used for testing. The query is given by a randomly selected trajectory or a portion of trajectory. The query is then converted to density distribution and compared with the trajectories in the database in the codeword space. The nearest-k trajectories are retrieved as the query result. In experiments, we use Euclidean distance to compare the trajectory in codeword space. Figure 3 shows an example of trajectory similarity search. The left plot is the query trajectory as an input to the similarity search algorithm. The right plot are the top 4 similar trajectories in the database. Figure 4 shows a similar example with partial trajectory as query data.

3.3. Trajectory classication We also test the bag of segments method for trajectory classication using nearest neighbor classier. 1 2 of the data is used to form the codebook of trajectory segments. 3 8 of the data is used to train the classier and 1 8 of the data is used for testing. The dataset we use is the same as in trajectory similarity search experiment. The data is divided into 3 classes: (1) people walking, (2) people running, (3) mixed - people walking, running and stopping. Table 1 demonstrates the experimental result of classication using bag of segments representation. The overall classication precision is 89.5%, which proves the effectiveness of bag of segments.

760