CMPE 699 Project:
Tracking Human Motion by using Motion Capture Data Işık Barış Fidaner
Introduction Tracking an active human body and understanding the nature of his/her activity on a video is a very complex problem. Psychological experiments show that human subjects can easily track and extract several variables from a video, including what the person is doing, the gender of the person, the emotional status, the identity of the person, and even if the person is oneself; all from a little amount of dynamic information. This is by means of the biological vision and cognition system that include inborn mechanisms for emphatizing with other humans and also adaptively incorporate several years of experience. Therefore, to build intelligent systems ourselves, we must find a way to algorithmically incorporate the information about the physical nature of the human body and the dynamical structure of different human activities. Human motion analysis has been studied in several fields for several applications, and motion capture data is one of the main sources of material in these studies. Motion capture technologies allow recording of human motion in terms of time sequences of exact body configurations. Open motion capture databases contain time sequences of body poses, such as CMU database [1] that include 2605 sequences captured from 144 people. By using this data, human actions like walking, running, sitting and standing can be mathematically analysed to improve our algorithms for tracking human motion. Extracting information from human body motion can help several application fields, including: • • •
Human Body Tracking, for improving the accuracy in model-fitting by eliminating unplausible body poses. Skeletal Animation, for believably blending between key-frame animation sequences captured seperately. Motion Capture Processing, for automatically segmenting complex MoCap sequences into the individual actions.
A basic problem in dealing with motion capture data is the high dimensionality and redundancy of the information. Motion data represents body pose in terms of large vectors. For example, in a CMU capture, for each time step, there is a 62-dimensional vector that place 29 joints of a human
body in 3D space. The captures include the angle of thumb and toe, as well as the angle of knee and elbow. Some of these angles are not relevant in detecting human activity. Moreover, in several human activities, most of these angles have a high degree of correlation, which reveals the redundancy in the raw motion capture data. Reduction of this dimensionality may be helpful in all related fields, because (a) processing less dimensional data needs smaller time-space complexity (compression). With same amount of resources, we can store and use the representation of a larger dataset. Moreover, (b) we know that the observed complexity of human motion is, in fact, a result of an inner neural mechanism that is can flexibly learn sets of very complex motion patterns. Reducing dimensionality can be used to construct models that can simulate the behaviour of human motor system. Principal Components Analysis (PCA) is a well-known mathematical method that is used to reduce dimensionality of data sets. In some studies, PCA is applied to motion capture data to segment it into single actions [2], [3] and for quantifying the degrees of freedom in skeletomuscular systems [4]. Also, PCA is compared to different methods such as Probabilistic PCA [3] or Pointwise Dimension Estimation [4]. In this project, I intend to examine the basics that are necessary for a human tracker that uses motion capture data to be implemented. I also investigate human motion analysis, by considering an example motion type: walking, an important motion type that is also periodic and regular. By reducing its dimension and comparing walk captures of different people, one can discriminate the general properties that define the motion itself from particular properties caused by personal style of walking. This project involves, • Applying PCA to analyse human walking motion • Investigating ways to construct a model that can be used for tracking walking people by a single fixed camera. Previous Work Dynamic analysis of human motion has been studied for a long time. An important motivation for analysing human motion data is a set of psychological experiments conducted by Gunnar Johansson in 1976. He attached light sources to a few points on a human body, and filmed different actions that showed only moving light sources. Despite the reduced information, the human in the film was easily recognized. This experiment demonstrated the huge amount of information encoded in the dynamics of human movement. Later on, it was also shown that gender, emotion and identity of the person was recognizable from these few moving light sources. One could tell men from women only by looking at moving light sources with the rate of 63%. An interesting finding was that when the film was shown upside down, the subjects' tendency was inversed, they thought men were women and women were men with a statistical difference from pure chance. Then, certain small features were considered like shoulder distance, arm swinging and torso movement that helped distinguish men from women. However, none of these features could explain the statistical significance, and if the naturalness of the motion was distorted, none of these features helped recognition, implying a more holistic biological recognition mechanism. In [5], Troje reviews these psychological findings and builds an algorithm that can distinguish men from women by analysing captured data of walking motion, represented by moving joints, similar to the moving light sources. He captured walking motions of 20 men and 20 women, and firstly
applied PCA to each of the motion sequences to get an "eigenposture" representation of a walking sequence. This representation was composed of a mean posture and four eigenvectors with periodic time series, similar to the implementation in this project. In the second step, this new representations were put through a second PCA operation to obtain the "eigenwalker" representation that allowed distinguishing between different styles of gait. In [6], another two stage PCA is applied, but with different parameters. Contrary to Troja's work, where only 3D positions of the joints were used, Das et al. included 3D angles between joints and also angular velocities in their analysis. They also focused the data on the legs and only recorded 6 components that included angles and angular velocities of thighs and knees. In the first stage of PCA, they obtained two components one of which could distinguish swing and pivot phases of the gait, and the other captured knee flexions inside these phases. For the second stage, they considered the gait cycles captured in these two components as a single data sample, and with a second PCA operation, managed to distinguish between different types of gait cycles like walking, running, jogging with high accuracy. To use the motion capture in tracking, we need the following: • A 3D dynamical model of human body, that is transformed to a different state space with meaningful axes directions, such as the nature of activity, and current phase in that activity. This model is constructed by the help of motion capture data carefully selected, aligned and analyzed to incorporate meaningful activity knowledge in our dynamical model. • A sequence of images captured in real-time. These can be human silhouette for single camera applications, or they can be point clouds captured with stereo camera computer vision systems. Typically, the real-time data contains several errors that need to be filtered by the previous knowledge of our dynamical model. • A method to compare the model and the images. This can either be probabilistic as in a generative model or a particle filter, or it can be a deterministic procedure such as the optimization of a differential objective function. In [7], a data driven probabilistic method is described for probabilistic object tracking, which is based on MCMC (Markov Chain Monte Carlo) methods. The high dimensionality of the scene space is divided into smaller structures with different properties, the object spaces and furthermore, the atomic spaces. In other words, an object space is the cartesian product of several atomic spaces, i.e. the combination of several atomic features such as edges and colours; and a scene space is the cartesian product of the object spaces that correspond to each of the objects present in the scene, each space having a distinct structure and dimensionality, related to the object class it represents. In addition to the object spaces, the scene space includes a color space, and a mapping of the image pixels to each of the objects. Finally, a Markov Chain is defined that transforms particles in the scene space in every time step. The transitions include transforming objects by themselves, destroying, creating, merging and dividing objects, and in every transition, the dimensionality of the scene space is subject to change. The requirements for an MCMC system include ergodicity and aperiodicity. Also the converse transitions like object birth/death must be balanced and reversible. Every transition probability is the combination of two probabilities, which are associated to choosing the nature of the transition (object birth or death etc.) and how this transition is applied (which object dies etc.). For each transition, an acceptance probability is calculated by the Metropolis-Hastings method to balance the combined effect of these probabilities. When updating the particle set, the new scene particles are selected so that the new distribution gives the least Kullback-Leibler divergence from the actual distribution. This measure prevents the high-dimensional particle set to be trapped in a single maximum and requires it to represent the multimodal structure of the actual distribution. A model-based pose estimation based
on data driven MCMC to is presented in [8]. This method builds a hierarchical structure that relates overall properties such as number of objects, to local image features that are used to detect individual body parts. Human body pose, shape and clothing are modeled separately and incorporated in the same probabilistic model that is balanced among different types of transitions, where each transition updates a single feature at a time. As a result, the algorithm is multi-layered and heterogeneous in nature. In [9], a deterministic body tracking method is presented, in contrast to the probabilistic approaches dominant in the field. A scalar field function is defined on the skeleton model by summing up elliptic distance functions of every body part. Human skin is defined as a level set surface of this field function. 3D point clouds of tracked people are compared to the skin surface by calculating the distances of each 3D point to the skin of the closest body part. This yields a differentiable objective function that can be directly optimized by gradient descent or similar methods. However, the hierarchical body model is very complex (composed of a global positionorientation and several joint angles), and an additional human activity knowledge must be incorporated by making use of motion capture data. For this purpose, they captured motions of some subjects walking and running in different speeds, and transformed the motion space to a less dimensional space by applying PCA over these data. The PCA is applied to the sample set of all motions involved, similar to the “eigenwalkers� introduced by Troje. To extract both phase and style related information, different periodic motions are aligned in a fixed time interval before applying PCA. As the phases of gait cycles are aligned in time, they could simulate transition from walking to running motion by modifying the components in the reduced motion space. After PCA, the differentiation of the objective function is applied to the reduced components that encode the style and type of the motion (walking, running, slow, fast etc.), and the current phase in the gait cycle (pivot or swing phase etc). To differentiate the objective function in the transformed motion space, its Jacobian is calculated with respect to the eigenvectors. The advantages of this deterministic method include the less computational cost required, and the specialized motion model that can capture holistic correlations involved in distinct human activities. However, the application is limited to periodic locomotion activities and a single person. Motion Capture format We used ASF/AMC motion capture files from Carnegie Mellon University Motion Capture database. It is a large database that includes thousands of motion files, available at [1]. ASF is Acclaim skeleton file, that contains the hierarchial skeletal structure of the subject, including initial bone angles and bone lengths. AMC is Acclaim motion capture file that contains the orientations of every bone in every axis in every time step of the motion. Details can be found in [10]. The motion data consists of changing angles of the bones in the skeleton. Every bone has different degrees of freedom in different axes. The bones of the skeleton are shown below. Tx signifies translation along x axis and Rx signifies rotation around x axis. Forward walking motions of five different subjects were used in the PCA experiments. For motion analysis, we use the angular representation of motion that is embedded in the motion capture files. We modify the angles in a preprocessing step where we weigh them by approximating the arc length they affect. It is also possible to numerically calculate and incorporate angular velocities and accelerations in every frame of motion sequences, but not included in the project.
Degrees of freedom of bones: Root: Tx Ty Tz Rx Ry Rz Head, Upperneck, Lowerneck, Thorax, Upperback, Lowerback: Each has Rx Ry Rz (Left & Right) Clavicle: Ry Rz Humerus: Rx Ry Rz Radius: Rx Hipjoint: None Wrist: Ry Femur: Rx Ry Rz Hand: Rx Rz Tibia: Rx Thumb: Rx Rz Foot: Rx Rz Fingers: Rx Toes: Rx Figure 1. The skeleton model used in the motion capture data
Projecting the 3D model We are going to examine some practical methods that can help human motion tracking by the help of a skeletal model. These do not constitute a general tracking approach, but they can be used in different schemes and frameworks. Thus, we did not conduct a systematic set of experiments by using these methods. To compare the captured images to the 3D body model, firstly, we have to convert the body model from joint angles to 3D positions of every joint. As we assume that the camera is fixed, there is a certain camera calibration matrix that can be used to convert 3D joint positions of the skeletal model to the 2D image positions. This matrix applies perspective projection to the skeleton, so we can use it to superimpose our model on any place in the tracked area. Moreover, if we have multiple cameras, we can superimpose the same 3D motion to images captured from all of these cameras. On the figure below, a walking 3D model is superimposed on several views captured by multiple cameras. The calibration matrix and the images are from the video database of INRIA perception group [11].
Figure 2. The skeleton superimposed on camera images based on calibration matrices
Estimating the model position The matrix transforms 3D points (X, 0, Z) on the plane Y=0 to the points that appear to be on the ground. Thus, we can reduce the 6-dimensional global position-orientation of our model to only three dimensions that encode the position of the body on the ground plane (X, Z) and which direction it is heading (Θ).
Θ
(X, Z)
Figure 3. a) 2D circular trajectory of the walking motion obtained by tracking the person’s silhouette. Red, green, blue lines denote X, Y, Z axis directions defined by camera calibration matrix. b) XZ plot of the 3D trajectory obtained by “unprojecting” the 2D points, i.e. intersecting the camera rays of each of 2D trajectory positions with the 3D ground plane Y=0.
If we approximate a 2D image point for the the ground base below the person (for example, position of the feet from a side-view silhouette), then we can “unproject” that 2D point to 3D by the assumption that the point is on the ground plane (Y=0). In this way, we can approximate X and Z at every time instant. These positions draw the 3D trajectory of the walking motion. If we assume that the subject is moving forward, we can approximate Θ as the slope of the line tangent to the 3D trajectory, in direction of velocity (vX,vZ) in every time interval. Moreover, we can integrate the distance travelled with respect to time, and obtain the actual distance travelled by the person, which can be used to update our 3D model, making it travel the same distance. However, these variables are subject to noise, so they must not be used directly, but they can be filtered with a probabilistic model, or used as the initial values in an algorithm that optimizes an objective function. To use these methods for tracking, we need to define a feasible human body posture model that also incorporates motion capture data to assure plausibility. Therefore, in the following sections, we are going to examine a way to represent the body poses in a less dimensional and more meaningful space.
Principal Component Analysis PCA is a mathematical method that is applied to an N-dimensional data set. PCA is used to linearly transform this dataset to an M-dimensional set where only M<N components are necessary to represent most of the data variance. In our experiments, we have 62-dimensional walking data, and we used PCA to transform it to 4-dimensional space. This involves transforming the data using four 62-dimensional eigenvectors. The representation coverages of these four components are represented by four eigenvalues, which are similar to the “eigenpostures� described by Troje. Before using our motion data, we have to modify it so that equal changes in values have equivalent effects in the motion. This is necessary, because PCA correlates these components based on their effects. Our original data includes both toe angles and knee angles. We have to modify them such that toe angle has less effect on our analysis than knee angle. For this purpose, we multiplied angle values by total length of the bones that it rotates. This value approximates the length of the arc created by the motion. For example, toe angles are multiplied only by toe length, whereas femur angles are multiplied by the total length of femur, tibia, foot and toe. Without this preprocessing, we could cover 99% of the variance by first 12 eigenvectors, whereas with preprocessing, only 4 eigenvectors were necessary, and these four components captured qualitatively different features. Experiments In the experiments, we used walking motions of five subjects, #07, #08, #35, #37 and #38 as numbered in the CMU database. We are going to refer these as subject A, B, C, D, E. The data is first preprocessed and PCA is applied to get four eigenvectors and eigenvalues. Our experiments show that first four components of the PCA cover most of the variance, and fifth eigenvalue is significantly smaller (less than half of the fourth eigenvalue). The components are referred as c1, c2, c3 and c4. As the coverages corresponding to different eigenvectors in all five subjects on the table show; c1 and c2 roughly forms the motion pattern, while c3 and c4 fine tunes the movement details. Total percentage is calculated by dividing sum of first four eigenvalues by the sum of all eigenvalues.
c1 c2 c3 c4 Total
A 68 22 6 2 98%
B 63 28 4 2 97%
C 85 10 2 2 99%
D 78 14 3 3 99%
E 73 20 4 3 99%
In all of the experiments, first component (c1) represents the global position change as a result of walking, relatively independent from body movements. This is an aperiodical component. A typical change of c1 in time is shown in graph. This behaviour is similar for all subjects:
When c1 is minimum, body is at starting position When c1 is maximum, body is at the ending position Observed on subject A, B, C, D, E c1 change of subject A
The second component (c2) is periodical and it represents the limb orientation from root. This graph is also similar in all subjects with changes only in period and range. When c2 is minimum, left leg and right arm put forward, right leg and left arm put backward When c2 is maximum, left leg and right arm put backward, right leg and left arm put forward c2 change of subject A
Observed on subject A, B, C, D, E
Third component (c3) usually represents contrary limb movements at elbows and knees in certain phases of the movement. This component is also periodic but has less ordered behaviour. C3 is different among subjects and it could carry information about walking style. To check this feature, we compared c3 plots from 5 different walking sessions with the same subject C, and they were similar. Below are c3 plots from different subjects: When c3 is minimum, left knee flexes, right knee extends and arms move inversely to c2 When c3 is maximum, left knee extends, right knee flexes and arms move inversely to c2 c3 change of subject A
Observed on subject A; on B with smaller effect, on C, D and E with some asymmetry
...c3 is minimum hip leans right
...c3 is minimum body leans right
...c3 is maximum hip leans left
...c3 is maximum body leans left
Observed on subject A
Observed on subject B
c3 change of subject D
The fourth component (c4) usually covers the physical, gravity-related aspects of movement. When we plot, we see a periodic function with two peaks on every period. This corresponds to the springing movement. While walking, we hop two times between the times we put our left foot on the ground. The plots are similar among subjects, except heights of the first and second hops, and sometimes minimum and maximum are reversed.
When c4 is minimum, body slightly raised and leaned back, knees flexed, legs lean slightly forward When c4 is maximum, body normal, knees extended, legs flat, foot on the ground c4 change of subject A
Observed on subject A, B, C, on D with some asymmetry, and on E
When c4 is minimum, feet closer to each other
When c4 is minimum, arms slightly flexed
When c4 is maximum, feet farther from each other
When c4 is maximum, arms slightly extended
Observed on subject A, B
Observed on subject A, B, C
When we plot for each subject, change in component 2 with 3 on a plane, we can see the shape of the gait cycle in a reduced PCA space. In the following plots, c3 is on the horizontal and c2 is on the vertical axis. We can see the similar pattern in all walking subjects.
Subject A
Subject B
Subject C
right knee extended (c3) left knee extended right leg extended (c2) Subject D
Subject E
left leg extended
The experiments show that if investigated closely, PCA analysis of walking motion reveals some characteristic features of the motion itself. The periodic and regular nature of walking also helps the analysis. Motion Exaggeration A direct application of the PCA analysis is the exaggeration of the captured human motion for animation purposes. After the PCA transformation, we can increase the intensity of a selected component, decrease others, and use this modified sequence to recover the motion in original space of joint angles. This operation results in a motion sequence that involved exaggerated motion patterns. For example, if we increase c2, the legs and arms are swinged more; if we increase c3, the knees are flexed-extended more; and if we increase c4, the model walks by springing and excessively extending knees. This method can be used for animating cartoonish characters. This application also reveals that if we implement a human tracking algorithm based on PCA, we can also track cartoonish and exaggerated movements. Conclusion In this project, we examined some practical methods that can be used for human tracking. As this problem is very complex, it involves several layers of subproblems, and there does not exist a single global approach that totally covers it. Therefore, even contrary approaches share common views and methods. Reduction of dimensionality and obtaining convertions between 2D images and 3D skeletons, which we investigated in this project, are some of these methods that would be included in a textbook or a course about human body tracking.
References [1] CMU Graphics Lab Motion Capture database http://mocap.cs.cmu.edu/ [2] Feng Zhou, Fernando De la Torre Frade, and Jessica K. Hodgins, "Aligned Cluster Analysis for Temporal Segmentation of Human Motion," IEEE Conference on Automatic Face and Gestures Recognition, September, 2008. [3] Jernej Barbič, Alla Safonova, Jia-Yu Pan, Christos Faloutsos, Jessica K. Hodgins, and Nancy S. Pollard. “Segmenting Motion Capture Data into Distinct Behaviors.” In Proceedings of Graphics Interface 2004, pages 185–194, May 2004. [4] Clewley, R.H.; Guckenheimer, J.M.; Valero-Cuevas, F.J., "Estimating Effective Degrees of Freedom in Motor Systems," Biomedical Engineering, IEEE Transactions on , vol.55, no.2, pp.430-442, Feb. 2008 [5] Troje, N. F. (2002). Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of Vision, 2:371-387, Full Text DOI 10.1167/2.5.2. [6] Sandhitsu R. Das, Robert C. Wilson, Maciej T. Lazarewicz, Leif H. Finkel: Two-Stage PCA Extracts Spatiotemporal Features for Gait Recognition. Journal of Multimedia 1(5): 9-17 (2006) [7] Song-Chun Zhu; Rong Zhang; Zhuowen Tu, "Integrating bottom-up/top-down for object recognition by data driven Markov chain Monte Carlo," Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on , vol.1, no., pp.738-745 vol.1, 2000 [8] Mun Wai Lee , Isaac Cohen, “A Model-Based Approach for Estimating Human 3D Poses in Static Images”, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.6, p.905916, June 2006 [9] R. Urtasun and P. Fua, “3D Human Body Tracking Using Deterministic Temporal Motion Models.” In European Conference on Computer Vision, Prague, Czech Republic, May 2004. [10] Information on ASF/AMC files: http://www.cs.wisc.edu/graphics/Courses/cs-838-1999/Jeff/ASF-AMC.html [11] The INRIA perception group’s database in available at: https://charibdis.inrialpes.fr/html/index.php