International Audio Laboratories Erlangen

Data-Driven Stabilization of Marker-Free Video-Based Motion Capture Systems (SMART)

Logo_DFG Teaser_SMART Logo_MPII Logo_UniSaarland Logo_UniHannover Logo_UniTUM

In the SMART project, we developed robust and efficient methods that allow marker-free tracking of complex human movements in video data. The project was funded by the German Research Foundation. On this website, we summarize the project's main outcomes and provide links to project-related publications.

  • Principal investigators:
  • Prof. Dr. Daniel Cremers, School of Computation, Information and Technology, Technische Universität München
  • Programme period: 2007–2011
  • Grant: MU 2686/2-2, RO 2497/6-2, CR 250/3-2

Project Description

Data-Driven Stabilization of Marker-Free Video-Based Motion Capture Systems

Teaser_SMART_example3

In this project, we developed robust and efficient methods that allow marker-free tracking of complex human movements in video data. Here, the tracking was supported and stabilized by including previous knowledge about specific motion aspects and using temporal coherence through comparison with previously learned motion patterns. One focus of the SMART project was exploring compact and flexible motion representation, generating prior knowledge from 3D motion data using statistical learning methods, integrating a priori knowledge in motion tracking, and developing efficient retrieval and classification techniques for multimodal motion data. Furthermore, we investigated feedback mechanisms between the tracked movement sequences and the knowledge database to enrich the prior knowledge. In this way, the introduction of movement dynamics for image analysis could be used in a top-down strategy, and conversely, the a priori knowledge of the databases could be tightened in a bottom-up process.

Projektbeschreibung

Datengestützte Stabilisierung markerfreier videobasierter Motion-Capture-Systeme

Teaser_SMART_example3

In diesem Projekt wurden robuste und effiziente Verfahren, die ein markerfreies Tracking komplexer menschlicher Bewegungen in Videodaten erlauben, entwickelt. Hierbei wurde das Tracking durch Einbeziehen von Vorwissen über geeignete Bewegungsaspekte und unter Ausnutzung zeitlicher Kohärenz durch Abgleich mit zuvor gelernten Bewegungsmustern unterstützt und stabilisiert. Schwerpunkte des SMART-Projekts war die Erforschung kompakter und fexibler Repräsentationsformen von Bewegungen, die Generierung von Vorwissen aus 3D-Bewegungsdaten mittels statistischer Lernverfahren, die Integration von A-priori-Wissen beim Bewegungstracking, sowie die Entwicklung effizienter Retrieval- und Klassifikationstechniken für multimodale Bewegungsdaten. Weiterhin wurden zur Anreicherung des Vorwissens Rückkopplungsmechanismen zwischen den getrackten Bewegungssequenzen und der Wissensdatenbank erforscht. Auf diese Weise konnte in einer Top-Down Strategie das Einbringen von Bewegungsdynamik zur Bildanalyse verwendet und umgekehrt in einem Bottom-Up Prozess das A-priori-Wissen der Datenbanken verschärft werden.

Projected-Related Publications

The following publications reflect the main scientific contributions of the work carried out in the SMART project.

  1. Andreas Baak, Thomas Helten, Meinard Müller, Gerard Pons-Moll, Bodo Rosenhahn, and Hans-Peter Seidel
    Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors
    In Proceedings of the 3nd International Workshop on Human Motion. In Conjunction with ECCV: 137–150, 2010. PDF 
    @inproceedings{BaakHMPRS10_EvaluatingTrackingInertial_ECCV-HMW,
author    = {Andreas Baak and Thomas Helten and Meinard M{\"u}ller and Gerard Pons-Moll and Bodo Rosenhahn and Hans-Peter Seidel},
title     = {Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors},
booktitle = {Proceedings of the 3nd International Workshop on Human Motion. In Conjunction with ECCV},
series    = {Lecture Notes of Computer Science (LNCS)},
volume    = {6553},
pages     = {137--150},
publisher = {Springer},
month     = sep,
year      = {2010},
address = {Hersonissos, Crete},
url-pdf   = {2010_BaakHMPRS_EvaluatingTrackingInertial_ECCV-HMW.pdf}
}
  2. Andreas Baak, Meinard Müller, Gaurav Bharaj, Hans-Peter Seidel, and Christian Theobalt
    A Data-Driven Approach for Real-Time Full Body Pose Reconstruction from a Depth Camera
    In Proceedings of the International Conference on Computer Vision (ICCV): 1092–1099, 2011. PDF 
    @inproceedings{BaakMBST11_DepthCamera_ICCV,
author       = {Andreas Baak and Meinard M{\"u}ller and Gaurav Bharaj and Hans-Peter Seidel and Christian Theobalt},
title        = {A Data-Driven Approach for Real-Time Full Body Pose Reconstruction from a Depth Camera},
booktitle    = {Proceedings of the International Conference on Computer Vision ({ICCV})},
year         = {2011},
pages        = {1092--1099},
ee           = {http://dx.doi.org/10.1109/ICCV.2011.6126356},
address      = {Barcelona, Spain},
url-pdf      = {2011_BaakMuellerBharajSeidelTheobalt_DataDrivenDepthTracking_ICCV.pdf},
}
  3. Andreas Baak, Bodo Rosenhahn, Meinard Müller, and Hans-Peter Seidel
    Stabilizing Motion Tracking Using Retrieved Motion Priors
    In Proceedings of the International Conference on Computer Vision (ICCV): 1428–1435, 2009. PDF 
    @inproceedings{BaakRMS09_StabilizedTracking_ICCV,
author       = {Andreas Baak and Bodo Rosenhahn and Meinard M{\"u}ller and Hans-Peter Seidel},
title        = {Stabilizing Motion Tracking Using Retrieved Motion Priors},
booktitle    = {Proceedings of the International Conference on Computer Vision ({ICCV})},
month        = sep,
year         = {2009},
address    = {Kyoto, Japan},
pages        = {1428--1435},
isbn         = {978-1-4244-4419-9},
url-pdf   = {2009_BaakRosenhahnMuellerSeidel_StabilizedMotionTracking_ICCV.pdf},
}
  4. Daniel Cremers and Kalin Kolev
    Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains
    IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(6): 1161–1174, 2011. Details 
    @article{CremersK11_MultiviewConvex_PAMI,
author       = {Daniel Cremers and Kalin Kolev},
title        = {Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains},
journal      = {{IEEE} Transactions on Pattern Analysis and Machine Intelligence},
volume       = {33},
number       = {6},
pages        = {1161--1174},
year         = {2011},
url-details  = {https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5567114}
}
  5. Meinard Müller, Bastian Demuth, and Bodo Rosenhahn
    An Evolutionary Approach for Learning Motion Class Patterns
    In Proceedings of the Annual Symposium of the German Association for Pattern Recognition (DAGM): 365–374, 2008. PDF Details 
    @inproceedings{MuellerDR08_EvolutionaryApproach_DAGM,
author    = {Meinard M{\"u}ller and Bastian Demuth and Bodo Rosenhahn},
title     = {An Evolutionary Approach for Learning Motion Class Patterns},
booktitle = {Proceedings of the Annual Symposium of the German Association for Pattern Recognition ({DAGM})},
address   = {Munich, Germany},
publisher = {Springer},
series    = {Lecture Notes in Computer Science},
volume    = {5096},
isbn      = {978-3-540-69320-8},
month     = jun,
year      = {2008},
pages     = {365--374},
url-pdf   = {2008_MuellerDemuthRosenhahn_EvolutionaryMotionPattern_DAGM.pdf},
url-details = {http://resources.mpi-inf.mpg.de/HDM05/}
}
  6. Gerard Pons-Moll, Andreas Baak, Thomas Helten, Meinard Müller, Hans-Peter Seidel, and Bodo Rosenhahn
    Multisensor-Fusion for 3D Full-Body Human Motion Capture
    In IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 663–670, 2010. PDF 
    @inproceedings{PonsBaHeMuSeRo10_MultisensorFusion_CVPR,
author     = {Gerard Pons-Moll and Andreas Baak and Thomas Helten and Meinard M{\"u}ller and Hans-Peter Seidel and Bodo Rosenhahn},
title      = {Multisensor-Fusion for 3D Full-Body Human Motion Capture},
booktitle  = {IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year       = {2010},
address    = {San Francisco, California, USA},
month      = jun,
pages      = {663--670 },
url-pdf    = {2010_PonsBaakHeltenMuellerSeidelRosenhahn_MultisensorFusionMocap_CVPR.pdf}
}
  7. Gerard Pons-Moll, Andreas Baak, Jürgen Gall, Laura Leal-Taixe, Meinard Müller, Hans-Peter Seidel, and Bodo Rosenhahn
    Outdoor Human Motion Capture using Inverse Kinematics and von Mises-Fisher Sampling
    In Proceedings of the International Conference on Computer Vision (ICCV): 1243–1250, 2011. PDF 
    @inproceedings{PonsBGMSR11_OutdoorHumanMotion_ICCV,
author       = {Gerard Pons-Moll and Andreas Baak and J\"urgen Gall and Laura Leal-Taixe and Meinard M{\"u}ller and Hans-Peter Seidel and Bodo Rosenhahn},
title        = {Outdoor Human Motion Capture using Inverse Kinematics and von Mises-Fisher Sampling},
booktitle    = {Proceedings of the International Conference on Computer Vision ({ICCV})},
year         = {2011},
pages        = {1243--1250},
ee           = {http://dx.doi.org/10.1109/ICCV.2011.6126375},
address    = {Barcelona, Spain},
url-pdf   = {2011_PonsBaakGallTaxieMuellerSeidelRosenhahn_OutdoorHumanMocap_ICCV.pdf},
}
  8. Licong Zhang, Jürgen Sturm, Daniel Cremers, and Dongheui Lee
    Real-time human motion tracking using multiple depth cameras
    In Intelligent Robots and Systems (IROS): 2389–2395, 2012. Details DOI 
    @inproceedings{ZhangSCL12,
author       = {Licong Zhang and J{\"{u}}rgen Sturm and Daniel Cremers and Dongheui Lee},
title        = {Real-time human motion tracking using multiple depth cameras},
booktitle    = {Intelligent Robots and Systems ({IROS})},
pages        = {2389--2395},
publisher    = {{IEEE}},
year         = {2012},
doi          = {10.1109/IROS.2012.6385968},
url-details  = {https://ieeexplore.ieee.org/document/6385968}
}

Projected-Related Ph.D. Theses

  1. Andreas Baak
    Retrieval-based Approaches for Tracking and Reconstructing Human Motions
    PhD Thesis, Universität des Saarlandes, 2012. PDF Details 
    @phdthesis{Baak12_TrackRecMotion_PhD,
author      = {Andreas Baak},
year        = {2012},
title       = {Retrieval-based Approaches for Tracking and Reconstructing Human Motions},
school      = {Universit{\"a}t des Saarlandes},
url-details = {https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/26465},
url-pdf = {2012_BaakAndreas_HumanMotionTracking_Thesis-PhD.pdf}
}
  2. Gerard Pons Moll
    Human Pose Estimation from Video and Inertial Sensors
    PhD Thesis, Gottfried Wilhelm Leibniz Universität Hannover, 2014. PDF Details 
    @phdthesis{Pons14_PoseEstimation_PhD,
author      = {Gerard Pons Moll},
year        = {2014},
title       = {Human Pose Estimation from Video and Inertial Sensors},
school      = {Gottfried Wilhelm Leibniz Universität Hannover},
url-details = {https://ps.is.mpg.de/publications/pons-moll_dissertation},
url-pdf = {2014_Pons_PoseEstimation_Thesis-PhD.pdf}
}