3D Human Reconstruction with Collaborative Aerial Cameras

Aerial vehicles are revolutionizing applications that require capturing the 3D structure of dynamic targets in the wild, such as sports, medicine and entertainment. The core challenges in developing a motion-capture system that operates in outdoors environments are: (1) 3D inference requires multiple simultaneous viewpoints of the target, (2) occlusion caused by obstacles is frequent when tracking moving targets, and (3) the camera and vehicle state estimation is noisy. We present a real-time aerial system for multi-camera control that can reconstruct human motions in natural environments without the use of special-purpose markers.

We present a multi-drone motion capture system for 3D human reconstruction in the wild. Our framework coordinates aerial cameras to optimally reconstruct the target’s body pose while avoiding obstacles and occlusions outdoors.

Multi-camera coordination

We formulate a multi-camera coordination scheme with the goal of maximizing the reconstructed 3D pose quality of dynamic targets. We develop a scalable two-stage system with long planning time horizons and real-time performance that uses a centralized planner for formation control and a decentralized trajectory optimizer that runs on each robot.

Real-life flight among obstacle. Our adaptive formation rotates clockwise avoiding the mound to maintain optimal reconstruction angle.

We provide studies evaluating system performance in simulation, and validate real-world performance using two drones while a target performs activities such as jogging and playing soccer.

3D reconstruction of a highly dynamic target playing soccer.

Video

Additional Info

Citation

@inproceedings{ho2021_human3d,
  author = {Ho, Cherie and Jong, Andrew and Freeman, Harry and Rao, Rohan and Bonatti, Rogerio and Scherer, Sebastian},
  title = {3D Human Reconstruction in the Wild with Collaborative Aerial Cameras},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year = {2021},
  month = sep,
  url = {https://arxiv.org/abs/2108.03936},
  video = {https://youtu.be/jxt91vx0cns}
}

Please refer to our paper for details.

Contributors

Cherie Ho <cherieh [at] cmu [dot] edu>
Andrew Jong
Harry Freeman
Rohan Rao
Rogerio Bonatti
Prof. Sebastian Scherer <basti [at] cmu [dot] edu>