Published: by
Rebecca Martin

3D Human Reconstruction with Collaborative Aerial Cameras

Aerial vehicles are revolutionizing applications that require capturing the 3D structure of dynamic targets in the wild, such as sports, medicine and entertainment. The core challenges in developing a motion-capture system that operates in outdoors environments are: (1) 3D inference requires multiple simultaneous viewpoints of the target, (2) occlusion caused by obstacles is frequent when tracking moving targets, and (3) the camera and vehicle state estimation is noisy. We present a real-time aerial system for multi-camera control that can reconstruct human motions in natural environments without the use of special-purpose markers.

We present a multi-drone motion capture system for 3D human reconstruction in the wild. Our framework coordinates aerial cameras to optimally reconstruct the target’s body pose while avoiding obstacles and occlusions outdoors.

Multi-camera coordination

We formulate a multi-camera coordination scheme with the goal of maximizing the reconstructed 3D pose quality of dynamic targets. We develop a scalable two-stage system with long planning time horizons and real-time performance that uses a centralized planner for formation control and a decentralized trajectory optimizer that runs on each robot.

Real-life flight among obstacle. Our adaptive formation rotates clockwise avoiding the mound to maintain optimal reconstruction angle.

We provide studies evaluating system performance in simulation, and validate real-world performance using two drones while a target performs activities such as jogging and playing soccer.

3D reconstruction of a highly dynamic target playing soccer.

Video

Additional Info

Citation

@inproceedings{ho2021_human3d,
  author = {Ho, Cherie and Jong, Andrew and Freeman, Harry and Rao, Rohan and Bonatti, Rogerio and Scherer, Sebastian},
  title = {3D Human Reconstruction in the Wild with Collaborative Aerial Cameras},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year = {2021},
  month = sep,
  url = {https://arxiv.org/abs/2108.03936},
  video = {https://youtu.be/jxt91vx0cns}
}

Please refer to our paper for details.

Contributors

Perception, Planning

Latest Research

AirIMU: Learning Uncertainty Propagation for Inertial Odometry
AirIMU: Learning Uncertainty Propagation for Inertial Odometry

AirIMU, a hybrid IO system designed for state estimation and uncertainty propagation.

SubT-MRS: Pushing SLAM Towards All-weather Environments
SubT-MRS: Pushing SLAM Towards All-weather Environments

Simultaneous localization and mapping (SLAM) is a fundamental task for numerous applications such...

TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks
TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks