Autonomous Aerial Cinematography

Autonomous Aerial Cinematography

Published: by

Autonomous Aerial Cinematography


Aerial cinematography is revolutionizing industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. However, safely piloting a drone while filming a moving target in the presence of obstacles is immensely taxing, often requiring multiple expert human operators.

Commercial drone products can tackle some automated tasks, but one thing those systems don’t address is filming artistically. Our team of researchers has proposed a complete system for aerial cinematography that learns humans’ visual preferences. The fully autonomous system does not require scripted scenes, GPS tags to localize targets or prior maps of the environment.

Drone filming
Drone autonomously filming a moving vehicle

We address the problem in its entirety and propose a complete system for real‐time aerial cinematography that for the first time combines: (a) vision‐based target estimation; (b) 3D signed‐distance mapping for occlusion estimation; (c) efficient trajectory optimization for long time horizon camera motion; and (d) learning‐based artistic shot selection. We extensively evaluate our system both in simulation and in field experiments by filming dynamic targets moving through unstructured environments. Our results indicate that our system can operate reliably in the real world without restrictive assumptions.

Project Members

  • Rogerio Bonatti
  • Wenshan Wang
  • Cherie Ho
  • Aayush Ahuja
  • Mirko Gaschwindt
  • Efe Camci

Publications

  • R. Bonatti, W. Wang, C. Ho, A. Ahuja, M. Gschwindt, E. Camci, E. Kayacan, S. Choudhury, S. Scherer, “Autonomous Aerial Cinematography Among Unstructured Environments With Learned Artistic Decision-Making”. Journal of Field Robotics, 2019 [Link][PDF][Video]
  • R. Bonatti, H. Cherie, W. Wenshan, S. Choudhury, S. Scherer, “Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments”. International Conference on Intelligent Robots and Systems (IROS), 2019 [PDF][Video]
  • M. Gschwindt, E. Camci, R. Bonatti, W. Wenshan, S. Scherer, “Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography”. International Conference on Intelligent Robots and Systems (IROS), 2019 [PDF][Video]
  • W. Wang, A. Ahuja, Y. Zhang, R. Bonatti, S. Scherer, “Improved Generalization of Heading Direction Estimation for Aerial Filming Using Semi-supervised Regression”. International Conference on Robotics and Automation (ICRA), 2019 [PDF][Video]
  • R. Bonatti, Y.Zhang, S. Choudhury, W. Wang, S. Scherer, “Autonomous drone cinematographer: Using artistic principles to create smooth, safe, occlusion-free trajectories for aerial filming”. International Symposium on Experimental Robotics (ISER), 2018 [PDF][Video 1][Video 2][Video 3]
  • Y.Zhang, W. Wang, R. Bonatti, D. Maturana, S. Scherer, “Integrating kinematics and environment context into deep inverse reinforcement learning for predicting off-road vehicle trajectories”. Conference on Robot Learning (CoRL), 2018 [PDF][Video]

Video Data

We release a few videos of the field experiments. These videos were taken from different stages of our tests containing multiple types of actors in various seasons.

Perception, Planning, Control, Project Overview

Latest Research

AirIMU: Learning Uncertainty Propagation for Inertial Odometry
AirIMU: Learning Uncertainty Propagation for Inertial Odometry

AirIMU, a hybrid IO system designed for state estimation and uncertainty propagation.

SubT-MRS: Pushing SLAM Towards All-weather Environments
SubT-MRS: Pushing SLAM Towards All-weather Environments

Simultaneous localization and mapping (SLAM) is a fundamental task for numerous applications such...

TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks
TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks