Air Lab Summer School

6-10 July 2020

General Information

The goal of this boot camp is to get each student started with the basics, broaden their horizon beyond their own research topic, understand the tradeoffs of different approaches in our context (real-time, flying robots, etc.) and make our students more “dangerous” with some code/tools that they can use to jump start their research.

There will be three sessions per day, covering the topics of Perception, State Estimation, Action, and Infrastructure. Each session will consist of a 1 hour lecture followed by a 1.5 hour lab.

Click on a session in the schedule to jump to its session overview.


Day Time Session Host Links
Mon. July 6 9:00-11:30 AM EST 3.1 Motion Planning Tutorial (Basics) Basti slides | repo
  1:00-3:30 PM EST 1.1 Graph Learning Chen slides | repo
  3:30-6:00 PM EST 4.1 Setup in Cloud and Cluster Kat slides | repo
Tues. July 7 9:00-11:30 AM EST 2.1 Visual Odometry Yafei slides | repo
  12:30-3:00 PM EST 3.2 Motion Planning (Advanced) Jay & Brady slides | repo
  3:30-6:00 PM EST 4.2 Air Lab Core Stack John slides | repo
Wed. July 8 9:00-11:30 AM EST 1.2 Object Detection and Tracking Sourish & Anish & Andrew detection slides | tracking slides | tools slides
detection colab 1 | detection colab 2
tracking tutorial
  12:30-3:00 PM EST 3.3 Exploration and OpenVDB Rohit  
  3:30-6:00 PM EST 4.3 Simulation & 3.4 Reinforcement Learning Wenshan & Rogerio & Jay & Sourish slides | repo-RL | repo-AirSim-CoreStack | repo-AirSim-ROS
Thurs. July 9 9:00-11:30 AM EST 4.4 Project Management Vai slides
  12:30-3:00 PM EST 3.5 Formulating and Enforcing Safety Cherie & Mohammad  
  3:30-6:00 PM EST 1.3 Stereo Basics and Advanced Stereo Yaoyu slides | C++, Deep models
Fri. July 10 9:00-11:30 AM EST 2.3 SLAM / GTSAM Weikun  
  12:30-3:00 PM EST 3.6 Modeling and Control Azarakhsh repo | slides


Various labs will require these specifications:

  • Ubuntu 18.04
  • Matlab installed from CMU with all the toolboxes
  • Python: Pytorch, torchvision, opencv-python, opencv-contrib-python
  • ROS Melodic
  • Docker

Session Contents


1.1 Graph Learning

  • Graph Convolutional Networks
  • Graph Sampling Methods
  • Application and PyTorch Implementation

1.2 Object Detection and Tracking

  • Review of object detection methods based on two flavors
    • Single-shot detectors (YOLO, SSD, etc.)
    • Region-based networks (R-CNN, R-FCN, FPN, etc.)
  • Review of tracking
    • MOSSE, MedianFlow, KCF, CSRT
  • On-platform tools
    • OpenVino
  • Lab: Google colab session for implementation of a tiny object detector and tracker from scratch
    • Load dataset
    • Train model
    • Deploy

1.3 Stereo Basics and Advanced Stereo

  • Review
    • Fundamentals of stereo vision
    • Review of introductory CV courses.
  • Review stereo camera calibration. The secret for real-world testing.
    • ROS camera_calibration and OpenCV api, MATLAB.
    • Kalibr
  • Public stereo dataset and benchmark.
    • Middlebury, Scene Flow, NYU, KITTI, etc
  • Stereo reconstruction by OpenCV.
    • SGBM.
    • CUDA BM.
  • SPS-stereo.
  • Reconstruction with sparse depth heuristic.
  • Confidence measure.
  • Deep-learning models for stereo vision.
  • Related topics
    • Multi-view stereo
    • Monocular depth. Unsupervised learning.
    • Optical flow.

State Estimation

2.1 Visual Odometry (Traditional)

  • Feature based visual odometry
    • Pose estimation method from 2D to 2D
    • Pose estimation method from 3D to 2D
    • Pose estimation method from 3D to 3D
  • Direct based visual odometry
    • Optical Flow
    • Epipolar Search

2.2 Visual Odometry (Learning-Based)

  • Deep Learning for feature detection and matching
  • Supervised learning for visual odometry
  • Self-supervised learning for monocular depth and pose estimation


  • Intro to factor graph
  • Example to solve factor graph
  • Tools for back-end SLAM: GTSAM/Ceres


3.1 Motion Planning Tutorial (Basics)

  • Introduction to motion planning
  • Problem formulation
  • Abstraction and approaches
    • Regular graph search: A*-grid search
    • Sampling-based: RRT*
    • Trajectory optimization: CHOMP
  • Results
  • Matlab exploration of planners

3.2 Motion Planning (Advanced)

  • Review of motion planning basics
  • Overview of newer planners and development
    • FMT*
    • Informed RRT*
    • BIT*
    • RABIT*
  • Open Motion Planning Library (OMPL) overview
  • Exercises using OMPL

3.3 Exploration and OpenVDB

  • Presentation:
    • Planning for exploration of a-priori unknown environments
    • vdbmap - an occupancy grid mapping pipeline that uses a specialized sparse volumetric data structure
    • Frontier based exploration and its integration with the core_autonomy stack
  • Hands-On Session:
    • Setting up the workspace
    • Walk through the grid mapping pipeline
    • Playing with the grid mapping parameters for different sensor types
    • Walk through the frontier based exploration planner
    • Running and visualizing the planner in simulation

3.4 Reinforcement Learning

  • Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations
  • Learning people avoidance in simulation using DQN.

3.5 Formulating and Enforcing Safety

  • Purpose: Building intuition on different safety tools and where they fail
  • Where things fit in?
  • Common use cases… I want to make a safety claim on my planning stack
    • AACUS - Emergency maneuver library
  • Key tools
    • Reachability
    • Curse of dimensionality
  • Control barrier functions
    • How does CBF makes a safety claim?
  • Interactive Lab Session with ECBF
    • Single robot (with safe control and without) with fill-ins on base QP optimization
    • Multirobot
    • Problems with CBF
      • Deadlock and how to address
      • Unsolvable cases

3.6 Modeling and Control

  • Modeling a multirotor (kinematics and dynamics)
  • The general architecture of the control system with PX4 as an example
  • PID for attitude and position controllers
  • MATLAB simulation with analysis for multirotor, attitude control and position control designs (including fully-actuated multirotors)


4.1 Setup in Cloud and Cluster

  • Docker
  • Cluster & SLURM
  • Azure
  • Continuous Intergration
  • Git, Ansible, Deployer

4.2 Air Lab Core Stack

4.3 Simulation

  • Presentation
    • Unreal Engine and AirSim basics
      • Basic concepts
      • Installation procedure
      • Basic operations, navigate, add, move, scale, rotation etc.
    • Use AirSim w/ ROS
      • AirSim interfaces
      • AirSim ROS adapter
      • Joystick/ keyboard
    • Use AirSim on cluster w/ docker
    • Integrate AirSim with the core stack and px4
    • Use AirSim on Azure
  • Lab
    • A toy example of using AirSim w/ ROS
    • People Avoidance using Reinforcement Learning
    • Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations
    • Integrate AirSim with PX4

4.4 Project Management

  • Why project management/systems engineering
  • Different methods
    • Gantt
    • Agile
    • Etc
  • Tools
    • Airtable
    • Trello