Air Lab Summer School

6-10 July 2020

General Information

The goal of this boot camp is to get each student started with the basics, broaden their horizon beyond their own research topic, understand the tradeoffs of different approaches in our context (real-time, flying robots, etc.) and make our students more “dangerous” with some code/tools that they can use to jump start their research.

Sessions cover topics in Perception, State Estimation, Action, and Infrastructure. Each session will consist of a lecture followed by hands-on exercises.

Click on a session in the overview below to jump to its summary, outline, links, and video.

Sessions Overview

Session Host Links
1.1 Graph Learning Chen slides | repo
1.2 Object Detection and Tracking Sourish & Anish & Andrew slides | repo
1.3 Stereo Basics and Advanced Stereo Yaoyu slides | repo
2.1 Visual Odometry Yafei slides | repo
2.2 SLAM / GTSAM Weikun  
3.1 Motion Planning Tutorial (Basics) Basti slides | repo
3.2 Motion Planning (Advanced) Jay & Brady slides | repo
3.3 Exploration and OpenVDB Rohit  
3.5 Formulating and Enforcing Safety Cherie & Mohammad slides | repo
3.6 Modeling and Control Azarakhsh slides | repo
4.1 Setup in Cloud and Cluster Kat slides | repo
4.2 Air Lab Core Stack John slides | repo
4.3 Simulation & 3.4 Reinforcement Learning Wenshan & Sourish slides | repo

Dependencies

Various labs will require these specifications:

  • Ubuntu 18.04
  • Matlab installed from CMU with all the toolboxes
  • Python: Pytorch, torchvision, opencv-python, opencv-contrib-python
  • ROS Melodic
  • Docker

Session Contents

Perception


1.1 Graph Learning

  • Graph Convolutional Networks
  • Graph Sampling Methods
  • Application and PyTorch Implementation
slides | repo

1.2 Object Detection and Tracking

An overview of popular methods used for object detection and tracking with a brief introduction to platform tools for inference
  • Review of object detection methods based on two flavors
    • Single-shot detectors (YOLO, SSD, etc.)
    • Region-based networks (R-CNN, R-FCN, FPN, etc.)
  • Review of tracking
    • MOSSE, MedianFlow, KCF, CSRT
  • On-platform tools: OpenVino
  • Lab: Google colab session for implementation of a tiny object detector and tracker from scratch
    • Load dataset, train model, and deploy
detection slides | tracking slides | tools slides
detection colab 1 | detection colab 2 | tracking tutorial

1.3 Stereo Basics and Advanced Stereo

Revision and implementation of the recent binocular stereo methods including non-leanring and learning-based
  • Review
    • Fundamentals of stereo vision
    • Review of introductory CV courses
  • Review stereo camera calibration. The secret for real-world testing
    • ROS camera_calibration and OpenCV api, MATLAB.
    • Kalibr
  • Public stereo dataset and benchmark
    • Middlebury, Scene Flow, NYU, KITTI, etc
  • Stereo reconstruction by OpenCV
    • SGBM
    • CUDA BM
  • SPS-stereo
  • Reconstruction with sparse depth heuristic
  • Confidence measure
  • Deep-learning models for stereo vision
  • Related topics
    • Multi-view stereo
    • Monocular depth, Unsupervised learning
    • Optical flow
slides | C++ | Deep models

State Estimation


2.1 Visual Odometry (Traditional)

  • Feature based visual odometry
    • Pose estimation method from 2D to 2D
    • Pose estimation method from 3D to 2D
    • Pose estimation method from 3D to 3D
  • Direct based visual odometry
    • Optical Flow
    • Epipolar Search

2.2 SLAM / GTSAM

  • Intro to factor graph
  • Example to solve factor graph
  • Tools for back-end SLAM: GTSAM/Ceres

Action


3.1 Motion Planning Tutorial (Basics)

An introduction to the unique challenges of motion planning for flying robots with interactive Matlab exercises
  • Introduction to motion planning
  • Problem formulation
  • Introduction to motion planning
  • Abstraction and approaches
    • Regular graph search: A*-grid search
    • Sampling-based: RRT*
    • Trajectory optimization: CHOMP
  • Results
  • Matlab exploration of planners
slides | repo

3.2 Motion Planning (Advanced)

An overview of newer motion planning methods as well as a walkthrough on how to use OMPL
  • Review of motion planning basics
  • Overview of newer planners and development
    • FMT*
    • Informed RRT*
    • BIT*
    • RABIT*
  • Open Motion Planning Library (OMPL) overview
  • Exercises using OMPL
slides | repo

3.3 Exploration and OpenVDB

  • Presentation:
    • Planning for exploration of a-priori unknown environments
    • vdbmap - an occupancy grid mapping pipeline that uses a specialized sparse volumetric data structure
    • Frontier based exploration and its integration with the core_autonomy stack
  • Hands-On Session:
    • Setting up the workspace
    • Walk through the grid mapping pipeline
    • Playing with the grid mapping parameters for different sensor types
    • Walk through the frontier based exploration planner
    • Running and visualizing the planner in simulation

3.5 Formulating and Enforcing Safety

  • Purpose: Building intuition on different safety tools and where they fail
  • Where things fit in?
  • Common use cases… I want to make a safety claim on my planning stack
    • AACUS - Emergency maneuver library
  • Key tools
    • Reachability
    • Curse of dimensionality
  • Control barrier functions
    • How does CBF makes a safety claim?
  • Interactive Lab Session with ECBF
    • Single robot (with safe control and without) with fill-ins on base QP optimization
    • Multirobot
    • Problems with CBF
      • Deadlock and how to address
      • Unsolvable cases

3.6 Modeling and Control

A practical introduction to modeling and control for Multirotor Aerial Vehicles with interactive MATLAB exercises
  • Introduction
  • Modeling Multirotors
  • Multirotor Controller Architecture (with PX4 as an example)
  • PID Controllers
  • Underactuated vs. Fully-Actuated UAVs
  • Matlab exercises
    • Attitude Controller and PID Tuning
    • Position Controller
    • Fully-Actuated Multirotors
slides | repo

Infrastructure


4.1 Setup in Cloud and Cluster

  • Docker
  • Cluster & SLURM
  • Azure
  • Continuous Intergration
  • Git, Ansible, Deployer

4.2 Air Lab Core Stack

4.3 & 3.4 Simulation and Reinforcement Learning

An introduction of doing drone simulation in AirSim, using the core stack package in the simulation, as well as doing reinforcement learning in it
  • Presentation
    • Unreal Engine and AirSim basics
      • Basic concepts
      • Installation procedure
      • Basic operations, navigate, add, move, scale, rotation etc.
    • Use AirSim w/ ROS
      • AirSim interfaces
      • AirSim ROS adapter
      • Joystick/ keyboard
    • Use AirSim on cluster w/ docker
    • Integrate AirSim with the core stack and px4
    • Use AirSim on Azure
  • Lab
    • A toy example of using AirSim w/ ROS
    • People Avoidance using Reinforcement Learning
    • Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations
    • Integrate AirSim with PX4
slides | repo-RL | repo-AirSim-CoreStack repo-AirSim-ROS