The goal of this boot camp is to get each student started with the basics, broaden their horizon beyond their own research topic, understand the tradeoffs of different approaches in our context (real-time, flying robots, etc.) and make our students more “dangerous” with some code/tools that they can use to jump start their research.
Sessions cover topics in Perception, State Estimation, Action, and Infrastructure. Each session will consist of a lecture followed by hands-on exercises.
Click on a session in the overview below to jump to its summary, video, outline, and links.
Sessions Overview
Dependencies
Various labs will require these specifications:
- Ubuntu 18.04
- Matlab installed from CMU with all the toolboxes
- Python: Pytorch, torchvision, opencv-python, opencv-contrib-python
- ROS Melodic
- Docker
Session Contents
Perception
1.1 Graph Learning
An introduction to graph neural network and its application in robotics, with an exercise and implementation in Github Classroom
- Graph Convolutional Networks
- Graph Sampling Methods
- Application and PyTorch Implementation
slides | repo
|
|
1.2 Object Detection and Tracking
An overview of popular methods used for object detection and tracking with a brief introduction to platform tools for inference
- Review of object detection methods based on two flavors
- Single-shot detectors (YOLO, SSD, etc.)
- Region-based networks (R-CNN, R-FCN, FPN, etc.)
- Review of tracking
- MOSSE, MedianFlow, KCF, CSRT
- On-platform tools: OpenVino
- Lab: Google colab session for implementation of a tiny object detector and tracker from scratch
- Load dataset, train model, and deploy
detection slides |
tracking slides |
tools slides
detection colab 1 |
detection colab 2 |
tracking tutorial
|
|
1.3 Stereo Basics and Advanced Stereo
Revision and implementation of the recent binocular stereo methods including non-leanring and learning-based
- Review
- Fundamentals of stereo vision
- Review of introductory CV courses
- Review stereo camera calibration. The secret for real-world testing
- ROS camera_calibration and OpenCV api, MATLAB.
- Kalibr
- Public stereo dataset and benchmark
- Middlebury, Scene Flow, NYU, KITTI, etc
- Stereo reconstruction by OpenCV
- SPS-stereo
- Reconstruction with sparse depth heuristic
- Confidence measure
- Deep-learning models for stereo vision
- Related topics
- Multi-view stereo
- Monocular depth, Unsupervised learning
- Optical flow
slides |
C++ |
Deep models
|
|
State Estimation
2.1 Visual Odometry (Traditional)
- Feature based visual odometry
- Pose estimation method from 2D to 2D
- Pose estimation method from 3D to 2D
- Pose estimation method from 3D to 3D
- Direct based visual odometry
- Optical Flow
- Epipolar Search
slides | repo
|
|
Action
3.1 Motion Planning Tutorial (Basics)
An introduction to the unique challenges of motion planning for flying robots with interactive Matlab exercises
- Introduction to motion planning
- Problem formulation
- Introduction to motion planning
- Abstraction and approaches
- Regular graph search: A*-grid search
- Sampling-based: RRT*
- Trajectory optimization: CHOMP
- Results
- Matlab exploration of planners
slides | repo
|
|
3.2 Motion Planning (Advanced)
An overview of newer motion planning methods as well as a walkthrough on how to use OMPL
- Review of motion planning basics
- Overview of newer planners and development
- FMT*
- Informed RRT*
- BIT*
- RABIT*
- Open Motion Planning Library (OMPL) overview
- Exercises using OMPL
slides | repo
|
|
3.3 Planning for Exploration
- Part I - Mapping
- Occupancy Grid Maps
- Ray Casting
- Underlying data structure - OpenVDB
- Example 2D/3D occupancy grid maps
- Hands-on session
- Part II - Frontier Based Exploration
- Frontiers for exploration planning
- Example Drone Autonomy Architecture - I
- Example Drone Autonomy Architecture - II
- Overview of modules
- Hands-on session
slides
|
|
3.5 Formulating and Enforcing Safety
- Purpose: Building intuition on different safety tools and where they fail
- Where things fit in?
- Common use cases… I want to make a safety claim on my planning stack
- AACUS - Emergency maneuver library
- Key tools
- Reachability
- Curse of dimensionality
- Control barrier functions
- How does CBF makes a safety claim?
- Interactive Lab Session with ECBF
- Single robot (with safe control and without) with fill-ins on base QP optimization
- Multirobot
- Problems with CBF
- Deadlock and how to address
- Unsolvable cases
slides | repo
|
|
3.6 Modeling and Control
A practical introduction to modeling and control for Multirotor Aerial Vehicles with interactive MATLAB exercises
- Introduction
- Modeling Multirotors
- Multirotor Controller Architecture (with PX4 as an example)
- PID Controllers
- Underactuated vs. Fully-Actuated UAVs
- Matlab exercises
- Attitude Controller and PID Tuning
- Position Controller
- Fully-Actuated Multirotors
slides | repo
|
|
Infrastructure
4.1 Setup in Cloud and Cluster
An intro to setting up infrastructure on the cloud & cluster and an overview of various AirLab development operations tools
- Docker
- Cluster & SLURM
- Azure
- Continuous Intergration
- Git, Ansible, Deployer
slides | repo
|
|
4.2 AirLab Core Autonomy Stack
An overview of the AirLab Core Autonomy Stack, including the setup, walkthrough, and running examples
- Behavior Trees
- Global and Local Planners
- Trajectory, Pose, and Velocity Controllers
- State Estimation
- Gazebo Sim
- Exercises
slides | repo
|
|
4.3 & 3.4 Simulation and Reinforcement Learning
An introduction of doing drone simulation in AirSim, using the core stack package in the simulation, as well as doing reinforcement learning in it
- Presentation
- Unreal Engine and AirSim basics
- Basic concepts
- Installation procedure
- Basic operations, navigate, add, move, scale, rotation etc.
- Use AirSim w/ ROS
- AirSim interfaces
- AirSim ROS adapter
- Joystick/ keyboard
- Use AirSim on cluster w/ docker
- Integrate AirSim with the core stack and px4
- Use AirSim on Azure
- Lab
- A toy example of using AirSim w/ ROS
- People Avoidance using Reinforcement Learning
- Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations
- Integrate AirSim with PX4
slides |
repo-RL |
repo-AirSim-CoreStack
repo-AirSim-ROS
|
|