Published: by
Joao Dantas

How do you train AI Pilots?

We need more pilots. AI can help! Advanced Aerial Mobility (AAM) is an inclusive term that covers urban (UAM), regional (RAM), intraregional (IAM), and suburban air mobility (SAM) solutions. All of these proposed solutions have one thing in common: they all envision a future where autonomous or semi-autonomous aerial vehicles are seamlessly integrated into the current airspace system. AAM solutions open doors to significant socio-economic benefits while at the same time presenting challenges in the seamless integration of these systems with human-operated aircraft and controlling agencies.

Today, manned and unmanned vehicles are separated, limiting the utility and flexibility of operations and reducing efficiency. Human operated aircraft follow one of the two rules for operation: Visual Flight Rules (VFR) or Instrument Flight Rules (IFR). The choice of flying under VFR or IFR is typically a function of weather conditions. Under VFR, an aircraft is flown just like driving a car within the FAA established rules of the road. IFR flying on the other hand is typically associated with flying aircraft under degraded weather conditions where separation is provided by the ATC. While a pursuit to integrate AAM can start either under IFR or VFR, automated-VFR operations often have better scalability than automated-IFR operations [1]. Another option involves UTM solutions [2​] which in its current iteration only focuses on small unmanned aerial aircraft operating close to the ground (<700 ft) in uncontrolled airspace.

Mastering visual flight rules (VFR) operations for autonomous aircraft has significant operational advantages at unimproved sites, as well as in achievable traffic density compared to instrument flight rules (IFR) or completely separated operations between manned and unmanned systems. For (semi-)autonomous aircraft to operate in tandem with human pilots and ATC controllers under VFR, technological advancements in certain key areas are required.


  1. Unmanned aircraft should be able to guarantee self-separation even in a tightly-spaced terminal airspace environment;
  2. Unmanned vehicles should be able to interpret high-level instructions by ATC to meet the expectations of a normal traffic flow;
  3. Autonomous aircraft need to understand the expected and unexpected behaviour of other aircraft;
  4. Communications by other pilots and ATC need to be parsed and valid, corresponding responses should be produced; and
  5. Other aircraft need to be detected and estimated, as well as their future trajectories need to be predicted.

Many of these challenges have parallels in the self-driving industry and the technological improvements there can be leveraged to produce domain specific solutions for AAM. While this is promising, VFR-like AAM integration introduces newer challenges while pushing boundaries on the current state of technology.

Given the domain transfer, we list below certain key areas of development where improvements on current state-of-the-art are needed to enable VFR-like autonomous aerial operations.

Aircraft Detection & Tracking

See-and-Avoid is one of the key tenets of VFR operation. The ability to spot other aircraft or aerial hazards like birds, balloons, gliders etc and execute maneuvers so as to mitigate a collision hazard is critical to successfully deploying AAM solutions in the NAS.

Visualization of the detected aircraft bounding boxes from the data onboard a general aviation aircraft. At long ranges, the SnR is really poor and our algorithms are trained to detect aircraft even in low SnR situations.

A more detailed update on the recent work is available in this blog post

Intent Prediction

Reasoning about the potential set of future trajectories that other aircraft can take is critical to ensure that the best actions are taken that also minimise risk. Typical prediction methods use short horizons and fail to capture the long term intent of other aircraft. The majority of trajectory prediction work has been explored in the pedestrian and autonomous vehicle domains. Within AAM, goal points such as airports, pilots and ATC communications, and rules of way (such as FAR §91.113) are often known. The use of this explicit source of information as well as implicit sources like weather can help decipher the intent of other aircraft and increase the length of reliable predictions.

The figure shows the dataset and its collection setup at the Pittsburgh-Butler Regional Airport (KBTP) — a non-towered GA airport that serves as a primary location for the dataset. Lighter colour indicates lower altitude. a) Shown is a snippet of the processed dataset with aircraft trajectories showing clear lobes for traffic patterns for both runways. b) The left traffic pattern and nomenclature for the runways at the airport. c) Picture of the data-collection setup.

The lack of datasets and baseline methods makes it difficult to conduct meaningful research. Towards this end, we have been collecting traffic data from transponders at select controlled and uncontrolled airports to understand and train models for intent prediction. To capture the influence of weather on pilot decisions, the weather data was also collected by parsing Meteorological Aerodrome Reports (METARs) to gather the relevant sections from the full weather report. We recently published the first chunk of the trajectory data TrajAir along with a baseline method christened TrajAirNet to predict aircraft trajectories trained on the collected data. More information on this is available on this blog post.

Patrikar, Jay, et al. “Predicting Like A Pilot: Dataset and Method to Predict Socially-Aware Aircraft Trajectories in Non-Towered Terminal Airspace.” IEEE International Conference on Robotics and Automation (2022). [Link] [Video]

Safe and Socially-Compliant Navigation

As the FAR rules only specify a rough guideline, autonomous vehicles must be equipped with the capability to make flexible decisions to comply with the traffic norm of arbitrary situations. The idea of social navigation is to learn to follow the observed traffic patterns in a socially-compliant manner. Generating actions that are not only safe but also socially compliant while following rules is thus critical in generating behaviour that is acceptable to human pilots co-habiting the same airspace. We use variants of Monte Carlo Tree Search algorithms that combine learned behaviour with logic specifications like Signal Temporal Logic to enable safe and socially compliant navigation.

Figure shows the proposed planning setup for a single agent. The offline policy is trained using Generative Adversarial Imitation Learning algorithm that learns system behaviour from the recorded trajectories from KBTP airport. Online this behaviour is executed using a Monte Carlo Tree Search algorithm that uses multiple roll-outs to model possible future states of the human agent. The roll-outs are constrained using Signal Temporal Logic specifications that ensure that the actions follow the rules established by the FAA.

Guaranteeing Safety

The safety system ensures that the future trajectory of the aircraft is always safe and a safe fallback trajectory always exists. The safety system also checks plans made by the planning/inference engine to ensure that these plans will not violate safety invariants. Effectively, the safety system serves to encode a safe envelope and guarantees collision-free flight.

Automated Speech Recognition and Production

It is critical to establish clear communication between a human operator/pilot and an AI system in our target problem domain. Specific to the aviation domain is the challenge of understanding and decoding aviation-specific terminology that is different from everyday speech constructions. Other challenges such as radio background noise, incomplete instructions and radio phraseology also need to be addressed.

High-fidelity Simulators

The figure shows the high-fidelity simulator setup that enables Human-AI interaction. Figure a) shows a top-down view with one human agent (magenta) interacting and one AI agent (lime) while trying to land on the same runway. The most likely branch of the MCTS forward propagation tree for both the agents is shown in cyan. White lines show the reference trajectories. Figure b) shows the physical simulator setup with an immersive environment for the human pilot. Figure c) shows a screengrab of the visual rendering of the simulator using the X-Plane 11 flight simulator backend.

For safety critical domains, accurate high-fidelity simulators are required to test algorithms before real-world testing. Given the lack of simulators in the public domain, we designed our simulator christened XPlaneROS. With X-Plane 11 as the high-fidelity flight simulator and ROSplane as the autopilot, we obtain realistic aircraft models and visuals of similar aircraft. The X-Plane Connect Toolbox interfaces between X-Plane and Robot Operating System (ROS) topics. Based on the high-level commands input by the planner, ROSPlane generates the actuator commands for ailerons, rudder, elevator, and throttle, which are in turn sent to X-Plane through XPlaneConnect.

More information is available in this blog post.

Big Picture

We believe that the current understanding of integrating unmanned aircraft within National Airspace System (NAS) needs a more human-pilot centered approach. Pilots and aircraft already operating in the national airspace are important stakeholders in the larger discourse. Technological developments enabling close-proximity safe and seamless operation of manned and unmanned aircraft in a shared airspace need an understanding of the rules and conventions already in place. The NAS is a dynamic environment with in-built flexibility and protocols to handle traffic volumes and emergencies. Our core understanding is then rather than having the current NAS adapt to changing autonomy needs, we need to move towards identifying technological requirements that enable unmanned systems to operate along with humans in a collaborative fashion.

“Our hope is to build true AI-pilots that are indistinguishable from a human pilot to enable a seamless integration within the current NAS.”


The video below gives a general overview of these challenges and potential applications.

Additional Info


This work is supported by the Army Futures Command Artificial Intelligence Integration Center (AI2C).


[1] Mueller, Eric R., Parmial H. Kopardekar, and Kenneth H. Goodrich. “Enabling airspace integration for high-density on-demand mobility operations.” 17th AIAA Aviation Technology, Integration, and Operations Conference. 2017.

[2] Aweiss, Arwa S., et al. “Unmanned Aircraft Systems (UAS) Traffic Management (UTM) National Campaign II.” 2018 AIAA Information Systems-AIAA Infotech@ Aerospace. 2018. 1727.


Jay Patrikar, Joao Dantas, Ingrid Navarro, Ian Higgins, Jasmine Aloor, Parv Kapoor, Milad Hamidi, Jimin Sun, Ben Stoler, Rohan Baijal, Brady Moon, Sourish Ghosh, Dr Jean Oh, Dr Sebastian Scherer

Term of use

BSD 4-Clause License

Learning, Perception, Planning