The Limits of Simulation

By Michael Berry, M.D. Federal Air Surgeon - FAA Safety Briefing Nov/Dec 2017

In these pages, we explain the virtues and limitations of simulator training. Simulators are profoundly valuable when it comes to training and certificating airmen for flying abilities. So why don’t we allow them for the assessment of neurocognitive impairments that can accompany conditions like traumatic brain injury, stroke, Transient Ischemic Attack (TIA), ADHD/ADD, substance abuse/dependence, and depression/SSRI medication use? That question and many other neurology policy questions prompted a meeting of the FAA’s neurologist consultants to review and update the FAA’s medical certification policy regarding many neurologic conditions. Participants also pondered whether brain imaging and a neurological clinical exam, along with a simulator evaluation, would be sufficient to detect significant neurocognitive deficits.

In both cases, the answer was no.

Why Sims Don’t Work

It’s not that we don’t see the value of simulators. It’s just that even the best multi-million dollar simulators are still poor instruments to measure subtle cognitive impairment in a pilot. One of our experts, Chris Front, Psy.D., is an aerospace clinical psychologist and GA pilot who has examined this issue. The research is quite clear that specific functions of the brain are critical to pilot performance. These include perceptual-motor abilities, spatial abilities, processing speed, and in particular, “executive functions” such as logical and flexible problem-solving, attentional skills, working memory, sequencing abilities, and so forth. These factors are very predictive of pilot performance in general and in determining what, if any, deficits exist in an individual airman.

So how do flight simulators do in isolating and testing executive functions? Not well, and the problems can’t be solved by simply improving simulator fidelity. The first, and largest, problem is data. In order to have a valid test, we need what’s called normative data. Normative data is used to show what performance to expect from the population so we have a measuring stick against which to evaluate the people we are testing. To determine whether there is a deficit, you must first define what a normal reading is. To have a valid simulator test for a neurocognitive deficit, therefore, we would need normative data for every simulator we would want to use. Compiling and maintaining such a database would be virtually impossible.

This issue leads to problem two, novelty. In order to best test the brain’s executive functions, you need novel scenarios. Anyone who has experienced more than a few simulator checks would probably agree that the scenarios presented in the simulator are not exactly novel. Even assuming we could create a novel simulator check, we would then need to collect normative data regarding each scenario. The novelty of the scenario would likely be short lived, because pilots quickly get “the gouge” and test validity would soon evaporate. There are other issues that also disqualify simulators for medical evaluation, but these are two of the most difficult ones.

What Does Work?

As Dr. Front noted, research has clearly identified the neurocognitive functions most closely aligned with flight performance. During the 1980s, the FAA contracted for the development of a specialized test to measure these functions. The result was CogScreen-Aeromedical Edition (CogScreen-AE), a computer-administered test that assesses elements such as attention, memory, visual-perceptual functions, sequencing functions, logical problem solving/executive functions, psychomotor speed and coordination, and simultaneous information processing abilities. CogScreen-AE is also “normed” on pilots (large commercial airline, and regional carrier) rather than the general population.

In testing against Flight Data Recorder (FDR) data overseas (where the laws allow such research), CogScreen-AE was determined to be highly predictive of airline pilot performance. It has also been shown to be highly effective in detecting neurocognitive deficits that could impact flight safety. In 2013, the FAA added GA pilot normative data (meaning that individual performance can also be measured against that of other GA pilots).

While simulation is an exceptional tool for training purposes, the bottom line is that neuropsychological assessment that includes CogScreen-AE is a far better tool for determining a pilot’s Aeromedical fitness status.