Factoring in the Human in Avionics Certification

Jeff Holland and Wes Ryan

Reprinted with permission from FAA Aviation News

Any mention of how human factors (HF) affect system design or usability may conjure images of programming your stubborn old VCR, which is just slightly more difficult than changing runways on early generation GPS navigators. (For those who don’t get this example, ask your parents or an older pilot.) Yet, system usability has never been more important, as avionics get more complex and more integrated by the minute. We’re almost to the point where the manuals for some new systems are so big that operators may need to consider them in their weight and balance calculations.

One might ponder what we at the FAA do to address the growing complexity of avionics systems. The short answer is: plenty.

Starting with the Human

FAA has HF specialists, flight-test pilots, and engineers who review and certify every new major avionics product seeking technical standard order (TSO) and/or installation approval. It may not be widely known, but FAA HF evaluations typically start very early in the product development cycle, often with the first prototype in a company’s laboratory. These early HF evaluations check basic physical characteristics, including control interface and display characteristics. The areas we concentrate on include: intended function, display appearance, symbology, color palette, menu structure, menu depth and complexity, knob/button size, labeling, and system usability.

Menu structure, knob shape, and labels are very important because pilots expect obvious and easy-to-recognize functions with a clear and distinct tactile feel. Pilots expect their actions to result in intuitive and obvious system responses. They also expect clearly displayed options with an obvious means of selection and a clear way to return to a standard or default condition. Our early HF evaluations tell us how close to this mark new systems are.

As the system matures and designers add more capability and functionality, FAA evaluates the extent to which users have the ability to select options, view information, and input data. These part-task evaluations look at performance of single distinct tasks with the system, such as entering a navigation frequency, changing a barometric setting, or entering a simple flight plan. FAA HF specialists record their findings, identify any noncompliance, and provide feedback to the company. This cooperative process should occur early in the company’s design process, when it is easier and less costly to make changes to the system.

Testing, Testing, and More Testing

Once the system has achieved some level of maturity and represents the end product, the company installs it into an airplane to begin in-flight evaluations. This may represent the first time anyone evaluates display dynamics and system interface in the airplane, so there are usually additional items to evaluate. At this point, FAA conducts additional evaluations to assess system usability under actual flight conditions.

In cases where the system is highly integrated, complex, and/or performs critical functions, FAA uses a formal evaluation process that involves scenario-based evaluations by multiple FAA pilots. The process, termed multiple pilot system usability evaluation (MPSUE), has become a standard approach to evaluating complex avionics systems targeting general aviation aircraft.

The FAA does not conduct MPSUEs on every product seeking certification. Neither is it our intent to do an in-depth human factors study on every aspect on every new system or component. Instead, we make an initial assessment of the complexity, novelty, and potential for controversy based on the system characteristics or functions. Next, we assess the potential impact each aspect may have on pilot awareness, performance, workload, and, ultimately, safety. Based on these findings, we make a determination whether we need a MPSUE and how in-depth and rigorous it needs to be to mitigate risks and ensure a safe usable product. We have conducted MPSUEs with as few as three pilots and as many as seven pilots.

We select the test pilots based on their training, background, and experience to fly canned scenarios representative of the types of VFR and IFR environments and situations general aviation pilots may encounter. The evaluation scenarios include situations that will expose pilots to various system aspects, including failure conditions, and require them to exercise system functions. Our HF specialists collect data from individual pilots throughout the flight using questionnaires and rating forms. At the conclusion of testing, participants gather together to discuss results and draw conclusions. All the findings are shared with the company. The benefit to the company and end user is a product that is easier to learn, easier to use, and subsequently safer.

The process sounds simple, but it takes a very special mix of skills to conduct, including knowledge of human behavior, human performance, required piloting skills, and an understanding of system design philosophy

Keeping Eyes on the Prize

FAA is not interested in whether a new product is the “best on the market.” Instead, our goal is to make sure the system meets all the pertinent regulations, performs its intended function reliably, is intuitive to use, and is safe. While we don’t expect different products to function, look, or feel exactly the same to the pilot, we do strive to bring a level of standardization to the process. Pilots should be able to acceptably operate and use a system with minimal training.

The good news is that companies are beginning to recognize the benefit of getting the human factors experts involved early in the development and certification process. Many of the aircraft and avionics manufacturing companies have hired their own human factors experts. For those smaller companies that do not have the finances or resources to hire their own human factors experts, FAA steps in to ensure those companies consider the users’ capabilities and limitations throughout the design and development process.

Considering all the amazing technology making it into the panel these days, such as touch screens, voice-activated flight management systems, and other novel user interfaces, a proper HF evaluation has never been more important. There is nothing worse than developing a new gadget that few can use.

Ultimately, our avionics manufacturers agree that, even if the process is a little painful, it results in a better, more user-friendly product in the end. The next time you slide into that rental airplane with its vast array of advanced avionics or purchase that new avionics component, rest assured that a lot of thought and time went into the design of that box with you in mind. It’s not a perfect process; sometimes we just plain miss something. But, we take HF work very seriously as do the manufacturers. Yet, even with all this great HF effort in the design and approval process, pilots still must learn their systems to fly safely and must have a clear understanding of a system’s functions and limitations.

Jeff Holland is the human factors specialist and Wes Ryan is the manager of the FAA Small Airplane Directorate’s Programs and Procedures Branch.