Scenario-Based Practical Test Standards

By Tom Glista
Reprinted with permission from FAA Aviation News

Question: How are most practical tests for a certificate or rating conducted?
Answer: Evaluating maneuvers and procedures.

For example in the private pilot certification, the applicant is asked to plan a cross-country flight, which he/she knows will not be fully utilized. Then the examiner administers an oral exam on regulations, aircraft systems, weather, airspace, performance, weight and balance, and aero-medical factors. Once past that hurdle, the applicant goes out to fly. The planned cross-country flight begins, but soon is diverted to another airport (which again may not be completed). From there the applicant is put under the hood for some maneuvers and then followed by visual steep turns and stalls. The examiner pulls the engine to idle and the applicant goes through engine failure procedures. When close to the ground, the applicant breaks off the landing and completes ground reference maneuvers, then flies back to his/her home airport for take-offs, landings, and go a rounds. If the applicant maintains certain parameters (+/- 100 feet, +/- 10 degrees, +10/-5 knots), he/she passes.

Is this how we fly day-today? Depending on which statistics you look at, between 75% and 85% of all general aviation fatal accidents are attributed to human factors (what we used to call “pilot error”). However, I believe most of them are due to the lack of Higher Order Thinking Skills (HOTS). These skills are aeronautical decision making, risk management, automation management, situational awareness, and Controlled Flight into Terrain (CFIT) awareness. Yes, the National Transportation Safety Board (NTSB) classifies many accidents on lack of “stick and rudder” skills. But, when a pilot loses control of a Cessna 172 on landing with a 35 knot cross-wind, is it really a stick and rudder skill problem or an aeronautical decision making and risk management problem? So to reduce accidents, we must be able to evaluate applicants’ Higher Order Thinking Skills. But how, during a maneuvers-based practical test, do you evaluate HOTS? What we need is a better way for examiners to evaluate the applicant’s HOTS.

We are always tweaking the FAA’s Practical Test Standards (PTS) to meet training and testing needs. For example, around 2002 the FAA started adding CFIT and Aeronautical Decision Making (ADM) as a special emphasis item. In the 1990s we incorporated a requirement that the examiner develop a written “plan of action” (POA). “The ‘plan of action’ shall include all TASKs in each AREA OF OPERATION…” The intent was that the practical test be conducted, as much as practical, as a scenario. Unfortunately, some examiners’ and inspectors’ written POAs were to conduct Area of Operation 1 first, Area of Operation 2 second, etc. Did it meet the letter of the PTS? Yes. Was it want we intended? No. What we need to do now, is clarify what we originally intended.

Why do we need to transform the PTS to a scenario-based test? Here are just a few reasons. Scenario-based training has shown to be more effective in developing HOTS than maneuvers-based training. The air carrier industry and military have been training and testing this way for decades, and the FAA/Industry Training Standards (FITS) program is promoting general aviation to adopt scenario-based training. Research into the FITS training methodologies show that it is more effective (click here), but there is disconnect between scenario-based training and a maneuvers-based test. Another reason is how, in a maneuvers-based test, does the inspector or examiner evaluate higher order thinking skills? Most of this kind of test is a rote stick and rudder skill test without decision making opportunities. A scenario allows the applicant to make decisions, thus the examiner/inspector can evaluate the applicant’s higher order thinking skills. Finally, scenario-based testing should help encourage the training community to adopt scenario-based training methodologies, including emphasis on HOTS. To help training providers, the FITS program has developed generic stand-alone syllabi for all of the tests. You can get free copies by clicking here.

The FITS Technical Team was tasked to develop recommendations for the FAA to transform the following PTS to a scenario-based test: Private Pilot Airplane Single and Multi-engine; Commercial Pilot Airplane Single and Multi-engine; Instrument Airplane; and Flight Instructor Single Engine, Multi-engine, and Instrument. These recommendations are complete. The recommendations included changes in the wording of the PTS, example scenarios, and a Judgment Assessment Matrix for each practical test.

Your next question is, “What’s a Judgment Assessment Matrix?” This matrix is a tool that breaks down the components of judgment into individual parts for scoring. Consequently some of the subjectiveness is taken out of evaluating judgment and makes it more objective. Shown is part of the DRAFT Private Pilot Airplane Single Engine Land and Sea Judgment Assessment Matrix. To use it, the examiner only needs to circle the tenet under the level of accomplishment that the applicant achieves (worst action-Red, Okay action-Yellow, Best action-Green) NOTE: This matrix may also be valuable to an instructor in his or her everyday teaching.

Another part of the work is teaching the inspectors and examiners how to give a scenario-based practical test. Under today’s PTS, an examiner can give scenario-based PTS. But I believe what is stopping them is lack of guidance and training. So the FITS team is also developing recommended changes to the FAA inspector guidance and the Designated Pilot and Flight Engineer Examiners’ Handbook (FAA Order 8710.3, as amended), and training for inspectors and examiners on how to construct and conduct a scenario-based practical test.

We are currently working towards implementation. First, as FITS Program Manager, I reviewed the recommended changes to the PTS developed by the FITS Technical Team. This included the example scenarios. Development of the Judgment Assessment Matrix included use and evaluation by flight instructors, checks pilots, and designated examiners. It went through at least 20 revisions before we came up with this format. Once this was done, all documents were sent to a FITS Review team for review and comment. This team includes representatives from across the GA spectrum including National Association of Flight Instructors (NAFI) Master Certificated Flight Instructors; representatives from the National Air Transportation Association (NATA), insurance industry, Aircraft Owners and Pilots Association (AOPA), AOPA-Air Safety Foundation (ASF), Cirrus Design, Cessna, major aviation universities, and small part 61 flight training providers; and pilot examiners. After those comments have been addressed, the final draft is sent back to me for final review and approval. This is where we stand today (I am writing this in November 2007).

The next step is to have the final recommendations reviewed by the General Aviation Joint Safety Committee’s Personal Transportation Subgroup. This subgroup includes members for the major general aviation manufacturers, AOPA, AOPA-ASF, NAFI, General Aviation Manufacturers Association (GAMA), Small Aircraft Manufacturers Association (SAMA), shared ownership organizations, insurance representatives, training developers, National Business Aircraft Association (NBAA), universities, part 61 pilot school representatives, and others. It also includes people from the FAA’s offices of Small Airplane Directorate, Airman Testing Standards, Certification and General Aviation Operations, Accident Investigation-Safety Analysis, and others. Once a consensus is reached, the changes can be implemented.

The Airman Testing Standards Branch, which is in charge of changes to the PTS, not only oversees the PTS, but several handbooks as well. Many of these handbooks and PTS are on a revision cycle (FYI-the last time the Private Pilot PTS was revised was in 2002.). Due to limited resources, we must work within their cycle to implement the changes. Additionally, we must get the training and changes to the guidance implemented at about the same time.

This change to a scenario-based PTS is really an enhancement and clarification of what the FAA intended back in the 1990s. Changes are coming, but don’t panic. They will be slow and purposeful. This time we will include all the tools to handle the changes.

Tom Glista is an Aviation Safety Inspector and the FITS Program Manager in Flight Standards Ser-vice’s General Aviation and Commercial Division.