Uncovering Resilient Behavior in the Aviation Safety Reporting System Using Large Language Models

Abstract

Verbatim from NASA NTRS. Not paraphrased, not summarized.

Resiliency is present in everyday life, both in system design and exhibited by the operators that function within these systems. This includes the National Airspace System (NAS) where pilots and controllers make positive decisions and take preventative or corrective actions every day even in unsafe situations. Pilot safety reports filed after an event are rich text narratives that detail the conditions around an event and can provide additional context leading up to, during, and describe how the situation was resolved. This yields useful insights into the resilient positive actions and corrective steps that may have transpired to prevented a safety incident from degrading further. Analyzing large archives of these reports can be impractical for subject matter experts to properly extract evidence of resilient behavior. However, Large Language Models have demonstrated the potential to extract useful insights from extensive bodies of text. This work proposes to utilize the Llama3.1 Instruct model to identify examples of resilient behavior within four categories on over 250,000 narratives from NASA's Aviation Safety Reporting System (ASRS). The analysis will reveal how similar and different resilient behaviors are present within various ASRS anomaly categories such as airborne conflict, near mid air collision, altitude deviation overshoot, runway excursion/incursions, and responses to external factors such as weather turbulence. Additionally, the analysis will compare resilient behavior between general aviation and commercial operation events as well as temporal trends within the archive of reports. The analysis aims to uncover how operators are practicing positive resilient behavior in situations described within the corpus of the report archive. This method provides a new lens into these valuable safety reports that can be used to inform and improve safety monitoring systems from this human resiliency perspective. The benefit can lead to highlighting operator proficiencies within the community and identify any knowledge gaps to ultimately improve safety within the NAS.

Author

• Bryan Matthews KBR (United States)

Keywords

• Large language model
• Aviation Safety Reporting System
• Aviation Safety
• Safety II
• Resilence