FAA Order JO 7210.633A_ATO_Quality_Assurance_with_CHG_1

CHANGE U.S. DEPARTMENT OF TRANSPORTATION FEDERAL AVIATION ADMINISTRATION ORDER JO 7210.633A CHG 1 Effective Date: Air Traffic Organization Policy September 1, 2021 SUBJ: Air Traffic Organization (ATO) Quality Assurance 1. Purpose. This change updates the Aviation Risk Identification and Assessment (ARIA) definition to better address Air Traffic Organization’s (ATO’s) expanding use of ARIA, removes Appendix A since ATO Barrier Analysis is evolving, and makes editorial changes as needed. 2. Who the Change Affects. This change affects anyone using Federal Aviation Administration (FAA) Order JO 7210.633A, Air Traffic Organization (ATO) Quality Assurance. 3. Disposition of Transmittal. Retain this transmittal until superseded by a new basic order. PAGE CHANGE CONTROL CHART Remove Pages Dated Insert Pages Dated ii 10/1/20 ii 9/1/21 1-1 10/1/20 1-1 9/1/21 A-1 10/1/20 Remove 4. Explanation of Policy Changes. Removes Appendix A, Barrier Analysis, and redefines the ARIA definition in Chapter 1, paragraph 7a to be more inclusive, as follows: a. Aviation Risk Identification and Assessment (ARIA). An automated system that helps employ risk-based, data-driven decision-making facilitating better insight into potential risk in the National Airspace System (NAS). Glen A. Martin Vice President, Safety and Technical Training Air Traffic Organization

U.S. DEPARTMENT OF TRANSPORTATION FEDERAL AVIATION ADMINISTRATION ORDER JO 7210.633A Air Traffic Organization Policy Effective date: 10/01/2020 SUBJ: Air Traffic Organization (ATO) Quality Assurance (QA) Air Traffic Organization (ATO) Safety and Technical Training oversees the ATO’s quality assurance (QA) functions. QA’s primary function is collecting and analyzing data to identify systemic risk in the National Airspace System (NAS). QA is integral to the successful execution of the ATO Safety Management System (SMS). Historically, the ATO has based the identification of risk on whether or not an operation was compliant or that separation was maintained; however, this approach has not always identified all aspects of risk in the most effective manner. Similarly, compliant operations have, on occasion and under certain applications, introduced varying levels of risk into the NAS. Simply utilizing compliance as a measurement of risk has not been the most effective method to identify all underlying issues. Moving forward, the ATO will place more value on discovering why adverse safety occurrences happen through evaluating safety barriers and identifying risk, rather than determining who was at fault. The ATO has adopted a risk-focused approach to collecting and analyzing data, rather than gathering and evaluating data based solely upon existing compliance standards. Understanding the amount of risk associated with an operation is a holistic safety method that can proactively look beyond compliance. Risk-based safety addresses shortcomings without the boundaries of traditional compliance-based analysis. Glen A. Martin Vice President, Safety and Technical Training Air Traffic Organization Distribution: Electronic Initiated By: AJI-0

09/01/2021 JO 7210.633A CHG 1 Table of Contents Paragraph Page Chapter 1. General................................................................................................................... 1-1 1. Purpose of This Order. ................................................................................................... 1-1 2. Audience......................................................................................................................... 1-1 3. Where Can I Find This Order?....................................................................................... 1-1 4. Cancellation.................................................................................................................... 1-1 5. Explanation of Policy Changes. ..................................................................................... 1-1 6. Distribution..................................................................................................................... 1-1 7. Definitions...................................................................................................................... 1-1 8. Related Publications....................................................................................................... 1-2 9. How to Suggest Recommendations to Change this Order. ............................................ 1-2 Chapter 2. Safety Data............................................................................................................. 2-1 1. Introduction. ................................................................................................................... 2-1 2. Release of Safety Data. .................................................................................................. 2-1 3. Data Usage. .................................................................................................................... 2-1 4. Collection Methods. ....................................................................................................... 2-1 5. Information Requests. .................................................................................................... 2-2 6. Documentation Retention............................................................................................... 2-3 Chapter 3. Analysis .................................................................................................................. 3-1 1. AJI Responsibilities........................................................................................................ 3-1 2. Identification and Calculation. ....................................................................................... 3-1 3. Quality Assurance Validation. ....................................................................................... 3-1 4. ARIA and Barrier Analysis............................................................................................ 3-2 Chapter 4. Communications.................................................................................................... 4-1 1. Recurring Reports. ......................................................................................................... 4-1 2. Corrective Action Requests............................................................................................ 4-1 ii

09/01/2021 JO 7210.633A CHG 1 Chapter 1. General 1. Purpose of This Order. This order explains the responsibilities of the Air Traffic Organization (ATO) Safety and Technical Training (AJI) Quality Assurance (QA) office, and other offices within the ATO, for conducting safety reviews and data analysis, and identifying system trends and facilitating mitigations. 2. Audience. This order applies to all applicable ATO personnel, Federal Aviation Administration (FAA) contract tower employees, and anyone using ATO directives. 3. Where Can I Find This Order? This order is available on the MyFAA employee website at https://employees.faa.gov/tools_resources/orders_notices/ and the FAA website at https://www.faa.gov/regulations_policies/orders_notices/. 4. Cancelation. This order cancels FAA Order JO 7210.633, Air Traffic Organization Quality Assurance Program (QAP). 5. Explanation of Policy Changes. This revision removes the following: Measure of Compliance, Separation Conformance, Risk Analysis Event (RAE), and the process for notification and interviews associated with RAEs. This revision makes general organizational and editorial updates and incorporates the following processes: Aviation Risk Identification and Assessment (ARIA), Barrier Analysis Review (BAR), Combined Safety Barrier Review (CSBR), Preliminary ARIA Reports (PARs), and Referred ARIA Reports (RARs). 6. Distribution. This order is distributed to the following ATO service units: Air Traffic Services (AJT), Technical Operations Services (AJW), Mission Support Services (AJV), System Operations Services (AJR), and AJI. This order is also distributed to the following: Office of Accident Investigation and Prevention (AVP); Air Traffic Safety Oversight Service (AOV); Flight Standards Service (FS); NextGen (ANG); the William J. Hughes Technical Center; the Mike Monroney Aeronautical Center; National Air Traffic Controllers Association (NATCA); Professional Aviation Safety Specialists (PASS); National Association of Government Employees (NAGE); and the interested aviation public. 7. Definitions. a. Aviation Risk Identification and Assessment (ARIA). An automated system that helps employ risk-based, data-driven decision-making facilitating better insight into potential risk in the National Airspace System (NAS). b. Barrier Analysis Review (BAR). The process used to assess severity, likelihood, and barrier effectiveness in Referred ARIA Reports. Barrier analysis is also used to identify and assess factors (mitigating, aggravating, or observed) for air traffic operations where at least one aircraft is receiving Air Traffic Control (ATC) services. c. Closest Proximity. The smallest lateral distance between two airborne aircraft with the associated vertical value. 1-1

10/01/2020 JO 7210.633A d. Combined Safety Barrier Review (CSBR). A cooperative process between QA and facilities to gather additional information from subject matter experts and inform all concerned individuals about potential areas of risk in the system. This process utilizes aggregate data from BAR (if available) and includes facility stakeholders in an effort to identify, assess, and mitigate risk present in the operation. e. Corrective Action Request (CAR). CARs are formal requests initiating action to resolve an identified concern. f. Mandatory Occurrence Report (MOR). An occurrence involving ATO services for which the collection of associated safety-related data and conditions is mandatory. See FAA Order JO 7210.632, Air Traffic Organization Occurrence Reporting, Appendix A, for a full listing of MORs. g. Occurrence. Any observed or suspected event that meets the definition of an MOR. h. Preliminary ARIA Report (PAR). An initial report of an air traffic operation identified by ARIA for further review by QA personnel. i. Referred ARIA Report (RAR). Subset of Preliminary ARIA Reports identified for Barrier Analysis Review. j. Report. This refers to any safety data generated and recorded, generally in the Comprehensive Electronic Data Analysis and Reporting (CEDAR) application. k. Service Delivery Point (SDP). An air traffic control facility, flight service station, or staffed/unstaffed technical operations facility. 8. Related Publications. a. FAA Order JO 7210.634, Air Traffic Organization (ATO) Quality Control b. FAA Order JO 7210.632, Air Traffic Organization Occurrence Reporting c. FAA Order JO 1030.3, Initial Event Response d. FAA Order JO 8020.16, Air Traffic Organization Aircraft Accident and Aircraft Incident Notification, Investigation, and Reporting e. FAA Order JO 1350.14, Records and Information Management (RIM) f. FAA Order JO 7200.21, Partnership for Safety Program 9. How to Suggest Recommendations to Change this Order. Submit recommendations to change this order through email to the QA Group via [email protected]. AJI must review suggestions periodically. 1-2

10/01/2020 JO 7210.633A Chapter 2. Safety Data 1. Introduction. A successful safety system is highly dependent on accurate and timely data collection. AJI is responsible for collecting all safety-related data within the ATO. AJI will use collected data to properly identify and categorize reports, occurrences, and risks. Data will be analyzed for system trends and potential risks and will be primarily used to evaluate and possibly mitigate identified risk, including national orders, processes, and procedures. These data are not used as the basis for individual performance management, nor should it alone typically drive SDP quality control (QC) programs or actions. ARIA data or the suspected presence of risk in an operation may be used to initiate the CSBR process, which will generate a cooperative discussion between QA and an SDP. 2. Release of Safety Data. AJI is the sole ATO office responsible for coordinating the release of safety data to organizations outside the ATO. Other organizations must not release safety data outside the ATO without prior coordination with AJI. 3. Data Usage. AJI will collect safety data to: a. Accurately identify, categorize, and reconcile reported occurrences. b. Accurately analyze aircraft encounters. c. Identify suspected National Airspace System (NAS) risk trends. d. Assess the effectiveness of NAS risk mitigation actions. e. Analyze NAS services. f. Analyze NAS policies and procedures for compliance and effectiveness. 4. Collection Methods. AJI will collect safety data via the following methods: a. CEDAR. AJI will analyze data provided by ATC facilities and Technical Operations districts through the MOR submission processes or collected automatically via ARIA within the CEDAR application. b. Remote Collection. To minimize impact on ATC facilities’ and Technical Operations districts’ operations, AJI will conduct remote data collection to the greatest extent practical, using automation such as the National Offload Program, FALCON, ARIA, Performance Data Analysis and Reporting System, Digital Audio Legal Recorder, Remote Monitoring and Logging System (RMLS), etc. c. Additional Data Requests. AJI may request additional data, such as voice and radar/replay data or NAS services data. The purpose of such requests is to accurately analyze an operation/occurrence and is not intended to initiate a review of the operation/occurrence by the facility. AJI will indicate the format in which the information should be prepared, the scope of the information requested, and the method or means by which the information will be delivered. All data requests must be fulfilled within two administrative days of the request being received. 2-1

10/01/2020 JO 7210.633A d. ATC Facility and Technical Operations Services District Support of Analysis Data. The facility must ensure that all pertinent data are compiled and forwarded to AJI in support of the QA safety analysis process. Upon notification by AJI that additional data is required to assess the operation/occurrence, the facility must ensure the following: (1) If requested by AJI, all pertinent voice files must be attached to their associated reports in CEDAR within two administrative days. At a minimum, these voice files must include all communications with and about the aircraft involved in the reported operation/occurrence from either initial contact with the operating position responsible for the aircraft at the time of the operation/occurrence or five minutes before the operation/occurrence, whichever happens last. The voice files must continue until five minutes after the operation/occurrence or until final contact with the operating position, whichever happens first. (2) Include any recorded telephone conversations with involved flight crews concerning the operation/occurrence. Ensure that operation/occurrence-related communications with flight crews are conducted over recorded lines wherever available. (3) Although not required, every effort should be made to provide a bookmark with audio if the capability exists, in lieu of just providing a requested voice file. If FALCON is unavailable, or audio only is provided, attach audio data files that are .wma, .avi, or .mp3 format, due to file size and bandwidth restrictions. NOTE – Once notified by AJI that an operation/occurrence has been classified as a pilot deviation or is supporting other litigation, facility management must ensure that the voice recordings or acceptable waveform audio file (.wav) is retained according to applicable orders (see Chapter 2, Paragraph 6, Documentation Retention). NOTE – Technical Operations Services uses databases other than CEDAR. Until CEDAR is fully implemented throughout the ATO, AJI data requests will require that Technical Operations Services personnel pull data from existing databases. 5. Information Requests. Information requests provide data needed to make informed decisions. They are for gathering additional data and do not require corrective action. Requests to facilities or other ATO organizations may be either informal or formal, but formal processes are documented and tracked. Formal information requests may be generated in those situations where informal data gathering may be impractical, not expedient, or otherwise inappropriate. As a result of receiving a request, recipients may choose to request a CSBR and/or initiate QC processes, which could result in corrective action. a. AJI initiates formal information requests. The request initiates a bottom-up information exchange directed at the level most closely related to the issue. (1) Recipients must provide a response within the identified timeline. (2) If recipients do not provide a timely response, the issue may be escalated to a CAR. b. If the response confirms the identified safety issue, AJI will work with the respondent for resolution. If further action is required, the issue may be elevated to a CAR. 2-2

10/01/2020 JO 7210.633A 6. Documentation Retention. FAA Order JO 1350.14 provides general requirements for data and record retention. FAA Order JO 8020.16 provides retention requirements for aircraft accidents, aircraft incidents, litigation, and enforcement support. The following are the retention requirements for reports and occurrences covered by FAA Order JO 7210.632 and this directive: AJI must retain all data collected through the MOR and ARIA processes, in accordance with retention policies. 2-3

10/01/2020 JO 7210.633A Chapter 3. Analysis 1. AJI Responsibilities. a. Providing risk-based trend information, statistical data, recommendations, and other pertinent information to assist field facilities and other ATO organizations with their mitigation efforts. Information and data should be used in conjunction with field facility QC data to understand and mitigate areas of risk. b. Analyzing safety data from NAS Services performance data (e.g., RMLS, National Airspace Performance Reporting System). c. Examining and reconciling reports collected through automated and MOR processes to ensure: (1) The quality of the data is of the highest standard. (2) Accurate categorization of operations/occurrences. (3) Identification of air traffic incidents that must be reported to other organizations (e.g. pilot deviations, surface events, etc.). d. Conducting risk-based analysis of air traffic operations in accordance with this order and supporting AJI standard operating procedures. Ensuring that findings are made available to the ATO (e.g., observed and identified national trends, recommended mitigations). 2. Identification and Calculation. AJI will identify or calculate the following from reconciled data: a. Potential systemic risk. b. Applicable ATO and agency safety metrics. c. All runway incursions/excursions. d. Pilot deviations and near-midair collision reports, which AJI will forward to the responsible Flight Standards office. e. Vehicle and pedestrian deviations, which AJI will forward to the Airports Division and other affected organizations. f. Spillouts, military deviations, etc., which AJI will forward to the Department of Defense. 3. Quality Assurance Validation. 3-1

10/01/2020 JO 7210.633A a. ARIA Validation. QA will analyze operations and occurrences to support the identification of potential risk in the NAS, regardless of compliance with separation standards. PARs are captured and reviewed: (1) Automatically, based upon a predetermined set of data points contained in the QA Standard Operating Procedures. (2) Manually, as the result of additional referral methods into the BAR process (see Chapter 3, paragraph 4.b.(1)(a), Additional Referral Methods for BAR). NOTE – Data that does not meet the requirements to become a PAR will be retained for possible analysis. b. MOR Validation. QA validates MORs. Data from validated occurrences will be reviewed for potential systemic risk and are not a trigger to initiate performance management. 4. ARIA and Barrier Analysis. a. Administration of ARIA. AJI is responsible for the administration of ARIA. QA is responsible for identifying and validating risk in the NAS. ARIA is a tool that facilitates this activity at a systemic level. The ARIA software platform and associated algorithms are designed to make an initial finding of potential risk in the NAS. When an initial finding meets a specific set of data points, it is referred to BAR. BAR is conducted by subject matter experts, where measurements of risk associated with individual operations are quantified and associated factors are assigned. Aggregate BAR data is then used to accurately identify the presence of risk on a systemic level, determine mitigation efforts, and monitor their effectiveness. ARIA and the associated QA and QC processes protect the confidentiality of individuals involved in affected operations. b. Barrier Analysis Processes. Barrier analysis processes are the means by which ARIA data is evaluated to determine risk. (1) Barrier Analysis Review. BAR is the assessment of safety data to determine systemic risk. Additionally, the review captures aggravating, mitigating, and observed factors, and includes an evaluation of the resiliency of available safety barriers. The aggregate data produced by this process assists in the identification of systemic trends and potential risk in the NAS. (a) Additional Referral Methods for BAR. In addition to RARs, the following methods are included in BAR: i. Potential operational risk identified during the QA validation process. ii. Service Area QA Manager referral. iii. NATCA Service Area Safety Representative referral. iv. AJI Headquarters Manager referral. 3-2

10/01/2020 JO 7210.633A v. NATCA National Safety Representative referral. vi. Random selection by QA. (b) BAR Referral Requests. The following BAR referral requests are subject to approval through a collaborative review within the applicable Service Area QA office: i. AJT Management referral request. ii. Local Safety Council (LSC) referral request. (2) CSBR. A CSBR utilizes aggregate data from the BAR process, and other operational data, that may indicate potential risk in the NAS at the SDP level or above. The review will be conducted cooperatively between the QA office and the facility or facilities. The review utilizes multiple data sources to analyze and identify systemic risk. The identification of systemic risk within the NAS allows the ATO to mitigate and track operations that might pose a risk. (3) CSBR Referral. Participation in the CSBR process initiated by QA is mandatory. CSBR panels are established: (a) If more information is required to fully understand an operation being reviewed within the BAR process. (b) If potential systemic risk has been identified from the BAR process, and mitigation is believed to be warranted. (c) As the result of a safety concern identified during an on-site event review by an Event Response Team. REFERENCE − FAA Order JO 1030.3, Chapter 5, On-Site Event Review. (4) Requesting CSBR. The following requests are subject to approval through a collaborative review within the applicable Service Area QA office: (a) Facility request. (b) District request. (c) LSC request. (5) Documenting CSBR. The review process must be documented in CEDAR in the Combined Safety Barrier Review Tool (CSBRT). All areas, as noted in the CSBRT question tree, must then be considered, and related documentation and data used to support the outcome of the review must be referenced or attached in CEDAR in the tool. Data to consider should include: (a) Associated previous BAR information and results, to include aggregate data from previous RARs. 3-3

10/01/2020 JO 7210.633A (b) SDP QC data, to include: i. Service Review data. ii. QC Operational Safety Assessment data. iii. MOR data. iv. Current and previous Corrective Action Plan (CAP) information. v. Compliance Verification data. vi. Partnership for Safety, Safety Data Portal (facility must have an LSC). vii. Any other data deemed relevant by the collaborative team. (6) CSBR Resolution. Once an issue is understood (assessed and analyzed), facilities must collaboratively develop a CAP in accordance with FAA Order JO 7210.634 and the NATCA/FAA Collective Bargaining Agreement, or document why mitigation is not required, and close the CSBRT in CEDAR. 3-4

10/01/2020 JO 7210.633A Chapter 4. Communications 1. Recurring Reports. AJI will provide NAS safety data and trend findings as required by the Chief Operating Officer and other senior executives. These reports will also be provided to external organizations and labor unions, as appropriate. 2. Corrective Action Requests. CARs are formal requests initiating action to resolve an identified concern. The CAR process is initiated when it becomes unlikely that identified safety issues will (or should) be resolved informally. CARs begin a top-down process to inform the ATO of reported safety issues. a. CARs typically identify systemic safety issues and are rarely based on a single data point. b. All available information must accompany the request. c. Recipients must provide a response within the identified timeline. Although some issues are complex and require additional time to develop a comprehensive CAP, a response indicating the ATO’s intentions is still required within the identified timeline. d. If recipients do not provide a timely response, or if the recipient requests an extension, AJI will notify the appropriate Vice President. e. If AJI does not initially concur with the response, AJI will work with the respondents in order to achieve resolution. f. If the corrective action plan is sufficient, the issue will be closed. g. If AJI determines that the CAP does not mitigate the issue, AJI may elevate the matter to the ATO Deputy Chief Operating Officer to pursue appropriate action. 4-1