FAA Order JO 7210.634A_ATO_Quality_Control_with_CHG_1

CHANGE U.S. DEPARTMENT OF TRANSPORTATION JO 7210.634A CHG 1 FEDERAL AVIATION ADMINISTRATION Effective Date: Air Traffic Organization Policy September 1, 2021 SUBJ: Air Traffic Organization (ATO) Quality Control 1. Purpose of This Change. This change updates guidance for the Technical Operations Services Quality Control Program and makes other editorial changes throughout the document. 2. Audience. This change applies to the following Air Traffic Organization (ATO) service units: Technical Operations Services (AJW), Safety and Technical Training (AJI), Mission Support Services (AJV), System Operations Services (AJR), and Air Traffic Services (AJT). 3. Where Can I Find This Change? This order is available on the MyFAA Employee website at https://employees.faa.gov/tools_resources/orders_notices/ and the Federal Aviation Administration website at http://www.faa.gov/regulations_policies/orders_notices/. 4. Explanation of Policy Change. a. Cancels the System Service Review Guidance memorandum signed by the Vice President of Technical Operations Services on May 2, 2018, incorporating all pertinent guidance into this Change. b. Replaces Chapter 8, Technical Operations Quality Control Programs. c. Adds new Appendix F, Technical Operations Services System Service Review (SSR), Corrective Action Plan (CAP), and Systemic Issue Review (SYSIR) Templates. d. Updates the definition of Aviation Risk Identification and Assessment (ARIA). e. Replaces references to Performance Skill Checks with Performance Assessments. f. Deletes a legacy reference to Electronic Occurrence Report (EOR). 5. Distribution. This change is distributed to the following ATO service units: AJT, AJW, AJR, AJI, and AJV. In addition, the order is distributed to the following: the Air Traffic Safety Oversight Service (AOV), the William J. Hughes Technical Center, the Mike Monroney Aeronautical Center, the National Air Traffic Controllers Association (NATCA), Professional Aviation Safety Specialists (PASS), the National Association of Government Employees (NAGE), and to the interested aviation public. 6. Background. This change updates the Technical Operations Services Quality Control (QC) Program by providing specific procedures and processes for SSR, SYSIR, CAP, and Compliance Distribution: Electronic Initiated By: AJI-0

09/01/2021 JO 7210.634A CHG 1 Verification used to measure the quality of AJW products and services. Additionally, this change incorporates several minor editorial updates and a change to the ARIA definition. 7. Disposition of Transmittal. Retain this transmittal until superseded by a new basic order. 8. Page Control Chart. See below. PAGE CHANGE CONTROL CHART Remove Pages Dated ii and iii 10/1/20 1-1 10/1/20 1-3 through 1-6 10/1/20 2-3 and 2-4 10/1/20 4-11 and 4-12 10/1/20 6-1 through 6-3 10/1/20 8-1 through 8-3 10/1/20 B-1 10/1/20 C-5 10/1/20 D-4 10/1/20 E-1 and E-2 10/1/20 Insert Pages Dated ii and iii 9/1/21 1-1 9/1/21 1-3 through 1-6 9/1/21 2-3 and 2-4 9/1/21 4-11 and 4-12 9/1/21 6-1 through 6-3 9/1/21 8-1 through 8-6 9/1/21 B-1 9/1/21 C-5 9/1/21 D-4 9/1/21 E-1 and E-2 9/1/21 F-1 through F-6 9/1/21 Glenn A. Martin Vice President, Safety and Technical Training Air Traffic Organization Distribution: Electronic Initiated By: AJI-0

U.S. DEPARTMENT OF TRANSPORTATION FEDERAL AVIATION ADMINISTRATION ORDER JO 7210.634A Air Traffic Organization Policy Effective date: 10/01/2020 SUBJ: Air Traffic Organization (ATO) Quality Control The ATO has moved to a systemic outlook of safety within the National Airspace System. This outlook places more value on discovering why adverse safety occurrences happen, and identifying risk, rather than determining who was at fault. Historically, the Federal Aviation Administration (FAA) has experienced external and internal organizational imperatives that have encouraged a safety system focused on the outcome of individual incidents, and required the rapid reporting of preliminary incident information. While the FAA has enjoyed success under this past safety model, our current Safety Management System provides for a more measured and systemic approach to safety risk analysis. Quality control is an essential component of this system. Glen A. Martin Vice President, Safety and Technical Training Air Traffic Organization Distribution: Electronic Initiated By: AJI-0

09/01/2021 JO 7210.634A CHG 1 Table of Contents Paragraph Page Chapter 1. General…………………………………………………………………………….1-1 1-1. Purpose of This Order………………….…...………………………………………… 1-1 1-2. Audience…………………………………………..……………………………...…… 1-1 1-3. Where Can I Find This Order? ……………………………………………………….. 1-1 1-4. Cancellation…………………………………………………………………………….1-1 1-5. Explanation of Policy Changes……………………………………………………….. 1-1 1-6. Distribution …………………………………………………………………………… 1-1 1-7. Organizational Responsibilities……………………………………………………….. 1-1 1-8. Definitions…………………………………………………………………………….. 1-3 1-9. Related Publications…………………………………………………………………... 1-6 Chapter 2. Quality Control Model ………………………………………………………….. 2-1 2-1. Quality Control (QC) Model ……………………………..…………………………... 2-1 2-2. Facility/District QC Orders …………………………….…………………………….. 2-3 2-3. Local (Facility) Safety Reports ………………………….…………………………… 2-4 Chapter 3. Quality Control Monitoring ……………………………………………………. 3-1 3-1. Introduction …………………………………………………………………………... 3-1 3-2. QC Operational Skills Assessment (OSA) …………………………………………… 3-2 3-3. Emphasis Items (EIs) ………………………………………………………………… 3-3 Chapter 4. Service Reviews…………………………………………………………………... 4-1 4-1. Introduction …………………………………………………………………………... 4-1 4-2. SSR ………………………………………………………………………………….... 4-2 4-3. TMR …………………………………………………………………………….......... 4-5 4-4. CER ……………………………………………………………………………........... 4-7 4-5. SYSIR …………………………………………………………………………........... 4-9 Chapter 5. Compliance Verification ……………………………………………………....... 5-1 5-1. Compliance Verifications (CVs) ……………………………………………………... 5-1 5-2. ICV …………………………………………………………………………………… 5-1 5-3. ECV …………………………………………………………………………………... 5-2 5-4. Findings ………………………………………………………………………………. 5-2 ii

09/01/2021 JO 7210.634A CHG 1 Paragraph Page 5-5. Responding to Findings ……………………………………………………………..... 5-3 Chapter 6. Quality Control Validations ………………..…………………………………... 6-1 6-1. Quality Control Validations (QCVs) ............................................................................. 6-1 Chapter 7. Corrective Action Plans ........................................................................................ 7-1 7-1. Introduction. .................................................................................................................. 7-1 7-2. Develop and Implement CAPs. ..................................................................................... 7-1 7-3. Monitoring. .................................................................................................................... 7-2 7-4. Documentation............................................................................................................... 7-4 Chapter 8. Technical Operations Services Quality Control Program ................................. 8-1 8-1. Purpose .......................................................................................................................... 8-1 8-2. Technical Operations Services System Service Review (SSR)..................................... 8-1 8-3. Technical Operations Services Systemic Issue Review (SYSIR).................................. 8-4 8-4. Technical Operations Services Corrective Action Plan (CAP) ……………….……… 8-4 8-5. Technical Operations Services Compliance Verification (CV) .................................... 8-5 Appendix A. Alaska Flight Service Station (FSS) and U.S. NOTAM Office (USNOF) Quality Control Compliance Verification (CV) .................................................................... A-1 Appendix B. Federal Contract Flight Service Station (FCFSS) Quality Control Appendix E. Sample Fiscal Year (by Month) QC Activity Plan Large .............. B-1 Appendix C. Quality Control Elements ................................................................................. C-1 Appendix D. Generic Facility Quality Control Order Large Tower/TRACON ................ D-1 Tower / TRACON ………………………………………………………………………. E-1 Appendix F. Technical Operations Services SSR, CAP, and SYSIR Templates ………... F-1 iii

09/01/2021 JO 7210.634A CHG 1 Chapter 1. General 1-1. Purpose of This Order. The purpose of quality control, as defined in the Air Traffic Organization (ATO), is to assess the output (whether a product or service) of a particular process or function and identify any deficiencies, risk, or problems that need to be addressed. Within this quality control concept, it is a primary responsibility to take action, particularly at the Service Delivery Point (SDP), to ensure that these products or services meet the requirements of the SDP and the ATO organizationally. This order outlines the processes and steps used to ensure the quality of products and services provided at the SDP level on an ongoing basis. 1-2. Audience. The chapters of this order apply to ATO service units as follows: Chapters 1–7 and related Appendices: Air Traffic Services, System Operations Services; Chapter 1, Chapter 8, and Appendix F: Technical Operations Services; and this entire order is applicable to Mission Support Services due to their unique support role for the service units. 1-3. Where Can I Find This Order? This order is available on the MyFAA Employee website at https://employees.faa.gov/tools_resources/orders_notices/ and the Federal Aviation Administration (FAA) website at http://www.faa.gov/regulations_policies/orders_notices/. 1-4. Cancellation. This order cancels FAA Order JO 7210.634, Air Traffic Organization (ATO) Quality Control. 1-5. Explanation of Policy Changes. This revision adds and deletes definitions in paragraph 1-8; incorporates Aviation Risk Identification and Assessment (ARIA), Barrier Analysis Review (BAR), Preliminary ARIA Report (PAR), Referred ARIA Report (RAR), and Combined Safety Barrier Review (CSBR); clarifies organizational responsibilities; adds Chapter 2 and the Quality Control Model; clarifies requirements of the Service Review and Compliance Verification (CV) processes; incorporates ATO Safety Guidance ATO-SG-15-03 and ATO-SG-12-05; incorporates Corrective Action Plans (CAPs) and processes; deletes Quality Control Checks and associated processes; adds Performance Assessment Validations; adds and updates the charts contained in the Appendices; and makes general organizational and editorial updates. 1-6. Distribution. This order is distributed to the following ATO service units: Air Traffic Services (AJT), Technical Operations Services (AJW), System Operations Services (AJR), Safety and Technical Training (AJI), and Mission Support Services (AJV). In addition, the order is distributed to the following: the Air Traffic Safety Oversight Service (AOV), the William J. Hughes Technical Center, the Mike Monroney Aeronautical Center, the National Air Traffic Controllers Association (NATCA), Professional Aviation Safety Specialists (PASS), the National Association of Government Employees (NAGE), and the interested aviation public. 1-7. Organizational Responsibilities. a. Vice Presidents, or designees, of AJT, AJR, AJW, and AJV must: (1) Develop all policies and procedures related to quality control. (2) Ensure that their respective organizations comply with the requirements of this order. 1-1

10/01/2020 JO 7210.634A (3) Provide oversight and support to their respective Director(s) of Operations, Service Center Directors, and Deputy Directors of System Operations to ensure they are able to meet their quality control requirements as described in this order, and ensure they are responding to data indicators of non-compliance and risk. (4) Continually review available quality control data and information and develop initiatives and/or take actions(s) when appropriate. (5) Coordinate with other ATO service units, as appropriate, when developing initiatives or taking actions(s) to ensure organizational consistency and effective resource management and communication. (6) Review safety data analysis provided by ATO Safety and Technical Training, and initiate action(s) when appropriate. Communicate the results of such reviews to ATO Safety and Technical Training. (7) Respond to requests and/or actions initiated by ATO Safety and Technical Training in their quality assurance role. b. Directors of Operations must: (1) Ensure their organization complies with the requirements of this order. (2) Provide oversight and support to their respective General Managers, Technical Operations District/Group Managers, and/or SDP Managers to ensure they are able to meet and comply with their quality control requirements as described in this order. (3) Continually review available quality control data and information and develop initiatives and/or take action(s) when appropriate. (4) Coordinate with other ATO organizations within their service area (Air Traffic Services, Technical Operations Services, the Service Center, ATO Safety and Technical Training, etc.) and service unit headquarters staff as appropriate when developing initiatives or taking actions(s) to ensure organizational consistency and effective resource management and communication. (5) Examine operational compliance and safety related data provided by Service Center Directors, delivered by the applicable service center group(s), and initiate action(s) when appropriate. (6) Review data analysis provided by ATO Safety and Technical Training and initiate action(s) when appropriate. Communicate the results of such reviews to ATO Safety and Technical Training. (7) Respond to requests and/or actions initiated by ATO Safety and Technical Training in their quality assurance role. c. Service Center Directors must: 1-2

09/01/2021 JO 7210.634A CHG 1 (1) Ensure that their organization complies with the requirements of this order. (2) Coordinate with the Director(s) of Operations and assist and ensure that the Director(s) of Operations meet their quality control requirements as described in this order. (3) Ensure that Service Center groups coordinate their actions to support Service Center requirements. d. SDP Manager, General Manager, Assistant General Managers, and Technical Operations Services District/Group Managers must: (1) Ensure that their respective organization complies with the requirements of this order. (2) Provide oversight and support to their subordinate managers to ensure that they are able to meet their quality control requirements as described in this order. (3) Continually review available quality control data and information and develop initiatives and/or take actions when appropriate. (4) Coordinate with their Director of Operations and/or Service Center Quality Control Group, and other SDPs as appropriate, when developing initiatives or taking actions to ensure organizational consistency and effective resource management and communication. (5) Respond to requests and/or actions initiated by Safety and Technical Training in their quality assurance role. e. Flight Service Stations (FSSs) / U.S. NOTAM Office / Federal Contract Flight Service Stations (FSSs) must conduct quality control measures according to Appendix A and Appendix B, respectively. 1-8. Definitions. a. Aviation Risk Identification and Assessment (ARIA). An automated system that helps employ risk-based, data-driven decision-making facilitating better insight into potential risk in the National Airspace System (NAS). b. Barrier Analysis Review (BAR). The process used to assess severity, likelihood, and barrier effectiveness in Referred ARIA Reports. Barrier analysis is also used to identify and assess factors (mitigating, aggravating, or observed) for air traffic operations where at least one aircraft is receiving Air Traffic Control (ATC) services. c. BAR Report. The output of the BAR process. d. Checklists. Checklists are used as minimum guidance in preparing for and conducting Internal Compliance Verifications (ICVs) / External Compliance Verifications (ECVs). e. Combined Safety Barrier Review Output. The resulting output from the Combined Safety Barrier Review process. 1-3

09/01/2021 JO 7210.634A CHG 1 f. Combined Safety Barrier Review (CSBR). A cooperative process between Quality Assurance (QA) and facilities to gather additional information from subject matter experts and inform all concerned individuals about potential areas of risk in the system. This process utilizes aggregate data from BAR (if available) and includes facility stakeholders in an effort to identify, assess, and mitigate risk present in the operation. g. Compliance Verification Tool (CVT). A national database that contains information related to the compliance verification process. Information includes checklists, reports, facility information, tracking information, response data, and other statistical information available on the CVT website. Information contained in reports, any corrective action, status reports, and closure is submitted through this database system. h. Comprehensive Electronic Data Analysis and Reporting (CEDAR). A web-based, comprehensive data reporting, collection, and analysis tool used by both Quality Control (QC) and QA to record data associated with their respective organizational responsibilities. i. CEDAR Question Tree. An electronic entry method used within the quality control service review processes to capture specific data points regarding information about operational circumstances during the period under review. Question trees are based on a user-friendly question and answer format that guides the user through the electronic “form.” j. Conformity Index (CI). This item applies to federal and federal contract flight service stations only. Each on-site ICV/ECV conducted by the Flight Service Directorate must include a CI. The CI must essentially be the result of aggregating the weighted indices for each of the functional areas (system safety, system efficiency, and system management) on the national checklist. System safety is weighted more than the other functional areas. Instructions for calculating the CI are available from the CVT. The Flight Service Directorate acknowledges that no two facilities are identical; therefore, CIs are not intended to compare facilities. The intent of the CI is to numerically depict a facility’s overall compliance with directives/regulations and to assist with identifying “at risk” facilities for non-compliance. k. Corrective Action Plan (CAP). CAPs are collaborative activities enacted to correct non-compliance and areas of risk that have been properly identified, validated, and understood through data collection and analysis. A CAP contains a description of the mitigation actions, the scope of the CAP, a timeframe for completion, a defined monitoring plan, and a defined effectiveness target. l. Desk Audit. A desk audit is an off-site method of assessing checklist and off-checklist items. It is accomplished through discussion with facility personnel and/or review of requested or obtained recordings, data, and/or documentation. m. Direct Monitoring. Monitoring an operational position real-time from the same position (for example, monitoring arrival east while physically sitting at arrival east). 1-4

09/01/2021 JO 7210.634A CHG 1 n. External Compliance Verification (ECV). An externally initiated assessment of a facility, conducted primarily by the Quality Control Group (QCG) and/or additional personnel, in response to data-driven indicators of potential risk and/or practices. An ECV may be conducted on-site, using a customized checklist, to assess a facility’s overall performance. ECVs are conducted on an as-needed basis as determined via indicators of potential risk and non-compliance. o. Internal Compliance Verification (ICV). A facility’s self-evaluation that is conducted by the facility / designated personnel using the checklists contained in the CVT and procedures outlined in this directive. p. Off-Checklist Item. An assessed item that is not specifically identified on a national checklist. q. Operational Skills Assessment (OSA) Worksheet. An electronic worksheet in CEDAR used to document an individual’s technical performance. r. Playback Monitoring. Indirectly monitoring an operational position at any date/time other than live/real-time using available playback tools (Falcon, etc.) synchronized with a voice file. s. Preliminary ARIA Report (PAR). An initial report of an air traffic operation identified by ARIA for further review by QA personnel. t. Quality Assurance Group (QAG). The office in each service area responsible for conducting occurrence classification, identifying and categorizing air traffic incidents, identifying reports for barrier analysis, and conducting barrier analysis reviews in support of the ATO Top 5 and other safety processes. The mission of the QAGs may encompass other duties, including identifying significant safety risk trends and identifying potential significant events. u. Quality Control Group (QCG). The office in each service center responsible for conducting data-driven external compliance verifications, identifying and delivering SDP non-compliance/risk data to the Directors of Operation and SDPs, assessing the effectiveness of SDP corrective actions, facilitating the significant event process, and monitoring SDP QC programs while ensuring that they are in compliance and completed in accordance with directives. v. Referred ARIA Report (RAR). Subset of Preliminary ARIA Reports identified for BAR. w. Remote Monitoring. Indirectly monitoring an operational position in real-time from a remote location (watch desk, operations supervisor desk, other operational position, etc.). x. Service Delivery Point (SDP). An air traffic control facility, flight service station, or staffed/unstaffed technical operations facility. y. Significant Safety Risk. Identified ATO exposure to risk that has a high likelihood of occurrence and/or severity. 1-5

09/01/2021 JO 7210.634A CHG 1 z. Special Evaluation. This item applies to federal contract flight service stations only and assesses specific areas, programs, offices, or organizations as directed by the Flight Service Directorate. aa. Systemic. An identified safety concern that has the potential to introduce risk into the national airspace system at the local, district, regional, service area, or national level. bb. Voluntary Safety Reporting Program (VSRP). A voluntary program that provides a confidential, non-punitive mechanism for employees to report safety events and problems. VSRPs use employee input to identify leading indicators and significant safety concerns and issues, operational deficiencies, non-compliance with regulations, deviations from policies and procedures, and potential risk in the system. Qualitative data received through this program helps identify indicators of potential hazards and areas of risk in the NAS. 1-9. Related Publications. a. FAA Order JO 7210.632, Air Traffic Organization Occurrence Reporting. b. FAA Order JO 7210.633, Air Traffic Organization (ATO) Quality Assurance (QA). c. FAA Order JO 3400.20, Individual Performance Management (IPM) for Operational Personnel. d. FAA Order JO 1030.3, Initial Event Response. e. FAA Order JO 7200.21, Partnership for Safety Program. f. FAA Order JO 7200.20, Voluntary Safety Reporting Programs. g. FAA Order JO 3120.4, Air Traffic Technical Training. h. FAA Order JO 7110.65, Air Traffic Control. i. FAA Order JO 1000.37, Air Traffic Organization Safety Management System. j. FAA Order JO 7110.10, Flight Services. k. FAA Order JO 6040.6, National Airspace System Technical Evaluation Program. l. FAA Order 6000.15, General Maintenance Handbook for National Airspace System (NAS) Facilities. 1-6

10/01/2020 JO 7210.634A Chapter 2. Quality Control Model 2-1. Quality Control (QC) Model. a. A strong QC program integrates five QC elements: the five-step process, Local (facility) Safety Reports, the Local Safety Council (LSC), local QC orders, and a QC activity plan. The core of these five elements is the five-step process. Together, the five elements and the five-step process form a dynamic “5 x 5” QC program model. The five-step process is a continuous and cyclical process. The five-step process and additional QC elements are listed below and displayed in Figure 2-1. More information about QC Elements are contained in Appendix C. (1) Collect data (supports initial identification of non-compliance or risk, and monitors implemented corrective actions). (2) Validate and understand potential facility problems/issues (assess/analyze). (3) Develop and implement Corrective Action Plans (CAPs). (4) Document findings/actions. (5) Review data for integrity. Figure 2-1: The Five-Step Process/5 QC Elements 2-1

10/01/2020 JO 7210.634A b. It is important to recognize that Individual Performance Management (IPM) is not a component of QC. IPM is a separate but parallel process that contributes to a strong proactive safety management system. Field facilities must operate QC processes as a systemic (de­ identified) data collection method and perform IPM as a separate process following the specific steps outlined in FAA Order JO 3400.20. (1) QC processes, procedures, and policies that use direct employee (identified) observation as a foundation are not in compliance. (2) Facilities must not establish a QC program that requires identified observation of employees outside of approved IPM OSA processes. (3) Facilities must not use QC activities to generate, or as the source of, IPM follow-up activities. (4) QC activities collect and analyze systemic compliance, risk, and drift and are not to be attributed to or identify an individual employee. c. Below is an outline of the five-step process. (1) Collect Data. Facilities must collect/review data through the following: (a) QC OSAs (b) Emphasis Items (c) Compliance Verifications (d) Partnership for Safety (PFS) Safety Data Portal (if applicable; must have a LSC) (e) Mandatory Occurrence Report (MOR) and Barrier Analysis Review (BAR) data (if applicable) to identify potential patterns or trends that may be indicators of facility non­ compliance and risk. The review should not exclusively focus on loss of separation occurrences. It is expected that MOR and BAR data be reviewed on a regular basis, however, based on the volume of potential occurrences that can be generated at some facilities, this review is not intended to create a workload that consumes all QC efforts or resources. (f) MORs and BAR data must not be the primary basis of a facility QC program. Facilities must ensure they comply with the requirements prescribed in the NATCA/FAA Collective Bargaining Agreement when using MOR data to support IPM activities. (g) Facilities should review aggregate groupings or trends of factors identified in barrier analysis and use them in conjunction with other QC data to provide additional information that will enable them to better focus their service reviews or to create Emphasis Items to collect additional data. (2) Validate and Understand Facility Problems (Assess/Analyze). Facilities must ensure they properly understand potential identified problems by rigorously assessing collected 2-2

09/01/2021 JO 7210.634A CHG 1 data through standardized processes. Before developing a CAP or invalidating a potential systemic issue, facilities must assess potential issues by using one of the following processes: (a) Service reviews. (b) Compliance verifications. (c) CSBR. Requests for CSBR are made through the respective service area QA office. REFERENCE– FAA Order JO 7210.633, Chapter 3, Requesting Combined Safety Barrier Review. NOTE– It is understood that with some significant events or compliance issues, facilities may need to implement corrective actions prior to conducting an assessment. This should happen only under unusual circumstances. When this occurs, a service review must be conducted as soon as possible to validate or modify the issue and CAP. (3) Develop and Implement CAPs. Once a problem is understood (assessed and analyzed), facilities must collaboratively develop a CAP. CAPs must be designed to address the specific problem and be implemented throughout the facility or applicable operational area. In addition, CAPs must include how the effectiveness of implemented mitigations will be assessed. Facilities must monitor implemented CAPs as they continue to collect data. This can be done through performing QC OSAs, designing specific Emphasis Items to assess a specific CAP, performing ICVs, reviewing reported/detected occurrence data, reviewing BAR/CSBR data, or analyzing data available in the PFS Portal through the LSC. (4) Document. Facilities must document CAPs within CEDAR to maintain a record of implemented corrections for mitigation monitoring and effectiveness determinations. CAPs resulting from the ICV and ECV processes are documented in the Compliance Verification Tool. LSC corrective actions, safety information, and problems should be documented in CEDAR and shared in ATC InfoHub. (5) Data Integrity. Facilities must ensure that data collected through QC OSAs and Emphasis Items accurately reflect demonstrated technical performance. In addition, facilities must validate documentation associated with On-the-Job Training (OJT) and Certification Skill Checks / Performance Assessments to ensure that these processes accurately reflect facility performance. This ensures a solid foundation of data upon which CAPs are built. 2-2. Facility/District QC Orders. a. Facilities and/or districts must not create QC orders that either duplicate or contradict the national order or collective bargaining agreement. b. Facility/district QC orders (see Appendix D) may only contain the following elements: (1) QC OSA sampling plan (see Appendix D). 2-3

09/01/2021 JO 7210.634A CHG 1 (2) Plan for conducting random/scheduled System Service Reviews (SSRs) (and Traffic Management Reviews (TMRs) for facilities with Traffic Management Units). (3) Designation of points of contact for Systemic Issue Reviews (SYSIRs). (4) QC OSA Validation sampling plan to be documented in CEDAR. (5) Certification Skill Check and Performance Assessment Validation process to be documented in CEDAR. (6) OJT Documentation Validation process to be documented in CEDAR. (7) Requirements for recurring reports on performed QC processes, results of analyses of safety data, implemented corrective action plans, and data monitoring activities. 2-3. Local (Facility) Safety Reports. a. Overview. (1) Facilities are encouraged to create regular reports of collected data to support the identification of non-compliance and risk, provide visibility into facility performance, and ensure that the facility’s QC program is operating as intended. Reports can be generated monthly, quarterly, or at some other frequency depending on the available resources, size, and complexity of the facility; the amount of available data; and identified compliance issues. A sample of possible facility safety report data is contained in Appendix D, and an example facility QC Activity Plan is contained in Appendix E. (2) Facilities are encouraged to include the data listed below in their recurring safety reports. (a) Status: This section should list the status of required QC processes. The following should be considered: i. QC OSAs conducted during the reporting period. ii. Emphasis Items (include pre-existing Emphasis Items, new Emphasis Items created during the reporting period, and Emphasis Items closed during the reporting period). iii. Service reviews conducted during the reporting period (include each different service review (SSR, Covered Event Review, TMR, SYSIR) and the reason for each). iv. Status of current fiscal year ICV (include the percentage of items completed and remaining). v. QC Validation information (include numbers of items reviewed and whether this is in accordance with the facility plan for each). 2-4

10/01/2020 JO 7210.634A (b) Facility reporting data: This section is intended to provide information on the health of the proactive safety culture of the facility and assist in trend identification when used with QC data and analyzed through the service review process. The following data should be considered: i. Trends found through review of MORs reported during a specific time period ii. Trends found through the BAR process. (c) Findings: This section should provide information on validated instances of systemic facility non-compliance and areas of risk, derived from available data sources. The following findings should be considered: i. Findings from service reviews (the report should include all topic areas from the CEDAR question tree in which the service review team validated an issue and include a brief synopsis on selected (most serious) identified issues) ii. Findings from the ICV (include any items rated non-compliant and their current status (open/closed)) iii. Findings from Quality Control Validations (QCVs) (include any compliance issues identified) iv. Findings from CSBR (d) CAPs. This section should include information on all facility CAPs regardless of the triggering process (for example, CAPs created from service reviews, CSBR, ICV/ECVs, and/or QCVs). The following types of CAPs should be considered. i. New CAPs created during the reporting period (include the associated monitoring plans) ii. CAPs closed during the reporting period (include data indicating how the CAP data target was met) iii. Status of pre-existing CAPs not closed during the reporting period (include the status of the associated monitoring plan(s) and any preliminary findings, if available) NOTE– Some of this information can be obtained via automated report in the ATO Manager Dashboard (some Service Delivery Points may be excluded). The report is accessed via the ATO Application Portal. 2-5

10/01/2020 JO 7210.634A Chapter 3. Quality Control Monitoring 3-1. Introduction. a. Quality Control (QC) monitoring is a key data-gathering component of an SDP QC program. Through QC monitoring, an SDP can identify areas of success and system risk by measuring compliance to requirements in FAA directives. QC monitoring collects technical performance data that measures facility performance. These data support other quality control processes that assess training, procedures, airspace, directives, equipment, and the technical performance of personnel. b. QC processes are separate from individual performance management. The purpose of data collected through QC monitoring is to measure organizational performance. 3-2. QC Operational Skills Assessment (OSA). A QC OSA is a QC sampling method used to collect data by assessing the technical performance demonstrated by operational personnel. A QC OSA must not be used as the basis of, or to support, IPM. a . When to Perform a QC OSA. Sufficient QC OSAs must be performed to provide a valid quality control sample of the various SDP positions and functions. Each SDP must establish a sampling plan that includes random and scheduled activities. QC OSA sampling plans must assess each type of operational/control position including watch supervision functions and traffic management functions. NOTE– The facility sampling plan for QC OSAs must ensure that all operational positions are assessed, and is not based on individuals. There is no requirement to ensure that individuals have received a QC OSA. OSAs are only conducted on positions being worked by position certified individuals. EXAMPLE– 1. An ATC-12 TRACON conducts a random QC OSA on the departure east position from 1620-1700UTC. 2. An ARTCC schedules a minimum of six (6) QC OSAs each week on specific positions, in a specific area, and for defined time periods. (1) Facilities are required to conduct a number of QC OSAs that will ensure an accurate assessment of facility performance. The quarterly targets, listed in Figure 3-1 and Figure 3-2 below, provide a 95% confidence level that QC OSA data collected will meet this accuracy threshold. The Air Traffic Control System Command Center (ATCSCC) will use the same target OSA numbers as noted for the ATCSCC/Terminal Quarterly QC OSA Plan by Facility, Figure 3-1. Quarterly targets were developed by analyzing facility types and the associated numbers of fully certified operational personnel or operational areas. Applying this analysis to a statistical model provides the resultant QC OSA target value. NOTE– Quarterly targets for fully certified operational personnel and operational areas, Figures 3-1 and 3-2, are determined at the beginning of each fiscal year. Once determined, that quarterly target is to be used for the remainder of the fiscal year. 3-1

10/01/2020 JO 7210.634A Figure 3-1: ATCSCC/Terminal Quarterly QC OSA Plan by Facility ATCSCC/Terminal Quarterly QC OSA Plan by Facility CPC Count Only OSA Target 0-50 Half of Staff 51-75 31 76-100 41 101-125 52 126-150 61 151-175 70 176-200 79 201-225 87 226-250 95 Figure 3-2: En Route Quarterly QC OSA Plan by Facility En Route Quarterly QC OSA Plan by Facility Number of Operational Areas Quarterly OSA Target (25 x # of Areas) 8 200 7 175 6 150 5 125 4 100 ZAN, ZSU, ZUA Half of Staff b. How to Perform a QC OSA. QC OSAs may be conducted via real-time remote monitoring, real-time direct monitoring, or via playback tools (playback monitoring). Facilities are encouraged to use available playback tools/monitoring as their primary method of conducting QC OSAs. QC OSAs contain the following steps: (1) Review. Employees conducting the review segment of a QC OSA must identify and record all technical performance demonstrated during the session, whether exemplary, meeting standards (compliant), and/or deficient (needs improvement). Reviewers are also expected to be mindful of, and identify, potential systemic issues associated with training, efficiency, airspace, 3-2

10/01/2020 JO 7210.634A procedures, directives, and equipment, when applicable. Potential systemic issues are addressed through the systemic issue review process. (2) Documentation. Results of QC OSAs must be documented in the OSA module in CEDAR. All identified items must be documented in the OSA. Any potential systemic issues are collected within the OSA entry. Select the applicable systemic issue option, and enter a brief summary. Items not observed during the OSA should be left blank. For observed items, select the following category: (a) Exemplary: the QC OSA item was completed in accordance with FAA Orders and environmental conditions or other factors made it very challenging, or the employee went above and beyond what was required. (b) Compliant: the QC OSA item was completed in accordance with FAA Orders. (c) Needs Improvement: the QC OSA item was not completed in accordance with FAA Orders. NOTE– Comments are not required to close out the OSA form, however, comments greatly assist in finalizing a full understanding of what happened during the skill assessment. If checking or not checking an item category creates possible ambiguity for an individual reviewing the OSA, use the comments section to assist in a full understanding of what was observed during the QC OSA related to a specific item. (3) QC OSAs are de-identified samplings of individual technical performance and may not be used for or to trigger IPM actions. (4) Each QC OSA must accurately capture the technical performance demonstrated during that session. Data captured during QC OSAs are the basis for many other QC processes; accurate documentation is essential to ensure proper identification of facility performance issues. (5) QC OSAs may be conducted by both management and non-management staff since they will only be used for identifying facility systemic issues and not used for IPM. (6) QC OSAs must only be used to identify facility/organizational systemic non­ compliance and/or monitor the effectiveness of CAPs. (7) All QC OSAs require a bookmark of the reviewed session (if the capability exists) to be attached to the OSA. (8) Facilities must regularly review aggregate results from completed QC OSAs and identify potential facility systemic non-compliance/risk. The “OSA Report” in CEDAR displays aggregate information on QC OSA results, for a user selected date range. (9) Potential systemic non-compliance/risk identified in QC OSAs must be analyzed through the service review or the Compliance Verification processes in order to validate and understand the potential issue. 3-3

10/01/2020 JO 7210.634A 3-3. Emphasis Items (EIs). EIs are prioritized custom sub-tasks in OSAs, and are used to collect data on specific focus items for a period of time defined by the facility. These items may be identified to be observed, documented, and tracked as a quality control area of emphasis. a. Who Identifies EIs. Facilities can choose to develop an EI, or may be directed to do so by their district, Director of Operations, or other service unit personnel. b. How to Identify an EI. An EI can be developed for any issue beyond the standard sub­ tasks in the CEDAR OSA forms. There are three primary purposes of EIs: (1) Monitoring CAPs: Collecting data to assess the effectiveness of corrective actions implemented to mitigate a specific hazard. (2) Collecting additional data to validate suspected trends or issues: Potential facility problems identified through other means (e.g., service reviews, Combined Safety Barrier Review, Compliance Verifications, Partnership for Safety Portal data, Local Safety Council identification) often require additional data to determine the validity and scope of the issue. Facilities are encouraged to use EIs as a method of collecting such data. (3) Emphasizing special focus issues: Facilities, Districts, Service Areas, and/or Service Units may use Emphasis Items to place special focus on specific issues by requiring a topic to be assessed in every OSA conducted within the applicable location/organization. Collected data must be used to assess compliance with the requirement under review. c. How to Create and Document EIs. EIs must be entered into CEDAR to ensure documentation and organizational visibility into collected data, and is done so in the following manner: (1) Facilities can create an EI by first creating a custom sub-task in their OSA module. Once a custom sub-task is created, checking the “emphasis item” radio button for that custom sub-task will move it to the top of the OSA form and identify it as an EI. (2) Each EI must have a defined start and stop time period associated with it. Facilities should select a time period that will ensure sufficient data are collected on the sub-task (a minimum of 60 days is recommended). (3) Facilities should regularly review both custom sub-tasks and EIs in their local OSAs to revalidate the need for each, and make revisions as necessary. Large numbers of custom sub­ tasks can overload OSA evaluators resulting in items being overlooked or not assessed. Conversely, small numbers of custom sub-tasks can result in data not being collected on items important to a facility. Routine review of these custom items ensures the list of sub-tasks remains fresh and relevant to the data needs of the facility, and allows OSA reviewers to concentrate on evaluating the most concise and relevant list of items. 3-4

10/01/2020 JO 7210.634A Chapter 4. Service Reviews 4-1. Introduction. a. Service reviews are tools to be used for the on-going evaluation and improvement of SDP services and to validate suspected systemic issues or best practices. Service reviews should normally be conducted outside of the operational quarters. b. There are four different types of service reviews: System Service Reviews (SSR), Traffic Management Reviews (TMR), Covered Event Reviews (CER), and Systemic Issue Reviews (SYSIR). Service reviews provide a methodical manner to take a broad look at all facets that affect the delivery of air traffic services. While some service reviews may be conducted post- event (SSR), some are required on a regular basis (TMR) and some are required after accidents (CER). It is expected that most other reviews will be originated absent a specific event (SSR and SYSIR). Service reviews not associated with an event should be originated on a regular schedule or conducted randomly to ensure that an SDP does not wait for an event to actually occur. c. When a service review is originated and an associated Air Traffic Safety Action Program (ATSAP) report has been accepted, SDP personnel must follow the requirements of ATSAP. The ATSAP report must be considered to be an accepted report unless advised by the Event Review Committee (ERC) that the report is excluded. If Quality Control (QC) Operational Skills Assessments (OSAs) are performed in association with a service review, and an involved specialist(s) indicates that he/she has filed an ATSAP report, properly annotate the OSA (check the “ATSAP” box). d. Service reviews must be conducted in a proactive manner and are intended to identify areas for facility improvement, including systemic risk, as well as exemplary performance and best practices, all striving to continually improve air traffic services. Multiple points of view are an asset to the service review process. The results of service reviews should be determined by concurrence of the personnel involved in each review. 4-2. SSR. The intent of an SSR is to review the air traffic services provided in any situation at any time under any circumstances in an in depth, comprehensive, and collaborative manner. The absence of a defined list of instances when an SSR must be conducted allows a SDP the latitude to use judgment and discretion in the determination of when to perform an SSR. SSRs may be specifically focused and limited in scope or may eventually evolve into a larger more in- depth review. NOTE– The filing or acceptance of an ATSAP report does not preclude an SDP from conducting an SSR. SSRs shall not be used solely to focus on individuals, their performance, or as a tool to conduct Individual Performance Management. a. Who May Request an SSR. Personnel performing watch supervision (Operations Supervisor (OS), Operations Manager (OM), Controller-in-Charge (CIC), etc.) may request an SSR whenever they identify circumstances in the operating quarters that they believe warrant such a review. Personnel performing watch supervision must give first priority to determining what immediate actions (if any) need to be taken to ensure the continued quality of SDP services 4-1

10/01/2020 JO 7210.634A for circumstances that trigger a request for an SSR. SDPs may be required to conduct an SSR, as noted in paragraph 4-2d. b. Review of SSR Requests. Personnel designated by the SDP manager must review each request for an SSR. This review must include appropriate documentation associated with each request to determine whether an SSR should be conducted. If an SSR is not conducted, document the findings in CEDAR and close out the request. If an SSR is conducted, follow the procedures for conducting an SSR in this section. c. Who May Conduct an SSR. SDP staff managers or other personnel as designated by the SDP staff manager(s). SSRs would not normally be conducted at the ATCSCC, rather, the TMR process is primarily utilized. d. When to Conduct an SSR. SDPs must continually review services provided and initiate SSRs on a regular basis. SSRs may be random, scheduled, due to public inquiries, or as a follow up to known operational activities (post-event). In addition, SDPs may be required to conduct a SSR by service unit management (ATM, General Manager, Director of Operations, Director of Safety and Operations Support, Vice President, and/or their designees) by the QCG on behalf of the Director of Operations, or CSBR. (1) In addition to random and scheduled SSRs, SSRs may be triggered for a variety of reasons. Primary triggers for conducting an SSR are: (a) To validate a suspected systemic issue identified through a review of collected QC data. (b) In response to potential systemic compliance issues associated with a single reported/detected safety occurrence. (c) To validate a suspected systemic issue identified through the CSBR process. (2) SDPs are encouraged to develop local plans, contained in the facility QC order, for conducting random or scheduled SSRs. When conducting random or scheduled SSRs, SDPs should focus on specific positions or operational areas (e.g., the arrival positions in a Terminal Radar Approach Control (TRACON), ground control in an Airport Traffic Control Tower (ATCT), an operational area in an Air Route Traffic Control Center (ARTCC), arrival sectors to a common approach control in an ARTCC) when determining when and where to focus the SSR. (3) SSRs must be conducted post-event for any non-fatal accident or occurrence color- coded yellow or red, in reference to the FAA Order JO 1030.3, Initial Event Response, and the NATCA/FAA Collective Bargaining Agreement. NOTE– There is no recommended target number of SSRs to conduct, however, facilities are encouraged to conduct SSRs as often as possible, resources permitting. A deficiency in quality or quantity of SSRs can be an indicator of an unhealthy QC program. Conducting comprehensive SSRs increases the opportunity to identify systemic facility issues and best practices. In the absence of unlimited personnel resources, conducting fewer but more thorough and comprehensive service reviews is preferred to conducting 4-2

10/01/2020 JO 7210.634A multiple service reviews that result in high counts of service review numbers, with no resultant identification of systemic issues or best practices. EXAMPLE– 1. An ATC-11 TRACON conducts a random SSR to review services on the arrival east position from 1625-1700UTC. 2. An ARTCC conducts a minimum of three (3) SSRs each week. These SSRs are conducted on specific positions for defined time periods and are rotated through areas of operation on a regular basis. 3. An ATC-7 Tower/TRACON facility receives a call from a pilot questioning services provided five (5) days earlier. The manager/staff manager requests that a SSR be conducted. 4. An operations manager at an ATC-11 Tower facility requests an SSR as a result of a runway incursion involving an air carrier and a vehicle on an active runway. The SDP reviews the request and conducts an SSR. 5. A CSBR team is convened, at the request of the Service Area QA office, based on past SDP BAR data indicating a potential issue with Instrument Flight Rules and Visual Flight Rules conflicts. As a result, the team requests a SSR be completed to review any issues in need of mitigation. The SSR is conducted and a need for training is identified. As a result, a CAP is developed and implemented. e. How to Conduct an SSR. (1) Decision point. Based on available information and local SDP or district policy, determine if an SSR should be conducted. Determinations of when to conduct a SSR must include the criteria described in paragraph 4-2d. SSRs directed by service unit management, or that are mandated via other process, are not subject to this decision point and must be conducted. (2) Data review. Determine the time period and position(s) to be reviewed based on known circumstances. Review as much data as needed (QC OSAs, BAR output, etc.) to develop an understanding of what occurred during the timeframe in question or to assess/analyze the potential systemic issue being considered. Use available tools (Traffic Flow Management System (TFMS), replay, voice, discussions, etc.) as appropriate to support this process. When conducting SSRs based on requests by personnel performing watch supervision, review all documentation associated with the request. This must include immediate issues identified and/or addressed, what (if any) data were reviewed, and any other issues that may have been identified. When conducting any SSR, consider the following areas as necessary: (a) Training: Classroom, OJT hours, Mandatory Briefing Items (MBI), air traffic simulation, etc. (b) Efficiency: Traffic Management Initiatives (TMI), delays, traffic volume/complexity, Airport Arrival Rate (AAR)/Airport Departure Rate (ADR), runway configuration, etc. (c) Airspace/Airport: Special Use Airspace (SUA), noise abatement procedures, etc. (d) Procedures: Letters of Agreement (LOAs), waivers, Standard Terminal Arrival Routes (STAR)/Standard Instrument Departure (SID), deicing, Line-Up and Wait (LUAW), etc. 4-3

10/01/2020 JO 7210.634A (e) Directives: Standard Operating Practices/Procedures (SOP), FAA Order JO 7110.65, Code of Federal Regulations (CFR) procedures, Notices to Airmen (NOTAM), etc. (f) Technical operations: Equipment configuration, performance, outages, etc. (g) Customer feedback: ATCSCC logs, pilot/airline comments/inquiries, airport operator comments/inquiries, etc. (h) Resource management: Operational position combinations/configurations, personnel assignments, available staffing, position relief management, leave approvals, etc. (i) Individual Performance: Actions of involved personnel (specialist, manager, etc.). (3) Issue identification. After concluding the data review, isolate all items of recognized risk in the above identified areas and/or determine if a potential issue is valid or not. (a) Once issues are identified, or an item is validated, work with operational staff, facility management, etc., to develop and enact a corrective action plan to address the issue(s) through the corrective action plan process. (b) Additional issue identification. Each section in the CEDAR question tree includes the option for the identification of a potential systemic issue. If during the course of service review systemic issues are identified (either related to the original service review purpose or not), those issues may be validated within the current SSR being conducted or they may be identified as potentially systemic, and handled via the SYSIR process (select the applicable systemic issue option and enter a brief synopsis). (4) Documentation. All identified/validated issues and actions taken must be documented through the use of the SSR question tree in CEDAR for analysis and SSR closure. The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree, including the possibility of any systemic issues. If CEDAR is unavailable, retain the results, and enter them in CEDAR as soon as it becomes available. (5) SSR closure. Once all issues have been identified, or forwarded via the SYSIR process, and all documentation is completed, close the SSR in CEDAR by selecting the “Close SSR” option. Issues identified through a SSR must result in either a CAP that addresses the specific issue, or be identified as a potential SYSIR. (a) SSRs that were generated as the result of a services rendered teleconference must be reviewed by the respective QCG prior to the SDP closing the SSR. This review is intended to be an assessment of the quality of the SSR conducted by the SDP. (b) Resultant SSR CAPs must be documented in CEDAR, and are generated via the “Create CAP” function within the SSR. CAPs must be developed, and mitigations initiated, within 30 calendar days of SSR closure. f. Employee Interviews. Interviews with employees are an essential element used to gain an understanding about what occurred during the period under review. Personnel conducting 4-4

10/01/2020 JO 7210.634A SSRs are encouraged to obtain multiple perspectives that may indicate what occurred and should consider all employees regardless of status, e.g., developmental, as potential interviewees. Employee participation is mandatory; however, interviews with any employee must be conducted in an atmosphere of shared concern that is designed to gain a better understanding about the operational environment. Any information obtained during such interviews should be used within the SSR process to aid in identifying risk that may require corrective action. g. Other SDPs. When elements of the SSR indicate another SDP may benefit from reviewing the information, the SSR must be forwarded to the other SDP staff for review, comment, and/or action, as appropriate. 4-3. TMR. TMRs are in-depth, comprehensive, and collaborative facility operational reviews. The intent of a TMR is to perform an evaluation of SDP traffic management operations with a focus on identifying issues that may impact system efficiency. This should include items that need correcting as well as identification and recognition of best practices. The TMR is the primary service review process for the ATCSCC. a. When to Initiate a TMR. TMRs may be random, scheduled, or specific (post-event). SDPs must ensure that services provided by Traffic Management Units (TMU)/ATCSCC are continually reviewed and initiate TMRs on a regular basis. TMRs may be initiated at any time, due to random or scheduled reviews, because of significant delay events (due to weather, equipment outages, etc.), associated with a special event (sporting event, fly-in, etc.), or at the request of management. (1) In addition to random and scheduled TMRs, the primary triggers for conducting a TMR are: (a) Significant delay events (due to weather, equipment outages, TMIs, etc.) (b) Special event activities (e.g., sporting events, fly-ins, Temporary Flight Restrictions) (c) A request from operational management. (d) CSBR (2) When conducting random or scheduled TMRs, facilities should concentrate on specific traffic management positions or functions when determining when and where to focus the TMR. (3) TMRs must be conducted when requested by Air Traffic Services or System Operations organizational leadership. NOTE– There is no recommended target number for TMRs to conduct. However, applicable facilities are encouraged to conduct TMRs often, resources permitting. Conducting comprehensive TMRs increases the opportunities to identify potential systemic issues. A deficiency in quality or quantity of TMRs can be an indicator of an unhealthy QC program. In the absence of unlimited personnel resources, conducting fewer 4-5

10/01/2020 JO 7210.634A but more thorough and comprehensive TMRs is preferred to conducting multiple TMRs that results in high count totals, with no resultant identification of systemic issues or best practices. b. Who May Initiate a TMR. (1) SDP TMR. Traffic Management Officer (TMO), SDP/General Managers, National Operations Manager (NOM), National Traffic Management Officer (NTMO), Supervisory Traffic Management Coordinator (STMC), Traffic Management Coordinator-in-Charge, OS, OM, CIC, etc., quality control managers/staff, or other personnel as designated by the SDP Manager. In addition, personnel within the SDP’s service unit (General Manager, Director of Operations, Vice President, and/or their designees), may require a SDP to initiate a TMR. Finally, the Deputy Director of System Operations (DDSO) and/or the QCG may also require an SDP to initiate a TMR on behalf of the Director of Operations at the service area. (2) ATCSCC. Any request made by an SDP to the ATCSCC to initiate a TMR must be routed through either the OM, TMO, SDP Manager (or designee), General Manager, or Assistant General Manager for awareness and approval. (3) CSBR panel. c. Who Must Perform a TMR. TMRs should only be conducted at SDPs with a TMU or at the ATCSCC. TMRs must be performed by SDP-designated staff. NOTE– Due to the national perspective of the ATCSCC, the ATCSCC may request individual SDP TMRs to be completed in support of a particular TMR being conducted by the ATCSCC. d. How to Perform a TMR. (1) Data review. The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree (including the possibility of any systemic issues). Personnel conducting a TMR must review traffic management performance metrics for the date(s) under review. (a) This review must include, at a minimum, the following metrics for that SDP (if applicable): i. TMIs, to include miles-in-trail restrictions, ground delay program/airspace flow program/ground stop. ii. Operations Network (OPSNET) delays. iii. Terminal area efficiency rating. (b) These metrics should be evaluated against those values that measure system capacity and availability in order to provide an overall picture of available capacity as measured against actual performance. When reviewing capacity consider, as a minimum, the following values: 4-6

10/01/2020 JO 7210.634A i. National Traffic Management Log runway accuracy. ii. Average daily capacity. iii. Monitor alert parameters. iv. AAR/ADR. v. Runway configuration. (c) In addition, reviewers should consider traffic management data from adjacent facilities when such data would be pertinent to the review being conducted. (d) When appropriate, reviewers may interview operational personnel (National Traffic Management Specialist, NTMO, NOM, Traffic Management Coordinator, STMC, OS, Air Traffic Control Specialist, CIC, OM, etc.) to obtain an operational perspective regarding certain aspects of the date(s) under review. In addition, reviewers should consider contacting adjacent SDPs (ARTCC, TRACON, ATCT, ATCSCC, etc.) and/or other sources (airspace operators, National Weather Service, etc.) to gain a perspective on what was or was not effective and what impact that date’s operations may have had on those SDPs. (2) Issue identification. When comparing metrics to available capacity, reviewers should assess whether those initiatives that were implemented were thorough, performed as intended, and implemented following FAA orders. In addition, reviewers must assess whether those triggers used to initiate any TMIs were appropriate, coordinated, and communicated to the appropriate end user. Reviewers must then identify performance gaps, best practices, and/or successes primarily at the SDP level. However, such items may be identified at other organizational levels as well. Once the issue(s) is identified, SDPs must work with operational staff, facility management, etc., to develop and enact a corrective action strategy to address the issue(s) in a timely fashion. For those issues identified at higher organizational levels, service units (at the appropriate organizational level) must develop strategies to overcome those gaps. e. Documentation. TMRs must be documented in CEDAR. Any issues confirmed through a TMR must result in a CAP that addresses the specific issue and is documented in CEDAR via the “Create CAP” function. TMR CAPs must be developed, and mitigations initiated, within 30 calendar days of TMR closure. When elements of the TMR involve another facility, the TMR must be forwarded to the appropriate SDP for review, comment, and/or action, as appropriate. If issues are identified above the SDP level, forward that information via CEDAR and/or email to the appropriate DDSO for follow-up. 4-4. CER. The intent of a CER is to supplement and document the existing required review of air traffic services rendered during an aircraft accident. CERs are an in-depth, comprehensive, and collaborative facility operational review. A CER must include a review of all aspects of service (individual performance, equipment issues, weather, etc.) and identify any issues that cannot be ruled out as potentially contributing to the accident. Instances of identified exemplary performance should be identified and documented as well. A CER may be used as a tool to identify systemic risk as well as a method to improve air traffic services following an aircraft accident. The filing of an ATSAP report does not preclude a SDP from conducting a CER. 4-7

10/01/2020 JO 7210.634A a. When to Initiate a CER. The Director/Deputy Director of Operations, Senior Advisor, or the QCG on behalf of the Director of Operations in each respective service area determines when facilities must conduct a CER. CERs are a supplement to other post-accident activities and occur after required notifications and services rendered teleconferences. (1) A CER must never be initiated until all search and rescue activities, required notifications, and services rendered teleconferences have been completed. (2) CERs must be conducted after any aircraft accident involving fatalities in which air traffic services were provided. (3) A CER must be completed within five administrative days of the fatal accident under review. b. Who Must Initiate a CER. SDP management, or their designated staff, must initiate a CER when directed by the Director of Operations, Senior Advisor, the QCG on behalf of the director(s) of operations, or when mandatory. c. How to Conduct a CER. (1) Use available tools (TFMS, replay, voice, discussions, etc.) to develop an understanding of what occurred during the timeframe in question. Include any previous findings from any review of services data compiled as required by FAA Order JO 1030.3, Initial Event Response. When completing the CER, assess each of the following: (a) Training: Classroom, OJT hours, MBIs, air traffic simulation, etc. (b) Efficiency: TMIs, delays, traffic volume/complexity, AAR/ADR, runway configuration, etc. (c) Airspace/Airport: Operational position combinations, SUA, noise abatement procedures, etc. (d) Procedures: LOAs, STARs/SIDs, deicing, LUAW, etc. (e) Directives: SOPs, FAA Order JO 7110.65, CFR procedures, NOTAMs, etc. (f) Technical operations: Equipment configuration, performance, outages, etc. (g) Resource management: Operational position combinations/configurations, personnel assignments, available staffing, position relief management, leave approvals, etc. (h) Individual Performance: Actions of involved personnel (specialist, manager, etc.). d. Documentation and Closure. SDPs must document the findings of a review of individual performance as a QC OSA for all involved employees and check the “ATSAP filed” box, if applicable. In addition, record any systemic findings in CEDAR in the CER module under the applicable category in the CER question tree. 4-8

10/01/2020 JO 7210.634A (1) Employee(s) last providing ATC services to an aircraft involved in a fatal accident must be relieved from the operational position as soon as feasible and must remain relieved from operational duties until the employees’ performance can be reviewed and associated training, if assigned, is completed. When an ATSAP report has been submitted, facilities must only administer ERC-assigned training. Documentation of ERC-assigned training must comply with existing requirements. NOTE– If involved employees do not submit an ATSAP report and the employee’s supervisor believes training is warranted, the supervisor will assign appropriate training and ensure completion and documentation in accordance with FAA Order JO 3120.4, Air Traffic Technical Training. If involved employee(s) submit an ATSAP report and the employee’s supervisor believes training is warranted, the facility must submit its training recommendation to the ATSAP ERC for consideration; joint, collaborative management/NATCA submissions are always preferred, however separate recommendations will be accepted by the ERC. ERC review of facility training recommendations as a result of a CER is required in accordance with FAA Order JO 7200.20, Voluntary Safety Reporting Program (VSRP). (2) The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree (including the possibility of any systemic issues). Each section in the CER question tree includes the option for the identification of a potential systemic issue. Any potential systemic issue found must be indicated as such in the CER and handled via the SYSIR process, or if already validated within the applicable CER, mitigated through the CAP process. (3) CERs must be documented in CEDAR. Any facility non-compliance or systemic issues validated through a CER must result in a corrective action plan that addresses the identified issue, and the CAP must developed, and mitigations initiated, within 30 calendar days of CER closure. A CAP is generated through the “Create CAP” function within the CER. (4) A CER generated due to a fatal accident must be reviewed by the respective QCG prior to the SDP closing the CER. This review is not intended to be a part of the SDP review of the applicable employee’s performance, but is a review of the quality of the overall CER conducted by the SDP. 4-5. SYSIR. SYSIRs are a comprehensive and collaborative review of potential systemic issues identified during OSAs, SSRs, CSBR, and CERs. SYSIRs are a method for SDP personnel that conduct an OSA, SSR, CSBR, or CER to identify areas that have potential system impact, whether at the SDP, service area, or service unit level. SYSIRs can also be generated directly through the Air Traffic SYSIR function in CEDAR if a suspected systemic issue is found outside of the OSA, SSR, CSBR, or CER process. Individuals who identify potential systemic issues initiate the process of identifying previously unrecognized risk. When conducting an OSA, SSR, or CER, SDP personnel must continually look for underlying issues beyond the initial “what” that may have occurred and look for the “why.” Asking “why” will help lead personnel to the initial identification of a systemic issue. a. Each SDP must identify designees for the systemic areas of training, efficiency, airspace/airport, procedures, directives, and technical operations. The designees may be permanent, or rotated, at the discretion of the SDP. SDPs with limited or no support staff will be supported by their district in addressing the above systemic areas. 4-9

10/01/2020 JO 7210.634A b. SDPs must conduct SYSIRs in the following manner: (1) Assess all available information of each OSA, SSR, CSBR, or CER conducted, looking for possible underlying issues beyond those that are operationally immediate. If such an issue is identified, select in CEDAR one of the following options for further review: (a) Training. (b) Efficiency. (c) Airspace/airport. (d) Procedures. (e) Directives. (f) Resource Management. (2) In the SYSIR comments section of the OSA, SSR, or CER, describe the item that is a potential systemic issue. Once the OSA, SSR, or CER is saved, CEDAR will forward the SYSIR to the appropriate designees. EXAMPLE– 1. Any individual conducting an OSA can identify a potential systemic issue. When conducting a QC OSA, the reviewer identifies the incorrect application of vectors to intercept the final approach course. The reviewer believes this may be more widespread beyond one QC OSA and that a possible systemic training issue may exist in the facility. The reviewer enters a potential systemic training issue in the OSA and describes the issue. The SYSIR is then forwarded to the facility designee for all Training SYSIRs for review. 2. A SSR or CER collaborative team may determine that the scope of an issue warrants review under the SYSIR process. When conducting a SSR, the collaborative team determines via interviews that the application of tower visual separation is not being properly applied by facility personnel due to knowledge gaps and poor understanding of the language in FAA Order JO 7110.65, Air Traffic Control. As a result, the team believes a potential systemic issue of directives and training exists in the facility. The team enters both a potential systemic directives and training issue in the SSR and respectively describes both issues. Each SYSIR is then forwarded to the facility designee for Directives and Training SYSIRs for review. The two SYSIRs will be linked through the original SSR to ensure they are reviewed in tandem. 3. A collaborative CSBR team has convened, at the request of the facility Local Safety Council, to review suspected systemic risk associated with helicopter operations in close proximity to the departure and arrival corridors. As a result of the review, and the analysis of facility QC data and Barrier Analysis Review (BAR) data, the team determines there is risk that is in need of mitigation. A SYSIR is submitted under the Training SYSIR category for review and processing by the facility designee. (3) The SYSIR collaborative team must examine the potential systemic issue to the extent necessary to determine its validity. It is the responsibility of the team to ensure each issue is objectively reviewed and that no issue is pre-judged prior to the team’s analysis. Systemic issue validation must employ a variety of techniques. When validating potential systemic issues, consider utilizing some or all of the following methods: 4-10

09/01/2021 JO 7210.634A CHG 1 (a) Review of collected data (e.g., QC OSA data, MOR data, BAR output, compliance verification data, and Emphasis Item data). (b) Interviews of operational and/or staff personnel. (c) An examination of training materials and documentation (e.g., OJT, recurrent training, refresher training, and MBIs). (d) Review of operational replays and/or voice recordings. (e) Review of local/national directives. (f) Review of operational reference material (approach plates, charts, etc.). (g) Review of sector/airspace/runway configuration/usage. (h) Review of equipment installation, operation, configuration, and availability. (i) Review of traffic management initiatives, procedures, and compliance. (j) Review of Partnership for Safety Portal data. (k) Collection of additional data through Emphasis Items or the facility ICV. NOTE– This list is not all-inclusive. There are other items that may need examination. Teams should ensure that all pertinent avenues are explored prior to closing the SYSIR. (4) Once the examination is complete, the designee must close the item in CEDAR with one of the following three responses: (a) Concur: This response is appropriate if the SYSIR identifies/validates a previously unrecognized risk or non-compliance. The designee must concisely document the systemic issue identified in the SYSIR comments and close the SYSIR. The SDP must then employ corrective action. A CAP must be created, and mitigations initiated, via the “Create CAP” function within the SYSIR within 30 calendar days of SYSIR closure. The corrective action does not have to be completed to close the SYSIR. (b) Concur and Elevated: This response is appropriate if the SYSIR collaborative team identifies an issue, but the means to mitigate the issue is above the SDP level. The designee must concisely document the reason(s) the SDP cannot correct the identified issue in the SYSIR comments, select “Concur and Elevated,” and close the SYSIR in CEDAR. Before submitting an issue as “Concur and Elevated,” the SDP must consult with applicable district staff. “Concur and Elevated” SYSIRs are forwarded automatically through CEDAR. They will be routed as follows: 4-11

09/01/2021 JO 7210.634A CHG 1 i. To the QCG to be worked collaboratively with the NATCA Service Area Safety Representative. If the QCG is able to coordinate mitigations for the item, they must document actions taken in the elevated SYSIR prior to closure. If the QCG non-concurs, they must document the reasoning for making this determination, inform the SDP, and then close the item. If the QCG is unable to facilitate a solution but agrees there is an issue, they must concisely document the reason(s) they cannot mitigate the identified issue in the SYSIR comments, select “concur and elevated,” and close the SYSIR in CEDAR. ii. The “concur and elevated” SYSIR will then be routed to the AJI Safety Services Group, AJI-15, for resolution. AJI-15 will work the item to closure. AJI-15 will either concur, and the item will be mitigated at the national level, or non-concur, and the item will be closed after documenting why this determination was made. The appropriate service center QCG must be notified by AJI-15 of the non-concur determination. NOTE– AJI-15 will contact applicable service unit(s) with the identified issue(s) and preliminary data to support the topic. AJI-15 will work with the applicable service unit(s) to identify any additional supporting data and develop potential mitigation(s). NOTE– If CEDAR is temporarily unavailable, retain the results and enter them in the SYSIR as soon as it becomes available. (c) Does Not Concur: This response is appropriate if the collaborative team has examined the issue identified in the SYSIR but cannot identify any unrecognized risk or non-compliance. The designee must document the steps taken to examine the issue and the reason(s) for selecting “Does Not Concur,” and close the SYSIR in CEDAR. 4-12

10/01/2020 JO 7210.634A Chapter 5. Compliance Verification 5-1. Compliance Verifications (CVs). CVs are a way of assessing SDP performance and identifying areas for improvement. CVs must be conducted in accordance with Article 51 of the FAA/NATCA Collective Bargaining Agreement, which defines union participation. ICVs are planned assessments accomplished through the use of a checklist and random sampling methods such as, but not limited to, direct operational observation, discussions with SDP personnel, review of voice or radar/surveillance data, equipment parameters, certification parameters, and examination of other documentation. ECVs are assessments of SDPs that are conducted on an as-needed basis as determined by the service unit, Director of Operations, and/or the QCG. Determinations to conduct ECVs will be based on data analysis that identifies potential risk within specific SDPs. ECVs are not intended to be conducted at regular time intervals or on a regularly scheduled basis. ECVs may be conducted through various methods that may include developing a custom checklist, review of available data (CEDAR, quality control (QC) and quality assurance data, barrier analysis data, etc.), direct observation, interviews with personnel, and other means as appropriate. There are unique requirements for compliance verification for Alaska Flight Service Station (FSS) and U.S. NOTAM Office (USNOF), these are covered in Appendix A. a. Facilities should use compliance verifications in three (3) ways: (1) CAP monitoring – use the results of ICV and ECV checklist items as indicators of the effectiveness of mitigations. (2) As an indicator of potential target areas for QC Operational Skills Assessments, non­ event driven service reviews (System Service Reviews (SSRs) and Traffic Management Reviews), and possible Emphasis Items. (3) Identification of potential systemic issues – potential systemic non-compliance identified through the evaluation of checklist items must be analyzed through the SSR process or by the CV team conducting the CV in order to validate and understand the potential facility non­ compliance. Facilities with established Local Safety Councils are encouraged to use their Local Safety Council to review, identify and mitigate facility systemic non-compliance and/or risk. NOTE– FSSs, USNOF, and Federal Contract FSSs will follow the applicable procedures outlined in Appendix A and B to complete CVs. 5-2. ICV. a. ICVs are expected to be conducted by an ICV team designated by the SDP Manager or General Manager following all applicable national collective bargaining agreements. b. All FAA air traffic control SDPs and federal contract towers must conduct an ICV annually each fiscal year. The schedule for conducting an ICV will be determined by the facility in order to produce the most comprehensive evaluation results within the annual time period. c. ICVs must evaluate all items on the Compliance Verification Tool (CVT) checklist 5-1

10/01/2020 JO 7210.634A except those not applicable to the specific facility type. Use of this standardized checklist requires facilities to assess requirements that may not normally be reviewed through other QC processes. Any Air Traffic Organization (ATO) entity may propose the addition or deletion of a checklist item(s). Requests for checklist modification must be forwarded to the appropriate service unit for evaluation and possible incorporation. (1) Facilities should use a variety of methods to assess compliance with checklist items. This includes monitoring of operations (live and via playback tools), reviewing collected data in CEDAR, and interviewing facility personnel. (2) It is imperative that facilities document all non-compliance identified through the ICV process. This ensures an accurate record of facility compliance and correction. (3) Facilities may use SSRs to better understand an issue identified through the ICV process. SSR findings must be documented in CEDAR. 5-3. ECV. a. The Directors of Operations, within each service area, must ensure that ECVs are conducted in response to data-driven indicators of potential systemic risk and/or noncompliance. ECVs may also be initiated by the Service Unit. ECVs will primarily be conducted by Service Center QCGs. A service unit or a Director of Operations, at the service area, may assign additional personnel to any ECV team. The QCG may request resources from the service unit/area to augment ECV staff. b. The QCG must collaborate with the initiating organization (applicable service unit) to determine the scope of the ECV. ECVs may be broad in scope and utilize a customized checklist, or may focus narrowly on just a few items. When defining ECV scope, the service center QCG must consider trend analysis from the ATO Safety and Technical Training Quality Assurance Group. An ECV may be conducted as a site visit or a desk audit. c. The Air Traffic Control Systems Command Center ECV team will be comprised of representatives from any of the service center QCGs. d. At any time during an ECV, the team may observe operational items that represent a significant safety risk. After advising the SDP manager, the ECV team should then focus on those items whether or not they were defined within the original scope of the ECV. e. The ECV team lead will ensure the visit is coordinated as needed, and the team will conduct an in-briefing with the SDP manager. The briefing will include an introduction of team members and ECV expectations. 5-4. Findings. a. The ICV and ECV teams will assess items using the following categories and enter applicable details into the CVT: 5-2

10/01/2020 JO 7210.634A (1) Exemplary (E): This finding is assigned to items that demonstrate exemplary performance in quality and efficiency. (2) Compliant (C): This finding is assigned to items that are completed in compliance with national, service area, and local requirements. Details are not required to be entered into the CVT. (3) Non-compliant Low Risk (NL): This finding is assigned to items that are non- compliant, but do not represent a moderate or significant safety risk to the National Airspace System (NAS). (4) Non-compliant Medium Risk (NM): This finding is assigned to items that are non- compliant and represent a moderate safety risk to the NAS. (5) Non-compliant High Risk (NH): This finding is assigned to items that are non- compliant and represent a significant safety risk to the NAS. (6) Not Observed (NO): This item is assigned to items that are not observed during the verification. The reason the item was not observed must be documented. (7) Informational (I): For ECV use only. This finding is assigned to any item or topic that is of interest or significance, or used for awareness of information. b. Prior to leaving the SDP, or concluding the desk audit, the ECV team lead must brief the SDP manager on all items that will be rated as non-compliant. c. QCGs will forward findings of the ECV pertaining to training to ATO Safety and Technical Training. d. ECV reports must be completed and submitted into the CVT within 10 administrative work days. The report must list all items rated and include associated details for all items rated “E,” “NL,” “NM,” or “NH.” 5-5. Responding to Findings. a. CAP. Items assessed as non-compliant require a corrective action plan as described in Chapter 7 of this order. CAPs must be documented in the CVT. The CAP communicates how specific risk/non-compliance will be resolved. The plan must specifically describe actions taken to effectively mitigate and monitor the identified issue(s). b. The SDP must respond to items identified as introducing risk or non-compliant in the following manner: (1) NL: Develop corrective action and enter the plan in the CVT within 30 calendar days of the finding. To close the item, document the processes used to ensure the effectiveness of the mitigation. 5-3

10/01/2020 JO 7210.634A (2) NM: Develop corrective action and enter the plan in the CVT within 20 calendar days of the finding. To close the item, document the processes used to ensure the effectiveness of the mitigation. (3) NH: Due to the severity of the finding and subsequent risk to the NAS, SDPs must, within seven calendar days of being notified of the rating by the ICV/ECV team, develop a corrective action plan and obtain associated Director of Operations concurrence at the applicable service center (through the QCG). To close the item, document the processes used to ensure the effectiveness of the mitigation and enter it into the Compliance Verification Tool. c. No action is required for checklist items rated “E,” “C,” “I,” or “NO.” 5-4

09/01/2021 JO 7210.634A CHG 1 Chapter 6. Quality Control Validations 6-1. Quality Control Validations (QCVs). QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all QC processes to ensure the effective identification of non-compliance and associated corrective actions. QCVs are the primary method that facilities must use to ensure the integrity of data collected in QC processes. QCVs require facilities to review samplings of QC OSAs, Certification Skill Checks / Performance Assessments, and OJT documentation to validate accuracy and completeness within each process. The QCV process is a statistical sampling of a particular process or task to ensure compliance and accuracy. This process may also identify potential systemic issues associated with training, efficiency, airspace, procedures, directives, and equipment. If risk is determined to exist, facilities must determine the appropriate mitigation strategy and document a CAP in CEDAR within 30 calendar days. a. SDPs must develop a local validation plan for each of the three required QCVs. Local validation plans must contain the following: (1) Target number of validations to be performed: Targets may be defined by any calendar unit (e.g., monthly, quarterly, annually). Validations must be conducted in sufficient numbers (a representative sampling) to ensure an accurate assessment of facility performance in conducting each of the processes being validated. It is recommended that SDPs validate a minimum of 15% of all QC OSAs, Certification Skill Checks / Performance Assessments, and OJT Instructor (OJTI) documentation each quarter at each SDP. (2) Sampling method: While each process should be sampled randomly, local validation plans should include selection methods that ensure a cross-section of sectors/positions are reviewed. For example, a local validation plan could include a requirement that at least one QC OSA for each operational position be validated each fiscal year. (3) Replay tools to be used: SDPs must use both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer/OJTI. Validations must be conducted within the maximum data retention periods for the facility to ensure availability of required data. (4) Feedback process: SDPs may provide feedback to individual non–bargaining unit personnel for QCVs when appropriate. Individual feedback must always come through the individual’s immediate supervisor. SDPs must only provide facility-wide feedback to bargaining unit employees for all QCVs. (5) Follow-up process: SDPs must follow up on issues identified through validations to ensure that the feedback or corrective action was effective in improving the respective process (QC OSAs, Certification Skill Checks / Performance Assessments, and OJT documentation). Follow-up processes related to non-corrective action plans should include a review of the previously identified feedback issue, a defined period for follow-up review, and closure if the issue is resolved. If the issue still exists, additional feedback must be provided. For matters that 6-1

09/01/2021 JO 7210.634A CHG 1 resulted in a CAP, the monitoring and effectiveness goals will indicate the success of any implemented corrective action. b. QCVs must be conducted in accordance with Article 51 of the FAA/NATCA Collective Bargaining Agreement, which defines union participation. Facilities are encouraged to establish collaborative teams to conduct QCVs. c. QCVs must only be used to identify organizational or systemic issues. d. OSA Validations. SDPs must establish a validation process whereby representative samplings of OSAs are evaluated to ensure accuracy and completeness. At a minimum, SDPs must use both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer. The purpose of this initiative is to ensure that the reviewer has captured the performance of the trainee accurately in the OSA. These reviews provide an opportunity to identify exemplary performance that can be shared in the facility as a best practice, as well as any performance deficiencies or risky behaviors that otherwise may have been overlooked. Districts must ensure that sufficient assistance is provided to facilities with limited managerial resources in order to comply with this requirement. The validation process must include: (1) Feedback to the person conducting the OSA through their immediate supervisor, except as noted in paragraph 6-1a.(4); and (2) Follow-up mechanisms to ensure that the feedback was effective if OSA improvement is required. e. Certification Skill Check and Performance Assessment Validations. SDPs must establish a validation process whereby representative samplings of Certification Skill Checks and Performance Assessments are evaluated to ensure accuracy and completeness. At a minimum, SDPs must utilize both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer. The purpose of this initiative is to ensure that the OS/STMC/NTMO has accurately captured the performance of the developmental, Certified Professional Controller-in-Training (CPC-IT), or the Traffic Management Controller-in-Training (TMC-IT) / Traffic Management Specialist-in-Training and/or Developmental/OS/STMC/NTMO during a Certification Skill Check or Performance Assessment. These reviews provide an opportunity for an SDP to ensure consistency in their skill checks / assessments as well as identify performance deficiencies or risky behaviors that otherwise may have been overlooked. Districts must ensure that sufficient assistance is provided to facilities with limited managerial resources in order to comply with this requirement. The validation process must include: (1) Feedback to the person conducting the skill check / assessment through their immediate supervisor and (2) Follow-up mechanisms to ensure that the feedback was effective if skill check / assessment improvement is required. 6-2

09/01/2021 JO 7210.634A CHG 1 f. On-the-Job Training Documentation (OJTD) Validations. SDPs must establish a validation process whereby they assess the accuracy and completeness of the documentation provided by OJT instructors. The purpose of the OJTD is to ensure that OJTIs accurately captured the performance of the CPC-IT/TMC-IT/NTMS-in-training/developmental/OS/STMC/ NTMO during OJT. These reviews provide an opportunity for an SDP to ensure consistency in the conduct and documentation of OJT and to identify performance deficiencies on the part of OJTIs. Districts must ensure that sufficient assistance is provided to facilities with limited managerial resources in order to comply with this requirement. (1) When conducting OJTD validations, SDPs must review a representative sampling of the OJT documentation for the year under review using available replay tools. (a) SDPs must use both voice and radar/surveillance data (where available) to compare actual performance to that documented by the reviewer. (b) Tower-only facilities without surface radar are expected to use voice files to the maximum extent possible to assess any OJT sessions reviewed in this process. (2) The validation process must include: (a) Feedback to the SDP OJTIs; (b) Refresher training, if appropriate, to SDP OJTIs. If assigned, document refresher training in the employee’s FAA Form 3120-1, Training and Proficiency Record, using CEDAR; and (c) Follow-up mechanisms to ensure that feedback and training were effective. 6-3

10/01/2020 JO 7210.634A Chapter 7. Corrective Action Plans 7-1. Introduction. Finding and fixing problems is the fundamental purpose of our safety management system and the quality control process. CAPs are the method SDPs must use to document implemented corrections to validated systemic non-compliance or identified risk. 7-2. Develop and Implement CAPs. CAPs are actions taken by SDPs to correct non-compliance or risk, which has been properly identified, validated, and understood through data collection and analysis. Once a problem is understood (in scope and causal factors), SDPs must develop corrective actions to address the identified issue(s). CAPs must be designed to correct the specific problem, and then implemented throughout the applicable area. In addition, CAPs must include how the effectiveness of implemented mitigations will be measured. Facilities must monitor implemented CAPs as they continue to collect data. This can be accomplished through performing QC OSAs, designing specific EIs to measure a specific CAP, performing ICVs, reviewing reported/detected occurrence data, or analyzing data available in the PFS Portal through the LSC. a. CAPs may be generated from any of the following. (1) Service Reviews (2) Compliance Verifications (3) QC Validations (4) Significant events/investigations (require a System Service Review (SSR) to validate the issue, possibly at a later date) (5) LSC analysis (validation of issues through SSRs is encouraged) (6) Runway Safety Action Teams (RSATs) (7) Combined Safety Barrier Review (8) For issues identified externally such as; AOV audits, NTSB investigations, and/or AJI audits. b. CAP development teams should ensure that they gather input from relevant facility personnel to ensure all information is considered in creating a CAP. Facility personnel could include the following depending on the specific issue. (1) Facility staff personnel (e.g., QC, airspace/procedures, training) (2) Operational staff (e.g., Developmentals, Certified Professional Controllers, Supervisors, Operations Managers, Traffic Management Coordinator, STMC) (3) Facility management (e.g., Support Managers, Facility Manager) c. The following steps are required to complete a CAP. 7-1

10/01/2020 JO 7210.634A (1) Describe the specific corrective actions that will mitigate the facility non-compliance/risk. Examples of corrective actions include, but are not limited to: (a) Training (must target the specific knowledge gap) (b) Changes to local procedures and/or processes (c) Realignment of airspace (d) Changes to letters of agreement with adjacent facilities, airport operators, etc. (2) Identify the scope of the correction (e.g., facility-wide or certain operational areas within a facility). (3) Identify a timeframe for completion of the action(s) taken. (4) All CAPs must include a specific monitoring plan. Identify a monitoring plan for determining the effectiveness of the implemented corrective actions. Monitoring plans should include a frequency of data review, what data must be reviewed, and an assigned focal within the facility/organization for ensuring monitoring is conducted. (5) Identify the target value for mitigation effectiveness. EXAMPLE– A collaborative SSR team validates systemic facility non-compliance with the application of vectors to intercept the final approach course. This was based on QC OSA and Barrier Analysis Review (BAR) data. The team develops a CAP requiring training for all radar certified personnel on the issue. Once the training is completed within the required 60 days, the facility will include a new Emphasis Item in all QC OSAs on vectoring to the final approach course to be collected for 90 days from the completion of all training. The facility will consider the non-compliance mitigated if compliance with this Emphasis Item meets or exceeds 90%. d. If the mitigation effectiveness target is not met, a revised CAP must be developed, documented, and enacted. This must include a new monitoring plan to assess effectiveness of the revised CAP. 7-3. Monitoring. Facilities must monitor all implemented CAPs in order to determine their effectiveness. Monitoring will result in CAP closure or a revised CAP if effectiveness targets are not met. CAP monitoring is primarily accomplished by assessing collected/reported data. a. There are several possible methods for collecting data for a monitoring plan: (1) Emphasis Items: Develop an Emphasis Item for all OSAs that collects data specific to the non-compliance/risk and associated mitigation. EXAMPLE– An En Route facility has completed training of all radar-certified personnel on the use of speed control with terminal arrivals, as required by a CAP. The facility subsequently implements a new Emphasis Item for all OSAs. This Emphasis Item requires each OSA conducted for the next 90 days to specifically assess 7-2

10/01/2020 JO 7210.634A compliance with the issue. At the end of that time period the data will be assessed to determine the level of compliance with the requirement. Compliance will be used to determine CAP effectiveness. (2) Log entries: Require entries on the Facility Operations Log (Form 7230-4) in CEDAR. Log entries must be for a specific type operation/occurrence and must include a specific keyword to support the word search function in CEDAR. EXAMPLE– A terminal facility has revised and implemented local policies regarding combining/decombining arrival and final positions in the Terminal Radar Approach Control, as required by a CAP. The facility will require a log entry each time any of these positions are combined/decombined for the next 60 days to determine the level of compliance with the new requirement. Each log entry must include the keyword “FINCAP” to flag the entry for word searches. At the end of that time period the data will be assessed to determine the level of compliance with the requirement. Compliance will be used to determine CAP effectiveness. (3) Compliance Verifications: Use the facility ICV process. There must be a checklist item directly related to the non-compliance and mitigation. EXAMPLE– A terminal facility has completed training on its local QC OSA sampling plan in response to a CAP (QC OSAs were not being conducted in sufficient numbers to meet the facility defined quarterly target). The facility will assess compliance with the CAP during its upcoming ICV under the checklist item “Quality Control Monitoring.” The facility will consider the non-compliance mitigated if the target is met in each quarter remaining in the fiscal year. (4) PFS Portal: Use data available through the PFS Portal to assess mitigation effectiveness. Facilities must have an LSC to use the PFS Portal. EXAMPLE– A terminal facility has completed training of all operational personnel on vectoring and speed control techniques when sequencing arrivals for final approach, as required by a CAP. The CAP was developed in response to a high number of go-arounds due to unstable approaches. After 120 days, the facility’s LSC will review track data and Air Traffic Safety Action Program reports (accessed through the PFS Portal) to assess the effectiveness of the CAP. The LSC will specifically compare go-arounds during this time period to the time period used in the CAP. (5) MOR data: Use a specific MOR that is directly related to the non-compliance and associated mitigation. EXAMPLE– An En Route facility has completed training of all operational personnel on the handling of formation flights, as required by a CAP. After 180 days, the facility will review all “Suspected Loss Involving Formation Flight” MORs to assess compliance with the requirements of joining and separating formation flights. Compliance will be used to determine CAP effectiveness. (6) BAR data: Use data that is directly related to the risk and associated mitigation to determine CAP effectiveness. EXAMPLE– 7-3

10/01/2020 JO 7210.634A An En Route facility has completed training in a particular operational area on improper/inadequate coordination, resulting in conflicts. After 180 days, the facility will review all BAR explanatory factor data and associated safety barrier data to assess compliance/effectiveness of point outs and automated point outs. Issues found will be used to determine CAP effectiveness. (7) SDPs may choose to use combinations of any of the above options to maximize their ability to properly assess compliance with a specific requirement and determine the effectiveness of a CAP. 7-4. Documentation. SDPs must document CAPs within CEDAR to maintain a record of implemented corrections for mitigation monitoring and effectiveness determinations. Resultant CAPs of the Internal Compliance Verification and External Compliance Verification processes are documented in the Compliance Verification Tool. LSC mitigations, safety information, and problems, should be documented in CEDAR and shared in ATC InfoHub. 7-4

09/01/2021 JO 7210.634A CHG 1 Chapter 8. Technical Operations Services Quality Control Program 8-1. Purpose. The Technical Operations Services Quality Control (QC) Program analyzes NAS events with a goal of identifying systemic trends and mitigating operational risks. The QC program also analyzes performance characteristics of NAS systems and services, as well as maintenance policy compliance by Technical Operations Services personnel. This chapter provides specific procedures and processes used to measure the quality of Technical Operations Services products and services provided. They consist of the following components: • System Service Review (SSR) • Systemic Issue Review (SYSIR) • Corrective Action Plan (CAP) • Compliance Verification (CV) a. The QC program components in this chapter apply to the Technical Operations Services directorates and organizations where they have been implemented. b. The success of a QC program is dependent upon the ability to focus on data collection, analysis, and systemic identification to reduce risk—not to assign blame or exert punishment. The discussion of the event with an employee is not an investigatory interview, and employees must be free to share their knowledge of the facts of an event without fear of retribution. c. SSRs, SYSIRs, and CAPs are documented in the safety database. The current safety database is CEDAR, located at https://cedar.faa.gov. To assist managers, the Technical Operations CEDAR Desk Guide can be found under the Tech Ops User Guide section of the “Help” tab at https://cedar.faa.gov. This guide provides detailed instructions on how to perform SSR, SYSIR, and CAP entries and document review in the CEDAR safety database. d. Voluntary Safety Reporting. The VSRP is a separate but parallel program that provides a method to identify and correct potential safety hazards by encouraging voluntarily submitted safety reports from employees. Information on the VSRP can be found in FAA Order JO 7200.20A, Voluntary Safety Reporting Programs, and/or applicable Memorandum of Agreement (MOA) / Memorandum of Understanding (MOU). Filing or acceptance of a VSRP report does not take the place of, or preclude, a SSR being conducted. 8-2. Technical Operations Services System Service Review (SSR). A key component of the Technical Operations Services Quality Control Program is the SSR. The SSR is a comprehensive review following events that have resulted in, or could have resulted in, operational consequences with discernable impacts to the NAS systems and/or services. SSRs are conducted to determine the causal factors of the event, determine why an event occurred, document those findings, and develop any necessary corrective actions to reduce risk to the NAS. SSRs are also used to review if policies or procedures can be enhanced or corrected to prevent similar events from occurring in the future. 8-1

09/01/2021 JO 7210.634A CHG 1 a. When to conduct a SSR. The following events require a SSR to be conducted: (1) Services Rendered Teleconference (SRT) that results in a color code Yellow or Red, with Technical Operations Services involvement; (2) Technical Operations Services deviation resulting in unscheduled system/service interruption; (3) Technical Operations Services coordination deviation or lack of coordination that results in unscheduled system/service interruption; (4) Errors in a documented process or procedure resulting in unscheduled system/service interruption; (5) Required system/service Notices to Airmen not properly published; (6) Surface Incident or Runway Incursion with Technical Operations Services involvement; (7) ATC Zero caused by Technical Operations Services system/service failure or personnel action/inaction; (8) If requested by Technical Operations Services management at the headquarters, directorate, district, or group level. b. How to conduct a SSR. The Group Manager and Front Line Manager (FLM) are responsible for leading an open comprehensive fact-finding review with all personnel involved in the planning, risk assessment, coordination, and execution of the event—as well as affected stakeholders—to determine the causes, document those findings, and develop any necessary corrective actions. The Group Manager or FLM must notify the appropriate labor union representative(s) and invite them to participate in the SSR. Personnel must continually look for underlying issues beyond the initial “what” may have occurred and look for the “why” it occurred. All aspects of an event should be evaluated considering all contributing factors. (1) The Technical Operations Services SSR Template, located in Appendix F, is based on the questions and required information in the safety database SSR Entry Form and is intended to assist management by providing a guide for preparing for and conducting a SSR. Only management can complete the SSR template and enter data in the safety database. (2) Discussions with employees are an essential element used to gather information and gain an understanding about what occurred during the event under review. Employee participation is mandatory; however, discussions with any employee must be conducted in an atmosphere of shared concern that is designed to gain a better understanding about the operational environment. Any information obtained during such discussion should be used within the SSR process to aid in determining why the event occurred and identifying operational risk that may require mitigation. (3) It is the responsibility of the Group Manager to ensure that the SSR process is 8-2

09/01/2021 JO 7210.634A CHG 1 followed and within the prescribed timeframes. The SSR process consists of a preliminary review, discussion, and data entry. (a) Preliminary Review: The FLM must perform a preliminary review in advance of the SSR discussion to collect information to develop an event summary that provides a short description of service being reviewed, the sequence of events, and operational impacts. This preliminary review may include, but is not limited to, log reviews, replays, and discussions with employees and stakeholders. The preliminary review should be a collection of basic facts only. The follow-on SSR discussion will be a comprehensive review of the event with involved personnel and affected stakeholders. (b) Discussion: The Group Manager and FLM are responsible for completing the SSR discussion within seven calendar days of when they become aware of the subject NAS event, or sooner if requested by management at the directorate level. The discussion may be completed in person, via teleconference, via video conference, or via any combination that allows maximum participation of involved personnel and affected stakeholders. i. The FLM has the responsibility to coordinate and schedule the SSR discussion and to include all personnel involved in the planning, risk assessment, coordination, and execution of the event. The FLM should also consider stakeholders affected by the event. The following should also be included in all SSR discussion invitations: • Technical Operations Services Service Area Operational Safety Program Manager (SAOSPM). • Service Center Quality Control Group (QCG) representative. • Service Area Technical Services and Technical Services Center (TSC) management representative(s). • Labor union representative(s). ii. The Group Manager is responsible for facilitating the SSR discussion. The Group Manager is expected to open the discussion with an introduction to the subject event and the purpose of the SSR. The Group Manager is responsible for facilitating a productive conversation in a manner that avoids blame and confrontation and instead focuses on fact-finding around all aspects of the event. It is important that the Group Manager ensures full exploration of the event and guides the participants to discovering why the event occurred, identifying any potential systemic issues and corrective actions. iii. Using the information collected in the Preliminary Review, the FLM is responsible for providing a summary of the event and then using the Technical Operations Services System Service Review (SSR) Template to guide the participants through the SSR discussion. The FLM is responsible for ensuring that all the required review categories in the template that are applicable to the SSR are thoroughly discussed in an effort to identify why the event took place, any potential systemic issues, and any necessary corrective actions. (c) Data Entry: The FLM must enter the SSR in the safety database via the SSR Entry Form as a draft within seven calendar days of completing the SSR discussion. This should 8-3

09/01/2021 JO 7210.634A CHG 1 include CAPs if corrective action is required. The draft SSR should document all identified issues and corrective actions. The FLM should attach all documents relied upon during the review. To maintain confidentiality, employee names or operating initials shall not be included. The Group Manager shall review the draft SSR for accuracy and completeness. i. The FLM must close and publish the SSR in the safety database within 14 calendar days of conducting the SSR. 8-3. Technical Operations Services Systemic Issue Review (SYSIR). SYSIRs are a method to identify areas that have potential systemic impact, whether at the local, group, district, directorate, or headquarters level. An issue should be identified as potentially systemic if there is a probability for recurrence at the same location or elsewhere in the NAS. Personnel must continually look for underlying issues beyond the initial “what” that may have occurred and look for the “why.” Asking “why” will help lead personnel to the initial identification of a systemic issue. a. When a potentially systemic issue is identified and recorded during the SSR process, a SYSIR must be created. A SYSIR may also be created without conducting a SSR. It is important to provide a detailed and thorough description of the potential systemic issue so that reviewers can better understand the context of the issue. b. The SYSIR process can identify and document systemic issues in the following categories: • NAS systems and services. • Coordination. • Procedures/directives/documentation. • Impact mitigation. • Resource management. • Training. c. The FLM is responsible for creating and documenting the SYSIR in the safety database. If a potentially systemic issue is identified and recorded during the SSR process, a SYSIR will be automatically generated in the safety database, and then the FLM is responsible to document and complete the SYSIR. This should include CAPs if corrective action is required. To maintain confidentiality, employee names or operating initials shall not be included. The Group Manager is responsible for reviewing the SYSIR and determining the disposition (“Do Not Concur,” “Concur,” “Concur and Elevate”). d. The Technical Operations Services Systemic Issue Review (SYSIR) Template, located in Appendix F, is intended to assist users in preparing information and data for preparing an SYSIR, which then can be entered into the safety database. This template is based on questions and required information in the SYSIR function in the safety database. 8-4. Technical Operations Services Corrective Action Plan (CAP). CAPs are intended to document the corrective actions that will be used to address the operational safety deficiencies identified during a SSR or SYSIR. Corrective actions may include, but not be limited to, 8-4

09/01/2021 JO 7210.634A CHG 1 changes to procedures, maintenance policy, implementation of new technologies, or training. CAPs must follow the appropriate safety assessment in accordance with SMS policy, if required. a. CAPs may be initiated by Technical Operations Services management at the local, group, district, directorate, or headquarters level. CAPs can only be developed at the level of the organization empowered to correct (fix) the operational safety deficiencies and/or other risks to operational safety. To maintain confidentiality, employee names or operating initials shall not be included. b. The FLM is responsible for creating and documenting the CAP in the safety database. A CAP can be created in the safety database by three methods: from a SSR, from a SYSIR, or from a stand-alone creation process. The benefit of creating a CAP through a SSR or SYSIR in the safety database is that a CAP will be automatically generated and the database will associate (link) the CAP with the SSR or SYSIR. For an automatically generated CAP, the FLM is responsible for documenting and completing the CAP. c. The Technical Operations Services Corrective Action Plan (CAP) Template, located in Appendix F, is intended to assist users in preparing information and data for preparing a CAP, which can then be entered into the safety database. This template is based on questions and required information in the CAP function in the safety database. d. The CAP must include the following required elements: • Basic information. • • • • • Identified safety risk or hazards. Background information. Pertinent regulations. Corrective action(s), including a targeted completion date, monitoring plan, and effectiveness target. Supporting data. e. The FLM must ensure that all CAP actions are completed within the timeframes identified in the CAP itself. The Group Manager is responsible for monitoring the progress of the CAP and is responsible for reviewing and determining if all mitigations are implemented and if the effectiveness targets are met prior to closure. Once the CAP actions are complete, the FLM will recommend the CAP for closure, and the Group Manager must finalize the CAP in the safety database by closing it. 8-5. Technical Operations Services Compliance Verification (CV). The Air Traffic Organization uses CV to assess compliance with directives, policies, and procedures, and to identify areas for improvement. CV within Technical Operations Services is accomplished through the Internal Compliance Verification (ICV) process. a. ICV. The purpose of an ICV is to identify latent safety issues through the verification of compliance with policy and procedures. The National Airspace System Technical Evaluation Program (NASTEP) accomplishes this task through the review of equipment used to provide 8-5

09/01/2021 JO 7210.634A CHG 1 NAS services. NASTEP policies and procedures are identified in FAA Order JO 6040.6, National Airspace System Technical Evaluation Program. Technical Operations Services ICVs are captured on the Integrated NASTEP Application (INA) website located at https://nastepweb.faa.gov/INA_By_Numbers/app/index.cfm. 8-6

10/01/2020 JO 7210.634A Appendix A Appendix A. Alaska Flight Service Station (FSS) and U.S. NOTAM Office (USNOF) Quality Control Compliance Verification (CV) This appendix outlines procedures for performing compliance verification for FSS/USNOF facilities only. A-1. Purpose. The purpose of CV is to verify and validate the conformity of the SDP with FAA/Air Traffic Organization (ATO) directives and overall quality of air traffic services. A-2. Background. The objective of CV is to ensure that Service Delivery Points (SDPs) meet or exceed ATO goals, thereby enhancing the safety of the flying public. CV is a formal process with documented evidence to determine the level of compliance at an SDP with FAA Orders and directives. CVs must be conducted in accordance with Article 51 of the FAA/NATCA Collective Bargaining Agreement, which defines union participation. A-3. Method of Accomplishment. CVs are planned assessments accomplished through the use of a checklist and random sampling methods such as, but not limited to, direct operational observation, discussions with SDP personnel, review of audio files, and examination of other documentation. ECVs are assessments of SDPs that are conducted on an as-needed basis as determined by the Manager, Alaska Flight Service Information Area Group (AFSIAG) Manager, Air Traffic Manager, and Mission Support Services (AJV). Determinations to conduct ECVs will be based on data analysis that identifies potential risk within specific SDPs. ECVs are not intended to be conducted at regular time intervals or on a regularly scheduled basis. ECVs may be conducted through various methods that may include developing a custom checklist, review of available data (CEDAR Quality Control (QC) and quality assurance data, etc.), direct observation, interviews with personnel, and other means as appropriate. A-4. Internal Compliance Verifications (ICV). a. ICVs are expected to be conducted by an ICV team designated by the SDP manager or operations manager following all applicable national collective bargaining agreements. b. All FAA air traffic control SDPs must conduct an ICV each fiscal year. The schedule for conducting an ICV will be determined by the facility in order to produce the most comprehensive evaluation results within the annual time period. c. ICVs must evaluate all items on the Compliance Verification Tool (CVT) checklist except those not applicable to the specific facility type. Use of a standardized checklist requires facilities to assess requirements that may not normally be reviewed through other QC processes. Any ATO entity may propose the addition or deletion of a checklist item(s). Requests for checklist modification must be forwarded to the appropriate service unit for evaluation and possible incorporation. (1) Facilities should use a variety of methods to assess compliance with checklist items. This includes monitoring of operations (live and via playback tools), reviewing collected data in the CEDAR tool, and interviewing facility personnel. A-1

10/01/2020 JO 7210.634A Appendix A (2) It is imperative that facilities document all non-compliance identified through the ICV process. This ensures an accurate record of facility compliance and correction. (3) Facilities may use System Service Reviews (SSRs) to better understand an issue identified through the ICV process. SSR findings must be documented in CEDAR. A-5. ECV. a. The Flight Service Directorate will conduct all ECVs at Alaska Hub (Parent) facilities. AJV will designate those responsible for conducting ECVs for the USNOF. The ECV should focus on the operations and training at each evaluated facility and items that have a direct impact on its effectiveness. AFSIAG will retain responsibility for ECVs conducted at FSS satellite facilities. b. The checklist used for ECVs will be developed by the Flight Service Directorate or AJV, depending on the applicable ECV being conducted. ECV checklists may include items provided by service units, directors of Operations, district managers, ATO Safety, and service area quality assurance offices. ECVs may be broad in scope and utilize a customized checklist, or may focus narrowly on just a few items. The ECV checklist must also include items assessing the efficiency and effectiveness of a facility’s technical training program when applicable. c. The ECV Team will normally consist of personnel from the Flight Service Directorate and AFSIAG at Alaska Parent facilities, or as designated by AJV for USNOF ECVs. d. Anytime during an ECV, the team may observe operational items that represent a significant safety risk. After advising the SDP manager, the ECV team would then focus on those items whether or not they are included on the checklist. e. The ECV team will conduct an in-briefing with the SDP manager. The briefing will include an introduction of team members and ECV expectations. A-6. Findings. a. The ICV and the ECV will assess checklist items using the following categories and enter applicable details into the Compliance Verification Tool: (1) Satisfactory (S): This finding is assigned to checklist items that are completed in compliance with national, service area, and local requirements. Details are not required to be entered into the Compliance Verification Tool. (2) Unsatisfactory (U): This finding is assigned to checklist items that are non-compliant. (3) Not Observed (N/O): This finding is assigned to checklist items that are not observed during the verification. The reason the item was not observed must be documented. b. Prior to leaving the SDP, the ECV team lead must brief the SDP manager on all items that will be rated as “U.” A-2

10/01/2020 JO 7210.634A Appendix A c. ECV reports must be completed and submitted into the Compliance Verification Tool within 10 administrative work days. The report must list all items rated and include associated details for all items rated “U.” A-7. Responding to Findings. a. Corrective Action Plan (CAP). Items assessed as unsatisfactory (U) require a CAP as described in Chapter 7 of this order. CAPs must be documented in the CVT. The CAP communicates how “U” items will be resolved. The plan must specifically describe actions taken to effectively mitigate and monitor the identified issue(s). b. SDPs must respond to applicable “U” checklist items by developing a CAP to correct the item and concurrently obtaining associated agreement from the AFSIAG (Flight Service Directorate for Parent Facilities) or AJV, depending on the applicable ECV conducted, within 10 administrative days of the finding of the “U” item. The CAP must be entered into the CVT. To close the item, the facility must document the processes used to ensure the effectiveness of the CAP and enter it into the CVT. A-3

09/01/2021 JO 7210.634A CHG 1 Appendix B Appendix B: Federal Contract Flight Service Station (FCFSS) Quality Control This appendix outlines procedures for obtaining operational performance data and compliance verification processes for Federal Contract Flight Service Station (FCFSS) facilities only. For clarity, Service Provider (SP) refers to Leidos, and Service Delivery Point is in reference to the facility-level activities. B-1. Operational Performance Monitoring. a. Background. A key component in the delivery of air traffic services is personnel (Air Traffic Control Specialists, Flight Service Specialists, and Managers). Effective monitoring of the delivery of air traffic services will help ensure that those services are the safest and highest quality possible. All field managers at every level must effectively communicate performance expectations to their employees at the beginning of each rating cycle and subsequently measure and discuss their performance against these expectations. Data gathered through the following methods should be used to support overall SP performance. The processes and procedures described in this appendix are applicable to all FCFSS facilities. b. OSA. A quality control OSA is a sampling method used to collect data by assessing the technical performance demonstrated by operational personnel for the overall evaluation of the facility. The purpose of this initiative is to ensure that the Flight Service Directorate staff and the SP have captured the facility performance at the service delivery point and that it is accurately reflected in the OSA. (1) The Flight Service Directorate staff must document the results of the OSAs on the appropriate CEDAR FCFSS OSA 3120-26 Worksheet, if available. (2) The SP uses the appropriate FAA Order JO 3120.4 OJT evaluation form (FAA Form 3120-26) as stipulated in FAA Order JO 3120.4 under conduct of performance assessments. (3) Requirements for SP quality control OSA evaluation are outlined in the applicable paragraphs in FAA Order JO 7210.634, Chapter 3, Quality Control Monitoring, paragraphs 3-1 through 3-2. NOTE– Figure(s) 3-1 and 3-2 may be used as guidance for the conduct of SP conducted OSA requirements. c. FAA Flight Service Directorate Responsibilities. The Flight Service Directorate will effectively monitor the SP performance requirements over the life of the contract. For FCFSS, the staff roles, responsibilities, and oversight are guided by processes and procedures contained within this order (inclusive of Appendix B), the Flight Service Directorate contractual requirements, and all applicable FAA orders. (1) OSAs will be performed and analyzed on an as-needed basis as the means to conduct surveillance, auditing, sampling, data gathering, inspection, evaluation, and reporting. These activities will be accomplished at the discretion of the Flight Service Directorate. B-1

10/01/2020 JO 7210.634A Appendix B (2) Tools contained within this order, which include CEDAR and the Compliance Verification Tool (CVT), will be used to the extent possible to ensure the data gathered, mechanisms for tracking, statistical analysis and the reporting of the SP performance is accurate. Additional resources may be developed or implemented by the Flight Service Directorate per contractual requirements asneeded. (3) The activities outlined in this Appendix are in no way are meant to replace, or otherwise interfere with the SP adherence to the applicable FAA orders or any other contractual requirements. (4) Quality control monitoring is necessary to ensure areas of performance that are Exemplary, Meets Requirements/Satisfactory or Needs Improvement/Unsatisfactory are identified. d. FAA Flight Service Directorate Quality Control Findings. Results of Performance Incentive evaluations, OSAs, ECVs, or special evaluations conducted by the Flight Service Directorate will be provided to the SP or otherwise made available through the CEDAR FCFSS OSA report process. If deficiencies are noted, this provides the SP opportunities for process improvements or corrections to current processes or procedures, and allows performance enhancements throughout the monitored period. e. Corrective Action Plan. A Corrective Action Plan (CAP) is a mitigation tool used to address areas identified as “Needs Improvement” during FAA conducted FCFSS OSAs. When the SP’s overall performance of a specific OSA subtask is deficient in a specific area or job function then a CAP may be required. This process allows both the FAA and the SP a more formal approach to correct OSA identified deficiencies prior to the SP annual ICV or an FAA ECV. The FAA may also require the SP to submit a CAP to address any noted performance deficiency(s) regardless of the evaluation method(s) or data sources used or identified as a result of an ICV, ECV, or special evaluation. f. Voice Reviews. Voice reviews are a quality control tool that may be used at any time. Results of the review must be documented on the appropriate CEDAR OSA Worksheet if available. Otherwise, FAA Form 3120-26, FSS OJT Instruction/Evaluation Report evaluation form may be utilized. g. Certification Skill Check Audits. SDPs must establish an audit process whereby a representative sampling of certification skill checks will be reviewed. The certification skill check audit process must include feedback to the person conducting the certification skill check and follow-up mechanisms to ensure the feedback was effective if certification skill check improvement is required. The purpose of this initiative is to ensure the Operations Supervisor (OS) certifier has captured the performance of the full performance level/Developmental/OS accurately in certification. These reviews provide an opportunity for a SDP to ensure consistency in their certifications, as well as identify performance deficiencies or risky behaviors that otherwise may have been overlooked. B-2. Compliance Verification (CV). B-2

10/01/2020 JO 7210.634A Appendix B a. Purpose. The purpose of CV is to verify and validate the conformity of the SP with FAA/Air Traffic Organization (ATO) directives and overall quality of air traffic services. This order does not apply to occupational safety, health, or environmental evaluations and audits. b. Background. The success of any organization’s safety culture depends on the ability to verify compliance with the rules and regulations of the organization. The objective of CV is to ensure that SP meet or exceed ATO goals, thereby enhancing the safety of the flying public. CV is a formal process with documented evidence to determine the level of compliance at an SDP with FAA orders and directives. c. Method of Accomplishment. CVs are planned assessments accomplished through the use of a checklist and random sampling methods such as, but not limited to, direct operational observation, discussions with SDP personnel, review of voice, text or graphical data, equipment parameters, certification parameters, and examination of other documentation. The CVT is a national database that contains information related to the CV process. Information includes checklists, reports, facility information, tracking information, response data, and other statistical information available on the CVT website. Information contained in reports, mitigation plans, status reports, and closure is submitted through this database system. d. ICV. (1) ICVs are to be conducted by an ICV team designated by the FCFSS SDP manager or designee. (2) All FCFSSs must conduct an ICV annually each fiscal year. The schedule for conducting an ICV will be determined at the SDP facility level in order to produce the most comprehensive evaluation results. (3) ICVs must evaluate all items on the CVT checklist except those not applicable to the specific facility type. Any ATO entity may propose the addition or deletion of a checklist item(s). FCFSS requests for checklist modification must be forwarded to the Flight Service Directorate for evaluation and possible incorporation. (4) SDPs must complete an assessment and report of the efficiency and effectiveness of the following training items at least once per calendar year: (a) Qualification training for ATCSs, specialists, OSs, and controllers-in­ charge/DLSs. (b) Proficiency training for ATCSs, specialists, OSs. (c) Position certification process for ATCSs, specialists, OSs. (d) Return-to-duty process. (e) OJT instructor selection, certification, and evaluation. B-3

10/01/2020 JO 7210.634A Appendix B (5) The ICV team will compile the assessment findings in a memorandum to the Air Traffic Manager/Operations Manager (OM). e. ECV. (1) The Flight Service Directorate must ensure that ECVs are conducted at all FCFSS SDPs. ECVs must be conducted by the Flight Service Directorate. The ECV should focus on the operations and training at an SDP and items that have a direct impact on its effectiveness. (2) The checklist used for ECVs will be developed by the Flight Service Directorate. ECV checklists must include items provided by service units, directors of Operations, district managers, ATO Safety, and service area quality assurance offices. The ECV checklist must also include items assessing the efficiency and effectiveness of a facility’s technical trainingprogram. (3) An ECV will be conducted at the SP level annually. NOTE– This is accomplished by conducting ECV evaluations at the SDP facility level alternately every other year. (4) Anytime during an ECV, the team may observe operational items that represent a significant safety risk. After advising the SDP manager, the ECV team would then focus on those items whether or not they are included on the checklist. (5) The ECV team will conduct an in-briefing with the SDP manager. The briefing will include an introduction of team members and ECV expectations. f. Findings. (1) The ICV and the ECV will assess checklist items using the following categories and enter applicable details into the CVT: (a) Satisfactory (S): This finding is assigned to checklist items that are completed in compliance with national, service area, and local requirements. Details are not required to be entered into the CVT. (b) Unsatisfactory (U): This finding is assigned to checklist items that are non- compliant. (c) Not Observed (N/O): This finding is assigned to checklist items that are not observed during the verification. The reason the item was not observed must be documented. (d) Not Applicable (N/A): This finding is assigned to checklist items that are not applicable to the facility being evaluated. (e) Not Rated (N/R): The N/R finding is assigned to any checklist item that is applicable to the facility but for various reasons, such as time limitations, is not evaluated. B-4

10/01/2020 JO 7210.634A Appendix B (2) Prior to leaving the SDP, the ECV team lead must brief the SDP manager on all items that will be rated as non-compliant. (3) ECV reports must be completed and submitted into the CVT within 10 calendar days. The report must list all items rated and include associated details for all items rated “U.” (4) Conformity Index (CI). Each on-site ECV conducted by the Flight Service Directorate must include a CI. The Flight Service Directorate acknowledges that no two SDP facilities are identical; therefore, CIs are not intended to compare facilities. The intent of the CI is to numerically depict a facility’s overall compliance with directives/regulations and to assist with identifying “at-risk” facilities for non-compliance. g. Responding to Findings for ICVs and ECVs. (1) Corrective Action Plan. The corrective action plan communicates how specific risks will be managed, the action steps that are required to carry them out, and a timeline to accomplish identified tasks. This promotes a clear sense of the actions that are expected to be taken and provides management with an understanding of what actions are being taken on their behalf to effectively manage risk. (2) SDPs must respond to applicable non-compliant checklist items by developing a corrective action plan to correct the item. To close the item, the facility must document the processes used to ensure the effectiveness of the action and enter it into the CVT. (3) For “U” items representing a significant safety risk to the NAS, SDPs must, within 10 calendar days of being notified by the ECV team, develop a CAP to correct the item and obtain Flight Service Directorate concurrence. If a significant safety risk to the NAS is identified during an ICV, the SP must notify the FAA immediately of the issue and forward to the FAA a CAP within 10 calendar days. To close the item, the facility must document the processes used to ensure the effectiveness of the action and enter it into the CVT. (4) All “U” items must be closed and appropriately documented in the CVT no later than 180 calendar days after the date of the associated ICV or ECV out briefing. (5) Special evaluation responses are at the discretion of Flight Service Directorate. (6) The FAA may specify alternate response methods for the SP when appropriate. h. Procedures for Evaluating Federal Contract FSS. (1) Responsibilities. (a) Flight Service Directorate must: i. Ensure that an SP annual evaluation program is developed and implemented. ii. Be responsible for conducting evaluations at FCFSSs. B-5

10/01/2020 JO 7210.634A Appendix B iii. Maintain a national database of evaluation information for analysis. iv. Review the evaluation process continually to ensure its efficiency and effectiveness. v. Review documentation on closed problem areas. (b) Facility managers/OMs must: i. Promptly initiate steps to correct “U” items representing a significant risk to the NAS when notified by the lead evaluator following paragraph g. ii. Prepare a response addressing measures taken to correct all “U” items identified during or appended to ICVs, ECVs, FUEs, and special evaluations. Responses must be prepared and submitted according to paragraph g. iii. When resources permit, provide personnel to participate in compliance verifications. B-3. Evaluation Process. a. ECV. (1) Preparation and notification. An ECV, utilizing the appropriate checklist(s) in the CVT, will be conducted at all FCFSS facilities. The Flight Service Directorate will notify the facility manager prior to conducting an ECV. (2) In-briefing. An in-briefing is conducted for the purpose of introductions and should include a short discussion of anticipated evaluation activities while on-site. (3) Conducting the evaluation. Evaluators must conduct the ECV using all or some of the following methods: direct observation, position and/or voice/audio/data monitoring, observation of training activities, review of administrative records, and interviews. To avoid unwarranted “N/O” ratings, evaluators will use every means available to verify items not readily observable. Additionally, representatives from adjacent air traffic control facilities, FAA, and non-FAA offices (customers, fixed-base operators, airport management personnel, etc.) may be interviewed. (4) Daily briefing. The lead evaluator will normally provide the facility manager or designee with a daily briefing on the progress of the evaluation. (5) Out-briefing. The facility manager or designee must be briefed on the evaluators’/evaluation team’s findings at the conclusion of the evaluation. This may take place at the facility or by telephone conference. (6) Re-identified items. Items that are re-identified as Unsatisfactory “U” during ECVs must retain that rating. The appropriate closure process is required. B-6

10/01/2020 JO 7210.634A Appendix B b. Special Evaluations. An evaluation, other than an ECV or ICV, done infrequently to support ad hoc requirements. (1) Preparation and notification. The Flight Service Directorate must coordinate with the requesting office and notify the subject facility or organization through the appropriate manager. (2) Conducting the evaluation. The in-briefing, evaluation, and out-briefing must be conducted at the direction of Flight Service Directorate and the requesting office. (3) Report Content. The Flight Service Directorate and the manager who requested the evaluation must determine the areas to be evaluated and the report format. If a special evaluation is conducted at a field facility, the findings must be documented in the CVT. The Flight Service Directorate must make appropriate distribution. c. Appended Items. Coordination with the Flight Service Directorate and facility/operation managers must be accomplished before appending a new item to the ECV report. A separate report identifying the appended item(s) must be entered into the CVT. The appropriate closure process is required. Under the following circumstances, new items and previously identified “U” items maybe appended to the most recent ECV report: (1) While monitoring inter-facility operations during evaluations. For example, a problem may be identified at one facility while evaluating another. (2) As a result of findings emanating from a Mandatory Occurrence Report, accident, or incident. B-7

10/01/2020 JO 7210.634A Appendix C Appendix C. Quality Control Elements Data Collection WHAT WHY HOW HOW OFTEN WHAT TO LOOK FOR QC OSA Primary method to assess facility technical performance – assessments NOT outcome driven Must have a facility sampling plan that ensures ALL operational positions are assessed on a continual basis Mandatory quarterly target by facility type, number of positions, areas, and/or personnel Potential trends of systemic facility non-compliance/risk to be assessed in the Service Review process EMPHASIS ITEMS x Monitoring CAP effectiveness x Validation of suspected trends x Emphasizing special focus issues Created from customized sub- tasks in OSA module in CEDAR EIs must have defined start/stop time periods No recommended target The need for EIs should be reviewed regularly based on suspected issues, implemented CAPs, and potential special focus issues x CAP effectiveness x Suspected issues valid/invalid x Initial data collected – was awareness heightened and/or does an issue exist for SSR review? MOR DATA Method to identify potential trends of systemic facility risk/ non­ compliance – should be used in conjunction with other QC data Review aggregate data (by type and geography) for potential trends – potential trends MUST be assessed through SSR process No recommended target MOR data should be reviewed on a regular basis – facilities may set a regular schedule (monthly/quarterly) Aggregate data associated with: x Sectors/positions x Specific phase of flight x Type of occurrence x Airspace configuration, etc. PFS SAFETY PORTAL DATA x Safety portal merges multiple data sources x Involves front-line employees x Provides powerful tools Local Safety Council may review Safety Portal data on a continual basis – defined in local scoping document No recommended target Frequency of data review determined locally through LSC scoping document x Potential systemic facility issues x Potential system improvements COMPLIANCE VERIFICATIONS x Monitoring CAP effectiveness x Indicator of targeted QC OSAs, service reviews and EIs x Identification of potential systemic facility non- compliance Internal – collaborative facility teams use standardized checklists to assess facility compliance External – collaborative teams use dynamic checklist to assess facility compliance Internal CVs must be conducted once each fiscal year External CVs are conducted on an “as-needed” basis – data-driven or directed by operational leadership x Assess compliance with each required checklist item – compare actual performance to requirement x Any non-compliance must be assessed for seriousness (high/low) and documented in the CV tool BARRIER ANALYSIS REVIEW DATA Severity and likelihood are assessed; explanatory factors are identified. Trends found, combined with QC data, can aid facilities in local problem identification. Suspected trends could result in CSBR. Explanatory factors can be requested from AJI QA staff located in the applicable service area office, or BAR data can be reviewed via available ATO dashboards. Recommended that facilities request regular updates to Barrier Analysis data from AJI QA staff, or review available safety data dashboards on a regular basis. x Aggregate trends of explanatory factors and safety barriers, along with specific operations of interest x Compare data to QC data x Use CSBR or SSR/SYSIR process to fully understand trends before taking corrective action(s) C-1

10/01/2020 JO 7210.634A Appendix C Assess/Analyze WHAT WHY HOW HOW OFTEN WHAT TO LOOK FOR SYSTEM SERVICE REVIEW (SSR) • Analyze collected data to validate facility systemic non­ compliance or risk • Understand systemic non­ compliance/risk • Develop corrective action plans • Must use collaborative teams • Must use CEDAR question tree to explore all subject areas • Must develop CAPs for all validated issues No recommended target • Required for all yellow/red (non-accident) occurrences • May be conducted post- event • Must be conducted on a regular and random basis – facility needs schedule plan • Systemic non-compliance and risk • Validate non-compliance/risk • Scope of valid non­ compliance/risk • Underlying causal factors of valid non- compliance/risk • Develop CAP for valid issues COVERED EVENT REVIEW (CER) • Review post-accident services • Validate/understand facility systemic non-compliance/risk • Review services provided by employees • Develop individual/facility corrective action plans • Collaborative teams must use CEDAR question tree • Must review both systemic issues and individual performance • Training via ERC when ATSAP is submitted • Must develop CAPs for all validated systemic issues No recommended target • Required for all fatal accidents in which AT services provided, and must be completed within 5 admin days of the accident • Must never be started until ALL required SAR, notifications, and SRTs are completed • Systemic non-compliance/risk • Validate non-compliance/risk • Individual provision of AT services – need for training • Scope of valid non­ compliance/risk • Underlying causal factors of valid non- compliance/risk • Develop CAP for valid issues TRAFFIC MANAGEMENT REVIEW (TMR) • Review significant delay events, special events, and randomly to identify issues that may impact system efficiency • Develop corrective action plans • Must use collaborative teams • Must use CEDAR question tree to explore all subject areas • Must develop CAPs for all validated issues No recommended target • Randomly • After significant delay events • After special events • When requested by organizational leadership Identify issues associated with: • TMIs – triggers, scope, effectiveness •Coordination/ communication/execution • Facility performance/knowledge gaps • Develop CAP for valid issues SYSTEMIC ISSUE REVIEW (SYSIR) Identify, review, validate and address potential facility systemic issues • SYSIRs can be identified by anyone conducting an OSA, SSR, CER, or CSBR • Collaborative review process • Facility assigned POC must review and No recommended target • Initial identification can be done by any OSA reviewer, or any SSR/CER or CSBR team • Facility POC(s) must address all SYSIRs • Validate non-compliance/risk • Scope of non-compliance/risk • Underlying causal factors of valid non-compliance/risk • Develop CAP for valid (concur) issues concur, concur/elevate, or non-concur with SYSIR • Concurrence requires CAP C-2

10/01/2020 JO 7210.634A Appendix C WHAT WHY HOW HOW OFTEN WHAT TO LOOK FOR INTERNAL COMPLIANCE Uses standardized checklists to Identify facility non­ compliance/risk in areas that may not be reviewed through other QC processes • Collaborative process • Assess facility compliance with specific requirements using a standardized checklist • ICV Teams must: • Monitor operations • Review facility data in Once each fiscal year • Instances of non­ compliance/risk • Nature/scope of valid non­ compliance/risk • Underlying causal factors of valid non-compliance/risk • Develop CAP for valid issues (non-compliant high requires immediate action) VERIFICATIONS (ICV) CEDAR • Review local orders, SOPs, LOAs • Conduct interviews • Findings must be documented in CV Tool • Non-compliance/risk requires CAP C-3

10/01/2020 JO 7210.634A Appendix C Corrective Action Plans Must be documented in CEDAR/CV Tool as appropriate WHAT WHY HOW HOW OFTEN WHAT TO LOOK FOR SPECIFIC MITIGATIONS To correct specific validated systemic facility non­ compliance and risk • Training • Changes to procedures/processes • Airspace realignment • LOA changes • Changes to local policies • Other actions Whenever systemic non­ compliance/risk is validated and understood through a QC process, corrective actions must be developed and implemented to fix the problem Corrective actions should: • Specifically address the problem • Target the appropriate audience • Be clearly defined SCOPE To ensure corrective actions are targeted to the correct audience (facility-wide, tower- only, ops area only) Assess the nature of the non­ compliance/risk and determine how broadly it exists across the facility Scope should be assessed for every CAP developed by a facility Determine prevalence of non-compliance/risk across: • Operational areas • Types of operations (runway crossing, final approach, en route, arrival sector) • Phases of flight (takeoff, climb/descent level TIME FRAME CAPs must be implemented within a specific time frame to ensure completion and to measure actual execution Assess specific corrective actions and scope to determine appropriate time frame for implementation Implementation time frame must be assessed and defined for every CAP developed by a facility Time frames should: • Allow sufficient time for implementation • Ensure facility is aggressive in correcting the problem MONITORING PLAN Facility performance must be monitored to assess effectiveness of corrective actions Using collected/reported data is the primary method for monitoring CAPs Methods: • Emphasis Items • Required 7230-4 entries • Internal CVs • PFS Safety Portal • MOR Data • Barrier Analysis Review Data and Combined Safety Barrier Review Data Must identify who is responsible for monitoring • All CAPs must have a monitoring plan to assess effectiveness • Monitoring plans must have defined time period • Nature of non­ compliance/risk • Type of corrective action(s) • Match monitoring methods to specific corrective actions • Use multiple monitoring methods • Sufficient time/data to assess CAP effectiveness EFFECTIVENESS TARGET Facility performance data collected in monitoring plans must be measured to a defined benchmark to determine CAP effectiveness Determine required level of compliance for monitoring method(s): • % of compliance • % reduction in causal factors • ICV checklist item rated compliant, etc. Each monitoring plan associated with every CAP must have a pre­ defined effectiveness target CAP modification required if target not met Effectiveness targets: • Must ensure improved facility compliance and exposure to risk • May vary depending on severity of non­ compliance or risk • Set high standard for compliance C-4

09/01/2021 JO 7210.634A CHG 1 Appendix C Data Integrity WHAT WHY HOW WHAT TO LOOK FOR QC OSA VALIDATIONS QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all quality control processes to ensure the effective identification of non-compliance/risk and associated CAPs. Validations require facilities to review specific data collection processes to ensure accurate documentation of facility performance. Facilities must develop local sampling plans that include: • Target number of validations by week/month/quarter: Recommend facilities validate a minimum of 15% of all QC OSAs conducted each quarter. • Sampling method to ensure randomness and all sectors/positions are reviewed. • Replay tools to be used (radar AND voice where available). • Schedule that ensures data availability (within data retention time periods). • Feedback process to reviewers. • Follow-up process to ensure effectiveness of feedback. • Must be conducted collaboratively. • Must be documented in CEDAR. • Compare demonstrated performance to documentation to ensure accuracy. • Note discrepancies. • Must only be used to identify organizational or systemic issues. Reminders: • Validations cannot negatively impact an employee’s overall performance evaluation (previous IPM OSA). • Employees must not be decertified as a result of a Certification Skill Check validation. CERTIFCATION SKILL CHECK/ PERFORMANCE ASSESSMENT QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all quality control processes to ensure the effective identification of non-compliance/risk and associated CAPs. Facilities must develop local sampling plans that include: • Target number of validations by week/month/quarter: Recommend facilities validate a minimum of 15% of all Certification Skill Checks / Performance Assessments conducted each quarter • Sampling method to ensure randomness and all sectors/positions are reviewed. • Replay tools to be used (radar AND voice where available). • Schedule that ensures data availability • Compare demonstrated performance to documentation to ensure accuracy. • Note discrepancies. • Must only be used to identify organizational or systemic issues. Reminders: • Validations cannot negatively impact an VALIDATIONS Validations require facilities to review specific data collection processes to ensure accurate documentation of facility performance. (within data retention time periods). • Feedback process to reviewers. • Follow-up process to ensure effectiveness of feedback. • Must be conducted collaboratively. • Must be documented in CEDAR. employee’s overall performance evaluation (previous IPM OSA). • Employees must not be decertified as a result of a Certification Skill Check validation. OJT DOCUMENTATION VALIDATIONS QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all quality control processes to ensure the effective identification of non-compliance/risk and associated CAPs. Validations require facilities to review specific data collection processes to ensure accurate documentation of facility performance. Facilities must develop local sampling plans that include: • Sampling method to ensure randomness and all sectors/positions are reviewed (required target numbers already defined in QC order for OJTD Validations). • Replay tools to be used (radar AND voice where available). • Schedule that ensures data availability (within data retention time periods). • Feedback process to reviewers. • Follow-up process to ensure effectiveness of feedback. • Must be conducted collaboratively. • Must be documented in CEDAR. • Compare demonstrated performance to documentation to ensure accuracy. • Note discrepancies. • Must only be used to identify organizational or systemic issues. Reminders: • Validations cannot negatively impact an employee’s overall performance evaluation (previous IPM OSA). • Employees must not be decertified as a result of a Certification Skill Check validation. C-5

10/01/2020 JO 7210.634A Appendix C Local QC Orders WHAT WHY CONTENT QC OSA SAMPLING PLAN QC order requires local sampling plan • Target number of OSAs per quarter (see recommended targets) • Schedule for conducting QC OSAs • Method to ensure all functions/positions are reviewed • Method to ensure random selection • Schedule for conducting QC OSAs RANDOM SSR/TMR PLAN QC order requires that SSRs and TMRs be conducted on a random or scheduled basis and not being solely conducted post-event – this ensures assessment of facility performance outside of reportable occurrences (negative outcomes) • Schedule for conducting SSR/TMRs • Method to ensure random selection • Method to ensure all functions/positions are periodically reviewed SYSIR POC DESIGNATIONS QC order requires facilities to designate a specific POC for each systemic issue subject area: • Training • Efficiency • Airspace/airport • Procedures • Directives • Equipment • Resource Management • Identify by position or name the POC for each subject area – this ensures appropriate routing of SYSIRs in CEDAR for review and closure FACILITY SAFETY REPORT REQUIREMENTS Recommend facilities develop local safety reports that provide operational management with visibility into facility performance • Type of data to be included • Frequency of report • Responsibility for creation of report QC VALIDATIONS SAMPLING PLANS QC order requires local sampling plans for all QC Validations • Target number of validations by week/month/quarter • Method to ensure randomness and all sectors/positions are reviewed • Replay tools to be used (radar AND voice where available) • Schedule that ensures data availability (within data retention time periods) • Feedback process to reviewers • Follow-up process to ensure effectiveness of feedback C-6

10/01/2020 JO 7210.634A Appendix C Facility Safety Reports WHAT WHY CONTENT STATUS OF REQUIRED QC PROCESSES Provides operational management with information about ongoing data collection and assessment/analysis of facility performance • QC OSAs conducted during reporting period • Emphasis Items – Existing EIs, new EIs, terminated EIs • Service Reviews conducted during reporting period – include each type service review and reason for each (post-event, random, targeted) • Status of current Internal CV – include percentage completion • QC Validations – include number reviewed and whether facility target met CORRECTIVE ACTION PLANS Provides status of all Corrective Action Plans; existing, new, and recently completed CAPs • New CAPs created during the reporting period – include monitoring plan • CAPs closed during the reporting period – include data on how effectiveness target was met • Status of existing CAPs not closed during reporting period – status of monitoring plans and preliminary findings if applicable VALIDATED FINDINGS Provides information on validated systemic facility non-compliance and risk (requires action plan) • Findings from service reviews and CSBR– topic areas from CEDAR question tree; include summary of selected (most serious) findings • Findings from Internal CV – non-compliant and exemplary items – include current status (open/closed) • Findings from QC Validations – level of compliance FACILITY REPORTING DATA Provides information on the health of the reporting culture and assists in potential trend identification to be assessed through service review processes • Number (by type) of MORs reported during reporting period C-7

10/01/2020 JO 7210.634A Appendix D Appendix D. Generic Facility Quality Control Order Large Tower/TRACON U.S. DEPARTMENT OF TRANSPORTATION FEDERAL AVIATION ADMINISTRATION XYZ 7210.634 Generic ATCT/TRACON Policy Effective date: MM/DD/YYYY SUBJ: Generic ATCT/TRACON Quality Control 1. Purpose of This Order. This order conveys requirements for facility Quality Control (QC) and audit processes in accordance with Federal Aviation Administration (FAA) Order JO 7210.634, Air Traffic Organization (ATO) Quality Control, and FAA Order JO 7210.633, ATO Quality Assurance Program (QAP). 2. Audience. This order applies to all Generic Air Traffic Control Tower (ATCT)/Terminal Radar Approach Control (TRACON) (XYZ) personnel. 3. Where Can I Find This Order? This order can be found on the facility ACE-IDS Facility Order page, in the facility library, and the FAA Facility Directives Repository website. 4. Distribution. This order is distributed to the Quality Control Group, Eastern Service Center, and to the Director, Air Traffic Services, (geographic location, e.g., South Central). 5. Background. Compliance with national QC and Quality Assurance (QA) directives requires air traffic facilities to develop local plans for operational sampling, analysis, validations, audits, checks, compliance verifications, emphasis items, Point of Contact (POC) designations, corrective actions, and internal safety reports. 6. Applicable Policy and Related Documents. a. FAA Order JO 7210.634 b. FAA Order JO 7210.633 c. FAA Order JO 1030.3, Initial Event Response d. NATCA FAA CBA 7. QC Operational Skills Assessment (QC OSA) Sampling Plan. a. Generic ATCT/TRACON will complete a minimum of 52 QC OSAs per quarter. This target includes QC OSAs on all types of positions (Certified Professional Controller (CPC), D-1

10/01/2020 JO 7210.634A Appendix D Front Line Manager (FLM) / Controller In Charge (CIC), Traffic Management Coordinator (TMC)). b. QC OSAs will be conducted by operational management and support staff. The support manager for Quality Control must collaborate with the operations managers to coordinate support staff and FLM accomplishment of QC OSAs and is responsible for ensuring quarterly targets are met. c. QC OSAs must be conducted on all operational positions in each quarter. The support manager for Quality Control is responsible for ensuring this requirement is met. d. QC OSAs will be no less than thirty (30) minutes in duration. e. QC OSAs will be conducted utilizing playback tools to the maximum extent possible. Falcon and voice data must be attached to each QC OSA in the Comprehensive Electronic Data Analysis and Reporting (CEDAR) tool. 8. Service Review Sampling Plans. Facilities must conduct service reviews randomly and post event. This section describes the specific triggers and methods for ensuring compliance with national requirements for service reviews. a. System Service Reviews (SSRs). (1) The purpose of SSRs is three-fold: To identify and understand active and latent factors that are causal to a reported safety occurrence; to validate (or invalidate) a potential facility-systemic issue and identify and understand the active and latent causal factors for validated issues; and to randomly assess air traffic services on a regular basis to ensure comprehensive reviews are conducted on all aspects of the facility’s operations. (2) SSRs are not a mechanism for individual performance management, however if performance is notable, it must documented in the SSR OSA (i.e., QC OSA, which is de­ identified) and combined with other QC OSAs to be used as aggregate data. (3) SSRs must be conducted outside of the operation. SSRs must be conducted collaboratively with union participation; the principal facility union representative (or their designee) must be afforded the opportunity to participate in SSRs. (4) SSRs must be documented in CEDAR. The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree (including the possibility of any systemic issues). (5) SSRs must be conducted after any non-fatal accident safetyoccurrence color-coded red or yellow per FAA Order JO 1030.3, JO 7210.634, and the NATCA/FAA CBA. (6) SSRs must be conducted to validate and understand a suspected facility-systemic issue initially identified through a review of facility data (OSA, Emphasis Item, Mandatory Occurrence Report (MOR), or Barrier Analysis Review (BAR)/Combined Safety Barrier Review (CSBR) data. D-2

10/01/2020 JO 7210.634A Appendix D (7) SSRs may be conducted in response to potential compliance issues associated with a reported safety occurrence. The support manager for Quality Control will determine when to conduct select post-event SSRs. (8) SSR-validated issues must have a corresponding Corrective Action Plan (CAP) or must be identified in CEDAR as a potential systemic issue to be addressed in the SystemicIssue Review (SYSIR) process. (9) Generic ATCT/TRACON must conduct a minimum of four (4) SSRs per month. An SSR must be conducted on every operational position a minimum of once per fiscal year. SSRs conducted post-event or triggered by potential systemic issues may satisfy this requirement. The support manager for Quality Control is responsible for ensuring that these requirements are met and will direct random SSRs as necessary. b. Traffic Management Reviews (TMRs). (1) The purpose of TMRs is to identify and understand active and latent factors that are causal to a significant delay or special event and to randomly assess traffic management services on a regular basis to ensure comprehensive reviews are conducted on facility operations that may impact system efficiency. (2) TMRs must be conducted outside of the operation. TMRs must be conducted collaboratively with union participation; the principal facility union representative (or their designee) must be afforded the opportunity to participate in TMRs. (3) TMRs must be documented in CEDAR. The collaborative team must thoroughly consider each of the required topics in the CEDAR question tree (including the possibility of any systemic issues). (4) The primary triggers for conducting a TMR are after significant delay events (due to weather, equipment outages, Traffic Management Initiatives, etc.), after special event activities (e.g., sporting events, fly-ins, Temporary Flight Restrictions, etc.), and at the request of operational management. (5) TMRs must be conducted when requested by Air Traffic Services or System Operations organizational leadership. (6) Generic ATCT/TRACON must conduct a minimum of two (2) TMRs per month. TMRs conducted post-delay or special event satisfy this requirement. The traffic management officer is responsible for ensuring that these requirements are met and will direct random TMRs as necessary. c. SYSIR POCs. The POCs for reviewing and adjudicating potential systemic issues are: (1) Training, directives, resource management – the support manager for Quality Control and training (2) Efficiency – the traffic managementofficer D-3

09/01/2021 JO 7210.634A CHG 1 Appendix D (3) Airspace/airport, procedures, and equipment – the support manager for airspace and procedures. 9. QC Validation Sampling Plans. The support manager for QC and training is responsible for ensuring that all QC Validations are conducted in accordance with the following requirements. a. OSA Validations. (1) Generic ATCT/TRACON will validate a minimum of four OSAs per month. This target includes QC OSAs on all types of positions (CPC, FLM/CIC, TMC). (2) OSA Validation samples should be randomly selected but must ensure that each sector/position is reviewed a minimum of once each six months. (3) Radar sector/position OSA Validations must use Falcon with voice to review and compare demonstrated technical performance against that documented in the original OSA. Tower validations must use voice data to review and validate the OSA. (4) Feedback. Systemic issues/trends identified through validations of OSAs will be forwarded to the support manager for QC and training for dissemination to personnel conducting OSAs. (5) Follow-up. The support manager for QC and training must review OSA Validation findings for a time period to be defined to determine if previously identified systemic issues/trends have been resolved. b. Certification Skill Check and Performance Assessment Validations. (1) Generic ATCT/TRACON will validate a minimum of 20% of all Certification Skill Checks and Performance Assessments conducted per quarter. (2) Certification Skill Check / Performance Assessment Validation samples must be randomly selected. Radar sector/position skill check / assessment validations must use Falcon with voice to review and compare demonstrated technical performance against that documented in the original Certification Skill Check / Performance Assessment. Tower validations must use voice data to review and validate the Certification Skill Check / Performance Assessment. (3) Feedback. Issues identified through these skill check / assessment validations will be forwarded to the appropriate operations manager for feedback to the supervisor performing the original Certification Skill Check or Performance Assessment. (4) Follow-up. The responsible operations manager is responsible for ensuring that identified issues are corrected. c. OJT Documentation Validations. (1) Validation samples for OJT documentation must be randomly selected. D-4

10/01/2020 JO 7210.634A Appendix D (2) Radar sector/position OJT Validations must use Falcon with voice to review and compare demonstrated technical performance against that documented in the original OJT documentation. Tower validations must use voice data to review and validate the OJT documentation. (3) Feedback. Systemic issues/trends identified through validations of OSAs will be forwarded to the support manager for Quality Control and training for dissemination to personnel conducting OJT. (4) Follow-up. The support manager for Quality Control and training must review OJT Documentation Validation findings for a time period to be defined to determine if previously identified systemic issues/trends have been resolved. 10. Facility Safety Report. Generic ATCT/TRACON must create a monthly safety report that reviews overall facility performance and ensures the facility’s QC program is operating as intended. The support manager for Quality Control and training is responsible for creating and updating this monthly report. The report shall be published no later than the 15th of each month and shall provide information from the previous month. The report shall include the following information. a. Update on QC processes conducted during the reporting period. QC process updates must include: (1) The number of QC OSAs conducted during the previous month and for the fiscal year through the previous month. (2) A list of all Emphasis Items to be reviewed during QC OSAs. The report must breakout Emphasis Items into new items created during the reporting month, items closed/deleted during the reporting month, and those that are still active. (3) Service reviews conducted during the previous month. Service reviews must be listed by type and include the reason (trigger) for each. (4) Status of the current fiscal year Internal Compliance Verification (ICV). Include the percentage of items completed and remaining. (5) The number of QC Validations conducted during the previous month and for the fiscal year through the previous month. Include whether these numbers are in compliance with the facility sampling plan. b. Update on facility reporting data during the period. Facility reporting updates must include the following items. (1) Number (by type) of MORs reported during the previous month and fiscal yearto date (2) Fatal accidents with air traffic control services during the previous month and fiscal year to date. D-5

10/01/2020 JO 7210.634A Appendix D (3) Significant events color-coded red or yellow during the previous month and fiscal year to date, per FAA Order JO 1030.3 and the NATCA/FAA CBA. c. Update on validated instances of systemic facility risk/non-compliance identified from available data and analyses. Updates on facility findings must include: (1) Validated systemic non-compliance/risk identified through any service review (SSR, Covered Event Review, TMR, SYSIR) or CSBR during the previous month. Issue descriptions must include the type service review conducted, the topic area identified in the CEDAR question tree, and brief synopsis of the issue. (2) Any compliance issues identified through QC Validations during the previous month. Issue descriptions must include the type validation conducted and a brief description of the non­ compliance. (3) Any items rated non-compliant during the current fiscal year facility ICV (update monthly). Issue descriptions must include the checklist item and its current status. d. Status of facility CAPs. This includes any CAP for Generic ATCT/TRACON. CAPs may be generated in response to service reviews, CSBR, ICV / External Compliance Validation, QC Validations, significant events, fatal accidents, and/or as required by the district manager/director of operations. Updates on facility CAPs must include: (1) Any new CAP generated during the previous month. The information for each CAP must include the process for identification, a description of the issue, the specific mitigations, and all aspects of the monitoring plan to determine mitigation effectiveness. (2) Any CAPs closed during the previous month. Information must include data collected in accordance with the monitoring plan and demonstrate how the mitigation effectiveness target was met. (3) Status of open CAPs not closed during the previous month. Information must include a brief description of the issue, status of the monitoring plan, and when applicable, any preliminary findings from data collected in accordance with the monitoring plan. D-6

09/01/2021 JO 7210.634A CHG 1 Appendix E Appendix E. Sample Fiscal Year (by Month) QC Activity Plan Large Tower / TRACON October Facility Quality Control Activity Plan (Example of one month of twelve) QC AREA ACTIVITY # WHAT REQUIREMENT REFERENCE Data Collection QC OSA 21 Conduct min. # QC OSAs 61 per quarter Facility OSA plan & FAA Order JO 7210.634 Data Collection ICV 0 Complete % of facility ICV Complete 25%/quarter Facility ICV plan & FAA Order JO 7210.634 Data Reviews QC OSA Subtasks/Emphasis Items 2 Review OSA data to ID trends Review OSA data –2x/mo. Facility QC plan/order Data Reviews MOR Data 2 Review MOR aggregate data (by type/location) to ID trends Review MOR data – 1x/mo. Facility QC plan/order Data Reviews Barrier Analysis Data 1 Review Barrier Analysis explanatory factor data/CSBR data Obtain from SA QA or via dashboard/CEDAR – 1x/mo. Facility QC plan/order Data Reviews ICV Data 1 Review completed ICV checklist items & compare to OSA & MOR Data Review completed checklists items – 1x/mo. Facility QC plan/order Analysis SSR 4+ x Conduct random/scheduled SSRs x Assess potential trends identified in Data Reviews x Conduct post-event for all red/yellow event (non­ accident) x Random/scheduled – 4x/mo. x Post-event and to assess potential trends ­ as necessary Facility SSR plan, FAA Order JO 7210.634, and FAA Order JO 1030.3 Analysis CER TBD Conduct post-accident CERs Conduct after ALL fatal accidents with ATC services FAA Order JO 1030.3 & FAA Order JO 7210.634 Analysis TMR 2+ x Conduct random/scheduled x Conduct after significant delay and special events x Random/scheduled – 2x/mo. x After significant delay and special events – as necessary Facility TMR plan & FAA Order JO 7210.634 Analysis SYSIR TBD Conduct SYSIRs on all potential systemic issues flagged through OSAs, Service Reviews, and CSBR Conduct – as necessary FAA Order JO 7210.634 Data Integrity OSA Validation 7 Validate min. # OSAs each month Validate min. 7 OSAs/mo. Facility QC plan & FAA Order JO 7210.634 Data Integrity Cert Skill Check and Performance Assessment Validation 7 Validate min. # Cert Skill Checks / Performance Assessments each month Validate min. 7 cert. skill checks/performance assessments/mo. Facility QC plan & FAA Order JO 7210.634 Data Integrity OJT Validation 5 Validate min. # OJT forms each month Validate min. 5 OJT forms/mo. Facility QC plan & FAA Order JO 7210.634 E-1

09/01/2021 JO 7210.634A CHG 1 Appendix E QC AREA ACTIVITY # WHAT REQUIREMENT REFERENCE CAPs Review CAPs generated from Service Reviews, CVs, and CSBR 1 NEW – ensure contains ALL elements/review monitoring plan Review min. 1x/mo. or as CAPs are created Facility QC plan CAPs Review CAPs generated from Service Reviews, CVs, and CSBR 1 OPEN – review monitoring data/assess mitigation performance Review min. 1x/mo. Facility QC plan CAPs Review CAPs generated from Service Reviews, CVs, and CSBR 1 CLOSED – ensure effectiveness target met Review min. 1x/mo. or as CAPs are closed Facility QC plan Reports Facility Safety Report 1 Produce monthly safety report Publish facility safety report 1x/mo. Facility QC plan E-2

09/01/2021 JO 7210.634A CHG 1 Appendix F Appendix F. Technical Operations Services SSR, CAP, and SYSIR Templates This appendix contains templates for Technical Operations Services SSRs, CAPs, and SYSIRs based on the questions and required information in the safety database. It is intended to assist users in preparing information and data for entering results into the safety database. The current safety database is the Comprehensive Electronic Data Analysis and Reporting (CEDAR) tool located at https://cedar.faa.gov. F-1. Technical Operations Services System Service Review (SSR) Template. * Indicates required item Review Categories Response Explain Preliminary Data: Date, Start and End Time (UTC) (of Event)*. Short Description (of Event)* e.g. event summary that provides a short description of event being reviewed, sequence of events, and operational impacts. Additional Information: FSEP Facility Type, FSEP Facility ID.* Location (City, State)* RMLS Data: RMLS Log ID* NAS Systems and Services: Were any equipment issues identified?* If Yes: Was there any maintenance occurring (PM’s, Correctives, Mod’s, Troubleshooting, etc.)?* (Whether response to previous question is Yes or No, the following 3 questions require date selection) When was the last PM completed? [Date]* When was the last certification completed? [Date]* When was the last unscheduled outage? [Date]* Were any telecommunications issues identified?* Did any TPR issues or trends exist?* Was any service degradation identified prior to the event?* Were there any system modification issues?* Were there any maintenance alert issues (not completed, expired, etc.)?* Were any adaptation issues identified?* Were any airspace and procedure issues identified?* Did diversity, redundancy, or back-up system issues exist?* F-1

09/01/2021 JO 7210.634A CHG 1 Appendix F Review Categories Response Explain Were any potential systemic NAS system and/or service issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Coordination: Were any coordination issues identified?* If Yes: Was coordination (pre, final, and post) properly completed with all customers and stakeholders?* Could coordination or communication with stakeholders have been improved?* Was a NOTAM issued?* Were any potential systemic coordination issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Procedures/Directives/Documentation: Were any procedural, directive, or documentation issues identified (local, service area, or national)?* If Yes: Were personnel unfamiliar with the application of the procedure or directive?* Were any issues identified with any other reference documentation or technical drawings?* Were any potential systemic procedural/directive/documentation issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Impact Mitigation: Were any Safety Risk Management and Operational Risk Management (e.g. IRMC) issues identified?* Were there recently similar outages or trends?* Were any contingency plan issues identified?* Did anything delay service or system recovery?* Was fatigue an issue?* Were any potential systemic impact mitigation issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Resource Management: Were any resource management issues identified?* If Yes: Did any supervision or technical oversight issues exist (FLM, DFM, or SSC Coordinator)?* Were any issues with operational coverage identified?* Were any potential systemic resource management issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) F-2

09/01/2021 JO 7210.634A CHG 1 Appendix F Review Categories Response Explain Training: Were any training issues identified?* If Yes: Was training in progress at the time of the event?* Were any potential systemic training issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Stakeholder and Customer Feedback: Was there any stakeholder or customer feedback?* If Yes: Did the feedback express praise?* Did the feedback express concern?* Was the feedback validated?* Individual Performance: Was any notable individual performance identified?* (Exemplary and/or needing improvement) If Yes: Where was individual performance identified?* Facility Type:* [Drop Down: Navigational, Communications, Surveillance, Weather, Environmental, Control Center] Maintenance Organization:* [Drop Down: SSC, Control Center, Technical Support, Engineering Services, Contractor] ATSAP/TSAP filed? Supporting Data: Add Attachment: [Drop Down: Falcon Bookmark, Airborne Replay, Surface Replay, Voice Recording, Other] Findings and Corrective Actions: Is corrective action required?* If Yes: Corrective Action:* [Link: Create CAP] QC Comments:* Forward: Recipient: [Select Name] (Note: Can add multiple recipients) [] Send as Informational []Delegate []Cancel Comments: F-3

09/01/2021 JO 7210.634A CHG 1 F-2. Technical Operations Services Corrective Action Plan (CAP) Template. * Indicates required item Basic Information: Subject of CAP:* Safety Monitoring Watchlist: Identified Safety Risk or Hazard: Risk or Hazard:* Background Information: Detailed Description of Validated Safety Issue(s):* Pertinent Regulations: Add Regulation: Order/Directive Number: [Drop Down: 6000.15, 7110.65, 7210.3, Other] Reference: Description: Corrective Action Plan:* Corrective Action #1: (Note: More than one corrective action can be added in the safety database) Type:* [Drop Down: Briefing, Review, Airspace, Directive, Procedure, Equipment, Training, Other] Level:* [Drop Down: Headquarters, Service Area, District, Facility] Target Completion Date:* Details:* F-4

09/01/2021 JO 7210.634A CHG 1 Appendix F Monitoring Plan:* Monitoring Item #1: Type:* [Drop Down: Review] Level:* [Drop Down: Headquarters, Service Area, District, Facility] Target Completion Date:* Details:* Effectiveness Target:* Supporting Data: Attachment Type: [Drop Down: Falcon Bookmark, Airborne Replay, Surface Replay, Voice Recording, Other] Forward: Recipient: [Select Name] (Note: Can add multiple recipients) [] Send as Informational []Delegate []Cancel Comments: F-5

09/01/2021 JO 7210.634A CHG 1 Appendix F F-3. Technical Operations Services Systemic Issue Review (SYSIR) Template. * Indicates required item Date: Systemic Issues: Select the type of systemic issue identified:* [Drop Down: NAS Systems and Services, Coordination, Procedural/Directive/Documentation, Impact Mitigation, Resource Management, Training] Systemic Issue:* Add Attachment (.xls, .xlsx, .doc, .docx, .pdf, .png, .jpg, .gif): Enter Specific Details:* F-6

09/01/2021 JO 7210.634A CHG 1 Appendix C Data Integrity WHAT WHY HOW WHAT TO LOOK FOR QC OSA VALIDATIONS QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all quality control processes to ensure the effective identification of non-compliance/risk and associated CAPs. Validations require facilities to review specific data collection processes to ensure accurate documentation of facility performance. Facilities must develop local sampling plans that include: • Target number of validations by week/month/quarter: Recommend facilities validate a minimum of 15% of all QC OSAs conducted each quarter. • Sampling method to ensure randomness and all sectors/positions are reviewed. • Replay tools to be used (radar AND voice where available). • Schedule that ensures data availability (within data retention time periods). • Feedback process to reviewers. • Follow-up process to ensure effectiveness of feedback. • Must be conducted collaboratively. • Must be documented in CEDAR. • Compare demonstrated performance to documentation to ensure accuracy. • Note discrepancies. • Must only be used to identify organizational or systemic issues. Reminders: • Validations cannot negatively impact an employee’s overall performance evaluation (previous IPM OSA). • Employees must not be decertified as a result of a Certification Skill Check validation. CERTIFCATION SKILL CHECK/ PERFORMANCE ASSESSMENT QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all quality control processes to ensure the effective identification of non-compliance/risk and associated CAPs. Facilities must develop local sampling plans that include: • Target number of validations by week/month/quarter: Recommend facilities validate a minimum of 15% of all Certification Skill Checks / Performance Assessments conducted each quarter • Sampling method to ensure randomness and all sectors/positions are reviewed. • Replay tools to be used (radar AND voice where available). • Schedule that ensures data availability • Compare demonstrated performance to documentation to ensure accuracy. • Note discrepancies. • Must only be used to identify organizational or systemic issues. Reminders: • Validations cannot negatively impact an VALIDATIONS Validations require facilities to review specific data collection processes to ensure accurate documentation of facility performance. (within data retention time periods). • Feedback process to reviewers. • Follow-up process to ensure effectiveness of feedback. • Must be conducted collaboratively. • Must be documented in CEDAR. employee’s overall performance evaluation (previous IPM OSA). • Employees must not be decertified as a result of a Certification Skill Check validation. OJT DOCUMENTATION VALIDATIONS QC data collected by facilities is the foundation for local, service area, and national compliance assessments and corrective actions. It is imperative that facilities accurately capture data collected in all quality control processes to ensure the effective identification of non-compliance/risk and associated CAPs. Validations require facilities to review specific data collection processes to ensure accurate documentation of facility performance. Facilities must develop local sampling plans that include: • Sampling method to ensure randomness and all sectors/positions are reviewed (required target numbers already defined in QC order for OJTD Validations). • Replay tools to be used (radar AND voice where available). • Schedule that ensures data availability (within data retention time periods). • Feedback process to reviewers. • Follow-up process to ensure effectiveness of feedback. • Must be conducted collaboratively. • Must be documented in CEDAR. • Compare demonstrated performance to documentation to ensure accuracy. • Note discrepancies. • Must only be used to identify organizational or systemic issues. Reminders: • Validations cannot negatively impact an employee’s overall performance evaluation (previous IPM OSA). • Employees must not be decertified as a result of a Certification Skill Check validation. C-5

09/01/2021 JO 7210.634A CHG 1 Appendix D (3) Airspace/airport, procedures, and equipment – the support manager for airspace and procedures. 9. QC Validation Sampling Plans. The support manager for QC and training is responsible for ensuring that all QC Validations are conducted in accordance with the following requirements. a. OSA Validations. (1) Generic ATCT/TRACON will validate a minimum of four OSAs per month. This target includes QC OSAs on all types of positions (CPC, FLM/CIC, TMC). (2) OSA Validation samples should be randomly selected but must ensure that each sector/position is reviewed a minimum of once each six months. (3) Radar sector/position OSA Validations must use Falcon with voice to review and compare demonstrated technical performance against that documented in the original OSA. Tower validations must use voice data to review and validate the OSA. (4) Feedback. Systemic issues/trends identified through validations of OSAs will be forwarded to the support manager for QC and training for dissemination to personnel conducting OSAs. (5) Follow-up. The support manager for QC and training must review OSA Validation findings for a time period to be defined to determine if previously identified systemic issues/trends have been resolved. b. Certification Skill Check and Performance Assessment Validations. (1) Generic ATCT/TRACON will validate a minimum of 20% of all Certification Skill Checks and Performance Assessments conducted per quarter. (2) Certification Skill Check / Performance Assessment Validation samples must be randomly selected. Radar sector/position skill check / assessment validations must use Falcon with voice to review and compare demonstrated technical performance against that documented in the original Certification Skill Check / Performance Assessment. Tower validations must use voice data to review and validate the Certification Skill Check / Performance Assessment. (3) Feedback. Issues identified through these skill check / assessment validations will be forwarded to the appropriate operations manager for feedback to the supervisor performing the original Certification Skill Check or Performance Assessment. (4) Follow-up. The responsible operations manager is responsible for ensuring that identified issues are corrected. c. OJT Documentation Validations. (1) Validation samples for OJT documentation must be randomly selected. D-4

09/01/2021 JO 7210.634A CHG 1 Appendix E Appendix E. Sample Fiscal Year (by Month) QC Activity Plan Large Tower / TRACON October Facility Quality Control Activity Plan (Example of one month of twelve) QC AREA ACTIVITY # WHAT REQUIREMENT REFERENCE Data Collection QC OSA 21 Conduct min. # QC OSAs 61 per quarter Facility OSA plan & FAA Order JO 7210.634 Data Collection ICV 0 Complete % of facility ICV Complete 25%/quarter Facility ICV plan & FAA Order JO 7210.634 Data Reviews QC OSA Subtasks/Emphasis Items 2 Review OSA data to ID trends Review OSA data –2x/mo. Facility QC plan/order Data Reviews MOR Data 2 Review MOR aggregate data (by type/location) to ID trends Review MOR data – 1x/mo. Facility QC plan/order Data Reviews Barrier Analysis Data 1 Review Barrier Analysis explanatory factor data/CSBR data Obtain from SA QA or via dashboard/CEDAR – 1x/mo. Facility QC plan/order Data Reviews ICV Data 1 Review completed ICV checklist items & compare to OSA & MOR Data Review completed checklists items – 1x/mo. Facility QC plan/order Analysis SSR 4+ x Conduct random/scheduled SSRs x Assess potential trends identified in Data Reviews x Conduct post-event for all red/yellow event (non­ accident) x Random/scheduled – 4x/mo. x Post-event and to assess potential trends ­ as necessary Facility SSR plan, FAA Order JO 7210.634, and FAA Order JO 1030.3 Analysis CER TBD Conduct post-accident CERs Conduct after ALL fatal accidents with ATC services FAA Order JO 1030.3 & FAA Order JO 7210.634 Analysis TMR 2+ x Conduct random/scheduled x Conduct after significant delay and special events x Random/scheduled – 2x/mo. x After significant delay and special events – as necessary Facility TMR plan & FAA Order JO 7210.634 Analysis SYSIR TBD Conduct SYSIRs on all potential systemic issues flagged through OSAs, Service Reviews, and CSBR Conduct – as necessary FAA Order JO 7210.634 Data Integrity OSA Validation 7 Validate min. # OSAs each month Validate min. 7 OSAs/mo. Facility QC plan & FAA Order JO 7210.634 Data Integrity Cert Skill Check and Performance Assessment Validation 7 Validate min. # Cert Skill Checks / Performance Assessments each month Validate min. 7 cert. skill checks/performance assessments/mo. Facility QC plan & FAA Order JO 7210.634 Data Integrity OJT Validation 5 Validate min. # OJT forms each month Validate min. 5 OJT forms/mo. Facility QC plan & FAA Order JO 7210.634 E-1

09/01/2021 JO 7210.634A CHG 1 Appendix E QC AREA ACTIVITY # WHAT REQUIREMENT REFERENCE CAPs Review CAPs generated from Service Reviews, CVs, and CSBR 1 NEW – ensure contains ALL elements/review monitoring plan Review min. 1x/mo. or as CAPs are created Facility QC plan CAPs Review CAPs generated from Service Reviews, CVs, and CSBR 1 OPEN – review monitoring data/assess mitigation performance Review min. 1x/mo. Facility QC plan CAPs Review CAPs generated from Service Reviews, CVs, and CSBR 1 CLOSED – ensure effectiveness target met Review min. 1x/mo. or as CAPs are closed Facility QC plan Reports Facility Safety Report 1 Produce monthly safety report Publish facility safety report 1x/mo. Facility QC plan E-2

09/01/2021 JO 7210.634A CHG 1 Appendix F Appendix F. Technical Operations Services SSR, CAP, and SYSIR Templates This appendix contains templates for Technical Operations Services SSRs, CAPs, and SYSIRs based on the questions and required information in the safety database. It is intended to assist users in preparing information and data for entering results into the safety database. The current safety database is the Comprehensive Electronic Data Analysis and Reporting (CEDAR) tool located at https://cedar.faa.gov. F-1. Technical Operations Services System Service Review (SSR) Template. * Indicates required item Review Categories Response Explain Preliminary Data: Date, Start and End Time (UTC) (of Event)*. Short Description (of Event)* e.g. event summary that provides a short description of event being reviewed, sequence of events, and operational impacts. Additional Information: FSEP Facility Type, FSEP Facility ID.* Location (City, State)* RMLS Data: RMLS Log ID* NAS Systems and Services: Were any equipment issues identified?* If Yes: Was there any maintenance occurring (PM’s, Correctives, Mod’s, Troubleshooting, etc.)?* (Whether response to previous question is Yes or No, the following 3 questions require date selection) When was the last PM completed? [Date]* When was the last certification completed? [Date]* When was the last unscheduled outage? [Date]* Were any telecommunications issues identified?* Did any TPR issues or trends exist?* Was any service degradation identified prior to the event?* Were there any system modification issues?* Were there any maintenance alert issues (not completed, expired, etc.)?* Were any adaptation issues identified?* Were any airspace and procedure issues identified?* Did diversity, redundancy, or back-up system issues exist?* F-1

09/01/2021 JO 7210.634A CHG 1 Appendix F Review Categories Response Explain Were any potential systemic NAS system and/or service issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Coordination: Were any coordination issues identified?* If Yes: Was coordination (pre, final, and post) properly completed with all customers and stakeholders?* Could coordination or communication with stakeholders have been improved?* Was a NOTAM issued?* Were any potential systemic coordination issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Procedures/Directives/Documentation: Were any procedural, directive, or documentation issues identified (local, service area, or national)?* If Yes: Were personnel unfamiliar with the application of the procedure or directive?* Were any issues identified with any other reference documentation or technical drawings?* Were any potential systemic procedural/directive/documentation issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Impact Mitigation: Were any Safety Risk Management and Operational Risk Management (e.g. IRMC) issues identified?* Were there recently similar outages or trends?* Were any contingency plan issues identified?* Did anything delay service or system recovery?* Was fatigue an issue?* Were any potential systemic impact mitigation issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Resource Management: Were any resource management issues identified?* If Yes: Did any supervision or technical oversight issues exist (FLM, DFM, or SSC Coordinator)?* Were any issues with operational coverage identified?* Were any potential systemic resource management issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) F-2

09/01/2021 JO 7210.634A CHG 1 Appendix F Review Categories Response Explain Training: Were any training issues identified?* If Yes: Was training in progress at the time of the event?* Were any potential systemic training issues identified?* (Note: probability for recurrence at the same location or elsewhere in the NAS) Stakeholder and Customer Feedback: Was there any stakeholder or customer feedback?* If Yes: Did the feedback express praise?* Did the feedback express concern?* Was the feedback validated?* Individual Performance: Was any notable individual performance identified?* (Exemplary and/or needing improvement) If Yes: Where was individual performance identified?* Facility Type:* [Drop Down: Navigational, Communications, Surveillance, Weather, Environmental, Control Center] Maintenance Organization:* [Drop Down: SSC, Control Center, Technical Support, Engineering Services, Contractor] ATSAP/TSAP filed? Supporting Data: Add Attachment: [Drop Down: Falcon Bookmark, Airborne Replay, Surface Replay, Voice Recording, Other] Findings and Corrective Actions: Is corrective action required?* If Yes: Corrective Action:* [Link: Create CAP] QC Comments:* Forward: Recipient: [Select Name] (Note: Can add multiple recipients) [] Send as Informational []Delegate []Cancel Comments: F-3

09/01/2021 JO 7210.634A CHG 1 F-2. Technical Operations Services Corrective Action Plan (CAP) Template. * Indicates required item Basic Information: Subject of CAP:* Safety Monitoring Watchlist: Identified Safety Risk or Hazard: Risk or Hazard:* Background Information: Detailed Description of Validated Safety Issue(s):* Pertinent Regulations: Add Regulation: Order/Directive Number: [Drop Down: 6000.15, 7110.65, 7210.3, Other] Reference: Description: Corrective Action Plan:* Corrective Action #1: (Note: More than one corrective action can be added in the safety database) Type:* [Drop Down: Briefing, Review, Airspace, Directive, Procedure, Equipment, Training, Other] Level:* [Drop Down: Headquarters, Service Area, District, Facility] Target Completion Date:* Details:* F-4

09/01/2021 JO 7210.634A CHG 1 Appendix F Monitoring Plan:* Monitoring Item #1: Type:* [Drop Down: Review] Level:* [Drop Down: Headquarters, Service Area, District, Facility] Target Completion Date:* Details:* Effectiveness Target:* Supporting Data: Attachment Type: [Drop Down: Falcon Bookmark, Airborne Replay, Surface Replay, Voice Recording, Other] Forward: Recipient: [Select Name] (Note: Can add multiple recipients) [] Send as Informational []Delegate []Cancel Comments: F-5

09/01/2021 JO 7210.634A CHG 1 Appendix F F-3. Technical Operations Services Systemic Issue Review (SYSIR) Template. * Indicates required item Date: Systemic Issues: Select the type of systemic issue identified:* [Drop Down: NAS Systems and Services, Coordination, Procedural/Directive/Documentation, Impact Mitigation, Resource Management, Training] Systemic Issue:* Add Attachment (.xls, .xlsx, .doc, .docx, .pdf, .png, .jpg, .gif): Enter Specific Details:* F-6