gao-18-552

AVIATION SECURITY Basic Training Program for Transportation Security Officers Would Benefit from Performance Goals and Measures Report to Congressional Committees July 2018 GAO-18-552 United States Government Accountability Office

United States Government Accountability Office Highlights of GAO-18-552 , a report to congressional committees July 2018 AVIATION SECURITY Basic Training Program for Transportation Security Officers Would Benefit from Performance Goals and Measures What GAO Found The Transportation Security Administration (TSA) established the Transportation Security Officer (TSO) Basic Training program at the TSA Academy at the Federal Law Enforcement Training Centers (FLETC) in Glynco, Georgia to obtain benefits from centralized training. Prior to the Basic Training program, TSO training was conducted at individual airports, often by TSOs for whom instruction was a collateral duty. According to a business case developed by TSA for Congress in 2017 and TSA officials, TSA expected implementation of the TSO Basic Training program to provide efficiencies to the delivery of new- hire training for TSOs and to enhance the professionalism and morale of newly hired screeners. For example, GAO observed that TSO Basic Training facilities have airport checkpoint equipment and x-ray image simulators for students to practice skills, eliminating the challenge of finding available equipment and training times in a busy airport environment. According to program officials, centralized training also provides trainees with an increased focus on the TSA mission and instills a common culture among TSOs. TSA’s Office of Training and Development (OTD) updates and modifies the TSO Basic Training curriculum in response to evolving security threats and evaluations of effectiveness, among other factors. For example, OTD holds regular meetings with TSA’s Office of Security Operations—the office responsible for managing TSO performance—to discuss issues such as imminent threats. The offices also discuss analyses of TSO effectiveness identified through covert tests, in which role players attempt to pass threat objects—such as knives, guns, or simulated improvised explosive devices— through the screening checkpoints. The two offices identify ways to address issues identified in covert testing, which are then incorporated into TSO Basic Training. OTD also gathers input from TSO Basic Training instructors and from participants to adjust training curriculum. TSA has implemented a training evaluation model but has not yet established specific program goals and performance measures to assess TSO Basic Training. The Kirkpatrick model used by TSA is a commonly-accepted training evaluation model endorsed by the Office of Personnel Management and used throughout the federal government. While TSA reported expected benefits of TSO Basic Training in its business case and implemented the Kirkpatrick model to begin assessing training, it has not yet identified specific goals that the program is expected to achieve, nor has it developed applicable performance measures to evaluate progress toward goals, as called for by leading management practices for training evaluation. TSA officials told GAO that TSO Basic Training is a relatively new program and they planned to collect more data on TSO screening performance before further evaluating the potential impacts of the training program. However, TSO Basic Training serves as the foundation for TSOs to learn core skills and procedures, and it is important to establish goals and measures to better assess the effectiveness of the training they receive. This will help TSA determine the extent to which TSOs are able to fulfill their important role in ensuring passenger safety while also showing results for the funds spent on such training each year. View GAO-18-552 . For more information, contact Nathan Anderson at (206) 287-4804 or andersonn@gao.gov . Why GAO Did This Study TSA is responsible for ensuring that all airline passengers and their property are screened for items that could pose a threat to airplanes and passengers at 440 airports across the United States. Since 2016, TSO Basic Training— initial training for newly hired TSOs, including both TSA-employed and private screeners—has consisted of an intensive two-week course at the TSA Academy located at FLETC. TSA has obligated about $53 million for the program from its inception through March 2018. In 2015 and 2017, the Department of Homeland Security Inspector General raised questions about the effectiveness of checkpoint screening, which prompted concerns about training. GAO was asked to review TSA’s training of new TSOs. This report (1) describes the reasons why TSA established the TSO Basic Training program; (2) discusses factors OTD considers when updating TSO Basic Training curriculum; and (3) assesses the extent to which TSA evaluates its TSO Basic Training program. GAO reviewed documents on the development and modification of TSO Basic Training curriculum; visited FLETC; interviewed TSA officials; and compared TSA’s program evaluation to leading practices. What GAO Recommends GAO recommends that TSA establish specific goals and performance measures for the TSO Basic Training program. TSA concurred with the recommendation.

Page ii GAO-18-552 Aviation Security Abbreviations AIT Advanced Imaging Technology DHS Department of Homeland Security FLETC Federal Law Enforcement Training Centers NHTP New Hire Training Program OOI TSA Office of Inspection OSO TSA Office of Security Operations OTD TSA Office of Training and Development SOP Standard Operating Procedures SPP Screening Partnership Program TSA Transportation Security Administration TSO Transportation Security Officer This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Page 1 GAO-18-552 Aviation Security 441 G St. N.W. Washington, DC 20548 July 26, 2018 The Honorable Michael T. McCaul Chairman The Honorable Bennie G. Thompson Ranking Member Committee on Homeland Security House of Representatives The Honorable John Katko Chairman The Honorable Bonnie Watson-Coleman Ranking Member Subcommittee on Transportation and Protective Security Committee on Homeland Security House of Representatives The screening of airport passengers and their property is a critical component in securing our nation’s commercial aviation system. Since the terrorist attacks of September 11, 2001, the Transportation Security Administration (TSA) has been tasked with screening airline passengers and their carry-on and checked baggage for prohibited and other potentially dangerous items that could pose a threat to aircraft and passengers. According to TSA, in fiscal year 2017, more than 771 million passengers were screened at 440 airports across the United States. Since its inception, TSA has frequently adapted its standard operating procedures (SOP) to address new and evolving threats to aviation security. For instance, after the discovery of a plot to take the ingredients for an improvised explosive device—including hydrogen peroxide disguised as soda—onto multiple aircraft, TSA limited the amount of liquids passengers could bring onto planes from outside the airport. Transportation Security Officers (TSOs) at airports follow SOPs that guide screening processes and utilize technology such as advanced imaging technology (AIT) scanners or walk-through metal detectors to screen Letter

Page 3 GAO-18-552 Aviation Security management directive, and the business case for TSO Basic Training submitted by TSA to Congress. 4 We also visited the Academy to better understand the benefits of establishing TSO Basic at FLETC. In addition, we interviewed OTD officials at headquarters and at the Academy, instructors at the Academy, and union officials representing TSOs employed by TSA to better understand the benefits and challenges of training all new TSOs at the Academy and how TSO Basic Training differs from NHTP in place prior to TSO Basic Training. To discuss what factors OTD considers when updating the TSO Basic Training curriculum, we reviewed documentation from OTD regarding changes made to TSO Basic Training since its inception, including a list of changes to the curriculum and a contractor’s report that recommended some of those changes. 5 We also interviewed officials from the Office of Security Operations (OSO), the office that manages TSOs at the nation’s airports, the Office of Inspection (OOI), which conducts covert testing to evaluate checkpoint effectiveness, and OTD to determine why modifications were made and how the offices communicate regarding updates to standard operating procedures and other processes that may impact TSO Basic Training. To assess the extent to which TSA evaluates TSO Basic Training, we reviewed TSA documents used for evaluating training courses, including end-of-course surveys administered to participants and surveys distributed to participants and their supervisors approximately 3 months after course completion. We also interviewed TSA headquarters officials responsible for evaluating TSO Basic Training and for developing and implementing TSO Basic Training as well as TSO Basic Training instructors. We compared the training evaluation documentation to the Kirkpatrick Model for training evaluation, which TSA uses as the model for 4 TSA submitted this business case pursuant to the Explanatory Statement accompanying the Consolidated Appropriations Act, 2017. See Pub. L. No. 115-31, 131 Stat. 135 (2017); 163 Cong. Rec. H3813 (daily ed. May 3, 2017). Specifically, the Explanatory Statement directed TSA to develop a business case to justify the cost of new screener training (before permanent investments in the campus are made related to this training), including metrics related to increased TSO performance, improved morale, and better managed attrition. 5 TSA hired a government contractor to perform an analysis on the soundness of the TSO Basic curriculum. In August 2016, the contractor delivered a draft report, which included recommendations on how to improve the curriculum. TSA officials told us a final report was not provided.

Page 4 GAO-18-552 Aviation Security its evaluations of TSO training, and to leading practices in evaluating training and development efforts. 6 We conducted this performance audit from November 2017 through July 2018, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. After the terrorist attacks of September 11, 2001, Congress passed and the President signed the Aviation and Transportation Security Act into law on November 19, 2001, with the primary goal of strengthening the security of the nation’s civil aviation system. 7 The act established TSA as the agency with responsibility for securing all modes of transportation, including civil aviation. 8 As part of this responsibility, TSA performs or oversees security operations at the nation’s nearly 440 commercial airports, including managing passenger and checked baggage screening operations. 9 TSOs inspect individuals and property to deter and prevent passengers from bringing prohibited items on board an aircraft or into the airport sterile area—in general, an area of an airport to which access is 6 The Office of Personnel Management has endorsed the Kirkpatrick Model as an effective tool to help agencies evaluate their training programs. The Kirkpatrick Model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development and the impact of the training program on the agency’s mission. For leading practices, see GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government , GAO-04-546G (Washington, D.C.: March 2004). 7 Pub. L. No. 107-71, 115 Stat. 597 (2001). 8 See 49 U.S.C. § 114(d). 9 For the purposes of this report, a commercial (or TSA-regulated) airport is an airport in the United States that operates under a TSA-approved security program in accordance with 49 C.F.R. part 1542 that, in general, regularly serves air carriers with scheduled passenger operations to and from that airport. In general, TSA must provide for the screening of all passengers and property that will be carried aboard a passenger aircraft operated by an air carrier or foreign aircraft operators to, from, and within the United States. See 49 U.S.C. § 44901(a). Background

Page 6 GAO-18-552 Aviation Security Figure 1: Transportation Security Officer (TSO) Tasks at Airport Checkpoints Within TSA, two offices work together to manage TSOs and ensure their training is current and relevant. OSO is responsible for allocating TSO staff to airports, scheduling TSO work hours and training availability, and developing SOPs that govern how TSOs screen passengers and baggage. OTD is responsible for developing initial and ongoing training curricula for TSOs based in part on SOPs. Within OTD, a dedicated team is located at the Academy to manage updates at TSO Basic Training. In accordance with the Aviation and Transportation Security Act, screeners must complete a minimum of 40 hours of classroom instruction, 60 hours of on-the-job training, and successfully complete an on-the-job training examination. 11 Until 2016, new TSOs completed these training requirements at or near their home airports through the New Hire Training 11 See 49 U.S.C. § 44935(g)(2). Additionally, TSOs may not use any security screening device or equipment in the scope of that individuals’ employment unless the individual has been trained on that device or equipment and has successfully completed a test on the use of the device or equipment. 49 U.S.C. § 44935(g)(3).

Page 7 GAO-18-552 Aviation Security Program (NHTP). Since TSA centralized the TSO Basic Training program in January 2016, TSOs fulfill these training requirements through classroom training at the Academy as well as training at their home airports prior to the Academy and on-the-job training after completion of TSO Basic Training. During the 2 weeks spent at the Academy, TSOs receive 80 hours of training on standard operating procedures, threat detection, and the use of screening equipment. Prior to attending TSO Basic Training, new TSOs complete computer-based prerequisite training and may shadow experienced TSOs at a checkpoint. TSO Basic Training allows for participants to be trained at a dedicated facility with more hands-on training than was possible for NHTP (see Appendix I for a comparison of the two programs). As shown in table 1, of the $53 million obligated from January 2016 through March 2018, TSA obligated $18.2 million for procurement and development of the modular buildings on the FLETC campus used for TSA training, as well as associated hardware and set-up obligations such as audio/video equipment and fully operational simulated checkpoints. TSA obligated an additional $12 million in fiscal year 2016 and $13.7 million in fiscal year 2017 for the delivery of TSO Basic Training, including associated student travel and related equipment. 12 TSA officials told us that due to continuing budget resolutions that funded the government between October 2017 and March 2018, TSA was not able to fully fund the interagency contract between TSA and FLETC to support the TSO Basic Training course in fiscal year 2018 at the beginning of the year. For this reason, TSA does not yet have 2018 training obligations available for reporting through its accounting system. However, based on the average cost per student in fiscal year 2017 of about $2,300 to attend TSO Basic Training, TSA estimates total training obligations of approximately $9.1 million in the first half of fiscal year 2018. 13 12 For context, according to TSA officials, the agency expended approximately $4 million in total travel and logistics support to train new hires at their home airports through NHTP in each of fiscal years 2014 and 2015. This dollar figure does not account for some related costs of the New Hire Training Program, such as training space, printing, or other shared costs borne by OSO as part of delivering training within the field airports by local personnel. 13 According to TSA data, approximately $1,450 of this total is comprised of meals, lodging, and related costs for training on the FLETC campus. $834 is the average estimate per student for airfare and related travel expenses.

Page 9 GAO-18-552 Aviation Security NHTP. 15 Among the potential efficiencies and improvements cited by TSA are: • Increased consistency and standardization. According to TSA documents and OTD headquarters officials, centralized training provides a standardized curriculum that serves as a foundation for the skills, knowledge, and equipment used across an array of different airport environments. The TSA business case and other supporting analyses note that such an approach offers greater consistency of training delivery and a better mechanism for developing, delivering, and evaluating course content. • Equipment availability and expanded course content. TSO Basic Training includes a full suite of dedicated checkpoint equipment and x-ray image simulators for students to practice learned skills, eliminating the challenge of finding available equipment and training times in a busy airport environment (see figure 2). Officials told us that being more familiar with the screening equipment increases TSOs’ readiness for on-the-job training when they return to their home airports. Initial test results also indicate that participants trained at the Academy receive higher pass rates on end-of course assessments of x-ray image interpretation skills than those who received their initial training at their home airports. Specifically, according to TSA data, of the 5,877 test-takers who received training at TSO Basic Training in 2016, 91.5 percent passed the Image Interpretation Test on their first attempt. In contrast, 83.2 percent of the 1,458 test-takers who received training at local airports in 2016 passed the test on their first attempt. 16 In addition, the Academy curriculum incorporates new learning opportunities, including a live demonstration of improvised explosive devices and an active shooter drill, both of which would be difficult to reproduce within the airport environment, according to TSA officials. 15 See, for example, TSA, Office of Security Operations, Operational and Technical Training Division: Training Sites Alternatives Analysis . Prepared by U.S. Department of Transportation, Research and Innovative Technologies Administration, Volpe National Transportation Systems Center (Oct. 31, 2008); and TSA, Training Delivery Implementation of the New Hire Training Program Recommendations from the July 2011 Training Sites Business Case Analysis (Oct. 20, 2015). 16 According to officials, in fall 2016, a hurricane disrupted operations at FLETC and airports were given the option to train new TSOs on-site or to wait for FLETC to re-open. While FLETC re-initiated training operations within two weeks following the hurricane, the delay affected TSO Basic Training for over two months, during which time many TSOs were trained at their home airports.

Page 10 GAO-18-552 Aviation Security Figure 2: Simulated Operational Checkpoints Used for Training at the TSA Academy • Dedicated faculty and instructor development. TSO Basic Training offers a dedicated faculty and support staff focused exclusively on training TSOs. According to TSA officials, before TSO Basic Training, training at individual airports was often conducted by TSOs for whom instruction was a collateral duty, whereas instructors at the Academy have full-time training responsibilities and enhanced opportunities to learn from each other, increase their professional training skills, and provide feedback on the delivery of course curriculum. • Centralized facility and shared logistics. By locating the TSA Academy at FLETC, TSA is able to take advantage of the services and logistical support that FLETC provides. Specifically, FLETC services and logistics include accommodations, meals, and transportation, thereby reducing the administrative demands on TSA personnel and allowing students a focused and efficient training experience. Additional efficiencies cited by TSA officials include lower overall costs for office space, janitorial services, and other operational costs because such costs are shared by the 96 agencies that use FLETC. According to TSA officials, conducting training at FLETC can also help TSA accommodate hiring surges and better augment future training, if needed. For example, TSA officials reported that the facility has surge capacity from its current capacity of 240 students up to 300 new students if sufficient instructors are available. According to TSA documents and training officials, another key benefit of centralized training is the opportunity to enhance professionalism and help foster camaraderie and esprit de corps. TSA anticipates that centralized, standardized training will not only provide trainees with an

Page 12 GAO-18-552 Aviation Security OTD updates and modifies the TSO Basic Training curriculum based, in part, on regular communications from OSO, the office responsible for developing SOPs for screening operations and managing TSO performance. Officials from both offices told us that OSO provides information to OTD on changes to SOPs as soon as changes are made so they can update the TSO Basic Training curriculum. 19 For instance, in 2017, when OSO began planning major changes to the SOPs, the office gave OTD information about the planned SOP revisions, as well as the airports where the new SOPs would be piloted. 20 In response, OTD modified its curriculum and was able to provide revised training for new TSOs based at airports that were piloting the program, while providing TSOs at all other airports the prior version of training. OTD officials noted that in some cases TSA must quickly update SOPs to reflect imminent threats. According to officials, a plan is in place to make changes to TSO Basic Training curriculum in response to emerging or imminent threats, although such threats have not been experienced since the establishment of TSO Basic Training in 2016. In addition to changes in SOPs, OSO officials indicated they may also change the timing of when TSOs employed by TSA attend TSO Basic Training. Specifically, officials told us that OSO plans to implement a new 19 According to OTD officials, the relationship between OSO and OTD is one of client and service provider. In general, OSO makes decisions regarding SOPs and TSO training availability and OTD incorporates those decisions into its curriculum and planning. 20 Under the new SOP, travelers are asked to remove electronics larger than a cell phone from their carry-on bags to help TSA officers obtain a clearer X-ray image of electronic devices. The new screening procedures were initially tested at 10 airports and officials told us they are currently rolling out to airports around the country. Factors Considered in Updating the TSO Basic Training Curriculum Include Evolving Security Threats and Input from Course Participants OTD Uses Information from OSO to Update the TSO Basic Training Curriculum to Address Evolving Security Threats

Page 13 GAO-18-552 Aviation Security model for TSO Basic Training in which TSOs will attend TSO Basic Training 2 to 6 months after they are hired rather than as soon as is practical. According to TSA, the agency is pursuing this change to, among other things, implement a transparent career path for TSOs employed by TSA and to encourage and reward skill development. During the 2 to 6 months prior to attending TSO Basic Training, TSOs will perform checkpoint tasks that require training that can be delivered at the airport as soon as they are hired, such as checking passengers’ travel documents and helping passengers move through the checkpoint. Once TSOs are able to perform these initial tasks, they will attend TSO Basic Training at the Academy, Officials told us they are preparing for the change by modifying the TSO Basic Training curriculum to eliminate subjects that will be covered at the airports and to emphasize skills that more experienced TSOs will need, such as performing physical searches of passengers. TSA plans to implement the revised model beginning in August 2018. Finally, OTD receives information on TSO performance and uses that information to inform TSO Basic Training curriculum. For example, two TSA offices—OSO and the Office of Inspections—perform regular effectiveness testing of airport checkpoints through covert testing and share the results with OTD. 21 After each covert testing event, each office conducts interviews with TSOs to determine the factors that contributed to their effectiveness at identifying prohibited items. Officials told us that OSO and OTD hold regular meetings to discuss the analyses of covert testing failures and ways in which training curriculum can be modified to address the reasons for the failures, which are then incorporated into the TSO Basic Training curriculum. Office of Inspections officials noted that they participated in the development of the TSO Basic Training curriculum and provide regular reports to OTD on covert testing results. 22 21 TSA headquarters oversees covert testing, which is implemented locally at all US airports, using unrecognizable role players who attempt to pass threat objects, such as knives, guns, or simulated improvised explosive devices, through the screening checkpoints. The tests are designed to assess various aspects of the aviation security system, including the operational effectiveness of screeners. 22 GAO is in the process of conducting research for a more detailed report on TSA’s use of covert testing.

Page 15 GAO-18-552 Aviation Security TSA has implemented three of the four levels of the Kirkpatrick Model, a training evaluation model that, in part, helps TSA collect feedback from course participants and evaluate the impact on individual development. However, the agency has not developed goals for the program or related performance metrics to demonstrate progress toward goals. To evaluate the TSO Basic Training program, TSA uses the Kirkpatrick Model, which is a commonly accepted training evaluation model endorsed by the Office of Personnel Management and used throughout the federal government. The Kirkpatrick Model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. To date, TSA has implemented the first three levels of the model by administering (1) course surveys to participants at the end of the training program; (2) an end-of-course written exam and an x-ray image interpretation test to assess achievement of learning objectives; and (3) course surveys to participants and their supervisors several months after completing training to collect information regarding how the training affected behavior or performance on the job. OTD officials told us that they have not yet implemented Level 4 of the model because they do not believe they have enough data. Table 2 provides a description of what each level within the Kirkpatrick model is to accomplish and TSA’s progress in implementing the levels. TSA Has Made Progress in Implementing a Training Evaluation Model but Has Not Established Specific Goals and Performance Measures to Assess TSO Basic Training TSA Has Made Progress in Implementing the Kirkpatrick Model to Evaluate Its TSO Basic Training Program

Page 16 GAO-18-552 Aviation Security Table 2: Transportation Security Administration’s (TSA) Implementation of Kirkpatrick Training Evaluation Model Kirkpatrick Evaluation Level Description TSA’s Current Practices Level 1 The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training. TSA uses a survey at the completion of each course to obtain feedback from participants regarding the overall length and structure of the course, instructor effectiveness, and the degree to which they thought they could implement what they learned in their job. Level 2 The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course. After completion of the 2-week Transportation Security Officer (TSO) Basic Training program, TSA administers an end of course written exam and an x-ray image interpretation test to evaluate the extent that participants can demonstrate sufficient knowledge of course content and suitability to proceed to on-the-job training at their home airports. Level 3 The third level measures how training affects changes in behavior or performance on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on job performance. TSA administers surveys to course participants and their supervisors several months after the participant attends TSO Basic Training to collect information on the perceived value and effectiveness of the course and the extent to which participants were able to apply specific course learning objectives during their on-the-job training duties. For instance, the survey asks supervisors to agree or disagree with statements about their employees’ preparedness to operate screening checkpoint equipment or conduct physical searches of passengers. Officials have performed one analysis of these surveys, for which the response rate for the participant surveys was 8.5 percent of the 5,948 course graduates in FY16. The supervisor response rate was 31.2 percent of the 3,353 surveys administered. OTD officials consider this a usable response rate to provide feedback on any changes that may be needed to the course curriculum. Level 4 The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before and after a training modification was made. TSA has not yet implemented a Level 4 evaluation. Officials stated that they plan to do so but noted that several more years of data on TSO screening performance are likely needed to assess the broader organizational impacts of the training program. Source: GAO analysis of Transportation Security Administration information. | GAO-18-552 While TSA reported potential benefits of TSO Basic Training in its business case and implemented the Kirkpatrick Model to assess training, it has not yet identified specific goals that the TSO Basic Training program is expected to achieve, nor has it developed performance measures to evaluate progress toward goals. The business case and the Kirkpatrick Model are positive steps and document certain benefits of TSO Basic Training, but without a set of specific training goals and associated performance measures for the program, TSA is not able to TSA Has Not Yet Developed Goals and Performance Measures for TSO Basic Training