DOT PROGRAM EVALUATIONS

Performance measures show if intended outcomes are occurring and assess any trends. Program evaluation uses analytic techniques to assess the extent to which our programs are contributing to those outcomes and trends. As required by Government Performance and Results Act of 1993, the Department's FY 2003–2008 Strategic Plan included an initial list of new program evaluations planned for those fiscal years. This section provides a summary of DOT's program evaluation efforts scheduled for completion in FY 2005. In addition, updates of FY 2004 evaluations that were not completed when last year’s PAR went to press are also included to maintain continuity across fiscal years.

TYPES OF PROGRAM EVALUATIONS

Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes. Evaluations are of the following types:

Impact Evaluations use empirical data to compare measurable program outcomes with what would have happened in the absence of the program. These represent the highest standard of program evaluations and are often the most difficult and expensive to construct and interpret.

Outcome Evaluations assess the extent to which programs achieve their outcome-oriented objectives. Outcome evaluations will use quantitative methods to assess program effectiveness, but fall short of the rigorous causal analysis of impact evaluations.

Process Evaluations assess the extent to which a program is operating as intended. While a true process evaluation will use objective measurement and analysis, it falls short of assessing the causal links between intervention and outcome.

Cost-Benefit and Cost-Effectiveness Analyses compare a program's outputs or outcomes with the costs to produce them. This type of analysis conforms with program evaluation when applied systematically to existing programs and when measurable outputs and outcomes are monetized.

PROGRAM EVALUATION MANAGEMENT

DOT staff, contractors, academic institutions, the Office of Inspector General (OIG), or the Government Accountability Office (GAO) may conduct program evaluations. Program evaluation efforts are designed to ensure that the finished evaluations are useful regardless of who conducts the evaluation or the methodology used.

The programs selected for evaluations are vetted through the Department’s strategic planning process. Each modal administration nominates programs that are then reviewed by a strategic planning executive committee to ensure two things: 1) adequate breadth of program evaluations across modal administrations; and 2) alignment to the strategic objectives developed through the planning process. The OIG and the GAO conduct their own program evaluations independent of this schedule, as appropriate.

FY 2005 PROGRAM EVALUATION SUMMARIES

A summary of DOT program evaluations scheduled for completion in FY 2005 follows.

EVALUATION OF FEDERAL MOTOR CARRIER SAFETY ADMINISTRATION’S (FMCSA’s) COMPLIANCE REVIEW (CR) IMPACT ASSESSMENT MODEL

FMCSA's CR program is a nationwide program in which FMCSA and State inspectors conduct on-site reviews of motor carrier compliance with Federal Motor Carrier Safety Regulations (FMSCRs). FMCSA expects that through enforcement of the FMCSR, and promotion of safety requirements, motor carriers will improve the safety of their operations and reduce their chances of being involved in crashes.

Related Strategic Objective: Safety

The purpose of this impact evaluation is to measure the effectiveness of FMCSA's CR program in terms of crashes avoided, injuries avoided, and lives saved. The objective of conducting this evaluation is to provide FMCSA management and State safety partners with a quantitative basis for optimizing the allocation of resources dedicated to the improvement of commercial motor vehicle safety. The Volpe National Transportation Systems Center conducted this evaluation.

The scope of this evaluation is the safety impact of all CRs performed by FMCSA and its State partners. In 2002, Federal and State enforcement personnel conducted 13,430 CRs. The model used to evaluate the impact of these CRs is designed to measure the direct impact of CRs on carrier safety. It is not designed to measure indirect aspects such as deterrence (i.e., the threat of having a CR).

The methodology used to conduct this impact evaluation is an analytic program evaluation model called the CR Effectiveness Model, which FMCSA developed in cooperation with the Volpe National Transportation Systems Center. The model is based on the individual and cumulative “before and after” changes in the safety performance of carriers that received CRs. The model compares a motor carrier's crash rate in a time period after a CR to its crash rate prior to that review. To make this comparison, the model uses crash and power unit data from the Motor Carrier Management Information System (MCMIS) snapshots taken before and after the CR.

The results of this evaluation are as follows:

Compliance Review Program Effectiveness
1999-2004

# FY 2004 data are preliminary projections

The latest available data from 2003 and 2004 projected results are provided in the table above. The data trend over the last four years continues to remain positive as more crashes and injuries are avoided and the number of lives saved increases. FMCSA's plan is to continue to conduct this evaluation of the CR Program on an annual basis in order to monitor the effectiveness of the agency's CR program. Completion of this evaluation is set as an annual agency milestone.

EVALUATION OF FMCSA ROADSIDE INSPECTION/TRAFFIC ENFORCEMENT

Roadside inspection and traffic enforcement (RI/TE) are two of the Federal Motor Carrier Safety Administration's (FMCSA) key safety programs. The roadside inspection program consists of roadside inspections of vehicle and driver safety performed by qualified safety inspectors. The traffic enforcement program is based on the enforcement of 21 moving violations noted in conjunction with a roadside inspection. State RI/TE activities are funded through FMCSA's Motor Carrier Safety Assistance Program.

Related Strategic Objective: Safety

The purpose of the evaluation is to measure the impact of the RI/TE program in terms of crashes avoided, injuries avoided, and lives saved. The objective of conducting this evaluation is to provide FMCSA management and State safety partners with a quantitative basis for optimizing the allocation of resources dedicated to the improvement of commercial motor vehicle safety. FMCSA expects that vehicle and/or driver defects discovered, and then corrected as the result of RI/TE interventions, will reduce the probability that these vehicles/drivers will be involved in subsequent crashes, which will reduce overall crash rates. The Volpe National Transportation Systems Center conducted this evaluation.

The scope of this evaluation includes all RI/TEs funded by the FMCSA. In 2002, approximately 3.0 million RI/TEs were conducted. The model which is used to conduct this evaluation is designed to measure both the direct and indirect impact of RI/TEs on improving safety, (i.e., crashes avoided, injuries avoided, and lives saved).

The methodology used to conduct this impact evaluation is an analytic program evaluation model called the Intervention Model, which FMCSA developed in cooperation with the Volpe National Transportation Systems Center. The Intervention Model is based on the premise that the two programs, roadside inspection and traffic enforcement, directly and indirectly contribute to the reduction of crashes. The model includes two sub-models that are used for measuring these both direct and indirect effects:

§ Direct effects are based on the assumption that vehicle and/or driver defects discovered and then corrected as the result of interventions reduce the probability that these vehicles/drivers will be involved in subsequent crashes. The model calculates direct-effect, prevented crashes according to the number and type of violations detected and corrected during an intervention.

§ Indirect effects are the byproducts of the carriers' increased awareness of FMCSA’s programs and the potential consequences that the programs could impose if steps were not taken to ensure and/or maintain higher levels of safety. In order to measure indirect effects, which are essentially changes in behavior involving driver preparation and practices and vehicle maintenance, the model calculates responses to the programs and the resulting reduction in potentially crash-causing violations.

The results of this annual evaluation are as follows:

Program Effectiveness
2001-2004

The latest available data from 2004 are provided in the table above. Although crashes and injuries avoided continue their positive trend, lives saved have stayed relatively flat over the four year period. FMCSA's plan is to continue to conduct this evaluation on an annual basis in order to monitor the effectiveness of the agency's RI/TE program. Completion of this evaluation is set as an annual agency milestone.

HAZARDOUS MATERIALS AIR TRANSPORTATION EVALUATION

Both air carriers and passengers play a critical role in the transportation of hazardous materials by air. Air carriers (including shippers and repair stations that ship hazardous materials) are required to transport the material in accordance with the existing Federal Regulations. Passengers are prevented from carrying unauthorized hazardous materials on to scheduled commercial flights if discovered during the security screening. To ensure compliance, Special Agents periodically inspect and conduct investigations of violations by air carriers, shippers, and repair stations that ship by air. These hazardous materials regulations are promulgated by DOT’s Pipelines and Hazardous Materials Safety Administration (PHMSA). The regulations are inter-modal, applying to ground, sea, rail, and air transportation.

Related Strategic Objective: Safety

DOT’s OIG conducted an independent evaluation to assess the management of FAA’s Hazardous Materials Air Transportation program and its effectiveness in ensuring compliance with existing hazardous materials regulations. The OIG found that since the transfer of this program from the Transportation Security Administration (TSA) to the FAA, the FAA has made considerable progress in reestablishing it as a bona fide program for overseeing and enforcing industry’s compliance with hazardous material regulations. Since October 2002, FAA has realigned the Program’s organizational structure, hired and trained new members of its workforce, developed and implemented guidance and work plans for conducting inspections and investigations, and created an automated system for collecting and reporting the results of inspection and enforcement activities. Nevertheless, the current situation is far from an “end state” for ensuring the safety of hazardous material shipments by air, and new approaches are needed in managing the program.

To improve the management of this effort, the OIG recommendations included the following:

• Institute guidelines and timeframes for conducting hazardous materials investigations, conducting legal reviews, and issuing Notices of Proposed Civil Penalties through the coordinated efforts of the Hazardous Materials Division and the FAA’s Office of the Chief Counsel.
• Develop and implement alternate means of administering hazardous materials enforcement cases, such as the ticketing system used by Pipeline and Hazardous Materials Safety Administration (PHMSA).
• Finalize and implement the FAA voluntary disclosure reporting program. FAA needs to take a systematic approach in effectively managing the program, to include disseminating all useful information to the air carriers, hazardous material shippers, and DOT Operating Administrations with hazardous materials oversight and enforcement responsibilities.
• Implement a pilot project with the TSA and one or more air carriers to determine the effectiveness and cost of an automated operating system to record and process violations of hazardous materials regulations discovered during screening of the passenger's carry-on and checked baggage. In the interim, collaborate with TSA to implement procedures for notifying FAA of hazardous materials incidents associated with passengers' carry-on baggage.
• Issue an Advisory Circular notifying all air carriers that they must report to FAA all unauthorized hazardous materials found in passengers' checked baggage and take enforcement actions against those air carriers not complying with the reporting requirements.

In response to the OIG recommendations, the following actions have been taken:

• FAA has issued written field guidance on the timeliness of civil penalty cases that reduces the time allowed for civil penalty enforcement investigative reports to be submitted for legal review from 120 to 90 days. In addition, the FAA’s Chief Counsel's Office expects to revise Order 2150.3A by December 2005 that will also provide more consistent guidance to all FAA inspectors.
• The FAA expects to publish an initial notice in the Federal Register by March 30, 2006, that will implement a notice of violation process similar to both the Pipelines and Hazardous Materials Safety Administration "ticketing" process and the process previously used by FAA to administer certain aviation security violations.
• FAA, in cooperation with its Chief Counsel's Office completed a review of the draft Voluntary Disclosure Advisory Circular for certain hazardous materials violations. The draft Voluntary Disclosure Advisory Circular has been circulated for notice to, and comment from, air carrier associations. FAA expects to publish this Advisory Circular by December 2005.
• FAA has developed and is using a department-wide intermodal shipper database that contains the Department's hazardous materials inspections, penalties, incidents, exemptions and registrations information. The system will help set shipper inspection priorities based on prior penalties and incidents on record.
• The FAA has made arrangements with the Air Transport Association to provide its member air carriers with summary results of FAA hazardous materials inspections of their operations. The FAA has conducted over 3,000 outreach visits to hazardous materials shippers in the last twelve months and FAA field agents will provide a summary of a shipper's prior incident records to the shipper during inspections. Finally, the FAA has provided its inspection and penalty records for the intermodal database system so they are available for the OAs.

The FAA has drafted a revision to its Memorandum of Agreement (MOA) with the TSA. This revised MOA will be the basis for a mechanism to share TSA data on security checkpoint seizures of hazardous materials with the FAA. The FAA will evaluate and prioritize the information in terms of the risks posed by the abandoned hazardous materials. Lower risk items will be entered into the FAA's system to generate an automated outreach notification to the relevant passenger, higher risk items will forwarded to the relevant FAA Regional Office for investigation. Concerning unauthorized hazardous materials discovered during checked baggage screening, the TSA Standard Operating Procedure advises screeners to refer their discoveries to the air carrier that checked the bag. The air carrier would have to notify the FAA. The FAA initiated its system to generate automated outreach notices in January 2005. The system of collaboration between FAA, TSA and the air carriers themselves, has resulted in the mailing of over 4,000 notices to passengers.

EVALUATION OF AUTOMOBILE SIDE IMPACT PROTECTION

Side impacts rank second only to frontal impacts as a cause of occupant fatalities in cars, light trucks and vans. Federal Motor Vehicle Safety Standard 214 sets minimum performance requirements in side impacts. The requirements were phased into passenger cars during model years 1994 to 1997 and extended to light trucks and vans in model year 1999.

Related Strategic Objective: Safety

In 1997 NHTSA began testing vehicle performance at higher speeds and publishing the results through the New Car Assessment Program (NCAP). Initially, manufacturers upgraded side structures and affixed padding to meet Standard 214. In 1996 and later years, they installed two types of side air bags—torso bags and head air bags—that substantially improved side impact performance on the NCAP tests.

NHTSA originally planned to evaluate only the initial upgrades to structure and padding and complete the study by 2005, with a separate evaluation of side air bags at a later date. But, side air bags have become an integral part of the occupant protection system in much of the vehicle fleet (30% of new cars by 2002).

Consumers and manufacturers want to know as soon as possible about the effectiveness of side air bags. To expedite the impact evaluation of side air bags, NHTSA is combining it with the study of structures and padding. To ensure the statistics are more meaningful, on this relatively new technology, 2004 Fatality Analysis Reporting System and General Estimates System crash data are being included. These data are just now becoming available. NHTSA anticipates completing the report, sending it for peer review outside the agency, and publishing it in FY 2006.

LARGE TRUCK CRASH CAUSATION STUDY

There is no National database that provides information on the causes of, or factors contributing to, large truck crashes. FMCSA recognizes the importance of having this information and began investigating methods to collect it several years ago. The Government Accountability Office and the Department of Transportation’s Inspector General stated in separate reports in 1999 that the lack of large truck crash causation data hampers FMCSA program effectiveness. In addition, the Motor Carrier Safety Improvement Act of 1999 authorized funding for a study of the causes of commercial vehicle crashes. In response, in cooperation with the National Highway Traffic Safety Administration (NHTSA), FMCSA initiated the Large Truck Crash Causation Study (LTCCS); the first-ever National study of the causes of crashes involving large trucks. Nationally representative data on the primary and secondary causes of serious large truck crashes were collected by teams of trained investigators from NHTSA’s National Automotive Sampling System and FMCSA-funded State truck inspectors.

Related Strategic Objective: Safety

The goal of the LTCCS is to determine the reasons for, and factors contributing to, serious large truck crashes, so FMCSA can take the results of this process evaluation and implement the most effective countermeasures to reduce crash occurrence and severity.

The LTCCS collected data on crashes in 24 sites in 17 States from 2000 through 2003. Each crash involved at least one large truck with a gross vehicle weight rating of more than 10,000 pounds, and resulted in at least one fatality or at least one incapacitating or non-incapacitating but evident injury. To get the highest quality data possible, the on-site investigations began as soon as possible after crashes occurred. Data collection was performed at each crash site by a two-person team consisting of a trained researcher and a State truck inspector. Researchers collected data at crash scenes through driver, passenger, and witness interviews.

Most of the crashes involved collisions with another motor vehicle, usually a passenger vehicle. About two­thirds of the trucks involved in the crashes were truck tractors pulling a single semi-trailer. Preliminary findings to date indicate the immediate reason for large truck crashes in an overwhelming majority of the cases was an action by the driver of the truck or the other vehicle involved. Driver recognition and decision errors were coded most often for drivers of both trucks and passenger vehicles. Truck drivers, however, were in better condition to drive and made fewer driving performance errors than passenger vehicle drivers.

All of the crash data has been collected and encoded into a database. An initial release of data from the study to Congress and the public is scheduled for the first quarter of FY 2006.

BUS CRASH CAUSATION STUDY

There is no National database that provides information on the causes of, or factors contributing to, bus crashes. In 2004, in cooperation with the National Highway Traffic Safety Administration (NHTSA), FMCSA initiated the Bus Crash Causation Study (BCCS) as a complementary and follow-on effort to the Large Truck Crash Causation Study (LTCCS). The purpose of the BCCS is to analyze the causes of crashes involving buses.

Related Strategic Objective: Safety

The goal of the BCCS is to determine the reasons for, and factors contributing to, serious bus crashes, so FMCSA can take the results of this process evaluation and implement the most effective countermeasures to reduce crash occurrence and severity. The BCCS will use the same methodology as followed in the LTCCS. Data collection was initiated in 2004 and will continue through 2006, with a release of initial data and findings to Congress and the public scheduled for FY 2007.

EVALUATION OF FMCSA COMMERCIAL DRIVERS LICENSE (CDL) PROGRAM

The Commercial Drivers License (CDL) program is one of the Federal Motor Carrier Safety Administration’s (FMCSA’s) most important safety programs. Since the first CDL was granted in 1989, over 11 million have been issued and approximately 40,000 new CDLs are issued every month. Congress passed the Commercial Motor Vehicle Safety Act of 1986 (CMVSA) (Public Law 99–570, 100 Stat. 3207–170, 49 U.S.C. 31301), which established the CDL program. CMVSA made it illegal for drivers to hold more than one license and required States to adopt testing and licensing standards for truck and bus drivers to check a person's ability to operate the type of vehicle he/she plans to operate. The goal of the CDL program is to improve highway safety by ensuring that drivers of large trucks and buses are qualified to operate those vehicles and to remove unsafe and unqualified drivers from the highways.

Related Strategic Objective: Safety

The objective of this evaluation is to determine whether the CDL program is meeting its goal of removing unsafe and unqualified drivers from the highways. More specifically, this evaluation will focus on assessing the effectiveness of State systems and procedures to identify and take action against unsafe CDL holders.

This outcome evaluation is being conducted with support from the U.S. Department of Energy’s Oak Ridge National Laboratories (ORNL). This is a two-phase evaluation. Phase I of this evaluation was completed in July 2005. Phase I focused on analyzing results of the CDL Reviews conducted on State CDL programs by FMCSA. ORNL analyzed these CDL reviews and recommended actions to strengthen the program’s ability to foster the removal of unsafe and unqualified drivers from the highways. Under Phase II of this evaluation, ORNL will perform a more in-depth statistical analysis of the effectiveness of systems and procedures to identify and take action against unsafe CDL holders. Phase II is expected to be completed by December 2005.

RAIL GRADE CROSSING WARNING DEVICE INSTALLATION STUDY

This study will evaluate the effectiveness of the various types of warning devices used to control traffic at highway-rail grade crossings in preventing grade crossing collisions. The purpose of the evaluation is to provide traffic engineers, and others involved in the selection of warning devices at highway-rail grade crossings, with information on the type of warning devices that provide the greatest safety improvement for crossings.

Related Strategic Objective: Safety

FRA has not performed this study for several reasons. First, the purpose of the study, to provide information to aid in the selection of appropriate warning devices, was achieved in a separate DOT effort. In November 2002, DOT published the Guidance on Traffic Control Devices at Highway-Rail Grade Crossings document. This is a guidance document for users who understand general engineering and operational concepts of highway-rail grade crossings and are involved in the selection of traffic control devices. The Guide serves as a reference to aid in decisions to install traffic control devices or otherwise improve such crossings. A traffic control device selection procedure and extensive list of quantitative guidance are the specific products of this document.

Secondly, there were several other studies initiated that cover the subject. The Volpe National Transportation Systems Center is currently conducting an analysis to determine the various factors during the last ten years that have led to an almost 50% reduction in the number of fatalities resulting from crossing collisions. This study is looking into the effectiveness of the various types of warning devices at the crossings as part of the study. Volpe is also conducting an evaluation of safety measures implemented on the North Carolina Sealed Corridor. A third evaluation that studies the effectiveness of educational and law enforcement efforts in Arlington Heights, Illinois is also nearing completion.

In June 2004, the Department released the Secretary’s Action Plan for Highway-Rail Crossing Safety and Trespass Prevention. One of the specific actions to be undertaken in the plan is an evaluation of current safety efforts to determine the effectiveness of the principal grade crossing collision mitigation methods. FRA’s actions will include the following:

• Use the results from these three studies to evaluate the effectiveness of current safety efforts. This will include both engineering and non-engineering safety treatments.
• Issue a report on the current status of crossing safety efforts.

FRA anticipates that this report will be issued by the end of December 2006. The report will provide guidance to those who are involved in determining the appropriate traffic control device that should be installed at highway-rail grade crossings. It also will provide information on non-engineering efforts that can be made to improve crossing safety.

EVALUATION OF INTELLIGENT TRANSPORTATION SYSTEM (ITS) OPERATIONS INVESTMENTS BY LOCAL GOVERNMENTS

Intelligent transportation systems (ITS) improve transportation safety and mobility through the use of advanced communications technologies. ITS encompasses a broad range of wireless and wire line communications-based information and electronics technologies. When integrated into the transportation system's infrastructure, and in vehicles themselves, these technologies relieve congestion, improve safety and enhance efficiency.

Related Strategic Objectives: Safety & Mobility

In 2002, the Federal Highway Administration (FHWA) sponsored a program evaluation to determine to what extent Federal funding is being used at the local level to fund operations. The evaluation included a survey of 267 people from 141 agencies (including 125 officials representing 67 municipalities or county organizations). Among other findings, the survey revealed that ITS and operations investments are difficult to ascertain because they are usually included as part of larger capital infrastructure investments, or are not typically undertaken with Federal funds because local governments lack operations expertise and have few incentives to undertake investments in operations vis-à-vis larger capital investments. As a result of this finding, a new evaluation in FY 2005 was not undertaken.

The FY 2002 program evaluation, however, included four key recommendations and the FHWA has responded to each. For example, in FY 2004, FHWA commissioned an effort to provide agencies with more outcome-oriented information regarding the benefits of using ITS and operations solutions for tackling surface transportation problems. FHWA estimated the benefits that might be derived from the full deployment of ITS technologies and operational strategies in medium and large metropolitan areas. The study revealed that total congestion-related delay could be reduced by up to 27%, fuel consumption by up to 24%, and emission of harmful pollutants by up to 25% from current levels. Widespread deployment of ITS technologies and operational strategies not only relieves congestion, but also makes travel on the highway system more reliable and predictable.

ALTERNATE DISPUTE RESOLUTION PROCESS EVALUATION

Experience in both the public and private sectors shows that alternative dispute resolution (ADR) can improve communication and achieve mutually acceptable solutions more effectively than traditional, non­collaborative processes. Established in May of 2003, the U.S. Department of Transportation’s Center for Alternative Dispute Resolution works with DOT organizations, its management teams, and employees to increase knowledge, quality, and use of ADR. The Center serves as an information resource to both DOT ADR providers and users, and advises senior DOT officials on procedural approaches to resolving disputes having significant impact on National transportation policy.

Related Strategic Objective: Organizational Excellence

The Center, staffed by one full-time and one part-time employee, partners with the Office of the Dispute Resolution Specialist at the Department of Health and Human Services to offer conflict management­related informational and skill based training courses. From 2003–2005, the Center trained over 650 DOT employees, mediated approximately 16 cases, facilitated 4 office interventions, helped DOT organizations establish and support two workplace ADR programs and one civil penalties arbitration program, provided support for both court-ordered and party-agreed-upon mediation of complex cases, and incorporated ADR into DOT’s procurement disputes process.

The goal of this process evaluation was to determine service quality, growth in use of alternative dispute resolution process, and cost-effectiveness throughout DOT and to gain information for improving ADR services and programs.

The evaluation results show an increase in knowledge and use of ADR across DOT. In addition, the Center’s training program has received high marks from customers. The evaluation also made the following recommendations. The Center should:

• Follow a co-mediation model to increase available number of mediators.
• Conduct a survey to determine what percentages of employees are aware of available ADR services;
• Market its program using positive employee comments;
• Conduct post-mediation and facilitation surveys; and
• Survey mediation and training participants to see whether relationships have improved.

Beginning in FY 2006, the Center will implement the recommendations as follows:

• Use employees trained in Basic Mediation Skills as co-mediators;
• Survey all mediation and facilitation participants; and
• Design and implement an evaluation tool to determine any effect of Conflict Management Skills for Managers training on targeted participants. Final analysis would be completed in FY 2008.

EVALUATION OF THE TITLE VI COMPLAINTS PROCESS

The Departmental Office of Civil Rights (DOCR) oversees external complaints processed in accordance with Title VI of the Civil Rights Act of 1964. Title VI prohibits discrimination on the basis of race, color, and National origin in programs and activities receiving Federal financial assistance. DOT Order 1000.12, Implementation of the DOT Title VI Program, provides guidance and procedural instructions for all modal administrations on processing Title VI complaints. The Operating Administrations (OAs) use the procedures set forth in the Order to investigate external complaints, filed in accordance with Title VI, that are directly related to their mode of transportation. Both DOCR and the OAs use the Order to define the shared responsibility for collecting, accepting and dismissing external complaints. The Order also supports the OA's responsibility for collecting complaint related information. This information is used by DOT to provide its customers with the status of individual complaints and the status of the Title VI complaints program in its entirety.

Related Strategic Objective: Organizational Excellence

This process evaluation was conducted by the DOCR Information Technology and Program Evaluation Division (S-31). The techniques employed for this evaluation were in accordance with generally accepted program evaluation methods and the Generally Accepted Government Audit Standards (GAGAS). [U.S. Government Accountability Office, Government Auditing Standard (2003 revision). GAO-03-673G.]

The evaluation identified several strengths in DOT’s Title VI civil rights process:

• The Department has people with expert knowledge of the Title VI requirements as they relate to each mode of transportation;
• DOCR and the OAs have established effective partnerships that informally support information dissemination methods;
• DOT's decentralized structure allows for timely intake of complaints; and,
• Decentralized investigations also enable the Department to obtain program specific expertise, which can provide for more effective investigations.

Conversely, the evaluation identified several factors that contribute to untimely complaint processing:

• DOT has not formalized a strategic commitment to early Title VI complaint intervention;
• Despite providing billions of dollars toward transportation services, the Department of Transportation has not reduced Title VI resource inefficiencies within DOCR and the OAs;§ Outdated guidance and standards inhibit DOT’s ability to collect complete and accurate Title VI complaint information;
• Annually scheduled formal training has been unavailable to employees with complaints intake and investigation responsibilities;
• Formal information dissemination methods have not been established, which prevents knowledge sharing; and,
• Ineffective complaint tracking and administrative automated tools do not supply functions required to process complaints expeditiously and are not useful for providing the status of individual cases, or the status of the DOT program in its entirety.

Prior to this evaluation, the DOT began taking actions to improve civil rights programs and their respective products and services. The DOCR, the OAs’ civil rights offices, and other DOT partners are reviewing the composition and organizational structure of their civil rights programs to ensure quality and timely services are provided to customers. In support of this review, a series of streamlining proposals are being discussed for the Title VI and Title VII programs that will improve complaints processing. Also, the DOCR External Policy and Program Development Division has drafted a complaints investigation manual, which will establish a standardized method for investigating complaints filed under Title VI and other nondiscrimination statutes. The document is expected to be approved and its processes implemented by the first quarter of FY 2006.

Nevertheless, the DOT must take additional actions to improve its Title VI program. The following actions are being recommended to improve timely complaint processing and program efficiency for DOT’s Title VI civil rights program:

1. Establish a committee to review the allocation of resources to implement complaint avoidance methods, ensure program compliance, and process/investigate cases;
2. Revise DOT Order 1000.12 and all supplemental guidance to ensure that external civil rights complaints are processed using similar, if not standard, Title VI procedures. The Order must clearly outline procedures for data entry and tracking complaint activity;
3. Establish a formal training program that periodically certifies DOT personnel responsible for processing complaints under Title VI;
4. Establish quarterly knowledge sharing sessions to ensure that DOT personnel, who support Title VI requirements, are knowledgeable of DOT processes and Federal changes impacting Title VI complaint processing procedures;
5. Expand the existing External Civil Rights Task Force initiative to include defining requirements for a Departmental Title VI Civil Rights complaint tracking system;
6. Establish a uniform intake process that ensures all Title VI complaints filed with the DOT are identified and accounted for to provide sufficient detail to meet external reporting requirements;
7. Identify personnel within each OA who is responsible for ensuring that information resources used to create formal electronic records are complete and accurate;
8. Provide each OA with access to quarterly reports that identify the status and processing times of complaints filed in accordance with Title VI of the Civil Rights Act of 1964; and,
9. Establish a Title VI complaint early resolution intervention program that utilizes mediation as a means of resolving complaints filed against DOT in accordance with Title VI of the Civil Rights Act of 1964.

DATA QUALITY REVIEWS

The Research and Innovative Technology Administration’s (RITA’s) Bureau of Transportation Statistics (BTS) employs the statistical expertise of its workforce to conduct reviews of data programs and provide recommendations for data quality improvements.

Related Strategic Objective: Organizational Excellence

The purpose of the data quality reviews is to assess the data systems and methodologies that support data production and maintenance. By strengthening the underlying systems and methodologies, agencies can improve the performance of their data programs and increase the cost-effectiveness of their budgetary investment in such programs.

In recent years, the focus of data quality reviews has been internal to BTS’s own data programs. In particular, various aspects of the Airline Statistics program have been the subjects of review. During FY 2005, four aspects of the airline data program underwent some level of review including the Airline On-Time Performance data, Foreign Air Carrier Traffic data, Airline Financial data and study by the Airline Data Total Quality Management Group. In addition, a review of the National Household Travel Survey was initiated. None of these reviews were completed due to reprioritization of statistical program activities following the creation of the RITA and BTS’s relocation to the new agency.

As necessary, BTS may be called upon to perform this function; however, there are no further reviews scheduled.

DOT RESEARCH & DEVELOPMENT STRATEGIC PLAN PROCESS EVALUATION

Research and technology innovation is critical to advancing the priorities of DOT and in completing the strategic objectives of the Department. Recognizing the need to improve planning and coordination of the Department’s research and development activities, a reorganization was proposed and subsequently approved when President Bush signed the Norman Y. Mineta Research and Special Programs Improvement Act on November 30, 2004. The newly created Research and Innovative Technology Administration (RITA) will be dedicated to the advancement of the Department’s priorities for innovation and research in transportation technologies and concepts.

Related Strategic Objective: Organizational Excellence

The scheduled program evaluation of the Department’s Research and Development Strategic Planning Process was not conducted as planned in FY 2005 because it coincided with the creation of RITA and the reorganization of the Department’s coordination, facilitation, and review of research activities. As a result, the Department’s approach to research, development and technology (RD&T) program evaluation and coordination was significantly strengthened. A new RD&T coordination and management process was adopted through DOT Order 1120.39A, which established the RD&T Planning Council and Team. The basic features of RD&T management and coordination include multiyear strategic planning, annual budget and performance planning, and annual program planning. Early accomplishments were the preparation and submittal to Congress of the mandated report on the Department’s research activities, FY 2005 RD&T program reviews, FY 2007 RD&T budget priorities and cross-modal initiatives, and the beginning of the transportation RD&T strategic plan.

The Department is implementing the coordination, facilitation and review of research activities according to the process of DOT Order 1120.39A and as reported to Congress in March 2005. The strategic planning process will be reviewed by the National Research Council (NRC), pending funding availability, in FY 2006.

TRANSPORTATION PLANNING CAPACITY BUILDING PROGRAM EVALUATION

The Transportation Planning Capacity Building (TPCB) program provides information, training, and technical assistance to help transportation professionals in State and local transportation agencies create plans and programs that respond to the needs of the many users of their local transportation systems. The program improves transportation planning to be more comprehensive, inclusive and efficient.

Related Strategic Objective: Organizational Excellence

The TPCB program began as a pilot for metropolitan areas in 2000 and was expanded to include statewide, rural and small communities, and Tribal transportation planning in 2001. The Federal Highway Administration (FHWA) assesses program accomplishments, communicates the results to program stakeholders, and adjusts the program as necessary to best meet its objectives with available resources. The program consists of four key task areas or tracks: outreach and dissemination; peer programs; training; and scans and research.

Because of the differences in work processes and products among the four tracks, FHWA applied the following outcome and process evaluation techniques to evaluate the different tracks:

• Quantitative measures of geography and other audience factors were employed to assess the market penetration of program materials; less formal qualitative feedback from meetings with stakeholders was used to assess quality and usefulness of outreach and dissemination methods. The results are summarized in annual reports.
• FHWA annually assesses information about participants such as agency type and location to assess gaps in program coverage. After the first two years, the program surveyed a balanced sample of peer program participants to assess the application of knowledge and skills gained in from participating in the peer programs.
• Training delivered through the National Highway Institute and National Transit Institute includes a pre- and post-evaluation of class participants to assess how course content and instructor ability affect participant learning.
• Customer responsiveness is measured using an annual assessment of matching research to program stakeholder topic area priorities, and the completion of research on time and on budget is monitored on a monthly basis.

The FHWA annually adjusts the content and delivery methods of the program based on the evaluation results. For example, based on the list of distributed booklets and Web site downloads, a CD-ROM was developed containing the most requested documents. This reduced the cost of distributing booklets, while increasing the types of media available to program audiences. In another instance, a review of 21 peer events, which included participants from 34 States and the District of Columbia, revealed the need to increase the diversity of participants. In FY 2004–2005, the program included participants from 12 additional States and increased diversity by including representatives from several different areas of transportation and all levels of government.

FY 2004 PROGRAM EVALUATION UPDATES

For those evaluations that were scheduled for completion in FY 2004 and did not have results available for publication in the FY 2004 Performance and Accountability Report, an update is provided below.

EVALUATION OF FMCSA COMPLIANCE REVIEW PHASE II

The Federal Motor Carrier Safety Administration's (FMCSA) enforcement and compliance programs are nationwide programs in which FMCSA and State partners conduct on-site compliance reviews (CR) and roadside inspections (RI) of motor carrier compliance with the Federal Motor Carrier Safety Regulations (FMCSR) and Federal Hazardous Materials Regulations (FHMR). FMCSA expects that through enforcement of these regulations, and promotion of safety requirements, motor carriers will improve the safety of their operations and reduce their chances of being involved in crashes.

Related Strategic Objective: Safety

This process evaluation is a management study conducted for the purpose of improving the effectiveness of FMCSA's enforcement and compliance programs. A workgroup of FMCSA headquarters and field personnel, with the support of the Volpe National Transportation Systems Center, conducted this evaluation. This is the second phase of a two-phase study. Phase I focused on developing short-term improvements to the existing CR process. The Phase II effort had the broader goal of developing long­term improvements to the agency's overall enforcement and compliance programs.

The scope of this evaluation was all aspects of FMCSA enforcement and compliance operations, which account for the great majority of all agency activities and resources.

The methodology used for this evaluation was to gather data on existing FMCSA enforcement and compliance operations, examine the current results of these operations, and assess the long-term efficacy of the agency's current operational model. In making this analysis, the study also compared FMCSA operations to those of similar operations of other Federal, State, and Canadian organizations.

Public listening sessions were conducted in San Diego, Atlanta, Dallas, Chicago, Falls Church, Virginia and Springfield, Massachusetts. The listening sessions, conducted in September and October 2004, were designed to collect public input regarding ways FMCSA can improve its process of monitoring and assessing the safety performance of the commercial motor carrier industry. These sessions help support the objective of obtaining input in advance so that formal proposals are crafted in a clear and concise manner to facilitate future deliberations through the rulemaking or legislative process.

The Compliance Review Work Group issued a Phase II final report in February 2005, Proposed Operational Model for FMCSA Compliance and Safety Program. The report concluded that FMCSA’s existing compliance and safety programs are resource intensive and reach only a small portion of the regulated community and that a new operational model for FMCSA was needed. As a result, the Comprehensive Safety Analysis 2010 (CSA 2010) initiative was formed to implement the necessary changes. The ultimate goal of CSA 2010 going forward is to develop an optimal operational model that will assess the safety posture of the entire regulated industry. The Agency wants to be able to globally say, “This is a safe operation,” for those motor carriers that are truly operating in a safe manner. Conversely, the FMCSA wants to be able to focus its limited resources on monitoring and improving poor safety performers.

EVALUATION OF FHWA STATE MOTOR FUEL DATA

This process evaluation set out to examine State motor-fuel data to reduce the risk of errors and increase the reliability of the information used to distribute Federal highway program funds to the States. State motor-fuel data reported to the Federal Highway Administration (FHWA) is used as an apportionment factor in Federal-aid Highway funds distribution.

Related Strategic Objective: Mobility

A June 2000 Government Accountability Office (GAO) Study stated that there was little assurance that the Federal-aid Highway funds distributed to the States were sufficiently accurate. The GAO made the following recommendations to FHWA as a means of increasing accuracy:

• Perform detailed oversight verifications of motor fuel data used in process;
• Fully document the current methodology;
• Conduct an independent review;
• Evaluate the potential reliability of the Internal Revenue Service (IRS) Excise Files Information Retrieval System (ExFIRS) data as a tool to verify validity of State data.

FHWA agreed with the above recommendations and set out an action plan to achieve the results.

The scope of the evaluation was comprehensive with every aspect of the motor fuel reporting and attribution process in every State being evaluated. High-risk areas and FHWA internal processing were given the highest priorities. Continuous process improvement model was the single most prominent feature of the evaluation design. Other methods included zero defect processing, modeling, and comparison of State data sets with Treasury results. FHWA found through a reassessment that its basic attribution process was sound but in need of updating. It set out a multi-pronged action plan that included outreach and data-provider training to improve accuracy.

With one exception, FHWA and GAO have agreed that all action plan items and milestones were met. The exception concerns a comparison of Internal Revenue Service (IRS) ExFIRS data set with FHWA State-reported data set. The Agency is currently working with the IRS to resolve this issue.

EVALUATION OF FEDERAL HIGHWAY ADMINISTRATION (FHWA) INTELLIGENT TRANSPORTATION SYSTEMS (ITS) DEPLOYMENT

The ITS program oversees the deployment and use of ITS technology to improve transportation on Federal, State and local highways, including private vehicular traffic as well as transit and commercial vehicle operations.

Related Strategic Objective: Mobility

The ITS deployment tracking program is intended to track the integration of ITS technology in major metropolitan areas. This is accomplished by tracking deployment outputs, including numbers of systems deployed, percentage of roadway miles under instrumentation, and percentage of vehicle fleets instrumented, as well as integration between key metropolitan agencies.

Data for the process evaluation are gathered through surveys of transportation agencies in the 75 largest metropolitan areas. A score is assessed for each ITS deployment based on five key areas: freeway, arterial, transit, public safety, and traveler information. An integration score is determined based on evaluation of real-time integration between freeway, arterial, and transit agencies. The deployment and integration rankings are combined into a single ranking of high, medium, or low for each metropolitan area. The overall goal is for each of the 75 metropolitan areas to achieve a ranking of medium or high by CY 2005.

The most recent rankings report was published in December 2004. The FY 2004 survey results indicated that a total of 62 areas achieved a medium or high level of deployment, six short of the 68 target. The final 2005 results will be published in May 2006.

In September 2005, the Government Accountability Office published report GAO-05-943, Intelligent Transportation Systems’ Promise for Managing Congestion Falls Short, and DOT Could Better Facilitate Their Strategic Use. The report found that progress has been made toward achieving DOT’s deployment goal, but DOT’s goal and measures have limitations and fall short of capturing ITS’s impact on congestion. In addition, the measures do not capture the extent to which deployed ITS technologies are effectively operated. DOT noted in their response to this report that the recently passed SAFETEA-LU legislation repealed the ITS integration deployment program. Therefore, DOT will no longer update ITS program goals.

EVALUATION OF THEEFFECTIVENESS OF FHWA DESIGN BUILD CONTRACTING

Design-build is an optional contracting mechanism that enables the design and construction of highway projects to be let in a single contract to one vendor in order to save time and money on highway construction. The Transportation Equity Act for the 21st Century (TEA-21) authorized the use of this approach; a final rule was issued effective January 2003.

Related Strategic Objective: Mobility

TEA-21 required the FHWA to evaluate the suitability of this project procurement and delivery technique for States engaged in highway capital projects. The FHWA examined highway and bridge capital projects, particularly those involved in the Special Experimental Project No. 14 Innovative Contracting (SEP-14) program; developed lessons learned from other types of capital projects, including other modes and industries; and considered the perspectives of both project sponsors and stakeholders.

The objectives of the comprehensive National impact evaluation were to:

• Compare the effect of design-build contracting on project delivery vis-à-vis the traditional design-bid-build approach;
• Determine the appropriate level of design for design-build procurements;
• Assess the impacts of design-build contracting on small business, particularly small contractors and design firms;
• Assess the variation, use, and fairness of cost and non-cost factors used in the award of contracts; and
• Recommend concerning design-build contracting procedures and implementation approaches.

The FHWA prepared a draft Report to Congress on the Assessment of Design-Build Contracting in July 2005. The Report, which is under Departmental review, concludes that the design-build approach saves time in the project delivery process. However, the results are inconclusive regarding cost savings. In addition, there was no appreciable difference between design and construction quality between the traditional design-bid-build method and the design–build method. The study also found that the preliminary design included in the request-for-proposal document should be advanced to no more than a 30 percent design level to assure an efficient use of resources in the project delivery process. The impact of design-build on small firms was inconclusive. Overall subcontracting levels were noted to be similar when comparing design-bid-build with design-build. Cost and non-cost factors were increasingly being used in the award of design-build contracts.

EVALUATION OF FHWA INNOVATIVE BRIDGE RESEARCH AND CONSTRUCTION (IBRC) PROGRAM

The Innovative Bridge Research and Construction (IBRC) Program was authorized under TEA-21. The legislation made funding available to the States for projects to demonstrate the application of innovative materials relating to repair, rehabilitation, and construction of bridges and other highway structures. The program has two main components:

• Funds for repair, rehabilitation, replacement or new construction of bridges and other highway structures using innovative materials; and,
• Support for research and technology transfer activities related to the program's goals.

Related Strategic Objective: Mobility

The IBRC program is essentially a discretionary bridge construction grant program to the States. The process evaluation examined the rate of usage by the States, the effectiveness of the program, and the feasibility and desirability of continuing it in future legislation. A summary assessment of the IBRC program was conducted in 2005, in accordance with the provisions of 23 U.S.C. 503(b).

This program was fully funded at $20 million for construction and $1 million for research during 1998–2005. Of the 92 proposals received in FY 2004, 60 projects were awarded totaling $18.8 million in 41 States, the District of Columbia, and the Commonwealth of Puerto Rico. The States must now execute the contracts related to these project awards and report the results from their construction and in-service performance monitoring programs. A total of 102 project proposals were received from 41 States and the Commonwealth of Puerto Rico in FY 2005. The proposals are currently being reviewed.

The IBRC has been effective in developing and deploying new bridge technologies. Almost all States are now using High Performance Concrete (HPC) regularly in projects. High performance steel technology has also moved into conventional practice. Fiber reinforced polymer (FRP) composites are more popular applications for bridge deck replacements. About 125 bridge projects have been built to date. The use of FRP for repair and strengthening has been fully recognized. Thousands of pier columns were retrofitted using the FRP technology in States located in high seismic zones. Stainless steel or galvanized steel reinforcement is being employed for bridge deck construction. In 2004, the program was expanded to include prefabricated bridges and few projects were selected to demonstrate its effectiveness on rural bridges.

Funding for the IBRC was continued in SAFETEA-LU with a new set-aside for high performance concrete bridge technology research. In addition, several new initiatives that address bridge life and performance were funded including Long-term Bridge Performance, High Performing Steel Bridge Research and Technology Transfer, and Steel Bridge Testing.

FHWA's EVALUATION OF THE NATION’S HIGHWAYS, BRIDGES AND TRANSIT (CONDITION AND PERFORMANCE REPORT)

The Conditions and Performance (C&P) Report provides Congress and other decision makers with an appraisal of highway, bridge and transit physical conditions, operational performance, financing mechanisms, and future investment requirements.

Related Strategic Objectives: Safety, Mobility, Environment, Global Connectivity, Security &Organizational Excellence

Executive Order 12893, Principles for Federal Infrastructure Investments (January 1994), directs each Executive Department and Agency with infrastructure responsibilities to base investments on systematic analysis of expected benefits and costs, including both quantitative and qualitative measures. The C&P Report consolidates conditions, performance, and finance data provided by States, local governments, and transit operators to provide a National summary. The highway investment requirements in the C&P Report are developed in part from the Highway Economic Requirements System (HERS), which quantifies user, agency and societal costs for various types and combinations of improvements including travel time, vehicle operating, safety, capital, maintenance, and emissions costs. The National Bridge Investment Analysis System uses engineering and benefit/cost analysis. Transit investment analysis is based on the Transit Economic Requirements Model (TERM), which consolidates engineering and cost/benefit analysis. TERM identifies the investments needed to replace and rehabilitate existing assets, improves operating performance, and expands transit systems to address the growth in travel demand and evaluates these needs to select future investments. The 2004 version of the C&P Report was completed in December 2004. The DOT will release it upon review and approval by the Office of Management and Budget.

EVALUATION OF MARAD's SHIP DISPOSAL PROGRAM

The Maritime Administration (MARAD) has more than 100 obsolete and deteriorating ships awaiting disposal that pose potentially costly environmental threats to the waterways near where they are stored. Congress, in 2000, mandated that MARAD dispose of them by September 30, 2006. While MARAD has various disposal options available, each option is complicated by legal, financial, and regulatory factors.

Related Strategic Objective: Environmental Stewardship

MARAD’s ship disposal evaluation, originally scheduled for completion in FY 2004, was re-scheduled for FY 2005. However, the planned DOT evaluation yielded to an evaluation that was being conducted by the GAO on the same ship disposal effort. GAO reviewed the ship disposal program and published a report on March 7, 2005 titled Maritime Administration: Improved Program Management Needed to Address Timely Disposal of Obsolete Ships, report number: GAO-05-264.

The GAO study concluded that MARAD is unlikely to meet its statutory deadline of September 30, 2006. As of September 2004, MARAD had disposed of 18 ships from its inventory, with over 100 ships left to dispose of by the deadline. MARAD's slow progress is due to not having a comprehensive management approach, which could address the myriad of environmental, legal, and regulatory challenges that the program faces. MARAD's approach lacks an integrated strategy with goals, milestones, performance measures, and a mitigation plan for overcoming anticipated impediments. In the absence of this comprehensive approach, MARAD's ship disposal program lacks the vision needed to sustain a long-term effort. Consequently, MARAD has not been able to assure Congress that it can dispose of these ships in a timely manner to reduce the threat of a costly environmental event, nor has it clearly articulated what additional congressional assistance, such as funding, may be needed.

In addition the study found that since FY 2002, MARAD had relied on an inappropriate procurement method—Program Research and Development Announcements (PRDAs)—to acquire ship scrapping services. Federal acquisition regulations dictate that PRDAs may only be used for contracting for research or development efforts.

To address the GAO findings, MARAD has taken the following steps:

• To improve program management, MARAD has developed and implemented additional performance measures that more completely cover all aspects of the program; and,
• MARAD has ceased using PRDAs as a contract procurement method for ship disposal efforts and has developed a draft Ship Disposal Comprehensive Management Plan (CMP) that will be finalized in the first quarter of FY 2006.

Further follow-up efforts by MARAD will be evaluated by OMB in FY 2006 when the program is scheduled to undergo a Program Assessment Rating Tool (PART) evaluation.