Skip Navigation
Home Citizen Services Business Services Government Services Contact Us
Image indicating top of page.

DOT Program Evaluluations

Performance measures show if intended outcomes are occurring and assess any trends. Program evaluation uses analytic techniques to assess the extent to which programs contribute to those outcomes and trends. As required by the Government Performance and Results Act of 1993, the Department's FY 2006 - 2011 Strategic Plan includes a schedule of program evaluations by fiscal year.
Types of Program Evaluations

Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes. Evaluations are of the following types:

Program Evaluation Management

The programs selected for scheduled evaluations are vetted through the Department's strategic planning process. Each modal administration nominates programs that are then reviewed by a strategic planning executive committee to ensure: 1) adequate breadth of program evaluations across modal administrations; and 2) alignment to the strategic objectives. The OIG and the GAO pursue program evaluations independent of this schedule.

DOT Agency Program Type of Evaluation Source of Evaluation Status
SAFETY
FAA Operational Error Outcome Internal In Progress/Results available in FY 2009
FMCSA Motor Carrier Safety Assistance Program Process/Cost-Effectiveness Independent Deferred to state by state reviews
FRA Railroad Safety Enforcement (deferred from 2007) Outcome Independent Complete/ No Recommendations
NHTSA National Highway Safety Mobilizations Outcome Independent Complete/ No Recommendations
FMCSA Compliance Review Effectiveness Outcome Independent Complete
FMCSA Roadside Inspection and Traffic Enforcement Effectiveness Outcome Independent Complete
REDUCED CONGESTION
FHWA/ FTA Infrastructure Investment Needs Cost-Effectiveness Internal Complete/ Under Review
FAA Aircraft Delay Reduction Process Independent Complete/ No Recommendations
FTA Job Access and Reverse Commute Formula Grants Outcome Independent Complete
SECURITY, PREPAREDNESS AND RESPONSE
MARAD Maritime Security (deferred from 2007) Impact Independent In Progress/ To be completed in FY 2009
ORGANIZATIONAL EXCELLENCE
MARAD Maritime Education Impact Independent Deferred to FY 2009
PHMSA Outreach and Training Process Internal In Progress
FMCSA Quality Assurance Review – Grants Financial Management Process Independent Complete/ Under Review
FRA Rail Transportation Research, Dev and Demonstration Process Independent Actions Initiated
FMCSA State Division Effectiveness Process/Cost-Effectiveness Independent Complete/ Under Review
PHMSA Information Technology (deferred from 2007) Process Independent Complete

FY 2008 Program Evaluations Supporting Safety

Operational Error Program

FAA An OIG audit of the FAA's Investigating and Reporting of Operational Errors was initiated in November 2007. The objectives of the audit are to: (1) determine whether FAA has adequate policies and procedures in place to ensure the accuracy and consistency of operational error reporting and (2) review the roles and responsibilities of the Air Traffic Organization and FAA's Aviation Safety line of business in reporting and investigating operational errors.

The OIG is in the process of concluding the study and will have a final report in early FY 2009.

Motor Carrier Safety Assistance

FMCSA The FMCSA planned to perform an independent program evaluation of the Motor Carrier Safety Assistance Program (MCSAP) in FY 2008. However, FMCSA recently instituted new approaches to standardize the commercial motor vehicle safety planning process and requested time to allow these innovative processes sufficient opportunity to take effect. The Agency began a nationwide assessment of each MCSAP program grantee which involves conducting approximately 12 grantee reviews per year. With 20 reviews completed to date, FMCSA expects to have all of the reviews completed by 2011. The FMCSA intends to leverage the results from the grantee reviews and make a future determination based on need to perform a MCSAP program evaluation to assess the program's overall effectiveness related to outcomes once the individual reviews are completed.

Railroad Safety Enforcement

FRA In 2005, FRA initiated implementation of a new element of the FRA safety program, called Railroad System Oversight (RSO), to replace a prior safety program. The FRA conducted a program evaluation of the RSO since its implementation in 2005 to evaluate: the effectiveness of the RSO function; the extent to which the RSO supervisor and managers comply with the implementation directive; perceptions and satisfaction level of FRA HQ and regional managers, railroad managers, and labor organizations as to RSO effectiveness as a safety tool; and the extent to which RSO has been effective in communicating with FRA HQ and field personnel, and with railroad and labor organization representatives.

A contractor conducted interviews over two months with 58 stakeholders involved in the RSO process, including key FRA, railroad, and labor personnel. Additionally, the contractor analyzed safety data and statistical information and participated in scheduled RSO conference calls.

Findings Railroad managers and labor representatives expressed highly positive comments about the RSO and its staff. Some were emphatic about the need for FRA to maintain or expand the collaborative and liaison relationship. With one exception, both groups felt that RSO has value and a positive impact on safety. FRA regional administrators and staff directors, however, felt that RSO could be more effective and that it has the potential to be of greater value to the FRA safety program.

Of major concern noted in the interviews was the need to strengthen the communication network among RSO managers, FRA regional personnel, and staff directors. Additional findings and recommendations concerned improving overall effectiveness; tracking accomplishments and activities; greater analysis and specificity of accident/incident data; and the availability to staff directors and regional administrators of safety and compliance issues identified to headquarters by RSO managers.

Recommendations A few of FRA's RSO program recommendations that have been adopted include:

National Highway Safety Mobilizations

NHTSA NHTSA conducted a program evaluation to evaluate Click It or Ticket (CIOT) Mobilization including the use of paid advertisements focusing on seat belt enforcement, measure motorists' awareness of seat belt campaigns, and ultimately measure the change in seat belt use rate. NHTSA also evaluated the high visibility enforcement campaign to reduce impaired driving including the Labor Day and December Drunk Driving: Over the Limit; Under Arrest national crackdowns. The evaluation included review of program data, including dollars spent placing paid advertisements and enforcement activity, state reported observational surveys of safety belt use, knowledge/attitude surveys at driver licensing offices and a national telephone survey conducted in pre/post intervals to track progress.

Findings An analysis of the 2005 CIOT program and an evaluation of the national impaired driving high visibility enforcement campaign to reduce impaired driving 2003-2005 were published in the fall of 2007. Both reports are available under Click It or Ticket Seat Belt Mobilization Evaluation Reports at: http:// www.nhtsa.dot.gov/ portal/ site/ nhtsa/ menuitem. 3d62007aac5298598fcb6010dba046a0. The results of the 2006 programs were published in the fall of 2008 and are available at: http:// www.nhtsa.dot.gov/ portal/ site/ nhtsa/ menuitem. 3d62007aac5298598fcb6010dba046a0/.

The CIOT evaluations have demonstrated the positive impact of the campaign on overall belt use, and that focusing on groups less likely to use safety belts (pickup truck drivers, rural drivers) can increase their belt use. Impaired driving results are more mixed. Some states have made significant progress in reducing impaired driving fatalities, while progress has been slight in other states. The Impaired Driving evaluations show that successful states have comprehensive enforcement and media plans, and vigorous 24/7 enforcement.

The analysis of the 2007 mobilizations is in progress and reports will be published in the fall of 2009.

Recommendations Following the GAO recommendation we have tracked CIOT expenditures and the 2006 mobilization reports break out media costs for Hispanic target audiences.

Compliance Review Effectiveness

FMCSA The FMCSA uses the Compliance Review Effectiveness Model to measure the impact of compliance reviews (CR) on motor carrier safety and to provide states with macro and micro analysis of CR data for grant planning and resource deployment. Based on the individual and cumulative before and after changes in the safety performance of carriers that received CRs, the model estimates the number of crashes, injuries, and fatalities prevented as a result of performing these activities. Outputs from the CR Effectiveness Model are from calendar year (CY) 2002, CY 2003, CY 2004 and FY 2005 (methodology changed from CY to FY in 2005).

Findings Findings for FY 2005 include the total number of compliance reviews conducted (11,431), the estimated percentage reduction in average crash rate due to compliance reviews (16.3 percent), estimated crashes avoided (fatal, injury, and tow-away - 2,306), estimated injuries avoided (1,561), and estimated lives saved (92).

Recommendations FMCSA, in cooperation with RITA's Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of roadside inspections and traffic enforcements in terms of crashes avoided, injuries avoided, and lives saved. The model assigns a crash risk probability to each violation of the Federal Motor Carrier Safety Regulations, then determines how many times each type of violation was detected and corrected at the roadside as a result of the roadside inspection program. Based on this information, the model estimates the number of crashes, injuries, and fatalities that have been avoided each year as a result of the program. Although this evaluation does not produce official recommendations, this information does inform managers on the impact of performing roadside inspections and traffic enforcement activities, enabling informed decision-making on program adjustments and enhancements.

Roadside Inspection and Traffic Enforcement Effectiveness

FMCSA The FMCSA uses an analytic model called the Intervention Model to measure the effectiveness of roadside inspections and traffic enforcement activities in terms of crashes avoided, injuries avoided, and lives saved. The model assigns a crash risk probability to each violation of the Federal Motor Carrier Safety Regulations. It then determines how many times each type of violation was detected and corrected at the roadside as a result of the roadside inspection program. Based on this information, the model estimates the number of crashes, injuries, and fatalities that have been avoided each year as a result of the program. Outputs from the Intervention Model are from CY 2004, CY 2005 and FY 2006 (methodology changed from CY to FY in 2006).

Findings Findings for FY 2006 include program exposure or total number of roadside inspections and traffic enforcement activities. This data shows that in FY 2006, FMCSA and our state partners performed the highest number of interventions (3,273,062). It also provides FY 2006 program effectiveness for estimated number of crashes avoided (19,754), estimated injuries avoided (13,241) and estimated lives saved (748).

Recommendations FMCSA, in cooperation with RITA's Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of roadside inspections and traffic enforcements in terms of crashes avoided, injuries avoided, and lives saved. The model assigns a crash risk probability to each of violation of the Federal Motor Carrier Safety Regulations then determines how many times each type of violation was detected and corrected at the roadside as a result of the roadside inspection program. Based on this information, the model estimates the number of crashes, injuries, and fatalities that have been avoided each year as a result of the program. Although this evaluation does not produce official recommendations, this information does inform managers on the impact of performing roadside inspections and traffic enforcement activities enabling informed decision-making on program adjustments and enhancements.

FY 2008 PROGRAM EVALUATIONS SUPPORTING REDUCED CONGESTION

Infrastructure Investment Needs (Conditions and Performance Report)

FHWA and FTA The Conditions and Performance (C&P) Report provides Congress and other decision makers an appraisal of highway, bridge and transit physical conditions, operational performance, financing mechanisms, and future investment requirements. The C&P Report consolidates conditions, performance, and finance data provided by States, local governments, and transit operators to provide a national summary.

The C&P Report is issued roughly every two years. The 2006 C&P is the most recent, though work on the 2008 C&P is well under way. The content of the report purposely remains similar in each edition to facilitate comparison of data and tracking of trends. The 2006 report was based on the 2004 National Transit Database data at the time it was written, and did not capture the effect of changes in funding levels from SAFETEA-LU.

Findings In the 2006 C&P Report, FHWA reported that combined investment by all levels of government in highway and transit infrastructure has increased sharply since TEA-21 was enacted. Highway capital spending rose 45.2 percent from $48.4 billion in 1997 to $70.3 billion in 2004. While the overall physical condition of the Nation's highway and bridge infrastructure has improved as a result, highway congestion has worsened over the past decade. In order to keep average highway user costs from rising above their 2004 levels for a period between 2005 and 2024, investment by all levels of government would need to increase to $78.8 billion annually in constant 2004 dollars, a 12.2 percent increase above actual. In order to eliminate the existing backlog of highway and bridge deficiencies, as well as address new deficiencies between 2005 and 2024 when it is cost-beneficial to do so, spending would need to increase to $131.7 billion annually.

Recommendations The C&P report presents a series of 20-year capital investment scenarios projecting the potential impacts of alternative levels of public and private investment on system performance, but does not endorse any of these scenarios, or make any specific recommendations regarding future funding levels.

The Transit Economic Requirements Model (TERM), which is the computer model used to predict future transit funding needs for the C&P report, has been used for several special studies over the last year. The National Surface Transportation Policy and Revenue Commission Report, released in December of 2007, and the Rail Modernization Report to Congress (projected release in January of 2009) are the most important of these. Without the continuing data collection and updating of the TERM model, which takes place for the C&P report, these analysis' would not be possible.

Aircraft Delay Reduction

FAA More than one in four flights either arrived late or was canceled in 2007-making it one of the worst years for delays in the last decade. Flight delays are typically the worst at the New York metropolitan airports. The purpose of this study was to assess the effect of FAA's Aircraft Delay Reduction Program on flight delays and cancellations which have plagued the U.S. aviation system. The U.S. Government Accountability Office (GAO) conducted a study of (1) the trends in the extent and principal sources of flight delays and cancellations over the last 10 years; (2) the status of Federal Government actions to reduce flight delays and cancellations by the summer of 2008; and (3) the extent to which these actions may reduce delays and cancellations for the summer 2008 travel season. The GAO-08-934T report is available at: http://www.gao.gov/new.items/d08934t.pdf. Although GAO's scope covers the national airspace system as a whole, their work highlighted the New York region.

The study based its conclusions on an analysis of DOT data on airline on-time performance, a review of relevant documents and reports, and interviews with officials from DOT, FAA, airport operators, and airlines, as well as aviation industry experts and associations on the status and potential impact of the Federal Government's actions to reduce delays.

Findings The annual number of domestic airline flight delays and cancellations has increased about 62 percent while the annual number of scheduled flights has increased by 38 percent since 1998. In the New York area, the trend is even more pronounced. Cancellations in recent years have become more problematic as the airlines are now operating with fewer empty seats per flight.

  1. Data provide an incomplete picture of the sources of flight delay. Current on-time performance data do not capture the full extent of delays or cancellations due to reporting practices by some airlines. Data also fail to capture the extent to which passengers' average travel times have increased due to the fact that DOT tracks flights not passengers, which leaves out passenger delays due to missed connections from other delays or overbooked flights.
  2. Actions are being implemented to reduce delays. The GAO study commended DOT and FAA for taking steps to reduce mounting flight delays and cancellations for the 2008 summer travel season. DOT and the FAA worked with the aviation industry to develop and implement several actions-capacity enhancing initiatives, demand management policies, and air traffic procedures-to reduce congestion and delays for the summer 2008 travel season.
  3. Actions may help reduce delays, but the extent of delay reduction in the summer of 2008 will likely be limited. The growing air traffic congestion and delay problem is the result of many factors, including airline practices, inadequate investment in airport and air traffic control infrastructure, and how aviation infrastructure is priced. Addressing this problem involves choices that affect the interests of passengers, airlines, airports, and local economies.

Recommendations No recommendations were made as part of this GAO evaluation - the findings and conclusions served as testimony to the U.S. Senate's Committee on Commerce, Science and Transportation, Subcommittee on Aviation Operations, Safety, and Security. However, DOT and FAA continue to work both on short-term mitigation and long-term planning.

Job Access and Reverse Commute (JARC) Formula Grant

FTA The JARC program provides grants to States and localities for improving the mobility of low-income persons seeking and maintaining employment. This evaluation summary is a combination of the findings of three evaluations of the JARC program that were funded by the Federal Transit Administration (FTA) and conducted between 2002 and 2007:

Findings

Program Results (1) The JARC program is meeting its mission of providing low-income persons with transportation to entry-level job opportunities. Two-thirds of the JARC users surveyed indicated they would not be able to access their destination without the service they were currently using. (2) The cost-per-ride of JARC services is comparable to other public transit services and the annual program cost of the JARC program compares favorably to other Federal grant programs designed to improve employment opportunities for low-income persons. (3) Every $1 spent on JARC services produces a return of between $1.61 and $1.99 in net economic gains that accrue to the user. Every $1 spent on JARC services produces a return of $1.10 to the taxpaying and traveling public, as measured by (a) the value of taxes paid by JARC users; (b) welfare payments that no longer need to be made to the users; and (c) the value of congestion reduced and traffic accidents avoided that can be attributed to the JARC program.

Partnership Requirements (1) In many cases JARC energized existing planning activities and made services possible. (2) JARC partnerships have jump-started a variety of other partnerships and coordination activity. (3) Planning requirement has led to improved analysis of the regions' job access needs.

Implementation and Sustainability (1) Delays in funding and inability to find local match challenged the ability to provide service. (2) Transportation coordinators have been effective in linking former welfare clients to services. (3) Much greater outreach is needed with private companies. (4) Planning partnerships were instrumental in achieving the job access goals for their area. The planning process resulted in financial partnerships where local agencies were able to coordinate funding and provide transportation services, leading to increased efficiency.

Recommendations

FTA is already working to address the land use barriers by supporting the National Center on Transit Oriented Development's work to promote affordable housing near transit stations.

FTA and Department of Labor already sponsor cooperative agreements with the Community Transportation Association of America to promote private sector involvement in the JARC program including the National Joblinks employment transportation initiative. This is a national peer-to-peer network that links local agencies with experienced practitioners familiar with the human services and workforce development environments and knowledgeable about special client transportation needs.

FY 2008 PROGRAM EVALUATIONS SUPPORTING
SECURITY, PREPAREDNESS AND RESPONSE

Maritime Security

MARAD The Maritime Administration planned to have an independent auditor conduct an impact assessment of the Maritime Security Program (MSP) in FY 2007. However, FY 2007 funding was not available for this project, so it was deferred until FY 2008. The Maritime Administration has awarded a program evaluation contract and the evaluation will be completed in FY 2009.

FY 2008 PROGRAM EVALUATIONS SUPPORTING
ORGANIZATIONAL EXCELLENCE

Maritime Education

MARAD Due to the higher than expected costs for a Maritime Security Program (MSP) program evaluation, funding was not available to conduct a program evaluation of the Maritime Education Programs in FY 2008. The Maritime Administration is deferring this evaluation, subject to availability of funds.

Outreach and Training

PHMSA PHMSA is in the process of conducting a process evaluation of its outreach/training programs-a structured assessment of the underlying logic, planning, and implementation of those efforts that leverage our resources by working with others. The general objectives are to (1) identify possible areas for improvement and/or lessons learned that might be applied in other areas, and (2) begin laying the groundwork for a planned program evaluation of the pipeline safety grants program in FY 2009. The general approach for the evaluation is to assemble documents, interview program managers, formulate/reconstruct logic models, evaluate/test the models to identify logical disconnects, determine how the programs actually work in practice, and evaluate differences between assumptions in the plan and actual implementation.

The first phase, initiated in 2008 and carrying into 2009, is an evaluability assessment-aimed at helping to determine what aspects of outreach/training would be most useful and feasible to evaluate in more detail. The second phase, commencing in 2009, is a more in-depth evaluation of selected program activities or initiatives based on the evaluability assessment.

Findings Preliminary results from Phase I will be available by the end of calendar year 2008.

Recommendations Recommendations from Phase II will be available by the end of FY 2009.

Quality Assurance Review - Grants Management

The FMCSA assessed its seven grant programs for adequacy, consistency, objectivity, efficiency and effectiveness. The evaluation team documented and analyzed policies and procedures; compared associated grant administration processes with applicable government laws and regulations; and reviewed other related work (i.e. A-123 process flowcharts and GAO reports). The purpose of this process evaluation was to assess the extent to which our grant management programs and practices are consistent, standardized, and applied in a uniform and timely manner.

Findings The overarching finding was a lack of uniform program and financial management policies and procedures, and a lack of training in specialized grant management and monitoring.

Recommendations The Agency is still reviewing the recommendations, but in most cases program managers have initiated improvements consistent with several of the following key recommendations:

Of these recommendations, FMCSA has accomplished the first four and has established an initiative to achieve the last two.

Rail Transportation Research, Development and Demonstration

The Federal Railroad Administration (FRA) completed a review of the Research, Development and Demonstration (RD&D) Program associated with freight and passenger rail programs. It was conducted by a committee of the Transportation Research Board through semi-annual meetings that included presentations from FRA program managers, discussions and debate by industry and subject matter experts, and various other data gathering methods. The review covered stakeholder involvement, project evaluation process, research priorities, future committee activities, and the role of academic research in railroad engineering and transportation.

Findings A review of research priorities identified during a 2006 Workshop on research to Enhance Rail Network Performance focused on three issues - safety, capacity, and efficiency. The committee found that several areas are being addressed such as lower cost options for positive train control, use of performance-based standards, cost-benefit analysis and risk analysis for train movement risk assessment, a 5-year plan for low-cost grade-crossing improvements, and a successful close-calls database and reporting pilot to improve safety analysis.

The committee expressed support for the vision and direction of FRA's research, development, and demonstration efforts. Additionally, of note, FRA continues to make good progress in developing a more consistent and priority driven program selection process.

Recommendations

State Division Effectiveness

FMCSA The FMCSA conducted a study to evaluate the effectiveness of its State Divisions in meeting FMCSA safety goals and to identify weaknesses and gaps in planning efforts to meet them. The study also explored steps to eliminate these weaknesses and gaps, and to strengthen the effectiveness of the State Division activities. Additionally, this study identified potential planning and management improvements at all levels that could positively impact the safety goal and that would provide states with macro and micro analysis for grant planning and resource deployment.

The FMCSA used a comprehensive approach to collect and analyze safety and performance data as well as empirical information from managers, staff, and partners. The methodology included five elements: (1) data analysis, (2) division safety plan analysis, (3) field staff focus groups, (4) state division visits, and (5) FMCSA headquarters review team discussions.

Findings
  1. The study found that overall, the State Division offices employ robust management practices that include a range of management strategies and tools to motivate, manage, and develop strong, collaborative working relationships with their state partners.
  2. The State Division offices continually work to strengthen the effectiveness of their enforcement processes, such as arranging to receive real time reports from state police on commercial motor vehicle fatal crashes and targeting at-fault carriers for review.
  3. The study found some shortcomings in the State Division Safety Plan (DSP), in that it lacks input and output feedback loops, which disconnects it from other key planning processes and limits its utility to the State Divisions, Service Centers, and Headquarters.

Recommendations The study provides recommendations to strengthen the DSP. The recommendations suggest a process that integrates the State Division and Headquarters planning and program development into FMCSA's strategic priorities, budget, performance plans, and performance targets. The FMCSA is currently evaluating these recommendations for incorporation into the Agency's business processes.

Information Technology

PHMSA PHMSA conducted a comprehensive Information Technology Program Review on the current IM capabilities in order to identify business and technology performance gaps that inhibit the ability of PHMSA's lines of business to execute mission objectives.

Findings
  1. Organizational roles and responsibilities have not been defined for the management of enterprise-level and shared data.
  2. Data and information are often not accessible through desktop tools. There are often integrity and quality challenges associated with operational data.
  3. Key PHMSA business processes and services are underserved by the PHMSA IT groups.
  4. IT services are provided by three independent teams reporting to separate leaders within PHMSA.
  5. Enterprise-wide IT planning and IT investment management needs to be strengthened.
  6. A gap exists between the data required to report on some external performance measures and the data collected within operational systems.
  7. Internal organizational performance measures that describe how well PHMSA is meeting its mission need to be strengthened.
Recommendations

Contingent on the availability of funding, execution and implementation of the recommendations will take two years. The projects identified under items one and two above are being executed with existing resources. Funding and additional IT resources are required to design, develop, and implement the Data and Technical Architecture projects described above. The core capabilities being provided within the two year window are intended to provide the essential building blocks necessary to support a dynamic data driven organization.

Return to the table of contents.