Performance measures show if intended outcomes are occurring and assess any trends. Program evaluation uses analytic techniques to assess the extent to which our programs are contributing to those outcomes and trends. As required by Government Performance and Results Act of 1993, the Department's FY 2000 - 2005 Strategic Plan included an initial list of new program evaluations planned for those fiscal years. This appendix provides a summary of DOT's program evaluation efforts and a report on program evaluations completed in FY 2004.
Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes. Evaluations are of the following types:
The aim of this plan is to identify areas of program evaluation for:
DOT staff, contractors, or academic institutions may conduct program evaluations. Internal Departmental reviews are designed to ensure that the finished evaluations are useful regardless of how they are accomplished.
The Office of Budget and Programs and the Office of the Inspector General manage the schedule of program evaluations, foster training and development of program evaluation skills, and review the quality of the program evaluation process. The Office of Budget and Programs works to ensure that the results of program evaluations are considered in the allocation of resources. The Office of the Inspector General continues its own program evaluations independent of this schedule, as deemed appropriate.
A summary of DOT program evaluations completed in FY 2004 follows.
This evaluation role is provided for in the Bureau of Transportation Statistics' (BTS) enabling legislation. The evaluative effort is designed to review data programs within the Department of Transportation in order to assess the reliability of transportation data emanating from within the Department.
The primary purpose of the review function is to inform Departmental data collectors and data providers of the strengths and weaknesses in such data programs and to learn where weaknesses in data collection and analysis exist. These are essentially process evaluations in terms of examining the process by which data is collected, stored, and manipulated.
These reviews are not intended to address every DOT data program nor is there any requirement that component agencies make programmatic changes based on review findings. Relevant information about data quality is used to determine whether or not measures of program effectiveness can be used in the Departmental reporting mechanisms, such as the annual performance report now coinciding with annual budget preparations. Managers may also choose to use the findings to make improvements or enhancements to existing data programs.
The areas examined include planning and design, data collection, data preparation, data dissemination, and evaluation. The data are also accessed and examined. Data systems are assessed in relation to the quality (accuracy, reliability, and objectivity), relevance, timeliness, comparability, and utility. Particular attention is paid to compliance with the DOT Information Dissemination Quality Guidelines and, for BTS data programs, the BTS Statistical Standards. Reviews are accompanied by recommendations and suggestions for data quality improvements.
This is an evaluation of the FAA's Information System Security (ISS) Program. A key element of this program is the Security Certification and Authorization Package (SCAP), which is the focus of this evaluation.
The objective of the evaluation was to determine the effectiveness of the FAA's ISS Program at accomplishing security remediation measures. Mitre Corporation's Center for Advanced Aviation System Develpment conducted the evaluation.
Only National Airspace System (NAS) systems were addressed. The validity of the remediation measures identified in the SCAPs was not assessed. The remediation status information that was provided by FAA Headquarters, was not independently verified by examining systems in operational use.
A random sample of 30 NAS SCAPs with 285 remediations was used to determine the extent to which remediation actions documented in the SCAPs have been completed in the NAS, and determine if the SCAP process has been sufficiently defined and executed to ensure that remediation measures have been accomplished in a timely manner.
The FAA should (1) develop uniform classifications for the remediation status data; (2) complete and validate the remediation status data; (3) implement a remediation status tracking process; and (4) periodically evaluate the status of all remediation activities.
The FAA should update the status of SCAP remediations as part of a larger effort to best allocate mitigation funding. ATO has documented their process and is expected to complete the updates by September of 2004. Lessons learned by ATO should be applied across other FAA lines of business.
The Federal Motor Carrier Safety Administration's (FMCSA) enforcement and compliance programs are Nation-wide programs in which FMCSA and State partners conduct on-site compliance reviews (CR) and roadside inspections (RI) of motor carrier compliance with the Federal Motor Carrier Safety Regulations (FMCSR) and Federal Hazardous Materials Regulations (FHMR). FMCSA expects that through enforcement of these regulations, and promotion of safety requirements, motor carriers will improve the safety of their operations and reduce their chances of being involved in crashes.
This evaluation is a management study conducted for the purpose of improving the effectiveness of FMCSA's enforcement and compliance programs. A workgroup of FMCSA headquarters and field personnel, with the support of the VOLPE National Transportation Systems Center, conducted this evaluation. This is the second phase of a two-phase study. Phase I focused on developing short-term improvements to the existing CR process. This Phase II effort had the broader goal of developing long-term improvements to the agency's overall enforcement and compliance programs.
The scope of this evaluation was all aspects of FMCSA enforcement and compliance operations, which account for the great majority of all agency activities and resources.
The methodology used for this evaluation was to gather data on existing FMCSA enforcement and compliance operations, examine the current results of these operations, and assess the long-term efficacy of the agency's current operational model. In making this analysis, the study also compared FMCSA operations to those of similar operations of other Federal, State, and Canadian organizations.
The findings of the evaluation indicate that there are avenues which FMCSA could explore for developing a new model for agency enforcement and compliance operations that would yield improvements in motor carrier safety. This issue will be the subject of a public outreach effort by the agency and subsequent redesign of agency safety programs and systems.
FMCSA plans to conduct combined stakeholder meeting(s) in FY 2005. The meeting(s) will provide a forum for stakeholders to share their ideas for long-term improvements to FMCSA enforcement and compliance programs. The results of this evaluation, in combination with the results of the agency's related public outreach efforts, will be used in the development and implementation of a new operational model for all agency enforcement and compliance operations.
The FMCSA's CR program is a Nation-wide program in which FMCSA and State inspectors conduct on-site reviews of motor carrier compliance with the FMCSR. FMCSA expects that through enforcement of the FMCSR, and promotion of safety requirements, motor carriers will improve the safety of their operations and reduce their chances of being involved in crashes.
The purpose of this evaluation is to measure the effectiveness of FMCSA's CR program in terms of crashes avoided, injuries avoided, and lives saved. The objective of conducting this evaluation is to provide FMCSA management and State safety partners with a quantitative basis for optimizing the allocation of resources dedicated to the improvement of commercial motor vehicle safety. The VOLPE National Transportation Systems Center conducted this evaluation.
The scope of this evaluation is the safety impact of all CRs performed by the FMCSA and its State partners. In 2002, Federal and State enforcement personnel conducted 13,430 CRs. The model used to evaluate the impact of these CRs is designed to measure the direct impact of CRs on carrier safety. It is not designed to measure indirect aspects such as deterrence (i.e., the threat of having a CR).
The methodology used to conduct this evaluation is an analytic program evaluation model called the CR Effectiveness Model, which FMCSA developed in cooperation with the Volpe National Transportation Systems Center. The model is based on the individual and cumulative before and after changes in the safety performance of carriers that received CRs. The model compares a motor carrier's crash rate in a time period after a CR to its crash rate prior to that review. To make this comparison, the model uses crash and power unit data from the Motor Carrier Management Information System (MCMIS) snapshots taken before and after the CR.
| 1999 | 2000 | 2001 | 2002 | |
|---|---|---|---|---|
| Crashes Avoided | 1,200 | 2,200 | 1,600 | 1,656 |
| Injuries Avoided | 822 | 1,395 | 1,105 | 1,261 |
| Lives Saved | 51 | 91 | 67 | 70 |
FMCSA's plan is to continue to conduct this evaluation of the CR Program on an annual basis in order to monitor the effectiveness of the agency's CR program. Completion of this evaluation is set as an annual agency milestone.
Roadside inspection and traffic enforcement (RI/TE) are two of the Federal Motor Carrier Safety Administration's (FMCSA) key safety programs. The roadside inspection program consists of roadside inspections of vehicle and driver safety performed by qualified safety inspectors. The traffic enforcement program is based on the enforcement of 21 moving violations noted in conjunction with a roadside inspection. State RI/TE activities are funded through FMCSA's Motor Carrier Safety Assistance Program (MCSAP).
The purpose of the evaluation of the RI/TE program is to measure the impact of the RI/TE program in terms of crashes avoided, injuries avoided, and lives saved. The objective of conducting this evaluation is to provide FMCSA management and State safety partners with a quantitative basis for optimizing the allocation of resources dedicated to the improvement of commercial motor vehicle safety. FMCSA expects that vehicle and/or driver defects discovered and then corrected as the result of RI/TE interventions will reduce the probability that these vehicles/drivers will be involved in subsequent crashes, which will reduce overall crash rates. The VOLPE National Transportation Systems Center conducted this evaluation.
The scope of this evaluation includes all RI/TEs funded by the FMCSA. In 2002, approximately 3.0 million RI/TEs were conducted. The model which is used to conduct this evaluation is designed to measure both the direct and indirect impact of RI/TEs on improving safety, (i.e., crashes avoided, injuries avoided, and lives saved).
The methodology used to conduct this evaluation is an analytic program evaluation model called the Intervention Model, which FMCSA developed in cooperation with the Volpe National Transportation Systems Center. The Intervention Model is based on the premise that the two programs, roadside inspection and traffic enforcement, directly and indirectly contribute to the reduction of crashes. The model includes two submodels that are used for measuring these different effects:
The results of this annual evaluation are as follows:
FMCSA's plan is to continue to conduct this evaluation on an annual basis in order to monitor the effectiveness of the agency's RI/TE program. Completion of this evaluation is set as an annual agency milestone.
This evaluation set out to examine State motor-fuel data to reduce the risk of errors and increase the reliability of the information used to distribute Federal highway program funds to the States. State motor-fuel data reported to FHWA is used as an apportionment factor in Federal-aid Highway funds distribution. The DOT Strategic Plan identified the Evaluation of State Motor Fuel Data as a Management Challenge.
A June 2000 Government Accountability Office (GAO) Study stated that there was little assurance that the Federal-aid highway funds distributed to the States were sufficiently accurate. GAO made these following recommendations to FHWA as a means of increasing accuracy:
FHWA agreed with the above recommendations and set out an action plan to achieve the results.
The scope of the evaluation was comprehensive with every aspect of the motor fuel reporting and attribution process in every State being evaluated. High-risk areas and FHWA internal processing were given the highest priorities.
Continuous process improvement model was the single most prominent feature of the evaluation design. Other methods included zero defect processing, modeling, and comparison of State data sets with Treasury results.
FHWA found through a re-assessment that its basic attribution process was sound but in need of updating. It set out a multi-pronged action plan which included outreach, Smart System, and data-provider training to improve accuracy.
With one exception, FHWA and GAO have agreed that all action plan items and milestones have been met, and program completed. The exception concerns a comparison of IRS ExFirs data set with FHWA State-reported data set (see fourth bullet of Related performance goal: Mobility above). FHWA is currently working with IRS to obtain their data set, and expects that a comparison of data sets for CY 2003 will be done in FY 2004.
The National ITS program oversees the deployment and use of ITS technology to improve transportation on Federal, State and local highways, including private vehicular traffic as well as transit and commercial vehicle operations. To support this program, DOT has developed a systems architecture for ITS deployments, worked with standards organizations to ensure needed standards are created, and produced a large number of guidance documents to assist State and local officials in deploying ITS. Funding is also provided to support a number of operational tests and model deployments of ITS technology.
The ITS deployment-goal tracking program is intended to track the deployment and integration of ITS technology in major metropolitan areas. Progress in the National ITS program can be measured by tracking the number of metropolitan areas with ITS deployment of significant breadth in terms of variety of transportation functions supported and depth in terms of coverage and market penetration. Information from the tracking effort can be used to guide program efforts to address local deployment and integration shortfalls. This is accomplished by tracking deployment outputs, including numbers of systems deployed, percentage of roadway miles under instrumentation, and percentage of vehicle fleets instrumented, as well as integration between key metropolitan agencies.
The evaluation is focused on 75 of the largest metropolitan areas as a measure of National progress. An ITS infrastructure is defined for metropolitan areas that specifies functions performed by agencies and how they interact and the evaluation scope is limited to a selection of key output measures to serve as surrogates for the complete ITS infrastructure. Integration is measured using a limited number of integration links defined between agencies, chosen to involve key levels of government, and highway and transit agencies.
Data for the evaluation were gathered through surveys of transportation agencies in the target metropolitan areas. The same agencies were surveyed over time to track progress. A score was assessed for deployment based on five key areas: freeway, arterial, transit, public safety, and traveler information. Integration is ranked based on evaluation of real-time integration between freeway, arterial, and transit agencies. The deployment and integration rankings are combined into a single ranking of high, medium, or low for each metropolitan area. The goal is achieved for a metropolitan area when it is ranked medium or high. The overall goal is for each of the 75 metropolitan areas to achieve a ranking of medium or high by CY 2005. The survey is a census, not a sample, and 100 percent return is desired. The 2004 survey update has been launched with a target response rate of 80 percent by October 1, 2004; 85 percent by December 31, 2004; and 90 percent by May 31, 2005. The response rate through July 13, 2004 is 41 percent.
Interim goals have been established for each year included in the period of the goal, and as of 2003, the most recent survey update, progress is on track to achieve the overall goal by CY 2005.
The most recent report was published in December of 2003. The survey for the FY 2004 update is currently underway. A snapshot report will be issued in the first quarter of FY 2005 and draft CY 2004 results will be available in December 2004, with final results available in May 2005. The 2005 survey will be conducted in the summer of 2005. A snapshot report will be issued October 1, 2005, draft 2005 results will be available in December 2005, and final 2005 results will be published in May 2006.
Design-build is an optional contracting mechanism that allows the design and construction of highway projects to be let in a single contract to a single vendor for the purpose of saving time and money on highway construction. Authorized by the Transportation Equity Act for the 21st Century (TEA-21), a final rule was issued effective January 9, 2003.
Perform a comprehensive National study of design-build contracting that evaluates the suitability of this project procurement and delivery technique for States engaged in highway capital projects.
Section 1307(f) of the TEA-21 requires the FHWA to assess the impacts of design-build contracting by June 9, 2003.
The IBRC Program was established by Congress under TEA-21 Section 5103-codified under 23 U.S.C. 503(b)(3)(A)(ii) and 503 (b)(3)(B)-and was funded for six years, FY 1998-2003. It was subsequently extended into FY 2004 as a result of temporary extensions of TEA-21 and the FY 2004 appropriations act. The program is intended to demonstrate the application of innovative material technology in the construction of bridges and other structures and has two components. The larger component provides funds for repair, rehabilitation, replacement or new construction of bridges and other highway structures using innovative materials. The smaller component is intended to support research and technology transfer activities related to the program's goals. Overall, the legislation authorized funding to be available to the States for projects to demonstrate the application of innovative materials relating to repair, rehabilitation, and construction of bridges and other highway structures.
This summary provides a process evaluation, as the IBRC program is essentially a discretionary bridge construction grant program to the States. The evaluation will look at the rate of usage by the States, the program's effectiveness at delivering its stated intentions, and the feasibility/desirability of continuing it in future legislation.
This summary assesses how the IBRC program is being conducted in 2004, in accordance with the guidance provided in the statutory reference provided above.
Under the provisions of 23 U.S.C. 503(b), the Secretary of Transportation shall make grants to and enter into cooperative agreements and contracts with States to pay the Federal share of the cost of repair, rehabilitation, replacement, and new construction of bridges and other highway structures that demonstrate the application of innovative materials. Funds are available for bridge projects that meet one or more of the seven program goals listed in Section 503(b)(2) of Title 23, United States Code. However, projects must be on any public roadway, including State and locally funded projects, and funds are available for costs of preliminary engineering, costs of repair, rehabilitation or construction of bridges or other structures and costs of project performance evaluation including instrumentation and performance monitoring of the structure following construction.
Specific selection criteria used in the program consider whether the project that is the subject of the grant meets the goals of the program, as described in the legislation, including:
Project applications were solicited from the State transportation agencies on April 1, 2004; the submission deadline was July 15, 2004. As of the date of printing, more than 70 project applications have been submitted by the States, and another 15 to 20 are anticipated (based on preliminary information provided by the FHWA Division Offices).
The program is being conducted in accordance with the requirements described in the authorizing legislation, and the overall process is therefore considered adequate and appropriate.
The Conditions and Performance (C&P) Report is intended to provide Congress and other decision makers with an objective appraisal of highway, bridge and transit physical conditions, operational performance, financing mechanisms and future investment requirements.
The C&P Report offers a comprehensive, factual background to support the development and evaluation of legislative, program, and budget options at all levels of government.
It also consolidates conditions, performance, and finance data provided by States, local governments, and transit operators to provide a National summary.
Executive Order 12893, Principles for Federal Infrastructure Investments (January 1994), directs each executive department and agency with infrastructure responsibilities to base investments on "systematic analysis of expected benefits and costs, including both quantitative and qualitative measures."
The highway investment requirements in the C&P Report are developed in part from the Highway Economic Requirements System (HERS), which quantifies user, agency and societal costs for various types and combinations of improvements, including travel time, vehicle operating, safety, capital, maintenance, and emissions costs.
The National Bridge Investment Analysis System uses engineering and benefit/cost analysis.
Transit investment analysis is based on the Transit Economic Requirements Model (TERM), which consolidates engineering and cost/benefit analysis. TERM identifies the investments needed to replace and rehabilitate existing assets, improves operating performance, and expands transit systems to address the growth in travel demand and evaluates these needs to select future investments.
The Administration's Safe, Accountable, Flexible and Efficient Transportation Equity Act (SAFETEA) proposal included a provision moving the due date for the biennial C&P Report from January of odd years to July of even years. However, since new legislation has not yet passed, we will instead be targeting the statutory deadline of January 2005 for the C&P Report.
MARAD's ship disposal evaluation was originally scheduled for completion in FY 2004. Due to other commitments, this evaluation has been rescheduled for completion in FY 2005. The evaluation is being prepared by the staff of MARAD's Office of Policy and Plans.
The Departmental Office of Civil Rights (DOCR) supports the Department of Transportation's human capital objectives by enforcing laws. DOCR serves as a guardian of fair treatment for the Department's employees, employment applicants, and former employees. Equal employment opportunity (EEO) services have been made available for the aforementioned customer base since the Office's inception. In 1995 the Departmental Office of Civil Rights decentralized its formal EEO complaint processing services to provide its customers more effective EEO services. Over the last five years, DOCR has begun focusing on streamlining operations to promote greater efficiency. More specifically, the parties are focusing on ensuring that the EEO services provided are done in accordance with the 180 day time frame established by EEOC.
The DOCR Compliance Operations Division administers DOT's formal EEO complaints process and has responsibility for processing and investigating EEO complaints in a timely manner. In 1999, DOCR devised its Investigation Procedures Manual (IPM) based on EEOC regulations and guidance. The procedures manual established an internal formal EEO process and where applicable reduced time frames allotted to complete pertinent stages of the process.
DOCR evaluated its formal EEO complaints process, developed recommendations for improving the process, and will continue to implement changes and refinements intended to reduce case processing time.
The DOCR Evaluation Team selected a process-based methodology to support DOT's requirement for evaluating complaints processing procedures and practices.
Data supporting this evaluation was collected from Federal and DOT guidance, surveys, and the data entered into DOT's EEO automated system of record, WebCMS. The data assisted members of the evaluation team with determining the timeliness of processing complaints at various stages of the formal EEO process. Specifically, the data identified total investigator caseload by region, total and average processing times from file date to case closure, and average processing times between major events in the formal complaints process.
This methodology will enable DOT to obtain answers to the following specific questions about its formal EEO process:
The eight questions identified by the Chief of the Compliance Operation Division helped to determine factors that may be contributing toward high processing times. These questions help to idenfity critical challenges regarding policies and procedures, differences in processing, resources constraints, training needs, communication challenges and potential changes to organizational structure. The findings simply require change if the DOCR will meet Federal and Departmental requirements for protecting DOT's human capital.
As the team examined the data to identify factors that negatively affect case processing times, it was discovered that the WebCMS tracking system contained erroneous data fields. There is a need to employ standard practices and procedures for collecting, monitoring, and maintaining information within an electronic EEO case tracking system.
Future evaluation efforts should be expanded to assess the customers' needs and satisfaction level. Future efforts may also require an examination of other external factors that affect EEO case processing times.
Highlights of the many procedural recommendations include: