DOT PROGRAM EVALUATIONS

Performance measures show if intended outcomes are occurring and assess any trends.  Program evaluation uses analytic techniques to assess the extent to which our programs are contributing to those outcomes and trends.  As required by GPRA, the Department’s 2000 - 2005 Strategic Plan included an initial list of new program evaluations planned for those fiscal years.  This appendix provides a summary of DOT’s program evaluation efforts and a report on program evaluations completed in FY 2003. 

Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes.  Evaluations are of the following types:

• Impact Evaluations use empirical data to compare measurable program outcomes with what would have happened in the absence of the program.  These represent the highest standard of program evaluations and are often the most difficult and expensive to construct and interpret.
• Outcome Evaluations assess the extent to which programs achieve their outcome oriented objectives.  Outcome evaluations will use quantitative methods to assess program effectiveness, but fall short of the rigorous causal analysis of impact evaluations.
• Process Evaluations assess the extent to which a program is operating as intended.   While a true process evaluation will use objective measurement and analysis, it falls short of assessing the causal links between intervention and outcome.
• Cost-Benefit and Cost-Effectiveness Analyses compare a program’s outputs or outcomes with the costs to produce them.  This type of analysis conforms with program evaluation when applied systematically to existing programs and when measurable outputs and outcomes are monetized.

The aim of this plan is to identify areas of program evaluation for:

• programs that represent significant DOT activities contributing to our strategic goals;
• programs that are cross-modal in nature, or would benefit from evaluation that is reviewed outside an Operating Administration; and
• programs where Department-wide expertise can assist in evaluation planning and review.

Program Evaluation Management

DOT staff, contractors, or academic institutions may do program evaluations.  Internal departmental reviews are designed to ensure that the finished evaluations are useful regardless of how they are accomplished.

The Office of Budget and Programs and the Inspector General manage the schedule of program evaluations, foster training and development of program evaluation skills, and review the quality of the program evaluation process.  The Office of Budget and Programs works to ensure that the results of program evaluations are considered in the allocation of resources.  The Office of the Inspector General continues its own program evaluations independent of this schedule, as deemed appropriate.

A summary of DOT program evaluations completed in FY 2003 follows.

FY 2003 Program Evaluation Summaries

BTS Data Quality Reviews

This evaluation role is provided for in BTS’ enabling legislation. The evaluative effort is designed to review data programs within the Department of Transportation in order to assess the reliability of transportation data emanating from within the Department.

Related performance goal(s):  Organizational Excellence

The data quality review efforts apply both to BTS-operated data programs as well as those from other DOT components. For those data programs operated by BTS, the intent is to ensure that BTS statistical data meet the highest professional standards for quality and confidentiality protection. Ongoing quality assurance is aimed at improving data validity as well the processes of planning and design, data preparation, data dissemination, and evaluation. Data systems are assessed for quality (accuracy, reliability and objectivity), relevance, timeliness, comparability, and utility.

For those data programs operated by other DOT components, the intent is to serve as consultant about statistical methods, furnishing recommendations, as appropriate, for improvements in the collection, interpretation, and use of statistical information. In addition, BTS provides continuous updates on its information quality guidelines in order that component agencies may improve their own data programs by consulting the guidance. There is no intention of being comprehensive of all data programs nor is there any requirement that component agencies made programmatic changes based on the findings. Relevant information about data quality is used to determine whether or not measures of program effectiveness can be used in the Departmental reporting mechanisms, such as the annual performance report now coinciding with annual budget preparations. Managers may also choose to use the findings to make improvements or enhancements to existing data programs.

Reviews of nine databases have been completed.  The first four data quality reviews (the Enhanced Traffic Management System, the Metropolitan Intelligent Transportation Systems (ITS) Deployment Tracking Database, the Office of Intelligence and Security (OIS) Survey of DOT Operating Administrations, and the Merchant Mariner Licensing and Documentation System) were conducted at the request of Congress.  The next five reviews were of the Hazardous Material Information System, Unified Shippers Enforcement Data System (UNISHIP), the Airline Passenger Origin-Destination Survey, the National Transit Database Safety and Security Module, and an overview of the National Aviation Safety Data Analysis Center.

BTS plans to complete its comprehensive reviews of the remaining nine airline data systems:  the Traffic and Capacity Summaries (three databases), Airframes and Aircraft Engines, Fuel Consumption, Airline On-Time Performance, and the Commuter and Air Taxi Operator Traffic and Finances (three databases).  BTS will then implement data quality improvements, perform more targeted, in-depth, studies of data quality issues arising during the reviews or subsequent implementation.

Evaluation of FAA’s Free Flight Phase 1 (FFP1)

Free Flight Phase 1 (FFP1) was established in 1998 to deliver new air traffic control technologies focused on early benefits to the National Airspace System.  FFP1 completed planned implementation of this new software in 2002. FFP1 capabilities include the Traffic Management Advisor (TMA), User Request Evaluation Tool (URET), Surface Movement Advisor (SMA), and Collaborative Decision Making (CDM).

Related performance goal(s):  Safety and Mobility

The focus of this evaluation is on measurable benefits delivered to users and customers of the National Airspace System.  Free Flight has a dedicated metrics team focused on analyzing changes in operational performance associated with Free Flight tools.  The team collects detailed data for a one-year period both before (i.e., baseline) and after implementation at each site. 

Data collected includes both performance (capacity/efficiency) and specific conditions (weather, airport configurations).  Statistical analyses are used to normalize for conditions and compare performance before and after implementation.  These analyses are published in semi-annual reports titled Free Flight Performance Metrics: Results to Date.  Seven reports beginning in June 2000 have been published and posted to the Free Flight web site.  The metrics reports contain detailed findings on a site-by-site basis.  Many of the metrics used in this report can be normalized and translated into delay savings, which is a commonly used measure of the value of improvements in National Airspace System (NAS) operations.  The intent is for these metrics analyses to quantify user benefits of early system deployments, and to be used in the development of benefit/cost estimates for future deployments

An integral part of the metrics analysis involves in-depth discussions with air traffic controllers who use the Free Flight Phase (FFP) tools.  These discussions often focus the analyses on specific conditions where improvements are expected.  The FFP metrics team was established at the beginning of Free Flight Phase 1 with the goal of evaluating the user benefits of Free Flight deployments.  The approach used to measure operational impact was developed in collaboration with the Radio Technical Commission for Aeronautics (RTCA) Free Flight Steering Committee.  The metrics team now includes research analysts, database specialists, and air traffic controllers from the following organizations: FAA, MITRE Center for Advanced Aviation System Development (CAASD), CNA Corporation (CNAC), and Crown Consulting.

Findings were:

• Traffic Management Advisor (TMA) was operational at all 7 sites planned under FFP1 and that capacity improved by 3-5% for traffic into Dallas/Ft. Worth, Los Angeles, Minneapolis, Denver and Atlanta airports. 
• The User Request Evaluation Tool (URET) was operational at 6 of 7 En Route Centers planned under FFP1 and direct routings increased by at least 15% at all 6 Centers. 
• The Surface Movement Advisor (SMA) program was operational at all 6 sites planned for FFP1, and customers reported reduced gate delays and diversions.

Detailed metrics reports are part of Free Flight’s on-going evaluation of performance.  These reports, spanning from June 2000 to June 2003, are available on the Free Flight website http://ffp1.faa.gov/approach/approach_ben_met.asp.  The quantitative results from Phase 1 formed the basis for an investment decision for Free Flight Phase 2 now under way and demonstrating similar success.  An investment analysis study for FFP2 based on measured performance improvements in FFP1 is also available upon request.

Evaluation of FAA’s Operational Errors and Deviation Abatement

This evaluation was originally scheduled for completion in FY 2003.   However, due to rescheduling, this evaluation will be completed in FY 2007.

Evaluation of FHWA Innovative Financing Techniques

"Innovative finance" for transportation is a broadly defined term that encompasses a combination of specially designed techniques that supplement traditional highway financing methods. While many of these techniques may not be new to other sectors, their application to transportation is innovative.  In the early 1990s, the U.S. Department of Transportation (DOT), recognizing the need to expand investment in the Nation's transportation infrastructure, launched a comprehensive initiative to create new funding tools and expand flexibility of the Federal-aid highway funding program. This "innovative finance" initiative was an attempt to meet the increasing gap between transportation capital needs and available resources, without direct increases in Federal grant funding. The initiative also responded to States' calls for greater flexibility in the use of their Federal-aid funds.

Related performance goal(s):  Mobility

This report is intended to measure the performance of innovative finance tools with respect to the original goals set by the program. The goals of the U.S. DOT innovative financing initiative were to:

• Accelerate projects by reducing inefficient and unnecessary constraints on States' management of Federal highway funds; and
• Expand investment by: 1) removing barriers to private investment in surface transportation infrastructure, 2) encouraging the introduction of new revenue streams, particularly for the purposes of retiring debt obligations, and 3) reducing financing and related costs, thus freeing up the savings for investment into the transportation system itself.

This report quantifies benefits where feasible, and illustrates, through detailed analysis of specific projects, the benefits that are more difficult to measure on a macro level, but have wide-ranging effects across state surface transportation programs.  This analysis provides the first comprehensive evaluation of U.S. DOT's innovative finance program since its inception.

The results in this report should help guide both Federal policy makers, as they continue to improve and enhance Federal surface transportation programs, and State and local officials, as they continue to make use of these programs to advance critical transportation projects.

The evaluation analyzed projects that have been advanced under one or more of the U.S. DOT innovative finance initiatives since 1994. The inventory records the available information on the four innovative finance categories, i.e., TE-045, Grant Anticipation Revenue Vehicles (GARVEE), Federal credit, and state-based credit, as well as relevant information on the capitalization of the State Infrastructure Banks (SIB) and the GARVEE bonds issued. These tools are defined in the glossary and in the FHWA Innovative Finance Primer (Publication Number FHWA-AD-02-004). The project information was compiled from readily available data sources, including Federal data collection and reporting systems and through direct contact with the states. Data were collected through the spring of 2001, and represent information current as of that time.

Summary of Projects by Innovative Finance Category

The overall analysis approach is both quantitative and qualitative. This two-tiered approach attempts to quantify what can be measured given available data. Next, it attempts to capture less quantifiable aspects of the tools through the analysis of specific projects.

Quantitative Approach

The information collected was captured in a single Microsoft Excel workbook. The Excel workbook contains a number of worksheets that track the unique data elements pertaining to each of the four categories of projects - TE-045, GARVEEs, Federal credit, and state-based credit - as well as the capitalization data elements in terms of the SIBs and GARVEEs.

The second step in the quantitative analysis was determining how to measure the effectiveness of innovative finance, with respect to the goals of the program. Since the primary goal was to increase transportation investment without increasing direct Federal-aid funding, the evaluation first considers the leveraging effect of the program: how much project investment occurred for each Federal dollar invested? This "leveraging ratio" is an important indicator of the effectiveness of Federal-aid funding. Simply put, greater leveraging means that each dollar invested has gone further - building more projects for the same amount of Federal-aid funding.

Next, the approach considers other ways of defining leverage.  Since project acceleration was also a primary goal, available data on project acceleration are gathered and analyzed. Attraction of private investment and new revenue sources are also evaluated. Finally, to show the overall impact of these accelerated projects, economic results are reviewed.

Case Studies

To supplement the quantitative, macro-level analysis of benefits, case studies show how innovative finance tools have benefited individual projects. These case studies address each of the four major categories of tools, and attempt to capture key aspects of the tools that are difficult to measure, or for which data were not available for the entire group of projects analyzed in this report.

By showing how the new tool or approach proved to be a better choice than traditional methods, the case studies illustrate the value of these tools to state and local governments seeking alternative ways to finance critical transportation projects.

The evaluation found that, at least $29.1 billion in innovative finance projects have been advanced, which were supported by $8.6 billion in Federal-aid funding. On average, for each Federal dollar invested in an innovative finance project, $3.40 of construction investment has been enabled, which compares quite favorably to the ratio of $1.25 to $1.00 for every dollar invested in the traditional grant program.

The evaluation summarizes the benefits, based on the case studies, accruing from use of the innovative finance tools, in terms of costs and/or time saved (as available from project sponsors) as well as the mobility, safety, environmental, and other benefits of the project's implementation. These case study projects provide real-world documentation of project acceleration and other benefits that were impossible to calculate over the entire universe of projects

Now that these innovative finance tools have been created, the next frontier for innovative finance will be for States to determine how best to use them in the context of their overall financing programs. For example, States will need to develop policies to determine what level of long-term borrowing is acceptable, and how projects will be selected. As innovative finance programs continue to evolve, FHWA anticipates that States, local governments, and the private sector will continue to be the primary engine for creativity as the nation enters a second decade of commitment to innovative transportation finance.

Evaluation of FHWA Design-Build Contracting Procedures

Innovative contracting techniques relating to project delivery represent an emerging area of focus for those concerned with highway project cost and schedule control. Among these techniques, one of the most promising is design-build project delivery. This is a method of project delivery in which the design and construction phases are contractually-integrated activities of the project development process.

Related performance goal(s):  Mobility

As a result of the successes achieved in other infrastructure development sectors, FHWA established Special Experimental Project (SEP) Number 14, entitled "Innovative Contracting." SEP-14, as it is known, allows State transportation agencies that administer Federal-aid highway projects to apply the design-build project delivery approach to a selected number of projects in order to ascertain the usefulness of this alternative project delivery technique. To date, approximately half the States and a few metropolitan areas have instituted at least one design build project under the SEP-14 program.

Required by Section 1307(f) of TEA-21, the evaluation is intended to provide an unbiased analysis of the cost and efficiency of the design-build project delivery technique relative to the traditional design-bid-build approach, and develop recommendations for implementing design-build contracting where determined to be suitable by the responsible State Transportation Agencies.

Scope of the Evaluation:

• Focus fact-finding and analysis efforts on highway and bridge capital projects, particularly those involved in the Special Experimental Projects 14 program.
• Include lessons learned from other types of capital projects, including other modes and industries.
• Consider perspectives of both project sponsors and stakeholders.

The study will be based on the results of an extensive literature search, interviews with key stakeholders involved in the Federal-aid highway program and the SEP-14 program, and surveys of state transportation agency representatives with design-build project experience. Representative projects will be selected from the SEP-14 program, involving comparable federal-aid highway projects that have been delivered using the design-build and the design-bid-build project delivery approaches.

The evaluation was scheduled to be completed in FY 2003, but funding was unavailable until that same year. FHWA contracted with Science Application International Corporation (SAIC) and AECOM Consult, Inc. to perform this study, which is approximately 30% complete. Results should be available in late FY 2004.

FHWA Safety in the Highway Project Development Process Evaluation

The program evaluation of “Safety in the Highway Project Development Process,” planned for completion in 2003, has been superseded by an evaluation of the “Impact of FHWA Safety Goals and Objectives,” which will be completed in 2006.

Related performance goal(s):  Safety

After FHWA reorganized in 1999, Safety went from a function of the Office of Infrastructure to a program office of its own, with a much stronger and broader mission. Subsequent emphasis on safety by FHWA (as part of its “Vital Few” performance goals) and from the DOT, led FHWA to reconsider its program evaluation plans. In developing its contribution to the DOT FY 2003-2008 Strategic Plan, with the Office of the Secretary of Transportation, FHWA decided not to pursue this evaluation. Issues to be examined in the study are included in the broader evaluation “Impact of FHWA Safety Goals and Objectives,” included in the DOT FY 2003-2008 Strategic Plan (to be completed in FY 2006). That study will evaluate the effectiveness of strategies and initiatives to achieve the Vital Few Safety objectives

FMCSA Compliance Review Impact Assessment Model

The on-site compliance review (CR) is one of the primary activities of the Federal Motor Carrier Safety Administration FMCSA).  Thousands of CRs are conducted each year on commercial motor carriers in an effort to improve their compliance with Federal Motor Carrier Safety Regulations (FMCSRs). 

Related performance goal(s):  Safety

The purpose of the evaluation of the CR program is to measure the effectiveness of the CR in terms of crashes avoided, injuries avoided, and lives saved. The objective of conducting this evaluation is to provide FMCSA management and State safety program managers with a quantitative basis for optimizing the allocation of resources dedicated to the improvement of commercial motor vehicle safety. FMCSA expects that through education, heightened safety regulation awareness, and the enforcement of the CR, motor carriers will improve the safety of their commercial vehicle operations, and, ultimately, reduce their crash rates.

The scope of this evaluation includes all CRs performed by the FMCSA and its State partners.  In fiscal year 2001, Federal and State enforcement personnel conducted almost 14,000 CRs on individual motor carriers.  The model used to conduct the evaluation is designed to measure all direct aspects of CRs on carrier safety, but not the indirect aspects such as “deterrent” effects (i.e., the “threat” of having a CR).

The methodology used to conduct this evaluation is an analytic program evaluation model called the CR Impact Assessment Model, which FMCSA developed in cooperation with the Volpe National Transportation Systems Center.  This design is a “cause and effect” type of program evaluation that shows the direct impact of compliance reviews on carrier safety based on the individual and cumulative “before and after” changes in the safety performance of carriers that received CRs. The model compares a motor carrier’s crash rate in a time period after a CR to its crash rate prior to that review. To make this comparison, the model uses crash and mileage data collected during compliance reviews and CR follow-up inquiries.

The results of this annual evaluation are as follows:

Compliance Review Program Effectiveness: 1999 - 2002

FMCSA’s plan is to continue to conduct this evaluation of the CR Program on an annual basis in order to monitor the effectiveness of the agency’s CR program.  Completion of this evaluation is set as an annual agency milestone.

FMCSA Roadside Inspection and Traffic Enforcement Effectiveness Assessment

Roadside inspection and traffic enforcement (RI/TE) are two of FMCSA’s key safety programs. The roadside inspection program consists of roadside inspections performed by qualified safety inspectors following the guidelines of the North American Standard, which was developed by FMCSA and the Commercial Vehicle Safety Alliance (CVSA). Most roadside inspections by the States are conducted under a grant program (MCSAP) administered by FMCSA. The traffic enforcement program is based on the enforcement of 21 moving violations noted in conjunction with a roadside inspection.

Related performance goal(s):  Safety

The purpose of the evaluation of the RI/TE program is to measure the effectiveness of RI/TE activities in terms of crashes avoided, injuries avoided, and lives saved. The objective of conducting this evaluation is to provide FMCSA management and State safety program managers with a quantitative basis for optimizing the allocation of resources dedicated to the improvement of commercial motor vehicle safety.  FMCSA expects that vehicle and/or driver defects discovered and then corrected as the result of RI/TE interventions will reduce the probability that these vehicles/drivers will be involved in subsequent crashes, which will reduce overall crash rates.

The scope of this evaluation includes all RI/TE’s funded by the FMCSA.  In fiscal year 2001, nearly 3 million RI/TEs were conducted.

The methodology used to conduct this evaluation is an analytic program evaluation model called the Intervention Model, which FMCSA developed in cooperation with the Volpe National Transportation Systems Center.  The Intervention Model is based on the premise that the two programs—roadside inspection and traffic enforcement—directly and indirectly contribute to the reduction of crashes. The model includes two submodels that are used for measuring these different effects:

• Direct effects are based on the assumption that vehicle and/or driver defects discovered and then corrected as the result of interventions reduce the probability that these vehicles/drivers will be involved in subsequent crashes. The model calculates direct-effect-prevented crashes according to the number and type of violations detected and corrected during an intervention.
• Indirect effects are the byproducts of the carriers’ increased awareness of FMCSA programs and the potential consequences that the programs could impose if steps are not taken to ensure and/or maintain higher levels of safety. In order to measure indirect effects, which are essentially changes in behavior involving driver preparation and practices and vehicle maintenance, the model calculates responses to exposure to the programs and the resulting reduction in potentially crash-causing violations.

This design is an “impact evaluation” type of program evaluation that shows the impact of the RI/TE program on crash rates for carriers that were the subject of an RI/TE intervention. 

The results of this annual evaluation are as follows:

^†Mean estimates.  Higher and lower bound estimates were based on different risk assumptions, which may be found in Intervention Model:  Roadside Inspection and Traffic Enforcement Effectiveness Assessment, Sept. 2002.

FMCSA’s plan is to continue to conduct this evaluation on an annual basis in order to monitor the effectiveness of the agency’s RI/TE program.  Completion of this evaluation is set as an annual agency milestone.

Management Study of FMCSA’s Commercial Vehicle Information Systems and Networks (CVISN) Deployment Program

The CVISN Deployment Program is managed by FMCSA.  The program’s goal is to advance deployment of cost effective information systems and communications networks that provide electronic access to timely and accurate motor carrier safety and other information.

Related performance goal(s):  Safety,  Mobility, and future initiatives in security.

The CVISN program review was conducted by FMCSA and the Department’s Intelligent Transportation Systems (ITS) Joint Program Office (JPO) as part of a broader review of ITS programs.  The review provided a venue for Federal, State, industry, and private sector stakeholders to discuss the best ways to build on the successful accomplishments of the CVISN program; determine its future direction; and identify the appropriate business model options to achieve CVISN’s goals and objectives. 

The review consisted of an external and internal component.  The external review included three partnering sessions with key stakeholder groups.  The sessions were conducted as a collaborative effort between FMCSA and JPO.  The internal review focused on the development of a CVISN baseline document to summarize the current content, guiding principles, roles, and deployment models.

Three partnering sessions were conducted:  (1) state and state-based associations, (2) private sector and public/private partnerships, and (3) motor carriers and related associations.  They were facilitated listening sessions designed to obtain input on challenges and opportunities for implementing CVISN objectives, stakeholder roles, and business models to accomplish current CVISN goals and objectives, as well as future expanded capabilities. 

Participants validated that present CVISN objectives are appropriate, and should evolve the capability to support future initiatives in security.  The review yielded recommendations for program policy, program operation, and roles and leadership opportunities, including: (1) a combined fourth partnering session, (2) an interagency summit on security and CVISN led by the Transportation Security Administration (TSA), (3) a carrier working group, and (4) an internal CVISN review team.

Proposed actions and milestone:

• Combined Stakeholder Meeting – FMCSA will hold a combined stakeholder meeting in late 2003.  The meeting will provide a forum to share with the stakeholders the direction for the future CVISN Deployment Program, address key issues (e.g., data quality), and discuss how to jointly achieve the future direction of the program.
• Interagency Summit on Security and CVISN – FMCSA believes security is clearly a major focus area in today’s environment, and proposes security be addressed as part of a separate stakeholder meeting.  While security has not been singled out as an explicit goal or objective of the CVISN Deployment Program, security applications, such as credentials verification, are implicit in many aspects of CVISN deployment and have been addressed insofar as they have impacts on safety.  Under the DOT’s SAFETEA proposal, the deployment of expanded CVISN capabilities includes efforts to enhance transportation security, once a state implements the CVISN core capabilities.  FMCSA also believes TSA should take the lead role, and will coordinate existing Department security-related activities and task forces to ensure they are aware of the CVISN deployment program and its architecture to maintain consistency.
• Carrier Working Group – FMCSA agrees that motor carriers should be more involved in setting the future direction of CVISN and will work aggressively to increase their participation.  FMCSA proposes a regular, annual stakeholder forum focused on CVISN, with emphasis to ensure broad motor carrier participation.  In addition, FMCSA proposes to increase its participation in meetings with key carrier associations.
• Internal CVISN Review Team – FMCSA will use CVISN staff, and involve FMCSA field offices and JPO, as well as FHWA’s Office of Freight Management and Operations, to address issues that can be remedied quickly.

Evaluation of FRA Passenger Equipment Safety Standards

The purpose of the evaluation was to monitor the review of the existing standards, and to determine what changes if any need to be made.

Related DOT performance goal(s):  Safety

The evaluation would review the process of evaluating the standards, determining which remain appropriate, which should be deleted, which required modification, and which new standards should be added.

The original plan was to perform an in-house review.  However, this review was overtaken by other events, and therefore indefinitely postponed.

OMB has circulated a draft bulletin that directs Agencies to do peer reviews of existing regulations.  FRA expects a final bulletin covering peer reviews will be forthcoming.  It would not be a good use of scarce resources to produce two separate documents that are essentially program reviews of regulations still under development such as the Rail Car Passenger Safety Standards.  One of the documents would be because of the Strategic Plan and the other would be for OMB’s peer review.

An action plan has been developed and is in place.  Work has already begun on an intensive review and update of these standards, based on a whole new methodology.  A working group was charged in May of 2003 to perform a comprehensive review of these standards, and has already begun work on the process.  This review will include not only federal personnel, but also industry stakeholders.

FTA Evaluation of the Chemical/Biological Agent Detection System

FTA is currently negotiating with Department of Homeland Security and others to expedite evaluation of the Chemical Agent Detection System deployed within a major U.S. transit system. The final report that documents capabilities and lessons learned from this research effort will be completed by December 30, 2003.  The evaluation report should be completed by March of 2004. However, due to the security sensitive nature of this project, the detailed description of the system and the results of the evaluation will not be included in the DOT FY 2003 or FY 2004 Performance and Accountability Reports.

Evaluation of MARAD’s Federally Funded Maritime Education and Training

The United States Merchant Marine Academy (USMMA) and the six State Maritime Schools (SMS) conduct Federally funded merchant marine officer education programs.

The availability of mariners to crew commercial and sealift vessels simultaneously is a vital component of strategic mobility.  Strategic mobility refers to the concept of ensuring that sufficient contingency sealift [both vessels and crew] and commercial outload ports are available to support DOD mobilization requirements in times of emergency.

Related performance goals:  Homeland and National Security

MARAD evaluated the Federally funded merchant marine officer education programs to determine whether these programs are aligned with MARAD and the Department of Transportation’s (DOT) strategic mobility goals.

The maritime education program evaluation was designed as an impact assessment.

Program Logic Model:

• Inputs: Dollars and staff to operate the USMMA and to provide financial assistance to the SMS. 
• Output: Cadets enrolled at USMMA and SMS. 
• Intermediate outcome: Increased number & quality of licensed merchant mariners graduating; increase number of licensed merchant mariners. 
• Ultimate outcome: sufficient trained personnel in maritime labor pool to meet commercial and DOD crewing requirements in national emergencies; reservists provided to the US naval reserve/US Merchant Marine Reserve Program.

This evaluation examined the influence of the mariner education programs on the achievement of DOT’s and MARAD’s national security strategic goals, over and above what would have occurred without the program or with alternative programs, to the extent that causality could be measured.

To assess the importance of MARAD-funded programs for national security, MARAD sought to determine the pool of officers with unlimited licenses, and the percentage of USMMA, State Student Incentive Payment (SIP) and non-SIP of the total.  Additionally, MARAD sought to determine whether any of these segments comprised a higher proportion of those officers who are employed in shipboard jobs, compared to the rest of the pool.

MARAD assessed the demand for and supply of officers to meet current and projected commercial and sealift requirements by analyzing peacetime and mobilization crewing needs in 2000 and 2002.  Estimates of the supply of officers were derived from United States Coast Guard (USCG) Merchant Mariner Licensing and Documentation (MMLD) data and the results from our 2001 Mariner Survey.  Demand requirements were estimated from DOD mobilization studies and MARAD sealift planning studies, MARAD forecasts of the U.S.-flag self-propelled oceangoing commercial fleet, and MARAD mobilization plans for crewing the DOD inactive sealift fleets.  From these, MARAD estimated the demand for officers based upon DOD moderate risk mobility requirements and full employment of the commercial oceangoing U.S.-flag fleet to support the U.S. economy.  This was the scenario that MARAD had to address in order to determine whether MARAD’s maritime education programs were meeting national security requirements.

For FY 2003, Congress appropriated a total of $49.161 million to operate the USMMA and $7.478 million for SMS financial support.

MARAD was able to examine Federally funded maritime education programs during 1987-2002 and trends in the officer labor pool from 1987 through 2002.  Due to the need for additional data to measure program impact directly, MARAD conducted a random sample survey of currently licensed mariners.  This 2001 Mariner Survey, undertaken in conjunction with the Bureau of Transportation Statistics (BTS), provided data to measure the impact of MARAD’s education programs on national security requirements as well as preliminary information on the pool of officers and documented mariners capable of supporting sealift during national defense emergencies.

The officer education programs contributed to the DOT national security goal by graduating an average of 597 officers each year from 1989 to 2002.  Of these, an average of 269 per year graduated with service obligations and the licenses and skills needed to crew commercial and Department of Defense (DOD) organic vessels during peacetime and mobilizations.

MARAD estimated that in 2002, a pool of 10,700 officers with USCG licenses appropriate for deep-sea service would be available to fill the demand for approximately 9,000 officers needed to meet sealift requirements for DOD’s most likely full mobilization scenario with concurrent full operation of the commercial fleet.  Although the supply exceeded the projected demand, the “cushion” may be too small to ensure that there are sufficient officers to meet crewing requirements.  The equivalent of nearly 80 percent of the current number of qualified officers would be required to meet DOD’s most likely full mobilization scenario.  Consequently, the contribution of officers from MARAD programs with service obligations to feed the pool of mariners has become more critical to the achievement of the ultimate outcome goal.

MARAD education programs were found to be essential to support sealift readiness. However, these programs alone cannot meet crewing requirements for officers.  Sufficient employment is needed for graduates to gain at-sea experience.

While the Federally funded maritime education and training programs have provided a workforce sufficient to meet mobilization officer crewing requirements, the uncertainties inherent in the reliance on market-based mechanisms to ensure that qualified mariners with the required skills are available, under potentially severe time constraints, remains a concern.  MARAD, in consultation with maritime industry and labor partners, will continue to explore crewing supply/demand balances to identify potential shortfalls and will identify cost-effective initiatives (e.g., mariner surveys, mariner tracking systems, reserve programs, and accelerated training programs) to reduce the uncertainties.

Evaluation of RSPA’s Hazmat Grants Programs

The Hazardous Materials Emergency Preparedness (HMEP) Grants program is a formula grants program providing financial assistance to local first responder agencies to conduct planning and training for how to respond to hazardous materials accidents. Grants are made to states that then distribute funding to localities.

Related performance goal(s):  Safety

This program was evaluated with the Program Assessment Rating Tool (PART). The PART, based on criteria developed by OMB, was used to assess HMEP program performance.  PART is an objective and transparent method to determine a program’s effectiveness by evaluating its purpose and design, planning, management, and results.  PART assesses the extent to which an agency manages for results and maximizes the program’s performance, both key requirements of GPRA.

Program outcomes were evaluated, specifically the number of hazardous materials plans completed and the number of first responders trained. 

As this PART review was conducted for the FY 2005 Budget, this evaluation will be reported in the FY 2005 Program and Accountability Report.

Evaluation of RSPA’s Continuity of Operations Plans

Due to the classified nature of the subject matter, this evaluation cannot be published.