DOT PROGRAM EVALUATIONS
Performance measures show if intended outcomes are occurring and assess any trends. Program evaluation uses analytic techniques to assess the extent to which our programs are contributing to those outcomes and trends. As required by GPRA, the Department’s 2000 - 2005 Strategic Plan included an initial list of new program evaluations planned for those fiscal years. This appendix provides a summary of DOT’s plan for managing program evaluations within the Department and a report on program evaluations completed in FY 2002. An updated list of program evaluations to be conducted in FY 2003 will be included in DOT’s updated Strategic Plan this September.
Types of Program Evaluations
Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes.
The purpose of this program evaluation plan is to improve the analytic content of evaluations Department-wide in order to manage DOT programs for results. This plan generally focuses on the following types of program evaluation:
▪ Impact Evaluations use empirical data to compare measurable program outcomes with what would have happened in the absence of the program. These represent the highest standard of program evaluations and are often the most difficult and expensive to construct and interpret.
▪ Outcome Evaluations assess the extent to which programs achieve their outcome oriented objectives. Outcome evaluations will use quantitative methods to assess program effectiveness, but fall short of the rigorous causal analysis of impact evaluations.
▪ Process Evaluations assess the extent to which a program is operating as intended. While a true process evaluation will use objective measurement and analysis, it falls short of assessing the causal links between intervention and outcome.
▪ Cost-Benefit and Cost-Effectiveness Analyses compare a program’s outputs or outcomes with the costs to produce them. This type of analysis conforms with program evaluation when applied systematically to existing programs and when measurable outputs and outcomes are monetized.
The aim of this plan is to identify areas of program evaluation for:
▪ programs that represent significant DOT activities contributing to our strategic goals;
▪ programs that are cross-modal in nature, or would benefit from evaluation that is reviewed outside an Operating Administration; and
▪ programs where Department-wide expertise can assist in evaluation planning and review.
Program Evaluation Management
DOT manages program evaluations through a Program Evaluation Council (PEC), comprised of representatives from each Operating Administration and select Secretarial Offices. The PEC reviews proposals for program evaluations, shares information across modes, and monitors ongoing evaluations.
DOT staff, contractors, or academic institutions may do program evaluations. Internal departmental reviews are designed to ensure that the finished evaluations are useful regardless of how they are accomplished.
The Office of Budget and Programs and the Inspector General manage the schedule of program evaluations, foster training and development of program evaluation skills, and review the quality of the program evaluation process. The Office of Budget and Programs works to ensure that the results of program evaluations are considered in the allocation of resources. The Office of the Inspector General continues its own program evaluations independent of this schedule, as deemed appropriate.
A summary of DOT program evaluations completed in FY 2002 follows.
FY 2002 Program Evaluation Summaries
The importance of reducing noise around airports is recognized by Congress, which provided that “non-compatible land uses around airports must be reduced and efforts to mitigate noise must be given a high priority”(49 USC Section 47101(c)). In section 47117(e), Congress directed that the Secretary of Transportation set aside 34 percent of available discretionary funding under the AIP for carrying out noise compatibility programs. Over the past 20 years, considerable effort has been expended to provide relief to noise impacted areas through funding of noise compatibility projects under the AIP.
In FY 2002, the FAA evaluated how effectively its noise set-aside grant program contributes to reducing the noise-impacted population around the Nation’s airports. Data were obtained for the FY 2000 and FY 2001 noise set-aside grants from a detailed questionnaire completed by each of the regional airports division offices. The following findings resulted from the evaluation:
▪ Many airports throughout the country benefit from AIP noise set-aside program grants. A total of 219 noise compatibility projects, spread among 84 airports and in 37 States were supported by the approximately $522M in FY 2000 and FY 2001 AIP monies committed to the noise set-aside program.
▪ The FY 2000 program benefits a residential population of 13,785 and the FY 2001 program benefits a residential population of 19,043.
▪ Funding for the AIP noise compatibility projects is variable from year to year, making it somewhat difficult to forecast annual population benefits.
▪ Many of the FY 2000 and FY 2001 projects were based on noise data representative of aircraft activity during the late 1980s and the early to mid 1990s, prior to the December 1999 completion of the phase out of air carrier aircraft that use older and louder engines (i.e. Stage 2 aircraft). Because of the phase-out, it is likely that for many air carrier airports, the current (and future year) noise contours are smaller than that same airport’s noise contours from the earlier time period. This most likely resulted in some of the reported population benefits occurring in areas of moderate noise impact, rather than all of them being in areas of significant noise impact, as was reported.
The evaluation concluded that the program is effective in benefiting a large number of people exposed to aircraft noise. In order to improve the accuracy of reporting on how the AIP noise set-aside program benefits a population that is impacted by significant levels of noise, starting with the FY 2003 AIP program, guidance will be issued to ensure that 100 percent of all AIP programming decisions are based on current noise contours.
Evaluation of the FAA General Aviation Safety Program Training Methodologies
In the mid-1960s, FAA launched a program aimed at reducing the number of accidents in general aviation (GA). It was called the GA Accident Prevention Program. The premise of the program was to reduce accidents and provide pilots with recurrent training after certification in the best safety practices. By 1995, the program’s name changed to the Aviation Safety Program (ASP) and its mission expanded to include all aspects of aviation, including air carriers and maintenance. Although experience and past surveys, such as the Customer Satisfaction Survey in 1998, affirmed the popularity of the ASP, FAA saw a need to adapt the program to extend its benefits to more airmen and airwomen in the aviation community.
The evaluation focused on the portion of the program devoted to pilots and the current instructional methodologies used. The ASP has many instructional tools, but the one primarily used is the safety seminar, which pilots are encouraged to attend. Also, the ASP sponsors an incentive program, the Pilot Proficiency Award Program, informally known as WINGS. Besides attending seminars, WINGS participants are offered 3 hours of training in an aircraft for each of the 20 levels presently available. As an incentive, they receive lapel pins in the shape of wings to signify their accomplishment. Participation in either a single seminar or WINGS is entirely voluntary and provides the individual pilot with the opportunity to establish a recurrent training program after certification.
To gather the data for the evaluation, FAA and the Bureau of Transportation Statistics conducted a nationwide survey of pilots. Survey questionnaires were distributed to 5,400 certificated pilots randomly selected from the Civil Airman Registry, and 1,859 pilots responded to the survey, resulting in a 34 percent rate of return. The rate of return makes this sample highly representative of the pilot community as a whole.
The evaluation concluded that the ASP appeals to pilots holding all levels of certificates and they agree that program participation makes safer pilots. FAA intends a stronger outreach effort to the 53 percent of the pilot population that has not attended in the past two years. Outreach will be based on the following: Web-based training, incentive programs that include continuing education credits, partnering with individual employers, and more pilots participating in WINGS. More analysis is necessary to determine seminar topics of interest to participants by their experience and pilot certificate.
Evaluation of the FAA Runway Safety Program
The objective of the Runway Safety Program evaluation is to assess the effectiveness of FAA’s internal mechanisms designed to accomplish its runway safety goals. The evaluation took place over a five-month period, and provided useful insights into the functioning of the program. Data collection, accomplished via document review and interviews, was limited to June-August 2002. The data analysis process consisted of aggregating interview data, identifying trends in sub-categories, and then finding general trends.
Data and analysis suggest that the Runway Safety Program is striving towards successfully meeting runway safety goals. The analysis suggests that the program is making significant progress and that there is extensive support for runway safety initiatives at all levels (headquarters, regions, and field) of the organization.
The evaluation team observed that the current Runway Safety Program is in the early stages of formalizing policies and procedures. Ongoing efforts will prove themselves effective in the long term.
Evaluation of the FHWA Congestion Mitigation and Air Quality Improvement (CMAQ) Program
Following TEA-21 reauthorization hearings, Congress requested that the National Academy of Sciences’ Transportation Research Board (TRB) conduct a cost-effectiveness study of the CMAQ program, which is authorized and described by section 149 of title 23, United States Code. As a result, FHWA elected not to conduct a separate evaluation as originally scheduled in the DOT FY 2000 Strategic Plan. Accordingly, future program evaluation activities will incorporate the findings and recommendations of TRB Special Report No. 264, The Congestion Mitigation and Air Quality Improvement Program: Assessing 10 Years of Experience.
A TRB Committee of experts concluded (based on very limited objective data) that while a broad range of transportation planners, operating agency staff, air quality officials and interest groups believe the CMAQ program is valuable and support its continuation, projects aimed at reducing emissions directly through the use of technology or fuel standards have been more successful than CMAQ program strategies that rely on changes in travel behavior.
The TRB recommended:
▪ the CMAQ program be continued with a high priority on air quality improvement, rather than congestion mitigation;
▪ the CMAQ program be broadened to address all pollutants regulated under the Clean Air Act, not just the transportation-related pollutants for which the EPA has set National Ambient Air Quality Standards;
▪ Congress consider authorizing the use of CMAQ funds for areas in violation of standards for fine particulate matter and ozone; and
▪ addressing additional categories for CMAQ funds including vehicle scrappage programs, expanded eligibility for operating expenses, and land use actions.
During FY 2003, DOT will work with Congress and the States to reauthorize the CMAQ program and develop future program evaluation plans.
Evaluation of the FMCSA Compliance Review Process
The compliance review (CR) is FMCSA’s essential compliance and enforcement safety intervention, and is an on-site examination of a motor carrier’s operations to determine whether the motor carrier meets the safety fitness standards and is in compliance with the Federal Motor Carrier Safety Regulations (FMCSRs) and Federal Hazardous Materials Regulations (FHMRs). Phase I of this two-phase process evaluation documented the current CR process, focusing on recommendations for improving the efficiency and effectiveness. The evaluation identified process improvements, including: policy and procedural changes, training improvements, and software and carrier data system upgrades. Phase II, which commenced in 2003, will focus on alternative approaches to ensuring compliance. The Volpe National Transportation Systems Center’s Compliance Review Impact Assessment Model documents the effectiveness of compliance reviews in contributing to motor carrier safety – available at http://ai.volpe.dot.gov/CarrierResearchResults/CarrierResearchResults.asp?file=HTML/FMCSA_RI_02_005.htm.
In Phase I, a FMCSA workgroup, supported by the Oak Ridge National Laboratory, documented the current CR process and examined aspects of the system affecting the efficiency and effectiveness of the FMCSA’s identification, evaluation, and prosecution of carriers with poor performance. The workgroup analyzed data from existing FMCSA sources and conducted an extensive field survey. The evaluation identified process improvements, including: policy and procedural changes, training improvements, and software and carrier data system upgrades. An Implementation Plan addressing all 18 recommendations outlined by the review has been developed and is underway. Phase II, which commenced in 2003, will focus on alternative approaches to ensuring compliance.
Impact Evaluation of the Coast Guard’s Maritime Safety Program
The Coast Guard is improving maritime safety programs through a risk-based decision making system. To evaluate this more systematic approach to maritime safety, risk-based safety program decision-making is being tested through a pilot project in one Captain of the Port zone (Charleston, South Carolina). During FY 2002, the USCG field operations organization in Charleston and the USCG Research and Development Center created a baseline risk profile through a typology of all Coast Guard and private sector maritime activities, a determination of the safety hazards inherent in those activities, and an inventory of prevention and response activities used by the Coast Guard to address the hazards and manage the risk.
Results of the risk profiling effort showed that:
▪ for vessel and facility types, recreational vessels and commercial fishing vessels contributed most to increased safety risk;
▪ for mishap types, personnel injuries contributed most to increased safety risk, followed by fires/explosions and loss of vessel control; and
▪ vessel transiting operations had higher relative risk than loading and unloading, shore facility operations, and waterways management operations.
During FY 2003, candidate safety program activities will be chosen for analysis. The maritime safety-as-a-system evaluation will capitalize on port-specific risk profiles by assessing risk sensitivity resulting from changes in specific program safety activities. Changes in risk and program costs will be used to determine risk reduction benefits for each safety activity. Activities demonstrating a significant potential for reducing risk in comparison to cost will be identified.
Evaluation of the Coast Guard’s Domestic Icebreaking Program
The Coast Guard sponsored a study conducted by the Center for Naval Analysis (CNA) to assess its capability to perform its domestic icebreaking mission and to determine an optimal force mix. Because of resource limitations, the capability study was limited to East Coast icebreaking. In addition, the CNA study examined the benefit/cost ratio of the domestic icebreaking program using a number of prior studies to assess the program’s worth.
To determine the Coast Guard’s icebreaking capability, a mathematical model was developed to calculate how long an icebreaking assignment should take given the length of the waterway involved, the probable thickness of ice and the vessels used. The model was populated with data and was run to calculate required icebreaking hours. These figures were compared to available assets to determine whether the Coast Guard can meet requirements or whether resource gaps exist.
CNA concluded that the Coast Guard would continue to be able to meet icebreaking demand on the East Coast during most winters. However, the study was inconclusive on the ability of the Coast Guard to meet demands in severe winters, noting that although standards of service are lower in severe winters, customer expectations may not be lower.
The economic analysis reviewed five (5) prior studies that estimated benefit/cost ratios ranging from 9:1 (almost $9 of benefit for every $1 spent) down to a ratio of almost 2.5 to 1. The study concluded a benefit/cost ratio of at least 2:1 stating that this figure represents only monetary benefits to industry on the Great Lakes and on the East Coast. The study also noted that overall benefits might be higher considering other missions conducted in ice-laden waters, including flood control and search and rescue.
Evaluation of the Coast Guard’s Strategy for Interdiction Illegal Immigrants
The purpose of this study is to develop a strategic plan for the Coast Guard migrant interdiction mission. The strategic goals, objectives, and organizational foundations of Alien Migrant Interdiction Operations (AMIO) are being identified in the study. The results of the study will form the basis of the new 10-year AMIO strategic plan, tentatively named SOVEREIGN SHORES. A final report will be completed in FY 2003.
Evaluation of the Coast Guard’s Process for Capturing Recreational Boating Fatality Data
This evaluation resulted in a significant correction to Boating Fatality Data deficiencies. In FY 1999, the Department of Transportation’s Office of Inspector General (OIG) conducted an audit of the Coast Guard’s Recreational Boating Safety (RBS) program to examine how it sets and measures its performance goal. As a result of the audit, the OIG found a discrepancy between the boating fatality data captured by the Boating Accident Report Database (BARD) and the Coast Guard’s Search and Rescue Management Information System (SARMIS). The Coast Guard used the IG’s analysis to improve recreational boating fatality data collection.
This evaluation reviewed and measured the effectiveness of transferring data from SARMIS to the BARD system. The evaluation examined SARMIS case data for the years 1998 through 2001 to identify SARMIS cases where Coast Guard Search and Rescue (SAR) personnel rendered assistance to a recreational vessel that involved loss of life. The evaluation also reported on the new process implemented in January 2001 that was designed to notify appropriate State agencies of recreational boating fatalities using SARMIS II. This new process provides timely casualty information to appropriate State officials in the jurisdiction where an appropriate SAR case occurred.
The study confirms that for 1998 through 2000 the BARD database underreported approximately 6 percent of the fatality cases based on SARMIS reviews. After the new process of notifying the States of potential BARD fatality cases from the SARMIS database as they occurred, the percent of under reporting of recreational boating fatalities fell to approximately 1 percent.
Evaluation of FHWA’s Transportation Infrastructure Finance and Innovation (TIFIA) Program
As part of its 1998 enactment of the Transportation Equity Act for the 21st Century (TEA 21), Congress established a unique Federal credit program for large transportation projects. Sections 1501 to 1504 of TEA 21, collectively the Transportation Infrastructure Finance and Innovation Act of 1998 (TIFIA), authorize the DOT to provide three forms of credit assistance – secured (direct) loans, loan guarantees and standby lines of credit – to surface transportation projects of national or regional significance. This evaluation fulfills the requirement in TEA 21 to summarize the financial performance of the projects assisted by TIFIA and to discuss alternatives for achieving the program objectives in the future.
In establishing TIFIA, Congress found that a “Federal credit program for projects of national significance can complement existing funding resources by filling market gaps, thereby leveraging substantial private co-investment.” Because credit assistance requires a small fraction of the contract authority needed to provide a similar amount of grant assistance, TIFIA promotes a cost-effective use of Federal resources to encourage co-investment in transportation infrastructure. Federal grant funds that otherwise might be required to support these large projects can then be redirected toward smaller but critical infrastructure investments.
An explicit goal of the TIFIA program is to induce private investment in transportation infrastructure. Private co-investment in the TIFIA project selections totals about $3.1 billion, comprised of more than $3 billion in debt (including State and local debt held by private investors) and nearly $100 million in equity. This co-investment totals approximately 20 percent of the nearly $15.4 billion in total costs.
The broad project eligibilities and flexible financial provisions in TIFIA have enabled the DOT to assist projects in meaningful ways other than facilitating market access. Project sponsors of higher-rated credits have found that TIFIA assistance can reduce costs, coalesce support and help remove other barriers in advancing projects.
Comparing total capital investment to the total budgetary cost of Federal credit and grant assistance, the TIFIA portfolio represents nearly five dollars in total investment for each dollar of Federal investment. This Federal cost leverage ratio of 4.80 for TIFIA projects compares favorably with the leverage ratio of 1.25 for a Federal-aid project receiving 80 percent of its funding from Federal grant sources.
As current TIFIA projects move into their construction, operation and repayment phases, and as additional projects obtain TIFIA assistance, DOT will acquire better information for determining future policy for transportation infrastructure innovative financing.
Evaluation of FTA’s Job Access and Reverse Commute Grant Program
The Transportation Equity Act for the 21st Century (TEA-21) established the Job Access and Reverse Commute (JARC) program, and DOT became an important partner in welfare-to-work initiatives. JARC grants to local agencies and authorities, non-profit organizations, and transit authorities, improve mobility for low-income individuals seeking employment. Reverse Commute funds are intended to increase access to suburban employment opportunities for everyone, including welfare recipients and low-income individuals. TEA-21 authorized $150 million annually for the JARC program, with no more than $10 million per year of the total being allocated for reverse commute activities. Congress provided $75 million for this program in both FY 1999 and FY 2000, $100 million in FY 2001, and $125 million in FY 2002.
Grantees have used JARC funds for a wide variety of services, ranging from expansion of fixed route bus systems to the provision of customer information. Through FY 2001, 60 percent of JARC funds had been obligated for fixed route services, 34 percent for demand response services, three percent for ridesharing, and three percent for information services.
JARC grant recipients have been highly successful in enlisting the financial participation of human services agencies. In areas that receive JARC funds, the program is successfully meeting the transportation needs of low-income individuals seeking reliable transportation to employment and related support services.
Evaluation of Phase 1 of NHTSA’s “Buckle Up America” Safety Belt Program
Buckle Up America (BUA) was a Presidential initiative announced in January 1997, directing DOT to prepare a plan to increase seat belt usage nationwide. In response, the DOT's National Highway Traffic Safety Administration (NHTSA) developed a plan with current goals to increase the national seat belt use to 78 percent by 2003 and reduce child occupant fatalities by 25 percent by 2005.
Program evaluation data included multiple sources of seat observation results, Fatality analysis Reporting System data (FARS), and the collection of law enforcement citation data. Seat belt use rates increased after the inception of BUA and, in 2002, the national use rate was measured at 75 percent. Ten States, Puerto Rico and the District of Colombia had reached 80 percent belt use in 2001; 11 out of 12 were jurisdictions with a standard enforcement seat belt law. Since the inception of BUA, the number of States adopting a standard enforcement seat belt law increased by seven plus the District of Columbia. Child restraint use improved markedly for children under age five, and more children are being transported in restraints and fatalities decreased dramatically (19 percent).
Research and evaluation data also have consistently shown that intensive well-publicized enforcement produces substantial increases in belt use, in both standard and secondary law locations. Publicity has included substantial paid media to ensure that the message reaches the target audience at the time that the enforcement is about to take place. The immediate direction for Buckle Up America is clearly toward larger and more encompassing publicized enforcement efforts. During May 2002, approximately 30 States conducted intensified enforcement with paid media, most using the Click It or Ticket theme.
Although Buckle Up America is still short of its goal for nationwide 78 percent belt use by 2003, several States have achieved rates above 80 percent and several more are expected to achieve 80 percent or better as part of the May 2003 mobilization.
Evaluation of MARAD’s Federally Funded Maritime Education and Training Program
The availability of mariners to crew commercial and sealift vessels simultaneously is a vital component of strategic mobility. MARAD evaluated Federally funded merchant marine officer education programs at the U.S. Merchant Marine Academy (USMMA) and State maritime academies to determine whether these programs are aligned with MARAD and DOT national security goals. MARAD examined Federally funded maritime education programs during 1987-2000 and trends in the officer labor pool from 1987 through 2002. In addition, the House Committee on Appropriations directed MARAD to evaluate specific areas of these programs in House Report 105-636.
The officer education programs contributed to the MARAD & DOT national security goals by graduating an average of 605 officers each year from 1989 to 1999. Of these, an average of 273 per year graduated with service obligations and the licenses and skills needed to crew commercial and Department of Defense (DOD) organic vessels during peacetime and mobilizations. USMMA and State academy cadets enrolled in MARAD-funded education programs are the primary source of new entrants to the officer pool. In 1999, the academies (USMMA and State) graduated a combined total of 540 licensed third mates and third engineers (260 with service obligations) from the MARAD-funded programs. During 1999, 275 shipboard jobs needed for full mobilization of DOD commercially crewed vessels were third mates or third engineers. This is equivalent to about 630 officers using normal crew rotation practices during a long-term mobilization. The licensed graduates from the USMMA and State merchant marine academies have the technical qualifications and unlimited USCG deck or engineering licenses in rough proportion to third mate/third engineer requirements in the DOD organic fleet.
MARAD estimates that in 2002, a pool of 10,300 active and inactive qualified officers with USCG licenses appropriate for deep-sea service would be available to fill the demand for approximately 9,000 officers. These offices will be needed to meet sealift requirements for DOD’s most likely full mobilization scenario with concurrent full operation of the commercial fleet. Although the supply exceeds the projected demand, the “cushion” may be too small to ensure that there are sufficient officers to meet crewing requirements. If a significant proportion of the inactive mariners do not become qualified under the Standards of Training, Certification, and Watchkeeping (STCW‑95), a potential shortage of immediately available mariners might result in delays in vessel activations. However, STCW-95 includes a clause allowing the USCG to waive crew certification in times of national emergency.
The Federally funded maritime education programs have provided a workforce sufficient to meet mobilization requirements for deep-sea qualified officers. MARAD, in consultation with maritime industry and labor partners, will continue to explore crewing potential supply/demand shortfalls and identify cost-effective initiatives to reduce uncertainties.