Summary of FY 2001 Program Evaluations:

Project Kimball (Coast Guard)

The Coast Guard chartered a project team to conduct an intensive examination of the boat forces (groups, stations, and aids to navigation teams) and how they function as an integrated, interrelated system and to identify issues and problems affecting mission performance. The project team worked concurrently with other reviews of the search-and-rescue program, including one by the Department of Transportation Inspector General. The team was tasked with making recommendations that would improve performance. The evaluation was initiated in response to increased boat forces operational failures, an identified lack of resources, and a perceived loss of focus on boat forces readiness. The team compared performance expectations at each type of unit to the resources and performance each organization actually provided. The team undertook its work making every effort throughout the study to ensure that all recommendations would support successful mission accomplishments and Coast Guard core and strategic values and goals.

The project employed Human Performance Technology (HPT) in a three-phase approach focusing on desired outcomes. HPT is an accepted and systematic approach to both solving problems and identifying opportunities for improvement. The evaluation was divided into three chronological (HPT) phases: Work, Workplace, and Worker. Each phase was designed to “build upon” the previous phase’s efforts.

Fourteen (14) principal recommendations were made in the study dealing with personnel management practices, staffing, training of boat forces personnel, engineering systems, command and control capabilities and organizational support structures. The evaluation results will be used in further developing parts of the Coastal Search and Rescue Strategic Plan, budget strategies, and numerous other efforts, including force allocation, response boat development and engineering and personnel support program improvements.

Drug Smuggling Deterrence Study (Coast Guard)

A drug smuggling deterrence study was co-sponsored by the Office of National Drug Control Policy (ONDCP), the U.S. Customs Service, and the U.S. Coast Guard. The study -- entitled Measuring the Deterrent Effect of Enforcement Operations on Drug Smuggling, 1991-1999 -- was completed in August and released by ONDCP in September.

The study, took a mixed qualitative-quantitative approach.

The quantitative analysis assessed whether interdiction operations or events affected drug trafficking activity. This approach was encumbered by an incomplete picture of drug enforcement resource levels and activities, and by a lack of regional price series for cocaine. Despite these shortcomings, there was evidence that interdiction operations and events significantly impacted the drug supply chain. In particular, the study found that:

▪ Source zone interdiction operations and the arrest or death of major drug traffickers caused increases to cocaine prices in the U.S.

▪ Most transit and arrival zone interdiction operations do not have a statistically significant impact on U.S. cocaine prices, but exhibit an impact on trafficker behavior (e.g., smugglers change their transportation mode or route).

The qualitative analysis attempted to replicate the 1989 Rockwell study Measuring Deterrence – Approach and Methodology, which interviewed convicted smugglers to assess deterrence.

▪ Following interviews with smugglers, the study concludes that the biggest deterrents are:

- Threat of informants, whether cooperating defendants or confidential informants.

- Prison terms of 25 years or greater.

- Ability to be prosecuted under “dry conspiracy” charges (i.e., they need not be caught in possession of the drug to be convicted).

▪ The smugglers did not perceive a threat on the water:

- They think that law enforcement lacks the necessary assets to spot them.

- If spotted, they are certain law enforcement can’t catch them, and, if caught, the use of sophisticated (hidden) compartments makes it virtually impossible for the drugs to be located.

Readiness Management System (Coast Guard)

In September 1999, the Coast Guard chartered a Readiness System Development Team (RSDT) to assess how the Coast Guard measures and manages its readiness and to develop specific guidelines to establish a standard, service-wide readiness management system. The RSDT determined that the Coast Guard needed an agreed-upon framework to better manage readiness. While some segments of the Coast Guard have systems or processes to measure readiness, large parts of the Coast Guard based their readiness information on anecdotal stories. In some cases, individual units or communities adopted their own measurement systems to meet their own needs. Thus, the Coast Guard was unable to measure readiness accurately on a service-wide basis.

The RSDT determined that every level of personnel -- from station commanders at remote locations to senior Headquarters decision-makers – need a readiness management system based on common models, measures, standards, and data. As a result of its research and interviews, the RSDT developed four foundational models upon which a new Readiness Management System (RMS) is based:

▪ Resources to Results: Depicts how the Coast Guard turns resources (i.e., people, information, and systems) into results.

▪ Three Tiers of Readiness: Depicts that readiness can and should be assessed at three different levels in the organization (i.e., unit, Area/District/MLC/, and Headquarters).

▪ Six Facets or Readiness: Depicts the six facets or categories (people, training, equipment, support, infrastructure, and information) that are used to measure readiness.

▪ Four Parts of the RMS: Depicts how the RMS will operate. It includes the use of the Readiness Smart Window, an intranet web-based computer display that extracts data from existing databases. It helps decision-makers assess the readiness of units that they are responsible for leading or supporting by answering three key questions: (1) am I ready and for how long; (2) if not, why not; (3) what might I do about it.

The RMS concept was initially tested using Coast Guard multi-mission boat stations, which primarily focus on search and rescue and law enforcement, but support all Coast Guard strategic goals. A proof-of-concept was tested at 12 boat stations starting in July 2000. It contained 23 measures of the six facets of readiness. Based on the favorable results of the proof-of-concept, the RMS is now underway for use by the entire Coast Guard.

Navigation Aid Mix Study (Coast Guard)

The Coast Guard maintains short-range (mostly visual) and long-range (radio) aids to navigation that guide mariners to safe waters and away from hazards. Recent developments in radio navigation, as well as technological developments in electronic charting and navigation system integration, provide additional services to mariners. As part of its broader research into aid mix and waterway risk management, the Coast Guard Research and Development Center (RDC) developed a web-based survey to help understand how mariners are actually using navigational aids. This survey sought to (1) identify the navigation information required by mariners and how they use aids to navigation to acquire this information and guide their vessels; and (2) reveal ways to manage the Coast Guard’s Aids to Navigation system more effectively, while maintaining an acceptable level of safety, mobility, security, and protection of natural resources.

A pilot test of the survey was conducted of several major maritime population segments: Commercial, Public/Military, and Recreational Vessel operators. The survey questions were designed to gain a better understanding of user preferences for and actual use of navigational aids as a function of operation, visibility, and user group. The results of the pilot study were used to develop findings concerning navigational aid use, an assessment of program service alignment with user needs, and suggestions for improving the overall survey process.

Findings in the study area indicate that mariners use nearly all navigational aids that are available to them, over the range of conditions and areas in which they navigate. Both navigational aid preferences and usage patterns vary with user group, area of operation, and visibility. Global Positioning System (GPS) technology has been widely accepted by all groups in the study area except the Small Port-based Fishing and Charter group. More than half of the operators in this group continue to rely on LORAN as their primary radio aid to navigation. The use of Differential GPS by Large Commercial and Public/Military vessel operators is significant, but is limited to one third or less of the operators in all other user groups. However, mariners in all user groups cited buoys and lighted buoys, in addition to GPS, as their most preferred navigational aid type. As users progress from the open ocean, through the near coastal area to port, there is a general shift in preference from radio aids to mixed preference (combinations of radio and short range aids) to short range aids as the primary source of information. Overall, the assessment revealed no clear areas of outdated or substantially misaligned services.

Strategy for Migrant Interdiction (Coast Guard)

The purpose of this study is to develop a strategic plan for the Coast Guard migrant interdiction mission. The strategic goals, objectives, and organizational foundations of Alien Migrant Interdiction Operations (AMIO) are being identified in the study. The results of the study will form the basis of the new 10-year AMIO strategic plan, tentatively named SOVEREIGN SHORES. A final report will be completed in FY2002.

Assessment of Runway Safety Program (FAA)

The National Blueprint for Runway Safety, issued in October of 2000, committed the Runway Safety Program Office to conducting an annual review of the Runway Safety Program. The first such review was completed in July 2001. The purpose of the review was to ensure that the staff of the Runway Safety Program Office, as well as all those who support them from the various FAA lines of business and the aviation community/industry, are provided with feedback on the results of actions since the inception of the office. The Assessment Team’s findings were based upon their review of existing documentation and the information derived from interview responses and facility visits, including regional offices. Their findings were used to further consider best practices and issues presented in the initial assessment report.

Best Practices are processes, procedures or practices that have been effective and should be more widely adopted, or that others believe should be implemented at other offices/facilities. Most noteworthy among the list are: (1) the increased involvement and participation between Regional Administrators and FAA Headquarters; (2) a multimedia approach to making airport marking information available to users; (3) runway safety initiatives by airport managers, industry and State aviation officials; (4) increased National Air Traffic Controller Association (NATCA) involvement and participation; and (5) shared runway incursion reduction goals (Category A & B) on all supporting organizations’ executive Short Term Initiatives.

Issues are areas within the processes, procedures or practices that are in need of improvement. The Assessment Team’s findings included analyses, conclusions and recommendations. The team’s recommendations included: (1) distribution of the revised Runway Safety Order, to spell out roles and responsibilities for all who have a part in runway incursion reduction efforts; (2) ensuring that efficient lines of communication and feedback channels remain open; (3) establishment of a national database for runway safety to reduce duplicative data collection efforts; (4) development of a process that ensures the right amount of industry participation in runway safety activities; and (5) development of Standard operating procedures that lay out the processes for conducting runway incursion action team (RIAT) meetings, and tracking action items and feedback. The Office of Runway Safety will use these and other findings as it focuses upon improved performance and measurable enhancement of runway safety surface operations.

Implementation of FAA Core Compensation Plan (FAA)

Section 347 of the Department of Transportation Appropriations Act of 1996 granted the FAA the authority to develop innovative and flexible Human Resource Management (HRM) systems to support the accomplishment of the agency’s mission and meet customer expectations. The FAA took advantage of this opportunity and designed a new compensation system that is tailored specifically to FAA’s competitive environment, its values, and organizational objectives. In June 1998, the agency piloted the Core Compensation Plan in the segment of the work force reporting to the Associate Administrator for Research and Acquisitions (ARA). The Plan was later broadened to other non-bargaining FAA organizations in April 2000.

An internal corporate evaluation was conducted to examine the implementation of the Core Compensation Plan’s components in ARA and the expansion of the Plan in other FAA organizations. Specifically, the evaluation focused on (1) compensation baseline measures, (2) manager and employee perceptions and (3) the implementation of the Plan’s components in ARA -- e.g. Organizational Success Increases (OSI), Superior Contribution Increases (SCI), Job Documentation, and Pay Setting Decision Tools. Data for the evaluation came from multiple methods and sources including document reviews, FAA’s Consolidated Personnel Management Information System (CPMIS), structured and informal interviews, employee/manager surveys, review of pay setting decision-support tools, and review of training evaluation results. Information gathered through these methods were analyzed and the following key findings were identified based on convergent information from the evaluation:

▪ The ARA pilot provided valuable lessons learned for broader implementation of the Core Plan in the agency. For example, the pilot highlighted the importance of having in place, an organizational performance plan with measurable outcomes and an effective performance management system creating a clear line-of-sight to the goals of the agency and supporting better mission accomplishment.

▪ Strong leadership commitment, management accountability, and employee involvement are required to improve organizational readiness and to build advocacy for acceptance of the new compensation system.

▪ FAA managers and HR professionals are faced with the challenge of learning and performing new roles and responsibilities under the Core Compensation Plan to manage more effectively the agency’s human resources.

▪ FAA employees are willing to embrace and participate in change initiatives supporting a new performance-based FAA culture.

▪ Continued implementation for a longer period of time and across a broader segment of the FAA workforce will provide more information to determine the impacts of the new compensation system and whether the intended compensation objectives are met.

The evaluation concluded that the agency should persevere with the implementation of the Core Compensation Plan and its components and develop appropriate interventions to address manager and workforce readiness issues. Future assessment and evaluation of the new pay system implementation and outcomes will be very important to determine if long-term objectives are achieved. Ultimately, the success of the Core Compensation Plan will be driven by commitment from top leadership, management accountability and responsibility, acceptance from employees, effective communication and training, and integration of supporting HRM systems.

Accuracy and Timeliness of Procurement Data in the FAA’s ACQUIRE System (FAA)

The FAA conducted an evaluation of fiscal year 2000 procurement data in its ACQUIRE system to determine the accuracy and timeliness of information reported to the Office of the Secretary of Transportation (OST). The evaluation objectives were to determine whether reports using data in the system were at least 95 percent accurate and whether correct contracting data was entered into the system within 30 days of contract award. To make this determination, the evaluation team reviewed a statistically significant random sample of procurement transactions that represented the total contract population in each of the agency’s 12 regions/centers.

The evaluation team concluded that on a consolidated basis, 92 percent of the procurement data reported to OST were accurate based on a comparison of the ACQUIRE data download and contract file documentation. On a regional basis, the accuracy percentage ranged from 88.3 percent (Headquarters) to 95.2 percent (Central Region). Based on contract file documentation, the evaluation team could not determine the accuracy of 4.5 percent of the consolidated procurement data. While the consolidated results did not reach the 95 percent accuracy requirement, there were mitigating factors. The evaluation team could not determine the accuracy of certain data elements based on information in the contract file and did not have the time or resources to take the additional steps that would have been necessary to validate this information, such as contacting suppliers directly. Also, the evaluation team could not confirm the accuracy of 2 percent and 1.3 percent of the data elements for the Aeronautical Center and the Southwest Region, respectively, because a critical data element had not been provided in the ACQUIRE data download received from the Office of Acquisitions. The evaluation team was not aware of this oversight until after the fieldwork was completed. It is possible that the consolidated results would have reached the 95 percent accuracy requirement if all data elements in the sample had been validated.

The evaluation team also concluded that on a consolidated basis, 79.2 percent of the ACQUIRE procurements were timely (i.e., entered in the ACQUIRE system within 30 days after contract award) based on a comparison of the system-generated Reserved/Approved Date and signed legal documentation in the contract file. On a regional basis, timeliness ranged from 59.7 percent (Western Pacific Region) to 93.8 percent (Aeronautical Center). The evaluation team could not validate the timeliness of 6.6 percent of the consolidated ACQUIRE procurements because the contract file did not include documentation indicating when the contracting officer signed the contract or modification. There was a lot of confusion in the regions/centers regarding timeliness because the 30-day timeliness requirement had not been communicated to the regions/centers. In addition, the FAA’s procurement guidance did not include timeliness criteria. As a result, the standard operating procedures for when data was to be entered in ACQUIRE varied widely from region to region. Also, certain real estate and utility procurements did not lend themselves to the 30-day timeliness requirement.

The evaluation included eight recommendations for improving the accuracy and timeliness of procurement data reported to OST. These recommendations included conducting periodic quality assurance reviews, expanding exception reports, modifying ACQUIRE guidance, addressing inconsistencies in ACQUIRE guidance, setting requirements for contract file documentation and vendor file maintenance, incorporating ACQUIRE guidance into the FAA’s Acquisition System Toolset, and providing training that includes “lessons learned” from the evaluation. The FAA’s Director of Acquisitions agreed with the recommendations and has taken action to implement the recommendations.

Switching Operations Facility Analysis (FRA)

The Federal Railroad Administration’s (FRA’s) Office of Safety sponsored a Switching Operations Fatalities Analysis Working Group to review fatal incidents and develop recommendations for reducing fatalities in switching operations. The working group included representatives from the FRA, Association of American Railroads, United Transportation Union, Brotherhood of Locomotive Engineers, and American Short Line and Regional Railroad Association. Its charter was to “conduct a detailed fact-finding review and analysis of these incidents to determine whether trends or patterns can be found, identify best practices, and, if possible, formulate recommendations for the entire industry based on its findings.”

The major findings of the working group were:

▪ The occurrence of fatalities in switching yards did not decrease over the period under investigation (January 1992 through July 1998).

▪ Fatalities are not often the result of a single precipitating cause. Almost always, they are a result of multiple possible contributing factors.

▪ Although a great deal of data was reviewed with regard to time, location, number of crew on duty, and several other possible contributing factors, none of the data could be interpreted reliably because there was not sufficient exposure data. Better exposure data are needed to understand the frequency of occurrence of various conditions in the absence of a fatality.

▪ Despite the voluminous amount of detail available and the quality of each technical summary, there were still information gaps in fatality reports that had originally been collected by the FRA. A much broader range of information would greatly improve the ability to interpret possible contributing factors.

The working group generated two sets of specific recommendations. First, based on the data reviewed, a set of five recommendations was made to improve the safety of switching operations. From these safety recommendations, the so-called five “LIFESAVERS” program was developed. Second, a series of recommendations were developed to improve the methodologies used by the FRA and the industry to report employee fatalities, with particular emphasis on improving data collection.

Selected Safety Initiatives (FHWA)

Each State is required to develop and implement, on a continuing basis, a highway safety improvement program (HSIP) that has the overall objective of reducing the number and severity of crashes and decreasing the potential for crashes on all highways. The requirements for a highway safety improvement program are to include components for planning, implementation, and evaluation of safety programs and projects. These projects are to be developed by the States and approved by the FHWA.

The FHWA Safety Core Business Unit, in conjunction with the Office of Corporate Management, conducted a program review of the HSIP in six States--Delaware, Oregon, Connecticut, Florida, Ohio and Iowa-- between February and April 2001. The primary objective of this review was to document best practices of the HSIP by highlighting those practices that are uniquely best in each State and sharing this information with the safety community.

The review team found numerous, noteworthy activities being carried out by the States. Among the general best practices identified in a number of the States visited were the following:

▪ Having safety as a major goal of the agency, with commitment at the highest levels. For example, in several of the States visited, the Governors played an active role in promoting safety.

▪ Having a good multi-disciplinary safety management process in place, with a strong component for roadway safety.

▪ Emphasizing safety on all projects.

▪ Having a Safety Engineer in the State DOT as the focal point for the HSIP.

▪ For the larger States with Regional structures, having Safety Coordinators in each Region.

▪ Having community-based Traffic Safety Programs.

▪ Assisting localities.

▪ Using current technologies (e.g. Photologging, GIS, and web-based systems)

Interim Report on U.S. Large Truck Crash Causation Study (FMCSA)

As required by the Motor Carrier Safety Improvement Act of 1999, the Federal Motor Carrier Safety Administration (FMCSA) is conducting a multi-year large truck crash causation project in collaboration with NHTSA. This project is the first national effort to collect crash data for the purpose of determining the causes of, or factors contributing to, large truck crashes so that FMCSA and others can implement countermeasures to reduce the occurrence and severity of these crashes. During 2001, FMCSA prepared an interim report describing the approach being used in the large truck crash causation project and issues that were identified during the implementation of the pilot phase of the project.

The large truck crash causation project involves the collection of data from crash scenes about the drivers, vehicles, roadway, and environment. Teams comprised of State truck inspectors and crash researchers from NHTSA’s National Automotive Sampling System (NASS) are dispatched to crash sites. The NASS researchers collect physical data about the accident and interview the drivers (or their surrogates) and witnesses, and the State truck inspectors conduct North American Standard Level 1 truck and truck driver inspections. Using statistical association, these data are used to determine the factors connected with crashes and how much each factor contributes to the increased risk of crashes.

In May 2000, pilot data collection and analysis efforts on crashes commenced at four study sites: Philadelphia, Chicago, Prince Georges and Charles counties in Maryland, and La Paz and Yuma counties in Arizona. The major challenges identified during the pilot were: (1) training police dispatchers to recognize crashes of interest to the large truck crash causation project and notifying the investigating teams; (2) finding, training, and retaining staff to serve on the investigation teams; (3) acclimating NHTSA’s NASS researchers and State truck inspectors to a new crash study approach; and (4) maintaining a new level of cooperation from hundreds of local police jurisdictions.

Following the pilot, the large truck crash causation project was rolled out nationwide. Investigation teams have been dispatched to 24 locations in 17 States. Data collection and analysis efforts will continue through 2003, with a report on preliminary findings to be issued by the end of 2002. The study will conclude in 2004 with the release of a Final Report of the results of the study.

Safety Data Quality Improvement (BTS)

The Safety Data Initiative began in response to DOT’s 1999 National Transportation Safety Conference, where stakeholders identified better data collection and reporting across all jurisdictions as one of the top priorities to improve safety. In September 2000, the Bureau of Transportation Statistics (BTS) drafted the Safety Data Action Plan (the Plan) under the direction of the DOT Safety Council. The plan recommended ten cross-modal projects to address specific data quality problems and data gaps.

Under the guidance of BTS, working groups have done background research aimed at the development of implementation plans for the projects outlined in the Safety Data Action Plan. The major goal of the project is to provide DOT with a new level of data quality, sufficient to identify, quantify, and minimize the risk factors in U.S. travel.

The Safety Data Action Plan identified ten projects to focus on addressing specific shortcomings in current data collection and data quality within the various DOT database systems. These ten projects were organized into four broad areas: (1) improving the quality, comparability, and timeliness of existing data; (2) collecting better data on accident circumstances, precursors, and leading indicators; (3) expanding the use of technology in data capture; and (4) improving analytical capability. To date, four of the ten projects have been completed.

▪ Develop common criteria for reporting injuries and deaths. Transportation-related deaths and injuries are key measures of interest in the Department's Strategic Plan. Currently, definitions and reporting criteria for injuries and deaths are inconsistent across the modes. This variety of criteria makes aggregate counts of transportation deaths and injuries misleading and cannot be used reliably to present trends or make comparisons across modes. The objective of this project was to develop recommendations for a standard for coding common data elements and injuries across databases. This would ensure uniformity in injury event definitions and reporting criteria across modes and include sufficient mechanistic causal information for development of intervention strategies.

Recommendations have been developed to promote commonality among modes and improve the quality and utility of mechanistic incident and injury data for development of preventive strategies. Common definitions have been proposed for a reportable event, a fatality, and an injury. Additional recommendations include: (1) the development of an injury reporting system including mode-specific codes when necessary; (2) sampling as a way of limiting the reporting burden when a large numbers of incidents occur (i.e., highways), and (3) exploring opportunities for linking transportation databases to hospital databases, State or territory vital statistics, and other medical databases.

▪ Develop common denominators for safety measures. Each of the modes uses a different set of denominators for evaluating changes in safety risk. This variety makes aggregation or comparison unworkable, limits researchers' ability to perform comparative risk analysis across modes, and limits the Department’s ability to reliably allocate resources across modes. The goal of this project was to define a set of denominators that can be used to characterize transportation safety in a comparable way for comparable circumstances -- i.e., to allow the risk of recreational boating to be compared to the risk of recreational flying or recreational driving.

Specific exposure measures, suitable for cross modal comparison, have been identified based on the particular transportation activity. These activities include: passenger transportation, freight transportation, recreational use, and occupational activities. Detailed recommendations have been made to ensure that appropriate exposure data are collected within each mode.

▪ Develop common data on accident circumstances. Currently, there is no consistency in the collection of data on accident circumstances across modes. This inconsistency inhibits the sharing of information and fails to take advantage of advances in different modes. The objective of this project is to expand our understanding of data needed to identify causes of accidents, and to facilitate and support statistical analysis of data across a wide variety of accidents - even in different modes.

Using research on classifying accident circumstances, a prototype set of data elements has been developed. The use of new technologies in collection efforts, such as event data recorders, is being explored as a way to generate consistent accident information across various environments.

▪ Explore options for using technology in data collection. Better use of technology could greatly facilitate more timely data collection and improve data quality, since the likelihood of human error will decrease, and it may also be more cost-effective. The objective of this project is to explore options for using new technology to improve data collection and reporting.

The main focus of this report is identifying technologies that can be used across modes and can significantly improve the timeliness, accuracy, and coverage of DOT data collection. Recommendations for further research include pilot studies in the following three areas: Electronic Identification/Security Smart Cards, Operator Performance Monitoring (alerts operators to lapses in concentration), and Hands-Free Operation (wearable computers for data collection).

Alternative Dispute Resolution (Office of the Secretary)

Alternative Dispute Resolution (ADR) describes a variety of collaborative and voluntary problem-solving processes that usually involve a neutral third party. The purpose of the ADR Program Evaluation was to: (1) collect data on how the Department is using ADR; (2) examine and assess whether DOT is resolving Equal Employment Opportunity/Equal Opportunity (EEO/EO) disputes in a cost-effective, mutually acceptable manner compared to the traditional processes for resolving these disputes; (3) identify best practices for resolving EEO/EO disputes through ADR; and (4) make recommendations for improving dispute resolution in the Department.

The data collection showed that the Department is using ADR to resolve disputes in a variety of areas, both where formal ADR programs exist and on an ad hoc basis. The areas in which ADR activity was reported included: procurement, environmental justice, rulemaking, workplace/personnel, and in discussions with regulated entities.

The main emphasis of the evaluation was to examine the three different EEO/EO ADR programs within the Department. The Departmental Office of Civil Rights is responsible for a Department-wide Sharing Neutrals Program. The FAA has a nationwide mediation program, and the U.S. Coast Guard (USCG) has established policies and procedures on the use of ADR to resolve discrimination complaints, has trained employees to act as mediators, and has a pilot program in place.

The evaluation found that all three programs were established in accordance with principles outlined by the Equal Employment Opportunity Commission and the Federal ADR Council. Although customer service appears high, the programs are not tracking whether they are meeting all of their intended goals, such as time and cost savings, early resolution of the complaint, and more hospitable work environment. To leverage resources and avoid redundancy of efforts, the evaluation recommended that the three programs work together to collect and track data on shared goals, use the same customer satisfaction survey, and coordinate budget requests.

With regard to the effectiveness of the three programs, the evaluation recommended extensive training and practical experience for neutrals. The evaluation recommended that the USCG assess the results of its pilot program before expanding it. The evaluation also recommended that all Departmental employees receive awareness briefings on the EEO/EO mediation programs. With regard to dispute resolution generally, the evaluation recommended that prior to the implementation of any ADR programs a needs assessment should be conducted, and the program should begin with a pilot program that is evaluated prior to program expansion.

Finally, the evaluation noted that some matters are taken to the EEO/EO mediation programs when miscommunication, rather than discrimination, is the cause of the conflict. The evaluation suggests further evaluation to determine whether there should be a Department-wide human resource strategy to make ADR available for workplace disputes. Such an effort would be consistent with the Department’s Organizational Excellence goal of improving employee satisfaction and effectiveness.

Maritime Security Program (MSP) and Volunteer Intermodal Sealift (VISA) Agreement (MARAD)

MARAD evaluated the Maritime Security Program (MSP) and the Voluntary Intermodal Sealift Agreement program (VISA) to determine whether the programs are helping to achieve the Department of Transportation (DOT) national security strategic goal. In particular, the evaluation assessed the impact of the MSP and VISA in achieving the DOT outcome of increasing the capability of the transportation system to meet national defense needs.

The purpose of the Maritime Security Program (MSP) is to ensure that an active U.S.-flag merchant fleet of militarily useful general cargo vessels in international trade, and the trained personnel needed to operate both active commercial and Government-owned reserve vessels, are available to meet U.S. economic and national security requirements. The MSP was implemented in response to the decline of the U.S.-flag general cargo fleet and concern over whether it could continue to serve adequately both the economic and national security objectives of U.S. maritime policy in the future.

VISA provides contractual arrangements with private U.S.-flag ship operators to make intermodal transportation services available in times of national emergency. VISA provides DOD with assured access to emergency intermodal sealift capacity that complements DOD’s organic sealift capabilities in a coordinated, seamless transition from peace to war. The MSP and VISA programs have not been tested in an actual mobilization because there has not been a major contingency since they were created.

To determine the impact of the MSP and VISA programs, MARAD compared the relevant outcomes of the members of two similarly situated groups of U.S.-flag privately owned general cargo ships in foreign trade. The program group is defined as U.S.-flag privately owned general cargo vessels that participated in MSP during 1996-2000. Similarly situated ships that were never in MSP represent the control group. By comparing changes in the two groups over time, differences between them could be attributed to the influences of the MSP. From the available quantitative and qualitative information, MARAD determined the effects and impact of MSP and VISA on the U.S.-flag fleet and reached the following conclusions:

▪ MSP and VISA are operating in accordance with the statutory intent of the Maritime Security Act to ensure the availability of vessels to meet U.S. economic and national security objectives. These programs also contribute to the achievement of DOT and MARAD national security goals.

▪ MSP and VISA provide a significantly improved mechanism over previous programs to obtain sealift to meet mobility requirements. These improvements include pre-planned committed capacity, which facilitates rapid deployment to meet DOD delivery requirements; commercial intermodal resources for cost-effective door-to-door service; and flexibility for operators to maintain commercial routing during contingencies.

▪ The overall health (as measured by average age and cargo capacity) of the group of U.S.-flag ships receiving MSP payments has improved compared to the control (non-MSP) group. By comparing these two similarly situated groups, it was found that the MSP payment was the most significant factor affecting the health of the U.S.-flag dry cargo fleet. If the MSP payment continues to decrease in value, or is eliminated, there are no apparent external factors that would halt the overall decline in the health of the U.S.-flag foreign trade fleet.

Federally Funded Maritime Education and Training (MARAD)

During FY 2001, MARAD continued its evaluation of the impact of federally funded officer education programs on the achievement of DOT national security goals. The program evaluation, which is also a congressionally required report, will be submitted to Congress in FY 2002.

Program Evaluation	Strategic Goals					OE	Methodology	Scope
Program Evaluation	S	M	EG	E	HS	OE	Methodology	Scope
Strategy for Migrant Interdiction Program (USCG)					X		Management Study	Evaluate inter-agency strategy for migrant interdiction
Recreational Boating Fatality Data Capture (USCG)	X					X	Management Study	Evaluate data collection and analysis of boating fatalities
Maritime Safety Program Impact (USCG)	X						Combination	Evaluate the impact of safety strategies on maritime fatalities, injuries, and property (Interim Report)
Great Lakes Icebreaking (USCG)		X	X				Combination	Evaluate the impact of Great Lakes ice-breaking on mobility of goods and customer requirements
Airport Noise (FAA)				X			Longitudinal	Evaluate effectiveness of the Airport Improvement Program’s noise set-aside in reducing the noise-impacted population around airports
Aviation Safety Program Instructional Methodologies (FAA)						X	Management Study

Summary of FY 2001 Program Evaluations:

Federally Funded Maritime Education and Training (MARAD)

Schedule for FY 2002 Program Evaluations