Performance Data and Performance Measurement

Performance measurement is dependent on the availability of useful data. Useful data will indicate level of performance and progress toward organizational goals. All data are imperfect in some fashion. Pursuing “perfect” data, however, may consume public resources without creating appreciable value. For this reason, there must be an approach that provides sufficient accuracy and timeliness but at a reasonable cost. This section of the Performance Plan/Performance Report provides information on how DOT reports on performance, verifies and validates data, assesses limitations of the data, and plans for improving DOT’s data.


PerformanceData Completeness and Reliability

In an attempt to bring consistency and quality to its performance reporting, DOT has implemented some general rules regarding the data it uses and how it is evaluated.

Annual data – Whenever available, the data in this document are reported on a Federal Government fiscal year basis. However, there are instances where this is not possible so calendar year data are used instead.  This often occurs when data are collected and reported to DOT by external sources and a calendar year reporting requirement is specified in the implementing regulation. The reporting timeframe (FY or CY) for each measure is included in the Data Details in Appendix I.

Annual results – If available, the results for the most recent year in the Report are listed as “Actual” in the Performance Goals & Results box for each performance measure. However, given the March deadline for submission of the Performance Report, quite often data have not been compiled and finalized for the entire year. When this occurs and an actual value is not available for the current year, either an estimate or projection is provided instead. In general, estimates are based on partial year data that are extrapolated to cover a full 12-month period. For example, if six months of data are available, they will be compared to prior years for the same six-month period to determine any variation from past levels. Historical trend information, supplemented by program expertise, will then be applied to estimate the remaining six months of performance. The result will be identified as a “preliminary estimate” in the Report. If partial year data are not available, then past trend information will be analyzed and supplemented by program knowledge to develop a projected value for the annual performance measure. The result will be identified as a “projection” in the Report. As data are finalized, the projections and preliminary estimates will be replaced by actual results.  Results may be amended as errors and omissions are identified in the data verification process, because updated information is provided by the reporting sources, or because of legal or other action that changes a previously reported value. For example, updated pipeline spill reports may change the status of a previously reported value used in performance measurement.

In measuring progress toward the majority of performance goals, DOT is moving to a system of monthly performance measurements.  This will make it much easier to internally gauge periodic progress toward goals as the year progresses, and will enable more timely performance reporting after the years’ end. 

Completeness of Data – As described above, actual data and “preliminary estimates” incorporate complete or partial data from 2001.  Results listed as “projections” are not based on data from 2001, but on trend data from prior years. 

Reliability of Measurement Data – Because performance results in a given year are influenced by multiple factors, some of which are beyond DOT’s control, and some of which are due to random chance, there may be considerable variation from year to year. (See discussion in Appendix I.) A better “picture” of performance may be gained by looking at results over time to determine if there is a trend. Therefore, graphs are provided for each measure showing trend lines back to 1990, or as many years as possible if data are not available back to 1990. Additionally, a table is included at the beginning of each strategic goal section giving the available data from 1995 through 2001 for measures with performance goals specified for 2001.

Verifying & Validating Performance Measures

Integral to performance measurement is understanding data limitations, addressing these limitations where necessary and cost-effective, and acknowledging those that remain when interpreting results. This section on verification and validation provides a DOT-wide overview of our plan for assessing the quality of the data DOT uses to measure its performance, and follows the GAO definitions for verification and validation:

“Verification is the assessment of data completeness, accuracy, consistency, timeliness, and related quality control practices.”

“Validation is the assessment of whether data are appropriate for the performance measure.”

Virtually all data have errors. In Appendix I we have provided the following information about the data used for each performance measure: source of the data, limitations of the data, observations about the quality of the data, work planned or ongoing to improve data quality, and any known biases.

Additionally, we have compiled Source and Accuracy Statements for each of the DOT data programs used in this report, which can be found at www.bts.gov/statpol/SAcompendium.html. The Source and Accuracy Statements give more detail on the methods used to collect the data, sources of variation and bias in the data, and methods used to verify and validate the data.

By validating data used in the DOT performance plan, we are ensuring that those data are reflective of the phenomena they purport to measure. The Office of the DOT Inspector General (OIG) plans to selectively verify and validate performance measurement data each year. When pertinent to the conduct of ongoing projects, OIG will also assess performance measures to determine their appropriateness for measuring progress toward stated goals. These assessments may lead to changes in the goals, improvements to or additions of data collection systems, or both.

Assessing and, where possible, eliminating sources of error in DOT data collection programs has always been an important task for data program managers. As a part of their ongoing work, managers of Departmental data programs use quality control techniques, such as flowcharting the data collection process, to identify where errors can be introduced into the data collection system. Program managers also use computerized edit checks and range checks to minimize errors that may be introduced into the data of their respective programs. In addition, quality measurement techniques are employed to measure the effects of unanticipated errors. These include verification of data collection and coding, as well as coverage, response and non-response error studies to measure the extent of human error affecting the data. As sources of error are identified, steps are initiated to improve the data collection process.

The data used in measuring performance come from a wide variety of sources. Much of the data originates from sources outside the Department and, therefore, outside the direct control of the Department. The data often come from administrative records or from sample surveys. While DOT may not have a strong voice in improving the quality of outside data, the Department takes all available information about the limitations and known biases in outside data into account when using the data.

The myriad data sources make the task of assessing and, where possible, eliminating error a challenging one for DOT. Different data systems contain different types of errors. For example, data from administrative records systems may have missing or incorrect records, and data from sample surveys will contain sampling error.

Several measures (particularly in safety) require aggregation across transportation modes. This can be particularly problematic because of the use of different definitions in different transportation modes. Also, data from outside the Department may have unknown error properties.

To help the operating administrations address these issues, the Bureau of Transportation Statistics (BTS) is developing a statistical policy framework where the operating administrations will work together to identify and implement the current statistical “best practices” in all aspects of their data collection programs. This project is consistent with the data capacity discussions found in the DOT Strategic Plan.

In 2001, a DOT intermodal working group addressing DOT data quality issues continued to:

          develop Departmental statistical standards;

      update Source and Accuracy Statements for all DOT data programs to document limitations and known errors and biases;

          improve quality assurance procedures;

          evaluate sampling and non-sampling error; and;

      develop common definitions for data across modes.

BTS's statistical staff is consulting with the DOT operating administrations’ data program managers to assist in data evaluation and validation, documenting data sources, and determining the reliability of performance measurement estimates.

Departmental data systems managers use these data verification methods:

      Comparisons with previous data from the same source.

      Comparisons with another reliable source of the same type of data within DOT for the same time period.

      Comparisons with another reliable source of the same type of data within DOT for a previous time period.

      Comparisons with another reliable source of the same type of data outside DOT for the same time period.

      Comparisons with another reliable source of the same type of data outside DOT for a previous time period.

In addition to computerized edit checks and clerical review procedures to look for outliers, duplicate records, and data inconsistencies, data managers also verify data quality at each step of the data collection process using these procedures:

      Re-collecting/re-interviewing all (or a sample of) records and reconciling with the original collection. (This applies to census or sample survey data collections from administrative records, organizations, or individuals.)

      Conducting 100 percent (or a sample of) data re-coding and reconciliation to assess and correct coding errors.

      Conducting 100 percent (or a sample of) data re-entry and reconciliation to assess and correct data entry errors.

The American Travel Survey’s re-interview program, in which a sample of households were re-contacted and differences reconciled, is an example of a verification system within a data collection program.

Data Limitations in Performance Measures

DOT Data Source Limitations – Timeliness is the most significant limitation for DOT performance measurement data. Some DOT data are not collected annually. For example, the National Household Travel Survey and the Commodity Flow Survey each collect data every five years.  Data that are collected each year (or more frequently) require time to analyze, confirm and report results. For example, Highway Performance Monitoring System vehicle-miles traveled (VMT) data require several months of post-collection processing, making final results unavailable for this performance report.

Other performance measurement data limitations can be found in the previously mentioned Source and Accuracy Statements for DOT data programs. These statements contain descriptions of data collection program design, estimates of sampling error (if applicable), and discussions of non-sampling errors. Non-sampling errors include under-coverage, item and unit non-response, interviewer and respondent response error, processing error, and errors made in data analysis.

As part of its mandate in the Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA), the Transportation Equity Act for the 21st Century (TEA-21), and its plans for a statistical policy framework in the Department, BTS is working on a program of research, technical assistance, and data quality enhancement to support the continued improvement of data programs in DOT. This will help data program managers throughout DOT improve data quality and better document known data limitations. BTS also assists operating administrations with data collection and documentation.

Many of DOT’s internal data programs rely on State DOTs to collect reliable statistics within cost constraints. While we work closely with our State DOT partners, we do not have direct control over these data.

External Data Source Limitations – Timeliness is also a significant limitation for external or third-party data. Other limitations of external data are noted in the comments for each performance measure in Appendix I. In some cases, DOT has replaced external data, where little is known about the quality of the data, with internal data. For example, DOT has used estimates of person-miles traveled (PMT) from private organizations, absent any better estimate. The 1995 Nationwide Personal Transportation Survey and American Travel Survey give DOT data with known error properties that allow a better estimate of PMT.

Our Data Needs

The DOT Strategic Plan 2000 – 2005 identifies data needs for each of the Department’s strategic goals. They include:

Safety – DOT is undertaking major efforts over the next several years to improve safety data. Safety has always been our primary strategic goal, and in 1999 DOT created a Safety Data Action Plan to better organize data improvement efforts. BTS will lead efforts to: 1) develop common criteria for reporting injuries and deaths; 2) develop common data on accident circumstances; 3) improve data quality; 4) develop better data on accident precursors; 5) expand the collection of near-miss data to all transportation modes; 6) develop a variety of common denominators for safety measures; 7) advance the timeliness of safety data; 8) link safety data with other data; 9) explore options for using technology in data collection; and 10) expand, improve and coordinate safety data analysis.

Homeland Security – Existing performance data sources are generally good, but DOT will collect data to better understand the transportation system’s vulnerability to intentional acts of disruption or destruction.

Mobility – All mobility outcomes present complex measurement issues. Accordingly, DOT will: 1) develop ways of measuring user transportation cost, time, and reliability with time-series data; 2) develop better approaches for measuring access; 3) develop straightforward measures of congestion and its costs; 4) produce more timely and comprehensive data on the condition and use of the transportation system; and 5) develop a more complete understanding of variables influencing travel behavior.

Economic Growth – DOT needs aggregate data for measuring the productivity, effectiveness and efficiency of the U.S. transportation system. We plan to collect, analyze and disseminate data and information that identify critical trends and issues relating to transportation’s nexus to the U.S. economy. DOT will: 1) develop a means of measuring transportation cost, time, and reliability – at an aggregate level – with time-series data; 2) develop a comprehensive measure of the transportation capital stock; 3) improve our view of changes in the transportation workforce; 4) develop better measures of productivity in the transportation sector, and other issues concerning use of Producer Price Indices; and 5) develop a better picture of transportation-related variables influencing U.S. competitiveness in the global economy.

Human and Natural Environment – DOT will: 1) develop comparable and complete data on transportation emissions, noise, hazardous materials releases, and wetlands impacts; 2) improve our understanding of collateral damage to the human natural environment; 3) create better leading indicators for potential environmental issues; and 4) develop a reliable method of measuring the environmental benefits of bicycling and walking.

 


Appendix I – Performance Measures (Detail)

 

Each table includes a description of a performance measure and associated data provided by the agencies in charge of the measure.  The Scope statement gives an overview of the data collection strategy for the underlying data behind the performance measure.  The Source statement identifies the databases used for the measure and their proprietary agencies.  The Limitations statement describes some of the shortcomings of the data in quantifying the particular performance characteristics of interest.  The Statistical Issues statement has comments, provided by the Bureau of Transportation Statistics (BTS) and the agency in charge of the measure, that discuss variability of the measure and other points.  The Verification and Validation statement indicates steps taken by the proprietary agencies to address data quality issues.

DOT feels strongly that full compliance with the Government Performance and Results Act requires impartial reporting of the statistical uncertainty associated with numerical performance measures.  A portion of this uncertainty is related to the methodology used to calculate the performance measure and the accuracy of the underlying data.  For example, the use of samples introduces uncertainty because estimates are used in lieu of actual counts.  Also, there may be errors in the data collected.  However, there are many other sources of variation (e.g., nonsampling errors, climate effects, new technology) and they are often difficult to quantify.  Nonetheless, a combination of past data and expert judgment can enable uncertainty statements that are order-of-magnitude correct for even the most difficult problems.

The standard error of a performance measure indicates the likely size of the chance variation in the reported number.  It incorporates both the effects of measurement error, survey error, and so forth, as well as the variation that occurs naturally from year to year (i.e., even if there were no change in laws, infrastructure conditions, or human behavior, there would still be chance variation in an annual count of fatalities).  DOT success in meeting GPRA goals must be viewed in the context of this background noise.

In many of the following Statistical Issues statements, BTS refers to regression standard error.  This is a modification of the standard error to take into account of linear trends in the recent past.  Such adjustment is generally needed to incorporate consistent trends due to cumulative effects of such things as education programs, changing demographics, the gradual adoption of new technologies, and so forth.  The underlying assumptions are that: over a short time period the trend of the measurement data is linear; for any given year the performance measure values are normally distributed; and the standard deviation is the same for all years.  We believe that these assumptions lead to a conservative estimate of variability.

The regression standard error is an estimate, calculated from the annual performance results, of this common standard deviation.  It may be used in the same way as a regular standard error to set confidence intervals or describe uncertainty.  For the purposes of performance measurement, it may be considered a rough approximation of the annual variability in a measure, and it will include the affects of program initiatives, influences beyond the control of DOT (e.g., weather, petroleum prices, etc.), random chance, and errors inherent in the data.

For further information about the source and accuracy (S&A) of these data, please refer to the BTS S&A compendium available at

www.bts.gov/statpol/SAcompendium.html

 


Details on DOT Measures of Overall Safety

Transportation Safety                                                                       Page 12

Measures:

1.      Transportation fatalities. (CY)

2.      Fatalities per 100 million passenger-miles. (CY)

3.      Fatalities per 100 million ton-miles of freight. (CY)

4.      Transportation injuries. (CY)

5.      Injuries per 100 million passenger-miles. (CY)

6.      Injuries per 100 million ton-miles of freight. (CY)

7.      Transportation incidents. (CY)

Scope:

This family of measures aggregates fatalities, injuries and incidents across all modes of transportation (air, highway, railroad, transit, waterborne and pipeline).

The fatality and injury rates per 100 million passenger-miles exclude pipeline fatalities and injuries due to minimal interaction with passenger miles.  Highway-rail grade crossing fatalities and injuries are not counted since they are included in data for highways.

The fatality and injury rates per 100 million ton-miles of freight include fatalities and injuries from large truck, rail, waterborne and pipeline transportation.  Highway-rail grade crossing fatalities and injuries are also included since these involve freight transportation-related fatalities and injuries that would not otherwise be counted.  Ton-miles of freight covers intercity truck, rail, water and oil pipeline transportation.  Aviation fatalities, injuries and ton-miles are excluded because the fatality and injury data are not separated from passenger air carriers. Transportation incidents include crashes, system failures, spills, releases, and other accidents of a similar nature.

Source:

The data for these measures are obtained from National Transportation Statistics published annually by the Bureau of Transportation Statistics.  Information is taken from the following tables: Transportation Fatalities by Mode; Injured Persons by Transportation Mode; U.S. Passenger-Miles (Millions); U.S. Ton-Miles of Freight (Millions); and Transportation Accidents by Mode.  The one exception is the data on large truck fatalities and injuries used for calculating fatality and injury rates per 100 million ton-miles of freight are obtained from the Federal Motor Carrier Safety Administration.

Limitations:

Double counting of fatalities and injuries may occur when an accident involves more than one mode of transportation.  Differing definitions of injuries or transportation-related fatalities makes comparison across modes of transportation problematic.  Highway injuries and incidents are obtained from a nationally representative probability sample and are estimates, while the totals for other modes of transportation are actual counts.  The highway estimates are based on crashes where a police accident report was completed and the crash resulted in property damage, injury or death.  Accidents that were not reported to the police or did not result in property damage are not included.  Highway passenger miles are calculated by multiplying vehicle-miles of travel (VMT) by the average number of occupants for each vehicle type.  VMT is based on a nation-wide sample of vehicle travel.  The average number of vehicle occupants comes from survey information.  Therefore, vehicle passenger miles is an estimate, whereas passenger-miles for other modes of transportation are calculated based on actual passenger counts and recorded trip lengths. 

Statistical Issues:

All fatality totals, and the injury and incident numbers where actual counts are recorded, are relatively accurate.  Any double counting or omissions are expected to be fairly small.  The primary source of uncertainty in these measures comes from sampling and survey errors related to estimation of highway injuries, incidents, VMT and vehicle occupancy.  Based on data from 1994-2000, the annual variations in the transportation safety measures are as follows:  the regression standard error for the  number of transportation fatalities is 0.5 thousand.  For fatality rates by passenger-miles and ton-miles, it is 0.010 and 0.007, respectively.  For number of injuries, it is 0.10 million.  For injury rates by passenger-miles and ton-miles, it is 2.50 and 0.24, respectively.  For incidents, it is 0.16 million.

Verification & Validation:

BTS compiles the data for the National Transportation Statistics from information it gathers directly in its own data systems (e.g. airlines information), information published by other sources (e.g. FHWA highway statistics), or by personal communication with the agency/organization responsible for collecting the data.  Each data source conducts error checks and monitors the accuracy of its data.  Most of these sources and their verification and validation procedures are described in subsequent data details in this report for performance measures of individual modes of transportation.

Comment:

While caution should be exercised in comparing fatalities, injuries and incidents between modes of transportation due to differences in definitions and calculations, the aggregation of these values still provides useful information.  Because the methodology for calculating these measures has remained consistent over the years, the trend information should provide a reasonably accurate picture of results. 

 

Highway fatality rate                                                                   Page 15

Measure:

Fatalities per 100 million vehicle-miles-traveled (VMT) (CY)

Scope:

The number of fatalities is the total number of motor vehicle traffic fatalities which occur on public roadways within the 50 states and Washington, D.C.

Vehicle Miles of Travel (VMT) represent the total number of vehicle miles traveled by motor vehicles on public roadways within the 50 states and Washington, D.C.

Source:

Motor vehicle traffic fatality data are obtained from NHTSA’s Fatality Analysis Reporting System (FARS).  To be included in FARS, a motor vehicle traffic crash must result in the death of a vehicle occupant or a non-motorist within 30 days of the crash.  The FARS database is based on police crash reports and other state data. FARS includes fatalities on all roadways open to the public, using the National Highways System classification of roads.  Pedestrian and bicycle fatalities that occur on public highways, but do not involve a motor vehicle, are not recorded in FARS.  However, they constitute only a small number of fatalities.

VMT data are derived from FHWA’s Traffic Volume Trends (TVT), a monthly report based on hourly traffic count data in the Highway Performance Monitoring System (HPMS).  Information is transmitted to NHTSA where it is reviewed for consistency and accuracy before being entered into the system. These data, collected at approximately 4,000 continuous traffic counting locations nationwide, are used to determine the percentage change in traffic for the current month from the same month of the previous year.  The percentage change is applied to the nationwide travel for the same month of the previous year to obtain an estimate of nationwide travel for the current month.  The data are recorded as monthly totals and cumulative yearly totals.

Limitations:

VMT data are subject to sampling errors, whose magnitude depends on how well the locations of the continuous counting locations represent nationwide traffic rates.  HPMS is also subject to estimating differences in the states, even though FHWA works to minimize such differences and differing projections on growth, population, and economic conditions that impact driving behavior.

Statistical Issues:

The primary source of uncertainty in estimating fatality rates is the denominator.  While the estimate of total fatalities used in the numerator is relatively accurate, the estimate of total vehicle miles in the denominator has far more variability.  Based on data from 1994-2000, the annual variation in the fatality rate has a regression standard error of 0.029.

The estimates of the number and percentages of persons killed in motor vehicle traffic crashes during 2001 are preliminary and are based on incomplete data and statistical models.  NHTSA’s first official estimates for 2001, the Early Assessment, are being developed and will be completed in early April 2002.  Differences between the Official Early Assessment estimates and those in this report are to be expected.

Verification & Validation:

Fatality data from FARS are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis.  Quality control procedures are built into annual data collection at 6 and 9 months, and at year’s end.  A study was completed in 1993, looking at samples of FARS cases in 1989 through 1990 to assess the accuracy of data being reported.  VMT data are reviewed by FHWA for consistency and reasonableness.

Comment:

This data program has been in use for many years and is generally accepted for describing safety on the Nation’s highways.  Adjusting raw highway fatalities and injuries by VMT provides a means of portraying the changes in highway fatalities on a constant exposure basis and facilitates year-to-year comparisons.

 

Large truck-related fatalities                                                       Page 15

Measure:

Number and rate (per million commercial VMT) of fatalities in crashes involving large trucks. (CY)

Scope:

The measure includes all fatalities (e.g., drivers and occupants of passenger cars, motorcycles, large trucks, or pedestrians) associated with crashes involving trucks with a gross vehicle weight rating of 10,000 pounds or more. The number of fatalities comes from NHTSA’s Fatality Analysis Reporting System (FARS) data, a census of fatal traffic crashes within the 50 states, Puerto Rico, and Washington, D.C.  The fatal crash rate is the number of fatalities per 100 million vehicle miles of large truck travel (VMT).

Source:

NHTSA’s Fatality Analysis Reporting System (FARS) provides fatality data.  The VMT data are derived from the Federal Highway Administration’s (FHWA) Highway Performance Monitoring System (HPMS).

Limitations:

FARS data elements are modified from year to year to respond to emphasis areas, vehicle fleet changes, and other needs for improvement. Large truck VMT reported to FHWA by each state is based on a sample of road segments and is not a census.  In addition, the methods used to calculate total VMT may vary from state to state. The methods used by the states to estimate the VMT contribution from rural and urban minor collectors are unknown. 

Statistical Issues:

The fatality counts in FARS are generally quite accurate.  The major sources of error are underreporting by some precincts and inconsistent use of the definition of a truck.  Based on 1994-2000 data, the chance variation in a given year has a regression standard error of approximately 157 fatalities.  Because the VMT data provided to FHWA from each state are estimates based on a sample of road segments, the numbers have associated sampling errors.  The methodology used by each of the states to estimate VMT is not known and may introduce additional non-sampling error.  Although states provide VMT estimates on an annual basis, they are only required to update their traffic counts at all sampling sites once every three years.  Thus an annual VMT estimate from a particular state may be based, in part, on data collected during a previous year.  Based on 1994-2000 data, the chance variation in a given year in the number of fatalities per 100 million vehicle miles of large truck travel has a regression standard error of 0.053.

Verification & Validation:

Fatality data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis.  Quality control procedures are built into data collection and data processing.  A study using samples of 1989-1990 FARS cases was completed in 1993 to assess the accuracy of data being reported.  FHWA routinely works with state data providers to modify reported VMT values that do not appear reasonable before incorporating them into its final master file.

Comment:

The FARS data have been around for many years and are generally accepted as a good source for describing fatal crashes on the Nation’s highways.  The large truck VMT data used to calculate fatal crash rates have both sampling and non-sampling (i.e., bias) error associated with it.  The impact of these errors on FMCSA’s estimates of large truck crash rates is considered to be minimal.

 

 

Alcohol related highway fatalities                                               Page 18

Measure:

1.  Alcohol-related fatalities per 100 million vehicle-miles traveled.

2.  Percentage of highway fatalities that are alcohol related. (CY) (2001)

Scope:

The number of fatalities resulting from motor vehicle traffic crashes that are alcohol related and occur on public roadways within the 50 states and Washington, D.C. 

Source:

Motor vehicle traffic fatality data are obtained from NHTSA’s Fatality Analysis Reporting System (FARS). FARS is a census of fatal motor vehicle traffic crashes within the 50 states, Puerto Rico, and Washington, D.C.  To be included in FARS, a crash must result in the death of a vehicle occupant or a non-motorist within 30 days of the crash.  The FARS data are based on police crash reports and other state data. FARS includes fatalities on all roadways open to the public, using the National Highways System classification of roads.  Pedestrian and bicycle fatalities that occur on public highways, but do not involve a motor vehicle, are not recorded in FARS.  However, they constitute only a small number of fatalities.  A fatal motor vehicle traffic crash is alcohol-related if either a driver or a non-motorist (such as a pedestrian) involved in the crash had a measured or estimated blood alcohol concentration (BAC) of 0.01 grams per deciliter or above.

Limitations:

Blood Alcohol Concentration test results are not available for all drivers and non-occupants involved in fatal crashes.  Missing data can result for a number of reasons -- the most frequent of which is that persons are not always tested for alcohol.  To address the missing data issue, NHTSA has developed a statistical model (Multiple Imputation) to estimate specific values of BAC across the full range of possible values.  Estimating missing BAC in this manner will permit the estimation of valid statistics such as variances, measures of central tendency, confidence intervals and standard deviations. The statistical model is based on important characteristics of the crash including crash factors, vehicle factors, and person factors.  While this measure does not link alcohol with fault in fatal crashes, the more comprehensive scope of the measure compensates for a possible undercount of the extent of the alcohol impaired driving problem.  Multiple Imputation differs from the statistical model used in previous years.  However, all historical series of alcohol involvement will be revised back to the 1982 data year to reflect the estimates from the new methodology.

Statistical Issues:

The primary sources of uncertainty in this performance measure arise from information gaps in the number of intoxicated non-motorists, and from using the statistical model to estimate the number of intoxicated drivers. 

The estimates of the number and percentages of persons killed in motor vehicle traffic crashes during 2001 included in this section are preliminary and are based on incomplete data and statistical models. They were provided to meet the time restraints required for this report.  NHTSA’s first official estimates for 2001, the Early Assessment, are being developed and will be completed in early April.  Differences between the Official Early Assessment estimates and those in this report are to be expected.

Verification & Validation:

Data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis. Quality control procedures are built into annual data collection at 6 and 9 months, and at year’s end.  In 1987 and 1988, an independent panel of academics reviewed and commented on the statistical methods used in measuring alcohol-related highway fatalities.  This report recommended that research and development utilize a model that would permit the imputation of missing BACs as a semi-continuous variable.

Comment:

This data program has been used for many years and is generally accepted for describing safety on the Nation’s highways. 

 

 

 

 

Highway injured persons rate                                                      Page 18

Measure:

Injured persons per 100 million vehicle-miles-traveled (VMT) (CY) (2001)

Scope:

The number of injured persons is an estimate of the total number of persons injured in motor vehicle traffic crashes that occur on public roadways in the 50 states and Washington, D.C.

Vehicle Miles of Travel (VMT) represent the total number of vehicle miles traveled by motor vehicles on public roadways within the 50 states and Washington, D.C.

Source:

The number of injured persons data are derived from the NHTSA’s National Automotive Sampling System (NASS) General Estimates System (GES).  The NASS GES is a nationally representative probability sample that yields national estimates of total nonfatal injury crashes, injured persons, and property-damage-only crashes. NASS GES data cover all roadways open to the public, using the National Highways System classification of roads.

VMT data are derived from FHWA’s monthly report, Traffic Volume Trends (TVT), a monthly report based on hourly traffic count data in the Highway Performance Monitoring System (HPMS).  Information is transmitted to NHTSA where it is reviewed for consistency and accuracy before being entered into the system. These data, collected at approximately 4,000 continuous traffic counting locations nationwide, are used to determine the percentage change in traffic for the current month from the same month of the previous year.  The percentage change is applied to the nationwide travel for the same month of the previous year to obtain an estimate of nationwide travel for the current month.  The data are recorded as monthly totals and cumulative yearly totals.

Limitations:

GES data are obtained from a nationally representative sample of 60 sites.  The results provide only national data, not state level data, and are subject to sampling error.  The magnitude of the sampling error depends on the number of Primary Sampling Units (PSUs) in the sample and the number of crash reports sampled within each PSU. 

VMT data are subject to sampling errors, whose magnitude depends upon how well the continuous counting locations represent nationwide traffic rates.  HPMS is subject to estimating differences in the states, although FHWA works to minimize such differences and differing projections on growth, population, and economic conditions which impact driving behavior.

Statistical Issues:

The estimate of the injury rate includes three main sources of uncertainty.  The numerator count of injuries has a standard error of 5.1% (cf. Appendix C of Traffic Safety Facts).  The denominator estimate of VMT contains both complex sampling and non-sampling errors.  Based on data from 1994-2000, the annual variation in the injury rate has a regression standard error of 4.04.

The estimates of the number and percentages of persons injured in motor vehicle traffic crashes during 2001 are preliminary and are based on incomplete data and statistical models.  NHTSA’s first official estimates for 2001, the Early Assessment, are being developed and will be completed in early April.  Differences between the Official Early Assessment estimates and those in this report are to be expected.

Verification & Validation:

Data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis. Quality control procedures are built into annual data collection at 6 and 9 months, and at year’s end.   A study was completed in 1993, looking at samples of FARS cases in 1989 through 1990 to assess the accuracy of data being reported.  VMT data is reviewed by FHWA for consistency and reasonableness.

Comment:

This data program has been in use for many years and is generally accepted for describing safety on the Nation’s highways.  GES records injury severity in four classes: incapacitating injury, evident but not incapacitating injury, possible but not visible injury, and injury of unknown severity.  Adjusting raw highway fatalities and injuries by VMT provides a means of portraying the changes in highway fatalities on a constant exposure basis – to facilitate year-to-year comparisons.

 

 

Large truck-related injured persons                                            Page 18

Measure:

Number and rate of injured persons involving large trucks. (CY) (2001)

Scope:

The measure includes all injured persons (e.g., drivers and occupants of passenger cars, motorcycles, large trucks, or pedestrians) associated with crashes involving trucks with a gross vehicle weight rating of 10,000 pounds or more. The number of injured persons is derived from NHTSA’s General Estimates System (GES). The injury rate is the number of injured persons per 100 million vehicle miles of large truck travel (VMT).

Source:

NHTSA’s General Estimates System (GES) provides injury data. VMT data are derived from the Federal Highway Administration’s (FHWA) Highway Performance Monitoring System (HPMS).

Limitations:

GES data are obtained from a nationally representative sample of 60 sites.  The results provide only national data, not state-by-state data.  Large truck VMT reported to FHWA by each state is based on a sample of road segments and is not a census.  In addition, the methods used to calculate total VMT may vary from state to state. The methods used by the states to estimate the VMT contribution from rural and urban minor collectors are unknown.

Statistical Issues:

The GES data have a standard error of 6.9% for injuries from truck and automobile crashes (cf. Appendix C of Traffic Accident Reports).  They are less accurate than the corresponding fatality counts.  Based on 1994-2000 data, the variation due to random chance in the number of injuries, which includes sampling variability, has a regression standard error of approximately 7,091.  Because the VMT data provided to FHWA from each state are estimates based on a sample of road segments, the numbers have associated sampling errors.  The methodology used by each of the states to estimate VMT is not known and may introduce additional non-sampling error into the estimates.  Although states provide VMT estimates on an annual basis, they are only required to update their traffic counts at all sampling sites once every three years.  Thus an annual VMT estimate from a particular state may be based, in part, on data collected during a previous year.  Based on 1994-2000 data, the chance variation in a given year in the number of injured persons per 100 million vehicle miles of large truck travel has a regression standard error of 4.39.

Verification & Validation:

Injury data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis. Quality control procedures are built into data collection and data processing.  FHWA routinely works with state data providers to modify reported VMT values that do not appear reasonable before incorporating them into its final master file.

Comment:

The data program has been around for many years and is generally accepted for describing safety on the Nation’s highways.  GES records injury severity in four classes: incapacitating injury, evident injury but not incapacitating, possible but not visible injury, and injury of unknown severity.  The large truck VMT data used to calculate injured persons rates have both sampling and non-sampling (i.e., bias) error associated with it.  The impact of these errors on FMCSA’s estimates of large truck crash rates is considered to be minimal.

 

Seat belt use                                                                                 Page 18

Measure:

Percentage of front occupants using seat belts. (CY) (2001)

Scope:

The proportion of front seat outboard passenger vehicle occupants using shoulder belts during daylight hours.

Source:

Data for 1998, 1999, and 2000 are from the National Occupant Protection Use Survey (NOPUS).  NOPUS is a National, multi-stage probability sample.  In the first stage, counties or groups of counties (Primary Sampling Units or PSUs) were grouped by region (Northeast, Midwest, South, and West), level of urbanization (metropolitan or not), and level of belt use (high, medium, or low).  Fifty PSUs were selected based on the vehicle miles of travel in those locations.  In the next stage, a random sample of eight (8) Census Tracts was selected within each of the PSUs.  In the final stage a sample of ten (10) roadway segments for all types of roads was selected within each Census Tract.  In the even numbered years, shoulder belt use of front seat outboard (driver and right front seat) passenger vehicle (passenger cars, vans, sport utility vehicles, and pickup trucks) occupants was observed during daylight hours at each of the 4,000 sampled roadway segments.  In 1999, a Mini-NOPUS consisting of observation at a subsample of 2,000 of the 4,000 roadway segments was conducted.

Estimates of national shoulder belt use for other years shown in the graph are based on state belt use surveys.  These surveys are conducted by most of the 50 States and the District of Columbia.  For the years shown, these surveys varied in coverage, design, and observation methods.  National averages were obtained by weighting the most recently provided state belt use estimate by the population of the state.

Limitations:

NOPUS data are based on a random sample of sites and, therefore, are subject to sampling error.  For the estimate of overall National shoulder belt use from the 2000 NOPUS Survey, sampling error was estimated to be 1.4 percentage points.  Additionally, observation of shoulder belt use is restricted to daylight hours.

State belt use surveys have been conducted in many different ways.  Less than half of the states conducted probability based surveys and the rest were based on other methods.  Additionally, most states conducted surveys that observed use only for those occupants and vehicles covered by their state belt use law.  After enactment of a grant program in the ISTEA of 1991, some 24 states had surveys that met design criteria specified by NHTSA.

Statistical Issues:

The primary source of uncertainty in NOPUS is sampling errors.  The most recent estimate shown in this report is based on a probability sample, and the survey bias and reweighting are complex.  For State surveys, uncertainty derives from disparities among the different surveys conducted by the states, the use of non-probability samples by many of the states, the differences in persons and vehicles observed, the differing methodologies and processes followed to collect data on the persons and vehicles observed, and the procedures used to estimate overall belt use.  To compute the National average from state rates for a specific year, when a state did not conduct a survey or provide NHTSA with an estimate, the most recent rate provided by that state was substituted.  Also, weighting state averages by population may have overstated the contributions of some states.  Based on data from 1994-2000, the annual variation in the seat belt use rate has a regression standard error of 1.31 percent.

Verification & Validation:

NOPUS data collection is managed by a survey research contractor who has responsibility to hire and train the data collectors/observers.  Before data collection begins, NHTSA reviews and approves all the training materials and Data collectors/observers must pass a 2-day training course.  The data contractor also conducts on-scene “surprise” quality control visits to ensure that observations are made correctly and data are coded properly.  Numerous edits are also employed in the data processing.  NHTSA reviews the data provided by the contractor for consistency. NHTSA reviewed and approved the survey designs and data collection procedures for 24 states as a result of a grant program authorized by the ISTEA of 1991.  NHTSA, however, did not conduct any quality review or validation of the data collection and estimation processes employed by the states during or after data collection for the years shown.

Comment:

None.

 

Air carrier fatal accident rate                                                      Page 21

Measure:

Fatal aviation accidents (U.S. commercial air carriers) per 100,000 departures. (FY)

Scope:

This measure includes both scheduled and nonscheduled flights of large U.S. air carriers (14 CFR Part 121) and scheduled flights of commuter airlines (14 CFR Part 135).  It excludes on-demand (i.e., air taxi) service and general aviation.

Source:

Part 121 and Part 135 departure data is submitted to BTS under 14 CFR Parts 241 and 298, respectively.  NTSB provides accident data.

Limitations:

The fatal accident rate in these categories is small and could significantly fluctuate from year to year due to the occurrence or non-occurrence of a single accident.

Statistical Issues:

The switch from calendar to fiscal year in 2001, combined with the use of departures rather than flight hours as the activity measure for the denominator, present new problems.  The FAA has no independent data sources to validate BTS-collected departure data as it did with flight hour data.  To overcome reporting delays of 60 to 90 days, FAA must rely on historical data, partial internal data sources, and Official Airline Guide (OAG) scheduling information to project at least part of the fiscal year activity data.  Due to the reporting procedures in place, it is unlikely that calculation of future fiscal year departure data will be markedly improved.  Lacking complete historical data on a monthly basis and independent sources of verification increases the risk of error in the activity data.  The regression standard error for the annual variation in the fatality rate, based on data from 1994 – 2000, is 0.023.

Verification & Validation

The FAA does comparison checking of the departure data collected by BTS; however, FAA has no independent data sources against which to validate the numbers submitted to BTS.  FAA compares its list of carriers to the DOT list to validate completeness of the reporting list and places the carriers in the appropriate category (i.e., Part 121 or Part 135).  NTSB and FAA’s Office of Accident Investigation meet regularly to validate the accident count.

Comment:

The joint government/industry group working on improving the level of safety for U.S. commercial aviation has determined that the number of departures is a better denominator measure to use for determining accident rates.  In a recent report on the Safer Skies effort the Government Accounting Office agreed and recommended that the FAA use departures. 

 

General aviation fatal accidents                                                  Page 21

Measure:

Number of fatal general aviation accidents. (FY)

Scope:

The measure includes on-demand (non-scheduled FAR Part 135) and general aviation.  General aviation comprises a diverse range of aviation activities.  The range of general aviation aircraft includes single-seat homebuilt aircraft, helicopters, balloons, single and multiple engine land and seaplanes including highly sophisticated extended range turbojets.

Source:

National Transportation Safety Board (NTSB).

Limitations:

The use of the 1996-1998 timeframe for the baseline represents one of the safest periods in general aviation history in terms of a decline in fatal accidents.  The number of general aviation accidents reported in any given year might change in subsequent years.  There are many reasons for these changes to the historical data.  Primary among them is that the accident had not been reported to the NTSB, or that it was misreported and the information corrected at a later date.

Statistical Issues:

There is no major error in the accident counts.  Random variation in air crashes results in a significant variation in the number of fatal accidents over time.  The regression standard error in this variation for 1996 through 2000 is 16.5. 

Verification & Validation:

NTSB and FAA’s Office of Accident Investigation meet regularly to validate the information on the number of accidents. 

Comment:

It would be preferable to use fatal accident rates rather than fatal accidents as the performance measure. However, general aviation flight hours are based on an annual survey conducted by the FAA.  Response to the survey is voluntary.  The accuracy of the flight hours collected is suspect and there is no readily available way to verify or validate the data.  For this reason, the General Aviation community is unwilling to use a rate measure until the validity and reliability of the survey data can be assured.

 

Operational Errors (Air Traffic)                                                    Page 24

Measures:

1.  Operational errors per 100,000 activities, or per 1 million activities. (2001)

2. Number of operational errors where less than 80 percent of required separation is maintained.

Scope:

An error occurs when separation between aircraft is less than the separation determined necessary for the specific phase of flight.  “Activities” are total facility activities, as defined in Aviation System Indicators 1997 Annual Report.  Total facility activities are the sum of en route and terminal facility activities.

Source:

FAA air traffic facilities have a software program called Operational Error Detection Patch (OEDP) that detects possible operational errors and sends alert messages to supervisory personnel.  Facility management reviews OEDP alerts and data provided from the National Track Analysis Program (NTAP) to determine if an operational error has occurred.  Controllers are required to report operational errors.  The information is summarized in the FAA Air Traffic Operational Error and Deviation Database.

Limitations:

There is a few months’ lag in reporting data because of the need to investigate major incidents.  The severity of errors is not measured.  Minor errors such as a 4.5-mile rather than a 5-mile separation are counted in the same way as more serious errors.  Data are available for 1994 and following years.

The DOT IG conducted an audit of reporting on operational errors.  The IG believes that there is a potential for underreporting of operational errors, as some errors are self-reported.  The FAA disagrees with this assessment because there are substantial penalties for not reporting an operational error.

Statistical Issues:

There are no major sources of systematic error in the operational errors data that have been quantified.  Again, random variation in operational errors results in a significant variation in the measured rates over time. The regression standard error in the operational error rate using 100,000 activities denominator and the 1 million activities denominator, based on 1994-2000 data, are .048 and .48, respectively.

Verification & Validation:

FAA performs system checks and counts daily against reported data to ensure the accuracy of information reported.

Comment:

In August 1998, the FAA discovered and corrected a misunderstanding of the procedures used in interpreting separation reported by the National Track Analysis Program and the data provided by the Operational Error Detection Patch.  The corrected application of these procedures, while not affecting safety, has resulted in an overall increase in the number of errors reported between 4.6 and 4.9 miles separation (Standard separation in these cases is 5 miles).

 

Runway incursions                                                                        Page 24

Measures:

1.  Number of runway incursions. (FY) (2001)

2.  Number and rate (per 100,000 operations) of highest risk runway incursions.

Scope:

Runway incursions are the result of ground collision hazards or loss of separation for aircraft in the process of taking off or landing. They are grouped in three general categories:  operational errors, surface pilot deviations, and vehicle/pedestrian deviations. Incursions are reported and tracked at airports that have an operational air traffic control tower. 

Source:

Air traffic controllers and pilots are the primary source of runway incursion reports. The data is recorded in the FAA National Incident Monitoring System (NAIMS).

Limitations:

Preliminary incident reports are evaluated when received. Evaluation can take up to 90 days.

Statistical Issues:

There are no major sources of systematic error in quantified runway incursion data. The regression standard error in the reported number of incursions, based on 1994-2000 data, is approximately 15.4.  Based on 1998 – 2001 data, the regression standard error for the number and rate of highest risk runway incursions are 8.8 and 0.01, respectively.

Verification & Validation:

Surface incidents are reported in the Administrator’s Daily Bulletin at the beginning of each weekday. Surface incidents are evaluated to determine if they should be classified as incursions. Incidents are evaluated against the official runway incursion definition. The Air Traffic Runway Safety Program Manager, ATP-20, makes the final decision regarding runway incursions.

Comment:

None.

 

Mariner Rescue                                                                             Page 26

Measure:

Percent of all mariners in imminent danger who are rescued. (FY)

Scope:

Includes people in water; on shore; and aboard a vessel, offshore structure, pier, or vehicle that is in distress or in urgent need of assistance.  The Coast Guard makes a final determination on scene whether there is imminent danger, based on criteria that include the nature of distress, the condition of the vessel, the people onboard, and the environmental conditions.  Criteria for this decision are discussed in search and rescue doctrine publications.

Source:

CG Search and Rescue Management Information System (SARMIS). Data is collected from Coast Guard field units that conduct search and rescue responses.

Limitations:

It is probable that some number of imminent danger cases, and the associated lives, are not reported in SARMIS.  This includes situations where no distress call was received by the Coast Guard and the persons in distress were rescued by private citizens or local government personnel, or where the persons in distress perished without trace. The extent of this under-reporting is not known. There is some judgment involved in assessing whether mariners are in danger. However, there is likely to be consistency in these assessments across years.  1994 data is skewed upward by a large surge of migrants interdicted at sea, most of whom were counted as “rescued,” thus increasing the percentage of lives reported as saved.  Reporting no longer includes migrants interdicted; they are counted directly as migrants interdicted under law enforcement activity.  Prior to the introduction of the next generation data system in October 2000, data entry was limited to closed cases, after a rescue has been successfully completed or after the recovery of a body.  The new data system now allows missing bodies to be tracked.  In this first year of data, more cases than expected were found where bodies were not recovered.  Before adding this number into our data analysis, we will track this number to assure that this represents a data trend and not an unusual aberration.  Errors may be introduced in SARMIS through data entry, but are likely rare for lives saved data elements.

Statistical Issues:

The primary source of uncertainty consists of non-sampling errors. The second generation data system, brought on-line on October 1, 2000, reduces error due to miscoding through the use of more extensive drop down menus, machine generated case numbers, structured data boxes, and more extensive business rules eliminating the selection of data not consistent with other entered data. The regression standard error for year-to-year chance variation is 2.6 percent mariners rescued, based on data from 1994 through 2000.

Verification & Validation:

SARMIS data entry system uses structured entry values, check boxes, and pull down selection lists to limit entry errors.  The use of plain language descriptions eliminates a majority of erroneous data code selection. Additional system business rules also eliminate the selection of data not appropriate with other entered data. The SAR Mission Coordinator (SMC) is responsible for accurate entry of particular case data by all units involved in the case. CG Program Managers annually validate the data in SARMIS. Entries are reviewed at Coast Guard District offices as first step in validation – errors and inconsistencies are identified and corrected.  Finally, Coast Guard Headquarters program managers review compiled data annually to assess consistency with historic variance and trends.  This review includes curvilinear regression analysis to compare current data to historic data and a program review analysis to identify and resolve aberrations.

Comment:

Beginning in FY01, this measure will cover all mariners in distress reported in SARMIS.  The previous measure covered only mariners reported in distress that were rescued.  The significance of the 87.5% result for FY99 is uncertain at this point; FY95-98 data show a flat trend at 84%.  It is not known if the FY99 result was produced by anomalous factors, or if it is the product of program strategies and a changing external environment.  Therefore, the goal target remains at 85% until more analysis is completed.  For FY 2001, the preliminary estimate of the measure was 84.2 percent of all lives, bringing the percentage about equal to the average since 1995 and slightly below the goal, but certainly within normal variation about the average. 

 

Recreational boating fatalities                                                    Page 28

Measure:

Number of recreational boating fatalities. (CY) (2001)

Scope:

Measure includes fatalities occurring aboard vessels that are being operated for recreational purposes.  Surfboards, iceboats, and vessels engaged in sanctioned racing events are not considered recreational vessels.  Fatalities are included if caused by a fire, explosion, sinking or other occurrence involving a recreational vessel, and the vessel or associated equipment caused or contributed to the fatality.  Fatalities are not included if they occurred aboard a recreational vessel, but were caused by self-inflicted wounds or natural causes.  Fatalities are also excluded if they occurred while the victim was engaged in other activity such as swimming or diving, where the vessel was used as a platform only and was not a contributing factor to the fatality. Beginning two years ago, the measure for Recreational Boating was revised by adding an additional 6% to the aggregate number of reported fatalities, to correct for an estimated 6% underreporting of recreational boating fatalities.

Source:

Coast Guard Boating Accident Report Database (BARD).  Data is entered into BARD by state administrators who collect data from boat owners and operators through formal Boating Accident Reports, as instructed in 33 CFR 173c. 

Limitations:

Fatality data is derived from reports submitted by the public along with accompanying state investigation reports. There is consensus among the Coast Guard, the states, safety professionals, and other researchers that most fatalities that occur on inland and most coastal waters are under-reported.  To better quantify the extent of possible under-reporting the Coast Guard initiated and funded an analysis of BARD data conducted by the Boat Owners Association of the United States (BOAT/U.S.) Foundation for Boating Safety.  The study found some fatalities involving recreational boating in the Coast Guard’s Search and Rescue Management Information System (SARMIS) that were not in BARD. However, although the study reported a 9% discrepancy, further analysis revealed that some of these findings would not be reportable as recreational boating fatalities. There is also consensus that under-reporting exists for fatalities occurring offshore, and aboard U.S. recreational boats operating overseas.  Also, although there are guidelines as to what constitutes a recreational boating fatality, there is still an element of interpretation at the state level in reporting fatalities.  It is probable that the states do not always interpret the guidelines in the same manner. Overall, the best estimate indicates that total fatalities are currently under-reported by at least 6%.

Statistical Issues:

The discrepancy between BARD and the Search & Rescue Management Information System (SARMIS) amounts to 6% of the total reports for those states covered by SARMIS.  The numbers given in this report have been adjusted to correct the deficiency.  Also, note that since the boating fatality counts are influenced by weather, gasoline prices and other external factors, annual chance variation should be large.  Using data from 1994 to 2000, the annual variation in the number of fatalities attributable to random chance has a regression standard error of 50.7.

Verification & Validation:

Fatality data in BARD is verified and validated by state boating administrators and Coast Guard program managers. At the end of the calendar year, the Coast Guard compiles state fatality data and sends a report to each state for confirmation.  Both State and Coast Guard officials review the statistics, including sampling of cases to ensure guidelines for classifying fatalities were followed.  Any discrepancy is reconciled jointly by the State and Coast Guard program manager.

Comment:

Data are not normalized for increases or decreases in the number or usage of boats, which tends to limit data use in making comparisons over time.  The number and usage of recreational boats has increased over the past 2 decades, while the raw number of fatalities has generally decreased.

The BOAT/US review of BARD data for 1993 through 1997 identified underreporting in BARD of 8% in 1993 and 1994, 12% in 1995, 13% in 1996 and 8% in 1997.  The Coast Guard reviewed BOAT/US’s findings for 1995, 1996, and 1997.  Each record for these years was checked and fatalities that were incorrectly labeled as recreational boating fatalities by BOAT/US were removed from the count.  Based on this revised count of recreational boating fatalities with mislabeled fatalities removed, the Coast Guard estimates that 7%, 8% and 4% of all recreational boating fatalities were not captured in its Boating Accident Report Database (BARD) in 1995, 1996 and 1997 respectively for purposes of this report. The median of these numbers – 6% - has been used to adjust recreational boating safety data for 1993, 1994, 1998 and 1999, and to reset the goals for 1999 through 2001. The original goal of 720 has been increased by 6% to 763 for 2000.

The Coast Guard is in the process of commissioning a comprehensive National Boating Survey to obtain valid and reliable information on boating practices, safety, and exposure.  This information will enable safety officials to assess boating risk, implement appropriate safety intervention strategies, and measure the effectiveness of program activities in reducing the risk and negative outcomes associated with the use of recreational boats. Data from this study will be used to further address underreporting issues and estimate reporting discrepancies in BARD.  The study was originally set to begin in Fall 2001, however data collection is now scheduled to begin in April 2002.

 

Passenger Vessel Fatalities                                                          Page 28

Measure:

Fatalities and rate (per million passenger capacity) aboard passenger vessels. (2001)

Scope:

This measure is an indicator of passenger safety.  It includes reportable marine casualties resulting in the death or disappearance of a passenger aboard any U.S. vessel (regardless of type or location) or aboard foreign flag vessels in U.S. waters. Exceptions include death/disappearance of “non-passengers”, whenever the cause of death/disappearance is classified as being from diving, natural causes, (e.g. heart attack) or whenever the death/disappearance is the result of an intentional act (e.g. suicide, altercation).  Fatalities on recreational vessels are not included for two principal reasons: Recreational vessels are prohibited from carrying “passengers” and recreational vessel fatalities are measured and reported separately.

Source:

Passenger fatality source data is obtained from the Coast Guard Marine Safety Information System (MSIS).  Passenger fatalities are reported to the Coast Guard as required by federal regulations.  Sources of reports are most often vessel masters, operators, owners, insurance companies, legal representatives, and other mariners.

Limitations:

The investigation, retrieval, analysis and reporting processes result in under-reporting for the most recent year, with the most significant effects over the most recent 5 months.  Estimates are often used to compensate for this known data-lag. The Coast Guard initiates about 40-50 civil penalty cases for failure to report marine casualties, although many of these are for minor casualties. In addition, some passenger fatalities may not be reported to the Coast Guard. This number is unknown.  Some passenger injuries may ultimately prove fatal and lead to death; some missing passengers may be found.  These numbers may not be updated to reflect the changes in status.  The number is believed to be small.  Duplicate casualty entries are sometimes entered into MSIS, and some casualties are mistakenly omitted or coded incorrectly. Verification procedures strive to correct these errors, but it is probable that a small number are not corrected. The data retrieval & reporting processes do not allow automated distinction between all death types (e.g. natural vs. accidental).  As a result, some natural deaths or suicides may be inadvertently included.

Statistical Issues:

The major sources of uncertainty in this measure are the estimation error (as a result of the data-lag) and the reporting error (as a result of the inability to distinguish between which deaths should be included and which should be excluded).

Verification & Validation:

Verification and validation occurs at several levels.  Edit checks within MSIS software can detect some incorrect or missing data and force review and correction before data entry is completed.  Selection lists for certain data fields also reduce the opportunity for data entry error.  All investigations go through review at the field unit for accuracy.  Investigations of serious marine casualties are also usually reviewed at district and headquarters offices.  The headquarters Data Administration staff conducts periodic quality control checks to identify entry errors such as missing data or miscoding, and corrects any errors identified.  Errors identified are referred to either the Data Administration staff or the Investigations and Analysis staff for correction.

Comment:

During FY 2002, the Marine Safety Information System (MSIS) will be replaced by the Marine Information System for Safety and Law Enforcement (MISLE).  While the new system will be a major improvement, it is expected to cause serious difficulties in making performance comparisons.  One factor is that many business processes were re-designed in conjunction with system development.  Another factor is that data quality under MISLE is expected to be superior to that of MSIS.  While this represents improvement, it may cause near-term problems in making meaningful comparisons of data between the two systems.

 

Rail fatality and Accident rates                                           Page 29 & 30

Measure:

1.  Train accidents per million train-miles.

2.  Rail-related fatalities per million train-miles. (CY) (2001)

Scope:

The fatality measure includes anyone on rail property, any on-duty railroad employee, and anyone killed by a train or its contents.  It does not include fatalities on trains or rail lines that do not connect to the national rail network, such as mass transit operations, certain excursion and tourist railroads, and some industrial railroads not connected to the general system.  The only railroad fatalities that are not counted are suicides (as determined by a public official) and death by natural cause not associated with railroad operations. Train accidents do not include those at grade crossings.  They are reported under the performance goal for highway-rail grade crossing accidents.

Source:

Railroad Safety Statistics – Annual Report.  Statistical data, tables, and charts depict the causes and nature of rail-related fatalities.  Data on fatalities and train miles are reported to FRA by railroad companies.

Limitations:

Because of the scope of the reporting criteria, some fatalities that are counted are not associated directly with operation of the trains, and some railroad fatalities are not counted.  This scope is consistent with the regulatory authority of the agency, but not consistent with other modes of transportation for comparative purposes.

Statistical Issues:

The reported estimates are based upon partially reported data from 2001.  Based on data from 1994-2000, chance variation from year to year, as reflected in the regression standard error, is 0.055 for rail fatalities.

Verification & Validation:

Railroads are required by law to submit monthly accident/incident reports to FRA.  They are also required to update any inaccurate or incomplete information.  FRA conducts routine data audits (records inspections) to verify the adequacy of railroad reporting and record keeping requirements.

Comment:

None.

 

Highway - Rail grade-crossing accidents                                     Page 29

Measure:

Grade-crossing accidents divided by the product of: 1) million train miles and 2) trillion vehicle-miles-traveled. (CY)

Scope:

The measure includes all collisions with on-track equipment and highway users at public and private grade crossings.

Source:

Collisions and train-miles are reported in FRA’s Railroad Safety Statistics – Annual Report.  Vehicle-miles-traveled (VMT) are obtained from the FHWA Office of Highway Information Management. 

Limitations:

Because the denominator includes all highway vehicle-miles-traveled (VMT), and not just VMT that are exposed to grade crossings, the rate portrayed may be lower than the actual risk.

Statistical Issues:

Trains and automobiles have different exposures at rail crossings---the denominator used here attempts to combine these.  The numerator is based on partially reported 2001 data. The annual variation by chance from year to year as measured by the regression standard error is 0.109, based on data from 1994-2000.

Verification & Validation:

FRA’s Office of Safety has a review process to ensure that railroads and the States comply with Federal reporting requirements in the preparation of the FRA Railroad Safety Statistics - Annual Report.

Comment:

None

 

Transit fatality and injured person rates                            Page 30 & 31

Measure:

1.      Transit fatalities per 100 million passenger miles traveled. (CY)

2.      Transit injured persons per 100 million passenger miles traveled. (CY)

Scope:

The data include both riders and employees.  A fatality is defined as a transit-caused death from collision, personal casualty, fire, derailment, or bus going off the road.  An injury is defined as any physical damage or harm to a person requiring medical treatment caused by a transit collision, personal casualty, fire, derailment, or bus going off the road.

Source:

FTA’s Safety Management Information System (SAMIS), with data reported by transit operators to the National Transit Database (NTB).

Limitations:

Because of the scope of the reporting criteria, some fatalities that are counted are not associated directly with transit operation.  This scope is consistent with the regulatory authority of the agency, but not consistent with other modes of transportation for comparative purposes.

Statistical Issues:

The fatality and injury counts in SAMIS are generally quite accurate---the major source of error in the measure comes from uncertainty in the passenger miles traveled.  Based on 1994-2000 data, the chance variation in a given year has a regression standard error of 0.039 for the transit fatality rates and 2.210 for the transit injury rates.

Verification & Validation:

An independent auditor and the transit agency’s CEO certify that data reported to the NTD are accurate.  Using data from the NTD to compile the SAMIS data, the Transportation Systems Center compares current safety statistics with previous years, identifies questionable trends, and seeks explanation from operators.

Comment:

None.

 

 

Pipeline failures                                                                            Page 33

Measure:

Excavation damages to natural gas and hazardous liquid pipelines. (FY)

Scope:

This measure is based on reported hazardous liquid and natural gas accidents that meet federal reporting criteria as defined in 49 CFR 191.1 and 191.15 for natural gas transmission pipeline incidents and in 49 CFR 195.50 for hazardous liquid pipelines. 

Source:

RSPA’s Natural Gas Distribution and Transmission Incident Reports and Hazardous Liquid Pipeline Accident Reports. Failure reports are filed within 30 days of the occurrence of reportable incidents.  Complete calendar year data are available by March 1 of the following year.  Data may change as operators file supplemental reports.

Limitations:

RSPA lacks adequate infrastructure information on pipeline operations and maintenance needed to fully characterize problems when they occur and lacks information on precursor conditions that contribute to incidents.  RSPA seeks further improvements in data collection in 2002 to address these concerns.

Statistical Issues:

Reduction in excavation damages is tied to economic growth and expansion as populations increasingly are encroaching on once rural areas where major interstate pipelines are located.  Because of delays in mail delivery associated with 9/11/2001 terrorist activities, statistical close-out of the 2001 tally requires an extrapolation of number of reports anticipated for the last quarter of 2001.

Verification & Validation:

RSPA reviews/verifies data provided for accuracy and requests supplemental reports where shortcomings are indicated.

Comment:

RSPA discontinues this measure after 2002, replacing this safety measure with pipeline excavation damages measure.

 

Pipeline failures                                                                            Page 34

Measure:

Failures of natural gas transmission pipelines. (CY) (2001)

Scope:

This measure is based on reported hazardous natural gas leaks that meet federal reporting criteria as defined in 49 CFR 191.1 and 191.15 for natural gas transmission pipeline incidents. 

Source:

RSPA’s Natural Gas Transmission Incident Report. Failure reports are filed within 30 days of the occurrence of reportable incidents.  Complete calendar year data are available by March 1 of the following year.  Data may change as operators file supplemental reports.

Limitations:

RSPA lacks adequate infrastructure information on pipeline operations and maintenance needed to fully characterize problems when they occur and lacks information on precursor conditions that contribute to incidents.  Joint Federal, state and industry teams have been formed to devise a new course to improve information availability.

Statistical Issues:

The number of failures of natural gas transmission pipelines is likely to be underreported.  The annual variation in the number of failures from year to year due to chance has a regression standard error of 528 for natural gas pipeline failures based on data from 1994 to 2000.

Verification & Validation:

RSPA reviews/verifies data provided for accuracy and requests supplemental reports where shortcomings are indicated.

Comment:

None.

 

 

Hazardous Materials Incidents                                                     Page 36

Measure:

Number of serious hazardous materials incidents in transportation. (CY)

Scope:

Serious reported hazardous materials incidents were initially defined by RSPA to be those that result in a fatality or major injury (for most purposes, an injury resulting in hospitalization) due to a hazardous material, closure of a major transportation artery or facility, or evacuation of six or more persons due to the presence of a hazardous material, or a vehicle accident or derailment resulting in the release of a hazardous material.  For the 2003 Plan, the definition is revised to include those incidents resulting in a fatality or major injury, the evacuation of 25 or more employees or responders or any number of the general public, the closure of a major transportation artery, the alteration of an aircraft flight plan or operation caused by the release of a hazardous material or the exposure of hazardous material to fire; plus any release of radioactive materials from Type B packaging, Risk Group 3 or 4 infectious substance, over 11.9 gallons or 88.2 pounds of a severe marine pollutant, or a bulk quantity (over 119 gallons or 882 pounds) of a hazardous material.  This measure tracks only transportation related releases of hazardous materials that are in commerce.  Volume of spills is not tracked, as this does not necessarily indicate risk.

Source:

Hazardous Materials carriers report data to RSPA for entry into the Hazardous Materials Information System (HMIS).

Limitations:

Data for all hazardous materials incidents is suspected of being incomplete due to under-reporting for minor incidents.  Most reportable serious incidents are in the system, making this a more consistent measure for program management.  However, it does not reflect all incidents.  RSPA has issued an NPRM to revise the reporting system.

Statistical Issues:

Although the number of incidents is likely to be underreported, such recording error is probably small in comparison to the annual variation due to chance.  The annual variation in the number of failures (original definition) from year to year due to chance has a regression standard error of 37.2 based on data from 1994 to 2000.  The new incident definition has a regression standard error of 30.6 based on data from 1997 to 2000.

Verification & Validation:

RSPA verifies the data by periodic follow-up reviews of data entry by the manager of the Hazardous Materials Information System, and verification audits of the data entry process.  RSPA crosswalks HMIS reports against the National Response Center log of accidents.  RSPA is improving compliance with reporting requirements by correlating HMIS reports with FRA’s Accident Report data and the HMIS telephonic data.  RSPA is piloting and plans to incorporate procedures to correlate HMIS reports with FHWA’s Safetynet Accident File data.

Comment:

None.

 

Details on DOT Measures of Homeland Security

Aviation security                                                                           Page 41

Measure:

1.  Average waiting time in minutes for passengers in line for screening. (FY)

2.  [Measure on passenger and baggage screening effectiveness.] (FY)

Scope:

  TBD

Source:

  TBD

Limitations:

 

Statistical Issues:

 

Verification & Validation:

 

Comment:

 

 

Aviation Security                                                                           Page42

Measure:

Detection rate for explosive devices and weapons that may be brought aboard aircraft. (FY) (2001)

Scope:

Machine performance test results, automated threat-image projection (TIP) and FAA field agent testing of aviation security screener proficiency to detect and resolve images or FAA test objects that simulate weapons and explosive devices in checked and carry-on baggage, or carried on the person through an airport security checkpoint.

Source:

FAA Office of Civil Aviation Security Airport and Air Carriers Information Reporting System (AAIRS).  Laboratory test results from the William J. Hughes Technical Center.

Limitations:

No comment.

Statistical Issues:

There is no major error present in the subject data.

Verification & Validation:

Special “red team” testing led by agents based at FAA headquarters is used to validate field test results.  AAIRS data is subject to multiple layers of review.

Comment:

The White House Commission recommended more aggressive, realistic testing.  Funding that began in 1997 enabled an increase in testing as more field agents were hired and trained.  Prior to 1998, data from realistic testing were too sparse to be conclusive.

 

Coastal and Seaport Security                                                       Page 43

Measure:

Percent of high interest vessels screened.

Scope:

High Interest Vessel (HIV) inspection or escort is measured by the a ratio of the number of HIV vessel inspected or escorted to the number of HIV vessels arriving at US ports.  HIV designation is determined using specific criteria.  Coast Guard inspection or escort standards are to inspect 100% of HIV.

Source:

The data for this measure is collected using a manual count from situation reports sent after a vessel inspection or escort.

Limitations:

This is an interim activity-based measure.  Appropriate outcome-based measures are under development that will improve our ability to measure and reduce security risks in US ports.

Statistical Issues:

This is a new measure and data systems have not yet been developed or modified to capture this information.  It is possible that errors in the data could result due to manual data collection.

Verification & Validation:

Verification and validation is conducted through cross checks with situation reports.  

Comment:

None.

 

Military Readiness                                                                        Page 44

Measure:

Percentage of days that the designated number of critical defense assets (high endurance cutters, patrol boats, and port security units needed to support Defense Department operational plans) maintain a combat readiness rating of 2 or better. (FY) (2001)

Scope:

Only high endurance cutters, patrol boats, and port security units that are designated as necessary for defense plans are included. The specific units required are classified.

Source:

DOD Status of Readiness and Training System (SORTS) – Database used by the Coast Guard in applying DOD standards to its assets to determine a readiness score.

Limitations:

SORTS uses a multi-factor matrix to calculate the readiness status.  Although specific criteria are outlined for each factor, some judgment is required in applying criteria.  Different units and personnel may apply standard criteria in slightly different ways depending on the nature of the unit’s mission.

Statistical Issues:

This particular performance measure in FY 2001 is based on readiness levels of two types of vessels, patrol boats and high endurance cutters, which have extremely different levels of readiness.  In addition, a third resource, Port Security Units (PSUs), is measured for its readiness.  PSUs are comprised of Coast Guard Reservists and Active Duty personnel, trained to protect foreign ports for expeditionary forces.  The drastic change between FY 1999 and FY 2000 performance was caused in large part due to the fact that the requirement to report the Contingency Personnel Requirements List (CPRL) (the full wartime personnel strength requirement) in the unit SORTS report was waived for FY 2000 and subsequent years pending validation of personnel requirements that have changed due to new equipment and operational procedures.  The Navy has been informed of this waiver and has not objected to reporting personnel strength using the less demanding Coast Guard standards for peacetime operations in view of the fact that Reserve Unit personnel are available to quickly bring Coast Guard units up to the full wartime personnel strength requirements in the event of a war.

Verification & Validation:

Units self assess and report readiness using objective standards.  Unit readiness is periodically validated through inspections, assistance visits, and in some cases training and assessment at Navy facilities.  These assessments are conducted by external, field level commands (such as Coast Guard areas, districts, and groups).

Comment:

Coast Guard will continue to reassess the overall adequacy of this measure.

 

Strategic Mobility                                                                         Page 47

Measure:

Percentage of DOD-required shipping capacity complete with crews available within mobilization timelines (FY)

Scope:

As of March 2002, this measure is based on the material availability of 76 ships in the Maritime Administration’s Ready Reserve Force (RRF) and 115 ships enrolled in the Voluntary Intermodal Sealift Agreement (VISA) program, which includes 47 ships enrolled in the Maritime Security Program (MSP).  A second factor pertinent to this measure is the availability of sufficient licensed and unlicensed mariners to operate the available ships.  The performance measure represents the number of available ships (compared to the total number of ships in the RRF and VISA) that can be fully crewed within the established readiness timelines.  While other Government (primarily Military Sealift Command) owned or controlled sealift type vessels are not included in this measure, they draw their crews from the same pool of mariners.  Accordingly, the availability measure is adjusted to reflect expected requirements during the early stages of a military crisis.

Source:

Material availability of ships: MARAD records (and reports to DOD) on the readiness/availability status of each RRF ship each month.  Typical reasons why a ship is not materially available include: the ship is in drydock, the ship is undergoing a scheduled major overhaul, or the ship is undergoing an unscheduled repair.  MARAD and DOD also maintain records of the sealift ships enrolled in the MSP and VISA and their crew requirements. Availability of mariners: Information on the available supply of licensed and unlicensed mariners is extrapolated from data received from the U.S. Coast Guard’s Merchant Mariner Licensing and Documentation (MMLD) system.

Limitations:

The information on the available supply of licensed and unlicensed mariners is an estimate. Because the MMLD also does not contain all of the information on individual mariners contained in their paper records, and provides no information on the availability and willingness of individuals to accept a sealift position in an emergency, it does not provide sufficient assurance of mariner availability.

Statistical Issues:

None

Verification & Validation:

The MARAD Regional Offices (and contracted ship managers) monitor the condition and overall readiness of each assigned RRF ship to meet its DOD mission.  When a ship is determined not capable of meeting its activation timeframe (mission), it is given one of several vessel condition ratings that are reported to DOD.  The monthly report contains an explanation of the deficiency and an estimated date when the ship will become fully capable of meeting its mission.  MSP contract performance is monitored throughout the year in order to assure proper payment of the MSP payment to the ship operators.  Recently, MARAD attempted to validate mariner availability estimates by conducting a survey of the mariner population.  A second survey is expected to commence in April 2002 to refine and improve the information needed to determine availability.  Because the decision to serve is a matter of individual choice and is subject to change, MARAD intends to develop a plan for maintaining current information on mariner availability based on the results of the 2002 mariner survey.

Comment:

None.

 

DOD-designated port facilities                                                     Page 47

Measure:

Percentage of DOD-designated commercial strategic ports for military use that are available for military use within DOD established readiness timelines.

Scope:

The measure consists of the total number of DOD-designated commercial strategic ports for military use that are assessed as able to meet DOD-readiness requirements on 48-hour notice, expressed as a percentage of the total number of DOD-designated commercial strategic ports.  Presently there are 14 DOD-designated commercial strategic ports.  Port readiness is based on monthly reports submitted by the ports and semi-annual port readiness assessments by MARAD in cooperation with other NPRN partners.  The MARAD/DOD semi-annual port assessments provide data or other information on a variety of factors, including the following: the capabilities of channels, anchorages, berths, and pilots/tugboats to handle larger ships; rail access, rail restrictions, rail ramp offloading areas, and rail storage capacities; the availability of trained labor gangs and bosses; number and capabilities of available cranes; long-term leases and contracts for the port facility; distances from ports to key military installations; intermodal capabilities for handling containers; highway and rail access; number of port entry gates; available lighting for night operations; and number and capacity of covered storage areas and marshalling areas off the port.

Source:

MARAD data are derived from monthly reports submitted by the commercial strategic ports and from MARAD/DOD semi-annual port assessments.

Limitations:

Port readiness assessments were not made prior to 1995; therefore, data are available only for 1995 and later years.  MARAD conducts a monthly survey of all strategic facilities to determine whether they meet the DOD availability requirement.  This information is provided to MARAD as a self-assessment by the port agency that owns the facility.  There is some degree of subjectivity in determining the availability of the port facilities.  As part of the overall planning process, MARAD and DOD conduct semiannual visits to independently verify and reassess port capability and availability.  The indicator is by definition a point-in-time judgment.  The results of the monthly and semi-annual reports used to measure port readiness can vary in accordance with the intensity of commercial activity at a given port at the time of the assessment.  Also, the monthly reports do not include the same level of detail as the semi-annual assessments, although MARAD is in continuous contact with port officials to minimize response error.

Statistical Issues:

The measurement of port readiness is an overall measure derived from MTMC comments, monthly readiness reports, and semi-annual assessments.  As such, it is a subjective measure.

Verification & Validation:

The MARAD/DOD semi-annual port visits independently verify and reassess not only the DOD-designated facilities, but also the total capability of the commercial strategic port.

Comment:

None.

 

Sealift capacity                                                                             Page 48

Measure:

Ship capacity (in twenty-foot container equivalent units, or TEUs) available to meet DOD’s requirements for intermodal sealift capacity. (FY) (2001)

Scope:

Includes the aggregate TEUs (or estimated square footage) of cargo capacity for ships enrolled in the Maritime Security Program (MSP) and Voluntary Intermodal Sealift Agreement (VISA).

Source:

MARAD/USTRANSCOM database of the militarily useful sealift capacity for ships enrolled in the MSP and VISA programs, based on vessel capacity data obtained from the vessel operators. 

Limitations:

MARAD, DOD and operator data on vessel characteristics (e.g., deck strength in pounds per square feet, deck height, container stowage factors), which are used to determine the portion of a vessel suitable for carrying military cargo, are not always consistent.  For example, the majority of ships in MSP/VISA are containerships, which normally are measured in TEUs; however, DOD generally measures surge sealift ships, most of which are Roll-on/Roll-off vessels, in square feet.  Historical data prior to FY 1997 are unavailable since the MSP and VISA programs were not enacted until that year.

Statistical Issues:

None.

Verification & Validation:

MARAD works with DOD and the maritime industry to use the most accurate information.  MARAD validates the vessel capacity data, which are obtained from the vessel operators, through comparisons with internationally recognized databases of vessel characteristics (such as Lloyd’s Register data), vessel trim and stability information, stowage plans and other cargo loading documents.

Comment:

None.

 

Ready Reserve Force (RRF) activation                                        Page 49

Measure:

1.      Percent of RRF no-notice activations that meet assigned readiness timelines. (FY) (2001)

2.      Percent of days that RRF ships are mission-capable while under DOD control. (FY 2001)

Scope:

DOD conducts no-notice exercises, called “Turbo-Activations,” annually to assess RRF activation readiness.  The USTRANSCOM, via MSC, randomly selects and orders the activation of a number of RRF ships on an annual basis to test their capability to be ready-for-sea (i.e., mission-capable) within their assigned readiness timeframes of 4, 5, 10, or 20 days.

Source:

MARAD maintains a database on the number of days it takes to activate each RRF ship and its operational reliability.  The MSC activation order is received either by phone call or message.  Documents produced during the no-notice activation period comprise the data source for determining the amount of time taken to activate each ship.  Non-performance time is based on the MSC Casualty Reporting (CASREP) system, which identifies casualties that are of a severity to prevent the ship from performing the mission.  These messages are passed from the ship's Captain to MSC, the Ship Manager, and MARAD.  The reliability of activated RRF ships is measured as the percent of days that RRF ships are mission-capable while under DOD control.  Mission-capability is determined, in part, by the number of days it takes to repair a ship that has become inoperative. For example, the low percent of mission capability in 1997 (95.2) was the result of one ship being out of service for 156 days while undergoing repairs.

Limitations:

None.

Statistical Issues:

Since the population of vessels covered by these measures often consists of a very small number of vessels (as low as 13 vessels in FY 2000), a large swing in results can occur from just one ship not being available on time or one ship having operational problems.

Verification & Validation:

The source of the activation data is the actual activation order from DOD to MARAD and the documents produced during a no-notice activation.  These fix the actual time of call-up and the time when the vessel is "Ready for Sea" (or tendered to MSC).  The Ready for Sea time is agreed to by MARAD and the on-board MSC representative and reported to DOD by official message.  The time taken to activate each ship is maintained in the ship’s logbook and in official DOD, MSC, and MARAD records.

The collection of data regarding mission capability under MSC operational control starts when MSC officially accepts delivery of RRF ships with date and time documentation.  The Captain of the ship reports all problems that are of a severity to prevent the ship from performing its mission to MSC, the Ship Manager, and MARAD.  The Captain also reports when the problem has been corrected.  This information is entered by MSC into its CASREP system.

 

Mariner availability                                                                      Page 49

Measure:

Of the mariners needed to crew combined sealift and commercial fleets during national emergencies, the percent of the total that are available. (FY) (2001)

Scope:

The pool of licensed and unlicensed mariners consists of mariners that have had sea service on U.S.-flag oceangoing merchant vessels over 1,000 gross tons within five years.  The mariner pool includes licensed and unlicensed actively sailing mariners and inactive mariners, employed shoreside, who have the necessary skills and retain the appropriate license/rating to operate sealift ships, defined by shipboard position and U.S. Coast Guard certification.  This pool is then compared to the DOD and commercial manpower requirements to determine sufficiency of the labor force.  Only oceangoing merchant vessels over 1,000 gross tons are considered because mariners on these vessels have skills required for emergency sealift operations.  The targets are based on a sealift operation that extends beyond 6 months, necessitating relief for the mariners who were sailing at the start-up of the operation.

Source:

U.S. Coast Guard Merchant Mariner Licensing and Documentation (MMLD) system.  The Coast Guard is the lead Federal agency for regulating, licensing, and documenting professional merchant mariners.  MMLD provides information on both actively sailing mariners and inactive mariners, including their skill level and seafaring employment.

Lloyd’s Maritime Information Systems.  MARAD obtains information to track the use of U.S.-flag commercial ships active in international trade and projects the size of the active, ocean-going, commercial fleet.  The size of this fleet has a direct correlation to the size of the commercial pool of mariners, based upon commercial crewing rules.

MARAD/DOT Mariner Survey. New for FY 2001, a random sample of mariners with current qualifications is now being surveyed, in conjunction with the Bureau of Transportation Statistics.  The Survey will provide a more accurate determination of the number of currently qualified mariners as well as information on mariner availability for sealift employment during national defense emergencies.

Limitations:

The size of the active and inactive mariner pool can be estimated from the MMLD.  MARAD integrates these data into its own system for analysis and reporting.  Because the MMLD does not contain all of the information on individual mariners contained in their paper records, and provides no information on the availability and willingness of individuals to accept a sealift position in an emergency, it does not provide sufficient assurance of mariner availability.

Statistical Issues:

The primary area of uncertainty lies within the MMLD system, which MARAD uses to define the population of available mariners.  While the accuracy of the data continues to improve as all licenses and documents are now on a five-year renewal cycle, gaps still exist in the database. Because the MMLD system was not designed to contact mariners, address and telephone information in the system is incomplete and out-of-date.  Also, operators of some large oceangoing vessels are not required to report mariner employment to the USCG, and evidence of sea service provided by individual mariners to fulfill requirements for upgrading their rating is not entered in the MMLD.

Verification & Validation:

The MMLD system is currently the only source of electronic data on mariner qualifications and employment.  MARAD continues to work with the USCG to improve the MMLD system.  The new MARAD/DOT Mariner Survey data will be used to estimate the number of qualified mariners available and willing to support sealift crewing positions.  Because this determination is a matter of individual choice and is subject to change, MARAD intends to develop a plan for maintaining current information on mariner availability based on the results of the Survey.

Comment:

None.

 

Drug interdiction                                                                          Page 50

Measure:

Amount of drugs seized or destroyed at sea (metric tons). (FY)

Scope:

Total amount of drugs (cocaine, marijuana, hashish, heroin, etc.) seized, jettisoned, or destroyed at sea by the United States Coast Guard.  Cocaine currently constitutes the largest drug threat to the U.S., but the Coast Guard seeks to interdict all illegal narcotics moving by non-commercial maritime conveyances. 

Source:

The amount of drugs seized is measured by Coast Guard crews and reported through the Coast Guard Law Enforcement Drug Interdiction Data Base.  Seizures are officially credited to the Coast Guard via Federal Drug Identification Numbers (FDINs) and are recorded in the federal Consolidated Counter-Drug Data Base, which is administered by the U.S. Interdiction Coordinator (USIC).

Limitations:

It is possible that non-entry, duplication, and coding errors are present in seizure amount data; however, the chance of this error is small. 

Statistical Issues:

None.

Verification & Validation:

Verification and validation occurs in several places in the data reporting and collection process.  Data entry software helps ensure data quality and consistency by employing selection lists and logic checks.  Internal analysis and review of published data by external parties help identify errors.  CG data is further reviewed at a quarterly Consolidated Counter-Drug Data Base Conference, where all agencies that input data into the database review all agency data for consistency and accuracy. 

Comment:

This measure aligns with the goals contained in the President’s National Drug Control Strategy. 

Measure:

Seizure rate for cocaine that is shipped through the transit zone. (FY) (2001)

Scope:

Seizure rate is a measure consisting of the amount of cocaine seized by the Coast Guard divided by the noncommercial maritime cocaine flow, expressed as a percentage.  Noncommercial is defined as any vessel or aircraft not engaged in port-to-port transfer of cargo/passengers for the primary purpose of business profit. Examples are pleasure craft, fishing vessels, offshore work-boats, or freighters carrying cargo as a means of disguising illegal drugs.

Source:

The amount of cocaine flow shipped by non-commercial means through the transit zone is estimated in the Interagency Assessment of Cocaine Movement (IACM) published by the Office of National Drug Control Policy (ONDCP).  The amount of cocaine seized is measured by Coast Guard crews and reported through the Coast Guard Law Enforcement Information System.

Limitations:

It is probable that non-entry, duplication, and coding errors are present in seizure amount data (numerator); however, this error is likely to be small.  The cocaine flow amount (denominator) is estimated through a complex process using many different sources of information. Due to the secretive nature of the illegal drug trade, cocaine flow estimates may contain significant errors.  The size of this error may vary from year to year; the extent of this is not known. The estimation process changes slightly each year as improvements are made, so year-to-year comparisons of the flow are not completely consistent. The accuracy of the official cocaine flow estimate has been questioned by some individuals and organizations outside of government that have an interest in U.S. drug policy.  ONDCP continuously attempts to refine this estimate to improve the measurement of interdiction activities.  This measure only addresses cocaine; formal flow assessments do not exist for other major drugs.  This measure is not designed to include cocaine shipped by commercial means such as large container vessels; however, it is probable that a small amount of cocaine included in the numerator is actually related to commercial shipping.  This distinction between commercial and noncommercial is better for program management; at-sea interdiction of commercially conveyed cocaine, particularly when shipped in containers, is extremely difficult, and not the focus of the Coast Guard program.   

Statistical Issues:

The primary source of uncertainty in estimating seizure rate for cocaine is the denominator.  Although the numerator estimate of cocaine seized is relatively accurate, the estimate of the amount shipped in the denominator is far more variable. The regression standard error for year-to-year chance variation in the cocaine seizure rate is 4.0 percent, based on data from 1995 through 2000.   

Verification & Validation:

Verification and validation occurs in several places in the data reporting and collection process.  Data entry software helps ensure data quality and consistency by employing selection lists and logic checks.  Internal analysis and review of published data by external parties help identify errors.

Comment:

This measure is consistent with the goals contained in the President’s National Drug Control Strategy.

 

Migrant interdiction                                                                      Page 50

Measure:

Interdict and/or deter at least 87 percent of undocumented migrants who consider attempting to enter the U. S. via maritime routes. (FY)

Scope:

Measure includes Cuban, Dominican, Haitian, and Chinese migrants, as these are the primary groups using maritime channels and the groups for which formal flow estimates exist.  Success rate is the estimated number arriving by maritime channels divided by those that pose a threat of migration (estimated intent).  The interdiction rate is just 1 minus the success rate.

Source:

Data is obtained from Coast Guard and from the Immigration and Naturalization Service (INS).  Estimates of migrants who successfully arrive and estimates of those with a high potential for undertaking the voyage are derived (with a consistent methodology) from investigations of incidents, interviews of detainees, and intelligence gathering.  Sources for this information are the Coast Guard, INS, and other authorities.

Limitations:

The numbers of illegal migrants entering the U.S., and the numbers of potential migrants, are derived numbers subject to estimating error.  Because of the speculative nature of the information used, and the secretive nature of illegal migration, particularly where professional smuggling organizations are involved, the estimated potential flow of migrants may contain significant error.  The measure only tracks four migrant groups at this time. A small number of migrants (approximately 10%) from various source countries are not included because formal flow estimates of migrants leaving these countries are not available.  Using the number of potential migrants in the denominator helps address the deterrence value of Coast Guard operations, but could lead to confusion of this measure with a simple interdiction rate. Trend information prior to 1995 is not available.

Statistical Issues:

The primary source of uncertainty in estimating the success rate for undocumented migrants is the denominator, which is an estimate of the flow of migrants, both documented and undocumented.

Verification & Validation:

The numbers of migrants reaching the U.S. via maritime routes and the number of “potential migrants” are estimated.  Methodologies and data are continuously reviewed.  The Coast Guard has developed the estimation techniques that support this indicator over the last six years in order to more consistently use intelligence information.  They are seeking independent assessment of the methods, and look to improve the process in the future.

Comment:

Partly because maritime threats of illegal migration have come from a limited number of sources, the Coast Guard and others have developed quantified threat estimates to better manage interdiction.  Over the past six years, estimation techniques have been improved to remove as much subjectivity and inconsistency as possible. It should be noted that past information reflects the success of intentional illegal activity.  While some DOT measures allow accurate projection of likely future outcomes, the highly variable nature of illegal migrant activity limits the ability to project future outcomes based on performance in the immediate past.

 

Critical transportation infrastructure protection                         Page 53

Measure:

Of those who need to act, percent who receive threat information within 24 hours. (FY) (2001)

Scope:

Threat information, in this context, is defined as credible information (both time-sensitive/action-oriented and informational) received by the Intelligence Community, analyzed by OIS and distributed in the form of a Transportation Security Information Report, generated by OIS for distribution by the Operating Administrations (OAs).  Figure is derived from the percentage of transportation security officials and industry representatives that receive threat information from OIS through the OAs within the 24-hour period.  Security representatives and officials will be randomly sampled by OIS within 48 hours of information dissemination and asked if and how soon they received the subject material. 

Source:

Internally prepared.  Survey conducted by OIS of both DOT personnel and industry security contacts.

Limitations:

Data:  Relies on the reporting of the customers and consumers of this information.  Reporting could be skewed to reflect positively on the dissemination process within the Operating Administrations.

Indicator:  This measure only identifies whether there are possible breakdowns and bottlenecks in the dissemination process.  It does not identify where those breakdowns specifically may be in the dissemination chain.

Statistical Issues:

Since these data are collected through a sample survey, they are subject to sampling and non-sampling errors.

Verification & Validation:

Customers will be randomly surveyed at all levels within the dissemination process, not solely the end users.  Consequently, the reporting of dissemination times and officials who are in receipt of the information can be crosschecked for verification and validity of data. 

Comment:

None.

 

 

 

 

Energy Efficiency                                                                          Page 55

Measure:

Transportation-related petroleum consumption (in quadrillion BTUs) per trillion dollars of Real Gross Domestic Product (GDP). (CY) (2001)

Scope:

Measure includes primary consumption of petroleum for transportation, in quadrillion BTUs.   This does not account for petroleum-produced electricity that is used in transportation; however, this is less than 1% of petroleum use.   Petroleum use is normalized to real GDP, in constant 1996 dollars.

Source:

U.S. Department of Energy Annual Energy Review 1999 and Annual Energy Review 2000.

Limitations:

Energy consumption does not include petroleum-produced transportation electricity.  Measure does not capture the fraction of this petroleum use that is imported, nor does it capture actual energy efficiency  (BTUs per passenger-mile-traveled).

Statistical Issues:

These data are external to DOT.  They are subject to both sampling and nonsampling errors.  Based on 1994-2000 data, chance variation from year to year in the transportation energy efficiency measure has a regression standard error of 0.016.

Verification & Validation:

Data is taken from external sources, which conduct their own verification and validation.

Comment:

Petroleum use is normalized to the nation’s real GDP in order to capture the nation’s economic exposure to petroleum use in transportation.  Beginning in 1999, the GDP baseline was changed from constant 1992 dollars to 1996 dollars.

 

 

Details on DOT Measures of Mobility and Economic Growth

Highway infrastructure condition                                                              Page 60

Measure:

Percentage of travel on the National Highway System (NHS) meeting pavement performance standards for acceptable ride. (CY)

Scope:

Data include vehicle miles traveled on the HPMS reported NHS sections and pavement ride quality data reported using the International Roughness Index (IRI).

IRI is a quantitative measure of the accumulated response of a "quarter-car" vehicle suspension experienced while traveling over a pavement.

Vehicle Miles of Travel (VMT) represent the total number of vehicle miles traveled by motor vehicles on public roadways within the 50 states and Washington, D.C.

Source:

Data collected by the State Highway Agencies and reported to FHWA for the Highway Performance Monitoring System (HPMS).  They are obtained from calibrated measurement devices that meet industry set standards.  Measurement procedures are included in the HPMS Field Manual.

VMT is a calculated product of the annual average daily traffic (AADT) and the centerline length of the section for which the AADT is reported. In the HPMS, travel is accumulated for each universe section to develop appropriate totals for the higher functional systems. AADT is required for each section of Interstate, NHS, and other principal arterial; as a result, travel is computed for these functional systems on a 100-percent basis. For minor arterial, rural major collector and urban collector systems, travel is calculated from samples using the AADT, centerline length reported for each sample section and the HPMS sample expansion factor for each section. Travel for the NHS on all functional systems is computed from the universe AADT data.

For the most part, travel for the rural minor collector and rural/urban local functional systems is calculated by the States using their own procedures and is provided in HPMS on a summary basis. Some States use supplemental traffic counts outside of the HPMS procedures; others employ estimating techniques, such as fuel use, to determine travel on these systems. In general, these methods are used in both rural and urban areas, including the donut areas of nonattainment areas to meet Clean Air Act requirements.

Limitations:

IRI data for the approved NHS exist from 1995 onward. Past data (1993 and 1994) contain some variation as this data was on the proposed, rather than the existing NHS. No NHS IRI data are available prior to 1993.  The HPMS requires States to report IRI data every two years; however, following the requirements is not mandated, but voluntary.

VMT estimates reported via the HPMS should be of reasonable quality particularly for the higher order functional systems. AADT and travel data are edited by the HPMS software for unusual values and for unusual changes to previously reported values. FHWA routinely works with State data providers to modify reported AADT values that do not appear to be reasonable before final use. Although AADT is required to be updated annually in HPMS, counts are only required to be updated on a 3-year cycle. For any reporting year, AADT for uncounted sections is usually derived by factoring the latest year's count for those sections.

Statistical Issues:

The major source of error in the percentages is probably the sampling error from selecting the segments of highway tested for smoothness. 

VMT data are subject to sampling errors, whose magnitude depends on how well the locations of the continuous counting locations represent nationwide traffic rates.  HPMS is also subject to estimating differences in the states, even though FHWA works to minimize such differences and differing projections on growth, population, and economic conditions which impact driving behavior.

Verification & Validation:

FHWA validates the data based on consistency reviews. States that follow the HPMS sampling instructions in developing traffic counting programs (Appendix F in the HPMS Field Manual) and the practices advocated in the Traffic Monitoring Guide have adequate counting and classification tools to prepare quality AADT and travel estimates for HPMS. The consistency of the sampling and counting procedures should also provide comparable State-to-State traffic data.

Comment:

None.

Measure:

Percentage of miles on the National Highway System (NHS) that meet pavement performance standards for acceptable ride. (CY) (2001)

Scope:

International Roughness Index (IRI) is compiled annually for every section of the NHS, using data reported from the States.

Source:

Data collected by the State Highway Agencies and reported to FHWA for the Highway Performance Monitoring System (HPMS).  They are obtained from calibrated measurement devices that meet industry set standards.  Measurement procedures are included in the HPMS Field Manual.

Limitations:

IRI data for the approved NHS exist from 1995 onward. Past data (1993 and 1994) contain some variation as this data was on the proposed, rather than the existing NHS. No NHS IRI data are available prior to 1993.  The HPMS requires States to report IRI data every two years; however, following the requirements is not mandated, but voluntary.

Statistical Issues:

The major source of error in the percentages is probably the sampling error from selecting the segments of highway tested for smoothness.  The annual variation in the percentage due to chance has a regression standard error of approximately 0.44 percent based on data from 1995-2000.

Verification & Validation:

FHWA validates the data based on consistency reviews

Comment:

None. 

 

Highway bridge condition                                                             Page 61

Measure:

Percentage of deficient bridges on the NHS. (CY) (2001)

Scope:

Measure includes the number of deficient (structurally deficient and functionally obsolete) bridges on the NHS functional system divided by the total number of NHS bridges in the inventory, expressed as a percent.  

Source:

Bridge information is collected by State DOTs and other bridge owners and provided to FHWA annually for inclusion in the FHWA maintained National Bridge Inventory (NBI).     

Limitations:

NBI includes information on all 114,567 NHS bridges.  States are required to update the system annually, but many States update quarterly.  The system contains 95 data items for each of the bridges, and 20 of these items relate to bridge condition and appraisal.  There are specific instructions as to how to assess bridges based on these items, including a grading scale from 0 to 9 with specific definitions and specific criteria to follow

Statistical Issues:

Even with the item specific grading system, differences in the grading between individual inspectors and between inspection days are probably the largest component of variation in the percentages.  Based on 1994-2000 data, the estimated regression standard error for year-to-year variation in the percentages due to chance is approximately 0.65 percent.

Verification & Validation:

DOT evaluates accuracy and reliability of the submitted NBI information through data checks and field reviews by both Headquarter and field office personnel.   This is done as a part of FHWA’s NBI, the National Bridge Inventory System (NBIS), and Highway Bridge Replacement and Rehabilitation Program.  Evaluation of the State’s compliance with the NBIS most often includes a sample of bridge inspection reports and a comparison of condition data with field visits to the bridge site.  In addition, there is an edit update program that identifies potential data errors in the NBIS.

Comment:

None.

 

Appalachian highway system                                                       Page 61

Measure:

Miles of the Appalachian Development Highway System (ADHS) completed. (FY) (2001)

Scope:

Measure includes actual miles completed on the 3,025 mile ADHS, within 13 member States.

Source:

States submit annual status updates on ADHS miles completed within their State each fiscal year to the Appalachian Regional Commission (ARC).  The ARC compiles the data.

Limitations:

This is an output measure.

Statistical Issues:

None.

Verification & Validation:

Completed by ARC.

Comment:

ARC estimates that the TEA-21 funding level will result in completion of approximately 37 additional miles each FY 1999 through 2003.

 

 

 

Highway congestion                                                             Page 60 & 61

Measure:

1.  Of total annual urban-area travel, percentage that occurs in congested conditions (CY)

2. Of annual urban-area peak period travel time, additional percentage of travel time attributable to congestion, (CY) (2001) and

3.  For the individual traveler in urban areas, average annual hours of extra travel time due to delays. (CY) (2001)

Scope:

Scope: Data for the three measures below stemmed from approximately 400 urban areas.  The data reflects the travel conditions of the freeway and principal arterial street networks.

Definitions:

1.       Urban area:  Developed area with a density of greater than 1000 persons per square mile.

2.       Congested travel:  Traveling below the posted speed limit(s).

3.       Peak Periods:  (Applicable to Travel Time Measure only.  Congested Travel and Traveler Delay represent daily travel.)  Monday-Friday morning and evening rush hours when slow speeds (below posted speed limits) are more likely to occur.  The length of peak periods varies, e.g., large urban areas are typically longer.  The Travel Time Measure accounts for the variations.

4.       Delay:  Extra travel time due to traffic volume and /or incidents.

Source:

Data collected and provided by the State Departments of Transportation from existing State or local government databases, including those of Metropolitan Planning Organizations.  The Federal Highways Administration’s Highway Performance Monitoring System serves as the repository of the data.  The Texas Transportation Institute utilizes HPMS data to derive the above measures.  

Limitations:

We have gathered data up through 2000.  We anticipate having 2001 data on/about Nov. 2002.  The proportion of congested travel figures used in calculating the measures are computed rather than measured values.  The computed values may understate congestion, as delay from incidents is not calculated.  Performance evaluation is process-oriented.  Transportation programs that help combat highway congestion possess outcome-oriented, objective methods within the specific program areas; however, the causal relationship between the programs and overall highway congestion is inconclusive.

Statistical Issues:

Methodology used to calculate performance measures has been developed by the Texas Transportation Institute and used in their annual Mobility Study.  A detailed description of TTI’s methodology is best described on their website at

http://mobility.tamu.edu/

.

Verification & Validation:

State-reported HPMS data are reviewed by FHWA for completeness, consistency, and adherence to reporting guidelines.  When necessary, and with close State cooperation, data may be adjusted to improve completeness, consistency, and uniformity.

Comment:

In the FY 2000 Performance Plan, we used hours of delay per 1,000 vehicle miles traveled (VMT) to measure this goal.  This metric attempted to provide a system-wide measure of congestion.  However, it represented only one dimension of congestion – delay -- and did not effectively reflect the actual performance of the highway system in places where congestion regularly happens, i.e., the measure showed delay decreasing nationwide when in fact congestion was worsening in urban areas.  Moreover, the measure was difficult to interpret by the general public.  Based on discussions with our partners and customers, we replaced this indicator with three new measures:  Congested Travel, Travel Time, and Traveler Delay.  Together, these new indicators will reflect changing travel conditions more comprehensively by focusing on three different aspects of inefficient road performance in a broad collection of urban areas across the nation where congestion regularly occurs.  The data supporting the three new measures stem from the Highway Performance Monitoring System (HPMS).  The availability of the data is approximately 9 months from the base year, e.g., 2001 actual numbers will not be available from HPMS until Sep/Oct 2002.  To accurately and reliably manage the transportation system, real-time (minute-by-minute) measurement of system speeds is needed and can only be achieved with automated instrumentation.  As the Intelligent Transportation System network is put in place, reliability will become a barometer of this strategic goal. Ten cities have been identified with sufficient instrumentation to permit the development of a reliability measure.  This is a first step in migrating from HPMS data to real-time, ITS-based data.

 

Intelligent Transportation Systems integrated deployment       Page 61

Measure:

Number of metropolitan areas where integrated ITS infrastructure is deployed. (FY) (2001)

Scope:

The level of integrated deployment in 75 of the nation’s largest metropolitan areas has been established using a set of indicators that consider two factors: (1) How much ITS infrastructure is in place at each metropolitan area; and, (2) How much integration is going on at each area.  The process for determining the level of “component” deployment in a metropolitan area employs a set of indicators that measure the magnitude of deployment for selected ITS components.  These are typically expressed as a ratio of actual deployment divided by the total possible, for example the number of freeway miles under electronic surveillance divided by the total freeway mileage.  Components are considered deployed once the level of deployment attains a specified threshold level based on the indicators.  Integration is defined as the sharing of data between agencies associated with the different jurisdictions responsible for ITS infrastructure.  Typically there are three: State DOTs responsible for management of freeways and incident management programs; city governments, which manage most of the traffic signal systems; and public transit authorities, which manage most bus and rail services.  The level of integration is determined by the extent that these three major transportation organizations employ technology to share and use transportation data to increase system capacity.  Two examples of integration are: 1) a city traffic signal system receiving data from the state freeway management center about the queues at freeway ramp meters and then adjusting the signal timings on the arterial streets, or 2) a transit agency providing the state freeway management center with the real-time location of the buses so that freeway speeds can be determined.  Metropolitan areas are rated as low, medium, or high separately for deployment and integration and then assigned an overall combined rating.  An overall score of medium or high meets the goal for a metropolitan area.

Source:

Metropolitan ITS Deployment Tracking Database developed by the Oak Ridge National Laboratory for the ITS Joint Program Office.  Data are collected by means of surveys from designated metropolitan areas.

Limitations:

This indicator is designed to track and encourage basic steps toward component deployment and systems integration.  However, it does not reflect the full breadth of deployment or integration activities.  For example, while it establishes the existence of basic integration of essential components, it does not confirm that all possible or desirable integration links exist in a metropolitan area.  Similarly, the attainment of a deployment threshold only confirms a substantial commitment to the use of ITS technology but does not indicate that all needed deployment is complete. 

Statistical Issues:

These data come from sample surveys that, like all sample surveys, contain sampling and non-sampling errors.

Verification & Validation:

The DOT Joint Program Office reviews deployment tracking indicators and methodology.  Results are distributed to DOT headquarters and field staff as well as to state and local survey responders for confirmation of accuracy and completeness before the final reports are issued.  Independent experts in statistics and transportation review procedures for survey construction and data collection prior to each survey iteration.  A steering committee of Federal, state, and local transportation officials review and approve changes to methodology and indicators prior to implementation.

Comment:

The FY 1997 baseline is 36 areas.

 

 

 

Transit ridership                                                                  Page 66 & 67

Measure:

1.  Billion transit passenger miles traveled. (CY) (2001)

2.  Average percent change in transit passenger-miles traveled per transit market. (FY)

Scope:

Includes revenue-passenger miles on publicly sponsored bus, transit rail, commuter rail, ferry, and vanpools in urbanized areas.

Source:

National Transit Database (NTD), with information gathered from transit operators.

Limitations:

Data is self-reported by transit agencies using an FTA-approved sampling methodology.  Although most data is reported in the National Transit Database each year, sample cycles may be annual, every three years, or every five years depending on the size of the urban area and the number of vehicles operated.  Ridership is an outcome indicator that reflects a variety of factors, including the capital investment of the Federal Government.  Ridership is also influenced by operational decisions of transit authorities, and the availability and cost of alternative modes of transportation.

Statistical Issues:

The sources of uncertainty include sampling error, annual chance variation, and auditing issues.  The regression standard error from 1994-2000 indicates that the magnitude of the combination of the first two sources of error is approximately 0.67.

Verification & Validation:

An independent auditor and the transit agency’s CEO certify that data reported to the NTD are accurate.  FTA also compares data to key indicators such as vehicle revenue miles, number of buses in service during peak periods, etc.

Comment:

None.

Measure:

Percentage of transit grants obligated within 60 days after submission of a completed application.

Scope:

FTA grants obligated during a fiscal year period for major programs: Urbanized area, non-Urbanized area, and Elderly and Persons with Disabilities formula grants; Capital grants; Job Access and Reverse Commute grants; Over-The-Road Bus grants; and Planning grants.

Source:

FTA TEAM database.

Limitations:

Several factors that contribute to grant delays are beyond FTA’s ability to control. These factors include the processing of flexible funds from FHWA through the Treasury, and the Congressional grant release process.

Statistical Issues:

Processing time is calculated from submission date to obligation date. $0 dollar non-funding grant amendments are excluded from analysis.

Verification & Validation:

TEAM output file is crosschecked against other system generated files for consistency; inconsistencies are investigated and corrected prior to reporting. Grants with missing or out-of-sequence dates are excluded for calculating averages.

Comment:

An FTA task force meeting was held in February 2002 to identify causes for grant processing delays. The resulting action plan is now being circulated for final review and approval. Implementation of the plan will follow.

 

 

 

Aviation Delay                                                                              Page 68

Measure:

Percentage of on-time flights. (FY)

Scope:

The time of arrival of completed, scheduled passenger flights to and from the 32 DOT large-hub airports is compared to their scheduled time of arrival.  The sum of flights arriving on or before 15 minutes of scheduled arrival time is divided by the total number of completed flights.

Source:

The Aviation System Performance Metrics (ASPM) database, maintained by the FAA’s Office of Aviation Policy and Plans.  By agreement with the FAA, ASPM flight data is filed by certain major air carriers for all flights to and from 21 large and medium hubs, and is supplemented by flight records contained in the Enhanced Traffic Management System (ETMS) and flight movement times provided by Aeronautical Radio, Inc. (AIRINC).  Data are sufficient to complete ASPM data files for 49 airports.

Statistical Issues:

There is little major error in the count of completed flights or the count of flights that arrive on-time.

Limitations:

Some ASPM data is constructed from ETMS records, a small portion of which may not be maintained in FAA traffic control computers when they are under heavy use.  

Verification & Validation:

Flight data is extracted from the Official Airline Guide (OAG) and compared to& data from carrier records, which contains carrier computer reservation flight schedule data.  Summary data is compared to data filed monthly with DOT under 14 CFR Part 234, Airline Service Quality Performance Reports, which separately requires reporting by major air carriers on flights to and from the 32 large hubs.

Comment:

FAA’s percentage of flights arriving on-time derived from ASPM data differs only by fractions of a percent from the on-time percentage derived from DOT’s slightly different database. 

Measure:

Aviation delays per 100,000 activities. (FY) (2001)

Scope:

An FAA reported delay occurs when an aircraft is delayed fifteen minutes or more because of constraints that prevent the aircraft from making a scheduled landing.  Delays are counted in five categories: FAA equipment, volume, weather, runway related, and other.  Delays due to airline equipment are not considered. “Activities” are total facility activities, as defined in Aviation System Indicators 1998 Annual Report.  Total facility activities are the sum of en route and terminal facility activities.

Source:

FAA air traffic facilities report the data to headquarters, which incorporates the data into the Air Traffic Operations Management System.

Statistical Issues:

There is no major error in either the delay counts (numerator) or the flight operations data (denominator) for this rate.  However, random variation in aviation delays results in a significant variation in the delay rate from year to year. The regression standard error in the delay rate, based on 1994-2000 data, is approximately 20.0.

Limitations:

By collecting information on delays of fifteen minutes or more, FAA does not capture the aggregate amount of system delay, but only the most significant delays.

Verification & Validation:

Data is analyzed and checked by an Air Traffic Service headquarters office on a daily basis to ensure accuracy of the information reported.

Comment:

Total delays in all five categories are what the traveling public experience.

 

 

Airport and en route efficiency improvements                            Page 69

Measure:

Cumulative increase in throughput during peak periods at certain major airports. (FY) (2001)

Scope:

This measure focuses on the arrival rates during peak traffic periods comparing pre-CTAS rates to post CTAS rates.

Source:

Radar system (HOST and ARTS) data is collected and aircraft flight tracks are obtained from those systems and analyzed to determine arrival and departure times. 

Limitations:

The radar systems produce very large data files requiring a substantial effort to extract relevant data for analysis. The extracted data sets need to be of sufficient size to produce statistically significant results.

Statistical Issues:

Conditions (weather, runways in use, aircraft mix) vary, affecting rates. Data must be normalized and data sets must be of sufficient size to produce valid results.

Verification & Validation:

Methodologies and detailed results are available for review in semi-annual FFP1 Metrics Updates (December and June).  Results are coordinated with FAA and User stakeholders.

Comment:

None.

Measure:

Cumulative increase in direct routings for en route flight phase. (FY) (2001)

Scope:

This measure focuses on the number of direct routings provided by en route controllers comparing pre and post-URET installation.

Source:

URET provides data on routing amendments, which is then analyzed to determine the number of direct amendments. 

Limitations:

The radar systems produce very large data files requiring a substantial effort to extract relevant data for analysis. The extracted data sets need to be of sufficient size to produce statistically significant results.

Statistical Issues:

Extreme weather conditions, particularly thunderstorms, will significantly affect routing amendments; therefore, data is sampled for days when weather is not a factor.

Verification & Validation:

Methodologies and detailed results are available for review in semi-annual FFP1 Metrics Updates (December and June).  Results are coordinated with FAA and User stakeholders.

Comment:

None.

 

Runway pavement condition                                                        Page 70

Measure:

Percent of runways in good or fair condition (commercial service, reliever, and selected general aviation airports). (CY) (2001)

Scope:

Paved runways at the 3,300+ airports in FAA’s National Plan of Integrated Airport Systems (NPIAS) are assessed for pavement condition.  The NPIAS airports include all commercial service and reliever airports and those general aviation airports that are significant to national air transportation.

Source:

The FAA’s Airport Safety Data Program (ASDP) provides extensive data about the facilities that are available at public-use airports.  Data are provided approximately annually by FAA inspectors for airports certified under FAR 139.  Data for other airports, including most public use general aviation airports, are provided under an FAA contract.

Limitations:

FAA contracts for a visual survey of the runways to categorize their condition based on criteria developed by the FAA Office of Airports.  “Good” condition means all cracks and joints are sealed; “fair” condition means there is mild surface cracking, unsealed joints, and slab edge spalling; and “poor” condition means there are large open cracks, surface and edge spalling, and vegetation growing through cracks and joints.  Since the reports are based on a visual inspection, underlying drainage or strength problems are not reported.  However, these problems normally create surface defects that are visible.  The more detailed pavement condition index (PCI) inspections require a section-by-section examination of the runway rather than an overall assessment used for this performance measure. FAA has been aggregating the ADSP data from all NPIAS airports only every several years for inclusion in the NPIAS report to Congress.  This information exists for 1993, 1997, and 1998.

Statistical Issues:

Less than half of the ADSP records were updated during CY 2000.  The relatively subjective nature of judging pavement quality means this measure is also subject to random variation due to measurement error.

Verification & Validation:

Efforts continue to correlate PCI and ADSP data.

Comment:

A contract was initiated in FY 2001 to coordinate efforts by state agencies to conduct safety inspections at selected general aviation airports.

 

All Weather Access to Airports                                                    Page 70

Measure:

Number of runways that are accessible in low visibility conditions. (FY) (2001)

Scope:

This performance measure counts the total number of airport runways with published ground-based and/or satellite-based landing systems.  The intent of this measure is to reflect increased accessibility using satellite-based technology for vertically guided approaches.

Source:

Internal FAA Aviation Systems Standards tracking system.

Limitations:

Increasing the number of runways with satellite- based landing systems as well as augmenting existing satellite-based landing systems with vertical altitude guidance will improve access to airports and increase schedule reliability.   Both improved access and increased reliability are considered benefits to the aviation industry and the individual air traveler.  However, individual use of landing systems is not tracked by current FAA information systems.  In addition, aircraft must be appropriately equipped to use the new technology.  The FAA does not track these equipment additions. 

Statistical Issues:

There is no major error in the counts of published landing systems.  However, like the above measure, random changes in the number of published approaches result in random variation in the count from year to year.

Verification & Validation:

The number of airport runways with a satellite-based landing system is computed monthly by Aviation Systems Standards.

Comment:

None.

 

 

 

 

 

Maritime navigation                                                                     Page 72

Measure:

Total number of commercial vessel collisions, allisions, and groundings. (FY)

Scope:

The measure includes all commercial ships regardless of tonnage. Intentional groundings are excluded.  “Allisions” refers to incidents wherein ships collide with a fixed object such as a bridge or aid to navigation.

Source:

Coast Guard Marine Safety Information System (MSIS). Sources of reports are most often vessel masters, operators, owners, insurance companies, legal representatives, and other mariners. Collisions, Allisions, and Groundings are reported to the Coast Guard as required by federal regulations.

Limitations:

The investigation, retrieval, analysis and reporting processes result in under-reporting for the most recent year, with the most significant effects over the most recent 5 months.  Estimates are often used to compensate for this known data-lag.  It is probable that some collisions, allisions and groundings are not reported to the Coast Guard. This number is unknown.  Serious events such as major collisions and hard groundings are more likely to be reported than minor events such as a temporary grounding where a vessel could remove itself without assistance.  Duplicate event entries are sometimes entered into MSIS, and some events are mistakenly omitted or coded incorrectly.  Verification procedures strive to correct these errors, but it is probable that a small number are not corrected. Because this count of incidents is not normalized to exposure, it does not provide a sensitive indicator of changes in risk.

Statistical Issues:

The major sources of uncertainty in these measures are the estimation error (as a result of the data-lag) the response error (as a result of parties failing to report casualties to the Coast Guard), and recording error (based on differences in the training and judgment of Coast Guard investigators in recording the accident). The regression standard error for year-to-year chance variation in the number of collisions, allisions and groundings under the new measure is approximately 70, based on data from 1996 through 2000.

Verification & Validation:

Verification and validation occur at several levels.  Edit checks within MSIS software can detect some incorrect or missing data and force review and correction before data entry is completed.  Selection lists for certain data fields also reduce the opportunity for data entry error.  All investigations go through one level of review at the field unit for accuracy.  Investigations of serious marine casualties are also usually reviewed at district and headquarters offices.  The headquarters Data Administration staff conducts periodic quality control checks to identify entry errors such as missing data or miscoding, and corrects any errors identified.  Each investigation involving a vessel accident is reviewed before it is included in the measure.  Errors identified are referred to either the Data Administration staff or the Investigations and Analysis staff for correction.

Comment:

During FY 2002, the Marine Safety Information System (MSIS) will be replaced by the Marine Information System for Safety and Law Enforcement (MISLE).  While the new system will be a substantial improvement, it is expected to cause serious difficulties in making performance comparisons.  One factor is that many business processes were re-designed in conjunction with system development.  Another factor is that data quality under MISLE is expected to be superior to that of MSIS.  While this represents improvement, it may cause near-term problems in making meaningful comparisons of data between the two systems.

 

 

 

 

 

St. Lawrence Seaway system availability                                    Page 72

Measure:

Percentage of days in the shipping season that the U.S. sectors of the St. Lawrence Seaway locks are available, including the two U.S. Seaway locks in Massena, N.Y. (CY)

Scope:

The availability and reliability of the U.S. sectors of the St. Lawrence Seaway, including the two U.S. Seaway locks in Massena, N.Y., are critical to continuous commercial shipping during the navigation season (late March to late December).  System downtime due to any condition (weather, vessel incidents, malfunctioning equipment) causes delays to shipping, affecting international trade to and from the Great Lakes region of North America.  Downtime is measured in minutes/hours of delay for weather (visibility, fog, snow, ice); vessel incidents (human error, electrical and/or mechanical failure); water level and rate of flow regulation; and lock equipment malfunction.

Source:

SLSDC gathers the data from internal Lock Operations records.

Limitations:

As the agency responsible for the operation and maintenance of the U.S. portion of the St. Lawrence Seaway, SLSDC’s lock operations unit gathers primary data for all vessel transits through the U.S. Seaway sectors and locks, including any downtime in operations.  Data is collected on site, at the U.S. locks, as vessels are transiting or as operations are suspended.  This information measuring the System’s reliability is compiled and delivered to SLSDC senior staff each month.  In addition, SLSDC compiles annual System availability data for comparison purposes.  Since SLSDC gathers data directly from observation, there are no limitations.

Statistical Issues:

None.

Verification & Validation:

SLSDC verifies and validates the accuracy of the data through review of 24-hour vessel traffic control computer records, radio communication between the two Seaway entities and vessel operators; and video and audiotapes of vessel incidents.

Comment:

SLSDC influences the measure primarily through capital planning, and consistent facilities maintenance and investment.

 

Domestic Icebreaking                                                                   Page 73

Measure:

Days critical waterways are closed due to ice. (FY) (2001)

Scope:

Seven waterways are designated critical to icebreaking on the Great Lakes based on historical ice conditions, volume of traffic, and potential for flooding due to ice dams on rivers.  The Coast Guard measure is the number of days critical waterways are closed for more than 24 hours due to ice.

Source:

Data comes from U.S. Coast Guard and U.S. Army Corps of Engineers observations.  Waterways closure data is reported to the Ninth Coast Guard District by operating units via operational situation reports.

Limitations:

The data set associated with this measure is relatively small and simple; hence it tends to be fairly accurate.  However, it is possible that small errors exist. This measure captures only Great Lakes winter navigation, and not all domestic icebreaking. The observation of closures in critical waterways is a surrogate for mobility over the whole Great Lakes waterway system.

Statistical Issues:

This particular performance measure is highly sensitive to the severity of winter weather, which will dramatically affect the ability to predict the number of days the waterways are closed due to ice.  The Coast Guard expects a lower rate of waterways closures due to ice during mild winters and a corresponding higher rate of waterways closures during severe winters.  The Coast Guard uses a standard severity index (based on average temperatures) to measure the severity of winter weather  (–6.2 or milder defines average severity; less than –6.2 defines severe, e.g. –6.5). The term “waterway closure” is also subject to differences in definition by districts or sub-units reporting the data.

Verification & Validation:

Coast Guard district program managers review and validate data from situation reports and provide Coast Guard headquarters with an End of Season report. 

Comment:

Great Lakes data reflect initial measurement methodology.  Further refinements are being explored that will make this measure a more comprehensive gauge of winter navigation.

 

Transportation accessibility                                                         Page 75

Measure:

1.      Percentage of bus fleets that are Americans with Disabilities Act (ADA) compliant. (CY)

2.      Percentage of key rail stations that are ADA compliant. (CY)

Scope:

Accessibility for bus fleet means that vehicles are lift or wheel chair ramp equipped.   Accessibility for key rail facilities is determined by standards for ADA compliance.

Source:

Data on bus accessibility is collected in the National Transit Database (NTD), with information gathered from transit operators.  Data on rail accessibility is reported to FTA by the transit authorities.

Limitations:

Measure does not capture ADA compliance (or transportation accessibility) for modes other than transit.

Statistical Issues:

None.

Verification & Validation:

For bus accessibility, an independent auditor and the transit agency’s CEO certify that data reported to the NTD are accurate.  Data are also compared with fleet data reported in previous years, and crosschecked with other related operating/financial data in the report.  Fleet inventory is reviewed as a part of FTA’s Triennial Review, and a visual inspection is made at that time.  FTA’s Office of Civil Rights conducts oversight reviews in order to verify the information on key rail station accessibility which has been self-reported by the transit authorities.

Comment:

FTA will primarily influence the goal through Federal transit infrastructure investment, which speeds the rate at which transit operators can transition to ADA-compliant facilities and equipment.

 

Access to jobs                                                                               Page 75

Measure:

Number of employment sites that are made accessible by Job Access and Reverse Commute transportation services. (FY) (2001)

Scope:

This measure assesses one part of the Job Access and Reverse Commute program – the number of employment sites made accessible that were not previously accessible.  An employment site is considered accessible if located within 1/4 mile of services provided by the grantee.  Employment sites must offer jobs that require a high school diploma or less.  Services that make an employment site accessible may include, but are not limited to, carpools, vanpools, and demand-responsive services as well as traditional bus and rail public transit.  The measure cannot account for those Job Access and Reverse Commute activities that encourage riders to use already existing sources of public transit. 

Source:

Data are provided to FTA by grantees of the Job Access and Reverse Commute program in their quarterly progress reports.

Limitations:

The goal and measurement is the primary evaluation measure aimed at capturing results of the Job Access and Reverse Commute program.  Three elements are key to job access – the residence of the employee, the commute, and the job location.  This measure includes the “goal” of the commute and the job, but it does not include the “starting line” of the commute, the rider’s home.  Although jobs may be made more accessible to transportation services, these services may not provide access to potential workers’ communities.  This measure also cannot account for improved accessibility due to lower fares or shorter commute times – it only addresses the gap in service delivery.  FTA requires a greater level of precision from larger, urban grantees than rural grantees that may have fewer resources at their disposal.

Statistical Issues:

There are major problems in obtaining accurate estimates of the number of entry-level jobs within a quarter-mile of grantee services.  Surveys are costly and prone to systematic biases.  The uncertainty in this estimate is both large and difficult to quantify.

Verification & Validation:

FTA will use an oversight contractor to verify reported information on a sample basis.

Comment:

None.

 

International air service                                                               Page 78

Measure:

Number of passengers (in millions) in international markets with open skies aviation agreements. (FY)

Scope:

These data are collected by DOT for all flight segments to/from a U.S. point. The data for this measure include all passengers on U.S. and foreign carrier flights to and from 47 “open-skies” countries and Canada.  This indicator reflects (barring significant, unrelated macroeconomic and political influences) the extent to which the competitive environment promoted by DOT increases travel opportunities.

Source:

U.S. air carriers file domestic and foreign data in the T-100 system.  Foreign carrier data are from the T-100F database.  Foreign air carriers file data for all nonstop flight segments involving a U.S. point. 

Limitations:

These data are considered a reliable measure of airline passenger traffic between the U.S. and foreign nations.  The annual increase in air traffic, however, is affected by economic strength as well as market liberalization in bilateral aviation trade agreements.  Furthermore, only part of the growth rate in open skies markets can be attributed to new traffic – some of the increase may reflect diversion of traffic from less competitive routes with higher taxes and/or inferior service options.  The goal of 3% annual growth reflects aviation analysts’ judgment of the net impact of these agreements above the estimated growth expected in the industry.  For these reasons, this goal must be considered more of a forecast than a "target." 

Statistical Issues:

Like other counts of aviation-related activities, there are no major sources of systematic error in these data that have been quantified.  However, random variation in the number and distribution of airline passengers, as well as the changes in the number of "open-skies" agreements, results in variation in the measure over time.  The regression standard error in this variation for 1994 through 2000 is 2.20.

Verification & Validation:

Airlines are required to certify that these data are accurate.  Also, these data are a 100% enumeration of traffic and capacity and can be verified for reasonableness against other databases, such as flight schedules.

Comment:

U.S. policy has favored the linking of networks.  Networks allow improved service and marketing in many thousands of small city-pair markets.  All of this traffic flows over flights captured by the T-100 and T-100F reports for international flights.

 

 

Essential air service                                                                      Page 80

Measures:

1.      Percent of subsidized communities with at least 2 round trips/day, 6 days/week (12 round trips/week).  (This measure will be discontinued after FY 2001.) (FY) (2001)

2.      Percent of subsidized communities with at least 3 round trips/day, 6 days/week (18 round trips/week). (FY) (2001)

Scope:

The measure shows the number of weekly round trips at subsidized EAS communities in the continental U.S.  EAS communities are those that were on the certificated airline map in 1978.

Source:

Air carrier filings, airport managers and community officials.

Limitations:

Service frequency is closely associated with program funding levels and the number of EAS communities that require subsidy; the number of communities may change.  Service frequency may also be affected by conditions such as an air carrier going out of business, airline strikes, or carrier shutdowns.  DOT’s goal assumes a fairly constant level of communities in the base (76 in 1998).   This measure will not show instances in which the Department is successfully able to effect a carrier transition to commercially viable service without a subsidy.  Data has only been gathered for 1996 and later years.

Statistical Issues:

There is no major error present in the subject data.

Verification & Validation:

Continued contact with airport and civic parties, carrier officials, and Congressional staffs.

Comment:

Consideration of alternate strategies or performance measures may be prompted by developments such as budget constraints and the makeup of commuter’s aircraft fleet.

 

Commercial shipbuilding                                                              Page 81

Measure:

Gross tonnage (in thousands) of commercial vessels on order or under construction in U.S. shipyards. (CY) (2001)

Scope:

Includes all commercial self-propelled vessels 100 GT or larger that are on order or under construction (i.e., the orderbook) in the United States, as of December 31.  Vessels such as drill rigs and inland barges are not included in these figures.

Source:

In addition to MARAD’s compilation of data, information is drawn from commercial suppliers of worldwide vessel characteristics data.  These include Lloyd’s Register of Shipping (marketed through Lloyd’s Maritime Information Services), Clarkson’s Research Service, and Fairplay.

Limitations:

No single commercial supplier of vessel data has complete information on shipyard orders and construction activity in the U.S.  None of the major data suppliers collect information on non-self-propelled vessels.  In 1998, MARAD began direct semi-annual shipyard surveys.  However, as the overall response rate was about 40 percent and did not produce any significant increase in either the quantity or quality of the data, MARAD is seeking alternative methods to obtain this data.  The commercial sources used are the best available, and consequently the data reported represents an amalgam of their collection efforts.

Statistical Issues:

One anomaly with the data is a gap in the statistics for vessels between 100 and 1,000 GT.  Only Lloyd’s data provides data in this category, but their data does not cover the full spectrum of vessels.  Orderbook data on December 31 of each year represents information available at that time and may not reflect complete information.

Verification & Validation:

MARAD compares information obtained from the different data sources to verify its accuracy.

Comment:

It has become evident that the available data does not adequately measure the value or complexity of the commercial shipbuilding program; therefore, MARAD plans to develop a new goal and measure.

 

Transportation and education                                                      Page 82

Measure:

Number of students graduating with transportation-related advanced degrees from universities receiving DOT funding. (SY) (2001)

Scope:

University Transportation Center (UTC) data includes recipients of Masters and Ph.D. degrees in programs considered to be transportation related. 

Source:

UTC data to be derived from university records provided to RSPA as part of the UTCs’ grant application.

Limitations:

While baseline data has been obtained for the UTC program, no data currently exists for other education programs that can result in graduate degrees.

Statistical Issues:

There is a possibility of undercounting, due to difficulty in specifying degree programs that are transportation-related.  Additionally, some universities may not fully comply.

Verification & Validation:

Comparison with data reported for all degree programs by host universities and specific reports on each recipient of an advanced degree.

Comment:

None. 

Measure:

Cumulative number of students (in thousands) reached through the Garrett A. Morgan Technology and Transportation Futures Program. (SY) (2001)

Scope:

Includes students of all ages reached through specific activities such as internships, job shadowing, career days, video conferences, classroom visits, and teacher externship visits that inform them of the opportunities available in the transportation field and ensure that they have the skills and knowledge required for transportation jobs.

Source:

RSPA maintained database to aggregate responses from program organizers.

Limitations:

The inherent nature of this measure does not allow us to gauge the quality of contacts made with students “reached” or provide a means to track outcomes in terms of students entering the transportation field as a direct result of the activities sponsored through the Garrett A. Morgan Technology and Transportation Futures Program.

Statistical Issues:

Some variability is inevitable in classroom attendance counts, videoconferences, and other measures of exposure.  But this uncertainty should be small.

Verification & Validation:

RSPA works to ensure that the quantitative data being reported is complete and accurately reflects the associated student activity before it is entered into RSPA’s database.

Comment:

None.

 

Amtrak ridership                                                                          Page 83

Measure:

Millions of passengers on Amtrak’s intercity routes. (FY) (2001)

Scope:

The measure includes all revenue paying passengers on intercity routes.

Source:

Amtrak Annual Report and Amtrak’s Monthly Train Earnings Report.

Limitations:

Data collection relies on accuracy of Amtrak report.  Ridership is an outcome indicator that reflects a variety of factors, not insignificantly the capital investment of the Federal Government.  Operational decisions of Amtrak and the availability and cost of alternative modes of transportation also influence ridership. 

Statistical Issues:

Chance variation from year to year, as estimated by the regression standard error from 1994-2000, is 0.81.  This calculation assumes stable operations over the seven-year period; since new runs and lines are added and removed fairly often, the standard error is only a rough approximation.

Verification & Validation:

Amtrak conducts monthly verification and validation of data.

Comment:

A 3.6 million increase in ridership was projected from 1998-2001 as a result of the initiation of the Northeast Corridor high-speed rail service. 

 

Impediments to port commerce                                                   Page 58

Measure:

Percentage of ports reporting landside and waterside impediments to the flow of commerce. (FY) (2001)

Scope:

81 U.S. deep and shallow draft ports.

Source:

Informal telephone surveys of some port officials.

Limitations:

The informal surveys did not encompass all of the intended ports within the scope of this measure.  These surveys were not scientifically rigorous and the questions asked varied from one region of the country to another.

Statistical Issues:

(See Verification and Validation section.)

Verification & Validation:

Impediments data was incomplete and inconsistent.  After reexamining the available data and the methods for obtaining it, MARAD has concluded that these data do not provide any valid indication as to whether the goal was met or not.  MARAD was not successful in clearing up inconsistencies or filling in data gaps.

Comment:

MARAD has also reached the conclusion that MARAD programs do not have a measurable impact in reducing impediments at U.S. ports.  MARAD efforts in this area are limited in scope to facilitating dialogue between stakeholders in the Marine Transportation System or technology demonstrations at one or two ports.  Therefore, this measure will no longer be used.

 

Transit system condition                                                              Page 84

Measure:

1.      Average condition of motor bus fleet (on a scale of 1 (poor) to 5 (excellent)). (CY) (2001)

2.      Average condition of rail vehicle fleet (on a scale of 1 (poor) to 5 (excellent)). (CY) (2001)

Scope:

The measure includes bus, demand response, and rail fleets.

Source:

National Transit Database (NTD), with information gathered from transit operators; Transit Economic Requirements Model (TERM), which estimates average vehicle condition using NTD data.

Limitations:

Average vehicle condition may not fully reflect the average condition that transit passengers face, since vehicles in worse condition tend to be utilized less. There are also lags in reporting of data to the NTD (thereby requiring preliminary estimates for recent years) and in the effects of federal government capital assistance (since it may take five years from the time that funds are appropriated to the time that new or rehabilitated vehicles are placed in service)

Statistical Issues:

Condition is generated from NTD data using an econometric model, which in turn is based on a random national sample of vehicles. Average condition changes very slowly due to the steady replacement of vehicles and the relationships in the estimated model.

Verification & Validation:

An independent auditor and transit agency’s CEO certify that data reported to the NTD are accurate. Data are also compared with fleet data reported in previous years, and crosschecked with other related operating/financial data in the report. The econometric model used to translate NTD data into average condition ratings is based on visual inspections of a national sample of bus and rail vehicles. The sample will need to be repeated periodically in the future in order to keep the econometric model current with developments in vehicle conditions.

Comment:

None.

 

Details on DOT Measures of Human & Natural Environment

Fishery protection                                                                         Page 89

Measure:

Number of significant domestic and foreign fishery violations found. (FY)

Scope:

Fishery protection is measured by the number of significant fishery violations recorded by the United States Coast Guard.  Significant violations are defined as those Living Marine Resource violations which result in one or more of the following conditions:  1) Significant damage/impact to the resource/fisheries management plan; or 2) Significant monetary advantage to the violator over their competitors. 

Source:

Significant fishery violations are detected by Coast Guard law enforcement units in the course of living marine resource law enforcement boardings.  The information from the boarding is reported through the Coast Guard Marine Information for Safety and Law Enforcement (MISLE) System. 

Limitations:

It is possible that non-entry, duplication, and coding errors are present in MISLE data; however, the likelihood of this error is small. 

Statistical Issues:

None.

Verification & Validation:

Verification and validation of data occurs in several places in the data reporting and collection process.  Data entry software helps ensure data quality and consistency by employing selection lists and logic checks.  Internal analysis and review of published data by external parties help identify errors. 

Comment:

 None.

Measure:

Percent change in number of species that are designated as overfished (includes only the areas where Coast Guard has enforcement responsibility in fisheries management plans). (FY) (2001)

Scope:

This measure includes species covered under formal fisheries management plans that contain Coast Guard enforcement responsibilities, and that are formally assessed by the National Marine Fisheries Service and designated as either over-fished, approaching over-fished, or not over-fished.

Source:

National Marine Fisheries Service.  Data is provided through the annual NMFS report to Congress "Status of Fisheries of the United States."  This report is mandated by the Sustainable Fisheries Act of 1996.

Limitations:

Historical data are limited – 1997 - 2000 only.  Not all species required to be assessed were formally assessed as over-fished or not over-fished until 2000. Hence, the number of reported over-fished species rose in NMFS’ 2001 assessment.  Assessments of over-fishing are based on biological sampling methods and estimations that are subject to error.

Statistical Issues:

As noted in the Limitations section, this measure is likely to rise as NMFS continues its search for currently unknown fish stocks.  In addition, NMFS revisions to data definitions of over-fished stocks, including their reclassification of over-fished into categories of over-fished and over-fishing has affected the calculation of this measure.  

Verification & Validation:

Data are provided by NMFS.  DOT does not independently verify or validate the data.

Comment:

This measure is aligned with the Sustainable Fisheries Act and the National Marine Fisheries Service (NMFS) related goal.

The Coast Guard also measures the rate of compliance with federal regulations as a critical component of enforcing fisheries management plans designed to improve species health, and prevent over-fishing.

 

Wetland protection and recovery                                                 Page 91

Measure:

On a program-wide basis, acres of wetlands replaced for every acre affected by Federal-aid Highway projects (where impacts are unavoidable). (FY)

Scope:

Measure includes wetlands associated with all Federal-aid highway projects each fiscal year.  To be included, wetland replacement (or investment in a wetland bank) must have begun.

Source:

State DOTs input Federal-aid related wetland degradation and replacement data into either locally developed wetland mitigation databases or the FHWA Wetlands Management Database.  FHWA compiles the final data. 

Limitations:

Data only exists on Federal-aid related wetland replacement.  Also, uniformity of the data is not guaranteed, as it is subject to interpretation by the reporting State DOTs.  In particular, there is no uniform understanding of what should be reported as mitigation acreage.  The FHWA has provided guidance on mitigation activities to report and will soon issue the Wetlands Management Database that should reduce the current variations in data received from the States.  Data on wetland replacement is available for the past five fiscal years (FY 1996 - FY 2000).

Statistical Issues:

The non-uniformity of the data is problematic.  Definitional ambiguity also makes formal statements of statistical uncertainty problematic.

Verification & Validation:

Data are gathered from established mitigation amounts required by section 404 (Clean Water Act) permits that states must acquire for their projects.  In addition, FHWA provides guidance to help states consistently report mitigation data.  This process will be further improved through a standard mitigation database under development for the states.  At present, there is no external audit of state data.

Comment:

All Federal agencies (including FHWA and other modes) must comply with National Environmental Policy Act (NEPA) and the Clean Water Act (specifically section 404(b)(1) of the CWA) regarding disruption of wetlands. These laws require agencies to identify project alternatives that would avoid or minimize impacts to wetlands as a first consideration.  These alternatives are subjected to analysis under both NEPA and the Clean Water Act.  Under the law, these alternatives must be chosen unless the project sponsors clearly demonstrate that they are not viable because they do not meet the project purpose and need, or will lead to other more significant environmental impacts.  If, in compliance with the law, wetland disruption is unavoidable, FHWA then works to achieve this goal of wetland replacement.

 

DOT facility cleanup                                                                     Page 93

Measure:

Percentage of DOT facilities categorized as No Further Remedial Action Planned (NFRAP) under the Superfund Amendments and Reauthorization Act (SARA). (FY)

Scope:

EPA maintains a Federal Facility Hazardous Waste docket (docket), which contains information regarding Federal facilities that manage hazardous wastes or from which hazardous substances have been or may be released.  DOT facilities listed on the docket are discussed in the Annual SARA report sent to Congress each year.  EPA regional offices make the determination to change facility status to NFRAPs on the docket.

Source:

Annual SARA Report to Congress.

Limitations:

The number of DOT facilities listed on the docket can and has fluctuated over the years.  Several of the DOT facilities listed have more than one site requiring cleanup and a facility is not removed from the list until all of the sites have no further remedial action planned.  Some facilities are listed erroneously and it may take several years to remove them from the docket.  NFRAP decisions may be reversed by EPA if future information reveals that additional remedial actions are warranted.

Statistical Issues:

There is no major error present in the subject data.

Verification & Validation:

The data used in measuring this performance is based on restoration activities at field locations for USCG, FAA, FHWA, and FRA.  These field sites report their activities to their respective headquarters management who verifies the data by periodic follow-up reviews.  The data is then reported yearly to the Office of the Secretary, who crosschecks it against data received from EPA and the states. 

Comment:

The primary criterion for NFRAP is a determination that the facility does not pose a significant threat to the public health or environment.  NFRAP decisions may be reversed if future information reveals that additional remedial actions are warranted. The Operating Administrations’ activities are controlled, to a degree, by interaction and decisions made by EPA Regional personnel.

Management Discussion

The number of obsolete vessels removed from the National Defense Reserve Fleet (NDRF) sites for subsequent disposal. (FY)

Scope:

As of January 2002, there were 136 vessels in the NDRF designated for disposal.  MARAD estimates this number will increase, as more Ready Reserve Force (RRF) merchant-type vessels become obsolete.  This increase is primarily due to obsolescence of additional non-combatant, merchant-type vessels from MARAD’s RRF, but also from other Federal agencies (e.g. Coast Guard, NOAA, etc.) for disposal.  MARAD notified the Navy in October 2001 that it would not accept titles to obsolete Navy merchant-type ships until significant progress is made in disposing of MARAD’s current backlog of obsolete ships.  A vessel is not removed from the list of vessels awaiting disposal until it is physically removed from the NDRF sites.

Source:

MARAD maintains records on each of the vessels located at its three Reserve Fleet sites and the entity responsible for disposal of each of the vessels.

Limitations:

None

Statistical Issues:

None

Verification & Validation:

Vessels removed from the NDRF sites are tracked by MARAD.  MARAD has oversight authority for the vessels that it has contracted to be scrapped and continually monitors the operation of the contract holders to make sure that the ships are scrapped in a safe and environmentally sound manner.  Additionally, the Environmental Protection Agency and State and local environmental agencies are made aware of ships being scrapped or recycled, and they also monitor progress.  MARAD requires written certification from respective entities that all recycled activities are completed in accordance with Federal, State and local laws.

Comment:

None

 

Mobile Source Emissions                                                              Page 96

Measure:

Monthly average number of area transportation emissions conformity lapses. (FY)

 

Scope:

The transportation conformity process is intended to ensure that transportation plans, programs, and projects will not create new violations of the National Ambient Air Quality Standards (NAAQS), increase the frequency or severity of existing NAAQS violations, or delay the attainment of the NAAQS in designated non-attainment (or maintenance) areas.  The publication,

Transportation Conformity: A Basic Guide for State and Local Officials

 contains the basic provisions of the conformity process.

 

Source:

FHWA and FTA jointly make conformity determinations within air quality non-attainment and maintenance areas to ensure that Federal actions conform to the purpose of State Implementation Plans (SIPs).  With DOT concurrence, the EPA has issued regulations pertaining to the criteria and procedures for transportation conformity, which were revised based on stakeholder comment.

 

Limitations:

Conformity determinations are required by law to be updated once every three years.  One reason for an area to be in a conformity lapse is due to the fact that it missed the deadlines for making a conformity determination on the transportation plan and program.  Under this scenario, the conformity lapse is not a result of the emissions problem in that area. 

In addition, certain State Implementation Plan (SIP)-related deficiency findings by EPA (such as a disapproval of a submitted SIP without a protective finding) may also put an area in a conformity lapse.  This may take a long time before the SIP-related issue(s) are addressed through the complex and time-consuming SIP revision process.  In this situation, FHWA/FTA will have little control over the duration of the conformity lapse.

 

Statistical Issues:

None.

 

Verification & Validation:

The MPO and U.S. DOT (FHWA/FTA) have a responsibility to ensure that transportation plans and programs within metropolitan boundaries conform to the SIP. In metropolitan areas, the governing board of each MPO must formally make a conformity determination on its transportation plan/TIP prior to submitting them to the U.S. DOT (FHWA/FTA) for review and approval. Conformity determinations for projects outside of these boundaries are the responsibility of the U.S. DOT (FHWA/FTA) and the project sponsor, which usually is the State DOT.  In addition, the National Memorandum of Understanding issued on April 19, 2001, provides the EPA and DOT with a framework for coordinating and working through issues in the conformity and SIP processes. Specifically, the MOU's provisions ensure that:

1. EPA and DOT consult on conformity determinations before DOT's approval process;

2. the conformity rule's provisions are appropriately applied with regard to conformity determinations; and

3. adequate interagency consultation persists through the planning and conformity processes to identify and resolve issues prior to a conformity lapse or freeze.

 

Comment:

If conformity cannot be determined within certain time frames after amending the SIP, or if three years has passed since the last conformity determination, a conformity lapse is deemed to exist and no new non-exempt projects may advance until a new determination for the plan and TIP can be made. This affects transit as well as highway projects.  During a conformity lapse, FHWA and FTA can only make approvals or grants for: projects that are exempt from the conformity process (pursuant to '93.126 and '93.127 of the conformity rule) such as safety projects, and transportation control measures (TCMs) that are included in approved SIP. Only those project phases that have received approval of the project agreement, and transit projects that have received a full funding grant agreement (FFGA), or equivalent approvals, prior to the conformity lapse may proceed during a conformity lapse.

 

Measure:

Tons (in millions) of mobile source emissions from on-road motor vehicles. (FY) (2001)

Scope:

Figure is the sum of on-road mobile source emissions of carbon monoxide, hydrocarbons, nitrogen oxides, and particulate matter less than 10 microns in diameter (PM-10).

Source:

National Air Quality and Emissions Trends Report published annually by EPA.  (EPA uses data from FHWA’s Highway Performance Monitoring System – HPMS.)

Limitations:

On-road mobile source emissions estimates are modeled using vehicle data.  Past data contain some variations due to changes in methodology used to obtain on-road mobile source emissions estimates.  EPA revises emission estimates periodically based on revised methodology.  In 1999, EPA increased the annual emission burden trend based on the knowledge that heavy-duty diesel trucks manufactured since the early 1990’s produce higher emissions during high-speed operations.  Emissions data are reported in a 2-year time lag.  Indicator captures all major mobile source emissions from on-road vehicles.  It does not capture off-road mobile sources, such as agriculture and construction machinery, lawn mowers, aircraft, trains, and boats.  

Statistical Issues:

The EPA’s use of a mathematical model poses issues of model validation.  The annual variation in the model’s estimates, as measured by the regression standard error for data from years 1994 to 1999, is 2.53.  The HPMS data used as input to the model are subject to sampling and non-sampling errors.

Verification & Validation:

EPA conducts verification and validation of data.  FHWA field offices perform annual reviews of HPMS data that EPA uses as a part of its model.

Comment:

The National Ambient Air Quality Standards (NAAQS), as revised in July 1997, may create new challenges for DOT in meeting the air quality goal. Targets may need to be modified to reflect these changes.

 

Greenhouse Gas Emissions                                                           Page 97

Measure:

Metric tons (in millions) of carbon equivalent emissions from transportation sources. (CY) (2001)

Scope:

Measure includes GHGs such as those subject to the Kyoto Protocol (e.g., CO2, CH4), but not other GHGs (e.g., water vapor).  Emissions from fossil fuels combusted in civilian and military ships and aircraft engaged in international transport of passengers and cargo (i.e., those that are recorded separately as international bunkers) are not included. Does not include emissions from non-transportation mobile sources such as farm and construction equipment.

Source:

Inventory of U.S. Greenhouse Gas Emissions and Sinks: 1990-1999, published by EPA, supplemented with EPA’s Draft Inventory of U.S. Greenhouse Gas Emissions and Sinks: 1990-2000.  Estimates are based on data from EPA and other agencies.

Limitations:

GHG emissions are estimated based on DOE estimates of aggregate supply of energy products such as motor gasoline and distillate fuel oil.  Further disaggregation (e.g., of transportation modes and other uses such as agriculture) is not always available.  Related “upstream” emissions and sequestration (e.g., from petroleum refining) are in separate categories.  Procedures for calculating and applying GHG credits and permits have not yet been established.

Statistical Issues:

These data are external to DOT.  They are subject to both sampling and non-sampling errors.

Verification & Validation:

EPA conducts verification and validation of data.  DOT will participate as appropriate in reviewing data, methodology, and results.

Comment:

None.

 

Maritime Oil Spills                                                                        Page 98

Measure:

Gallons spilled per million gallons shipped by maritime sources. (FY)

Scope:

Spills from vessels and waterfront facilities that are the target of Coast Guard regulatory prevention efforts are counted; other spills are not. Oil spills of 1 million gallons or more are excluded (or shown separately) from data since they are rare (they do not occur every year) and would have an inordinate influence on statistical trends.

Source:

Spill amounts (numerator) are obtained from the Coast Guard Marine Safety Information System (MSIS). By regulation, spills are reported to the National Response Center or to the Coast Guard Federal On-scene Coordinator.  Spill reports are normally made by the representatives of the party spilling the oil. Sometimes spill reports are received from third parties, or spills are discovered by Coast Guard personnel.  Data on waterborne oil shipments (denominator) is from US Army Corps of Engineers “Waterborne Commerce Statistics”.

Limitations:

The investigation, retrieval, analysis and reporting processes result in under-reporting for the most recent year, with the most significant effects over the most recent 5 months.  Estimates are often used to compensate for this known data-lag.  It is probable that some spills are not reported.  Large spills that impact a large area, or are located in heavily transited areas are more likely to be reported than small spills or spills in remote locations.  The actual amount of oil spilled may vary significantly from the amount estimated.  The significance of this error depends on the unique circumstances of each case.  However, the error rate for volume of oil spilled is estimated to be less than 5% because large spills receive a high level of review and account for most of the volume spilled. Duplicate spill entries are sometimes entered into MSIS, and some spills are mistakenly omitted or entered incorrectly.  Verification procedures strive to correct these errors, but it is probable that some are not corrected. By excluding non-regulated sources and major oil spills, the measure does not capture the amount spilled annually from all sources. However, the exclusions are helpful in assessing the impact of existing Coast Guard regulations and policies (program management).

Statistical Issues:

The major sources of uncertainty in this measure are the reporting error (as a result of the data-lag), estimation error (actual amount of oil spilled may vary from the amount estimated), and response error (as a result of spills not being reported to or discovered by the Coast Guard). The regression standard error for year-to-year chance variation is 1.8 for the number of gallons spilled per million gallons shipped, based on data from 1995 through 2000. 

Verification & Validation:

Verification and validation occurs at several levels.  Edit checks within MSIS can detect some incorrect or missing data and force review and correction before data entry is completed.  Selection lists for certain data fields also reduce the opportunity for data entry error.  All investigations go through one level of review at the field unit for accuracy.  Investigations of spills are also usually reviewed at district and headquarters offices.  The headquarters Data Administration staff conducts periodic quality control checks to identify entry errors such as missing data or miscoding, and corrects any errors identified.  Each spill involving 1,000 gallons or more is reviewed before it is included in the measure.  Errors identified are referred to either the Data Administration staff or the Investigations and Analysis staff for correction.

Comment:

During FY 2002, the Marine Safety Information System (MSIS) will be replaced by the Marine Information System for Safety and Law Enforcement (MISLE).  While the new system will be a significant improvement, it is expected to cause serious difficulties in making performance comparisons.  One factor is that many business processes were re-designed in conjunction with system development.  Another factor is that data quality under MISLE is expected to be superior to that of MSIS.  While this represents improvement, it may cause near-term problems in making meaningful comparisons of data between the two systems.

 

Hazardous materials spills                                                           Page 98

Measure:

Tons of hazardous liquid materials spilled per million ton-miles shipped by pipelines. (CY)

Scope:

Hazardous liquid pipeline incidents are those that result in a fatality or injury resulting in hospital treatment or hospitalization, property damage equal to or greater than $50,000, or more than 50 barrels spilled.  (A rulemaking proposes to lower the reporting threshold for spill amount from 50 barrels to five gallons.)  This measure tracks only releases from hazardous liquid pipelines to the environment.  Natural gas pipeline releases vaporize into the atmosphere and do not have long-term significant impact on the environment, and thus are not included in this measure.

Source:

Pipeline operators report to RSPA on form 7000-1, Hazardous Liquid Accident Report.  RSPA records the data in RSPA’s Hazardous Materials Information System.

Limitations:

Because of the magnitude and frequency of fluctuations in the historical data for this measure, a short-term goal will be of limited use in tracking program performance.  RSPA does not collect volume shipped data but uses the Association of Oil Pipelines annual Fact Sheet as source for this part of the measure.

Statistical Issues:

These spill incidents are rare and probably not independent events.  The performance measure is a ratio, so uncertainty in the denominator can have a large effect on the overall uncertainty. 

Verification & Validation:

RSPA reviews the data for accuracy.  Supplemental reports are requested where obvious reporting shortcomings are indicated.  Additionally, the ASME B31.4 liquid pipeline data review subcommittee performs an annual examination of the hazardous liquid incident reports.  Known problems with under-reporting property damages and spill quantities are being addressed by a rulemaking to revise accident reporting requirements to implement a new “open and closed” status to insure that operators continue to file supplemental reports until the spill consequence is fully reported.  A new industry data improvement effort piloted in 1999 will provide better precursor data and more extensive data about impacts to the environment of hazardous liquid pipeline spills.  The American Petroleum Institute is housing the voluntary data repository, which will collect information on spills down to five gallons (down to one gallon in water) for all pipeline spills, including those currently not jurisdictional to RSPA.

Comment:

The data for this measure fluctuate year to year.  RSPA is studying the spill data to determine the nature of this fluctuation and improve this measure.

 

Aircraft noise exposure                                                              Page 102

Measure:

Number of people in the U.S. (in thousands) who are exposed to significant noise levels (65 decibels or more). (FY)

Scope:

Residential population exposed to aircraft noise above Day-Night Sound Level of 65 decibels around U.S. airports with the greatest number of commercial jet take-offs and landings.

Source:

A statistical modeling technique (the MAGENTA model) is applied using U.S. population data from the Department of Commerce, locally developed traffic distribution (route and runway utilization), and aircraft distributions developed using the Official Airline Guide and current aircraft registration databases. The local traffic utilization data is available for the busiest U.S. airports in the form of studies developed for the FAA’s Integrated Noise Model (INM). For smaller airports, a generic statistical procedure was employed.

Limitations:

No actual count (i.e., using a local survey) is made of the number of people exposed to aircraft noise.  No military or general aviation aircraft are included in the FAA’s model. Aircraft type and event level can be considered current. However, the majority of the databases used to establish route and runway utilization were developed from 1990 to 1997, with many of them now over seven years old. Changes in airport layout including expansions may not be reflected. The benefits of federally funded mitigation, such as sound insulation or buyout, are not accounted at present. Future development of the methodology will attempt to quantify the gains (reduction in people exposed) due to these actions.

Statistical Issues:

This measure is derived from model estimates that are subject to errors in model specification.  The estimates of population data will be revised once the new U.S. Census data for 2000 is released and the model software is updated accordingly.

Verification & Validation:

The Integrated Noise Model has been validated with actual acoustic measurements at both airports and other environments such as areas under aircraft at altitude.  External forecasts data are from primary sources. The MAGENTA population exposure methodology has been thoroughly reviewed by an ICAO task group and was validated for several airport specific cases.

Comment:

FY 2000 was the first year measuring using the MAGENTA model.

 

Transit service                                                                            Page 104

Measure:

Percent of urban population living within a quarter mile of a transit stop with service frequency of 15 minutes or less (non-rush-hour). (CY) (2001)

Scope:

A transit stop is defined as a bus stop, but does not include rail stations unless associated with a bus stop. 

Source:

FTA compiled information from bus schedules across the country.  Population statistics come from the Census Bureau.  Information from both of these sources was formatted using the Geographic Information System.

Limitations:

Transit stops do not include rail stations (such as light rail or subway).  However, rail stations are almost always served by bus lines, so most persons who live near a rail station also live near a bus line. 

Statistical Issues:

The extrapolation of population statistics from the Census Bureau at a level fine enough to support inferences within a geographic radius of a quarter mile is difficult.  The measurement aspects of this estimate require careful examination.

Verification & Validation:

Under development.

Comment:

The Federal Transit Administration is working to develop the Transit Performance Monitoring System.  Fully instituted, the TPMS will allow the Department to measure not only how many people live close to public transit, but also how many people use public transit for basic mobility.

 

 

Details on DOT Measures of Organizational Excellence

Small disadvantaged and women-owned small business contracting  Page 111

Measure:

1.  Percent share of the total dollar value of DOT direct contracts that are awarded to women-owned businesses. (FY)

2.  Percent share of the total dollar value of DOT direct contracts that are awarded to small disadvantaged businesses. (FY)

Scope:

Includes contracts awarded by DOT contracting activities (except FAA) through direct procurement (i.e., not including contracts issued by grantees).

Source:

All DOT contracting activities except the FAA report data to the Contract Information System (CIS).  This data is reported to the Federal Procurement Data Center (FPDC) by the CIS.

Limitations:

Contracting data is reported by procurement offices directly into the CIS. Data can still be entered into CIS and reported to FPDC after performance measurement results are submitted so small variations in prior year performance data may result.

Statistical Issues:

There is no major error present in the subject data.  However, random variation in the number of DOT contracts as well as the number of women-owned and small-disadvantaged businesses each year results in some random variation in these measures from year to year.  The regression standard error for 1994-2000 is 0.64 percent for women-owned small businesses and 1.23 percent for small-disadvantaged businesses.

Verification & Validation:

DOT conducts comparison checks of CIS data with FPDC data to reconcile discrepancies.  On occasion, GSA audits the accuracy of DOT contracting data.

Comment:

None.

 

Environmental Justice                                                                Page 114

Measure:

Number of environmental justice cases that remain unresolved after one year. (FY)

Scope:

Data will cover complaints filed with DOT under Title VI of the Civil Rights Act of 1964 and which have had environmental justice elements, such as allegations of substantially adverse environmental or health impact on a minority or low-income community by a transportation project. Case resolutions are actions that end or administratively close out complaints.  These include such actions as determinations of no jurisdiction, withdrawals by complainants, resolutions achieved through alternative dispute resolution, findings of no violation, and negotiated settlements after discrimination findings under Title VI.

Source:

DOT will collect this data through the External Complaint Tracking System (XTRAK).

Limitations:

This measure is an initial indicator of how well DOT processes EJ complaints.  Variables that will not necessarily be assessed include such factors as magnitude of injury, number of beneficiaries adversely affected, pervasiveness, and time constraints before irreparable damage occurs. Other statutory requirements exist for NEPA concerns.

Statistical Issues:

There is no major error present in the subject data.

Verification & Validation:

Data will cover the entire universe of external complaints received by DOT, and will be entered into the system by operating administrations and DOT Office of Civil Rights staff.

Comment:

This indicator does not measure the impact of DOT’s efforts to prevent the conditions that give rise to complaints.  It does provide an initial measure of response timeliness, which is important to the public.  The measure was expanded in 2000 to include the percent of cases that remain unresolved after one year as a further indicator of the timeliness of resolution.  All environmental justice cases by definition relate to the concerns of a community of low income and/or minority people.  In addition, the number of cases indicates the pervasiveness of community perception of significantly adverse environmental and health concerns.