Performance Data and Performance Measurement

Performance measurement is dependent on the availability of useful data. Useful data will indicate level of performance and progress toward organizational goals. All data are imperfect in some fashion. Pursuing “perfect” data, however, may consume public resources without creating appreciable value. For this reason, there must be an approach that provides sufficient accuracy and timeliness but at a reasonable cost. This section of the Performance Plan/Performance Report provides information on how DOT reports on performance, verifies and validates data, assesses limitations of the data, and plans for improving DOT’s data.


PerformanceData Completeness and Reliability

In an attempt to bring consistency and quality to its performance reporting, DOT has implemented some general rules regarding the data it uses and how it is evaluated.

Annual data – Whenever available, the data in this document are reported on a Federal Government fiscal year basis. However, there are instances where this is not possible so calendar year data are used instead.  This often occurs when data are collected and reported to DOT by external sources and a calendar year reporting requirement is specified in the implementing regulation. The reporting timeframe (FY or CY) for each measure is included in the Data Details in Appendix I.

Annual results – If available, the results for the most recent year in the Report are listed as “Actual” in the Performance Goals & Results box for each performance measure. However, given the March deadline for submission of the Performance Report, quite often data have not been compiled and finalized for the entire year. When this occurs and an actual value is not available for the current year, either an estimate or projection is provided instead. In general, estimates are based on partial year data that are extrapolated to cover a full 12-month period. For example, if six months of data are available, they will be compared to prior years for the same six-month period to determine any variation from past levels. Historical trend information, supplemented by program expertise, will then be applied to estimate the remaining six months of performance. The result will be identified as a “preliminary estimate” in the Report. If partial year data are not available, then past trend information will be analyzed and supplemented by program knowledge to develop a projected value for the annual performance measure. The result will be identified as a “projection” in the Report. As data are finalized, the projections and preliminary estimates will be replaced by actual results.  Results may be amended as errors and omissions are identified in the data verification process, because updated information is provided by the reporting sources, or because of legal or other action that changes a previously reported value. For example, updated pipeline spill reports may change the status of a previously reported value used in performance measurement.

In measuring progress toward the majority of performance goals, DOT is moving to a system of monthly performance measurements.  This will make it much easier to internally gauge periodic progress toward goals as the year progresses, and will enable more timely performance reporting after the years’ end. 

Completeness of Data – As described above, actual data and “preliminary estimates” incorporate complete or partial data from 2001.  Results listed as “projections” are not based on data from 2001, but on trend data from prior years. 

Reliability of Measurement Data – Because performance results in a given year are influenced by multiple factors, some of which are beyond DOT’s control, and some of which are due to random chance, there may be considerable variation from year to year. (See discussion in Appendix I.) A better “picture” of performance may be gained by looking at results over time to determine if there is a trend. Therefore, graphs are provided for each measure showing trend lines back to 1990, or as many years as possible if data are not available back to 1990. Additionally, a table is included at the beginning of each strategic goal section giving the available data from 1995 through 2001 for measures with performance goals specified for 2001.

Verifying & Validating Performance Measures

Integral to performance measurement is understanding data limitations, addressing these limitations where necessary and cost-effective, and acknowledging those that remain when interpreting results. This section on verification and validation provides a DOT-wide overview of our plan for assessing the quality of the data DOT uses to measure its performance, and follows the GAO definitions for verification and validation:

“Verification is the assessment of data completeness, accuracy, consistency, timeliness, and related quality control practices.”

“Validation is the assessment of whether data are appropriate for the performance measure.”

Virtually all data have errors. In Appendix I we have provided the following information about the data used for each performance measure: source of the data, limitations of the data, observations about the quality of the data, work planned or ongoing to improve data quality, and any known biases.

Additionally, we have compiled Source and Accuracy Statements for each of the DOT data programs used in this report, which can be found at www.bts.gov/statpol/SAcompendium.html. The Source and Accuracy Statements give more detail on the methods used to collect the data, sources of variation and bias in the data, and methods used to verify and validate the data.

By validating data used in the DOT performance plan, we are ensuring that those data are reflective of the phenomena they purport to measure. The Office of the DOT Inspector General (OIG) plans to selectively verify and validate performance measurement data each year. When pertinent to the conduct of ongoing projects, OIG will also assess performance measures to determine their appropriateness for measuring progress toward stated goals. These assessments may lead to changes in the goals, improvements to or additions of data collection systems, or both.

Assessing and, where possible, eliminating sources of error in DOT data collection programs has always been an important task for data program managers. As a part of their ongoing work, managers of Departmental data programs use quality control techniques, such as flowcharting the data collection process, to identify where errors can be introduced into the data collection system. Program managers also use computerized edit checks and range checks to minimize errors that may be introduced into the data of their respective programs. In addition, quality measurement techniques are employed to measure the effects of unanticipated errors. These include verification of data collection and coding, as well as coverage, response and non-response error studies to measure the extent of human error affecting the data. As sources of error are identified, steps are initiated to improve the data collection process.

The data used in measuring performance come from a wide variety of sources. Much of the data originates from sources outside the Department and, therefore, outside the direct control of the Department. The data often come from administrative records or from sample surveys. While DOT may not have a strong voice in improving the quality of outside data, the Department takes all available information about the limitations and known biases in outside data into account when using the data.

The myriad data sources make the task of assessing and, where possible, eliminating error a challenging one for DOT. Different data systems contain different types of errors. For example, data from administrative records systems may have missing or incorrect records, and data from sample surveys will contain sampling error.

Several measures (particularly in safety) require aggregation across transportation modes. This can be particularly problematic because of the use of different definitions in different transportation modes. Also, data from outside the Department may have unknown error properties.

To help the operating administrations address these issues, the Bureau of Transportation Statistics (BTS) is developing a statistical policy framework where the operating administrations will work together to identify and implement the current statistical “best practices” in all aspects of their data collection programs. This project is consistent with the data capacity discussions found in the DOT Strategic Plan.

In 2001, a DOT intermodal working group addressing DOT data quality issues continued to:

          develop Departmental statistical standards;

      update Source and Accuracy Statements for all DOT data programs to document limitations and known errors and biases;

          improve quality assurance procedures;

          evaluate sampling and non-sampling error; and;

      develop common definitions for data across modes.

BTS's statistical staff is consulting with the DOT operating administrations’ data program managers to assist in data evaluation and validation, documenting data sources, and determining the reliability of performance measurement estimates.

Departmental data systems managers use these data verification methods:

      Comparisons with previous data from the same source.

      Comparisons with another reliable source of the same type of data within DOT for the same time period.

      Comparisons with another reliable source of the same type of data within DOT for a previous time period.

      Comparisons with another reliable source of the same type of data outside DOT for the same time period.

      Comparisons with another reliable source of the same type of data outside DOT for a previous time period.

In addition to computerized edit checks and clerical review procedures to look for outliers, duplicate records, and data inconsistencies, data managers also verify data quality at each step of the data collection process using these procedures:

      Re-collecting/re-interviewing all (or a sample of) records and reconciling with the original collection. (This applies to census or sample survey data collections from administrative records, organizations, or individuals.)

      Conducting 100 percent (or a sample of) data re-coding and reconciliation to assess and correct coding errors.

      Conducting 100 percent (or a sample of) data re-entry and reconciliation to assess and correct data entry errors.

The American Travel Survey’s re-interview program, in which a sample of households were re-contacted and differences reconciled, is an example of a verification system within a data collection program.

Data Limitations in Performance Measures

DOT Data Source Limitations – Timeliness is the most significant limitation for DOT performance measurement data. Some DOT data are not collected annually. For example, the National Household Travel Survey and the Commodity Flow Survey each collect data every five years.  Data that are collected each year (or more frequently) require time to analyze, confirm and report results. For example, Highway Performance Monitoring System vehicle-miles traveled (VMT) data require several months of post-collection processing, making final results unavailable for this performance report.

Other performance measurement data limitations can be found in the previously mentioned Source and Accuracy Statements for DOT data programs. These statements contain descriptions of data collection program design, estimates of sampling error (if applicable), and discussions of non-sampling errors. Non-sampling errors include under-coverage, item and unit non-response, interviewer and respondent response error, processing error, and errors made in data analysis.

As part of its mandate in the Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA), the Transportation Equity Act for the 21st Century (TEA-21), and its plans for a statistical policy framework in the Department, BTS is working on a program of research, technical assistance, and data quality enhancement to support the continued improvement of data programs in DOT. This will help data program managers throughout DOT improve data quality and better document known data limitations. BTS also assists operating administrations with data collection and documentation.

Many of DOT’s internal data programs rely on State DOTs to collect reliable statistics within cost constraints. While we work closely with our State DOT partners, we do not have direct control over these data.

External Data Source Limitations – Timeliness is also a significant limitation for external or third-party data. Other limitations of external data are noted in the comments for each performance measure in Appendix I. In some cases, DOT has replaced external data, where little is known about the quality of the data, with internal data. For example, DOT has used estimates of person-miles traveled (PMT) from private organizations, absent any better estimate. The 1995 Nationwide Personal Transportation Survey and American Travel Survey give DOT data with known error properties that allow a better estimate of PMT.

Our Data Needs

The DOT Strategic Plan 2000 – 2005 identifies data needs for each of the Department’s strategic goals. They include:

Safety – DOT is undertaking major efforts over the next several years to improve safety data. Safety has always been our primary strategic goal, and in 1999 DOT created a Safety Data Action Plan to better organize data improvement efforts. BTS will lead efforts to: 1) develop common criteria for reporting injuries and deaths; 2) develop common data on accident circumstances; 3) improve data quality; 4) develop better data on accident precursors; 5) expand the collection of near-miss data to all transportation modes; 6) develop a variety of common denominators for safety measures; 7) advance the timeliness of safety data; 8) link safety data with other data; 9) explore options for using technology in data collection; and 10) expand, improve and coordinate safety data analysis.

Homeland Security – Existing performance data sources are generally good, but DOT will collect data to better understand the transportation system’s vulnerability to intentional acts of disruption or destruction.

Mobility – All mobility outcomes present complex measurement issues. Accordingly, DOT will: 1) develop ways of measuring user transportation cost, time, and reliability with time-series data; 2) develop better approaches for measuring access; 3) develop straightforward measures of congestion and its costs; 4) produce more timely and comprehensive data on the condition and use of the transportation system; and 5) develop a more complete understanding of variables influencing travel behavior.

Economic Growth – DOT needs aggregate data for measuring the productivity, effectiveness and efficiency of the U.S. transportation system. We plan to collect, analyze and disseminate data and information that identify critical trends and issues relating to transportation’s nexus to the U.S. economy. DOT will: 1) develop a means of measuring transportation cost, time, and reliability – at an aggregate level – with time-series data; 2) develop a comprehensive measure of the transportation capital stock; 3) improve our view of changes in the transportation workforce; 4) develop better measures of productivity in the transportation sector, and other issues concerning use of Producer Price Indices; and 5) develop a better picture of transportation-related variables influencing U.S. competitiveness in the global economy.

Human and Natural Environment – DOT will: 1) develop comparable and complete data on transportation emissions, noise, hazardous materials releases, and wetlands impacts; 2) improve our understanding of collateral damage to the human natural environment; 3) create better leading indicators for potential environmental issues; and 4) develop a reliable method of measuring the environmental benefits of bicycling and walking.

 


Appendix I – Performance Measures (Detail)

 

Each table includes a description of a performance measure and associated data provided by the agencies in charge of the measure.  The Scope statement gives an overview of the data collection strategy for the underlying data behind the performance measure.  The Source statement identifies the databases used for the measure and their proprietary agencies.  The Limitations statement describes some of the shortcomings of the data in quantifying the particular performance characteristics of interest.  The Statistical Issues statement has comments, provided by the Bureau of Transportation Statistics (BTS) and the agency in charge of the measure, that discuss variability of the measure and other points.  The Verification and Validation statement indicates steps taken by the proprietary agencies to address data quality issues.

DOT feels strongly that full compliance with the Government Performance and Results Act requires impartial reporting of the statistical uncertainty associated with numerical performance measures.  A portion of this uncertainty is related to the methodology used to calculate the performance measure and the accuracy of the underlying data.  For example, the use of samples introduces uncertainty because estimates are used in lieu of actual counts.  Also, there may be errors in the data collected.  However, there are many other sources of variation (e.g., nonsampling errors, climate effects, new technology) and they are often difficult to quantify.  Nonetheless, a combination of past data and expert judgment can enable uncertainty statements that are order-of-magnitude correct for even the most difficult problems.

The standard error of a performance measure indicates the likely size of the chance variation in the reported number.  It incorporates both the effects of measurement error, survey error, and so forth, as well as the variation that occurs naturally from year to year (i.e., even if there were no change in laws, infrastructure conditions, or human behavior, there would still be chance variation in an annual count of fatalities).  DOT success in meeting GPRA goals must be viewed in the context of this background noise.

In many of the following Statistical Issues statements, BTS refers to regression standard error.  This is a modification of the standard error to take into account of linear trends in the recent past.  Such adjustment is generally needed to incorporate consistent trends due to cumulative effects of such things as education programs, changing demographics, the gradual adoption of new technologies, and so forth.  The underlying assumptions are that: over a short time period the trend of the measurement data is linear; for any given year the performance measure values are normally distributed; and the standard deviation is the same for all years.  We believe that these assumptions lead to a conservative estimate of variability.

The regression standard error is an estimate, calculated from the annual performance results, of this common standard deviation.  It may be used in the same way as a regular standard error to set confidence intervals or describe uncertainty.  For the purposes of performance measurement, it may be considered a rough approximation of the annual variability in a measure, and it will include the affects of program initiatives, influences beyond the control of DOT (e.g., weather, petroleum prices, etc.), random chance, and errors inherent in the data.

For further information about the source and accuracy (S&A) of these data, please refer to the BTS S&A compendium available at

www.bts.gov/statpol/SAcompendium.html

 


Details on DOT Measures of Overall Safety

Transportation Safety                                                                       Page 12

Measures:

1.      Transportation fatalities. (CY)

2.      Fatalities per 100 million passenger-miles. (CY)

3.      Fatalities per 100 million ton-miles of freight. (CY)

4.      Transportation injuries. (CY)

5.      Injuries per 100 million passenger-miles. (CY)

6.      Injuries per 100 million ton-miles of freight. (CY)

7.      Transportation incidents. (CY)

Scope:

This family of measures aggregates fatalities, injuries and incidents across all modes of transportation (air, highway, railroad, transit, waterborne and pipeline).

The fatality and injury rates per 100 million passenger-miles exclude pipeline fatalities and injuries due to minimal interaction with passenger miles.  Highway-rail grade crossing fatalities and injuries are not counted since they are included in data for highways.

The fatality and injury rates per 100 million ton-miles of freight include fatalities and injuries from large truck, rail, waterborne and pipeline transportation.  Highway-rail grade crossing fatalities and injuries are also included since these involve freight transportation-related fatalities and injuries that would not otherwise be counted.  Ton-miles of freight covers intercity truck, rail, water and oil pipeline transportation.  Aviation fatalities, injuries and ton-miles are excluded because the fatality and injury data are not separated from passenger air carriers. Transportation incidents include crashes, system failures, spills, releases, and other accidents of a similar nature.

Source:

The data for these measures are obtained from National Transportation Statistics published annually by the Bureau of Transportation Statistics.  Information is taken from the following tables: Transportation Fatalities by Mode; Injured Persons by Transportation Mode; U.S. Passenger-Miles (Millions); U.S. Ton-Miles of Freight (Millions); and Transportation Accidents by Mode.  The one exception is the data on large truck fatalities and injuries used for calculating fatality and injury rates per 100 million ton-miles of freight are obtained from the Federal Motor Carrier Safety Administration.

Limitations:

Double counting of fatalities and injuries may occur when an accident involves more than one mode of transportation.  Differing definitions of injuries or transportation-related fatalities makes comparison across modes of transportation problematic.  Highway injuries and incidents are obtained from a nationally representative probability sample and are estimates, while the totals for other modes of transportation are actual counts.  The highway estimates are based on crashes where a police accident report was completed and the crash resulted in property damage, injury or death.  Accidents that were not reported to the police or did not result in property damage are not included.  Highway passenger miles are calculated by multiplying vehicle-miles of travel (VMT) by the average number of occupants for each vehicle type.  VMT is based on a nation-wide sample of vehicle travel.  The average number of vehicle occupants comes from survey information.  Therefore, vehicle passenger miles is an estimate, whereas passenger-miles for other modes of transportation are calculated based on actual passenger counts and recorded trip lengths. 

Statistical Issues:

All fatality totals, and the injury and incident numbers where actual counts are recorded, are relatively accurate.  Any double counting or omissions are expected to be fairly small.  The primary source of uncertainty in these measures comes from sampling and survey errors related to estimation of highway injuries, incidents, VMT and vehicle occupancy.  Based on data from 1994-2000, the annual variations in the transportation safety measures are as follows:  the regression standard error for the  number of transportation fatalities is 0.5 thousand.  For fatality rates by passenger-miles and ton-miles, it is 0.010 and 0.007, respectively.  For number of injuries, it is 0.10 million.  For injury rates by passenger-miles and ton-miles, it is 2.50 and 0.24, respectively.  For incidents, it is 0.16 million.

Verification & Validation:

BTS compiles the data for the National Transportation Statistics from information it gathers directly in its own data systems (e.g. airlines information), information published by other sources (e.g. FHWA highway statistics), or by personal communication with the agency/organization responsible for collecting the data.  Each data source conducts error checks and monitors the accuracy of its data.  Most of these sources and their verification and validation procedures are described in subsequent data details in this report for performance measures of individual modes of transportation.

Comment:

While caution should be exercised in comparing fatalities, injuries and incidents between modes of transportation due to differences in definitions and calculations, the aggregation of these values still provides useful information.  Because the methodology for calculating these measures has remained consistent over the years, the trend information should provide a reasonably accurate picture of results. 

 

Highway fatality rate                                                                   Page 15

Measure:

Fatalities per 100 million vehicle-miles-traveled (VMT) (CY)

Scope:

The number of fatalities is the total number of motor vehicle traffic fatalities which occur on public roadways within the 50 states and Washington, D.C.

Vehicle Miles of Travel (VMT) represent the total number of vehicle miles traveled by motor vehicles on public roadways within the 50 states and Washington, D.C.

Source:

Motor vehicle traffic fatality data are obtained from NHTSA’s Fatality Analysis Reporting System (FARS).  To be included in FARS, a motor vehicle traffic crash must result in the death of a vehicle occupant or a non-motorist within 30 days of the crash.  The FARS database is based on police crash reports and other state data. FARS includes fatalities on all roadways open to the public, using the National Highways System classification of roads.  Pedestrian and bicycle fatalities that occur on public highways, but do not involve a motor vehicle, are not recorded in FARS.  However, they constitute only a small number of fatalities.

VMT data are derived from FHWA’s Traffic Volume Trends (TVT), a monthly report based on hourly traffic count data in the Highway Performance Monitoring System (HPMS).  Information is transmitted to NHTSA where it is reviewed for consistency and accuracy before being entered into the system. These data, collected at approximately 4,000 continuous traffic counting locations nationwide, are used to determine the percentage change in traffic for the current month from the same month of the previous year.  The percentage change is applied to the nationwide travel for the same month of the previous year to obtain an estimate of nationwide travel for the current month.  The data are recorded as monthly totals and cumulative yearly totals.

Limitations:

VMT data are subject to sampling errors, whose magnitude depends on how well the locations of the continuous counting locations represent nationwide traffic rates.  HPMS is also subject to estimating differences in the states, even though FHWA works to minimize such differences and differing projections on growth, population, and economic conditions that impact driving behavior.

Statistical Issues:

The primary source of uncertainty in estimating fatality rates is the denominator.  While the estimate of total fatalities used in the numerator is relatively accurate, the estimate of total vehicle miles in the denominator has far more variability.  Based on data from 1994-2000, the annual variation in the fatality rate has a regression standard error of 0.029.

The estimates of the number and percentages of persons killed in motor vehicle traffic crashes during 2001 are preliminary and are based on incomplete data and statistical models.  NHTSA’s first official estimates for 2001, the Early Assessment, are being developed and will be completed in early April 2002.  Differences between the Official Early Assessment estimates and those in this report are to be expected.

Verification & Validation:

Fatality data from FARS are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis.  Quality control procedures are built into annual data collection at 6 and 9 months, and at year’s end.  A study was completed in 1993, looking at samples of FARS cases in 1989 through 1990 to assess the accuracy of data being reported.  VMT data are reviewed by FHWA for consistency and reasonableness.

Comment:

This data program has been in use for many years and is generally accepted for describing safety on the Nation’s highways.  Adjusting raw highway fatalities and injuries by VMT provides a means of portraying the changes in highway fatalities on a constant exposure basis and facilitates year-to-year comparisons.

 

Large truck-related fatalities                                                       Page 15

Measure:

Number and rate (per million commercial VMT) of fatalities in crashes involving large trucks. (CY)

Scope:

The measure includes all fatalities (e.g., drivers and occupants of passenger cars, motorcycles, large trucks, or pedestrians) associated with crashes involving trucks with a gross vehicle weight rating of 10,000 pounds or more. The number of fatalities comes from NHTSA’s Fatality Analysis Reporting System (FARS) data, a census of fatal traffic crashes within the 50 states, Puerto Rico, and Washington, D.C.  The fatal crash rate is the number of fatalities per 100 million vehicle miles of large truck travel (VMT).

Source:

NHTSA’s Fatality Analysis Reporting System (FARS) provides fatality data.  The VMT data are derived from the Federal Highway Administration’s (FHWA) Highway Performance Monitoring System (HPMS).

Limitations:

FARS data elements are modified from year to year to respond to emphasis areas, vehicle fleet changes, and other needs for improvement. Large truck VMT reported to FHWA by each state is based on a sample of road segments and is not a census.  In addition, the methods used to calculate total VMT may vary from state to state. The methods used by the states to estimate the VMT contribution from rural and urban minor collectors are unknown. 

Statistical Issues:

The fatality counts in FARS are generally quite accurate.  The major sources of error are underreporting by some precincts and inconsistent use of the definition of a truck.  Based on 1994-2000 data, the chance variation in a given year has a regression standard error of approximately 157 fatalities.  Because the VMT data provided to FHWA from each state are estimates based on a sample of road segments, the numbers have associated sampling errors.  The methodology used by each of the states to estimate VMT is not known and may introduce additional non-sampling error.  Although states provide VMT estimates on an annual basis, they are only required to update their traffic counts at all sampling sites once every three years.  Thus an annual VMT estimate from a particular state may be based, in part, on data collected during a previous year.  Based on 1994-2000 data, the chance variation in a given year in the number of fatalities per 100 million vehicle miles of large truck travel has a regression standard error of 0.053.

Verification & Validation:

Fatality data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis.  Quality control procedures are built into data collection and data processing.  A study using samples of 1989-1990 FARS cases was completed in 1993 to assess the accuracy of data being reported.  FHWA routinely works with state data providers to modify reported VMT values that do not appear reasonable before incorporating them into its final master file.

Comment:

The FARS data have been around for many years and are generally accepted as a good source for describing fatal crashes on the Nation’s highways.  The large truck VMT data used to calculate fatal crash rates have both sampling and non-sampling (i.e., bias) error associated with it.  The impact of these errors on FMCSA’s estimates of large truck crash rates is considered to be minimal.

 

 

Alcohol related highway fatalities                                               Page 18

Measure:

1.  Alcohol-related fatalities per 100 million vehicle-miles traveled.

2.  Percentage of highway fatalities that are alcohol related. (CY) (2001)

Scope:

The number of fatalities resulting from motor vehicle traffic crashes that are alcohol related and occur on public roadways within the 50 states and Washington, D.C. 

Source:

Motor vehicle traffic fatality data are obtained from NHTSA’s Fatality Analysis Reporting System (FARS). FARS is a census of fatal motor vehicle traffic crashes within the 50 states, Puerto Rico, and Washington, D.C.  To be included in FARS, a crash must result in the death of a vehicle occupant or a non-motorist within 30 days of the crash.  The FARS data are based on police crash reports and other state data. FARS includes fatalities on all roadways open to the public, using the National Highways System classification of roads.  Pedestrian and bicycle fatalities that occur on public highways, but do not involve a motor vehicle, are not recorded in FARS.  However, they constitute only a small number of fatalities.  A fatal motor vehicle traffic crash is alcohol-related if either a driver or a non-motorist (such as a pedestrian) involved in the crash had a measured or estimated blood alcohol concentration (BAC) of 0.01 grams per deciliter or above.

Limitations:

Blood Alcohol Concentration test results are not available for all drivers and non-occupants involved in fatal crashes.  Missing data can result for a number of reasons -- the most frequent of which is that persons are not always tested for alcohol.  To address the missing data issue, NHTSA has developed a statistical model (Multiple Imputation) to estimate specific values of BAC across the full range of possible values.  Estimating missing BAC in this manner will permit the estimation of valid statistics such as variances, measures of central tendency, confidence intervals and standard deviations. The statistical model is based on important characteristics of the crash including crash factors, vehicle factors, and person factors.  While this measure does not link alcohol with fault in fatal crashes, the more comprehensive scope of the measure compensates for a possible undercount of the extent of the alcohol impaired driving problem.  Multiple Imputation differs from the statistical model used in previous years.  However, all historical series of alcohol involvement will be revised back to the 1982 data year to reflect the estimates from the new methodology.

Statistical Issues:

The primary sources of uncertainty in this performance measure arise from information gaps in the number of intoxicated non-motorists, and from using the statistical model to estimate the number of intoxicated drivers. 

The estimates of the number and percentages of persons killed in motor vehicle traffic crashes during 2001 included in this section are preliminary and are based on incomplete data and statistical models. They were provided to meet the time restraints required for this report.  NHTSA’s first official estimates for 2001, the Early Assessment, are being developed and will be completed in early April.  Differences between the Official Early Assessment estimates and those in this report are to be expected.

Verification & Validation:

Data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis. Quality control procedures are built into annual data collection at 6 and 9 months, and at year’s end.  In 1987 and 1988, an independent panel of academics reviewed and commented on the statistical methods used in measuring alcohol-related highway fatalities.  This report recommended that research and development utilize a model that would permit the imputation of missing BACs as a semi-continuous variable.

Comment:

This data program has been used for many years and is generally accepted for describing safety on the Nation’s highways. 

 

 

 

 

Highway injured persons rate                                                      Page 18

Measure:

Injured persons per 100 million vehicle-miles-traveled (VMT) (CY) (2001)

Scope:

The number of injured persons is an estimate of the total number of persons injured in motor vehicle traffic crashes that occur on public roadways in the 50 states and Washington, D.C.

Vehicle Miles of Travel (VMT) represent the total number of vehicle miles traveled by motor vehicles on public roadways within the 50 states and Washington, D.C.

Source:

The number of injured persons data are derived from the NHTSA’s National Automotive Sampling System (NASS) General Estimates System (GES).  The NASS GES is a nationally representative probability sample that yields national estimates of total nonfatal injury crashes, injured persons, and property-damage-only crashes. NASS GES data cover all roadways open to the public, using the National Highways System classification of roads.

VMT data are derived from FHWA’s monthly report, Traffic Volume Trends (TVT), a monthly report based on hourly traffic count data in the Highway Performance Monitoring System (HPMS).  Information is transmitted to NHTSA where it is reviewed for consistency and accuracy before being entered into the system. These data, collected at approximately 4,000 continuous traffic counting locations nationwide, are used to determine the percentage change in traffic for the current month from the same month of the previous year.  The percentage change is applied to the nationwide travel for the same month of the previous year to obtain an estimate of nationwide travel for the current month.  The data are recorded as monthly totals and cumulative yearly totals.

Limitations:

GES data are obtained from a nationally representative sample of 60 sites.  The results provide only national data, not state level data, and are subject to sampling error.  The magnitude of the sampling error depends on the number of Primary Sampling Units (PSUs) in the sample and the number of crash reports sampled within each PSU. 

VMT data are subject to sampling errors, whose magnitude depends upon how well the continuous counting locations represent nationwide traffic rates.  HPMS is subject to estimating differences in the states, although FHWA works to minimize such differences and differing projections on growth, population, and economic conditions which impact driving behavior.

Statistical Issues:

The estimate of the injury rate includes three main sources of uncertainty.  The numerator count of injuries has a standard error of 5.1% (cf. Appendix C of Traffic Safety Facts).  The denominator estimate of VMT contains both complex sampling and non-sampling errors.  Based on data from 1994-2000, the annual variation in the injury rate has a regression standard error of 4.04.

The estimates of the number and percentages of persons injured in motor vehicle traffic crashes during 2001 are preliminary and are based on incomplete data and statistical models.  NHTSA’s first official estimates for 2001, the Early Assessment, are being developed and will be completed in early April.  Differences between the Official Early Assessment estimates and those in this report are to be expected.

Verification & Validation:

Data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis. Quality control procedures are built into annual data collection at 6 and 9 months, and at year’s end.   A study was completed in 1993, looking at samples of FARS cases in 1989 through 1990 to assess the accuracy of data being reported.  VMT data is reviewed by FHWA for consistency and reasonableness.

Comment:

This data program has been in use for many years and is generally accepted for describing safety on the Nation’s highways.  GES records injury severity in four classes: incapacitating injury, evident but not incapacitating injury, possible but not visible injury, and injury of unknown severity.  Adjusting raw highway fatalities and injuries by VMT provides a means of portraying the changes in highway fatalities on a constant exposure basis – to facilitate year-to-year comparisons.

 

 

Large truck-related injured persons                                            Page 18

Measure:

Number and rate of injured persons involving large trucks. (CY) (2001)

Scope:

The measure includes all injured persons (e.g., drivers and occupants of passenger cars, motorcycles, large trucks, or pedestrians) associated with crashes involving trucks with a gross vehicle weight rating of 10,000 pounds or more. The number of injured persons is derived from NHTSA’s General Estimates System (GES). The injury rate is the number of injured persons per 100 million vehicle miles of large truck travel (VMT).

Source:

NHTSA’s General Estimates System (GES) provides injury data. VMT data are derived from the Federal Highway Administration’s (FHWA) Highway Performance Monitoring System (HPMS).

Limitations:

GES data are obtained from a nationally representative sample of 60 sites.  The results provide only national data, not state-by-state data.  Large truck VMT reported to FHWA by each state is based on a sample of road segments and is not a census.  In addition, the methods used to calculate total VMT may vary from state to state. The methods used by the states to estimate the VMT contribution from rural and urban minor collectors are unknown.

Statistical Issues:

The GES data have a standard error of 6.9% for injuries from truck and automobile crashes (cf. Appendix C of Traffic Accident Reports).  They are less accurate than the corresponding fatality counts.  Based on 1994-2000 data, the variation due to random chance in the number of injuries, which includes sampling variability, has a regression standard error of approximately 7,091.  Because the VMT data provided to FHWA from each state are estimates based on a sample of road segments, the numbers have associated sampling errors.  The methodology used by each of the states to estimate VMT is not known and may introduce additional non-sampling error into the estimates.  Although states provide VMT estimates on an annual basis, they are only required to update their traffic counts at all sampling sites once every three years.  Thus an annual VMT estimate from a particular state may be based, in part, on data collected during a previous year.  Based on 1994-2000 data, the chance variation in a given year in the number of injured persons per 100 million vehicle miles of large truck travel has a regression standard error of 4.39.

Verification & Validation:

Injury data are reviewed and analyzed by NHTSA’s National Center for Statistics and Analysis. Quality control procedures are built into data collection and data processing.  FHWA routinely works with state data providers to modify reported VMT values that do not appear reasonable before incorporating them into its final master file.

Comment:

The data program has been around for many years and is generally accepted for describing safety on the Nation’s highways.  GES records injury severity in four classes: incapacitating injury, evident injury but not incapacitating, possible but not visible injury, and injury of unknown severity.  The large truck VMT data used to calculate injured persons rates have both sampling and non-sampling (i.e., bias) error associated with it.  The impact of these errors on FMCSA’s estimates of large truck crash rates is considered to be minimal.

 

Seat belt use                                                                                 Page 18

Measure:

Percentage of front occupants using seat belts. (CY) (2001)

Scope:

The proportion of front seat outboard passenger vehicle occupants using shoulder belts during daylight hours.

Source:

Data for 1998, 1999, and 2000 are from the National Occupant Protection Use Survey (NOPUS).  NOPUS is a National, multi-stage probability sample.  In the first stage, counties or groups of counties (Primary Sampling Units or PSUs) were grouped by region (Northeast, Midwest, South, and West), level of urbanization (metropolitan or not), and level of belt use (high, medium, or low).  Fifty PSUs were selected based on the vehicle miles of travel in those locations.  In the next stage, a random sample of eight (8) Census Tracts was selected within each of the PSUs.  In the final stage a sample of ten (10) roadway segments for all types of roads was selected within each Census Tract.  In the even numbered years, shoulder belt use of front seat outboard (driver and right front seat) passenger vehicle (passenger cars, vans, sport utility vehicles, and pickup trucks) occupants was observed during daylight hours at each of the 4,000 sampled roadway segments.  In 1999, a Mini-NOPUS consisting of observation at a subsample of 2,000 of the 4,000 roadway segments was conducted.

Estimates of national shoulder belt use for other years shown in the graph are based on state belt use surveys.  These surveys are conducted by most of the 50 States and the District of Columbia.  For the years shown, these surveys varied in coverage, design, and observation methods.  National averages were obtained by weighting the most recently provided state belt use estimate by the population of the state.

Limitations:

NOPUS data are based on a random sample of sites and, therefore, are subject to sampling error.  For the estimate of overall National shoulder belt use from the 2000 NOPUS Survey, sampling error was estimated to be 1.4 percentage points.  Additionally, observation of shoulder belt use is restricted to daylight hours.

State belt use surveys have been conducted in many different ways.  Less than half of the states conducted probability based surveys and the rest were based on other methods.  Additionally, most states conducted surveys that observed use only for those occupants and vehicles covered by their state belt use law.  After enactment of a grant program in the ISTEA of 1991, some 24 states had surveys that met design criteria specified by NHTSA.

Statistical Issues:

The primary source of uncertainty in NOPUS is sampling errors.  The most recent estimate shown in this report is based on a probability sample, and the survey bias and reweighting are complex.  For State surveys, uncertainty derives from disparities among the different surveys conducted by the states, the use of non-probability samples by many of the states, the differences in persons and vehicles observed, the differing methodologies and processes followed to collect data on the persons and vehicles observed, and the procedures used to estimate overall belt use.  To compute the National average from state rates for a specific year, when a state did not conduct a survey or provide NHTSA with an estimate, the most recent rate provided by that state was substituted.  Also, weighting state averages by population may have overstated the contributions of some states.  Based on data from 1994-2000, the annual variation in the seat belt use rate has a regression standard error of 1.31 percent.

Verification & Validation:

NOPUS data collection is managed by a survey research contractor who has responsibility to hire and train the data collectors/observers.  Before data collection begins, NHTSA reviews and approves all the training materials and Data collectors/observers must pass a 2-day training course.  The data contractor also conducts on-scene “surprise” quality control visits to ensure that observations are made correctly and data are coded properly.  Numerous edits are also employed in the data processing.  NHTSA reviews the data provided by the contractor for consistency. NHTSA reviewed and approved the survey designs and data collection procedures for 24 states as a result of a grant program authorized by the ISTEA of 1991.  NHTSA, however, did not conduct any quality review or validation of the data collection and estimation processes employed by the states during or after data collection for the years shown.

Comment:

None.

 

Air carrier fatal accident rate                                                      Page 21

Measure:

Fatal aviation accidents (U.S. commercial air carriers) per 100,000 departures. (FY)

Scope:

This measure includes both scheduled and nonscheduled flights of large U.S. air carriers (14 CFR Part 121) and scheduled flights of commuter airlines (14 CFR Part 135).  It excludes on-demand (i.e., air taxi) service and general aviation.

Source:

Part 121 and Part 135 departure data is submitted to BTS under 14 CFR Parts 241 and 298, respectively.  NTSB provides accident data.

Limitations:

The fatal accident rate in these categories is small and could significantly fluctuate from year to year due to the occurrence or non-occurrence of a single accident.

Statistical Issues:

The switch from calendar to fiscal year in 2001, combined with the use of departures rather than flight hours as the activity measure for the denominator, present new problems.  The FAA has no independent data sources to validate BTS-collected departure data as it did with flight hour data.  To overcome reporting delays of 60 to 90 days, FAA must rely on historical data, partial internal data sources, and Official Airline Guide (OAG) scheduling information to project at least part of the fiscal year activity data.  Due to the reporting procedures in place, it is unlikely that calculation of future fiscal year departure data will be markedly improved.  Lacking complete historical data on a monthly basis and independent sources of verification increases the risk of error in the activity data.  The regression standard error for the annual variation in the fatality rate, based on data from 1994 – 2000, is 0.023.

Verification & Validation

The FAA does comparison checking of the departure data collected by BTS; however, FAA has no independent data sources against which to validate the numbers submitted to BTS.  FAA compares its list of carriers to the DOT list to validate completeness of the reporting list and places the carriers in the appropriate category (i.e., Part 121 or Part 135).  NTSB and FAA’s Office of Accident Investigation meet regularly to validate the accident count.

Comment:

The joint government/industry group working on improving the level of safety for U.S. commercial aviation has determined that the number of departures is a better denominator measure to use for determining accident rates.  In a recent report on the Safer Skies effort the Government Accounting Office agreed and recommended that the FAA use departures. 

 

General aviation fatal accidents                                                  Page 21

Measure:

Number of fatal general aviation accidents. (FY)

Scope:

The measure includes on-demand (non-scheduled FAR Part 135) and general aviation.  General aviation comprises a diverse range of aviation activities.  The range of general aviation aircraft includes single-seat homebuilt aircraft, helicopters, balloons, single and multiple engine land and seaplanes including highly sophisticated extended range turbojets.

Source:

National Transportation Safety Board (NTSB).

Limitations:

The use of the 1996-1998 timeframe for the baseline represents one of the safest periods in general aviation history in terms of a decline in fatal accidents.  The number of general aviation accidents reported in any given year might change in subsequent years.  There are many reasons for these changes to the historical data.  Primary among them is that the accident had not been reported to the NTSB, or that it was misreported and the information corrected at a later date.

Statistical Issues:

There is no major error in the accident counts.  Random variation in air crashes results in a significant variation in the number of fatal accidents over time.  The regression standard error in this variation for 1996 through 2000 is 16.5. 

Verification & Validation:

NTSB and FAA’s Office of Accident Investigation meet regularly to validate the information on the number of accidents. 

Comment:

It would be preferable to use fatal accident rates rather than fatal accidents as the performance measure. However, general aviation flight hours are based on an annual survey conducted by the FAA.  Response to the survey is voluntary.  The accuracy of the flight hours collected is suspect and there is no readily available way to verify or validate the data.  For this reason, the General Aviation community is unwilling to use a rate measure until the validity and reliability of the survey data can be assured.

 

Operational Errors (Air Traffic)                                                    Page 24

Measures:

1.  Operational errors per 100,000 activities, or per 1 million activities. (2001)

2. Number of operational errors where less than 80 percent of required separation is maintained.

Scope:

An error occurs when separation between aircraft is less than the separation determined necessary for the specific phase of flight.  “Activities” are total facility activities, as defined in Aviation System Indicators 1997 Annual Report.  Total facility activities are the sum of en route and terminal facility activities.

Source:

FAA air traffic facilities have a software program called Operational Error Detection Patch (OEDP) that detects possible operational errors and sends alert messages to supervisory personnel.  Facility management reviews OEDP alerts and data provided from the National Track Analysis Program (NTAP) to determine if an operational error has occurred.  Controllers are required to report operational errors.  The information is summarized in the FAA Air Traffic Operational Error and Deviation Database.

Limitations:

There is a few months’ lag in reporting data because of the need to investigate major incidents.  The severity of errors is not measured.  Minor errors such as a 4.5-mile rather than a 5-mile separation are counted in the same way as more serious errors.  Data are available for 1994 and following years.

The DOT IG conducted an audit of reporting on operational errors.  The IG believes that there is a potential for underreporting of operational errors, as some errors are self-reported.  The FAA disagrees with this assessment because there are substantial penalties for not reporting an operational error.

Statistical Issues:

There are no major sources of systematic error in the operational errors data that have been quantified.  Again, random variation in operational errors results in a significant variation in the measured rates over time. The regression standard error in the operational error rate using 100,000 activities denominator and the 1 million activities denominator, based on 1994-2000 data, are .048 and .48, respectively.

Verification & Validation:

FAA performs system checks and counts daily against reported data to ensure the accuracy of information reported.

Comment:

In August 1998, the FAA discovered and corrected a misunderstanding of the procedures used in interpreting separation reported by the National Track Analysis Program and the data provided by the Operational Error Detection Patch.  The corrected application of these procedures, while not affecting safety, has resulted in an overall increase in the number of errors reported between 4.6 and 4.9 miles separation (Standard separation in these cases is 5 miles).

 

Runway incursions                                                                        Page 24

Measures:

1.  Number of runway incursions. (FY) (2001)

2.  Number and rate (per 100,000 operations) of highest risk runway incursions.

Scope:

Runway incursions are the result of ground collision hazards or loss of separation for aircraft in the process of taking off or landing. They are grouped in three general categories:  operational errors, surface pilot deviations, and vehicle/pedestrian deviations. Incursions are reported and tracked at airports that have an operational air traffic control tower. 

Source:

Air traffic controllers and pilots are the primary source of runway incursion reports. The data is recorded in the FAA National Incident Monitoring System (NAIMS).

Limitations:

Preliminary incident reports are evaluated when received. Evaluation can take up to 90 days.

Statistical Issues:

There are no major sources of systematic error in quantified runway incursion data. The regression standard error in the reported number of incursions, based on 1994-2000 data, is approximately 15.4.  Based on 1998 – 2001 data, the regression standard error for the number and rate of highest risk runway incursions are 8.8 and 0.01, respectively.

Verification & Validation:

Surface incidents are reported in the Administrator’s Daily Bulletin at the beginning of each weekday. Surface incidents are evaluated to determine if they should be classified as incursions. Incidents are evaluated against the official runway incursion definition. The Air Traffic Runway Safety Program Manager, ATP-20, makes the final decision regarding runway incursions.

Comment:

None.

 

Mariner Rescue                                                                             Page 26

Measure:

Percent of all mariners in imminent danger who are rescued. (FY)

Scope:

Includes people in water; on shore; and aboard a vessel, offshore structure, pier, or vehicle that is in distress or in urgent need of assistance.  The Coast Guard makes a final determination on scene whether there is imminent danger, based on criteria that include the nature of distress, the condition of the vessel, the people onboard, and the environmental conditions.  Criteria for this decision are discussed in search and rescue doctrine publications.

Source:

CG Search and Rescue Management Information System (SARMIS). Data is collected from Coast Guard field units that conduct search and rescue responses.

Limitations:

It is probable that some number of imminent danger cases, and the associated lives, are not reported in SARMIS.  This includes situations where no distress call was received by the Coast Guard and the persons in distress were rescued by private citizens or local government personnel, or where the persons in distress perished without trace. The extent of this under-reporting is not known. There is some judgment involved in assessing whether mariners are in danger. However, there is likely to be consistency in these assessments across years.  1994 data is skewed upward by a large surge of migrants interdicted at sea, most of whom were counted as “rescued,” thus increasing the percentage of lives reported as saved.  Reporting no longer includes migrants interdicted; they are counted directly as migrants interdicted under law enforcement activity.  Prior to the introduction of the next generation data system in October 2000, data entry was limited to closed cases, after a rescue has been successfully completed or after the recovery of a body.  The new data system now allows missing bodies to be tracked.  In this first year of data, more cases than expected were found where bodies were not recovered.  Before adding this number into our data analysis, we will track this number to assure that this represents a data trend and not an unusual aberration.  Errors may be introduced in SARMIS through data entry, but are likely rare for lives saved data elements.

Statistical Issues:

The primary source of uncertainty consists of non-sampling errors. The second generation data system, brought on-line on October 1, 2000, reduces error due to miscoding through the use of more extensive drop down menus, machine generated case numbers, structured data boxes, and more extensive business rules eliminating the selection of data not consistent with other entered data. The regression standard error for year-to-year chance variation is 2.6 percent mariners rescued, based on data from 1994 through 2000.

Verification & Validation:

SARMIS data entry system uses structured entry values, check boxes, and pull down selection lists to limit entry errors.  The use of plain language descriptions eliminates a majority of erroneous data code selection. Additional system business rules also eliminate the selection of data not appropriate with other entered data. The SAR Mission Coordinator (SMC) is responsible for accurate entry of particular case data by all units involved in the case. CG Program Managers annually validate the data in SARMIS. Entries are reviewed at Coast Guard District offices as first step in validation – errors and inconsistencies are identified and corrected.  Finally, Coast Guard Headquarters program managers review compiled data annually to assess consistency with historic variance and trends.  This review includes curvilinear regression analysis to compare current data to historic data and a program review analysis to identify and resolve aberrations.

Comment:

Beginning in FY01, this measure will cover all mariners in distress reported in SARMIS.  The previous measure covered only mariners reported in distress that were rescued.  The significance of the 87.5% result for FY99 is uncertain at this point; FY95-98 data show a flat trend at 84%.  It is not known if the FY99 result was produced by anomalous factors, or if it is the product of program strategies and a changing external environment.  Therefore, the goal target remains at 85% until more analysis is completed.  For FY 2001, the preliminary estimate of the measure was 84.2 percent of all lives, bringing the percentage about equal to the average since 1995 and slightly below the goal, but certainly within normal variation about the average. 

 

Recreational boating fatalities                                                    Page 28

Measure:

Number of recreational boating fatalities. (CY) (2001)

Scope:

Measure includes fatalities occurring aboard vessels that are being operated for recreational purposes.  Surfboards, iceboats, and vessels engaged in sanctioned racing events are not considered recreational vessels.  Fatalities are included if caused by a fire, explosion, sinking or other occurrence involving a recreational vessel, and the vessel or associated equipment caused or contributed to the fatality.  Fatalities are not included if they occurred aboard a recreational vessel, but were caused by self-inflicted wounds or natural causes.  Fatalities are also excluded if they occurred while the victim was engaged in other activity such as swimming or diving, where the vessel was used as a platform only and was not a contributing factor to the fatality. Beginning two years ago, the measure for Recreational Boating was revised by adding an additional 6% to the aggregate number of reported fatalities, to correct for an estimated 6% underreporting of recreational boating fatalities.

Source:

Coast Guard Boating Accident Report Database (BARD).  Data is entered into BARD by state administrators who collect data from boat owners and operators through formal Boating Accident Reports, as instructed in 33 CFR 173c. 

Limitations:

Fatality data is derived from reports submitted by the public along with accompanying state investigation reports. There is consensus among the Coast Guard, the states, safety professionals, and other researchers that most fatalities that occur on inland and most coastal waters are under-reported.  To better quantify the extent of possible under-reporting the Coast Guard initiated and funded an analysis of BARD data conducted by the Boat Owners Association of the United States (BOAT/U.S.) Foundation for Boating Safety.  The study found some fatalities involving recreational boating in the Coast Guard’s Search and Rescue Management Information System (SARMIS) that were not in BARD. However, although the study reported a 9% discrepancy, further analysis revealed that some of these findings would not be reportable as recreational boating fatalities. There is also consensus that under-reporting exists for fatalities occurring offshore, and aboard U.S. recreational boats operating overseas.  Also, although there are guidelines as to what constitutes a recreational boating fatality, there is still an element of interpretation at the state level in reporting fatalities.  It is probable that the states do not always interpret the guidelines in the same manner. Overall, the best estimate indicates that total fatalities are currently under-reported by at least 6%.

Statistical Issues:

The discrepancy between BARD and the Search & Rescue Management Information System (SARMIS) amounts to 6% of the total reports for those states covered by SARMIS.  The numbers given in this report have been adjusted to correct the deficiency.  Also, note that since the boating fatality counts are influenced by weather, gasoline prices and other external factors, annual chance variation should be large.  Using data from 1994 to 2000, the annual variation in the number of fatalities attributable to random chance has a regression standard error of 50.7.

Verification & Validation:

Fatality data in BARD is verified and validated by state boating administrators and Coast Guard program managers. At the end of the calendar year, the Coast Guard compiles state fatality data and sends a report to each state for confirmation.  Both State and Coast Guard officials review the statistics, including sampling of cases to ensure guidelines for classifying fatalities were followed.  Any discrepancy is reconciled jointly by the State and Coast Guard program manager.

Comment:

Data are not normalized for increases or decreases in the number or usage of boats, which tends to limit data use in making comparisons over time.  The number and usage of recreational boats has increased over the past 2 decades, while the raw number of fatalities has generally decreased.

The BOAT/US review of BARD data for 1993 through 1997 identified underreporting in BARD of 8% in 1993 and 1994, 12% in 1995, 13% in 1996 and 8% in 1997.  The Coast Guard reviewed BOAT/US’s findings for 1995, 1996, and 1997.  Each record for these years was checked and fatalities that were incorrectly labeled as recreational boating fatalities by BOAT/US were removed from the count.  Based on this revised count of recreational boating fatalities with mislabeled fatalities removed, the Coast Guard estimates that 7%, 8% and 4% of all recreational boating fatalities were not captured in its Boating Accident Report Database (BARD) in 1995, 1996 and 1997 respectively for purposes of this report. The median of these numbers – 6% - has been used to adjust recreational boating safety data for 1993, 1994, 1998 and 1999, and to reset the goals for 1999 through 2001. The original goal of 720 has been increased by 6% to 763 for 2000.

The Coast Guard is in the process of commissioning a comprehensive National Boating Survey to obtain valid and reliable information on boating practices, safety, and exposure.  This information will enable safety officials to assess boating risk, implement appropriate safety intervention strategies, and measure the effectiveness of program activities in reducing the risk and negative outcomes associated with the use of recreational boats. Data from this study will be used to further address underreporting issues and estimate reporting discrepancies in BARD.  The study was originally set to begin in Fall 2001, however data collection is now scheduled to begin in April 2002.

 

Passenger Vessel Fatalities                                                          Page 28

Measure:

Fatalities and rate (per million passenger capacity) aboard passenger vessels. (2001)

Scope:

This measure is an indicator of passenger safety.  It includes reportable marine casualties resulting in the death or disappearance of a passenger aboard any U.S. vessel (regardless of type or location) or aboard foreign flag vessels in U.S. waters. Exceptions include death/disappearance of “non-passengers”, whenever the cause of death/disappearance is classified as being from diving, natural causes, (e.g. heart attack) or whenever the death/disappearance is the result of an intentional act (e.g. suicide, altercation).  Fatalities on recreational vessels are not included for two principal reasons: Recreational vessels are prohibited from carrying “passengers” and recreational vessel fatalities are measured and reported separately.

Source:

Passenger fatality source data is obtained from the Coast Guard Marine Safety Information System (MSIS).  Passenger fatalities are reported to the Coast Guard as required by federal regulations.  Sources of reports are most often vessel masters, operators, owners, insurance companies, legal representatives, and other mariners.

Limitations:

The investigation, retrieval, analysis and reporting processes result in under-reporting for the most recent year, with the most significant effects over the most recent 5 months.  Estimates are often used to compensate for this known data-lag. The Coast Guard initiates about 40-50 civil penalty cases for failure to report marine casualties, although many of these are for minor casualties. In addition, some passenger fatalities may not be reported to the Coast Guard. This number is unknown.  Some passenger injuries may ultimately prove fatal and lead to death; some missing passengers may be found.  These numbers may not be updated to reflect the changes in status.  The number is believed to be small.  Duplicate casualty entries are sometimes entered into MSIS, and some casualties are mistakenly omitted or coded incorrectly. Verification procedures strive to correct these errors, but it is probable that a small number are not corrected. The data retrieval & reporting processes do not allow automated distinction between all death types (e.g. natural vs. accidental).  As a result, some natural deaths or suicides may be inadvertently included.

Statistical Issues:

The major sources of uncertainty in this measure are the estimation error (as a result of the data-lag) and the reporting error (as a result of the inability to distinguish between which deaths should be included and which should be excluded).

Verification & Validation:

Verification and validation occurs at several levels.  Edit checks within MSIS software can detect some incorrect or missing data and force review and correction before data entry is completed.  Selection lists for certain data fields also reduce the opportunity for data entry error.  All investigations go through review at the field unit for accuracy.  Investigations of serious marine casualties are also usually reviewed at district and headquarters offices.  The headquarters Data Administration staff conducts periodic quality control checks to identify entry errors such as missing data or miscoding, and corrects any errors identified.  Errors identified are referred to either the Data Administration staff or the Investigations and Analysis staff for correction.

Comment:

During FY 2002, the Marine Safety Information System (MSIS) will be replaced by the Marine Information System for Safety and Law Enforcement (MISLE).  While the new system will be a major improvement, it is expected to cause serious difficulties in making performance comparisons.  One factor is that many business processes were re-designed in conjunction with system development.  Another factor is that data quality under MISLE is expected to be superior to that of MSIS.  While this represents improvement, it may cause near-term problems in making meaningful comparisons of data between the two systems.

 

Rail fatality and Accident rates                                           Page 29 & 30

Measure:

1.  Train accidents per million train-miles.

2.  Rail-related fatalities per million train-miles. (CY) (2001)

Scope:

The fatality measure includes anyone on rail property, any on-duty railroad employee, and anyone killed by a train or its contents.  It does not include fatalities on trains or rail lines that do not connect to the national rail network, such as mass transit operations, certain excursion and tourist railroads, and some industrial railroads not connected to the general system.  The only railroad fatalities that are not counted are suicides (as determined by a public official) and death by natural cause not associated with railroad operations. Train accidents do not include those at grade crossings.  They are reported under the performance goal for highway-rail grade crossing accidents.

Source:

Railroad Safety Statistics – Annual Report.  Statistical data, tables, and charts depict the causes and nature of rail-related fatalities.  Data on fatalities and train miles are reported to FRA by railroad companies.

Limitations:

Because of the scope of the reporting criteria, some fatalities that are counted are not associated directly with operation of the trains, and some railroad fatalities are not counted.  This scope is consistent with the regulatory authority of the agency, but not consistent with other modes of transportation for comparative purposes.

Statistical Issues:

The reported estimates are based upon partially reported data from 2001.  Based on data from 1994-2000, chance variation from year to year, as reflected in the regression standard error, is 0.055 for rail fatalities.

Verification & Validation:

Railroads are required by law to submit monthly accident/incident reports to FRA.  They are also required to update any inaccurate or incomplete information.  FRA conducts routine data audits (records inspections) to verify the adequacy of railroad reporting and record keeping requirements.

Comment:

None.

 

Highway - Rail grade-crossing accidents                                     Page 29

Measure:

Grade-crossing accidents divided by the product of: 1) million train miles and 2) trillion vehicle-miles-traveled. (CY)

Scope:

The measure includes all collisions with on-track equipment and highway users at public and private grade crossings.

Source:

Collisions and train-miles are reported in FRA’s Railroad Safety Statistics – Annual Report.  Vehicle-miles-traveled (VMT) are obtained from the FHWA Office of Highway Information Management. 

Limitations:

Because the denominator includes all highway vehicle-miles-traveled (VMT), and not just VMT that are exposed to grade crossings, the rate portrayed may be lower than the actual risk.

Statistical Issues:

Trains and automobiles have different exposures at rail crossings---the denominator used here attempts to combine these.  The numerator is based on partially reported 2001 data. The annual variation by chance from year to year as measured by the regression standard error is 0.109, based on data from 1994-2000.

Verification & Validation:

FRA’s Office of Safety has a review process to ensure that railroads and the States comply with Federal reporting requirements in the preparation of the FRA Railroad Safety Statistics - Annual Report.

Comment:

None

 

Transit fatality and injured person rates                            Page 30 & 31

Measure:

1.      Transit fatalities per 100 million passenger miles traveled. (CY)

2.      Transit injured persons per 100 million passenger miles traveled. (CY)

Scope:

The data include both riders and employees.  A fatality is defined as a transit-caused death from collision, personal casualty, fire, derailment, or bus going off the road.  An injury is defined as any physical damage or harm to a person requiring medical treatment caused by a transit collision, personal casualty, fire, derailment, or bus going off the road.

Source:

FTA’s Safety Management Information System (SAMIS), with data reported by transit operators to the National Transit Database (NTB).

Limitations:

Because of the scope of the reporting criteria, some fatalities that are counted are not associated directly with transit operation.  This scope is consistent with the regulatory authority of the agency, but not consistent with other modes of transportation for comparative purposes.

Statistical Issues:

The fatality and injury counts in SAMIS are generally quite accurate---the major source of error in the measure comes from uncertainty in the passenger miles traveled.  Based on 1994-2000 data, the chance variation in a given year has a regression standard error of 0.039 for the transit fatality rates and 2.210 for the transit injury rates.

Verification & Validation:

An independent auditor and the transit agency’s CEO certify that data reported to the NTD are accurate.  Using data from the NTD to compile the SAMIS data, the Transportation Systems Center compares current safety statistics with previous years, identifies questionable trends, and seeks explanation from operators.

Comment:

None.

 

 

Pipeline failures                                                                            Page 33

Measure:

Excavation damages to natural gas and hazardous liquid pipelines. (FY)

Scope:

This measure is based on reported hazardous liquid and natural gas accidents that meet federal reporting criteria as defined in 49 CFR 191.1 and 191.15 for natural gas transmission pipeline incidents and in 49 CFR 195.50 for hazardous liquid pipelines. 

Source:

RSPA’s Natural Gas Distribution and Transmission Incident Reports and Hazardous Liquid Pipeline Accident Reports. Failure reports are filed within 30 days of the occurrence of reportable incidents.  Complete calendar year data are available by March 1 of the following year.  Data may change as operators file supplemental reports.

Limitations:

RSPA lacks adequate infrastructure information on pipeline operations and maintenance needed to fully characterize problems when they occur and lacks information on precursor conditions that contribute to incidents.  RSPA seeks further improvements in data collection in 2002 to address these concerns.

Statistical Issues:

Reduction in excavation damages is tied to economic growth and expansion as populations increasingly are encroaching on once rural areas where major interstate pipelines are located.  Because of delays in mail delivery associated with 9/11/2001 terrorist activities, statistical close-out of the 2001 tally requires an extrapolation of number of reports anticipated for the last quarter of 2001.

Verification & Validation:

RSPA reviews/verifies data provided for accuracy and requests supplemental reports where shortcomings are indicated.

Comment:

RSPA discontinues this measure after 2002, replacing this safety measure with pipeline excavation damages measure.

 

Pipeline failures                                                                            Page 34

Measure:

Failures of natural gas transmission pipelines. (CY) (2001)

Scope:

This measure is based on reported hazardous natural gas leaks that meet federal reporting criteria as defined in 49 CFR 191.1 and 191.15 for natural gas transmission pipeline incidents. 

Source:

RSPA’s Natural Gas Transmission Incident Report. Failure reports are filed within 30 days of the occurrence of reportable incidents.  Complete calendar year data are available by March 1 of the following year.  Data may change as operators file supplemental reports.

Limitations:

RSPA lacks adequate infrastructure information on pipeline operations and maintenance needed to fully characterize problems when they occur and lacks information on precursor conditions that contribute to incidents.  Joint Federal, state and industry teams have been formed to devise a new course to improve information availability.

Statistical Issues:

The number of failures of natural gas transmission pipelines is likely to be underreported.  The annual variation in the number of failures from year to year due to chance has a regression standard error of 528 for natural gas pipeline failures based on data from 1994 to 2000.

Verification & Validation:

RSPA reviews/verifies data provided for accuracy and requests supplemental reports where shortcomings are indicated.

Comment:

None.

 

 

Hazardous Materials Incidents                                                     Page 36

Measure:

Number of serious hazardous materials incidents in transportation. (CY)

Scope:

Serious reported hazardous materials incidents were initially defined by RSPA to be those that result in a fatality or major injury (for most purposes, an injury resulting in hospitalization) due to a hazardous material, closure of a major transportation artery or facility, or evacuation of six or more persons due to the presence of a hazardous material, or a vehicle accident or derailment resulting in the release of a hazardous material.  For the 2003 Plan, the definition is revised to include those incidents resulting in a fatality or major injury, the evacuation of 25 or more employees or responders or any number of the general public, the closure of a major transportation artery, the alteration of an aircraft flight plan or operation caused by the release of a hazardous material or the exposure of hazardous material to fire; plus any release of radioactive materials from Type B packaging, Risk Group 3 or 4 infectious substance, over 11.9 gallons or 88.2 pounds of a severe marine pollutant, or a bulk quantity (over 119 gallons or 882 pounds) of a hazardous material.  This measure tracks only transportation related releases of hazardous materials that are in commerce.  Volume of spills is not tracked, as this does not necessarily indicate risk.

Source:

Hazardous Materials carriers report data to RSPA for entry into the Hazardous Materials Information System (HMIS).

Limitations:

Data for all hazardous materials incidents is suspected of being incomplete due to under-reporting for minor incidents.  Most reportable serious incidents are in the system, making this a more consistent measure for program management.  However, it does not reflect all incidents.  RSPA has issued an NPRM to revise the reporting system.

Statistical Issues:

Although the number of incidents is likely to be underreported, such recording error is probably small in comparison to the annual variation due to chance.  The annual variation in the number of failures (original definition) from year to year due to chance has a regression standard error of 37.2 based on data from 1994 to 2000.  The new incident definition has a regression standard error of 30.6 based on data from 1997 to 2000.

Verification & Validation:

RSPA verifies the data by periodic follow-up reviews of data entry by the manager of the Hazardous Materials Information System, and verification audits of the data entry process.  RSPA crosswalks HMIS reports against the National Response Center log of accidents.  RSPA is improving compliance with reporting requirements by correlating HMIS reports with FRA’s Accident Report data and the HMIS telephonic data.  RSPA is piloting and plans to incorporate procedures to correlate HMIS reports with FHWA’s Safetynet Accident File data.

Comment:

None.

 

Details on DOT Measures of Homeland Security

Aviation security                                                                           Page 41

Measure:

1.  Average waiting time in minutes for passengers in line for screening. (FY)

2.  [Measure on passenger and baggage screening effectiveness.] (FY)

Scope:

  TBD

Source:

  TBD

Limitations:

 

Statistical Issues:

 

Verification & Validation:

 

Comment:

 

 

Aviation Security                                                                           Page42

Measure:

Detection rate for explosive devices and weapons that may be brought aboard aircraft. (FY) (2001)

Scope:

Machine performance test results, automated threat-image projection (TIP) and FAA field agent testing of aviation security screener proficiency to detect and resolve images or FAA test objects that simulate weapons and explosive devices in checked and carry-on baggage, or carried on the person through an airport security checkpoint.

Source:

FAA Office of Civil Aviation Security Airport and Air Carriers Information Reporting System (AAIRS).  Laboratory test results from the William J. Hughes Technical Center.

Limitations:

No comment.

Statistical Issues:

There is no major error present in the subject data.

Verification & Validation:

Special “red team” testing led by agents based at FAA headquarters is used to validate field test results.  AAIRS data is subject to multiple layers of review.

Comment:

The White House Commission recommended more aggressive, realistic testing.  Funding that began in 1997 enabled an increase in testing as more field agents were hired and trained.  Prior to 1998, data from realistic testing were too sparse to be conclusive.

 

Coastal and Seaport Security                                                       Page 43

Measure:

Percent of high interest vessels screened.

Scope:

High Interest Vessel (HIV) inspection or escort is measured by the a ratio of the number of HIV vessel inspected or escorted to the number of HIV vessels arriving at US ports.  HIV designation is determined using specific criteria.  Coast Guard inspection or escort standards are to inspect 100% of HIV.

Source:

The data for this measure is collected using a manual count from situation reports sent after a vessel inspection or escort.

Limitations:

This is an interim activity-based measure.  Appropriate outcome-based measures are under development that will improve our ability to measure and reduce security risks in US ports.

Statistical Issues:

This is a new measure and data systems have not yet been developed or modified to capture this information.  It is possible that errors in the data could result due to manual data collection.

Verification & Validation:

Verification and validation is conducted through cross checks with situation reports.  

Comment:

None.

 

Military Readiness                                                                        Page 44

Measure:

Percentage of days that the designated number of critical defense assets (high endurance cutters, patrol boats, and port security units needed to support Defense Department operational plans) maintain a combat readiness rating of 2 or better. (FY) (2001)

Scope:

Only high endurance cutters, patrol boats, and port security units that are designated as necessary for defense plans are included. The specific units required are classified.

Source:

DOD Status of Readiness and Training System (SORTS) – Database used by the Coast Guard in applying DOD standards to its assets to determine a readiness score.

Limitations:

SORTS uses a multi-factor matrix to calculate the readiness status.  Although specific criteria are outlined for each factor, some judgment is required in applying criteria.  Different units and personnel may apply standard criteria in slightly different ways depending on the nature of the unit’s mission.

Statistical Issues:

This particular performance measure in FY 2001 is based on readiness levels of two types of vessels, patrol boats and high endurance cutters, which have extremely different levels of readiness.  In addition, a third resource, Port Security Units (PSUs), is measured for its readiness.  PSUs are comprised of Coast Guard Reservists and Active Duty personnel, trained to protect foreign ports for expeditionary forces.  The drastic change between FY 1999 and FY 2000 performance was caused in large part due to the fact that the requirement to report the Contingency Personnel Requirements List (CPRL) (the full wartime personnel strength requirement) in the unit SORTS report was waived for FY 2000 and subsequent years pending validation of personnel requirements that have changed due to new equipment and operational procedures.  The Navy has been informed of this waiver and has not objected to reporting personnel strength using the less demanding Coast Guard standards for peacetime operations in view of the fact that Reserve Unit personnel are available to quickly bring Coast Guard units up to the full wartime personnel strength requirements in the event of a war.

Verification & Validation:

Units self assess and report readiness using objective standards.  Unit readiness is periodically validated through inspections, assistance visits, and in some cases training and assessment at Navy facilities.  These assessments are conducted by external, field level commands (such as Coast Guard areas, districts, and groups).

Comment:

Coast Guard will continue to reassess the overall adequacy of this measure.

 

Strategic Mobility                                                                         Page 47

Measure:

Percentage of DOD-required shipping capacity complete with crews available within mobilization timelines (FY)

Scope:

As of March 2002, this measure is based on the material availability of 76 ships in the Maritime Administration’s Ready Reserve Force (RRF) and 115 ships enrolled in the Voluntary Intermodal Sealift Agreement (VISA) program, which includes 47 ships enrolled in the Maritime Security Program (MSP).  A second factor pertinent to this measure is the availability of sufficient licensed and unlicensed mariners to operate the available ships.  The performance measure represents the number of available ships (compared to the total number of ships in the RRF and VISA) that can be fully crewed within the established readiness timelines.  While other Government (primarily Military Sealift Command) owned or controlled sealift type vessels are not included in this measure, they draw their crews from the same pool of mariners.  Accordingly, the availability measure is adjusted to reflect expected requirements during the early stages of a military crisis.

Source:

Material availability of ships: MARAD records (and reports to DOD) on the readiness/availability status of each RRF ship each month.  Typical reasons why a ship is not materially available include: the ship is in drydock, the ship is undergoing a scheduled major overhaul, or the ship is undergoing an unscheduled repair.  MARAD and DOD also maintain records of the sealift ships enrolled in the MSP and VISA and their crew requirements. Availability of mariners: Information on the available supply of licensed and unlicensed mariners is extrapolated from data received from the U.S. Coast Guard’s Merchant Mariner Licensing and Documentation (MMLD) system.

Limitations:

The information on the available supply of licensed and unlicensed mariners is an estimate. Because the MMLD also does not contain all of the information on individual mariners contained in their paper records, and provides no information on the availability and willingness of individuals to accept a sealift position in an emergency, it does not provide sufficient assurance of mariner availability.

Statistical Issues:

None

Verification & Validation:

The MARAD Regional Offices (and contracted ship managers) monitor the condition and overall readiness of each assigned RRF ship to meet its DOD mission.  When a ship is determined not capable of meeting its activation timeframe (mission), it is given one of several vessel condition ratings that are reported to DOD.  The monthly report contains an explanation of the deficiency and an estimated date when the ship will become fully capable of meeting its mission.  MSP contract performance is monitored throughout the year in order to assure proper payment of the MSP payment to the ship operators.  Recently, MARAD attempted to validate mariner availability estimates by conducting a survey of the mariner population.  A second survey is expected to commence in April 2002 to refine and improve the information needed to determine availability.  Because the decision to serve is a matter of individual choice and is subject to change, MARAD intends to develop a plan for maintaining current information on mariner availability based on the results of the 2002 mariner survey.

Comment:

None.

 

DOD-designated port facilities                                                     Page 47

Measure:

Percentage of DOD-designated commercial strategic ports for military use that are available for military use within DOD established readiness timelines.

Scope:

The measure consists of the total number of DOD-designated commercial strategic ports for military use that are assessed as able to meet DOD-readiness requirements on 48-hour notice, expressed as a percentage of the total number of DOD-designated commercial strategic ports.  Presently there are 14 DOD-designated commercial strategic ports.  Port readiness is based on monthly reports submitted by the ports and semi-annual port readiness assessments by MARAD in cooperation with other NPRN partners.  The MARAD/DOD semi-annual port assessments provide data or other information on a variety of factors, including the following: the capabilities of channels, anchorages, berths, and pilots/tugboats to handle larger ships; rail access, rail restrictions, rail ramp offloading areas, and rail storage capacities; the availability of trained labor gangs and bosses; number and capabilities of available cranes; long-term leases and contracts for the port facility; distances from ports to key military installations; intermodal capabilities for handling containers; highway and rail access; number of port entry gates; available lighting for night operations; and number and capacity of covered storage areas and marshalling areas off the port.

Source:

MARAD data are derived from monthly reports submitted by the commercial strategic ports and from MARAD/DOD semi-annual port assessments.

Limitations:

Port readiness assessments were not made prior to 1995; therefore, data are available only for 1995 and later years.  MARAD conducts a monthly survey of all strategic facilities to determine whether they meet the DOD availability requirement.  This information is provided to MARAD as a self-assessment by the port agency that owns the facility.  There is some degree of subjectivity in determining the availability of the port facilities.  As part of the overall planning process, MARAD and DOD conduct semiannual visits to independently verify and reassess port capability and availability.  The indicator is by definition a point-in-time judgment.  The results of the monthly and semi-annual reports used to measure port readiness can vary in accordance with the intensity of commercial activity at a given port at the time of the assessment.  Also, the monthly reports do not include the same level of detail as the semi-annual assessments, although MARAD is in continuous contact with port officials to minimize response error.

Statistical Issues:

The measurement of port readiness is an overall measure derived from MTMC comments, monthly readiness reports, and semi-annual assessments.  As such, it is a subjective measure.

Verification & Validation:

The MARAD/DOD semi-annual port visits independently verify and reassess not only the DOD-designated facilities, but also the total capability of the commercial strategic port.

Comment:

None.

 

Sealift capacity                                                                             Page 48

Measure:

Ship capacity (in twenty-foot container equivalent units, or TEUs) available to meet DOD’s requirements for intermodal sealift capacity. (FY) (2001)

Scope:

Includes the aggregate TEUs (or estimated square footage) of cargo capacity for ships enrolled in the Maritime Security Program (MSP) and Voluntary Intermodal Sealift Agreement (VISA).

Source:

MARAD/USTRANSCOM database of the militarily useful sealift capacity for ships enrolled in the MSP and VISA programs, based on vessel capacity data obtained from the vessel operators. 

Limitations:

MARAD, DOD and operator data on vessel characteristics (e.g., deck strength in pounds per square feet, deck height, container stowage factors), which are used to determine the portion of a vessel suitable for carrying military cargo, are not always consistent.  For example, the majority of ships in MSP/VISA are containerships, which normally are measured in TEUs; however, DOD generally measures surge sealift ships, most of which are Roll-on/Roll-off vessels, in square feet.  Historical data prior to FY 1997 are unavailable since the MSP and VISA programs were not enacted until that year.

Statistical Issues:

None.

Verification & Validation:

MARAD works with DOD and the maritime industry to use the most accurate information.  MARAD validates the vessel capacity data, which are obtained from the vessel operators, through comparisons with internationally recognized databases of vessel characteristics (such as Lloyd’s Register data), vessel trim and stability information, stowage plans and other cargo loading documents.

Comment:

None.

 

Ready Reserve Force (RRF) activation                                        Page 49

Measure:

1.      Percent of RRF no-notice activations that meet assigned readiness timelines. (FY) (2001)

2.      Percent of days that RRF ships are mission-capable while under DOD control. (FY 2001)

Scope:

DOD conducts no-notice exercises, called “Turbo-Activations,” annually to assess RRF activation readiness.  The USTRANSCOM, via MSC, randomly selects and orders the activation of a number of RRF ships on an annual basis to test their capability to be ready-for-sea (i.e., mission-capable) within their assigned readiness timeframes of 4, 5, 10, or 20 days.

Source:

MARAD maintains a database on the number of days it takes to activate each RRF ship and its operational reliability.  The MSC activation order is received either by phone call or message.  Documents produced during the no-notice activation period comprise the data source for determining the amount of time taken to activate each ship.  Non-performance time is based on the MSC Casualty Reporting (CASREP) system, which identifies casualties that are of a severity to prevent the ship from performing the mission.  These messages are passed from the ship's Captain to MSC, the Ship Manager, and MARAD.  The reliability of activated RRF ships is measured as the percent of days that RRF ships are mission-capable while under DOD control.  Mission-capability is determined, in part, by the number of days it takes to repair a ship that has become inoperative. For example, the low percent of mission capability in 1997 (95.2) was the result of one ship being out of service for 156 days while undergoing repairs.

Limitations:

None.

Statistical Issues:

Since the population of vessels covered by these measures often consists of a very small number of vessels (as low as 13 vessels in FY 2000), a large swing in results can occur from just one ship not being available on time or one ship having operational problems.

Verification & Validation:

The source of the activation data is the actual activation order from DOD to MARAD and the documents produced during a no-notice activation.  These fix the actual time of call-up and the time when the vessel is "Ready for Sea" (or tendered to MSC).  The Ready for Sea time is agreed to by MARAD and the on-board MSC representative and reported to DOD by official message.  The time taken to activate each ship is maintained in the ship’s logbook and in official DOD, MSC, and MARAD records.

The collection of data regarding mission capability under MSC operational control starts when MSC officially accepts delivery of RRF ships with date and time documentation.  The Captain of the ship reports all problems that are of a severity to prevent the ship from performing its mission to MSC, the Ship Manager, and MARAD.  The Captain also reports when the problem has been corrected.  This information is entered by MSC into its CASREP system.

 

Mariner availability                                                                      Page 49

Measure:

Of the mariners needed to crew combined sealift and commercial fleets during national emergencies, the percent of the total that are available. (FY) (2001)

Scope:

The pool of licensed and unlicensed mariners consists of mariners that have had sea service on U.S.-flag oceangoing merchant vessels over 1,000 gross tons within five years.  The mariner pool includes licensed and unlicensed actively sailing mariners and inactive mariners, employed shoreside, who have the necessary skills and retain the appropriate license/rating to operate sealift ships, defined by shipboard position and U.S. Coast Guard certification.  This pool is then compared to the DOD and commercial manpower requirements to determine sufficiency of the labor force.  Only oceangoing merchant vessels over 1,000 gross tons are considered because mariners on these vessels have skills required for emergency sealift operations.  The targets are based on a sealift operation that extends beyond 6 months, necessitating relief for the mariners who were sailing at the start-up of the operation.

Source:

U.S. Coast Guard Merchant Mariner Licensing and Documentation (MMLD) system.  The Coast Guard is the lead Federal agency for regulating, licensing, and documenting professional merchant mariners.  MMLD provides information on both actively sailing mariners and inactive mariners, including their skill level and seafaring employment.

Lloyd’s Maritime Information Systems.  MARAD obtains information to track the use of U.S.-flag commercial ships active in international trade and projects the size of the active, ocean-going, commercial fleet.  The size of this fleet has a direct correlation to the size of the commercial pool of mariners, based upon commercial crewing rules.

MARAD/DOT Mariner Survey. New for FY 2001, a random sample of mariners with current qualifications is now being surveyed, in conjunction with the Bureau of Transportation Statistics.  The Survey will provide a more accurate determination of the number of currently qualified mariners as well as information on mariner availability for sealift employment during national defense emergencies.

Limitations:

The size of the active and inactive mariner pool can be estimated from the MMLD.  MARAD integrates these data into its own system for analysis and reporting.  Because the MMLD does not contain all of the information on individual mariners contained in their paper records, and provides no information on the availability and willingness of individuals to accept a sealift position in an emergency, it does not provide sufficient assurance of mariner availability.

Statistical Issues:

The primary area of uncertainty lies within the MMLD system, which MARAD uses to define the population of available mariners.  While the accuracy of the data continues to improve as all licenses and documents are now on a five-year renewal cycle, gaps still exist in the database. Because the MMLD system was not designed to contact mariners, address and telephone information in the system is incomplete and out-of-date.  Also, operators of some large oceangoing vessels are not required to report mariner employment to the USCG, and evidence of sea service provided by individual mariners to fulfill requirements for upgrading their rating is not entered in the MMLD.

Verification & Validation:

The MMLD system is currently the only source of electronic data on mariner qualifications and employment.  MARAD continues to work with the USCG to improve the MMLD system.  The new MARAD/DOT Mariner Survey data will be used to estimate the number of qualified mariners available and willing to support sealift crewing positions.  Because this determination is a matter of individual choice and is subject to change, MARAD intends to develop a plan for maintaining current information on mariner availability based on the results of the Survey.

Comment:

None.

 

Drug interdiction                                                                          Page 50

Measure:

Amount of drugs seized or destroyed at sea (metric tons). (FY)

Scope:

Total amount of drugs (cocaine, marijuana, hashish, heroin, etc.) seized, jettisoned, or destroyed at sea by the United States Coast Guard.  Cocaine currently constitutes the largest drug threat to the U.S., but the Coast Guard seeks to interdict all illegal narcotics moving by non-commercial maritime conveyances. 

Source:

The amount of drugs seized is measured by Coast Guard crews and reported through the Coast Guard Law Enforcement Drug Interdiction Data Base.  Seizures are officially credited to the Coast Guard via Federal Drug Identification Numbers (FDINs) and are recorded in the federal Consolidated Counter-Drug Data Base, which is administered by the U.S. Interdiction Coordinator (USIC).

Limitations:

It is possible that non-entry, duplication, and coding errors are present in seizure amount data; however, the chance of this error is small. 

Statistical Issues:

None.

Verification & Validation:

Verification and validation occurs in several places in the data reporting and collection process.  Data entry software helps ensure data quality and consistency by employing selection lists and logic checks.  Internal analysis and review of published data by external parties help identify errors.  CG data is further reviewed at a quarterly Consolidated Counter-Drug Data Base Conference, where all agencies that input data into the database review all agency data for consistency and accuracy. 

Comment:

This measure aligns with the goals contained in the President’s National Drug Control Strategy. 

Measure:

Seizure rate for cocaine that is shipped through the transit zone. (FY) (2001)

Scope:

Seizure rate is a measure consisting of the amount of cocaine seized by the Coast Guard divided by the noncommercial maritime cocaine flow, expressed as a percentage.  Noncommercial is defined as any vessel or aircraft not engaged in port-to-port transfer of cargo/passengers for the primary purpose of business profit. Examples are pleasure craft, fishing vessels, offshore work-boats, or freighters carrying cargo as a means of disguising illegal drugs.

Source:

The amount of cocaine flow shipped by non-commercial means through the transit zone is estimated in the Interagency Assessment of Cocaine Movement (IACM) published by the Office of National Drug Control Policy (ONDCP).  The amount of cocaine seized is measured by Coast Guard crews and reported through the Coast Guard Law Enforcement Information System.

Limitations:

It is probable that non-entry, duplication, and coding errors are present in seizure amount data (numerator); however, this error is likely to be small.  The cocaine flow amount (denominator) is estimated through a complex process using many different sources of information. Due to the secretive nature of the illegal drug trade, cocaine flow estimates may contain significant errors.  The size of this error may vary from year to year; the extent of this is not known. The estimation process changes slightly each year as improvements are made, so year-to-year comparisons of the flow are not completely consistent. The accuracy of the official cocaine flow estimate has been questioned by some individuals and organizations outside of government that have an interest in U.S. drug policy.  ONDCP continuously attempts to refine this estimate to improve the measurement of interdiction activities.  This measure only addresses cocaine; formal flow assessments do not exist for other major drugs.  This measure is not designed to include cocaine shipped by commercial means such as large container vessels; however, it is probable that a small amount of cocaine included in the numerator is actually related to commercial shipping.  This distinction between commercial and noncommercial is better for program management; at-sea interdiction of commercially conveyed cocaine, particularly when shipped in containers, is extremely difficult, and not the focus of the Coast Guard program.   

Statistical Issues:

The primary source of uncertainty in estimating seizure rate for cocaine is the denominator.  Although the numerator estimate of cocaine seized is relatively accurate, the estimate of the amount shipped in the denominator is far more variable. The regression standard error for year-to-year chance variation in the cocaine seizure rate is 4.0 percent, based on data from 1995 through 2000.   

Verification & Validation:

Verification and validation occurs in several places in the data reporting and collection process.  Data entry software helps ensure data quality and consistency by employing selection lists and logic checks.  Internal analysis and review of published data by external parties help identify errors.

Comment:

This measure is consistent with the goals contained in the President’s National Drug Control Strategy.

 

Migrant interdiction                                                                      Page 50

Measure:

Interdict and/or deter at least 87 percent of undocumented migrants who consider attempting to enter the U. S. via maritime routes. (FY)

Scope:

Measure includes Cuban, Dominican, Haitian, and Chinese migrants, as these are the primary groups using maritime channels and the groups for which formal flow estimates exist.  Success rate is the estimated number arriving by maritime channels divided by those that pose a threat of migration (estimated intent).  The interdiction rate is just 1 minus the success rate.

Source:

Data is obtained from Coast Guard and from the Immigration and Naturalization Service (INS).  Estimates of migrants who successfully arrive and estimates of those with a high potential for undertaking the voyage are derived (with a consistent methodology) from investigations of incidents, interviews of detainees, and intelligence gathering.  Sources for this information are the Coast Guard, INS, and other authorities.

Limitations:

The numbers of illegal migrants entering the U.S., and the numbers of potential migrants, are derived numbers subject to estimating error.  Because of the speculative nature of the information used, and the secretive nature of illegal migration, particularly where professional smuggling organizations are involved, the estimated potential flow of migrants may contain significant error.  The measure only tracks four migrant groups at this time. A small number of migrants (approximately 10%) from various source countries are not included because formal flow estimates of migrants leaving these countries are not available.  Using the number of potential migrants in the denominator helps address the deterrence value of Coast Guard operations, but could lead to confusion of this measure with a simple interdiction rate. Trend information prior to 1995 is not available.

Statistical Issues:

The primary source of uncertainty in estimating the success rate for undocumented migrants is the denominator, which is an estimate of the flow of migrants, both documented and undocumented.

Verification & Validation:

The numbers of migrants reaching the U.S. via maritime routes and the number of “potential migrants” are estimated.  Methodologies and data are continuously reviewed.  The Coast Guard has developed the estimation techniques that support this indicator over the last six years in order to more consistently use intelligence information.  They are seeking independent assessment of the methods, and look to improve the process in the future.

Comment:

Partly because maritime threats of illegal migration have come from a limited number of sources, the Coast Guard and others have developed quantified threat estimates to better manage interdiction.  Over the past six years, estimation techniques have been improved to remove as much subjectivity and inconsistency as possible. It should be noted that past information reflects the success of intentional illegal activity.  While some DOT measures allow accurate projection of likely future outcomes, the highly variable nature of illegal migrant activity limits the ability to project future outcomes based on performance in the immediate past.

 

Critical transportation infrastructure protection                         Page 53

Measure:

Of those who need to act, percent who receive threat information within 24 hours. (FY) (2001)

Scope:

Threat information, in this context, is defined as credible information (both time-sensitive/action-oriented and informational) received by the Intelligence Community, analyzed by OIS and distributed in the form of a Transportation Security Information Report, generated by OIS for distribution by the Operating Administrations (OAs).  Figure is derived from the percentage of transportation security officials and industry representatives that receive threat information from OIS through the OAs within the 24-hour period.  Security representatives and officials will be randomly sampled by OIS within 48 hours of information dissemination and asked if and how soon they received the subject material. 

Source:

Internally prepared.  Survey conducted by OIS of both DOT personnel and industry security contacts.

Limitations:

Data:  Relies on the reporting of the customers and consumers of this information.  Reporting could be skewed to reflect positively on the dissemination process within the Operating Administrations.

Indicator:  This measure only identifies whether there are possible breakdowns and bottlenecks in the dissemination process.  It does not identify where those breakdowns specifically may be in the dissemination chain.

Statistical Issues:

Since these data are collected through a sample survey, they are subject to sampling and non-sampling errors.

Verification & Validation:

Customers will be randomly surveyed at all levels within the dissemination process, not solely the end users.  Consequently, the reporting of dissemination times and officials who are in receipt of the information can be crosschecked for verification and validity of data. 

Comment:

None.

 

 

 

 

Energy Efficiency                                                                          Page 55

Measure:

Transportation-related petroleum consumption (in quadrillion BTUs) per trillion dollars of Real Gross Domestic Product (GDP). (CY) (2001)

Scope:

Measure includes primary consumption of petroleum for transportation, in quadrillion BTUs.   This does not account for petroleum-produced electricity that is used in transportation; however, this is less than 1% of petroleum use.   Petroleum use is normalized to real GDP, in constant 1996 dollars.

Source:

U.S. Department of Energy Annual Energy Review 1999 and Annual Energy Review 2000.

Limitations:

Energy consumption does not include petroleum-produced transportation electricity.  Measure does not capture the fraction of this petroleum use that is imported, nor does it capture actual energy efficiency  (BTUs per passenger-mile-traveled).

Statistical Issues:

These data are external to DOT.  They are subject to both sampling and nonsampling errors.  Based on 1994-2000 data, chance variation from year to year in the transportation energy efficiency measure has a regression standard error of 0.016.

Verification & Validation:

Data is taken from external sources, which conduct their own verification and validation.

Comment:

Petroleum use is normalized to the nation’s real GDP in order to capture the nation’s economic exposure to petroleum use in transportation.  Beginning in 1999, the GDP baseline was changed from constant 1992 dollars to 1996 dollars.

 

 

Details on DOT Measures of Mobility and Economic Growth

Highway infrastructure condition                                                              Page 60

Measure:

Percentage of travel on the National Highway System (NHS) meeting pavement performance standards for acceptable ride. (CY)

Scope:

Data include vehicle miles traveled on the HPMS reported NHS sections and pavement ride quality data reported using the International Roughness Index (IRI).

IRI is a quantitative measure of the accumulated response of a "quarter-car" vehicle suspension experienced while traveling over a pavement.

Vehicle Miles of Travel (VMT) represent the total number of vehicle miles traveled by motor vehicles on public roadways within the 50 states and Washington, D.C.

Source:

Data collected by the State Highway Agencies and reported to FHWA for the Highway Performance Monitoring System (HPMS).  They are obtained from calibrated measurement devices that meet industry set standards.  Measurement procedures are included in the HPMS Field Manual.

VMT is a calculated product of the annual average daily traffic (AADT) and the centerline length of the section for which the AADT is reported. In the HPMS, travel is accumulated for each universe section to develop appropriate totals for the higher functional systems. AADT is required for each section of Interstate, NHS, and other principal arterial; as a result, travel is computed for these functional systems on a 100-percent basis. For minor arterial, rural major collector and urban collector systems, travel is calculated from samples using the AADT, centerline length reported for each sample section and the HPMS sample expansion factor for each section. Travel for the NHS on all functional systems is computed from the universe AADT data.

For the most part, travel for the rural minor collector and rural/urban local functional systems is calculated by the States using their own procedures and is provided in HPMS on a summary basis. Some States use supplemental traffic counts outside of the HPMS procedures; others employ estimating techniques, such as fuel use, to determine travel on these systems. In general, these methods are used in both rural and urban areas, including the donut areas of nonattainment areas to meet Clean Air Act requirements.

Limitations:

IRI data for the approved NHS exist from 1995 onward. Past data (1993 and 1994) contain some variation as this data was on the proposed, rather than the existing NHS. No NHS IRI data are available prior to 1993.  The HPMS requires States to report IRI data every two years; however, following the requirements is not mandated, but voluntary.

VMT estimates reported via the HPMS should be of reasonable quality particularly for the higher order functional systems. AADT and travel data are edited by the HPMS software for unusual values and for unusual changes to previously reported values. FHWA routinely works with State data providers to modify reported AADT values that do not appear to be reasonable before final use. Although AADT is required to be updated annually in HPMS, counts are only required to be updated on a 3-year cycle. For any reporting year, AADT for uncounted sections is usually derived by factoring the latest year's count for those sections.

Statistical Issues:

The major source of error in the percentages is probably the sampling error from selecting the segments of highway tested for smoothness. 

VMT data are subject to sampling errors, whose magnitude depends on how well the locations of the continuous counting locations represent nationwide traffic rates.  HPMS is also subject to estimating differences in the states, even though FHWA works to minimize such differences and differing projections on growth, population, and economic conditions which impact driving behavior.

Verification & Validation:

FHWA validates the data based on consistency reviews. States that follow the HPMS sampling instructions in developing traffic counting programs (Appendix F in the HPMS Field Manual) and the practices advocated in the Traffic Monitoring Guide have adequate counting and classification tools to prepare quality AADT and travel estimates for HPMS. The consistency of the sampling and counting procedures should also provide comparable State-to-State traffic data.

Comment:

None.

Measure:

Percentage of miles on the National Highway System (NHS) that meet pavement performance standards for acceptable ride. (CY) (2001)

Scope:

International Roughness Index (IRI) is compiled annually for every section of the NHS, using data reported from the States.

Source:

Data collected by the State Highway Agencies and reported to FHWA for the Highway Performance Monitoring System (HPMS).  They are obtained from calibrated measurement devices that meet industry set standards.  Measurement procedures are included in the HPMS Field Manual.

Limitations:

IRI data for the approved NHS exist from 1995 onward. Past data (1993 and 1994) contain some variation as this data was on the proposed, rather than the existing NHS. No NHS IRI data are available prior to 1993.  The HPMS requires States to report IRI data every two years; however, following the requirements is not mandated, but voluntary.

Statistical Issues:

The major source of error in the percentages is probably the sampling error from selecting the segments of highway tested for smoothness.  The annual variation in the percentage due to chance has a regression standard error of approximately 0.44 percent based on data from 1995-2000.

Verification & Validation:

FHWA validates the data based on consistency reviews

Comment:

None. 

 

Highway bridge condition                                                             Page 61

Measure:

Percentage of deficient bridges on the NHS. (CY) (2001)

Scope:

Measure includes the number of deficient (structurally deficient and functionally obsolete) bridges on the NHS functional system divided by the total number of NHS bridges in the inventory, expressed as a percent.  

Source:

Bridge information is collected by State DOTs and other bridge owners and provided to FHWA annually for inclusion in the FHWA maintained National Bridge Inventory (NBI).     

Limitations:

NBI includes information on all 114,567 NHS bridges.  States are required to update the system annually, but many States update quarterly.  The system contains 95 data items for each of the bridges, and 20 of these items relate to bridge condition and appraisal.  There are specific instructions as to how to assess bridges based on these items, including a grading scale from 0 to 9 with specific definitions and specific criteria to follow

Statistical Issues:

Even with the item specific grading system, differences in the grading between individual inspectors and between inspection days are probably the largest component of variation in the percentages.  Based on 1994-2000 data, the estimated regression standard error for year-to-year variation in the percentages due to chance is approximately 0.65 percent.

Verification & Validation:

DOT evaluates accuracy and reliability of the submitted NBI information through data checks and field reviews by both Headquarter and field office personnel.   This is done as a part of FHWA’s NBI, the National Bridge Inventory System (NBIS), and Highway Bridge Replacement and Rehabilitation Program.  Evaluation of the State’s compliance with the NBIS most often includes a sample of bridge inspection reports and a comparison of condition data with field visits to the bridge site.  In addition, there is an edit update program that identifies potential data errors in the NBIS.

Comment:

None.

 

Appalachian highway system                                                       Page 61

Measure:

Miles of the Appalachian Development Highway System (ADHS) completed. (FY) (2001)

Scope:

Measure includes actual miles completed on the 3,025 mile ADHS, within 13 member States.

Source:

States submit annual status updates on ADHS miles completed within their State each fiscal year to the Appalachian Regional Commission (ARC).  The ARC compiles the data.

Limitations:

This is an output measure.

Statistical Issues:

None.

Verification & Validation:

Completed by ARC.

Comment:

ARC estimates that the TEA-21 funding level will result in completion of approximately 37 additional miles each FY 1999 through 2003.

 

 

 

Highway congestion                                                             Page 60 & 61

Measure:

1.  Of total annual urban-area travel, percentage that occurs in congested conditions (CY)

2. Of annual urban-area peak period travel time, additional percentage of travel time attributable to congestion, (CY) (2001) and

3.  For the individual traveler in urban areas, average annual hours of extra travel time due to delays. (CY) (2001)

Scope:

Scope: Data for the three measures below stemmed from approximately 400 urban areas.  The data reflects the travel conditions of the freeway and principal arterial street networks.

Definitions:

1.       Urban area:  Developed area with a density of greater than 1000 persons per square mile.

2.       Congested travel:  Traveling below the posted speed limit(s).

3.       Peak Periods:  (Applicable to Travel Time Measure only.  Congested Travel and Traveler Delay represent daily travel.)  Monday-Friday morning and evening rush hours when slow speeds (below posted speed limits) are more likely to occur.  The length of peak periods varies, e.g., large urban areas are typically longer.  The Travel Time Measure accounts for the variations.

4.       Delay:  Extra travel time due to traffic volume and /or incidents.

Source:

Data collected and provided by the State Departments of Transportation from existing State or local government databases, including those of Metropolitan Planning Organizations.  The Federal Highways Administration’s Highway Performance Monitoring System serves as the repository of the data.  The Texas Transportation Institute utilizes HPMS data to derive the above measures.  

Limitations:

We have gathered data up through 2000.  We anticipate having 2001 data on/about Nov. 2002.  The proportion of congested travel figures used in calculating the measures are computed rather than measured values.  The computed values may understate congestion, as delay from incidents is not calculated.  Performance evaluation is process-oriented.  Transportation programs that help combat highway congestion possess outcome-oriented, objective methods within the specific program areas; however, the causal relationship between the programs and overall highway congestion is inconclusive.

Statistical Issues:

Methodology used to calculate performance measures has been developed by the Texas Transportation Institute and used in their annual Mobility Study.  A detailed description of TTI’s methodology is best described on their website at

http://mobility.tamu.edu/

.

Verification & Validation:

State-reported HPMS data are reviewed by FHWA for completeness, consistency, and adherence to reporting guidelines.  When necessary, and with close State cooperation, data may be adjusted to improve completeness, consistency, and uniformity.

Comment:

In the FY 2000 Performance Plan, we used hours of delay per 1,000 vehicle miles traveled (VMT) to measure this goal.  This metric attempted to provide a system-wide measure of congestion.  However, it represented only one dimension of congestion – delay -- and did not effectively reflect the actual performance of the highway system in places where congestion regularly happens, i.e., the measure showed delay decreasing nationwide when in fact congestion was worsening in urban areas.  Moreover, the measure was difficult to interpret by the general public.  Based on discussions with our partners and customers, we replaced this indicator with three new measures:  Congested Travel, Travel Time, and Traveler Delay.  Together, these new indicators will reflect changing travel conditions more comprehensively by focusing on three different aspects of inefficient road performance in a broad collection of urban areas across the nation where congestion regularly occurs.  The data supporting the three new measures stem from the Highway Performance Monitoring System (HPMS).  The availability of the data is approximately 9 months from the base year, e.g., 2001 actual numbers will not be available from HPMS until Sep/Oct 2002.  To accurately and reliably manage the transportation system, real-time (minute-by-minute) measurement of system speeds is needed and can only be achieved with automated instrumentation.  As the Intelligent Transportation System network is put in place, reliability will become a barometer of this strategic goal. Ten cities have been identified with sufficient instrumentation to permit the development of a reliability measure.  This is a first step in migrating from HPMS data to real-time, ITS-based data.

 

Intelligent Transportation Systems integrated deployment       Page 61

Measure:

Number of metropolitan areas where integrated ITS infrastructure is deployed. (FY) (2001)

Scope:

The level of integrated deployment in 75 of the nation’s largest metropolitan areas has been established using a set of indicators that consider two factors: (1) How much ITS infrastructure is in place at each metropolitan area; and, (2) How much integration is going on at each area.  The process for determining the level of “component” deployment in a metropolitan area employs a set of indicators that measure the magnitude of deployment for selected ITS components.  These are typically expressed as a ratio of actual deployment divided by the total possible, for example the number of freeway miles under electronic surveillance divided by the total freeway mileage.  Components are considered deployed once the level of deployment attains a specified threshold level based on the indicators.  Integration is defined as the sharing of data between agencies associated with the different jurisdictions responsible for ITS infrastructure.  Typically there are three: State DOTs responsible for management of freeways and incident management programs; city governments, which manage most of the traffic signal systems; and public transit authorities, which manage most bus and rail services.  The level of integration is determined by the extent that these three major transportation organizations employ technology to share and use transportation data to increase system capacity.  Two examples of integration are: 1) a city traffic signal system receiving data from the state freeway management center about the queues at freeway ramp meters and then adjusting the signal timings on the arterial streets, or 2) a transit agency providing the state freeway management center with the real-time location of the buses so that freeway speeds can be determined.  Metropolitan areas are rated as low, medium, or high separately for deployment and integration and then assigned an overall combined rating.  An overall score of medium or high meets the goal for a metropolitan area.

Source:

Metropolitan ITS Deployment Tracking Database developed by the Oak Ridge National Laboratory for the ITS Joint Program Office.  Data are collected by means of surveys from designated metropolitan areas.

Limitations:

This indicator is designed to track and encourage basic steps toward component deployment and systems integration.  However, it does not reflect the full breadth of deployment or integration activities.  For example, while it establishes the existence of basic integration of essential components, it does not confirm that all possible or desirable integration links exist in a metropolitan area.  Similarly, the attainment of a deployment threshold only confirms a substantial commitment to the use of ITS technology but does not indicate that all needed deployment is complete. 

Statistical Issues:

These data come from sample surveys that, like all sample surveys, contain sampling and non-sampling errors.

Verification & Validation:

The DOT Joint Program Office reviews deployment tracking indicators and methodology.  Results are distributed to DOT headquarters and field staff as well as to state and local survey responders for confirmation of accuracy and completeness before the final reports are issued.  Independent experts in statistics and transportation review procedures for survey construction and data collection prior to each survey iteration.  A steering committee of Federal, state, and local transportation officials review and approve changes to methodology and indicators prior to implementation.

Comment:

The FY 1997 baseline is 36 areas.

 

 

 

Transit ridership                                                                  Page 66 & 67

Measure:

1.  Billion transit passenger miles traveled. (CY) (2001)

2.  Average percent change in transit passenger-miles traveled per transit market. (FY)

Scope:

Includes revenue-passenger miles on publicly sponsored bus, transit rail, commuter rail, ferry, and vanpools in urbanized areas.

Source:

National Transit Database (NTD), with information gathered from transit operators.

Limitations:

Data is self-reported by transit agencies using an FTA-approved sampling methodology.  Although most data is reported in the National Transit Database each year, sample cycles may be annual, every three years, or every five years depending on the size of the urban area and the number of vehicles operated.  Ridership is an outcome indicator that reflects a variety of factors, including the capital investment of the Federal Government.  Ridership is also influenced by operational decisions of transit authorities, and the availability and cost of alternative modes of transportation.

Statistical Issues:

The sources of uncertainty include sampling error, annual chance variation, and auditing issues.  The regression standard error from 1994-2000 indicates that the magnitude of the combination of the first two sources of error is approximately 0.67.

Verification & Validation:

An independent auditor and the transit agency’s CEO certify that data reported to the NTD are accurate.  FTA also compares data to key indicators such as vehicle revenue miles, number of buses in service during peak periods, etc.

Comment:

None.

Measure:

Percentage of transit grants obligated within 60 days after submission of a completed application.

Scope:

FTA grants obligated during a fiscal year period for major programs: Urbanized area, non-Urbanized area, and Elderly and Persons with Disabilities formula grants; Capital grants; Job Access and Reverse Commute grants; Over-The-Road Bus grants; and Planning grants.

Source:

FTA TEAM database.

Limitations:

Several factors that contribute to grant delays are beyond FTA’s ability to control. These factors include the processing of flexible funds from FHWA through the Treasury, and the Congressional grant release process.

Statistical Issues:

Processing time is calculated from submission date to obligation date. $0 dollar non-funding grant amendments are excluded from analysis.

Verification & Validation:

TEAM output file is crosschecked against other system generated files for consistency; inconsistencies are investigated and corrected prior to reporting. Grants with missing or out-of-sequence dates are excluded for calculating averages.

Comment:

An FTA task force meeting was held in February 2002 to identify causes for grant processing delays. The resulting action plan is now being circulated for final review and approval. Implementation of the plan will follow.

 

 

 

Aviation Delay                                                                              Page 68

Measure:

Percentage of on-time flights. (FY)

Scope:

The time of arrival of completed, scheduled passenger flights to and from the 32 DOT large-hub airports is compared to their scheduled time of arrival.  The sum of flights arriving on or before 15 minutes of scheduled arrival time is divided by the total number of completed flights.

Source:

The Aviation System Performance Metrics (ASPM) database, maintained by the FAA’s Office of Aviation Policy and Plans.  By agreement with the FAA, ASPM flight data is filed by certain major air carriers for all flights to and from 21 large and medium hubs, and is supplemented by flight records contained in the Enhanced Traffic Management System (ETMS) and flight movement times provided by Aeronautical Radio, Inc. (AIRINC).  Data are sufficient to complete ASPM data files for 49 airports.

Statistical Issues:

There is little major error in the count of completed flights or the count of flights that arrive on-time.

Limitations:

Some ASPM data is constructed from ETMS records, a small portion of which may not be maintained in FAA traffic control computers when they are under heavy use.  

Verification & Validation:

Flight data is extracted from the Official Airline Guide (OAG) and compared to& data from carrier records, which contains carrier computer reservation flight schedule data.  Summary data is compared to data filed monthly with DOT under 14 CFR Part 234, Airline Service Quality Performance Reports, which separately requires reporting by major air carriers on flights to and from the 32 large hubs.

Comment:

FAA’s percentage of flights arriving on-time derived from ASPM data differs only by fractions of a percent from the on-time percentage derived from DOT’s slightly different database. 

Measure:

Aviation delays per 100,000 activities. (FY) (2001)

Scope:

An FAA reported delay occurs when an aircraft is delayed fifteen minutes or more because of constraints that prevent the aircraft from making a scheduled landing.  Delays are counted in five categories: FAA equipment, volume, weather, runway related, and other.  Delays due to airline equipment are not considered. “Activities” are total facility activities, as defined in Aviation System Indicators 1998 Annual Report.  Total facility activities are the sum of en route and terminal facility activities.

Source:

FAA air traffic facilities report the data to headquarters, which incorporates the data into the Air Traffic Operations Management System.

Statistical Issues:

There is no major error in either the delay counts (numerator) or the flight operations data (denominator) for this rate.  However, random variation in aviation delays results in a significant variation in the delay rate from year to year. The regression standard error in the delay rate, based on 1994-2000 data, is approximately 20.0.

Limitations:

By collecting information on delays of fifteen minutes or more, FAA does not capture the aggregate amount of system delay, but only the most significant delays.

Verification & Validation:

Data is analyzed and checked by an Air Traffic Service headquarters office on a daily basis to ensure accuracy of the information reported.

Comment:

Total delays in all five categories are what the traveling public experience.

 

 

Airport and en route efficiency improvements                            Page 69

Measure:

Cumulative increase in throughput during peak periods at certain major airports. (FY) (2001)

Scope:

This measure focuses on the arrival rates during peak traffic periods comparing pre-CTAS rates to post CTAS rates.

Source:

Radar system (HOST and ARTS) data is collected and aircraft flight tracks are obtained from those systems and analyzed to determine arrival and departure times. 

Limitations:

The radar systems produce very large data files requiring a substantial effort to extract relevant data for analysis. The extracted data sets need to be of sufficient size to produce statistically significant results.

Statistical Issues:

Conditions (weather, runways in use, aircraft mix) vary, affecting rates. Data must be normalized and data sets must be of sufficient size to produce valid results.

Verification & Validation:

Methodologies and detailed results are available for review in semi-annual FFP1 Metrics Updates (December and June).  Results are coordinated with FAA and User stakeholders.

Comment:

None.

Measure:

Cumulative increase in direct routings for en route flight phase. (FY) (2001)

Scope:

This measure focuses on the number of direct routings provided by en route controllers comparing pre and post-URET installation.

Source:

URET provides data on routing amendments, which is then analyzed to determine the number of direct amendments. 

Limitations:

The radar systems produce very large data files requiring a substantial effort to extract relevant data for analysis. The extracted data sets need to be of sufficient size to produce statistically significant results.

Statistical Issues:

Extreme weather conditions, particularly thunderstorms, will significantly affect routing amendments; therefore, data is sampled for days when weather is not a factor.

Verification & Validation:

Methodologies and detailed results are available for review in semi-annual FFP1 Metrics Updates (December and June).  Results are coordinated with FAA and User stakeholders.

Comment:

None.

 

Runway pavement condition                                                        Page 70

Measure:

Percent of runways in good or fair condition (commercial service, reliever, and selected general aviation airports). (CY) (2001)

Scope:

Paved runways at the 3,300+ airports in FAA’s National Plan of Integrated Airport Systems (NPIAS) are assessed for pavement condition.  The NPIAS airports include all commercial service and reliever airports and those general aviation airports that are significant to national air transportation.

Source:

The FAA’s Airport Safety Data Program (ASDP) provides extensive data about the facilities that are available at public-use airports.  Data are provided approximately annually by FAA inspectors for airports certified under FAR 139.  Data for other airports, including most public use general aviation airports, are provided under an FAA contract.

Limitations:

FAA contracts for a visual survey of the runways to categorize their condition based on criteria developed by the FAA Office of Airports.  “Good” condition means all cracks and joints are sealed; “fair” condition means there is mild surface cracking, unsealed joints, and slab edge spalling; and “poor” condition means there are large open cracks, surface and edge spalling, and vegetation growing through cracks and joints.  Since the reports are based on a visual inspection, underlying drainage or strength problems are not reported.  However, these problems normally create surface defects that are visible.  The more detailed pavement condition index (PCI) inspections require a section-by-section examination of the runway rather than an overall assessment used for this performance measure. FAA has been aggregating the ADSP data from all NPIAS airports only every several years for inc