Performance measurement is dependent on the availability of useful data. Useful data will indicate level of performance and progress toward organizational goals. All data are imperfect in some fashion. Pursuing “perfect” data, however, may consume public resources without creating appreciable value. For this reason, there must be an approach that provides sufficient accuracy and timeliness but at a reasonable cost. This section of the Performance Plan/Performance Report provides information on how DOT reports on performance, verifies and validates data, assesses limitations of the data, and plans for improving DOT’s data.

Performance Data Completeness and Reliability

In an attempt to bring consistency and quality to its performance reporting, DOT has implemented some general rules regarding the data it uses and how it is evaluated.

Annual data – Whenever available, the data in this document are reported on a Federal Government fiscal year basis. However, there are instances where this is not possible so calendar year data are used instead. This often occurs when data are collected and reported to DOT by external sources and a calendar year reporting requirement is specified in the implementing regulation. The reporting timeframe (FY or CY) for each measure is included in the Data Details in Appendix I.

Annual results – If available, the results for the most recent year in the Report are listed as “Actual” in the Performance Goals & Results box for each performance measure. However, given the March deadline for submission of the Performance Report, quite often data have not been compiled and finalized for the entire year. When this occurs and an actual value is not available for the current year, either an estimate or projection is provided instead. In general, estimates are based on partial year data that are extrapolated to cover a full 12-month period. For example, if six months of data are available, they will be compared to prior years for the same six-month period to determine any variation from past levels. Historical trend information, supplemented by program expertise, will then be applied to estimate the remaining six months of performance. The result will be identified as a “preliminary estimate” in the Report. If partial year data are not available, then past trend information will be analyzed and supplemented by program knowledge to develop a projected value for the annual performance measure. The result will be identified as a “projection” in the Report. As data are finalized, the projections and preliminary estimates will be replaced by actual results. Results may be amended as errors and omissions are identified in the data verification process, because updated information is provided by the reporting sources, or because of legal or other action that changes a previously reported value. For example, updated pipeline spill reports may change the status of a previously reported value used in performance measurement.

In measuring progress toward the majority of performance goals, DOT is moving to a system of monthly performance measurements. This will make it much easier to internally gauge periodic progress toward goals as the year progresses, and will enable more timely performance reporting after the years’ end.

Completeness of Data – As described above, actual data and “preliminary estimates” incorporate complete or partial data from 2001. Results listed as “projections” are not based on data from 2001, but on trend data from prior years.

Reliability of Measurement Data – Because performance results in a given year are influenced by multiple factors, some of which are beyond DOT’s control, and some of which are due to random chance, there may be considerable variation from year to year. (See discussion in Appendix I.) A better “picture” of performance may be gained by looking at results over time to determine if there is a trend. Therefore, graphs are provided for each measure showing trend lines back to 1990, or as many years as possible if data are not available back to 1990. Additionally, a table is included at the beginning of each strategic goal section giving the available data from 1995 through 2001 for measures with performance goals specified for 2001.

Verifying & Validating Performance Measures

Integral to performance measurement is a proper understanding of data limitations, cost-effectively addressing these limitations where necessary, and acknowledging those that remain when interpreting results. This section on verification and validation provides a DOT-wide overview of our plan for assessing the quality of the data DOT uses to measure its performance, and follows the GAO definitions for verification and validation:
“Verification is the assessment of data completeness, accuracy, consistency, timeliness, and related quality control practices.”

“Validation is the assessment of whether data are appropriate for the performance measure.”

Virtually all data have errors. In Appendix I we have provided the following information about the data used for each performance measure: source of the data, limitations of the data, observations about the quality of the data, work planned or ongoing to improve data quality, and any known biases.

Additionally, we have compiled Source and Accuracy Statements for each of the DOT data programs used in this report, which can be found at www.bts.gov/statpol/SAcompendium.html. The Source and Accuracy Statements give more detail on the methods used to collect the data, sources of variation and bias in the data, and methods used to verify and validate the data.

By validating data used in the DOT performance plan, we are ensuring that those data are reflective of the phenomena they purport to measure. The Office of the DOT Inspector General (OIG) plans to selectively verify and validate performance measurement data each year. When pertinent to the conduct of ongoing projects, OIG will also assess performance measures to determine their appropriateness for measuring progress toward stated goals. These assessments may lead to changes in the goals, improvements to or additions of data collection systems, or both.

Assessing and, where possible, eliminating sources of error in DOT data collection programs has always been an important task for data program managers. As a part of their ongoing work, managers of Departmental data programs use quality control techniques, such as flowcharting the data collection process, to identify where errors can be introduced into the data collection system. Program managers also use computerized edit checks and range checks to minimize errors that may be introduced into the data of their respective programs. In addition, quality measurement techniques are employed to measure the effects of unanticipated errors. These include verification of data collection and coding, as well as coverage, response and non-response error studies to measure the extent of human error affecting the data. As sources of error are identified, steps are initiated to improve the data collection process.

The data used in measuring performance come from a wide variety of sources. Much of the data originates from sources outside the Department and, therefore, outside the direct control of the Department. The data often come from administrative records or from sample surveys. While DOT may not have a strong voice in improving the quality of outside data, the Department takes all available information about the limitations and known biases in outside data into account when using the data.

The myriad data sources make the task of assessing and, where possible, eliminating error a challenging one for DOT. Different data systems contain different types of errors. For example, data from administrative records systems may have missing or incorrect records, and data from sample surveys will contain sampling error.

Several measures (particularly in safety) require aggregation across transportation modes. This can be particularly problematic because of the use of different definitions in different transportation modes. Also, data from outside the Department may have unknown error properties.

To help the operating administrations address these issues, the Bureau of Transportation Statistics (BTS) is developing a statistical policy framework where the operating administrations will work together to identify and implement the current statistical “best practices” in all aspects of their data collection programs. This project is consistent with the data capacity discussions found in the DOT Strategic Plan.

BTS's statistical staff is consulting with the DOT operating administrations’ data program managers to assist in data evaluation and validation, documenting data sources, and determining the reliability of performance measurement estimates.

Departmental data systems managers use these data verification methods:

• Comparisons with previous data from the same source.

• Comparisons with another reliable source of the same type of data within DOT for the same time period.

• Comparisons with another reliable source of the same type of data within DOT for a previous time period.

• Comparisons with another reliable source of the same type of data outside DOT for the same time period.

• Comparisons with another reliable source of the same type of data outside DOT for a previous time period.

In addition to computerized edit checks and clerical review procedures to look for outliers, duplicate records, and data inconsistencies, data managers also verify data quality at each step of the data collection process using these procedures:

• Re-collecting/re-interviewing all (or a sample of) records and reconciling with the original collection. (This applies to census or sample survey data collections from administrative records, organizations, or individuals.)

• Conducting 100 percent (or a sample of) data re-coding and reconciliation to assess and correct coding errors.

• Conducting 100 percent (or a sample of) data re-entry and reconciliation to assess and correct data entry errors.

The American Travel Survey’s re-interview program, in which a sample of households were re-contacted and differences reconciled, is an example of a verification system within a data collection program.

As part of its mandate in the Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA), the Transportation Equity Act for the 21st Century (TEA-21), and its plans for a statistical policy framework in the Department, DOT is working on a program of research, technical assistance, and data quality enhancement to support the continued improvement of data programs in DOT. This will help data program managers throughout DOT improve data quality and better document known data limitations.

Many of DOT’s internal data programs rely on State DOTs to collect reliable statistics within cost constraints. While we work closely with our State DOT partners, we do not have direct control over these data.

External Data Source Limitations – Timeliness is also a significant limitation for external or third-party data. Other limitations of external data are noted in the comments for each performance measure in Appendix I. In some cases, DOT has replaced external data, where little is known about the quality of the data, with internal data. For example, DOT has used estimates of person-miles traveled (PMT) from private organizations, absent any better estimate. The 1995 Nationwide Personal Transportation Survey and American Travel Survey give DOT data with known error properties that allow a better estimate of PMT.

National Security – Existing performance data sources are generally good, but DOT will collect data to better understand the transportation system’s vulnerability to intentional acts of disruption or destruction.

Mobility – All mobility outcomes present complex measurement issues. Accordingly, DOT will: 1) develop ways of measuring user transportation cost, time, and reliability with time-series data; 2) develop better approaches for measuring access; 3) develop straightforward measures of congestion and its costs; 4) produce more timely and comprehensive data on the condition and use of the transportation system; and 5) develop a more complete understanding of variables influencing travel behavior.

Economic Growth – DOT needs aggregate data for measuring the productivity, effectiveness and efficiency of the U.S. transportation system. We plan to collect, analyze and disseminate data and information that identify critical trends and issues relating to transportation’s nexus to the U.S. economy. DOT will: 1) develop a means of measuring transportation cost, time, and reliability – at an aggregate level – with time-series data; 2) develop a comprehensive measure of the transportation capital stock; 3) improve our view of changes in the transportation workforce; 4) develop better measures of productivity in the transportation sector, and other issues concerning use of Producer Price Indices; and 5) develop a better picture of transportation-related variables influencing U.S. competitiveness in the global economy.

Human and Natural Environment – DOT will: 1) develop comparable and complete data on transportation emissions, noise, hazardous materials releases, and wetlands impacts; 2) improve our understanding of collateral damage to the human natural environment; 3) create better leading indicators for potential environmental issues; and 4) develop a reliable method of measuring the environmental benefits of bicycling and walking.