The President’s Management Agenda (PMA) has at its core a Government that is citizen centered and results oriented. Each initiative (Human Capital, Competitive Sourcing, E-Gov, Financial Management, and Integrating Performance and Budget) has stated goals to achieve color-coded measurement scores. In keeping with the emphasis on results oriented Government, the Department of Transportation (DOT) procurement offices have and will continue to use performance measures to assist the Department’s business processes.

Purpose:

To provide the DOT procurement managers with an assessment system, using a balanced scorecard approach that offers them a snap shot perspective of the progress towards offering our customers world-class business solutions. We fully intend to evaluate our performance towards our established goals, but we feel strongly that this process must be constructive for our managers and our workforce. We intend to use our performance measures as a management tool to assist managers in determining what they have achieved, as well as areas where they may want to focus attention in the coming year. These measurements will gauge the performance of our fiduciary responsibility as well as our progress toward reaching our strategic goal of becoming a world-class acquisition system.

The data gathered is for individual DOT Operating Administrations (OAs) assessment purposes and comparisons between OAs will not be conducted without their permission. The only results that will be published are the cumulative DOT results and statistical data that is public information.

Responsibility for PPMS:

The DOT Procurement Management Council (PMC) has overall responsibility for developing and maintaining the PPMS and may charter groups or subgroups (e.g., Acquisition Performance Measurement Committee (APMC)) as they determine necessary.

Role of the Office of the Senior Procurement Executive’s (OSPE):

In partnership with the Procurement Management Council (PMC), the Senior Procurement Executive (SPE) will establish DOT-wide measurement goals. The OSPE will manage the overall process of the performance measures to include both the surveys and the metrics/statistics. OSPE will support OA Administrators during the survey period and will assist OA Administrators in the presentation of their measurements, should they request help. The SPE will hold annual meetings with each PMC member to present the DOT-wide measurement results as it relates to the OA and the Department as a whole. During these meetings, management strategies for improvement and Action Plans will be investigated. Externally, DOT will participate in the benchmarking of common Government-wide measurements and measurement system with other Departments.

PPMS Guiding Principles

The Balance Scorecard (BSC) measurement approach will be the approach used to assess our overall performance.
Performance measurements will be established from DOT-wide and Government-wide measurements in use. Every effort will be made to minimize unique data collection requests.
Annually, the OSPE will collect, analyze and compile data, and debrief each member of the PMC on their Operating Administration (OA) results. Meetings with OA field offices may be held upon request of the PMC member.
OA measurements resulting from the surveys may be used by the OA as a whole or by individual offices for their own unique needs. Data for these measures will be collected, distributed and analyzed by the applicable OA/office.
Publication, dissemination, or comparison of individual OA results will not be done without prior consent of the PMC member or their representative.
Waivers or requests for exceptions to this document are to be submitted through the OA’s Head of the Contracting Activity (HCA) to the Office of the Senior Procurement Executive (OSPE). The request is to contain sufficient detail to explain the basis of the request, procedures to be waived, and any recommended alternative action.

Background:

In 1993, an interagency group created the Procurement Measurement Action Team. Their task was to assess the state of the acquisition system, to identify a structured methodology to measure and improve acquisition performance, and to develop strategies for measuring the health of agency acquisition systems.

The Team found that organizations were using top-down management reviews to determine compliance with established process-oriented criteria and to certify the adequacy of the acquisition system. This method was found to lack a focus on the outcomes of the processes used and was largely ineffective in obtaining dramatic and sustained improvements in the quality of the operations, because of its orientation on process rather than results.

The Team did extensive research and made site visits to leaders in performance measurement/management to identify an assessment methodology appropriate for federal organizations. The Team chose a model developed by Drs. David Norton and Robert Kaplan—the BSC-because it is:

Focused on high impact measures
Intended to be easy and economical to use
Balanced with emphasis on prevention rather than detection
Customer-oriented and cross-functional
Empowered the procurement organization to make improvements, and
Designed to compare the quality of service

OSPE adopted the Balance Scorecard (BSC) approach as a structured methodology for using performance measurement results to gauge progress and to help reach our goal of creating a world-class acquisition system oriented to business solutions and facilitating mission accomplishment.

In 1995, we replaced our formal Procurement Management Reviews (PMRs) with a less costly and less resource-intensive performance measurement system based largely on surveys provided to relevant acquisition personnel. The Performance Measurement Assessment Tool (PMAT), as the survey tool is referred to and specific metric or statically measurements provided us with both a subjective and objective method to assess the state of our business solutions. This process has resulted in a tool that emphasizes to our acquisition managers’ the need for appropriate business solutions rather than the previous method of procurement reviews to detect non-compliance with process.

Numerous statutes and initiatives affected our choice of using the BSC to view performance. They include: the Government Performance and Results Act (GPRA) of 1993, the Federal Acquisition Streamlining Act of 1994, the Government Management Reform Act of 1994, the Federal Acquisition Reform Act of 1995, the Information Technology Management Reform Act of 1996 (the Clinger-Cohen Act) and the President’s Management Agenda. Each of these requirements, in some form, requires agencies to strategically plan how they will deliver high-quality supplies and services to their customers, and specifically measure program performance in meeting their strategic commitments.

In FY04 the APMC completed a full review of the PPMS process. This review resulted in major changes to the surveys for Employees, Customers and Metric Measures used in the BarBell Chart. The Procurement Management Council (PMC) approved these changes, which will reflect the concepts of business solutions and strategic planning. Two new measurements where added to the surveys of customers and employees, Technology Satisfaction and Strategic Value. Several new measurements were added to the metric portion of the PPMS. In FY04 there will be detailed tracking of the Purchase Card program and a start toward Commodity/Spend Analysis for the department. Measurements have also been added to reflect COTR Certification levels in the department. OSPE will continue to explore software packages that will allow for the total automation of the PPMS, currently only the survey portion is Web-based.

Development of a Strategic Path for Acquisition

DOT’s Strategic Path to the PPMS

DOT Strategic Plan

DOT’s Strategic Plan spans five Fiscal Years (FY 2003 to FY 2008). It is the one document that sets forth the overall direction, vision and mission of DOT. The Strategic Plan sets performance goals and identifies measures to gauge progress towards those goals. The acquisition portion of the Strategic Plan is under Organizational Excellence. The DOT Strategic Plan performance goals and measures include the Presidents Management Initiatives and Scorecard Method.

DOT Performance Plan

To accomplish the Strategic Plan, an annual Performance Plan is developed. This plan defines the measures, activities, and goals that, when taken together, indicate how well an agency is achieving its goals. The goals of the Performance Plan are achieved through the efforts of DOT personnel by tasks outlined in their Performance Agreement.

Budget Process

The budget defines the resources needed to accomplish the strategic goals. As Program Managers develop their budget requests, the involvement of contracting officials is vital to accomplishment of their goals.

Performance Agreement

A Performance Agreement is used to translate and flow down the goals and objectives of the Strategic Plan, Performance Plan and Budget. The achievement of these goals can be reflected in an individual’s annual Performance Agreement or by tying the individual’s achievements to the organization’s strategic goals.

Procurement Performance Measurement System (PPMS) Process Overview

In developing and refining the process by which we are looking at the health of our organization, we decided to look at both survey and metric (statistical) tools. We have defined five overarching perspectives, which will reflect our accomplishments towards meeting our goal of a world-class acquisition system. These are:

DOT’s Five Overarching Perspectives

1. Customer Perspective – A customer is a person who uses contracting services to obtain a product or service through a contractual agreement. Using a series of survey questions, we measure our customer’s satisfaction with specific Business Measures. Each OA may ask all or some of their customers to respond to the survey. OAs are encouraged to obtain an adequate representation for a true picture of their results.

2. Employee Perspective – A DOT employee is a person in the GS-1101, 1102, or 1105 series working in a contracting office. Using a series of survey questions, we measure our employee’s satisfaction with specific Business Measures. Each OA is encouraged to invite all contracting personnel to participate in the survey. Some OAs may invite employees who are not in the GS-1101, 1102, or 1105, but are involved in the contracting offices functions. Individuals serving under a contract to DOT to perform “contracting functions” are not considered DOT Employees.

3. Internal Business Perspective – Contains a set of statistical business measures, which reflect the Government and DOT’s achievement of business goals.

4. Learning and Growth Perspective – Contains informational business measures, which reflect the level of training and certification acquired by the acquisition workforce.

5. Financial Perspective - Contains a set of statistical business measurement, which reflects our effectiveness at employing business decisions.

In obtaining the results for these five perspectives, we turn to survey and statistical measurements. Two of our surveys, the customer and employee, specifically look at measures that we call, Business Measures, to determine customer and employee satisfaction. We focus on the following business measures as part of the Customer and Employee Perspectives:

Survey Business Measures

Timeliness – The degree of satisfaction with the delivery of products and services and other factors affecting the acquisition schedule.

Quality – The degree that Goods and Services meet or exceed the minimum requirements for performance and price/cost.

Service/Partnership – The degree of responsiveness of the acquisition team, the success of mechanisms that support teaming and the degree of satisfaction with communications, problem solving, innovative business solutions and facilitating mission accomplishment.

Work Environment – The employee’s satisfaction with items such as tools provided (information technology, reference material etc.), working conditions, alternate work schedules and reward or recognition process.

Leadership/Management – The measure of a professional environment that promotes the efficient and effective acquisition of goods and services from responsible contactors. This measure includes an assessment of the employee’s perception of organizational professionalism, culture, values and empowerment.

Strategic Value – A measurement of the impact of acquisition accomplishment on the DOT mission.

Technology Satisfaction – A measure of the uses and effectiveness of the various information technology tools, which are available to acquisition personnel to accomplish their tasks.

Customer Satisfaction – This is a weighted average of the degree of satisfaction with quality, timeliness, strategic value, service/partnership, and technology.

Employee Satisfaction – This is a weighted average of the degree of satisfaction with, timeliness, strategic value, leadership/management, work environment, service/partnership, and technology.

Quality Assurance Surveillance Program[1] - This is the way we will measure the business procedures and methods used to achieve business solutions for our customers.

Attachment A is a diagram of the PPMS process.

PPMS Goals

Goals for all PPMS measures (Survey and Metrics) will be established annually in Attachment B.

PPMS Timeline

Please see Attachment C for the timeline of events by fiscal year.

Survey Overview

The Survey or PMAT is the first step in DOT’s PPMS. It is a qualitative method for determining our progress in becoming a world-class acquisition system.

DOT uses five surveys--Customer, Employee, Chief of the Contracting Officer (COCO), Office of the Senior Procurement Executive (OSPE) Customer Survey and Quality Assurance Surveillance Program.

Survey	Description	Attachment
Employee	This survey is intended for persons in the 1101, 1102, 1105 GS series for procurement and contracting personnel. OA’s may choose to invite other contracting office staff members to take the survey. If results are sought from other than the procurement or contracting personnel, the survey results should be annotated to indicate what other types of personnel results are included. Individuals serving under a contract to DOT to perform “contracting functions” are not considered DOT Employees.	D
Customer	This survey is intended for individuals who are “customers” of the procurement or contracting office. OA’s choose to invite their customers to take the survey. The actual population may differ from OA to OA. It is expected that the customer population will involve Program Managers, Project Manager, Contracting Officer Technical Representatives (COTR), engineers, etc. Vendors or contractors are not considered as customers for the purpose of this survey. Some OA’s may selectively invite certain customers to take the survey. The same individuals do not need to take the survey every year. If results are sought from other than the identified group, the survey results should be annotated to indicate what other types of personnel results are included.	E
COCO	This survey is our “manager” survey. A COCO is defined in TAR Part 2 as “the individual(s) responsible for managing the contracting office(s) within an operating administration.” Currently, four OAs (FAA, FHWA, RSPA, and MARAD) have field offices and there may be more than one COCO Survey, to account for headquarters and field offices. This survey is meant to have the managers reflect on the past fiscal year’s results and what events, issues or problems may be reflected in their customer and employee surveys. Managers should comment on their Action Plans from the previous fiscal year.	F
OSPE Customer Survey	This survey is optional for employees in each Operating Administration and COCOs only. The survey’s results are used by the SPE to assess how well it is meeting the needs of the acquisition community. Priorities and program support are defined using the feedback from this survey. This survey could be considered as a ‘customer survey’ of the Office of the Senior Procurement Executive.	G
Quality Assurance Surveillance	TBD for FY05 Survey. (This was the “Effective Business Standards” Results.	H

The APMC reviews each survey before they are issued to determine if DOT’s "core questions" need changes or additions. In addition, the APMC member, in conjunction with their PMC member, may develop/select OA specific questions (Unique Questions) for the surveys. However, only “core” survey question results are tracked by DOT and made a part of the BarBell Chart.

The surveys have been structured for ease of use and minimal time for completion. The APMC members continually look for ways to increase the overall response rates by reducing the burden on the individuals responding to the surveys through a fully automated user-friendly Web-based survey process. The goal of the APMC in having a set of “core” questions for each survey is to ensure that only necessary questions are asked which will help to identify specific areas where the acquisition process could improve. To maintain data integrity, the responses are completely anonymous.

The survey is administered through a web-based application. The OSPE has developed a site that provides a unique entryway and location for the OA surveys. The survey is made available on the DOT/OA website annually, generally, during November so that results can be collected and analyzed to support the SPE’s annual reviews with the OA PMC representatives in January. OA Administrators can access the Survey Tool and tailor their OA survey by adding questions to the “core” questions in each survey. Guidance and directions are provided at http://pmat.dot.gov in three different types of guides:

PMAT² Administrator User Guide - A guide to help PMAT administrators use the application.

OA Administrator User Guide - A guide to help OA administrators use the application.

PMAT² Reports User Guide - A guide to help OA administrators use the reporting system

Metric Overview Measurements

The metric or statistical measurement is the second area in DOT’s PPMS. It is a quantifiable method for determining how we are progressing in meeting our business solutions goals. These measurements are shown in the Internal Business, Financial, Learning, and Growth Perspectives. Metric data will be obtained annually in November and reflect the data results from the previous fiscal year. The OSPE prepares this data.

The measurement results are based upon data derived from management information systems used within or by DOT such as the Federal Procurement Data System – Next Generation (FPDS-NG) or the bank serving the Purchase Card program. Each measurement, which is reflected on the BarBell Chart, has a targeted goal (when practicable) against which the data results are compared. This comparison indicates how well we are progressing toward our overall business goals.

The OSPE has also designated an additional set of statistics or metrics that are reviewed on quarterly bases. These metrics serve as the foundation for the measures used in the BarBell Charts. This data is displayed on the DOT Procurement Performance Spreadsheet or Metrics Spreadsheet.

The measurements rely heavily on the accuracy, reliability, and availability of data systems. The measurements that are chosen may deal with special DOT interests, initiatives from outside organizations such as Congress, issues of public policy, as well as Government-wide initiatives such as Performance Based Service Acquisition (PBSA). The specific measures are chosen by the APMC and the OSPE. This committee and the OSPE will review on a yearly basis our metric measures to insure we are measuring and providing OAs with useful information.

Cost to Spend Ratio

"Cost to Spend" data will be collected annually. OAs respond to a data call in October to provide the OSPE with their Cost to Spend Data. Data elements for the Cost to Spend Ratio are provided at Attachment L. OAs will be requested by email to submit this data on a specific form (Attachment M) to the OSPE in November.

1102s Meeting Qualification Standards

In 1996, the Clinger-Cohen Act (P.L. 104-106) changed Section 37 of the Office of Federal Procurement Policy Act to direct OFPP and agencies to establish Acquisition Workforce certifications and training programs for all agencies not covered by the Defense Acquisition Workforce Improvement Act (DAWIA) of 1990.

OFPP Policy Letter # 97-01, issued on September 12, 1997, implemented the Acquisition Workforce requirements of Clinger-Cohen. On August 3, 1998, DOT issued our requirements in “Acquisition Career Development Policy for Contracting Professionals.”

The data for these measurements are obtained from the appropriate personnel management information system by the OSPE in October. The data reflects the previous fiscal year accomplishments.

This measurement looks at three areas:

Education – This measures the percentage of the workforce that has achieved the required educational levels.

Training - This measures the percentage of the workforce that has achieved the required training levels.

Certified at all Levels – Measures the percentage of all contracting personnel that are certified at their appropriate level.

COTR Training

The same legislation and OFPP policy letter that established the criteria for contracting personnel also established mandatory training for all COTRs. On August 8, 1999 DOT issued our requirements in “Contracting Officer’s Technical Representative (COTRs) Training Standards.”

The data for this measurement will be obtained from OA systems that track the certification requirements for their COTRs. This information will be requested by the same email that is requesting Cost to Spend Ratio. OAs will state the percentage of their COTRs that have met the certification requirements for the previous fiscal year.

Information Technology (IT) Program Manager Certification

TBD

Cost, Schedule and Performance

TBD. This data will be obtained from the CIO and CFO through a data call in October each year.

BarBell Chart

The BarBell Chart is DOT’s unique way of displaying performance measurements. The BarBell Chart is designed in the Balance Scorecard concept, and the BarBell Chart reflects the five perspectives. Within these five perspectives, the results of the Survey’s Business Measures and Metric Measures are displayed.

We have two BarBell Charts, one is a “standard” (See Attachment I) display of the results and the second reflects the survey and metric measures under the appropriate Presidents Management Agenda (PMA) (See Attachment J) items; Human Capital, Competitive Sourcing, E-Gov, Performance and Budget, and Financial Management. The Competitive Sourcing data is provided by the Competitive Sourcing Office and is only reflected on the BarBell Chart for the PMA.

Each year the APMC reviews both BarBell Charts to insure that the measures reflected are current and meeting the objects of the PPMS.

Action Plans For Improved Performance

A necessary ingredient for continuous improvement is discussing how current and past results will affect the organization’s future strategies. Changes in the environment (e.g., new technology, legislative initiatives) may create new opportunities or risks not anticipated when the initial strategies were developed. The PPMS process serves as a foundation for effective improvement and risk management. It also completes the feedback loop created by performance measurement that supports management at all levels.

Action Plans are a critical component of the PPMS. OA Action Plan reviews are part of the OSPE annual PPMS meeting with the members of the PMC. We do not wish to impose any specific format or content requirement, because it is more important that Action Plans address the OAs recognized needs, rather than conform to a reporting requirement. The following are some suggestions and guidelines for OAs to use in preparing their Action Plans.

Purpose: Collecting data by itself is not useful information, but when viewed from the context of goals and objectives, it can be very useful. Proper analysis is imperative in deciding how effective a measurement is and if the results are contributing to our objectives. By using data from the two PPMS measurement tools--Survey, and Metric,—it can determine the level of progress made in reaching goals. To understand how this data can be used and what the results mean may best be answered by asking the following questions:

Does the data indicate any performance trends over time? How does the trend data look and does it reveal a shortfall in one area because of a gain in another? By comparing current and past results, it can be determined if we are on the proper course. Example: Our goal is to have both employee and customer satisfaction high. If you have had a trend of customer satisfaction being high but employee satisfaction low, you may have happy customers but your staff feels used or abused. If the trend lines are beginning to merge, then your action plan is working and you can determine what other areas need focus.
Were performance goals met? If not, explore what impacted successful performance.
Were performance goals exceeded? If so, perhaps other benefits to the organization can be gained such as operating cost reductions. Sharing such information may also spread the gain to other OAs.
Can the data be used to improve performance in areas other than the one(s) assessed or can it pinpoint specific areas for consideration? Data can help identify specific areas within the larger assessed area that needs attention.
Where improvement efforts should be focused to achieve the greatest or most needed return? Many times the amount of available resources dictates what areas to focus on to receive the greatest return on the resources invested.

Importance of trend data: To evaluate our performance level, trend data is perhaps the most critical of all. The more data available for a particular measurement, the better chance we have of getting an accurate picture of progress. While PPMS uses annual data, there are many measurements, especially in the DOT Procurement Performance Comparison Spreadsheet that can be reviewed on a monthly or quarterly basis. This data can provide a larger database from which to gauge performance and to see any fluctuations in performance.

Sharing Results: As with any performance management system, feedback is vital for the system to grow and flourish. Retaining results without sharing them, especially among its participants, will be the demise of this system. Conversely, to share and inform, is to nurture and promote the system. It is critical to let respondents and other interested parties know the results, but also it is important to collaborate and partner with them to develop strategies for improvement. We feel this is a necessary part of any Action Plan.

Strategies for Improvement: Once measurement results for DOT wide and OAs are analyzed and areas needing improvement determined, it is time to decide the strategies needed to achieve desired results. Those affected by the process are normally the best ones to reveal what needs to be done. Experience has shown us that identifying and solving one or two problem areas each year is much more effective than trying to solve all identified areas. It is this information that sets your objectives, goals and outcomes and improvements for the future.

Goal Setting: Every measure needs to have an associated goal to know where your organization is relative to your goals. It will help determine how fast or slow you need to go, and when you are there. Of critical importance, is to have clear, quantifiable goals for those measures contained in the Strategic and Performance Plans. OSPE has established goals (See Attachment B) for each of the items contained on the BarBell Chart. OAs are encouraged to develop unique measures and goals that address specific issues within an OA. Where individual OA measures are achieving above the DOT goal, the OA may want to establish a goal above the DOT goal. DOT Goals will be evaluated each year by the APMC and revised as appropriate.

Keys to Success: The key to success is to achieve as much balance as possible for all measures on the PPMS and to demonstrate the procurement organization’s benefit towards helping the program office achieve its mission. If one measure has a deep fluctuation creating a need to shift performance back, make adjustments slowly so problems do not occur in areas critical to a particular operation. Being able to use the measurement results to make good management decisions is the foundation of performance management. Attachment P provides a number of performance assessment tools that may be helpful when making decisions for improvement.

Future Success: The success of the PPMS is dependent upon many factors. These include:

Demonstrating strong support for the System at all levels of management.
Integrating procurement’s critical measures into DOT’s Strategic and Performance Plans and flowing those measurements down through individual employee performance agreements.
Spreading the word as to what performance measurement and performance management is, the purpose and the benefits.
Ensuring that the data systems provide accurate and reliable information.
Promoting participation in completing surveys and other evaluation tools.
Sharing results and providing feedback to participants to help nurture and promote improvement through performance management.
Using measurement results and partnering with customers to determine improvement strategies.
Translating management strategies into action for improved outcomes.
Setting achievable goals.

As we progress on our improvement journey, we expect the system to evolve, grow and become more effective.

[1] In the past, this measure was known as Effectiveness of Business Standards, under Internal Business Perspective. Because of the FY04 Review and changes, the measure has changed its name and the process is under development.

[2] The goal of 80% was selected, because the dollar value of the purchase could be greater than $25,000. This percentage will be reevaluated in FY05.

[3] Tam 1213, Appendix B, VI, “…Approving officials shall have no more that 10 cardholders under their purview, unless unavoidable circumstances make it impossible to achieve this ratio.”

[4] There was no PBSA goal established by OFPP for FY04. Each agency was left to adopt its own goals. We will remain at the FY03 goal of 30%. Please be advised the OFPP PBSA Goal for FY05 is 50%. This goal will be revised upward in FY05 to reflect the final OFPP Goal.

[5] We have not established a goal for Contract Bundling in FY04 because the historical data has been affected by a change in definition for a “bundled” contract. Based on further research with SBA and results in FY04 a DOT will be established in FY05.