NATIONAL ADVISORY COUNCIL
OFFICE OF MANAGEMENT AND BUDGET
PERFORMANCE MEASUREMENT ADVISORY COUNCIL
SUMMARY OF MEETING

March 3, 2003

The Performance Measurement Advisory Council (PMAC) was convened for its third and final meeting at 1:00 P.M. on March 3, 2003, at the White House Conference Center, Washington, D.C. Reid Cramer, acting as the Council’s Designated Federal Officer, opened the meeting along with Mortimer Downey, the Council’s Chair.

In accordance with the provisions of Public Law governing Federal Advisory Committees, the meeting was open to the public.

Council members present:

Mortimer L. Downey
William Eggers
Harry P. Hatry
Patricia Ingraham
Donald F. Kettl
Joseph Wright, Jr.

Meeting Called to Order

Mortimer Downey opened the meeting. He stated that the main focus of the meeting would be to offer feedback on the presentation of the program assessments in the 2004 Budget. He stated his view that this was an incremental process, where improvements should be made along the way.

Reid Cramer reported that input from the PMAC’s first two meetings was helpful in developing both the PART assessment tool and the budget presentation. The release of the President’s Budget provides the public with an unprecedented amount of information revealing what the government achieves with its public resources. At this point, OMB intends to use the PART again as a central element of the 2005 budget process, rating an additional 20% of federal programs. OMB also anticipates communicating the findings of these analyses in a similar presentation. Although wholesale changes to the PART are not expected, OMB is interested in strengthening the process, the instrument, and the budget presentation.

Presentation by Individual Members

Members were asked to summarize their responses to the set of questions that had been distributed to members prior to the meeting and were included on page 52 and 53 in the Budget volume.

The following questions were considered by the Council:

• What do you see as the strengths and weaknesses of the presentation of performance information in the PART summaries?
  
  
• How can the PART be improved or revised? Page 52 of the Budget described a series of issues for the PART and invited public comments. These issues included increasing consistency, defining “adequate” measures, minimizing subjectivity, and measuring progress towards results.
  
  
• Given that the PART provides an incentive for agencies to improve performance information, how should the PART treat programs that have acceptable measures but bad targets?
• Many programs lack outcome measures either because they cannot pinpoint the contribution of the program to achievement of the overall outcome or because the nature of their work is process-oriented. Are there acceptable alternative measures when there are no outcome goals?
  
  
• To get credit for results, the PART required that a program have acceptable measures, either long-term or annual. Should the PART continue to enforce a linkage between strategic planning and results?

Strengths and weaknesses

Harry Hatry articulated his views that the results information should be the primary basis for program assessment and he liked the separation of the results score. He thought the distinction between long-term and annual goals is confusing and not constructive.

Patricia Ingraham thought the presentation was clear and understandable, achieving significant transparency. She was concerned that the objectives of a program do not always match the measures. Contextual issues, such as changing missions, should be taken into account.

Don Kettle stated he thought that the presentation cries out for web access to make it accessible and available to be manipulated for analysis.

Mort Downey observed that the assessment volume and the worksheets offer a rich data set, but comparisons are important. More crosscutting analysis should be pursued in the future.

William Eggers said that he thought the PART material was well integrated into the individual chapters of the Budget, but attention should be paid to making it a web-based document. He believes the common measures exercise is an important and related exercise that needs to be highlighted further.

Joe Wright liked the format. He said the assessment volume should look like a budget book and it does. He appreciated the graphics. He suggested OMB consider talking to its oversight committees in Congress to ask that they require it. This would facilitate institutionalization of the process. Many of the assessments would benefit from have more long-range targets. He thought that some attention should be given to establishing a sense of priorities because currently everything is given the same weight. He believes that the Budget is a policy document and the assessments should not shy away from asking each program if there is a justifiable federal role.

Mort Downey recognized that the PART was valuable in helping to shed light on the work of small programs, many of which were looedk at for the first time. This was constructive.

Improving process

Don Kettle stated that he would like to see much discussion of goals and purpose because for many programs the purpose was unclear. The linkage between the assessment findings and the budget recommendation should also be made stronger. Consistency is a major issue that is related to OMB resource commitments to this process.

Patricia Ingraham noted that as much training as possible would be helpful.

Harry Hatry suggested that the first three sections of the PART be given a grade but the weighting of each individual question could be dropped. Harry also expressed concerns that the PART is compatible with GPRA. OMB should provide clear guidance on the relationship between the two exercises. In the future measures need to be improved, both in the selection and data collection procedures. Intermediate outcomes and ultimate outcomes are both important. In-depth program evaluations should be used when there are recent and of good quality. Sometimes the findings of these reports are more important than the program’s outcome measures. Also, OMB should look at ways of capturing the distributional effects of a program, if they are important.

Joe Wright cited some examples of where he thought the PART was applied inconsistently. A similar program with a better set of targets scored lower than another. He suggested that the process could be improved by engaging with the community of Inspector Generals to assist in the auditing of program assessments. This could help address OMB’s workload issue as it expands the number of programs assessed.

William Eggers stated that OMB could do a better job of stating the purpose of the assessments and also improving the connection between the budget recommendations and the assessments. He thought the assessments need to describe the trajectory of a program (whether it was improving over time). OMB should use the PART to identify statutory changes and regulatory barriers but assess program on what they should be doing. The selection of programs to be assessed should be done according to themes so that it supports the common measures effort. In terms of training, OMB should find ways of highlighting PART best practices.

Mort Downey thought that transparency will help with subjectivity. The process should continue to be as transparent as possible. OMB should systematically survey participants over ways to improve the process.

Joe Wright expressed satisfaction that the PART represents a good effort to date and put the “M” back in OMB. Still OMB needs to look for every hook to make the process stick.

Adjournment

The meeting adjourned at 3:30 P.M.

I hereby certify that, to the best of my knowledge, the foregoing minutes are accurate and complete.

Reid Cramer, Designated Federal Officer for the March 3^rd meeting.