Assessment Of The Results Of External Independent Reviews For U S Department Of Energy Projects

Peer review is an essential component of engineering practice and other scientific and technical undertakings. Peer reviews are conducted to ensure that activities are technically adequate, competently performed, and properly documented; to validate assumptions, calculations, and extrapolations; and to assess alternative interpretations, methodologies, acceptance criteria, and other aspects of the work products and the documentation that support them. Effective peer reviews are conducted in an environment of mutual respect, recognizing the contributions of all participants. Their primary objective is to help the project team achieve its goals. Reviews also contribute to quality assurance, risk management, and overall improvement of the management process. The U.S. Department of Energy (DOE) conducts different types of peer reviews at the different stages of a project, including reviews to assess risks and other factors related to design, safety, cost estimates, value engineering, and project management. Independent project reviews (IPRs) are conducted by federal staff not directly affiliated with the project or program and management and operations (M&O) contractors. External independent reviews (EIRs) are overseen by the Office of Engineering and Construction Management and conducted by contractors external to the department. EIRs are the primary focus of this report. However, the committee found that, in many cases, IPRs are explicitly used as preparation for or as preliminary reviews prior to EIRs. Thus, because IPRs are integral to the review process in DOE, they are also discussed because they might have an effect on EIRs. In October 2000, DOE issued Order 413.3, Program and Project Management for the Acquisition of Capital Assets (DOE, 2000). The order established a series of five critical decisions (CDs), or major milestones, that require senior management review and approval to ensure that a project satisfies applicable mission, design, security, and safety requirements: approve mission need, approve alternative selection and cost range, approve performance baseline, approve start of construction, and approve start of operations or project closeout. Assessment of the Results of External Independent Reviews for U. S. Department of Energy Projects summarizes the results.

The Office of Science and Technology (OST) of the U.S. Department of Energy's (DOE's) Office of Environmental Management (EM) recently has instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development. OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. This is the first of two reports to be prepared by this committee on OST's new peer review program. OST requested this interim report to provide a preliminary assessment of OST's new peer review program. In the final report, the committee will provide a more detailed assessment of OST's peer review program after its first complete annual cycle.

The U.S. Department of Energy has been at the center of many of the greatest achievements in science and engineering in this century. DOE spends billions of dollars funding projects-and plans to keep on spending at this rate. But, documentation shows that DOE's construction and environmental remediation projects take much longer and cost 50% more than comparable projects undertaken by other federal agencies, calling into question DOE's procedures and project management. What are the root causes for these problems?

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organizations refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. When the organization under discussion is formally labeled a laboratory, it is referred to as such. The question arises: How does one know whether an organization and its programs are achieving excellence in the best interests of its parent? Does the organization have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support the interests of its parent? This report offers assessment guidelines for senior management of organizations and of their parents. The report lists the major principles of assessment, noting that details will vary from one organization to another. It provides sufficient information to inform the design of assessments, but it does not prescribe precisely how to perform them, because different techniques are needed for different types of organizations. Best Practices in Assessment of Research and Development Organizations covers three key factors that underpin the success of an R&D organization: (1) the mission of the organization and its alignment with that of the parents; (2) the relevance and impact of the organization's work; and (3) the resources provided to the organization, beginning with a high-quality staff and management.

A nuclear weapon or a significant quantity of special nuclear material (SNM) would be of great value to a terrorist or other adversary. It might have particular value if acquired from a U.S. facility-in addition to acquiring a highly destructive tool, the adversary would demonstrate an inability of the United States to protect its nuclear assets. The United States expends considerable resources toward maintaining effective security at facilities that house its nuclear assets. However, particularly in a budget-constrained environment, it is essential that these assets are also secured efficiently, meaning at reasonable cost and imposing minimal burdens on the primary missions of the organizations that operate U.S. nuclear facilities. It is in this context that the U.S. Congress directed the National Nuclear Security Administration (NNSA)-a semi-autonomous agency in the U.S. Department of Energy (DOE) responsible for securing nuclear weapons and significant quantities of SNM-asked the National Academies for advice on augmenting its security approach, particularly on the applicability of quantitative and other risk-based approaches for securing its facilities. In carrying out its charge, the committee has focused on what actions NNSA could take to make its security approach more effective and efficient. The committee concluded that the solution to balancing cost, security, and operations at facilities in the nuclear weapons complex is not to assess security risks more quantitatively or more precisely. This is primarily because there is no comprehensive analytical basis for defining the attack strategies that a malicious, creative, and deliberate adversary might employ or the probabilities associated with them. However, using structured thinking processes and techniques to characterize security risk could improve NNSA's understanding of security vulnerabilities and guide more effective resource allocation.

In 1997, Congress, in the conference report, H.R. 105-271, to the FY1998 Energy and Water Development Appropriation Bill, directed the National Research Council (NRC) to carry out a series of assessments of project management at the Department of Energy (DOE). The final report in that series noted that DOE lacked an objective set of measures for assessing project management quality. The department set up a committee to develop performance measures and benchmarking procedures and asked the NRC for assistance in this effort. This report presents information and guidance for use as a first step toward development of a viable methodology to suit DOE's needs. It provides a number of possible performance measures, an analysis of the benchmarking process, and a description ways to implement the measures and benchmarking process.

The Department of Energy (DOE) is engaged in numerous multimillion- and even multibillion-dollar projects that are one of a kind or first of a kind and require cutting-edge technology. The projects represent the diverse nature of DOE's missions, which encompass energy systems, nuclear weapons stewardship, environmental restoration, and basic research. Few other government or private organizations are challenged by projects of a similar magnitude, diversity, and complexity. To complete these complex projects on schedule, on budget, and in scope, the DOE needs highly developed project management capabilities. This report is an assessment of the status of project management in the Department of Energy as of mid-2001 and the progress DOE has made in this area since the National Research Council (NRC) report Improving Project Management in the Department of Energy (Phase II report) was published in June 1999.