Posted by Making Risk Management Tools More Credible: Calibrating the Risk Cube
CEBok Reference Track
Downloadable Files:
Making Risk Management Tools – Coleman
Abstract
The so-called Risk Cube is arguably the most widely used tool in Risk Management (RM) today. The method involves creating a risk register of all foreseeable risks. The attendant register is intended to give risk managers and program offices a tool to identify and manage risks to allow mitigation. The quantitative version of this method assigns a probability of failure (Pf) and a consequence of failure (Cf) to each risk. The qualitative version uses some measure of “likelihood” and consequence which are usually low, and high or ordinal scales. This paper will discuss an improvement to the qualitative method. In reviewing the qualitative method the authors have observed three general shortcomings. First, many programs, despite having the means at their disposal, do not roll up the risk register to determine the expected outcome in total. This is easily remedied and if Monte Carlo methods are used (which will be briefly explained in the paper) a distribution of outcomes can be produced. Secondarily, many programs fail to allow for any “unknown unknowns” which are well established as necessary in risk literature; accordingly, they yield an unreasonably low expected value. Thirdly, the authors have noted that virtually all Risk Cube users fail to have their methodology calibrated in a way that lends itself to producing reasonable outcomes. Many, even most programs have scales for consequence whose highest end is below the average cost growth for their commodity. The authors will discuss the virtues of a scale that is based upon (calibrated to) historical cost growth, in such a way as to yield typical cost and schedule growth for the commodity when average cost and schedule consequences are assessed. If the program has unusually high or low consequences assessed, the answer will deviate from historical averages, just as it should. The correct way to do this calibration is discussed, and calibrations for typical DoD and ship programs will be presented. The advantage of this method is clear, in that it will yield a reasonable and historically grounded result.
Author(s):
Jessica R. Summerville
Ms. Summerville graduated with a B.A. in Mathematics from the University of Michigan in 1994. She received an M.S. in Operations Research from the College of William and Mary in 1995. During her Masters program, she worked as a Programmer/Analyst at SAIC in support of NASA’s Earth Radiation Budget Experiment. Upon graduation, she joined TASC, Inc., where she works as a cost estimator and risk analyst in support of the Ballistic Missile Defense Organization (BMDO), NAVAIR, the intelligence community, and the Cost Development Team for the Gold Team of the Navy’s 21st Century Destroyer Program (DD 21). Her cost estimating duties have involved lead analyst for National Missile Defense (NMD) radar systems and the Patriot Advanced Capability (PAC-3) missile system. In her capacity as Lead Analyst for Cost Risk for several organizations, she oversees the conduct of Schedule and Technical Risk Assessments and Cost Risk Analysis. In addition, she is a key member of multiple teams and working groups that perform research and modeling to advance the development of cost risk analysis.
Richard L. Coleman
Mr. Coleman was commissioned upon graduation from the United States Naval Academy on 5 June 1968, where he received a B.S. in Naval Engineering with a minor in Operations Analysis. He received a Master of Science Degree in Operations Research (With Distinction) from the Naval Postgraduate School in Monterey, California in September 1974. Graduating first in his class, he was the recipient of the Chief of Naval Operations Award for Excellence in Operations Research. In the Navy, his career culminated in tours as Commanding Officer of USS Dewey (DDG 45) and Director, Naval Center for Cost Analysis. His awards include the Legion of Merit, the Meritorious Service Medal with two Gold Stars, and the Navy Commendation Medal with one Gold Star. He retired from active duty as a Captain, U. S. Navy in 1993. Since retirement, he has worked for TASC, Inc. He conducts risk analysis at the Ballistic Missile Defense Office and NAVAIRSYSCOM. He supports the Navy’s Acquisition Reform Office in Cost as An Independent Variable (CAIV) & Total Ownership Cost. He leads the Gold Team in cost analysis of the Navy’s 21st Century Destroyer (DD 21). Additionally, he supports the intelligence community in cost, risk and CAIV. He is a Regional Vice President of the Society of Cost Estimation and Analysis. He was a co-author of two papers at the 1998 ISPA/SCEA International Conference, one on CAIV and Total Ownership Cost, awarded Best Paper in Acquisition Reform, and one on Cost Risk, winning Best Paper Overall, and a co-author of a paper that won Best Paper in CAIV at the 1999 ISPA/SCEA International Conference.