Posted by Understatement of Risk and Uncertainty by Subject Matter Experts
Risk Track
RS11_Presentation_UnderstatementofRiskandUncertbySMEs_Braxton
Abstract:
The authors’ paper “The Correct Use of Subject Matter Experts in Cost Risk Analysis,” presented at the Naval Postgraduate School (NPS) Acquisition Research Symposium (ARS) and Department of Energy Cost Analysis Symposium (DoECAS), explored alternative methods for conflating the risk distributions of multiple subject matter experts (SMEs). This paper builds on the results therein by establishing a mathematical foundation for the conflation of triangular distributions and by presenting the results of an independent experiment on expert assessment of uncertain quantities.
The basic conflation methods presented in the cited paper include averaging random draws from the expert distributions, with or without correlation, for each run of the Monte Carlo; “averaging” the distributions themselves by averaging the means (or modes) and either averaging the extrema or taking the extrema of the extrema; and sampling the expert distributions. This paper provides mathematical proofs of which of these methods are equivalent and how they compare in general. It reiterates recommendations for best conflation method in the cases of “single reality” and “multiple realities.”
In much of the literature cited in the previous paper, experiments were conducted to gauge the degree to which SMEs tend to underestimate uncertainty (the range of possible values). These experiment generally focused on knowable but uncertain quantities, such as the height of Mount Kosciuszko. The authors improved the results of these studies by conducting their own experiments to include: (1) unknown and uncertain quantities, such as the number of points scored in the upcoming Super Bowl; (2) a direct (self-)assessment of the degree of the SME’s expertise in both the subject of the assessment and the (meta-)subject of risk assessment; (3) a quantification of the understatement of risk (tendency for growth) in addition to understatement of uncertainty (CGFs in addition to CVs); and (4) cost estimators and other defense acquisition professionals as the subject of the experiments.
Together, these results hold the promise to significantly increase the quality and fidelity of expert quantification of risk.
Author(s):
Peter J. Braxton
Technomics, Inc.
Peter J. Braxton holds an AB in Mathematics from Princeton University and an M.S. in Applied Science (Operations Research) from the College of William and Mary. A Senior Cost Analyst at Technomics, Inc., he is a Certified Cost Estimator/Analyst (CCEA) and currently serves as Director, Body of Knowledge, for the Society of Cost Estimating and Analysis (SCEA). He was named SCEA’s 2007 Estimator of the Year for contributions in Education and received both a TASC President’s Award for Operational Excellence and a Northrop Grumman Corporate Contracts and Pricing Award in 2008. He served as SCEA’s Training Chair from 2004 to 2009 and as a Northrop Grumman Technical Fellow from 2006 to 2009.
He currently supports the Naval Center for Cost Analysis (NCCA) and Air Force Cost Analysis Agency (AFCAA) on data collection and cost research efforts, and the Naval Postgraduate School (NPS) and Defense Acquisition University (DAU) on curriculum development. Throughout more than a dozen years in the community, he has worked to advance the state of knowledge of cost estimating and risk analysis, Cost As an Independent Variable (CAIV), and Target Costing on behalf of the Navy Acquisition Reform Office (ARO), the DD(X) development program, and other ship and intelligence community programs. He has conducted independent cost evaluations (ICEs) on all of Northrop Grumman’s major shipbuilding programs, and on several IT systems and services programs. He served as managing editor for development and maintenance of the acclaimed Cost Estimating Body of Knowledge (CEBoK) and its predecessor, Cost Programmed Review Of Fundamentals (CostPROF), and as the Training Track chair for the last seven SCEA international conferences. He has taught extensively at government, corporate, and society training events throughout the United States, Europe, and Australia. He is lead author or co-author of over a dozen professional papers on cost, risk, and CAIV, including two SCEA Best Paper winners.
Richard L. Coleman
TASC
Dick Coleman is a Naval Academy graduate, received an M. S. with Distinction in Operations Research from the U. S. Naval Postgraduate School, where he received the Chief of Naval Operations Award for Excellence in Operations Research. He retired from active duty as a Captain, USN, in 1993. His service included tours as Commanding Officer of USS Dewey (DDG 45), and as Director, Naval Center for Cost Analysis. At TASC and then Northrop Grumman, he worked extensively in cost, CAIV, and risk for the Missile Defence Agency (MDA), Navy ARO, the intelligence community, NAVAIR, and the DD(X) Design Agent team. He was the Director of Independent Cost Estimation for Northrop Grumman Information Systems and conducted Independent Cost Evaluations on over 150 Northrop Grumman programs in ships, IT, electronics, mission and space systems. He supported numerous ship programs including the DDG 1000 class, DDG 51 class, Deepwater, NSC 1 class, LHD 8 and LHA 6, LPD 17 class, Virginia class SSNs, CVN 77, and CVN 78. At Technomics, Inc., he was a Senior Technical Fellow and supported the Naval Center for Cost Analysis, the Department of Homeland Security, the Naval Postgraduate School, and the Defence Acquisition University in cost estimation and risk analysis. He retired from full-time employment in 2011. He has over 80 professional papers to his credit, including five ISPA/SCEA and SCEA Best Paper Awards and two ADoDCAS Outstanding Contributed Papers. He was a senior reviewer for CostPROF and CEBoK and lead author of the Risk Module. He has served as Regional and National Vice President of SCEA and is currently a Board Member. He received the SCEA Lifetime Achievement Award in 2008.