Posted by An Elicitation Method to Generate Minimum-Bias Probability Distributions
Models and Methods Track
MM15_Presentation_ElicitationMethodtoGenerateMinBiasProbDistr_Greenberg
Abstract:
As the US Department of Defense develops less mature, more advanced and more complex systems, there is an ever-increasing burden on the cost analyst to employ methods of eliciting requirements, schedule and cost uncertainties from one of more subject matter experts (SMEs). Arguably, the most common technique a cost analyst uses today to elicit such data is to ask each SME for the lowest, most likely and highest value which, consequently, produces a triangular distribution.
Eliciting and using a triangular distribution has its advantages. Getting the SME to provide the three input values takes only a few minutes, the SME can provide a reasonable basis for his or her input values and the distribution provides the SME’s first-order approximation of what s/he believes to be the uncertainty. However, this common process of depicting uncertain input parameters typically produces optimistic cost estimates. More specifically, structural limitations inherent to the triangular distribution coupled with the optimistic bias of the SME tend to produce optimistic cost estimates.
This paper proposes an approach that may enhance the SME-elicitation process while, at the same time, provides a means for the cost analyst to employ uncertainty distributions that are best suited for the given type of cost estimate. The approach begins with a “risk analysis-method” matrix that aids in the decision to (a) only do a “most-likely” point estimate, (b) perform sensitivity analysis around the “most-likely” estimate, or (c) develop an uncertainty distribution upon completing the sensitivity analysis.
Assuming the matrix leads to the decision to develop an uncertainty distribution, there are still several paths the cost analyst can take (i.e. the matrix includes guidance on what uncertainty distributions are recommended for the given type of estimate). For illustrative purposes, a beta distribution was selected as the most preferred. An elicitation template demonstrates how the cost analyst can interview the SME in order for the cost analyst to develop a triangular distribution. This template includes elicitation techniques so that the cost analyst can adjust the triangular distribution for SME bias.
The paper concludes with an approach to transform the “bias adjusted” triangular distribution into a beta distribution. Such a transformation of the triangular distribution produces what is considered a more complete depiction of uncertainty. The cost analyst can then use the beta parameters as inputs into a Monte Carlo simulation tool (such as Crystal Ball or @Risk), where such inputs that are now readily traceable to assumptions and rationale.
Author:
Marc Greenberg
Naval Center for Cost Analysis
Hired by the Naval Center for Cost Analysis (NCCA) in November 2008, Mr. Greenberg has led an effort for estimating the Navy’s Operating & Support costs for Ballistic Missile Defense, developed cost estimates for the Navy’s Enterprise Resource Planning (ERP) program and performed a cost assessment for the Ship-to-Shore Connector concept. As the Head of NCCA’s Cost Research Branch, he continues to develop methods to improve upon existing cost risk methods and policy.
From October 2005 through October 2008, Mr. Greenberg taught for the Defense Acquisition University (DAU). At DAU, he taught acquisition professionals the principles of cost analysis and was involved in the development and delivery of various course curricula. Prior to teaching at DAU, Mr. Greenberg worked as a cost analysis for the Naval Sea Systems Command (NAVSEA, Carderock) for fourteen years. At NAVSEA Carderock, he conducted cost studies on Navy ships, submarines and emerging technologies. His efforts concentrated in risk analysis, cost modeling, technology cost and cost-benefit analysis. Mr. Greenberg also served on co-located cost teams that supported acquisition efforts for the New Attack Submarine Program, Future Aircraft Carrier Program and the Littoral Combat Ship Program. Prior to his career with the Navy, Mr. Greenberg worked as an electronics engineer for the US Army Information Systems Engineering Command where he provided support in simulation, design and construction of high frequency and microwave communication systems.
In 1987, Mr. Greenberg received his bachelor’s degree in ceramic science and engineering from the Pennsylvania State University. In May 1998, he received his master’s degree in engineering management from the George Washington University. Mr. Greenberg is professionally certified by the Society of Cost Estimating and Analysis as a Cost Estimator/Analyst and is DAWIA Level III certified in Business Cost Estimating and Financial Management. In 2003, he joined the Omega Rho International Honor Society, an organization that recognizes academic achievements in operations research and management science.