Posted by A Step-Wise Approach to Elicit Triangular Distributions
Risk II Track
R2-11_Presentation_AStepWiseApproachToElicitTriangularDistributions_Greenberg
Abstract:
As the federal government acquires less mature, more advanced and more complex systems, there is an ever-increasing burden on the cost analyst to employ methods of eliciting requirements, schedule and cost uncertainties from one of more subject matter experts (SMEs). Arguably, the most common technique a cost analyst uses today to elicit such data is to ask each SME for the lowest, most likely and highest value which, consequently, produces a triangular distribution.
Eliciting and using a triangular distribution has its advantages. Getting the SME to provide the three input values takes only a few minutes, the SME can provide a reasonable basis for his or her input values and the distribution represents the SME’s first-order approximation of what s/he believes to be the uncertainty. However, this common process of depicting uncertain input parameters typically produces optimistic estimates. More specifically, structural limitations inherent to the triangular distribution coupled with the optimistic bias of the SME tend to produce optimistic estimates.
This paper proposes an approach that may enhance the SME-elicitation process while, at the same time, provides a means for the cost analyst to employ uncertainty distributions that are best suited for the given type of estimate. The approach assumes a scenario where the SME agrees that there is some measure of uncertainty about a “most-likely” estimate. Under such a scenario, the SME presents bias towards the most-likely estimate (aka “anchoring”) which, in turn, typically results in a probability distribution having underspecified dispersion.
In order to counter his/her “anchoring” to the most-likely estimate, the interviewer asks the SME twelve questions. Once the questions are completed, the interviewer asks the SME eight of the same questions. However, in this second iteration, these repeated questions are revised to account for “contributing risk factors” identified during the first iteration. The value-added of this iterative process is that the SME can take advantage of lessons-learned during the first iteration as well as use a visual aid to evaluate how his/her opinion impacts the parameters and shape of the uncertainty distribution.
Such an elicitation process demonstrates how a cost analyst can interview experts in a repeatable fashion in order to develop a triangular distribution. The process includes elicitation and mathematical techniques so that the cost analyst can adjust the triangular distribution for SME bias. The paper provides both the twelve-step methodology and a case study on eliciting a triangular distribution of commuting time.
The paper concludes with an approach to transform the “bias adjusted” triangular distribution into a PERTbeta distribution. Such a transformation of the triangular distribution produces what is considered a more complete depiction of uncertainty. The cost analyst can then use the beta parameters as inputs into a Monte Carlo simulation tool (such as Crystal Ball or @Risk), where such inputs that are now readily traceable to assumptions and rationale.
Author:
Marc W. Greenberg
NASA Cost Analysis Division (CAD)
Marc Greenberg has been working for NASA’s Cost Analysis Division (CAD) since October 2012. At NASA, he is helping to develop technology cost estimating methods and has facilitated the stand-up of the NASA Cost IPT. Prior to being hired by NASA, Mr. Greenberg worked for the Department of Homeland Security (DHS). At DHS, he helped programs implement and/or improve their cost estimating practices. His efforts included policy development, consolidating best practices, providing cost-specific guidance, conducting workshops and providing on-site support. He also provided cost oversight and guidance to a variety of programs within DHS’ National Protection and Programs Directorate (NPPD).
From November 2008 to June 2011, Mr. Greenberg worked for the Naval Center for Cost Analysis (NCCA). At NCCA, he performed a cost assessment of the Ship-to-Shore Connector concept, developed cost estimates for the Navy’s Enterprise Resource Planning (ERP) program and led a one-year effort estimating Operating & Support costs for Ballistic Missile Defense. In October 2010, Mr. Greenberg became NCCA’s Cost Research Branch head where he led efforts on improving existing cost methods, revising DoN cost policy and collaborating with DoD’s cost research community.
From October 2005 through October 2008, Mr. Greenberg taught for the Defense Acquisition University (DAU). At DAU, he taught acquisition professionals the principles of cost analysis and was involved in the development and delivery of various course curricula. Prior to teaching at DAU, Mr. Greenberg worked as a cost analysis for the Naval Sea Systems Command (NAVSEA, Carderock) for fourteen years. At NAVSEA Carderock, he conducted cost studies on Navy ships, submarines and emerging technologies. His efforts concentrated in risk analysis, cost modeling, technology cost and cost-benefit analysis. Mr. Greenberg also served on co-located cost teams that supported acquisition efforts for the New Attack Submarine Program, Future Aircraft Carrier Program and the Littoral Combat Ship Program.
Prior to his career with the Navy, Mr. Greenberg worked as an electronics engineer for the US Army Information Systems Engineering Command where he provided support in simulation, design and construction of high frequency and microwave communication systems.
In 1987, Mr. Greenberg received his bachelor’s degree in ceramic science and engineering from the Pennsylvania State University. In May 1998, he received his master’s degree in engineering management from the George Washington University. Mr. Greenberg is professionally certified by the International Cost Estimating and Analysis Association (ICEAA) as a Cost Estimator/Analyst and is Level III certified in Business – Cost Estimating (BUS – CE). In 2003, he joined the Omega Rho International Honor Society, an organization that recognizes academic achievements in operations research and management science.