Posted by The Unseen: Statistical Inference with Limited Data
Risk Track
RSK14_Presentation_TheUnseen-StatisticalInterferencewithLimitedData_VanAtta
RSK14_Paper_TheUnseen-StatisticalInterferencewithLimitedData-VanAtta
Abstract:
“Objective measurements of probability are often unavailable, and most significant choices under risk require an intuitive evaluation of probability.” -Nobel Laureates Daniel Kahneman and Amos Tversky
What are the odds of rolling a sum total of seven when tossing two dice? What is the probability of red turning up after a spin of a European roulette wheel? Most analysts, given a little time and a calculator, could answer these two questions with exact precision. For both of these questions, there is only one true correct answer. Such is the nature of probability analysis for questions that are decompositional (all possible outcomes can be determined), frequentistic (the experiment can be repeated an infinite number of times), and algorithmic (the results can be measured with numbers). Unfortunately, as pointed out in the quote above, not all questions involving uncertainty can be measured with precise probabilities, and, more often than not, we must rely on our intuition to evaluate risk. For example, there is no perfect probability measure when evaluating the odds that a specific applicant will be a successful employee if hired, or assessing the likelihood that a witness is telling the truth during testimony.
But what about cost estimates? Can we calculate the precise probability of a project overrunning one million dollars through the use of statistics? Unfortunately, no matter how much data gathering and analysis we do, we cannot place limits on the real world. We could calculate a 70% confidence interval for an estimate, but some unexpected event could occur (i.e. an earthquake, a financial collapse, etc.) that alters the odds and throws the cost of the project spinning out of control. Real world scenarios are not decompositional such that we can account for all possible outcomes, so an objective, perfect probability calculation isn’t possible regarding cost risk.
So then, how do we evaluate cost risk? Must we default to our intuitions and give up on statistics altogether? This question is very real in the Department of Defense, especially because we often don’t have more than a few reliable data points upon which to formulate our cost estimates. Nevertheless, there are practical means by which we can bridge the gap between intuition and mathematics. We can combine psychological research on intuitive judgment from recent decades with mathematics to understand how to draw distributions when the data that we’ve gathered isn’t enough to perform parametric analysis or a Goodness-of-Fit test. Cost risk analysis requires both mathematical and intuitive processes, so exploring and understanding the flaws and powers of our intuition is the only way to connect the real world with mathematics so that we can formulate meaningful statistical inferences.
Author:
Trevor VanAtta
US Army Tank-automotive & Armaments Command (TACOM)
Trevor L. VanAtta is a graduate of the University of Michigan, where he obtained a B.A. in Economics in 2006. He is currently an Operations Research Analyst at the U.S. Army Tank-automotive and Armaments Command (TACOM), where he specializes in cost and risk analysis for tracked combat vehicles. He has performed, analyzed, and validated cost estimates across the entire life-cycle of numerous systems, including tracked combat vehicles, route clearance vehicles, unmanned helicopters, autonomous and non-autonomous robotic vehicles, unmanned aerial vehicles, ground sensors, wheeled tactical vehicles, and combat support systems. Mr. VanAtta also worked in the Office of the Secretary of Defenses Cost Analysis and Program Evaluation (OSD CAPE) office in 2009, where he helped perform Independent Cost Estimates (ICEs) for ground robots, unmanned aerial vehicles, ground sensors systems, and the Navys EA-18G Growler aircraft.
Mr. VanAtta has also served as Chair of the U.S. Army Cost Risk Working Group since 2008, leading the effort to write the Armys Cost Uncertainty and Risk Analysis Guidance in 2009. He also published the Armys January 2011 Cost Uncertainty & Risk Analysis Quick Reference Guide. He has performed cost uncertainty and risk analysis for the Armys Ground Combat Vehicle (GCV) program, consulted with numerous tactical and combat vehicle program offices at TACOM in their development of cost risk analysis, and provided detailed instruction on the topic of cost risk analysis across the Army. In addition to performing cost risk analysis for Program Office Estimates, Mr. VanAtta has also successfully applied the concepts of cost risk analysis in support of Army program affordability assessments.