Posted by Mathematical Lessons Learned from a Year’s Worth of ICEs
Management Track
MT14_Presentation_MathLessonfromICEs_Boulais
Abstract:
Having conducted over 20 Independent Cost Estimates (ICEs) across the Intelligence Community (IC) over the past year, there were a few common mathematical concerns that needed addressing in multiple estimates. These concerns appeared time and again from a cost method and application point of view. Three such topics will be discussed in this paper: calculating and applying correlation, calculating a cost factor from an average “rate”, and properly implementing distributions in Monte Carlo simulations such that the “Mean” point estimate and the “Mean” output from the simulation are the same.
Implementing correlation in cost estimates happens to a varying degree. While some estimates have no basis for the correlation used, others are based on a set of data using the “Correl” function in MS Excel. The Correl function uses Pearsons Product Moment Correlation Coefficient to determine the correlation between two variables. Once the correlation is determined, the coefficient is then placed in a MC simulation package such as @Risk or Crystal Ball to develop an S-Curve. What is often overlooked in the application of this process is that most simulation packages used Spearman’s Rank Correlation during implementation. The two methods are different and in some cases the same dataset can yield very different results. This paper shows how to deal with this problem and recommend a solution that provides consistent results from a correlation perspective.
SW Productivity is of particular interest when developing ground based estimates. While there are varying definitions of SW productivity, it often comes down to some measure of effort vs. some measure of time (otherwise known as a “rate”). What weve seen (and where the problems arise) is when analysts use a dataset to calculate an average SLOC/HR, but then apply the productivity factor in a model as if the inverse of average SLOC/HR equals the average HR/SLOC. Based on the most common method of calculating an average (the arithmetic mean), this is not necessarily true. This paper mathematically explores the three different Pythagorean Means: arithmetic, geometric, and harmonic and discusses which is best to use for calculating an average SW productivity factor and why.
One final area of note is in the implementing of distributions in Monte Carlo Simulations and their impact on the mean of the results. Dividing by distributions in a simulation can provide skewed results in that the mean estimate based on the inputs will not equal the mean estimate based on the simulation. The paper examines those impacts both mathematically and via simulation and recommends a solution for analysts producing cost estimates in the future.
Author(s):
Ryan W. Boulais
Scitor Corporation
Mr. Boulais began his career in 1998 as a Military Intelligence officer with assignments in Bamberg and Wurzburg, Germany, Fort Monroe, VA, and a deployment to Kosovo. In 2003, he accepted a position with Northrop Grumman TASC as a cost analyst supporting customers at the NGA, MDA, Navy, and internal corporate customers. In 2004 he was promoted to Section Manager. In 2006, Mr. Boulais was recalled from the Army’s individual ready reserve (IRR), re-designated as a Civil Affairs officer and deployed to Baghdad, Iraq for a year where he served as the Economic Team Leader for the Baghdad Provincial Reconstruction Team (PRT) and the Executive Officer to the US Ambassador in charge of reconstruction for Iraq. Upon his return, he resumed his career at TASC as a Section Manager with customers at the NGA, NRO, and JFCOM.
In January 2009, Mr. Boulais joined Scitor Corporation. He has supported various customers within the NRO including leading a business process re-engineering (BPR) effort in the FM environment and developing a risk-adjusted independent cost estimate for a source selection effort. In his current role, Mr. Boulais is the contractor research lead at the ODNI Cost Analysis Improvement Group.
Mr. Boulais received a B.S. in Operations Research from the United States Military Academy at West Point in 1998 and an M.A. in Systems and Information Engineering from the University of Virginia in 2004.
Brett Dickey
Scitor Corporation
Brett Dickey received his B.S. and M.S in Systems Engineering from the University of Virginia in 2006 and 2008, respectively. After graduating, Mr. Dickey worked as a cost analyst at TASC, providing Independent Life-Cycle Cost Estimates (I-LCCEs), TASC-internal cost estimates, and proposal support. Mr. Dickey currently works at Scitor Corporation where he provides support to the Office of the Director of National Intelligence (ODNI) Systems and Resources Analysis (SRA) Cost Analysis (CA) group, primarily developing Independent Cost Estimates (ICEs) for the Intelligence Community (IC) and conducting research to improve upon current costing methods and tools.