Posted by Base Realignment and Closure (BRAC) Savings and Acquisition Risk
Risk II Track
R2-1_Presentation_BaseRealignment&ClosureSavings&AcquisitionRisk_Braxton
R2-1_Paper_BaseRealignment&ClosureSavings&AcquisitionRisk_Braxton
Abstract:
The Government Accountability Office (GAO) recently released the study Military Base Realignments and Closures: Updated Costs and Savings Estimates from BRAC 2005 (GAO-12-709R, June 29, 2012). Its appendices contain a wealth of risk data, with initial estimates (2005 BRAC commission) and final costs (Fiscal year 2011 DOD budget) for 175 distinct BRAC initiatives. Applying an innovative method for modeling within-program risk and uncertainty using cross-program data, this paper derives cost growth factors (CGFs) and coefficients of variation (CVs) for BRAC initiatives. Furthermore, the pattern in these data is astoundingly similar to that found in major defense acquisition program (MDAP) data, a strong confirmation of this modeling approach (meta analysis).
The BRAC initiatives ranged in size from very small (a few million dollars or less) to large (a few billion dollars), and they experienced a very wide range of cost growth, from -100% (i.e., a final cost of zero!) up to more than 1800% (19-fold!). The average CGF was 2.02 (102% growth), but median growth (36%) was much smaller, and dollar-weight growth (68%) fell in between. These are typical results, especially in view of the well-documented “size effect,” wherein smaller programs have much greater risk and opportunity (and hence uncertainty). While the GAO report examined how the increased one-time cost reduced the net present value (NPV) of the initiatives, which nonetheless remained positive due to real operational savings, our focus was on that up-front cost itself, as a potential analogy for acquisition programs.
Using the method pioneered in “The Perils of Portability: CGFs and CVs” (Braxton, et al., SCEA/ISPA 2011), we modeled the BRAC data using a maximum likelihood estimation (MLE) regression of final cost as a function of initial cost, with a heteroskedastic error term. This models the size effect by allowing the variation around smaller programs to be smaller in absolute (dollar) terms but larger in percent terms. For example, the BRAC data show a 30% CV at about a half billion dollars, with higher CVs for smaller projects, and asymptotically smaller CVs for larger projects (down to about 10% in the observed range). While we fail to reject the null hypothesis that the normalized errors follow a normal distribution, the eerie and unmistakable similarity of the pattern of normalized errors to that produced by the same model for the entirely distinct MDAP data leads us to investigate a different error form.
Drawing from more than 400 Selected Acquisition Report (SAR) baselines from more than 300 MDAPs as reported in “SAR Data Analysis, CV Benchmarks, and the Updated NCCA S-Curve Tool” (Lee, et al., SCEA/ISPA 2012), we updated the same MLE regression analysis and found that, like the BRAC data, the normalized errors showed a clustering below mean growth and other evidence that a skew-right distribution such as lognormal may be more appropriate. It has long been hypothesized that within-program risk is normal, in consonance with the application of the Central Limit Theorem to probabilistic cost estimates, while cross-program risk is lognormal, due to the presence of a few extremely risky programs and many moderately risky programs. This research offers unprecedented insight into within-program risk, and indications are that it too may be lognormal.
Author(s):
Peter J. Braxton
Technomics, Inc.
Peter J. Braxton is a Senior Cost Analyst and Technical Officer at Technomics, Inc., where he supports the Naval Center for Cost Analysis (NCCA) on data collection and cost research efforts, and Defense Acquisition University (DAU) on curriculum development.
He is a Certified Cost Estimator/Analyst (CCEA) and currently serves as Body of Knowledge Chair for the the International Cost Estimating and Analysis Association (ICEAA). He was named the Society of Cost Estimating and Analysis (SCEA) 2007 Estimator of the Year for contributions in Education. He is the managing editor for development and maintenance of the acclaimed Cost Estimating Body of Knowledge (CEBoK(R)) and its predecessor, Cost Programmed Review Of Fundamentals (CostPROF). He served as SCEA’s Training Chair from 2004 to 2009 and as Training Track chair for nine consecutive SCEA international conferences. He has taught extensively at government, corporate, and society training events throughout the United States, Europe, and Australia.
He holds an AB in Mathematics from Princeton University and an M.S. in Applied Science (Operations Research) from the College of William and Mary. He is lead author or co-author of over two dozen professional papers on cost, risk, and Cost As an Independent Variable (CAIV), including three SCEA Best Paper winners and a Journal of Cost Analysis and Parametrics article.
Kevin Cincotta
Technomics, Inc.
Kevin Cincotta is a Technical Officer and Division Operations Manager at Technomics, Inc. His primary areas of expertise are cost analysis, database creation and management, and statistics. Mr. Cincotta leads or contributes to projects for the Defense Acquisition University (DAU), Department of Homeland Security (DHS), and Deputy Assistant Secretary of the Army for Cost and Economics (DASA-CE). In addition, Kevin created and serves as Director of the Technomics Training Institute, which trains junior and mid-level costs analysts with the aim of building core knowledge and facilitating professional certification.
Prior to joining Technomics, Mr. Cincotta was a Research Fellow at LMI (formerly the Logistics Management Institute). He led myriad projects for clients in the Departments of Defense and Homeland Security. These include analysis of cost per flying hour calibration factors for the Air Force Cost Analysis Agency (AFCAA), development of the Program-Budget/Joint Capability Area (P-B/JCA) data structure for the Office of the Secretary of Defense, Capability Assessment and Program Evaluation (OSD CAPE), and various applied estimates and comparative analyses for the Coast Guard Research and Development Center (RDC). He was also a lead instructor for LMI’s internal cost estimating and analysis training.
Mr. Cincotta is an International Cost Estimating and Analysis Association (ICEAA)-Certified Cost Estimator/Analyst (C/CEA). He created several model questions for the current C/CEA exam, and currently serves at the ICEAA Chapter and Regional Training Chair. Kevin holds a master’s degree in economics and philosophy from the London School of Economics and Political Science, and a bachelor’s in the same fields from the University of Virginia.
Richard Lee
Technomics, Inc.
Richard C. Lee received his B.S. in Mechanical Engineering in 2007 from Virginia Polytechnic Institute and State University. He joined the Advanced Vehicle Dynamics Laboratory at the same institution and started his graduate studies on modeling and simulation of terrain profile models using numerical methods and vehicle dynamics. He received his M.S. in Mechanical Engineering in the spring of 2009. He is a member of the Society of Cost Estimating and Analysis (SCEA), the American Society of Mechanical Engineers (ASME), and the Institute For Operations Research and the Management Sciences (INFORMS). A Cost Analyst at Technomics, Inc., he has supported the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE), the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE), and the Naval Center for Cost Analysis (NCCA) on projects spanning Earned Value Management (EVM) analysis, data collection and cost research, and risk analysis.