Posted by SAR Data Analysis, CV Benchmarks, and the Updated NCCA S-Curve Tool
Risk Track
RSK05_Presentation_SARDataAnalysisCVBenchmarksandUpdatedNCCASCurveTool_Lee
Abstract:
To support the development of better probabilistic cost estimates, the Naval Center for Cost Analysis (NCCA) has championed the development of the S-Curve Tool, which was well received at both the 44th Annual Department of Defense Cost Analysis Symposium (ADoDCAS) in February, 2011 (1), and the joint Society of Cost Estimating and Analysis (SCEA) / International Society of Parametric Analysts (ISPA) conference in June, 2011. This paper presents ongoing research to support both continued improvement of the S-Curve Tool and greater understanding of the nature of cost growth for major acquisition programs; its mean value (risk) and variability (uncertainty); and the components thereof. The refinement of historical benchmarks presented in the previous paper on analysis of Selected Acquisition Report (SAR) Summaries for Department of the Navy (DON) programs (2), including cost growth factors (CGFs) and coefficients of variation (CVs), enables more realistic estimates and supports better decision-making.
This paper presents the results of extensive data collection, validation, normalization, and analysis using cost variance data from SARs across all Services DoD components. By shifting from the SAR Summaries to the SARs themselves, the authors were able to decompose the previous data, which were at the level of total Acquisition cost with Quantity and Economic adjustments only, into appropriation types – Research, Development, Test, and Evaluation (RDT&E), Procurement, Military Construction (MILCON), and (Acquisition-phase) Operating and Support (O&S) – and all seven SAR Cost Variance categories. We identified and quantified two additional categories, Baseline Adjustments (identified elsewhere in the SAR) and Inter-Phase growth, which occurs when the initial Baseline Estimate of one phase does not match the final Current Estimate of the previous phase. We identified several distinct validation steps to ensure the soundness of the data, and used those steps to identify and resolve any apparent anomalies. In addition to significantly improving the granularity of the data, we more than tripled the number of data points by incorporating Army, Air Force, and DoD-level SAR programs. The data, comprising more than 400 milestone estimates from more than 300 programs, are stored in a Microsoft Access-based relational database in 3rd normal form. This allows thousands of query types (based on any combination of Service, phase, appropriation type, program year, milestone, etc.) to be run quickly without any manual manipulation of data, and ensures referential integrity by storing all data in only one place.
We re-tested previous hypotheses (2) regarding historical cost growth and variability, including tests for differences in CGFs and CVs by commodity, era, and milestone, and examined more closely the decomposition of CGF and CV by Cost Variance category, beyond just the previous Quantity and Economic (Then Year vs. Base Year) adjustments. We also revisited the comparison of the two primary CV data analysis approaches, the CV of CGFs presented in (1) and the size-effect maximum-likelihood estimation (MLE) regression approach presented in (3). For the latter, we introduce standardized residuals based on the heteroscedastic variance model to enable additional hypothesis testing.
The paper includes a brief demonstration of the use of the new expanded benchmarks within the updated S-Curve Tool.
Author(s):
Richard Lee
Technomics, Inc.
Richard C. Lee received his B.S. in Mechanical Engineering in 2007 from Virginia Polytechnic Institute and State University. He joined the Advanced Vehicle Dynamics Laboratory at the same institution and started his graduate studies on modeling and simulation of terrain profile models using numerical methods and vehicle dynamics. He received his M.S. in Mechanical Engineering in the spring of 2009. He is a member of the Society of Cost Estimating and Analysis (SCEA), the American Society of Mechanical Engineers (ASME), and the Institute For Operations Research and the Management Sciences (INFORMS). A Cost Analyst at Technomics, Inc., he has supported the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE), the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE), and the Naval Center for Cost Analysis (NCCA) on projects spanning Earned Value Management (EVM) analysis, data collection and cost research, and risk analysis.
Peter J. Braxton
Technomics, Inc.
Peter J. Braxton holds an AB in Mathematics from Princeton University and an M.S. in Applied Science (Operations Research) from the College of William and Mary. A Senior Cost Analyst and Technical Officer at Technomics, Inc., he is a Certified Cost Estimator/Analyst (CCEA) and currently serves as Director, Body of Knowledge, for the Society of Cost Estimating and Analysis (SCEA). He was named SCEAs 2007 Estimator of the Year for contributions in Education and received both a TASC Presidents Award for Operational Excellence and a Northrop Grumman Corporate Contracts and Pricing Award in 2008. He served as SCEAs Training Chair from 2004 to 2009 and as a Northrop Grumman Technical Fellow from 2006 to 2009.
He currently supports the Naval Center for Cost Analysis (NCCA) on data collection and cost research efforts, and Defense Acquisition University (DAU) on curriculum development. Throughout more than a dozen years in the community, he has worked to advance the state of knowledge of cost estimating and risk analysis, Cost As an Independent Variable (CAIV), and Target Costing on behalf of the Navy Acquisition Reform Office (ARO), the DD(X) development program, and other ship and intelligence community programs. He has conducted independent cost evaluations (ICEs) on all of Northrop Grummans major shipbuilding programs (now Huntington Ingalls Industries), and on several IT systems and services programs. He served as managing editor for development and maintenance of the acclaimed Cost Estimating Body of Knowledge (CEBoK(R)) and its predecessor, Cost Programmed Review Of Fundamentals (CostPROF), and as Training Track chair for the last nine SCEA international conferences. He has taught extensively at government, corporate, and society training events throughout the United States, Europe, and Australia. He is lead author or co-author of over a dozen professional papers on cost, risk, and CAIV, including two SCEA Best Paper winners.
Kevin Cincotta
Technomics, Inc.
Kevin Cincotta is a Technical Officer at Technomics. He was recently awarded the 2011 Estimator of the Year Award for Technical Achievement from the Washington Area Chapter of the Society of Cost Estimating and Analysis (SCEA). His primary areas of expertise are cost analysis, database creation and management, and statistics. He leads projects for the Defense Acquisition University (DAU) and Naval Center for Cost Analysis (NCCA). The projects focus on development and maintenance of training materials for cost analysts at all levels; and analysis of cost growth in time series data relating to contract obligations, respectively. In addition, he serves as Director of the Technomics Training Institute, which trains junior and mid-level costs analysts with the aim of building core knowledge and facilitating professional certification.
From 2003 to 2009, Kevin was a Research at Fellow at LMI (formerly the Logistics Management Institute). He led myriad projects for clients in the Departments of Defense and Homeland Security. These include analysis of cost per flying hour calibration factors for the Air Force Cost Analysis Agency (AFCAA), development of the Program-Budget/Joint Capability Area (P-B/JCA) data structure for the Office of the Secretary of Defense, Capability Assessment and Program Evaluation (OSD CAPE), and various applied estimates and comparative analyses for the Coast Guard Research and Development Center (RDC). He was also a lead instructor for LMIs internal cost estimating and analysis training.
From 2001 to 2003, Kevin served as a Senior Cost Analyst at MCR, LLC. He worked closely with government clients at the Missile Defense Agency (MDA) to develop a radar cost model, which was presented by MCR at the 2004 SCEA conference.Kevin’s DoDCAS and SCEA presentations include Interval Maximum Likelihood Estimation (2005), A Physics-Based Model of Maintenance Demands (2006), Multicollinearity: Coping with the Persistent Beast (2007), Agreements, Planning and Execution System (2008), The Percentile Problem: How Much Is Enough? (2008), New Research in General Error Regression Model (GERM) Significance Testing (2009), Cost Estimating as a Linear Programming Problem (2009), Use of Life Cycle Cost Estimates in OMB 300 Reporting (2010), and Multicollinearity Zero Intercept Regression (2011). More recently, Kevin has focused on coordinating the training for the SCEA/ISPA National Conference, and serves as national Training Chair for SCEA.Kevin also led several cost analysis-related tasks at the New Vectors (formerly Vector Research, Incorporated and the Altarum Institute) from 1997 to 2001. As a Senior Cost Analyst and Systems Developer, he assisted in creating life cycle cost estimates (LCCEs) for myriad DOD projects, including the Standard Procurement System (SPS), the Defense Occupational Health Readiness System (DOHRS), and the Simplified Tax and Wage Reporting System (STAWRS). He created several model questions for the current C/CEA exam, and currently serves at the SCEA Training Chair. Known as Crystal City’s original Nature Boy, Kevin holds a masters degree in economics and philosophy from the London School of Economics and Political Science, and a bachelors in the same fields from the University of Virginia.
Brian Flynn
Technomics, Inc.
Technomics Senior Cost Analyst Brian Flynn is based at the firms Arlington, Virginia, headquarters where he supports national security efforts and, more specifically, the Naval Center for Cost Analysis.
Dr. Flynn has a deep understanding of and strong working relationships with the Naval Center for Cost Analysis; the Naval Systems Commands; the Office of the Secretary of Defense; Army, Air Force and Marine Corps cost analysis organizations; and major cost-analysis entities within the Ministries of Defense in the North Atlantic Treaty Organization. His areas of expertise include:
– Defense planning and capability portfolio analysis
– Weapon system acquisition strategy and cost estimating
– Econometric modeling
– Software cost estimating
– Quantitative risk analysis
– Earned value management
– Defense industrial base analysis
– Economic analysis
– Corporate financial health analysis.
Before joining the firm in 2012, Dr. Flynn worked in the Naval Center for Cost Analysis as a plank owner and more recently as Special Assistant to the Deputy Assistant Secretary of the Navy(Cost and Economics). He served simultaneously as Study Director and U.S. DoD representative of a NATO/Partnership for Peace task group on independent cost estimating and defense planning. His team has been nominated for NATO’s prestigious Scientific Achievement Award.
A recognized analyst and leader in the international defense community, Dr. Flynn received the Department of the Navy’s Distinguished Civilian Service Award, the highest honorary award the Secretary of the Navy can confer on a civilian employee, as well as the Superior Civilian Service Award and the Meritorious Civilian Service Award.
Dr. Flynn has authored or co-authored significant cost-analysis guidelines, studies, and white papers including A Partial-Adjustment Model for Explaining Changes in Overhead Costs, Effects of Competitive Procurement on Weapon System Prices, and Weapon Systems Acquisition Reform Act (WSARA) and the Enhanced Scenario-Based Method (eSBM) for Cost Risk Analysis, all of which were presented at government symposia.
Dr. Flynn holds a Ph.D. in Economics, with distinction, from Georgetown University and M.A. and B.A. degrees in Economics from Virginia Polytechnic Institute and State University.
Most importantly of all, Dr. Flynn has undying pride in his service, over a generation ago, in the United States Marine Corps.
Benjamin Breaux
NCCA
Mr. Breaux works in the Naval Center for Cost Analysis (NCCA), within the Office of the Deputy Assistant Secretary of the Navy for Cost and Economics. As a senior cost analyst, he conducts and supports Independent Cost Assessments (ICAs) and Independent Cost Estimates (ICEs) of Navy ship and weapon systems acquisition programs.
Prior to joining NCCA in 2011, he was employed by Northrop Grumman as a line manager and operations research analyst leading cost estimating support efforts for several Navy and United States Marine Corps (USMC) acquisition programs as well as providing analytical support to OSD and USMC customers. Before joining Northrop Grumman, he was a consulting manager for Robbins-Gioia LLC, providing analytical and estimating support to projects for the Defense Logistics Agency, OSD and USMC customers.
Mr. Breaux holds a Bachelors Degree in Industrial Engineering from the Georgia Institute of Technology as well as a Masters of Science in Operations Research from the Naval Postgraduate School.