Posted by Enhanced Scenario-Based Method for Cost Risk Analysis: Theory, Application, and Implementation
Risk Track
RSK06_Update_eSBM for Cost Risk Analysis_Theory_Application_Implementation
RSK06_Presentation_EnhancedScenarioBasedMethodforCostRiskAnalysis_Flynn
RSK06_Paper_EnhancedScenarioBasedMethodofCostRiskAnalysis_Flynn
Abstract:
There is a growing realization within the cost-analysis community that estimates of cumulative probability distributions of cost, or S-curves, too often understate true, underlying risk and uncertainty. Several organizations cite cases where return program acquisition costs, or actuals, fall at the 99th+ percentile on S-curves estimated years previously. This degree of deviation from the mean is a legitimate possibility for any one acquisition program. After all, there’s no such thing as an “average” program. Variation is expected. However, the frequency of occurrence of outliers, coupled with an apparent disconnect between estimated and historically-based coefficients of variations (CVs) reflected in S-curves, suggests a systemic rather than an isolated problem. To address this problem, this paper presents recent enhancements to the Scenario-Based Method (SBM), the application of which should result in more realistic S-curves for program cost.
Among the pantheon of procedures for performing cost risk and uncertainty analysis, such as Monte Carlo simulation and sensitivity analysis, the enhanced Scenario-Based Method, or eSBM, offers a unique top-level, historically-based, issue-oriented perspective. Sets of risk events, or scenarios, are defined which represent plausible alternatives to the baseline. Using CVs based on historical results of DoD acquisition programs, S-curves are generated and risks assessed. eSBM can be used as a stand-alone technique to support major milestone decisions, or, as a best practice, in conjunction with other methods.
In 2006, the Scenario-Based Method (SBM) was introduced as an alternative to advanced statistical methods for generating measures of cost risk. Since then, enhancements to SBM have been made. These include integrating historical data into SBM’s algorithms and providing a context for applying SBM from a WSARA perspective. Together, these improvements define the enhanced SBM – an historical, data-driven application of SBM.
This paper presents
– eSBM theory;
– Historically-based CVs to employ in generating S-curves, or in checking their accuracy;
– A case study that employs eSBM;
– An illustration of the consequences of using inaccurate S-curves; and
– An easy-to-use tool for generating and comparing S-curves.
In summary, eSBM and the S-curve tool, with benchmark CVs as their foundation, provide invaluable assistance in (1) estimating the risks and uncertainties of major acquisition programs and (2) gauging the accuracy of cost risk assessments, generated by whatever means.
Author(s):
Brian Flynn
Technomics, Inc.
Technomics Senior Cost Analyst Brian Flynn is based at the firms Arlington, Virginia, headquarters where he supports national security efforts and, more specifically, the Naval Center for Cost Analysis.
Dr. Flynn has a deep understanding of and strong working relationships with the Naval Center for Cost Analysis; the Naval Systems Commands; the Office of the Secretary of Defense; Army, Air Force and Marine Corps cost analysis organizations; and major cost-analysis entities within the Ministries of Defense in the North Atlantic Treaty Organization. His areas of expertise include:
– Defense planning and capability portfolio analysis
– Weapon system acquisition strategy and cost estimating
– Econometric modeling
– Software cost estimating
– Quantitative risk analysis
– Earned value management
– Defense industrial base analysis
– Economic analysis
– Corporate financial health analysis.
Before joining the firm in 2012, Dr. Flynn worked in the Naval Center for Cost Analysis as a plank owner and more recently as Special Assistant to the Deputy Assistant Secretary of the Navy(Cost and Economics). He served simultaneously as Study Director and U.S. DoD representative of a NATO/Partnership for Peace task group on independent cost estimating and defense planning. His team has been nominated for NATO’s prestigious Scientific Achievement Award.
A recognized analyst and leader in the international defense community, Dr. Flynn received the Department of the Navy’s Distinguished Civilian Service Award, the highest honorary award the Secretary of the Navy can confer on a civilian employee, as well as the Superior Civilian Service Award and the Meritorious Civilian Service Award.
Dr. Flynn has authored or co-authored significant cost-analysis guidelines, studies, and white papers including A Partial-Adjustment Model for Explaining Changes in Overhead Costs, Effects of Competitive Procurement on Weapon System Prices, and Weapon Systems Acquisition Reform Act (WSARA) and the Enhanced Scenario-Based Method (eSBM) for Cost Risk Analysis, all of which were presented at government symposia.
Dr. Flynn holds a Ph.D. in Economics, with distinction, from Georgetown University and M.A. and B.A. degrees in Economics from Virginia Polytechnic Institute and State University.
Most importantly of all, Dr. Flynn has undying pride in his service, over a generation ago, in the United States Marine Corps.
Peter J. Braxton
Technomics, Inc.
Peter J. Braxton holds an AB in Mathematics from Princeton University and an M.S. in Applied Science (Operations Research) from the College of William and Mary. A Senior Cost Analyst and Technical Officer at Technomics, Inc., he is a Certified Cost Estimator/Analyst (CCEA) and currently serves as Director, Body of Knowledge, for the Society of Cost Estimating and Analysis (SCEA). He was named SCEAs 2007 Estimator of the Year for contributions in Education and received both a TASC Presidents Award for Operational Excellence and a Northrop Grumman Corporate Contracts and Pricing Award in 2008. He served as SCEAs Training Chair from 2004 to 2009 and as a Northrop Grumman Technical Fellow from 2006 to 2009.
He currently supports the Naval Center for Cost Analysis (NCCA) on data collection and cost research efforts, and Defense Acquisition University (DAU) on curriculum development. Throughout more than a dozen years in the community, he has worked to advance the state of knowledge of cost estimating and risk analysis, Cost As an Independent Variable (CAIV), and Target Costing on behalf of the Navy Acquisition Reform Office (ARO), the DD(X) development program, and other ship and intelligence community programs. He has conducted independent cost evaluations (ICEs) on all of Northrop Grummans major shipbuilding programs (now Huntington Ingalls Industries), and on several IT systems and services programs. He served as managing editor for development and maintenance of the acclaimed Cost Estimating Body of Knowledge (CEBoK(R)) and its predecessor, Cost Programmed Review Of Fundamentals (CostPROF), and as Training Track chair for the last nine SCEA international conferences. He has taught extensively at government, corporate, and society training events throughout the United States, Europe, and Australia. He is lead author or co-author of over a dozen professional papers on cost, risk, and CAIV, including two SCEA Best Paper winners.
Dr. Paul Garvey
MITRE; Center for Acquisition and Systems Analysis
Paul R. Garvey is Chief Scientist for the Center for Acquisition and Systems Analysis a Division of The MITRE Corporation. He has nearly 30 years of experience in systems cost analysis, uncertainty analysis, and in the application of risk-decision analytic methods to a variety of problems in the federal government.
He is the author of over 25 peer-reviewed publications and three textbooks: Probability Methods for Cost Uncertainty Analysis, 2000, Analytical Methods for Risk Management, 2008, and Advanced Risk Analysis in Engineering Enterprise Systems, 2012. These works are from Chapman-Hall/CRC-Press, Taylor & Francis Group.
Dr. Garvey has an A.B. and M.Sc. in pure and applied mathematics from Boston College and Northeastern University. He earned a Ph.D. in engineering systems risk analysis and decision theory from Old Dominion University (ODU) and was awarded the doctoral dissertation medal from the faculty of the College of Engineering.
Richard Lee
Technomics, Inc.
Richard C. Lee received his B.S. in Mechanical Engineering in 2007 from Virginia Polytechnic Institute and State University. He joined the Advanced Vehicle Dynamics Laboratory at the same institution and started his graduate studies on modeling and simulation of terrain profile models using numerical methods and vehicle dynamics. He received his M.S. in Mechanical Engineering in the spring of 2009. He is a member of the Society of Cost Estimating and Analysis (SCEA), the American Society of Mechanical Engineers (ASME), and the Institute For Operations Research and the Management Sciences (INFORMS). A Cost Analyst at Technomics, Inc., he has supported the Office of the Secretary of Defense, Cost Assessment and Program Evaluation (OSD CAPE), the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE), and the Naval Center for Cost Analysis (NCCA) on projects spanning Earned Value Management (EVM) analysis, data collection and cost research, and risk analysis.