Acquisition Track
Track Chairs: Maureen (Mo) Deane, Rick Garcia
Download All Acquisition Track Papers & Handouts (.zip)
Observation of Production Rate Effect Observed in Historical Weapon Systems Cost
Acquisition Track (AC01)
William Banks
Timothy Lawless
Production Rate Effect has been observed in several weapon systems program’s historical cost data during the procurement phase of the acquisition life-cycle. The main contribution of this paper is to illustrate real life observations of production rate effect and the economies of scale shown when purchasing end items from major defense contractors.
Keywords: Budgeting, Data Collection, DoD, Government, Regression
Federal Agency Independent Government Cost Estimate (IGCE) Starter Kit
Acquisition Track (AC03)
Cassandra M. Capots
Lauren Nolte
Lavanya Yeleswarapu
To achieve mission success in today’s constrained fiscal environment, federal agencies must establish a fair and reasonable cost baseline to make informed decisions. This baseline is commonly called an Independent Government Cost Estimate (IGCE). We have developed a customizable process that forms a starter kit for any agency that wishes to quickly and effectively perform IGCEs. We discuss best practices, lessons learned, and implementation tips for both DoD and civilian agencies.
Keywords: Budgeting, DoD, Early Cost, Government, International, Methods, Program Management, Acquisition; Applied Estimating; Decision Support; Independent Government Cost Estimates; Starter Kit; Source Selection
Early Stage Cost Estimating for Radars and Sensors
Acquisition Track (AC04)
Jeremy Goucher
The majority of costs for programs are locked in even before a program enters production, which makes accurate early stage cost estimates vital for effective resource management and program success. This study proposes a method for analyzing combat system cost prior to a complete requirements description. The data set includes old, new, big, small, ground, sea, air, domestic, and foreign systems. The result is a model that requires limited data and is widely applicable.
Keywords: Budgeting, DoD, Early Cost, Government, Life Cycle
The ABCs of the UCF: A look at Section L and Section M
Acquisition Track (AC05)
Jennifer Leotta
This paper will examine the government cost analyst’s role in the acquisition/contracting process. Specifically, it will look at the Uniform Contract Format (UCF) and how to create effective language for Section L that provides clear directions for offerors and includes all information necessary to complete a thorough proposal evaluation. It will also look at developing Section M language for varying contract types (cost plus, fixed price, etc.) that is consistent with the Federal Acquisition
Keywords: Communication, Data Collection, Government, Contracting, Acquisition
GAO-COFC Concurrent Bid Protest Jurisdiction: Are Two Fora Too Many?
Acquisition Track (AC06)
James Linick
The Government Accountability Office (GAO) and the Court of Federal Claims (COFC) are the bid protest fora where a disappointed offeror may challenge an agency’s procurement decision. Critics claim that concurrent bid protest fora, which have differences in jurisdiction, standards of review and remedies, creates inefficiency in the procurement system, e.g. serial protests. This presentation examines whether concurrent bid protest fora are healthy for the US federal procurement system.
Keywords: DoD, Government, Bid, award protest, GAO, COFC, federal government procurement system
A Weapons Acquisition Case Study: Cost Overruns and Schedule Slips
Acquisition Track (AC07)
Randy Bowen
David L. Peeler, Jr.
This story is true. Certain details have been omitted to “stump the audience” and protect the guilty. The trials and tribulations of an acquisition program cost overrun are explored for persistent lessons and future applications. This saga includes materiel difficulties, workforce issues, congressional interest, funding and schedule perturbations. The aim is to educate, as the persistence of fundamental acquisition problems and the insignificance of acquisition reforms are framed.
Keywords: DoD, Government, Labor, Learning Curves, Manufacturing, Program Management, Risk, Scheduling
Business Case Analysis Track
Track Chairs: Melvin R. Etheridge, Jr., Kammy Mann
Download All Business Case Analysis Track Papers & Handouts (.zip)
Bridging the Gap Between Capital Investment Valuation in Private & Public Sector
Business Case Analysis Track (BC01)
George Bayer, Jr.
The public and private sectors have different valuation techniques and considerations when evaluating capital budgeting decisions. Understanding the difference between private and public sector capital investment analysis (discounted cash flow, cost/benefit, tax impacts, stakeholders) helps decision-makers make better informed decisions. In the cost estimating community, understanding investment distinctions makes us better stewards of information and more effective Finance professionals.
Keywords: Budgeting, Cost/Benefit Analysis, Government, Methods, Modeling, Monte Carlo, Risk, Variables, Valuation, taxes, cost of capital, discount rates, stakeholders, discount rates
ROS vs. ROC: Government Can Lower Price by Raising Fee
Business Case Analysis Track (BC02)
Sandy Burney
Many Government Contractors measure profitability using Return on Sales (ROS), which includes Fee in the denominator. The FAR regulates fee percentages or Return on Cost (ROC). In negotiating sole-source contracts, this metric difference can result in the Contractor negotiating a higher risk adjusted cost and price to offset lower fees offered by the Government. The briefing provides a mathematical example of the tradeoff between Cost and Fee for a CPFF contract.
Keywords: Government, Performance Management, Risk, Contract Type; FAR; Fee
Fixing the Flawed Nature of DoD BCA’s
Business Case Analysis Track (BC04)
William G. Williamson
The government is not a business that is profit-oriented. Monetary business metrics as a means of determining what items to consume are inherently flawed for the government since metrics such as Return on Investment’s assume the organization is a provider of the goods or services, not a consumer. A better approach for the government would be to measure the areas that the government most values, i.e. schedule and requirements. Using those areas as anchors, a best value metric can be developed, pointing the government to smarter acquisition decisions.
Keywords: Cost/Benefit Analysis, DoD, Early Cost
2016 Best Paper: Acquisition & Business Cases Category
3D Printing in the DoD; Employing a Business Case Analysis
Business Case Analysis Track (BC05)
Nicole Santos
Richard Shea
Robert Appleton
3D Printing is the family of technologies that enables users to produce items on demand from CAD files or 3D scanning. The potential benefits to military logistics include cost savings, weight reduction, and responsiveness to the warfighters’ needs. To demonstrate and measure the benefits in the Department of Defense (DoD), a rigorous Business Case Analysis (BCA) will identify benefits and challenges to implementation including evaluating its costs, risks, and benefits.
Keywords: Cost/Benefit Analysis, DoD, Manufacturing, 3D Printing, Business Case Analysis
Training Cost Analysts, a Cohesive Pedagogical Framework for Success
Business Case Analysis Track (BC06)
Kammy Mann
Danielle Lainez
How do we train cost analysts? This paper will take a critical look at peer-reviewed academic research regarding pedagogy, or the method of teaching. Leveraging existing knowledge, current best practices and resources available, a framework for teaching and training will be proposed to the cost community. By establishing a consistent methodology for educating our professional workforce we can ensure that new members of the cost analysis field have the tools and skills to succeed.
Keywords: Training, Pedagogy, Teaching, Education, Professional Workforce Development
Analyses of Alternatives for Space Systems
Business Case Analysis Track (BC07)
R. Alex Wekluk
This presentation provides an overview of recent experiences performing satellite analyses of alternatives, including technical and cost trades. Our team developed a scalable mechanism to perform satellite sizing using physics and stochastic modeling. Once modeled, an array of costing methods is applied to develop repeatable and defensible estimates. The team made great strides in presenting the large number of nuances to decision makers.
Keywords: Cost/Benefit Analysis, DoD, Government, Life Cycle, Methods, Modeling, Parametrics, Space, Analysis of Alternatives
Begin With a Joke? What John Oliver Can Teach Us About Communicating Cost Estimates
Business Case Analysis Track (BC08)
Erin K. Barkel
Are you having a hard time holding management’s attention when you explain the results of your estimates? It’s time to change your approach. Consider this: millions of people tune-in and click to watch John Oliver talk about cost estimates gone wild, and they stay tuned-in for 20 whole minutes. What can you learn from a comedian about explaining cost estimates? Maybe you should start with a joke.
Management, EVM & Scheduling Track
Track Chairs: William S. Barfield, Rodney Olin
Download All MS Track Presentations and Handouts (.zip)
Organization Cost Estimation as a Human Intensive Systems Engineering Problem
Management, EVM & Scheduling Track (MS01)
David Bloom
Danny Polidi
Robert Wright
This paper will compare and contrast the difference between Intensive and Non-Intensive Human Interactive Systems. Further, this paper will investigate the role of Governance, Culture, Process and Tools (GCPT) in each of the 2 systems, Intensive and Non-intensive. Finally, this paper will apply the GCPT Framework to the system of Organizational Cost Estimation and provide examples of results that Raytheon has seen in its effort to improve the accuracy, affordability and accountability of the current bidding/estimation process.
Conducting Root Cause Analysis for Should Cost Management
Management, EVM & Scheduling Track (MS02)
Nancy A. R. Droz
Robert L. Hernandez
Root-cause analysis (RCA) is typically performed after a technical failure or after a program anomaly is found. However, the technique is key to the development and selection of effective initiatives to reduce program cost. RCA is a critical aspect of successful Should Cost Management execution. The presentation will provide a how to approach of applying root cause analysis during the evaluation of program data to develop more effective costs reductions. Lessons learned will be presented.
Keywords: Cost/Benefit Analysis, DoD, Government, Program Management, Risk, Should Cost
EVM’s Potential for Enabling Integrated Cost-Risk Management
Management, EVM & Scheduling Track (MS03)
David R. Graham
Bryn Turner
Integrating the cost estimating, EVM analysis and technical risk management disciplines to realize comprehensive cost-risk management has proven elusive. Recently developed EVM cost-risk tools offer the potential to successfully enable integrated cost-risk management. The paper will illuminate the process of how integrated cost-risk management can be realized through cooperation of cost estimating, EVM analysis and technical risk management disciplines and briefly describe these new EVM cost-risk tools.
Keywords: Data Collection, DoD, Early Cost, EVM, Government, Microsoft Excel, Monte Carlo, Performance Management, Program Management, Regression, Risk, Space
Applying Earned Value to Agile Development Programs
Management, EVM & Scheduling Track (MS04)
Bob Hunt
Michael Thompson
Agile Software and Agile Development continue to dominate the Federal acquisition arena. This presentation will address current trends in estimating and applying Earned value to Agile programs.
Keywords: Agile, EVM, Performance Management, Scheduling, Story Points
Provide Agile Analysis to Stakeholders Using Category Tagging Approach
Management, EVM & Scheduling Track (MS05)
Eric Hong
It is becoming more and more important for organizations to use data analytics for improved decision-making to create a competitive advantage and/or operate efficiently. This paper will describe the challenges in creating and implementing a strategy. It will identify potential reporting requirements and how to leverage existing work products as input into a standard structure feasible to perform analysis for different stakeholders. The proposed solution will increase the accuracy and agility of an organizations’ analysis.
Keywords: Program Management, Data Analytics, Budgeting
2016 Best Paper: Management, EVM & Scheduling Category
Putting Schedule Quality Checks to the Test
Management, EVM & Scheduling Track (MS06)
Eric M. Lofgren
Analysts often use the 14-Point Assessment for Schedule Quality as a benchmark for determining the overall reliability of a schedule. But how much of the variation in schedule error can be explained by the results of the 14-Point check? This paper will use actual defense contract data to find the correlates of schedule reliability, measured using both the accuracy and the timeliness with which the schedule slip is predicted.
Keywords: EVM, Program Management, Regression, Scheduling, Statistics
Preventing Program Management Pitfall Using Portfolio Estimating
Management, EVM & Scheduling Track (MS07)
Nick Morales
Christopher Dewberry
Estimating the effort, time, and resources needed to complete project activities is one of the most challenging tasks that project managers must face. Along with the inherent uncertainty associated with managing these activities, additional management uncertainty occurs when there are multiple incremental efforts associated with that project. In order to handle the complication of managing several program increments, each with their own budget lines, portfolio estimating is a method that helps to provide program managers with a fully comprehensive overview of the resource requirements across multiple projects. Program managers can use the portfolio estimate as a tool to help make informed management decisions at the appropriate level and identify how those decisions impact the entire program.
Cost Consistency and Completeness as an Impossible Exercise
Management, EVM & Scheduling Track (MS08)
David L. Peeler, Jr.
In previous papers, Hilbert’s Problems were used to propose and revisit Hilbert’s Problems for Cost Estimating. This paper employs Gödel’s theorem w/respect to Hilbert’s application onto cost. What can we learn about ourselves as estimators and where can we exert greatest impact with our estimates? Using Godel’s two theorems of undecidability as catalyst, we explore the effect and utility of exacting math and other notions on cost estimates specifically and programmaitics generally.
Keywords: DoD, Government, Learning Curves, Modeling, Monte Carlo, Parametrics, Regression, Risk, Statistics
The Performance Metrics Model and Study (PMMS)
Management, EVM & Scheduling Track (MS09)
Derreck Ross
Haitham Ghannam
Richard C. Lee
The Performance Metrics Model and Study (PMMS) is a comprehensive technique for selecting a performance metric TCPI. The PMMS uses historical data of completed products to determine the best EVM estimation technique for similar products that are in progress. The result is an auditable and repeatable approach that increases the objectivity of an Estimate at Completion. This approach can be applied across any program that has historical data.
Keywords: DoD, EVM, Methods
Fiscal Sustainability of Canada’s National Defence Program
Management, EVM & Scheduling Track (MS10)
Peter Weltman
This report provides PBO’s estimate of the fiscal gap between the status quo budget allocations and the cost of sustaining Canada’s status quo national defence forces. Parliamentarians may wish to examine scenarios that will reduce or eliminate the gap between the cost of maintaining the current force structure and the amount of funding being allocated to paying for it.
Keywords: Budgeting, Government, Parametrics, Program Management
Liars! Why Cost Estimators and Budgeters Inflate and Underestimate Costs!
Management, EVM & Scheduling Track (MS11)
Travis Winstead
Ann Hawpe
Projects and programs regularly suffer from poor cost estimating techniques that result in insufficient budgets and unanticipated cost overruns. To issue a contract or set a realistic budget, the cost analyst must verify and validate the mountain of cost information and determine a fair and reasonable price for a contractor’s goods and services. This paper will discuss strategies to assist program analysts in reviewing cost estimates and strategies used to support Government negotiations.
Keywords: Government, Labor, Learning Curves, Manufacturing, Microsoft Excel, Modeling, Operations
Methods & Models Track
Track Chairs: Brandon Bryant, Ruth Dorr, Marc Greenberg
Download All Methods & Models Track Papers & Handouts (.zip)
Department of Defense Commercial-Off-The-Shelf (COTS) Pricing Analysis
Methods & Models Track (MM01)
Heather L. Brown
NCCA partnered with DoD Enterprise Software Initiative (ESI) to periodically receive commercial-off-the-shelf (COTS) software orders with the intent of improving software cost estimating. As of today, more than 250,000 orders from the Army, Navy, Marine Corps, and other federal entities have been recorded. Together, these represent over six billion dollars in sales. NCCA examines this data and categorizes various product descriptions as licensed products, maintenance and support, or services in order to present statistical trends.
Keywords: Data Collection, DoD, Government, IT, Labor, Methods, Modeling, Parametrics, Program Management, Statistics, Commercial-Off-The-Shelf (COTS), Enterprise Software Initiative (ESI)
Naval Sea Systems Ship Command Logistics Requirements Funding Summary Tool
Methods & Models Track (MM02)
Jeremy Eden
The United States Navy has focused on planning and estimation of operations and support costs for programs. A task force created by Secretary Stackley (Assistant Secretary Navy Research, Development & Acquisition) to identify actions for achieving this goal recommended creating a logistics tool similar to that developed by the United States Marine Corps (USMC). This paper demonstrates the tool, development approach, cost estimating methodologies, interface, and deployment to the next generation
Keywords: Budgeting, Data Collection, DoD, Early Cost, Government, Life Cycle, Microsoft Excel, Modeling, Operations, Tool, logistics cost, support cost, visual basic
Footprints in the Sand: A Conversational Approach to Basis of Estimate (BOE)
Methods & Models Track (MM03)
Frank R. Flett
The Basis of Estimate (BOE) is the mechanism by which the cost estimator or cost proposal preparer hopes to convince the evaluator or reviewer that the method he/she took in preparing the estimate was “reasonable” (not the same as “correct”). This paper will present a technique of writing BOEs as if they were a part of a conversation with the evaluator/reviewer, who is, after all, a human being who will early on develop a distinct impression of the “goodness” of the end result.
Keywords: Cost Proposal Preparation
Estimating the Cost of the Aegis Flight III Test Platform
Methods & Models Track (MM04)
Jeremy Goucher
The Flight III upgrade to the Aegis Destroyer program brings unparalleled new capabilities to the surface Navy fleet which will require in depth testing prior to deployment. For live fire testing, the Navy is considering four unique options to procure, install, and test the required systems on an unmanned vessel. This paper discusses the available data, methodology, and results of the cost estimate. The paper will also discuss some of the technical challenges associated with each option.
Keywords: Budgeting, Cost/Benefit Analysis, Data Collection, Government, Methods, Modeling, Program Management, Risk, Scheduling
2016 Best Paper: Methods & Models Category
Beyond Anderlohr: An Alternate Approach To Estimating Production Breaks
Methods & Models Track (MM05)
Brent Johnstone
Estimating the cost impacts of production breaks has long been problematic. Use of the Anderlohr method is widely accepted, but requires a significant degree of estimating judgment and can produce substantially different answers based on the individual user’s assumptions. This paper suggests an alternate empirical methodology based on recent research on organizational learning and forgetting.
Keywords: Anderlohr method, learning curve, methods, modeling, organizational learning, production breaks, regression analysis
Delphi Methodology for Cost Estimating
Methods & Models Track (MM06)
Cole J. Kupec, II
The Delphi methodology can be a practical and valuable tool for cost estimators. This paper explores its proper implementation based on current research to include group selection, question formulation, statistical analysis, and the survey process. Special attention is paid to tailoring the Delphi method to cost estimating applications. A case study is presented for example purposes. The history of the Delphi method as well as modifications to the traditional Delphi method are also presented.
Keywords: Data Collection, Methods
Beyond CSDRs: Collecting Contractor Cost Data for Detailed Cost Estimates
Methods & Models Track (MM07)
Tim Lawless
Quality cost estimates are based on actual costs, and for the Department of Defense, usually are acquired through Cost and Software Data Reports (CSDRs). However, CSDRs may not be available, complete or consistent for quality analytical purposes. This paper examines the collection and use of detailed contractor actual costs for estimates, such as with Priced Bills of Material and invoices, and their analytical benefits, perhaps supplanting CSDRs as the preferred data source for cost estimators.
Keywords: Data Collection, DoD, Manufacturing, Methods
Using Physics Based Reliability Methods for Should Cost Estimates
Methods & Models Track (MM08)
Patrick K. Malone
Value based should cost methodologies supporting Better Buying Power tenants can help realize significant MDAP cost savings over the life cycle. The use of Physics Based Reliability techniques in program development is described to meet affordability goals, identify cost drivers affecting reliability, availability and maintainability of weapon systems, and demonstrate PBR advantages over the classical reliability growth model with actionable plans to maintain technological superiority.
Keywords: Cost/Benefit Analysis, DoD, Early Cost, Government, Life Cycle, Reliability, Better Buying Power
Workload-Adjusted Labor Cost Model
Methods & Models Track (MM09)
Michael Mender
Government estimators typically use Forward Pricing Rate Agreements (FPRAs) to estimate future labor costs. FPRAs make assumptions about future workload, and can be inaccurate as realized workload often differs from projections. To enable understanding of this uncertainty, the Naval Center for Cost Analysis (NCCA) developed a lagged regression model that predicts future overhead costs as a function of workload, allowing NCCA to quickly and accurately assess the impact of Navy procurement decisions on acquisition programs.
Challenges in Estimating the Development Cost of Microcircuit Technology
Methods & Models Track (MM10)
Christopher Price
Gurney Thompson
Electronic Technology continues to change at a rapid pace. Significant changes in the level of integration of electronic components dictate that circuit designs using these parts be redesigned every three to five years. Estimating the development cost of these new designs can be challenging. This paper will discuss these challenges in more detail, and describe some methods and tools available to rapidly and accurately estimate the development, production and O&S cost of these new technologies.
Keywords: DoD, Early Cost, Government, Life Cycle, Manufacturing, Methods, Modeling, Parametrics, Microcircuits, Electronic, Technology, Development, Production, Operation and Support
Estimating the Costs of Future Air-to-Ground Weapons
Methods & Models Track (MM11)
Tom Sanders
The weapons acquisition community has been acquiring air-launched weapons in a slowly-evolving manner for years. While technologies experience great leaps forward, acquisition strategies evolve at a slower pace. The AF Research Lab is evaluating revolutionary weapons designs that facilitate newly-emerging acquisition strategies…thereby presenting real challenges to cost estimators. This paper documents techniques for estimating costs of those future weapons.
Keywords: Cost/Benefit Analysis, Infrastructure, Parametrics, Defense Industrial Base
The Space Situational Awareness (SSA) Cost Model
Methods & Models Track (MM12)
James Smirnoff
Brennen Woodruff
Today there are over a thousand operating satellites orbiting the earth providing a variety of critical functions such as communications, navigation, weather sensing, reconnaissance, and astronomy. As the space environment becomes more crowded, Space Situational Awareness (SSA) is increasingly important. This paper describes a new tool called the Space Situational Awareness (SSA) Cost Model which provides an objective and data-driven framework to estimate the cost of ground-based electro-optical components of future SSA architectures.
Keywords: Cost/Benefit Analysis, Data Collection, DoD, Government, Methods, Microsoft Excel, Modeling, Monte Carlo, Parametrics, Regression, Risk, Space, Statistics
Crew and Space Transportation Systems Cost Model (CASTS)
Methods & Models Track (MM13)
Richard Webb
As part of the MSFC Engineering Cost Office’s new Program Cost Estimating Capability (PCEC) suite of cost estimating tools and capabilities, we are developing the Crew and Space Transportation Systems Cost Model (CASTS), a new, unique cost model for use in estimating crew and space transportation systems. This paper will provide an overview of the capabilities, estimating approach, historical database, and key features of CASTS as well as plans for future improvements.
Keywords: Modeling, Parametrics, Space
Parametrics Track
Track Chairs: Ann Hawpe, Danny Polidi
Download All Parametrics Track Papers & Handouts (.zip)
Multiple Regression: Prediction & Confidence Intervals Demystified
Parametrics Track (PA02)
Stacy M. Dean
We all know the value of prediction/confidence intervals as a measure of the certainty in the best fit line for a regression equation; however, once we get past bivariate regressions how to actually calculate these statistics gets a little murky. This paper attempts to de-mystify multivariate confidence/prediction interval calculation as well as provide some fun facts on the origin and history of these infamous intervals.
Keywords: Microsoft Excel, Modeling, Parametrics, Regression, Statistics, Uncertainty
How Predictive Analytics is Improving Parametric Cost Estimation
Parametrics Track (PA03)
Anthony A. DeMarco
Predictive analytics is defined as the use of data, statistical algorithms and machine-learning techniques to identify the likelihood of future outcomes based on historical data. This paper will explore the history and parallels of parametric estimating and predictive analytics. It will highlight how cost management professionals can use current predictive analytics tools, methods and talent to improve their estimates and their success rates.
Keywords: Methods, Modeling, Parametrics, Regression, Statistics, Statistics analytics
The Collinearity Kill Chain: Detect, Classify, Localize, Neutralize
Parametrics Track (PA04)
Brian Flynn
Adam James
This paper examines the issue of multicollinearity from the fresh perspective of a statistical warfighter. A kill-chain of steps is offered to defeat this illusive foe. Heuristics, chi-squared tests, t-tests, and eigenvalues are the Aegis, Predator, Bradley, and Ma Deuce 50-caliber statistical equivalents used to detect, classify, localize, and neutralize the problem.
Keywords: DoD, Parametrics, Regression, Statistics
Dialing for Dollars: Improving Cost Estimating Accuracy Through Data Mining
Parametrics Track (PA05)
Brittany Holmes
James Glenn
Dr. Christian Smart
This paper shows the benefits of leveraging obligation and expenditure data from DoD Enterprise Resource Planning (ERP) Systems to supplement cost data from CCDRs or IPMRs. One benefit is that cost trends and funding patterns can be analyzed with more precision because the ERP data is updated real-time vice monthly or annually. Another benefit is that more comprehensive CERs can be created because ERP systems capture all cost vice only contracts with cost reporting CDRLs.
Keywords: Budgeting, Data Collection, Government, Methods, Microsoft Excel, Parametrics, Regression, Data Mining
Develop PRESS for Nonlinear Equations
Parametrics Track (PA06)
Shu-Ping Hu
Predicted residual sum of squares (PRESS) and Predicted R2 statistics are commonly used in regression analysis outside the cost analysis industry to determine (1) how well the model predicts new observations and (2) if the model includes more independent variables than necessary. Currently, these two leave-one-out statistics are only available for ordinary least squares. This paper develops these two statistics for nonlinear equations and demonstrates their value in practical applications.
Keywords: Methods, Parametrics, Regression, Statistics, Variables, Predicted Residual Sum of Squares (PRESS), Predicted R-Square, Cross-Validation, Leave-One-Out, Leverage Value, Hat Matrix
Including Escalation in Cost Estimating Relationships
Parametrics Track (PA07)
Christopher Jarvis
Prior to developing a Cost Estimating Relationship (CER) the data should be normalized for effects not explained by the assumed CER model. This typically includes the effects of inflation and/or escalation which has recently generated considerable discussion within the DoD. In this paper we demonstrate two common CERs models and augment them to include an escalation term. Discussion will be made regarding the reliability of the solutions of the augmented models.
Keywords: DoD, Learning Curves, Modeling, Regression, Escalation
2016 Best Paper: Parametrics Category
2016 Best Paper: Overall
Dangers of Parametrics
Parametrics Track (PA08)
Andy Prince
What if our models are not solving our estimating problems, but instead are the source of our problems? The purpose of this paper is to address this question. We will look at what a cost model is, and what it isn’t. We will examine how cost models appeal to our need for certainty and helps us create a good story for our cost estimate. We will take a look at a simple cost model to see what can go wrong when we trust the model over trusting the data. Finally, we will identify specific actions.
Keywords: Methods, Modeling, Parametrics, Regression, Statistics
Development of an Additive Manufacturing Cost Model (PA09)
F. Gurney Thompson, III
Grady Noll
This paper discusses our research and development of a cost model for additive manufacturing (AM), also known as 3D printing. Working with academia and multiple AM companies, we investigated cost drivers, collected cost and technical data, derived AM CERs, and developed an approach for including AM components within a larger parametric cost estimate. This paper discusses the research approach, data collection, results, cost model implementation, and future work.
Keywords: 3D Printing, Additive Manufacturing, Manufacturing, Parametrics, Modeling
X-Planes to X-Wings ~ Developing a Parametric Cost Model
Parametrics Track (PA10)
Steve Sterk
In today’s cost-constrained environment, NASA needs an X-Plane database and parametric cost and schedule model that can quickly provide rough order of magnitude predictions of cost and schedule from initial concept to first flight of potential X-Plane aircraft. This paper takes a look at the steps taken in developing such a model and reports the results. The challenges encountered in the collection of historical data and recommendations for future database management are discussed.
Keywords: Data Collection, Government, Learning Curves, Manufacturing, Methods, Microsoft Excel, Modeling, Monte Carlo, Parametrics, Regression, Statistics, Predictive Analysis
Integrating Cost Estimating and Data Science Methods in R
Parametrics Track (PA11)
Josh Wilson
Laura Barker
Data science is a growing, interdisciplinary field about the extraction of knowledge from data. It emphasizes the utilization of statistics and advanced analysis techniques to answer questions in a well-documented and reproducible manner. R is a free, open source programming language that is popular among data scientists. This presentation discusses the benefits of using R to supplement traditional cost estimating and data analysis practices.
Keywords: Data Collection, Methods, Microsoft Excel, Modeling, Data Science, R, Cost Estimating
AoAs the Right Way
Parametrics Track (PA12)
Amanda Wilson
David Macdonald
Mariam Uzunyan
The follow-ons to two large space acquisition programs from the Space and Missile Systems Center completed the AOA processes last year. They were set apart from past AoAs and AoA-like studies with the leadership and involvement of OSD CAPE and the AFCAA teams, with broad community involvement strengthening the teams. This change resulted in benefits, challenges and lessons learned; various visual tools were created to highlight risks, opportunities, and investment strategies to senior leaders.
Keywords: Cost/Benefit Analysis, DoD, Early Cost, Government, Life Cycle, Risk, Space, AFCAA, Analysis of Alternatives, CAPE
Cost Estimating Relationship Development Handbook
Parametrics Track (PA13)
John S. Fitch
Adam James
Alfred Smith
Developing appropriate and well-defined parametric relationships is a foundational requirement for Cost Estimators. Current training for analysts focuses on various ordinary least squares regression (OLS), techniques, but ignores logical decision sequences employed by good analysts applying OLS and omits methods beyond OLS. The CER Handbook includes these techniques in a flow-chart format and guides analysts along each step in developing a parametric relationship.
Keywords: Methods, Microsoft Excel, Modeling, Monte Carlo, Operations, Parametrics, Regression, Risk, Statistics, Regression, Cost Estimating Relationships, CERs
Risk & Uncertainty Track
Track Chairs: Karen Mourikas, Donna Nguyen
Download All Risk & Uncertainty Track Papers & Handouts (.zip)
Cost Risk for Firm Fixed-Price Contracts
Risk & Uncertainty Track (RU01)
David Biron
Christian Smart
The terms “cost risk” and “firm fixed-price contracts” seem contradictory. By design the contractor bears all risk under a firm fixed-price (FFP) contract. In spite of this, overruns often occur, and contractors have recourse to a Request for Equitable Adjustment (REA) when cost grows beyond the contract value. We present statistics on cost overruns for FFP contracts, some of which are significant, and show how to model risk for such contracts.
Keywords: DoD, Government, Program Management, Risk, Statistics, Cost Growth, Contracts
Modeling Prediction Intervals using Monte Carlo Simulation Software
Risk & Uncertainty Track (RU02)
James Black
Qing Qing “Q” Wu
The use of a prediction interval (PI) is a simple method of quantifying risk and uncertainty for a Cost Estimating Relationship (CER) derived from an Ordinary Least Squares (OLS) regression. Yet, few cost estimators implement PIs in their estimates despite their frequent use of CERs. This presentation will provide a step-by-step tutorial for modeling a PI for an example CER using Monte Carlo Simulation software and will identify the beneficial impact on the coefficient of variation (CV).
Keywords: DoD, Government, Methods, Microsoft Excel, Modeling, Monte Carlo, Regression, Risk
2016 Best Paper: Risk & Uncertainty Category
Introducing RIFT to Protect Your Uncertain Schedule
Risk & Uncertainty Track (RU03)
Nicholas DeTore
Peter Frederic
There are industry-accepted methods for allocating cost risk and uncertainty analysis results to detailed WBS elements; schedule results cannot be allocated the same way since duration behaves differently than cost. We present an innovative solution to this issue. The RIFT algorithm calculates a threshold date, for any task or milestone, that if exceeded puts the probabilistic project finish date in jeopardy. RIFT provides a new tangible metric to guide decision makers.
Keywords: Program Management, Risk, Scheduling, Statistics
Risk vs. Uncertainty – What’s the difference?
Risk & Uncertainty Track (RU04)
Melvin R. Etheridge, Jr.
After years of applying “risk”, many in the profession still do not understand the difference between program risk and cost uncertainty in cost estimates. Determining confidence levels goes beyond simply applying the effects of programmatic risks. Simply put, risks have a probability of occurrence, while we know that quantitative inputs have some degree of uncertainty and must be treated as random variables. This paper explores analytical methods in Excel-based cost models using Monte Carlo s
Keywords: Data Collection, Monte Carlo, Risk, Cost Uncertainty
A ‘Common Risk Factor’ Method to Estimate Correlations Between Distributions
Risk & Uncertainty Track (RU05)
Marc Greenberg
A ‘common risk factor’ method uses expert-derived risk factors to estimate correlation between two distributions. The premise is to estimate mutual information among risk factors that underlie each distribution. Approximation of such mutual information leverages properties of the joint probability distribution of a unit square. Geometric outputs of each pair of common random variables are compared to estimate common risk factor “intersections” that are, in turn, proxies of correlation.
Keywords: Bias, Government, Life Cycle, Methods, Microsoft Excel, Modeling, Monte Carlo, Parametrics, Risk, Scheduling, Statistics, Variables, Correlation, Correlation Coefficient, Covariance, Variance, Risk Factors, Uncertainty, Expert Opinion, Expert Judgment, Expert Elicitation, Triangular Distribution, Random Variables, Probability, Probabilistic, Joint Probability Distribution, Pearson, Spearman
How Regression Methods Impact Uncertainty Results
Risk & Uncertainty Track (RU06)
Boyan Jonov
Shu-Ping Hu
Alfred Smith
Log transformation and weighted least squares are commonly used to develop multiplicative error cost estimating relationships. We objectively compare the two regression techniques and provide a sound defense of log-linear ordinary least squares (LOLS) to counter arguments against its use. We then demonstrate how uncertainty modeling varies substantially based on regression method. Lastly, we establish criteria to use so that LOLS leads to a justifiable uncertainty assignment.
Keywords: Regression, Statistics, Variables, Uncertainty, Multiplicative Error Cost Estimating Relationships
The Zone System of Uncertainty Analysis
Risk & Uncertainty Track (RU08)
Jeff McDowell
This paper presents a methodology for selecting cost uncertainty distributions and their dispersion inspired by the photography Zone System. The method seamlessly addresses five groupings of uncertainty across a sliding complexity scale. The distribution shapes and their parameters are drawn from patterns developed from the AFCAA Cost Risk and Uncertainly Metrics Manual (CRUAMM) body of work.
Keywords: Methods, Modeling, Risk
Software & IT Track
Track Chairs: Daniel French, Daniel J. Harper
Download All Software & IT Track Papers & Handouts (.zip)
Cloud Total Ownership Costing: Considering the Technologies, Costs and Benefits
Software & IT Track (SI01)
Dan Galorath
Steven Woodward
Cloud abounds with promises of drastically reduced costs, which are all attractive. However, Cloud computing can be ineffective in terms of technology, affordability and total cost of ownership in many situations. The assumption that cloud technology reduces government costs is often flawed. The benefits from cloud computing relating to time-to-market, savings, innovation and agility should be analyzed, providing balanced perspectives for the cloud opportunities presented.
Keywords: Budgeting, Cost/Benefit Analysis, Government, Infrastructure, IT, Operations, Cloud
Data Collection for Agile Projects
Software & IT Track (SI02)
Blaze Smallwood
Agile software development projects produce a unique set of metrics, such as points and velocity, that can provide interesting insight into project progress. However, collecting this data requires specialized mechanisms, since no established standards exist, like SRDRs. This paper will discuss the types of data that are useful to collect for agile software development projects and mechanisms that have been used to collect them for several government projects.
Keywords: Agile, Data Collection, DoD, Government, IT, Program Management, Story Points, CDRL
Apples and Oranges: a Presentation and Analysis of Results of Cloud Cost Calculators and Rate Cards
Software & IT Track (SI03)
Daniel J. Harper
A recent effort for an Army customer examined over a dozen calculators and rate cards for estimating storage and hosting costs for cloud applications. This presentation will provide an overview of several calculators and tools, guidance for cost estimators on interpreting IT-centric inputs, and a discussion of similarities and variation in results. We will also present a cloud complexity plotter which provides a visual tool for explaining cloud cost and complexity drivers.
Keywords: Early Cost, IT, Program Management, Metrics, IT metrics, Cloud
Agile Software Development: Theory, Cost Estimating Methods, and Case Studies
Software & IT Track (SI04)
Eric Hawkes
Cole J. Kupec II
As Agile software development continues to become more common, cost estimators are compelled to adapt their methods to the Agile approach. This paper begins by explaining what Agile really is and how it differs from the common waterfall methodology. It also presents two case studies that detail how Agile was successfully implemented in a DoD acquisition program. Lastly, the paper proposes four innovative cost estimating methods for an Agile environment.
Keywords: Agile, DoD
The Cost of Cyber
Software & IT Track (SI05)
Ann Hawpe
Jeffrey Voth
A digital revolution is changing the nature of warfare and the cyber domain is now a top priority for all MDAP and MAIS programs. As the continued advancement of cyber warfare changes the future of land, air, sea, and space over the next decade, the cost community will need to consider the challenges facing defense programs and how to estimate the cyber security requirements within the construct of a reformed acquisition process.
Keywords: DoD, Infrastructure, IT, Cyber Security, Cyber Defense
Maturing the Economic Aspects of Agile Development in the Federal Government
Software & IT Track (SI06)
Jennifer Manring
Agile development has emerged in the commercial domain as a leading software development methodology, with growing adoption across federal agencies. This new approach for software development challenges traditional acquisition areas including cost estimation, economic analysis, and system engineering. This research provides a deeper understanding of the economic implications of agile development for the Federal Government and provides insights needed to understand the value proposition of an agile development environment compared with a traditional waterfall environment.
Keywords: Agile, Cost/Benefit Analysis, DoD, Government, IT, Software, Story Points
Using Predictive Analytics for Cost Optimization Across Cloud Workloads
Software & IT Track (SI07)
Zachary Jasnoff
David A. Cass
As organizations move to the cloud, estimating costs proves difficult, presenting many challenges. Once estimated, cloud costs must be optimized taking into account workloads. Organizations failing to estimate cost based on workloads will either underestimate or over spend on cloud services, often with disastrous results. A framework for optimizing costs (spanning IaaS, PaaS, and SaaS) and determining the cost/benefit between public clouds, private clouds and hybrid clouds will be presented.
Keywords: Budgeting, Cost/Benefit Analysis, Infrastructure, Methods, Modeling, Operations, Program Management, Cloud
SURF Process Summary & Initial Findings: A Deeper Focus on Software Data Quality
Software & IT Track (SI08)
Nicholas Lanham
Corinne Wallshein
Ranae P. Woods
Dan Strickland
As a result of the joint Software Resource Data Report Working Group (SRDRWG), several recommendations were generated by the supporting, multi-agency SRDRWG members to the Cost Leadership Forum. One of which led to the development of the first joint-agency SRDR Validation and Verification (V&V) guide as well as the SRDR User Review Function (SURF) subgroup. This study summarizes the results and data-quality improvement areas identified from 319 V&V comments generated by the SURF team.
Keywords: Data Collection, DoD, Government, Methods, Performance Management
Cloud Solutions – Infrastructure, Platform or Software – Where Should You Go?
Software & IT Track (SI09)
Arlene F. Minkiewicz
Ashley Hoenigke
Cloud computing technologies promise to create some new and interesting challenges for the cost estimating community. How does one determine the effort and costs associated with migration of existing applications to the cloud? What challenges do the various cloud solutions (IaaS vs. PaaS,vs. SaaS) present? This paper presents a case study outlining the migration activities and cost implications for migrating the same application on each of the three cloud solution models.
Keywords: Modeling, Parametrics, Cloud, Platform as a Service, Infrastructure as a Service, Software as a Service
Software Sizing Track
Track Chairs: David Bloom, Eric Plumer
Download All Software Sizing Track Papers & Handouts (.zip)
Estimating Small Scale Software Integration Efforts with a Large Scale Code Base
Software Sizing Track (SS01)
Joe Bauer
Estimating the cost and effort for testing and integration is often one of the most challenging aspects of software estimation. Integration effort depends greatly on the components under evaluation. When relatively small capabilities are introduced into large existing code bases, the integration impact due to the existing code may surprise stakeholders even when requirements are reduced. This presentation highlights software integration modeling for a major defense acquisition program.
Keywords: Government, Modeling, Parametrics
Alternatives to SLOC in the Modern Environment
Software Sizing Track (SS02)
Jeremy Goucher
Source Lines of Code (SLOC) based estimating relies heavily on subject matter expert input. However, viable alternatives to SLOC are few. By mapping deliverables to functionality and functionality to effort, the authors propose an alternate technique with reduced reliance on SLOC. Agile development and some recent innovations represent alternatives as well. The authors will present an overview of the topic with examples, required data inputs, and estimating methodology.
Keywords: Early Cost, Function Points, Modeling
Agile and Function Points: A Winning Combination
Software Sizing Track (SS03)
Daniel B. French
Many users of Agile have an incorrect belief that projects do not need to adhere to proper project management practices. However, it doesn’t advocate getting rid of metrics or PM processes. The most critical aspect of software development is software size. Function Points can be effectively used with Agile development and this presentation will demonstrate the benefits of using FP in many aspects of project management including estimation, schedule and product backlog.
Keywords: Agile, DoD, Early Cost, Function Points, Government, IT, Life Cycle, Parametrics, Program Management, Scheduling, Story Points, SCRUM, Product Backlog, Project Management, Software Estimation
Cost Element Structure for Cloud Migration Business Case Analysis
Software Sizing Track (SS06)
Lorena Aguilar
In 2010, the White House released its initial “cloud first” policy. Agencies have since struggled to identify all of the costs associated with cloud migration, and recent audits have criticized agencies for not properly assessing cloud-related costs and benefits. We have developed a cloud migration cost element structure (CES), which simplifies and standardizes cost data collection and estimating. We will present the structure and its applicability to business case and cost benefit analyses.
Keywords: Data Collection, DoD, Infrastructure, Labor, Cloud Computing
Collaborative Scoring: An Innovative Approach for Sizing Software Projects
Software Sizing Track (SS07)
Blaze Smallwood
A major challenge facing government organizations is developing defendable cost and schedule estimates for software development projects, driven by adequately estimating software size. This paper will describe a viable new method for sizing software requirements that utilizes a collaborative scoring methodology similar to those used during agile sprint planning. It has been successfully implemented on several government projects and is a viable option on all software projects.
Keywords: Agile, DoD, Early Cost, Functional Requirements, IT, Methods, Story Points, Collaborative, Scoring, Sizing, Complexity
2016 Best Paper: Software Sizing & IT Category
Process-Related Effort and Schedule Estimating Relationships for Software Cost Estimating
Software Sizing Track (SS08)
Corinne Wallshein
Nichlas Lanham
Wilson Rosa
The Naval Center for Cost Analysis will present comprehensive, updated findings of software size growth factors, effort estimating relationships (EER), and schedule estimating relationships (SER) with subsets of Department of Defense Computer Software Configuration Item records. This presentation focuses on software size (new, modified, reused, auto-generated, and total code). Subsets include maturity, application and super domain, language, contract type, and operating environment.
Keywords: DoD, Early Cost, Government, Labor, Methods, Operations, Parametrics, Regression, Scheduling, Statistics, Variables, Software cost estimation, software source lines of code, software effort hours, software duration, application domain, super domain, operating environment, language type, contract type, maturity, software size, empirical analysis, regression