2022 Workshop Breakout Sessions – International Cost Estimating and Analysis Association

Analytical Methods Track

Using TRLs to Predict the Future of Nuclear Weapons
Analytical Methods Track (AM01)
Abby Schendt
M. Michael Metcalf
Raymond Vera

Technology Readiness Levels (TRL) are used to measure and assess technology maturity. This paper presents innovative research demonstrating how historical TRL data can be used to perform credible, data-driven schedule analysis for programs early in development. Originally designed to help schedule analysis for the nuclear weapons stockpile, the resultant methodology combines historical milestone data and statistical methods to generate a Monte Carlo simulation of a risk-adjusted schedule for complex programs.

Keywords: monte carlo, TRL, schedule risk, MRL

AM01 – Schendt – Paper – Using TRLs to Predict the Future

AM01 – Schendt – Using TRLs to Predict the Future

Software Licenses: A Bill You Can’t Pay?
Analytical Methods Track (AM02)
Cheryl L. Jones
Brad Clark

For the past 20 years the DoD has emphasized the purchase and use of commercial software licenses as an approach to cut overall software development costs. Has this approach achieved its desired effect? How can one tell? This presentation will show the challenges and results of analyzing over 4,500 license records collected for the Army Software Maintenance Initiative. Implications to DoD’s future maintenance costs as well as implementing its DevSecOps objectives will be discussed.

Keywords: Cost Management, Decision Analysis, Parametrics, Performance Management

AM02 – JonesCheryl – Paper – Software Licenses

AM02 – JonesCheryl – Software Licenses

8D Cost Trades With Entanglement
Analytical Methods Track (AM04)
Doug Howarth

Many modern goods have elements that have substantial costs or create hazards if unavailable. If a component begins to add more cost than value to a final product or becomes hard to get, the project’s viability may be at risk. Here, we study the connections between a subcomponent and an ultimate product across interrelated market pairs. We look at their seven primary dimensions to visualize market mechanisms and add time as an eighth dimension.

Keywords: 4D, 7D, 8D, Hypernomics, Multidimensional Economics

AM04 – Howarth – Paper – 8D Cost Trades With Entanglement

AM04 – Howarth – 8D Cost Trades With Entanglement

Sustainment Analysis Methodology for Cost Models and Business Cases
Analytical Methods Track (AM05)
George Bayer
Austin Lutterbach

Government infrastructure systems often require detailed sustainability analyses – historical parts failures analysis, parts’ procurement, end-of-life, and economic analyses – to (1) forecast the optimum date for an acquisition to replace aging infrastructure with a New Investment, (2) conduct a cost-benefit analysis to justify further F&E investment, and (3) provide a standard sustainment cost estimate (corrective and preventative maintenance) for acquisitions. Using parametrics, research, and software algorithms, the team demonstrates an optimum approach for sustainment analysis.

Keywords: Cost/Benefit Analysis, Data-Driven, Decision Analysis, Government, Life Cycle, Methods, Parametrics, Regression, Risk, Statistics, Business Case, Python, Optimization, Process, Critical Thinking

AM05 – Bayer – Sustainment Analysis Methodology

AM05 – Bayer – Paper – Sustainment Analysis Methodology

Implementing Quantile Regression for Capital Acquisition Project Estimating
Analytical Methods Track (AM06)
Zachary Matheson
Jeff Beck
Charles Loelius
Gregory Stamp

The National Nuclear Security Administration (NNSA) is investigating the use of quantile regression on NNSA capital acquisition data to determine the cost drivers of project costs at values above the median, instead of via use of Log-Ordinary Least Squares. Utilizing this methodology will provide a better understanding and quantification of the uncertainty inherent in the NNSA’s capital project budgets, which are required to be budgeted at between the 70% and 90% confidence level.

Keywords: Budgeting, Parametrics, Regression, Uncertainty

AM06 – Matheson – Implementing Quantile Regression

The Progression of Regressions
Analytical Methods Track (AM07)
Jennifer Aguirre
Kyle Davis

Do you lose sleep at night over not being able to find the best regression for your data? Well fear no more, for this brief will alleviate those concerns. We review a case study creating a CER with a limited data set exploring multiple forms of regression (Linear, Nonlinear, Cubic, Polynomial), segmenting data into applicable portions while comparing predicted vs. actual results, and using predictive statistics and guidelines to assess best fit.

AM07 – Aguirre – The Progression of Regressions

Predicting and Minimizing Industrial Base Risks to Emerging Technologies
Analytical Methods Track (AM08)
Kevin Ray

The global pandemic has highlighted the fragility of the U.S. industrial base relied on to develop the next emergent technologies. Programs are routinely impacted by industrial base issues causing schedule delays and cost growth without the analytical means to predict and prepare for them. An analytical framework was developed that leverages cost estimating methodologies to predict these industrial base issues, quantify their impact, establish a new risk adjusted baseline, and reduce risk through targeted investments.

Keywords: Cost/Benefit Analysis, Risk, Uncertainty

AM08 – Ray – Predicting and Minimizing Industrial

Data Science Track

Developing Conceptual Cost Estimates for Good Ideas
Data Science Track (DS01)
Adrian Mitchell

This paper outlines the use of a super learner method for generating conceptual estimates of Australian Department of Defence facilities and infrastructure products. Many organisations require quick turnaround cost estimates for good ideas looking for budget funding. Developing estimates in the absence of firm requirements and a full understanding of proposal scope is challenging. A transformative approach to cost and duration estimation is needed to solve this problem.

Keywords: Modeling, Super learner, conceptual estimates

DS01 – Mitchell – Paper – Developing Conceptual Cost

DS01 – Mitchell – Developing Conceptual Cost

Supercharging Machine Learning with Cloud Computing
Data Science Track (DS02)
Conner Lawston
Bryan Eckle
Ketan Ganatra

Modern cloud computing environments can speed up machine learning models by up to 100X! In this case study with the DoD Joint Artificial Intelligence Center (JAIC), see how AWS and Databricks can be utilized to improve existing machine learning (ML) models for detecting fraud correcting unmatched transactions. When combined with robotic process automation (RPA), this effort saved 100,000 manual hours in FY20, and identified potential fraud with over 90% accuracy.

Keywords: Data-Driven, Statistics, Machine Learning, Cloud Computing, Robotic Process Automation

DS02 – Lawston – Supercharging Machine Learning with Cloud Computing

Statistical Techniques to Improve Software Effort Estimation Data Quality for Cost Engineers
Data Science Track (DS03)
Tomeka Williams

Since the topic of improving data quality has not been addressed for the U.S. defense cost estimating discipline beyond changes in public policy, the goal of the study was to close this gap and provide empirical evidence that supports expanding options to improve software cost estimation data matrices for U.S. defense cost estimators. The purpose of this quantitative study was to test and measure the level of predictive accuracy of missing data theory techniques that were referenced as traditional approaches in the literature, compare each theories’ results to a “ground truth” data set.

Keywords: Engineering Economics, Statistical Imputation, Cost Engineering, Missing Data Theory, Data Cleansing

DS03 – Williams – Paper – Statistical Techniques

DS03 – Williams – Statistical Techniques

Integrating Data Science & Cost Analysis
Data Science Track (DS04)
Sarah Green
Dakotah W. Hogan
Jacob Share

Data is more abundant than ever before. How can we as cost analysts leverage the full power of the data science world to get the most out of our data and utilize things like automation, machine learning, interactive dashboards and more? The author will demonstrate the innovative ways that AFCAA is utilizing a cloud based platform and various data science tools to transform our entire process from data ingestion/ storage to end products and even end to end cost modeling.

DS04 – Green – Integrating Data Science and Cost Analysis

Managing Data Science: A Stacked Approach to Integrating Advanced Data Analytics
Data Science Track (DS05)
John W. Maddrey
Eric J. Hagee

In this presentation, we will define the levels of the data science stack, what success in the stack means, and how to organize the work force with in the stack. Furthermore, we will discuss global processes for collecting, managing, and analyzing data including desired skill sets and required technologies.

Keywords: Communication, Cost Management, Data Collection, Data-Driven, Functional Requirements, Government, Methods, Modeling, Operations, Performance Management, Program Management, Risk, Scheduling, Statistics, Data Science, Stack, Data Analytics

DS05 – Hagee – Managing Data Science

Cost Estimating Maturity and a Vision for the Future
Data Science Track (DS06)
Patrick Malone
Henry Apgar
William King

The science of cost estimating has matured. From the Egyptians to modern methods, cost analyses advances were progressive, recent methods further the science. This paper reflects on rich heritage and looks to the future of cost estimating and analysis using emerging and maturing techniques. We look at postmodern tools and methods like Natural Language Processing and other methods enhance our ability accurately forecast costs of more complex systems. Last, we discuss future research and trends.

Keywords: Bayesian, Data-Driven, Decision Analysis, Methods, Program Management, Risk, Space, Advanced Tools

DS06 – Malone – Cost Estimating Maturity

DS06 – Malone – Paper – Cost Estimating Maturity

Machine Learning/NLP Track

Utilizing Artificial Intelligence to Evaluate and Estimate Requirements
Machine Learning/NLP Track (ML01)
Amar Zabarah
Kevin McKeel

Business Analysts face the challenge of parsing through hundreds/thousands of requirements to define system needs or to accurately estimate the software development cost. This is a grueling process and mistakes are still an inevitability. However, AI can serve analysts in automating requirements analysis, producing high-level concepts, and informing total lifecycle estimates in a short period of time. This presentation provides an overview of select AI capabilities and shows their applications to requirements analysis and cost estimation.

Keywords: Budgeting, Cost/Benefit Analysis, Functional Requirements, Process Engineering, Software, Artificial Intelligence (AI), Systems Analysis, Requirements Analysis, Software applications, Information systems, Software Sizing, Function Point, Function Point Analysis, Automation, Natural Language Processing (NLP)

ML01 – Zabrah – Utilizing AI to Evaluate Requirements

Estimation of Application Maintenance by Means of Machine Learning
Machine Learning/NLP Track (ML02)
Eric van der Vliet
Kimmo Kettula

This presentation is about an estimation model for application maintenance. The model calculates the required effort and costs for maintaining software application based on incidents with different service levels and priorities. The calculation is based on historical data collected of different business domains. To manage the large amount of data, application characteristics and domains, advanced data analytics and machine learning techniques are required. The development approach as well as the prototype results will be shown in this presentation.

Keywords: Data-Driven, Machine Learning, Data Analytics, Managed Services

ML02 – van der Vliet – Estimation of Application Maintenance

Linear Regression: How to Make What’s Old New Again
Machine Learning/NLP Track (ML03)
Kimberly Roye
Sara Jardine
Christian Smart

With the booming popularity of machine learning techniques, data scientists may have you believe there is no place for linear regression. Though many cost estimating applications are nonlinear, when there are linear relationships among features, ordinary least squares regression often performs better than the most powerful machine learning techniques. Scenarios for which linear regression should be chosen over more complicated algorithms are presented, as well as techniques such as regularization, gradient descent, and Bayesian methods.

Keywords: Regression

ML03 – Roye – Paper – Linear Regression

ML03 – Roye – Linear Regression

Adopting a Data Science Paradigm
Machine Learning/NLP Track (ML04)
Kyle Ferris
Zoe Keita

The availability of large unstructured datasets accelerated via autonomous data collection and warehousing, makes the use of advanced analytical skillsets a necessity. The incorporation of data science methodologies can yield meaningful improvements to cost estimating practices through effective data governance, streamlined data collection/normalization, and increased opportunities for exploratory analysis. This presentation will evaluate the software and skillsets required for advanced modeling and analysis, as well as planning considerations for data science training development.

Keywords: Communication, Data Collection, Data-Driven, Decision Analysis, IT, Methods, Modeling, Software, Data Science, Data Analysis, Programming

ML04 – Ferris – Adopting a Data Science Paradigm

Advanced Natural Language Processing for Work Breakdown Structures
Machine Learning/NLP Track (ML05)
Maura Lapoff

The National Nuclear Security Administration (NNSA) collects Work Breakdown Structure (WBS) data for Capital Asset Projects. Implementing the structure across projects grows exponentially in complexity due to varying scope, contextual changes, and vendor requirements. This paper will demonstrate how Natural Language Processing can automate the process of identifying and classifying WBS elements used for Capital Asset Projects. NLP can improve the Cost Analyst’s workflow by reducing time-consuming aspects of their work.

Keywords: Data-Driven, Government, Methods

ML05 – Lapoff – Advanced Natural Language

ML05 – Lapoff – Paper – Advanced Natural Language

Strategic Investment Planning using Machine Learning Techniques
Machine Learning/NLP Track (ML06)
Wendy E. Robello

Early identification of anticipated customer needs is a major component to winning new business in competitive environments. This allows bidders to make strategic investment decisions that will improve core competencies to meet anticipated customer needs. Current approaches to predict customer needs are subjective, costly, and time-consuming. This research will provide bidders an economical and objective approach using supervised machine learning to predict customer needs deduced from prioritized customer selection criteria to strengthen their proposed solution.

Keywords: Decision Analysis, strategic investment, machine learning, competitive positioning, customer needs

ML06 – Robello – Strategic Investment Planning

A Deep Look into Optimistic Bias based on a NASA’s STEM 4th Grade Activity
Machine Learning/NLP Track (ML07)
Steve Sterk

Beyond this 4th, Grade NASA STEM activity found on NASA JPL’s web site, this paper is a direct result based on independent research being conducted by Steve A. Sterk who works in the Program Planning and Control (PPC) Branch at NASA Armstrong Flight Research Center. It is a believed optimistic cognitive bias, is the number one problem while developing cost and schedule estimates and thus digging into human behavioral, a heuristic technique in completing the task at hand. This presentation will include; How people think, the start of an Optimistic Cognitive Bias data base for Machine Learning.

Keywords: Bias, Learning Curve, How People Think

ML07 – Sterk – A Deep Look into Optimistic Bias

Management & Risk Track

Delusions of Success: Overcoming Optimism Bias in Schedule Forecasting
Management & Risk Track (MR02)
Jeffrey Voth
Maxwell Moseley
Ann Hawpe

Schedule performance of major defense acquisition programs remains a challenge, leading to an average capability delay of more than two years and substantial cost growth, according to the latest U.S. Government Accountability Office assessment. Stakeholders must de-risk schedule estimates and improve performance through data-driven approaches based on realized prior program histories to prevent optimism bias. The authors evaluate the merits of taking an ‘outside view’ to mitigate risk through reference class forecasting, utilizing empirical distributional information from 116 programs across six commodity classes to develop more realistic and reliable front-end schedule estimates.

Keywords: Bias, Data-Driven, Methods, Scheduling, Statistics, Uncertainty

MR02 – Voth – Delusions of Success

MR02 – Voth – Paper – Delusions of Success

Real-World Data Transformation Challenges in the DoD Supply Chain
Management & Risk Track (MR03)
Michael Ahearn

Merger and acquisitions have driven tremendous growth of the top one hundred government contractors. This growth has also embedded land mines in the historical data of these “compete-mates.” The presenter will discuss real-world lessons from the extraction and transformation of programmatic, cost, schedule, and effort data in support of processes from Should Cost to Basis of Estimate for bid and proposal pursuit.

MR03 – Ahearn – Real World Data Transformation

Cost and Schedule Risk Analysis of Megaprojects
Management & Risk Track (MR04)
Michael Trumper
Lev Virine

Cost and schedule risk analysis of megaprojects can be complex, time consuming, and may not give meaningful results. The presentation demonstrates a practical and easy to implement steps how to perform risk analysis of projects of any size. They include schedule diagnostics consolidation. The large megaproject schedule can be separated on subprojects, which can be analyzed individually. The analysis can be performed for any milestone and multiple times during the course of project.

Keywords: Cost Management, Monte Carlo, Risk, Scheduling, Uncertainty, Megaprojects

MR04 – Trumper – Cost and Schedule Risk Analysis of Megaprojects

Uncertainty of Expert Judgment in Agile Software Sizing
Management & Risk Track (MR05)
Peter Braxton
Ken Rhodes
Alex Wekluk
David Brown

Agile software estimating and planning often rely on expert judgment to assess the size of the development effort at various levels of granularity and stages of maturity. Previous research by the author quantified the inherent risk and uncertainty of the self-similar scales (e.g., T-shirt sizing) commonly used in these assessments. This paper expands those a priori mathematical results and empirically tests the accuracy of experts in applying those scales. It elucidates the ideal ratio to align with the desired confidence interval, and recommends feedback mechanisms to improve consistency.

Keywords: Early Cost, Functional Requirements, Risk, Software, Uncertainty, Agile, Story Points

MR05 – Braxton – Paper – Uncertainty of Expert Judgment

MR05 – Braxton – Uncertainty of Expert Judgment

Alternative EAC Methodologies – Calculating EACs without Standard EVM Data
Management & Risk Track (MR06)
Richard Lee
Emily Goldhammer
Kate Malcolm

Developing an Estimate at Complete (EAC) typically involves establishing a performance-based estimate founded in the best practices of Earned Value Management (EVM) Gold Card analysis. But what if you are missing EVM data or do not have a performance baseline? This presentation explores alternative approaches to developing an EAC by leveraging burn rate profiles. Estimates are then developed by applying regression analysis and/or linear interpolation techniques to the historical burn rates observed.

MR06 – Lee – Alternative EAC Methodologies

Decision Support and Operation Design in Mission Critical Applications
Management & Risk Track (MR07)
Amin Rahimian

This talk covers key concepts of decision making under uncertainty. Building on applications in soldier-robot teaming, we explain how to balance exploration and exploitation tradeoffs using multi-armed bandits and reinforcement learning. The second half of the talk is concerned with challenges of operation design in costly and limited-information environments. Using examples from adversarial information operations, we explain how to balance costs and benefits of data collection and how to mitigate risks to data acquisition resources.

MR07 – Rahimian – Decision Support and Operation Design

Modeling & Analysis Track

Second Source Manufacturing: Lessons from the Second World War
Modeling & Analysis Track (MA01)
Brent M. Johnstone

Manufacturing defense systems at different sites is increasingly common due to foreign coproduction and international cooperative ventures. These situations challenge estimators, posing questions about the transfer of learning and relative efficiency of multiple production sites. This paper examines cost history from World War II, when U.S. bomber production lines were shared across multiple companies. The conclusions are tested against modern experience and guidance provided to estimators seeking help.

Keywords: Data-Driven, Labor, Learning Curves, Manufacturing, Methods, Modeling

MA01 – Johnstone – Paper – Second Source Manufacturing

MA01 – Johnstone – Second Source Manufacturing

Model-based Cost Engineering launches with Space Missions Estimating
Modeling & Analysis Track (MA03)
Vincent Delisle
Shawn Hayes
Mark Jacobs

NASA continually strives to improve cost estimation for the highly advanced technology flown on planetary as well as earth orbiting space missions. Over the years it has been proven that parametric cost models are a desired way to obtain accurate estimates. Still there is still room for improvement. This paper will discuss two of the latest and best methods for obtaining accurate cost estimates using best-of-breed model-based cost engineering techniques.

Keywords: Modeling, Parametrics, Space, NASA, Space Missions, Planetary, Earth Orbiting, electric propulsion, ion thrusters, lasers, parachutes, radar altimeters, thermal protection, case studies, validation

MA03 – Delisle – Model based Cost Engineering

MA03 – Delisle – Paper – Model based Cost Engineering

NRO CAAG Parametric Model for Spacecraft-to-Launch-Vehicle Integration
Modeling & Analysis Track (MA04)
Daniel Barkmeyer

New, innovative launch service providers and spacecraft designs have broadened the tradespace of launch service options for government customers, placing new importance on launch cost, an often understudied component of space programs’ enterprise-level cost. This paper demonstrates a parametric approach to estimating the most mission-dependent component of launch cost, spacecraft-to-launch-vehicle integration engineering. It addresses primary and secondary cost drivers and typical time phasing, and demonstrates the model’s performance compared to NRO and NASA historical costs.

Keywords: Data-Driven, Government, Methods, Modeling, Parametrics, Regression, Space, Launch, Rocket, Integration, ULA, SpaceX, Blue Origin

MA04 – Barkmeyer – NRO CAAG Parametric Model

Integrating Cost into Model-based Systems Engineering Environments
Modeling & Analysis Track (MA05)
Daniel Kennedy
Karen Mourikas

Model-based Systems Engineering (MBSE) incorporates digital models to represent system-level physical attributes and operational behavior throughout the system life-cycle. To date, many MBSE efforts have focused on technical requirements with little emphasis on cost. Integrating cost models and MBSE enables rapid exploration of design trades and associated cost impacts, traceability between requirements and cost targets, and earlier identification of affordability issues. We present results from several ongoing efforts as a continuation of our 2019 presentation.

Keywords: Decision Analysis, Manufacturing, Modeling, MBSE, Digital Engineering

MA05 – Kennedy – Integrating Cost into Model-based Systems

NASA Instrument Cost Model (NICM 9) Introductory Training
Modeling & Analysis Track (MA06)
Joseph Mrozinski

The NICM 9 Tool suite allows for cost and schedule estimation for space flight instruments. This introductory training session will examine a case study utilizing both the System and Subsystem tools, moving from inputs and assumptions to the cost and schedule estimates and interpretations. An overview of the NICM model development methodology will be given. Instructions to obtain follow-on training and to download NICM 9 will conclude the talk.

MA06 – Mrozinski – NASA Instrument Cost Model

Cost and Throughput Analysis for the NASA Ames Arc Jet Modernization Program
Modeling & Analysis Track (MA07)
Jennifer Scheel
Christian Smart

The facilities of NASA’s Arc Jet Complex are used to simulate the aerothermodynamic heating that a spacecraft endures throughout hypersonic atmospheric entry, and to test candidate Thermal Protection System (TPS) materials and systems. Ms. Scheel and Dr. Smart analyze the cost and throughput impact of several different modernization options. Estimating the cost requires independent research into highly specialized subsystems and customizing inflation application. Throughput analysis of the number of possible test runs per year is conducted via a probabilistic simulation.

MA07 – Scheel – Paper – Cost and Throughput Analysis

MA07 – Scheel – Cost and Throughput Analysis

Is Your Organization Ready for Model-Based Cost Engineering?
Modeling & Analysis Track (MA08)
Michael Ahearn

Model-based design continues to yield unprecedented speeds in technology solution development. Integrated and data-driven Model-Based Cost Engineering (“MBCE”) promises to estimate concurrently at the speed of design. Taking advantage of MBCE requires vision and a level of organizational maturity. The presenters will discuss critical attributes regarding people, processes, data and technology that impact the success of model-based costing.

Keywords: Early Cost, Functional Requirements, Methods, Process Engineering, Model-based design, MBSE, Cost Engineering

MA08 – Ahearn – Is Your Organization Ready

Mission Operations Cost Estimation Tool (MOCET) 2022 Research
Modeling & Analysis Track (MA09)
Marc Hayhurst

The Mission Operations Cost Estimation Tool (MOCET) is a model developed by The Aerospace Corporation in partnership with NASA’s Science Office for Mission Assessments (SOMA). MOCET provides the capability to generate cost estimates for the operational, or Phase E, portion of full NASA space science missions. Research topics being studied in 2022 will be presented, including: Level 2 Work Breakdown Structure (WBS) cost modeling, and extended mission cost modeling.

Keywords: Modeling, Tools, & Analysis Case Studies

MA09 – Hayhurst – Mission Operations Cost Estimation Tool

DICEROLLER: Estimating D&D Costs for the NNSA
Modeling & Analysis Track (MA10)
Zachary Matheson
Charles Loelius

The National Nuclear Security Administration Office of Programming, Analysis, and Evaluation has developed a model for estimating the cost of Decontamination and Disposition (D&D) activities. This effort involved collecting and normalizing cost data from past D&D projects, and then generating a parametric cost estimating relationship to predict future D&D project costs. The resulting model, named DICEROLLER, will be used to make lifecycle cost estimates and one-for-one replacement cost estimates of capital acquisition projects.

Keywords: Early Cost, Government, Infrastructure, Life Cycle, Parametrics, Regression

MA10 – Matheson – DICEROLLER

MA10 – Matheson – Paper – DICEROLLER

Applying System Readiness Levels to Cost Estimates – A Case Study
Modeling & Analysis Track (MA11)
Patrick Malone

Estimating cost, schedules and expected technical performance of large complex systems pre-development is difficult. When programs are executed, they are plagued with cost growth and schedule delays due to less than required maturity of some elements. We investigate the James Webb Space Telescope program actual cost and schedule history using System Readiness Level (SRL) methods to reveal trouble areas early. The resulting approach will support future estimating accuracy through higher fidelity information for early decision-making.

Keywords: Budgeting, Cost Management, Methods, Program Management, Risk

MA11 – Malone – Applying System Readiness Levels

Processes & Best Practices Track

Cracking Open the ‘Black Box’ of Product Technical Support Contracts
Processes & Best Practices Track (PB02)
Alexander Bonich
Patrick McCarthy
Rhys Bergeron

Product Technical Support contracts represent a significant annual cost to the DoD. While these contracts serve as useful “catch-alls” for various programmatic requirements, estimating these costs for future systems can be challenging. This paper analyzes historical technical support costs for various Army ground vehicle systems and demonstrates how combining contracts, work directives, and cost data sources along with categorizations and tags for “Service Categories” provides a comprehensive understanding and management of technical support cost requirements.

Keywords: Bayesian, Data Collection, Data-Driven, Government, Methods, Contract Pricing, Predictive Analytics

PB02 – Bonich – Paper – Cracking Open the Black Box

PB02 – Bonich – Cracking Open the Black Box

Visual Exploration of Data – The Missing Element in CER Development
Processes & Best Practices Track (PB03)
Benjamin Kwok

Within cost estimating training literature, it is common for the discussion around Cost Estimating Relationship (CER) development to focus primarily on its statistical parameters (e.g. correlation, equation form, etc.). An underemphasized component of CER development is the need to first visualize and explore your data. This presentation will show how integrating these processes into CER development leads to faster and better results.

Keywords: data visualization, cers, space force, space systems command

PB03 – Kwok – Visual Exploration of Data

Data Management for Cost Engineering Projects
Processes & Best Practices Track (PB04)
Cara Cuiule
Arlene F. Minkiewicz

As cost estimation data becomes more plentiful, data management becomes essential for organizing and preparing data for analysis. Even if a formal data management system is unnecessary for a project, concepts from database management can still be incorporated to ensure higher data quality. This paper will contain an overview of relational and non-relational models, along with a case study of a database that contains structured, unstructured, and semi-structured data.

Keywords: Data Collection, IT, Databases

PB04 – Cuiule – Data Management

PB04 – Cuiule – Paper – Data Management

Everything You Always Wanted to Know About Affordability… But Were Afraid to Ask
Processes & Best Practices Track (PB06)
Karen Mourikas
Denise Nelson

The term “Affordability” means different things to different people, depending upon one’s employer, organization, function, background, etc. The term has also morphed over the years – expanded or narrowed – based on one’s viewpoint. During ICEAA’s OEM-COG discussion on Affordability, diverse opinions were revealed. We will assemble various interpretations & implementations of Affordability from multiple perspectives, provide historical background, and explore how our community can clarify, standardize, and promote the concept of Affordability.

Keywords: Cost/Benefit Analysis, Life Cycle, Methods, Modeling, Affordability Analysis

PB06 – Mourikas – Everything You Always Wanted To Know

Predictive Thresholds for Schedule Execution Metrics
Processes & Best Practices Track (PB07)
Michelle Jones

We called it the search for a unicorn. After an intense week of schedule analysis, checking the numbers, updating visual basic code, then presenting objective evidence that the contractor is not achieving the baseline plan, we got the action: provide the metrics of a good program. We tried, but after reviewing countless schedules, we realized that a “good” program is as rare as a unicorn, because every program deviates from the baseline plan. Although we never found a unicorn (“good program” with good metrics), we developed data driven thresholds predictive of significant milestone slip.

Keywords: Data-Driven, Decision Analysis, Government, Methods, Scheduling, Space, Statistics

PB07 – JonesMichelle – Predictive Thresholds for Schedule Execution Methods

The BS in BoeS – Oh the Games that are Played
Processes & Best Practices Track (PB08)
Sandy Burney

A good Basis of Estimate (BoE) uses a historical comparison, and a complexity factor. These two choices are not statistically based, meaning a bias is introduced into most BoEs. This paper explores the bias in BoEs through the framework of Game Theory. A BoE goes through a creation, internal review, internal approval, and external review process, where each player in this process will likely have a different payoff matrix. Each player in this process will want to insert their bias into the BoE, which this paper terms “Bias Selectivity” (BS).

Keywords: Bias, BoE, Game Theory, OEM, SETA, Government

PB08 – Burney – Paper – BS in BoeS

PB08 – Burney – BS in BoeS

No Estimation Without Escalation – The Inflation Revolution
Processes & Best Practices Track (PB09)
Shannon Cardoza
Edward Smith

I downloaded the latest USG Inflation Indices, so my work here is done, right? OSD guidance encourages the analyst to explore program-specific escalation to maximize estimate accuracy vice defaulting to existing published indices. This briefing will provide a basics refresher background on common appropriation inflation rates, their outlay profiles, and how to interpret them. It will then explore how to improve to account for escalation, commodity-specific demand, and project-appropriate expenditure profiles increasing estimate accuracy.

Keywords: Inflation, Escalation, Expenditure Profile, Research, 3%

PB09 – Cardoza – No Estimating Without Escalation

Transforming Cost Estimation Services
Processes & Best Practices Track (PB10)
Kevin McKeel
Ed Spriggs
Tyrese Johnson

In the last 10-15 years the government contracting industry has quickened the pace of technology delivery. This emerging change has forced cost estimators, as well as acquisition professionals of all kinds, to change to their own processes to keep pace. Cost estimation shops can benefit by changing their service to match customer needs and delivery tempo, add software product instruction with analyst training programs, and extend the scope of “cost” to include budget formulation. This presentation touches on transformation such cost organization infrastructure and repeatable processes.

Keywords: Communication, Methods, Modeling, Services Transformation

PB10 – McKeel – Transforming Cost Estimation Services

Software & Agile Track

What Does Agile Software Development Need: Predictable Costs or Predictable Outcomes?
Software & Agile Track (SA03)
Christina Kosmakos

Despite ongoing laudable attempts to advance the state-of-the-art in agile software estimating processes and techniques, the software development and cost analysis communities are far from ‘cracking the nut’. There is a dire need for a cost estimating approach that accurately predicts agile project costs with the correct granularity to enable budget and execution planning. This paper will detail the exploration of three approaches that utilize extrapolation from actuals to implement a solution.

Keywords: Agile, Extrapolation, Software, Estimating

SA03 – Kosmakos – What Does Agile Software Development Need

SA03 – Kosmakos – Paper – What Does Agile Software Development Need

Simplifying Software Sizing with Simple Function Points
Software & Agile Track (SA04)
Daniel B. French
Carol Dekkers

Simple Function Points (SFP), modelled on the IFPUG Function Point Analysis method was introduced with the goal of simplifying the method while reducing cost, time, and difficulty estimating software size without sacrificing accuracy. We’ll introduce the IFPUG version of SFP and highlight the challenges/opportunities when using to size software. We’ll explore when to use SPF and key differences between SFP and IFPUG FP while providing guidance on using FP measures in software cost estimates.

Keywords: Function Points, Software Sizing, Software Metrics, Software Estimating, SFP, Simple Function Points, SiFP

SA04 – French – Paper – Simplifying Software Sizing

SA04 – French – Simplifying Software Sizing

Agile Team Performance Measurement as a Basis for Accurate Cost Estimation
Software & Agile Track (SA05)
H.S. van Heeringen

Agile team performance metrics Productivity, Cost Efficiency, Delivery Speed, Sprint Quality, and Product quality can be measured in an objective, repeatable and verifiable way, compared to each other, and benchmarked against industry data. The measurement data is used to recalibrate long-term effort, duration, and cost estimates based on the actual productivity delivered, resulting in increased predictability. I’ll show a recent study of 4 teams of one organization, each in a different European country.

Keywords: agile teams, performance measurement, agile estimation, software cost estimation, benchmarking

SA05 – van Heeringen – Agile Team Performance Measurement

Dynamic Software Effort Estimation: How SWEET It Is!
Software & Agile Track (SA07)
William Gellatly
David Brown
Lindsey Jones
Peter Braxton
Alex Wekluk

Software estimation is a complex and diverse field that must accommodate a variety of technical inputs across the life cycle. Our team developed an Excel-based effort estimation model designed for both flexibility and transparency. It dynamically builds an effort estimating relationship based on analyst-selected data fields and source data. Our paper explores how we used data from ISBSG, statistical analysis, and practical experience to drive the development of this “clear-box” model.

SA07 – Gellatly – Paper – Dynamic Software Effort Estimation

SA07 – Gellatly – Dynamic Software Effort Estimation

Agile Product Roadmap Estimating and Progress Tracking
Software & Agile Track (SA08)
Blaze Smallwood
Ryan Blackburn

Agile project managers often struggle developing and maintaining a full project plan, given the churn in requirements and priorities typical of an agile environment. PMs often use a Product Roadmap to communicate high-level work priorities and phasing, but these often lack the quantitative information needed to enable tracking progress against overall project goals. This paper will offer an innovative solution to this problem, which has recently gained traction on an actual government agile project.

Keywords: agile, product roadmap, software

SA08 – Smallwood – Agile Product Roadmap

Let’s Go Agile: Data-Driven Agile Software Cost and Schedule Models Derived from DHS Projects
Software & Agile Track (SA09)
Wilson Rosa
Sara Jardine
Chad J Lucas

This paper offers effort and schedule estimating models for agile software development projects using contractor backlog, monthly contract reports, and requirements documents. The regression models predict effort or schedule using six different sizing measures: Unadjusted Function Points, Simple Function Points, Functional Stories (Requirements), Issues, Stories, and Story Points. We also offer effort and schedule benchmarks. The analysis is based on data collected from 18 agile projects implemented within DHS and DoD between years 2014 to 2021.

Keywords: Backlog, JIRA, CONOPS, Requirements Document, Agile, Data-Driven, Early Cost, Functional Requirements, Regression

SA09 – Rosa – Paper – Lets Go Agile

SA09 – Rosa – Lets Go Agile

Software Phasing and Schedule Growth Analysis
Software & Agile Track (SA10)
Daniel Long

DoD software research lacks an understanding of development phasing, effort allocation, and schedule. We evaluated conventional rules of thumb for effort allocation against projects in the SRDR database. We compared how effort allocation varies between projects with high or low schedule growth. Results showed that increasing effort in early phases consistently decreases the total schedule growth, and led to improved allocation guidelines. These findings were significant across multiple categories such as Service and project size.

Keywords: Agile, Cost Management, Early Cost, Government, Scheduling, Software, Phasing

SA10 – Long – Software Phasing Schedule Growth

CEBoK® Training Track

Basics & Techniques
CEBoK® Training Track (CEB01)
Jennifer Kirchhoffer, CCEA®
The Basics & Techniques session introduces an overview of cost estimating and analysis and the reasons for doing cost estimates, as well as four essential cost estimating techniques most often used to develop realistic and credible estimates. Additionally, we will review cost estimating products and related topics such as schedule and operations and support estimating, providing the background information and fundamental knowledge from CEBoK® Modules 1-3.

Data Collection & Normalization
CEBoK® Training Track (CEB04)
Justin Knowles, CCEA®
This session covers the Core Knowledge section of CEBoK® Module 4: Data Collection. All estimating techniques and cost estimating models require credible data before they can be used effectively. In this module we will discuss the various types of data, processes needed to collect and analyze the data used in parametric applications, as well as data types, sources, and adjustment techniques.

Inflation
CEBoK® Training Track (CEB05)
Markie Harris
Katiana (Kat) Lemmons
This session covers the Core Knowledge section of Inflation and Index Numbers (CEBoK® Module 5). Proper inflation analysis is essential to the success of any cost estimate or economic analysis. Calculating inflation correctly and understanding the fundamental concepts will enable you to produce cost estimates that are timely, accurate, and credible to support your program’s lifecycle needs. It will also empower you to communicate with key stakeholders on the need to adjust your financial estimates based on changes in the economy.

Analysis, Probability, & Statistics
CEBoK® Training Track (CEB06)
Mike Allen, CCEA®
This session discusses the analytical steps to take after obtaining a set of cost data and covers techniques for displaying and analyzing data graphically and statistical and graphical analysis of univariate and bivariate data sets (CEBoK® Modules 6 & 10). Other topics include measures of central tendency and dispersion and important probability distributions. We also introduce the concept of a random variable; Monte Carlo simulation; and the differences between the normal and lognormal distributions. Finally, we discuss hypothesis testing.

Learning Curves
CEBoK® Training Track (CEB07)
Sriram Krishnan, CCEA®
This is a training track presentation of the CEBoK® Module 7 (Learning Curves) will cover the key ideas, analytical constructs, and applications of the module. Beyond the theoretical information, we will present the study questions for Module 7 with steps required to solve the problems using only a calculator as is required on the certification exam.

Regression Analysis
CEBoK® Training Track (CEB08)
Bill Barfield, CCEA®
This course introduces the basic concepts of regression and provides a demonstration of a simple linear ordinary least squares model (CEBoK® Module 8). This session focuses on the basics required to build and evaluate a simple linear model such as a Cost Estimating Relationship (CER). Key concepts include correlation, minimizing error, homoscedasticity, statistical significance, goodness of fit, confidence intervals, uncertainty, and analysis of variance. The better you understand these concepts, the better you will be able to make inferences about cost data and employ more complicated regression techniques.

Risk Analysis
CEBoK® Training Track (CEB09)
Sam Easterly, CCEA®
This session will provide motivation for the need for risk analysis and introduce the basic types and uses of risk (CEBoK® Module 9). It will focus on the practical execution of the general risk analysis process: develop a point estimate; identify the risk areas in the point estimate; determine uncertainty around the point estimate; apply correlation between uncertainty distributions; run the Monte Carlo simulation; assess the reasonableness of results; calculate, allocate, and phase risk dollars; and show the results.

Manufacturing Cost Estimating
CEBoK® Training Track (CEB11)
Corey Boone, CCEA®
The goal of the Manufacturing Cost Estimating module (CEBoK® Module 11) is to arm the student with a set of techniques used to address issues unique to estimating in the manufacturing environment. It will be our objective in this module to raise a few of the most common general issues, considerations and concerns the estimator must be aware of in a typical major manufacturing environment and to provide techniques for addressing them. Depending on time and interest of attendees, example problems can be worked as exam preparation.

Software Cost Estimating Using CEBoK-S
CEBoK® Training Track (CEB12)
David Brown, CCEA®
Kevin Cincotta, CCEA®
This session covers the core knowledge of Software Cost Estimating using CEBoK-S (all PCEA/CCEA testable topics are included). It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides an introduction to the basics of the software development and maintenance processes and how to estimate the related effort. The key ideas of Software Cost Estimating include the cost drivers of size, complexity, and capability. In the sizing area, we’ll focus on understanding the physical size, functional size, relative effort measures (agile software development), and RICE(FW)1 objects. We’ll also discuss the primary software development paradigms – Predictive (waterfall), Predictive with modification (incremental, evolutionary, and spiral methods), Agile (iterative, scrum, SAFe), and Hybrid – and how to model them from a cost estimating perspective.

Economic Analysis
CEBoK® Training Track (CEB13)
Richard Shea, CCEA®
This session covers the Core Knowledge section of Module 13 Economic Analysis of CEBoK®. It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides a practitioner’s perspective for conducting an economic analysis (EA) by reviewing EA concepts, terminology, variables and measures-of-merit. By accounting for monetized costs, monetized benefits, opportunity costs and time-value-of-money (“discounting”), an EA enables one to calculate economic measures-of-merit.

Contract Pricing
CEBoK® Training Track (CEB14)
Paul Neubert, CCEA®
This session explores the basics of contract pricing (CEBoK® Module 14). We explore various contract types and the factors and considerations related to choosing a contract type. We also explore fee, shared risk, cost-price proposal preparation, the makeup of a good Basis of Estimate (BOE), and evaluation efforts. This session also provides an introduction to cost management. Some methods discussed include Total Ownership Cost (TOC), Cost As an Independent Variable (CAIV), Target Costing, and Activity Based Costing (ABC).

Cost Management and EVM
CEBoK® Training Track (CEB15)
Earl J. Gee II
This session will provide an introduction to the basic concepts of earned value management (CEBoK® Modules 15 & 16), with a focus on implementation, governance, and practical application in support of a project or program. Specific topics will include basic EVM components and data elements, as well as standard earned value analysis techniques. We will use practice problems throughout the presentation to demonstrate and reinforce the basic principles of EVM.