Analytical Methods Track

Design to Cost: What it is and Why You Should Care
Analytical Methods Track (AM01)
Karen Mourikas
Henry Apgar
Lisa Colabella

Design to Cost (DTC) has been around since the 1970s, falling in and out of favor. But what exactly is DTC? And will it help control program or product costs? Googling it yields various definitions, sometimes contradictory. Examples fluctuate from large defense programs to mass-produced commercial components. Implementation differs depending upon company, life-cycle phase, and objectives. This presentation is the culmination of an effort by the SoCal ICEAA Chapter to educate our community on DTC.

Keywords: Cost/Benefit Analysis, Decision Analysis, Methods, Parametrics, Project Controls, DTC

AM01 – Mourikas – DtC What it is and Why – ppt

Accuracy, Precision and Uncertainty in Cost Estimates
Analytical Methods Track (AM02)
Timothy P. Anderson

Cost proposals for large government contracts are necessarily very detailed and typically highly precise, expressed down to the nearest dollar. This presentation explores the differences between precision and accuracy in cost proposals in the context of uncertainty and cost growth. Moreover, the presentation is interactive, with thought-provoking questions peppered throughout, providing numerous opportunities for audience participation for improved knowledge and retention.

Keywords: Precision, Accuracy, Uncertainty

AM02 – Anderson – Accuracy, Precision and Uncertainty – ppt

Innovative Risk-Driven Contract Pricing Strategy
Analytical Methods Track (AM03)
Brian Flynn
Robert Nehring
Andi Hylton

*2023 Best Paper: Analytical Methods Category -and- Best Paper Overall

This paper presents a framework or scoring matrix that quantifies contract risk using a numerical evaluation of the many factors that make or break a program, such as: experience of the contractor, stretch in technology, solidity of requirements, and degree of competitive procurement. The framework, in turn, is made operational by leveraging benchmarks from over 60 U.S. contracts, enabling data-driven selection of contract type, incentives, and share lines for use in evaluating future contract prices.

Keywords: Contracts, Strategy, Pricing Approach, Risk

AM03 – Flynn – Innovative Risk-Driven Contract Pricing Strategy – ppt
AM03 – Flynn – Innovative Risk-Driven Contract Pricing Strategy – paper

One Number to Correlate Them All – Efficient Implementation of Correlation
Analytical Methods Track (AM04)
Anh (Anne) Harris
Karen McRitchie
Christian Smart

Correlation is a key consideration in cost and schedule risk analysis, as its exclusion causes significant underestimation of uncertainty. When assigning values in the absence of functional correlation, this can be accomplished by considering every WBS estimate. However, this can be time consuming for a detailed estimate. In this presentation, we discuss an alternative method that uses a single value, which offers significant time savings and discuss its implementation in the SEER model suite.

Keywords: Monte Carlo, Parametrics, Risk, Uncertainty

AM04 – Harris – One Number to Correlate Them All – ppt

AM04 – Harris – One Number to Correlate Them All – paper

Incorporating Risk into Analysis of Alternatives Results: A Novel Methodology
Analytical Methods Track (AM05)
Brittany Clayton
Lauren Mayer

While Analysis of Alternatives (AoAs) commonly calculate the expected cost and effectiveness of alternatives, risks to cost overruns and degraded mission effectiveness are analyzed separately. RAND developed and employed a novel methodology for incorporating these risks into cost and effectiveness results. We collaborated with each AoA working group to identify, quantify and map risks to existing cost and effectiveness model variables, allowing for results to be displayed as a comparison of the alternatives’ risk-adjusted cost-effectiveness.

AM05 – Mayer – AoA Risk Methodology – ppt

Using Bayes’ Theorem to Develop CERs – Extending the Gaussian Model
Analytical Methods Track (AM06)
Christian Smart
David Jo

Bayes’ Theorem is a mathematical method to combine prior experience with new data and is extremely important in leveraging limited information, which is often the case in cost estimating. This paper is an extension of a previous ICEAA paper that dealt with the application of Bayes’ Theorem to cost estimating. In this update we show how the Bayesian approach for linear models can be extended for different, more realistic assumptions.

Keywords: Bayesian, Bias, Methods, Parametrics, Regression, Statistic

AM06 – Smart – Using Bayes Theorem to Develop CERs – ppt

AM06 – Smart – Using Bayes Theorem to Develop CERs – paper

Alternative Risk Measures for Determining Program Reserves
Analytical Methods Track (AM07)
Louis Fussell

NASA project managers hold cost and schedule reserves equal to a 50% joint confidence level and the managing directorates hold reserves to a 70% confidence level. These joint confidence levels are quantile risk measures. This paper discusses the drawbacks of quantile risk measures and proposes the use of super quantiles for determining project reserves. Several projects are analyzed and a comparison of risk measures is presented.

Keywords: Monte Carlo, Program Management, Risk Management, EVM & Scheduling, Risk, Planning

AM07 – Fussell – Alternative Risk Measures – ppt

AM07 – Fussell – Alternative Risk Measures – paper

The Space Between Us: A Novel Collaborative Spacecraft Estimating Framework
Analytical Methods Track (AM08)
Alex Wekluk
Benjamin Truskin

Government space acquisitions are renowned for their complexity and can suffer from notable cost and schedule overruns. The space cost community is composed of disparate organizations employing different cost estimating methods, even when the same industry partners are building the spacecraft. This paper leverages a Technomics internally-developed space estimating framework, industry-released Cost Estimating Relationships from multiple agencies, and NRO CAAG external collaboration experience to examine and contrast previously incompatible methods, and improve synchronization.

AM08 – Wekluk – The Space Between Us – ppt

AM08 – Wekluk – The Space Between Us – paper

Measuring Schedule Uncertainty
Analytical Methods Track (AM09)
Tommie (Troy) Miller

The growing popularity of Joint Cost & Schedule Analysis has highlighted the need for quality SRAs. SRAs require that uncertainty distributions be assigned to all schedule activities. The most popular technique for making this assignment employs a binning structure that assigns each task to a particular bin. Most binning structures have little theoretical support. This paper attempts to improve on this technique by discussing how uncertainty determination and allocation is conducted within the Sentinel program.

AM09 – Miller – Measuring Schedule Uncertainty – ppt

Spread Thin: Managing Coefficient of Variation in Monte-Carlo Based Cost Models
Analytical Methods Track (AM10)
Stephen Koellner

Coefficient of Variation (CV) can be utilized to determine whether sufficient uncertainty is captured in Monte-Carlo based estimates. This topic explores common barriers to capturing program level risk using the interpretation of a WBS as a linear combination of distributions. A WBS CV equation is provided to model perturbations of a baseline case and then randomized WBSs are generated to analyze CV at scale. Estimators can apply these insights to improve program estimate risk calculations.

AM10 – Koellner – Spread Thin Managing CV – ppt

AM10 – Koellner – Spread Thin Managing CV – paper

Data Science Track

Follow the Money in Government Estimates
Data Science Track (DS01)
Geoffrey P. Boals

Struggle with finding the correct source for actuals? Contracts for RDTE items are only bundles with no breakouts for the data you need? This session we will assist estimators in how to use government contract, Military Interdepartmental Purchase Requests (MIPRs), and other sources to find the relevant data needs for a strong basis for estimating. The session will detail a proposed data structure to store relevant data from the separate data sources to help power your estimates.

Keywords: Budgeting, Data Collection, DOD/MOD, Government, Variables

DS01 – Boals – Follow the Money in Government Estimates – ppt

DS01 – Boals – Follow the Money in Government Estimates – paper

“Cracking the Code” : Demystifying Programming Languages for Data Analytics
Data Science Track (DS02)
Kyle Ferris
John Maddrey

In recent years, buzzwords such as “Machine Learning” and “Artificial Intelligence” have made their way into the cost community’s vocabulary. While data science concepts such as these certainly advance the cost estimating industry, their significance is lost when analysts find themselves unfamiliar with fundamental programming principles. This presentation aims to demystify and compare commonly used programming languages such as R and Python, with the purpose of helping analysts who wish to develop coding expertise but don’t know where to begin.

Keywords: Data-Driven, IT, Methods, Modeling, Software, Statistics, Programming, Coding, Data Science, Data Analytics, Software Development

DS02 – Ferris – Cracking the Code – ppt

Level up your Cost Analysis with Data Science
Data Science Track (DS03)
Sarah Green

How can developments in the field of data science help us transform many of our analytics capabilities in the Cost Community? AFCAA is leveraging the AF’s cloud-based platform VAULT and state-of-the-art toolsets to in order to increase process automation, data curation and predictive models, and therefore significantly increase both quality and efficiency of our data analytics. This presentation will include a live demonstration including a demo of custom cost applications, tools and dashboards to help you in your everyday estimating- come see how the cost community is being revolutionized!

DS03 – Green – Level up your Cost Analysis – ppt

Web Applications for Cost Analysis Use: R Shiny vs Python Plotly Dash
Data Science Track (DS04)
Eric J. Hagee
Christine K. Clay

Web applications are a new tool for the cost community. Here, we compare and contrast two common web application tools, R Shiny and Python Dash. We develop a dashboard in each tool to display various USA government inflation measures and compare/contrast the use of either tool. Our findings are summarized, future thought points identified, and relevant background on the tools, inflation measures, use cases, and tool implementation provided.

Keywords: Communication, Data-Driven, R, Python, Shiny, Dash, Data Visualization, Web Application, Tools Comparisons, Dashboards

DS04 – Hagee – Web Applications for Cost Analysis Use – ppt

Using Machine Learning to Improve Cost Data Normalization
Data Science Track (DS06)
Bryan Mariscal

Cost data normalization is time intensive, but remains critical to building a defensible cost model. SSC’s Unmanned Space Vehicle Cost Model (USCM) database contains over 120 normalized datasets that span nearly a 50-year history. Developing machine learning models from the older data that generalize well to newer data would be difficult due to data obsolescence. However, we have identified an example use case with promising initial results that will be examined in detail.

Keywords: Data-Driven, Methods, Modeling, Machine Learning, Algorithms, Data Normalization, Process Improvement

DS06 – Mariscal – Using Machine Learning to Improve Cost Data – PPT

Automating the Data Preparation Process using R Programming
Data Science Track (DS08)
Zachary West

One of the most challenging aspects of the cost estimating process is the collection and consolidation of data prior to carrying out any analysis. Initial data collection and preparation steps are often manual, messy, and time consuming. I plan to present an automated solution using R Programming that efficiently collects and merges structured data reports, organizes and prepares the relevant data, and exports the results to support cost estimating efforts.

DS08 – West – Automating the Data Preparation Process using R – ppt

IT& Cloud Computing Track

Interesting Results from EVAMOSC or “Wow There is a Lot of O&S Data”
IT & Cloud Computing Track (IT01)
Daniel Germony

EVAMOSC is an new OSD-CAPE developed database which ingests and normalizes Operations & Sustainment (O&S) data on over a thousand weapon systems for the DoD. In this presentation, the EVAMOSC team will review interesting results it has found related to O&S data using the hundreds of millions of datapoints it has ingested and normalized. Pareto rule? Try hyper-Pareto rule! How did COVID impact vehicle maintenance? Not how you would have expected. Are field units experiencing inflation/escalation? All this and more.

IT01 – Germony – Interesting Results from EVAMOSC – ppt

Applying Simple Function Point Analysis to an 804 Rapid Acquisition Program Cost
IT & Cloud Computing Track (IT02)
Bob Hunt
Rainey Southworth
Chad Lucas

When software is developed under the Middle Tier of Acquisition (Section 804), Adaptive Acquisition Framework (AAF), these programs tend to be actual agile software development programs. These programs focus on delivering capability in a period of 2-5 years with rapid prototypes and rapid fielding with proven technology. Sizing a Software development program remains a major challenge. This paper outlines the advantages and issues associates with using Simple Function Point analyses to size and estimating the cost for a Major Rapid Acquisition Software Program.

Keywords: Agile, Acquisition, Software, Cost Estimation

IT02 – Hunt – Applying Simple Function Point Analysis to an 804 – paper

IT02 – Hunt – Applying Simple Function Point Analysis to an 804 – ppt

Cloud-Based Machine Learning in the DoD Environment
IT & Cloud Computing Track (IT03)
Conner Lawston
Bryan Eckle

Follow up to the 2022 Presentation “Supercharging Machine Learning with Cloud Computing.” This project has made huge strides in the past year- having made over 5,000 predictions since February, all 100% autonomously and within the DoD ADVANA environment. There have been numerous improvements (like increasing ML accuracy via PCA, & speeding up calculation time with SPARK) and lessons learned as we expanded into production for the Joint Chiefs, MDA, Navy, Marine Corps, and more!

Keywords: Modeling, Machine Learning, Cloud, Robotic Process Automation

IT03 – Lawston – Cloud Based Machine Learning – ppt

Addressing CI/CD with Automated Testing in Your Software Estimate
IT & Cloud Computing Track (IT04)
Arlene F. Minkiewicz

Agile development and DevOps/DevSecOps practices are becoming increasingly common in software development projects in commercial, government and government contractor organizations. Three practices arising are Continuous Integration, Continuous Delivery and Continuous Deployment. Automated testing is clearly an investment, but one that promises to lead to productivity and quality improvements. This paper presents the general concept of automated testing focused on practices highlighted above. The cost impacts of employing automated testing practices will be demonstrated and discussed.

Keywords: Software, Agile Software Development, DevSecOps

IT04 – Minkiewicz – Automated Testing – ppt

IT04 – Minkiewicz – Automated Testing – paper

Comparing the Cost of Cloud Services: AWS vs Azure
IT & Cloud Computing Track (IT05)
Gunnar Nichols
Patrick Casey
Sergey Kozin

Cloud Services continue to play a growing role in how the internet is used to provide capabilities to users. Amazon Web Services and Azure are the two biggest Cloud Service Providers on the market today. This research investigates cost differences for cloud services comprised of comparable virtual machines for workloads with defined computational requirements. This analysis is utilized to characterize scenarios where cost savings should be considered when supporting a Cloud Service Provider decision.

Keywords: Data-Driven, IT, Software

IT05 – Nichols – Comparing the Cost of Cloud Services – ppt

Shining Rays of Light & Savings on Cloud Portfolios: An Important Advance!
IT & Cloud Computing Track (IT06)
Kenneth Rhodes
Alex Wekluk
RJ Krempasky

Existing cloud cost estimating and pricing tools have at least two significant drawbacks: 1) they require many assumptions, therefore a detailed understanding of current/future architectures and 2) they do not provide insights required to manage costs. Technomics developed a groundbreaking alternative that will revolutionize how cloud costs are estimated and managed. This paper describes our parametric-based toolset, which enables cloud lifecycle costs estimation based on known programmatic design requirements and cost reduction and efficiency analysis.

IT06 – Rhodes – Shining Rays of Light – ppt

IT06 – Rhodes – Shining Rays of Light – paper

Modeling & Case Studies Track

Budget Execution and Margin Simulation (BEAMS)
Modeling & Case Studies Track (MC01)
Erik Burgess

ODNI works with organizations in the Intelligence Community to submit multibillion-dollar budget requests to Congress, based on cost estimates that are not exact. For an agency portfolio, this study quantifies the impact of budgeting to higher or lower confidence levels. By modeling the interactions among programs and the mechanics of the annual budget closure process, we show that estimates somewhat higher than 50 %tile are the “sweet spot” between minimizing growth and maximizing missions delivered.

Keywords: Budgeting, Space

MC01 – Burgess – Budget Execution Margin SImulation – ppt

Foundational Cost Models
Modeling & Case Studies Track (MC02)
Paul Broaddus Franklin Jr.

This study will unveil more than forty-five previously unpublished simple and learning-augmented cost estimating relationships for common construction materials and equipment. These relationships and other data allow us to investigate the hypothesis that the error distribution of conventional building components follow a skew exponential power distribution rather than the lognormal one usually applied to defense estimates. We also examine whether backcasting to the intercept can isolate approximate labor cost underlying composite expense data. This paper shares those relationships in the hope of expediting cost estimating from the ground-up.

Keywords: Parametric Estimation, Machine Learning, Construction, Infrastructure, Program Management, Operations Research, Defense

MC02 – Franklin – Foundational Cost Models – ppt

CSI EU (Cost Scene Investigation – European Union)
Modeling & Case Studies Track (MC03)
Doug Howarth

*2023 Best Paper: Modeling & Case Studies Category

In 2000, several Airbus units from European Union member countries launched the Airbus A380. The program was projected to sell 1250 units with a development cost of €9.5 billion but shut down after selling 251 copies, as costs more than doubled. This paper examines its causes and how the entire debacle might have been prevented, as all the information needed to assess its viability existed before it started.

Keywords: Development Costs, Sales, Demand

MC03 – Howarth – CSI EU – ppt

MC03 – Howarth – CSI EU – Paper

Trouble With the Curve: Engineering Changes and Manufacturing Learning Curves
Modeling & Case Studies Track (MC04)
Brent M. Johnstone

Engineering changes pose a dilemma for estimators: If learning curves assume cost improvement due to repetitive build, what happens when that repetition is interrupted by a change of task? Design changes are common occurrences, but rarely addressed in learning curve literature. This paper addresses how to analyze an engineering change by breaking it into its pieces and outlines techniques to calculate the reversionary impact on the learning curve to derive the estimated cost of change.

Keywords: Labor, Learning Curves, Methods, Modeling, Operations, Manufacturing, Engineering Changes

MC04 – Johnstone – Trouble With the Curve – Paper

MC04 – Johnstone – Trouble With the Curve

Developing a Schedule Model From a Cost Modeler’s Perspective
Modeling & Case Studies Track (MC05)
Benjamin Kwok
Daniel Newkirk

Cost estimators need to utilize and develop estimating methods for project components beyond cost. Schedule is one of those components. The team that brought you USCM is now applying their model building experience to the world of schedules. This journey has yielded some exciting new methods, products, and processes. It has also brought challenges as we could not address schedule exactly as we would with cost.

Keywords: Data Collection, Data-Driven, Modeling, Schedule, Databases, Metrics, Django, Python

MC05 – Kwok – Developing a Schedule Model – PPT

MC05 – Kwok – Developing a Schedule Model – paper

All in the Hierarchy: Meta-Estimators to Standardize Work Breakdown Structures
Modeling & Case Studies Track (MC06)
Maura Anne Lapoff
Patrick Shelton

The National Nuclear Security Administration (NNSA) has demonstrated how to use Machine Learning and Natural Language Processing to map disparate cost data to a standard, high-level Work Breakdown Structure (WBS). However, mapping data to deeper WBS levels becomes increasingly complex due to the hierarchical relationship between levels, rendering common machine learning models inadequate. Here we demonstrate how to implement a Hierarchical Classification Machine Learning scheme to map multi-level, hierarchical cost data to a common WBS.

Keywords: Data-Driven, Government, Modeling, Hierarchical, Construction, Classification

MC06 – Lapoff – All in the Hierarchy – paper

MC06 – Lapoff – All in the Hierarchy – ppt

The Cost of Bureaucracy
Modeling & Case Studies Track (MC07)
Zachary Matheson
Jenna Vandervort
Daniel Mulrow

The National Nuclear Security Administration (NNSA) has separate management methods for capital construction projects, with enhanced requirements for larger projects. Naively, one might expect that enhanced requirements might increase program management costs and possibly impact schedule. However, as the enhanced management requirements are tied to a dollar threshold, comparing costs is not straightforward. In this paper, we share results of a regression analysis that developed separate cost and schedule regressions for each management method and a comparison of the costs by management type.

Keywords: Decision Analysis, Early Cost, Government, Parametrics, Regression, Statistics

MC07 – Matheson – Cost of Bureaucracy – ppt

MC07 – Matheson – Cost of Bureaucracy – paper

Unprecedented – Accurate Estimating in the Hypersonics Era
Modeling & Case Studies Track (MC08)
Eric Sick
Christian Smart

Many cutting-edge programs and emergent domains, such as hypersonics, have few direct historical precedents. This presents challenges in estimating the cost of such programs at the system level. A key insight is that many of the components on these systems have been implemented on numerous other historical programs, so one way to successfully tackle this challenge is by estimating at a component-level Work Breakdown Structure. This paper provides comparative examples of accurate estimates for aerospace advanced technology programs, including commercial space and government hypersonics programs

Keywords: Cost/Benefit Analysis, Early Cost, Methods

MC08 – Sick – Unprecedented Accurate Estimating Hypersonics – ppt MC08 – Sick – Unprecedented Accurate Estimating Hypersonics – paper

Systems Engineering Cost Estimation Aligned with MIL-STD-881F
Modeling & Case Studies Track (MC09)
F. Gurney Thompson III
Vivian Tang

This presentation reviews advancements in estimating top-level Systems Engineering efforts. We will propose a new model structure for alignment with MIL-STD-881F definitions, and improved scope control and output usefulness. We will discuss the drivers of effort distribution within systems engineering activities and resources, including system criticality level, safety risks, logistics scenarios, human factors considerations, technology readiness level and more. This presentation also discusses ongoing data analysis efforts to support model validation and prototype demonstration.

MC09 – Thompson – Practical Estimating of Hardware Lifecycle – ppt

Cost Estimating: The Estimator’s Guide to a Parametric Universe
Modeling & Case Studies Track (MC10)
Taryn Anne Reilly

Imagining tomorrow’s cost estimating using parametric methods. Experience the advantages of parametric estimating throughout the opportunity and program lifecycle. Learn ways to significantly improve your estimating efficiency, accuracy and traceability through historical data calibration. Develop cost early so it influences the design process. Understand the importance of a standard WBS and curating completed program data to build calibrated models. Augment information in the Cost Community through data sharing, improving publicly available sources, methods, models.

MC10 – Reilly – Estimators Guide to a Parametric Universe – ppt

Processes & Best Practices Track

Shortfall Analysis – Creative Approaches to Problem Quantification
Processes & Best Practices Track (PB01)
George Bayer
Brian Carroll

Government acquisitions start with problem identification as a means of solving public sector problems or “shortfalls” and inefficiencies with technical hardware and software solutions. To justify government business cases, analysts are tasked to (1) define the problem, (2) identify impacted stakeholders, and (3) creatively quantify and monetize impacts to operations and services. Using a proven approach, the team demonstrates how to effectively perform shortfall analyses and monetize the largest underlying value to justify business cases.

Keywords: Cost/Benefit Analysis, Decision Analysis, Process Engineering, Regression, Statistics, Business Case, Shortfall Analysis, Critical Thinking, Root Cause Analysis, Cost Avoidance, Monetization

PB01 – Bayer – Shortfall Analysis – ppt

PB01 – Bayer – Shortfall Analysis – paper

You’ve Invested, Now Sustain it: Insights into the Sustainment Review Process
Processes & Best Practices Track (PB02)
Adam Kidwell
John Liss
Alex Bonich

Congress, in NDAA 2017, established Sustainment Reviews to assess the planning and execution of Major Defense Acquisition Program (MDAP) Product Support strategies. Readiness is a critical aspect of Defense Operations, and rising supply chain costs highlight the fact that effective planning, budgeting, and execution of a program’s Sustainment Strategy is crucial to its long-term success. This paper provides an objective, experience-based perspective of Sustainment Reviews and best practices to ensure successful outcomes.

Keywords: DOD/MOD, Life Cycle, Performance Management, Sustainment, Congress, Policy, Readiness

PB02 – Kidwell – Youve Invested Now Sustain – ppt

PB02 – Kidwell – Youve Invested Now Sustain – paper

Educating Future Cost Engineers in the Space Station Design Workshop (SSDW)
Processes & Best Practices Track (PB03)
Fabian Eilingsfeld
Nicolaus Millin

The space sector seeks young cost engineers who can work in a concurrent engineering environment. One means for scouting young talent is the annual Space Station Design Workshop (SSDW) at the University of Stuttgart, Germany. Two competing teams (size: 20) each must design a space station within one week. SSDW’s concurrent design facility (CDF) format offers students a unique opportunity to learn in a hands-on, international, and interdisciplinary environment. The use of cost estimating tools for teaching at SSDW is described. Lessons learned from the 2022 post-workshop survey are presented.

Keywords: Space, Education, Young Talent

PB03 – Eilingsfeld – Educating Future Cost Engineers – ppt

From Software CONOPS to ROM Estimate in Six Easy Steps
Processes & Best Practices Track (PB04)
Carol Dekkers
Dan French

Creating software estimates using a Concept of Operations (ConOPS) document is a challenge for software estimators due to the low fidelity of the requirements. This presentation will illustrate how to create a defensible ROM (Rough Order of Magnitude) Estimate using the new IFPUG software functional size measure. Carol and Dan will also explain how to use this size metric to develop a ROM Cost and Schedule by leveraging various software estimating techniques.

PB04 – Dekkers – From Software CONOPS to ROM Estimate – ppt

PB04 – Dekkers – From Software CONOPS to ROM Estimate – paper

Interviewing Subject Matter Experts
Processes & Best Practices Track (PB05)
Melissa Stone

As estimators, we often must obtain technical information from Subject Matter Experts (SMEs) in order to provide thorough and accurate analysis. But sometimes getting the information you need is easier said than done. This presentation discusses the steps for interviewing SMEs in order to make the most of their time and your time. It explores the benefits of building a cooperative relationship of mutual respect and the “dos” and “don’ts” of SME consultations.

PB05 – Stone – Interviewing Subject Matter Experts – ppt

NATO Transformation Command: Launching a Cost Estimating Capability
Processes & Best Practices Track (PB06)
Candace Mahala
Ryan Feeks
Michael Thompson

After over 70 years of establishing NATO, the Transformation Command recognized a cost estimating gap in its capabilities. They launched the development of a new cost estimating capability, for the pre-concept and concept stages remaining solution-agnostic at programmatic level. This is the story of how to grow and cultivate cost estimating from program inception, establishing the fundamentals, integrating with other NATO organizations’ cost communities and developing a closed-loop cost track for robust data.

PB06 – Mahala – NATO Transformation Command – ppt

Managing Schedule Risk Expectations During Program Execution
Processes & Best Practices Track (PB07)
Patrick Malone

Program managers use Integrated Master Schedules to complete projects. It is a time phased logical view of the work and activity network. Organizations identify risks using rule-based risk taxonomies; or selecting risk exposures and impacts from tables grading them low to high. These assessments are carried forward but not always implemented in the schedule. We investigate schedule risk methods to provide an approach to more accurate forecasting resulting in a process to implement across programs.

Keywords: Schedule, Risk, Program Management, Analysis

PB07 – Malone – Managing Schedule Risk Expectations – paper

PB07 – Malone – Managing Schedule Risk Expectations – ppt

Take the Blue Pill!
Processes & Best Practices Track (PB08)
Javier Provencio
Chris Hutchings
Cassidy Shevlin

Digital ecosystems are distributed technical systems that are scalable, sustainable, and often self-organizing; much like ecosystems found in the natural world. We will offer insight into what a digital ecosystem encompasses, along with relatable examples, and then explicate the virtues and value for our community in areas such as on-demand design to cost, MBSE, in-service support, and simulation. Finally, we will examine emergent relevant and significant use cases.

Keywords: Cost Management, Cost/Benefit Analysis, Data-Driven, Decision Analysis, Infrastructure, Methods, Operations

PB08 – Provencio – Take the Blue Pill – paper

Accidental Proof of Calculus for Cost Analysts
Processes & Best Practices Track (PB09)
Jack Snyder
Brittany Smith

Ever get derailed chasing your tail checking calculations, only to discover the cross check initiating the train wreck was flawed to begin with? Join us on an unflattering journey that began with a quick integral cross check that snowballed into the perfect proof for why calculus is a requirement for cost analysts. MATLAB, calculator, or by hand were the academic tools of the calculus trade, we’ll also share how we use Excel for integral calculus.

PB09 – Snyder – Accidental Proof of Calculus – ppt

Methodology for Assessing Reasonableness of Large Scientific Facilities’ Costs
Processes & Best Practices Track (PB10)
Raymond D. Woods
Valerie Rockwell

Aerospace analysts developed a methodology to address the questions and challenges that arise when looking at reasonableness of large scientific facilities’ costs. This methodology is an iterative process of developing tailored reasonableness ranges and systematically addressing lower WBS levels to refine estimates. Aerospace will share their experience in using this method, when it is applicable, the benefits of implementing this, and associated products – including a specialized ‘Scorecard’ – to provide insights to program management.

PB10 – Woods – Assessing Reasonableness of Large Scientific Facilities – ppt

Software & Agile Track

NLP: A New Approach to SFP Estimation
Software & Agile Track (SA01)
David H. Brown

The need for data-driven methods for predicting software size has never been greater. This is especially true for agencies that rely on SME opinion or T-shirt based sizing as the primary method for estimating the size and cost of agile-developed software. This paper offers an alternative method — use of Natural Language Processing to automate the estimation of Simple Function Point counts – that offers two important advantages: 1) consistency across multiple counts and 2) speed.

Keywords: Data-Driven, Early Cost, Software, Agile, SFP, NLP

SA01 – Brown – NLP A New Approach – ppt

SA01 – Brown – NLP A New Approach – paper

The Symbiotic Nature of Requirements Quality and Software Cost Estimation
Software & Agile Track (SA02)
Colin Hammond

*2023 Best Paper: Software & Agile/IT & Cloud Computing Category

It is widely recognised that high quality software requirements lead to reliable functional size estimates, what is less apparent is the potential contribution of functional sizing to help assess and improve the quality of requirements. This paper looks at the observations from assessments of software user stories using our NLP (Natural Language Processing) requirements analysis tool, and how this symbiotic relationship is revealed and enhanced with automated analysis tooling.

Keywords: Cost/Benefit Analysis, Data-Driven, NLP, Quality, Sizing, Function Points, Cost Reduction, Reduce Risk, Reduce Waste

SA02 – Hammond -Symbiotic Relationship between Requirements Quality – Paper

SA02 – Hammond -Symbiotic Relationship between Requirements Quality – ppt

A Novel Approach for Early Phase Agile Software Estimating and Sizing
Software & Agile Track (SA03)
Wilson Rosa
Sara Jardine

This study investigates how well three high-level size measures (Capability Gaps, Capabilities, and Epics) accurately predict total agile development effort and schedule at the earliest acquisition phase. These measures were obtained from early program documents including Mission Needs Statement, Concept of Operations, and Release Roadmap. Analysis of Variance and Goodness-of-Fit were used to introduce six new estimation models based on data from 20 agile projects implemented between 2014 to 2022 in the DHS and DoD.

Keywords: Data-Driven, IT, Parametrics, Regression, Software, Agile Software Development, Cost Estimation, Mission Needs Statement, Analysis of Alternatives, Rough Order Magnitude, Capability Gaps, Capabilities, Epics, Software Acquisition, Software process, Time Estimation

SA03 – Rosa – Novel Approach for Early Phase Agile – ppt

SA03 – Rosa – Novel Approach for Early Phase Agile – paper

Scheduling Agile-Fall – Best Practices
Software & Agile Track (SA04)
Sam Kitchin
Ryan Webster

A high-quality schedule can improve a program’s ability to clarify requirements, predict cost, and communicate effectively by creating a common denominator that links complex efforts across a project. Many Agile programs operate using a hybrid waterfall approach making a properly built schedule critical for understanding the relationship between agile development and government requirements. This presentation will demonstrate best practices for schedule construction in an Agile environment and its implications on cost, delivery, and performance.

SA04 – Kitchin – Scheduling AgileFall – ppt

Tales from the Trenches – Challenges of Data Analytics in a DevSecOps Program
Software & Agile Track (SA05)
Alex Smith

The large databases of information captured in software planning tools within DevSecOps environments contain a wealth of information. However, this data is user-input and not created with data analysis in mind, which results in several challenges when attempting to analyze. This presentation walks through the real-world process that resulted from collecting, cleaning, reviewing, and analyzing over 20,000 observations of agile software data to facilitate DevSecOps planning and road-mapping activities for a program entering the Software Acquisition Pathway.

Keywords: Government, IT, Story Points, DevSecOps, Data Analysis

SA05 – Smith – Tales from the Trenches – ppt

How Efficient are Your Efficiency Levers?
Software & Agile Track (SA07)
Eric van der Vliet

For application service projects an important aspect is making services more efficient by means of efficiency levers like juniorization, test automation, robotics, etc. The challenge is to combine these efficiency improvements and determine the effort and cost impact during the maintenance period. This presentation presents lessons learned with respect to applying efficiency levers in large application service projects and shows an estimation tool that supports a combination of multiple efficiency levers with different impacts and durations and calculates the monthly impact.

Keywords: Budgeting, Data-Driven, Parametrics, Software

SA07 – vander Vliet – How Efficient are Your Efficiency Levers – ppt

Technical Innovations Track

Ticking Time Bomb of Telework: Communication and Time Management
Technical Innovations Track (TI01)
Jennifer Aguirre
Shannon Cardoza
Annie Bagay

Today’s evolving work environment has demonstrated the importance of emphasizing soft skills in the workplace. Employers must seek candidates with high interpersonal abilities in conjunction with technical expertise. Hard skills often dominate in job-related training, while critical foundations like the ability to communicate and manage workload effectively are often overlooked. This presentation will focus on expanding a person’s marketability by discussing techniques to open these communication pipelines and increase productivity despite an ever-growing workload.

Keywords: Communication, Program Management

TI01- Aguirre – Ticking Time Bomb of Telework – ppt

AI and Cost Estimation: Data Science’s Expanding Role in Cost Estimating
Technical Innovations Track (TI03)
Kevin McKeel
Daniel Harper
Greg Forman

Data Science has taken on an expanded presence in Cost Analysis. E.g., NLP is used to automate functional software sizing in commercial models. Data Science methods and tools such as NLP can be also used to extract data to identify spending trends and agency funding. We will present an overview of modern usages of data science, to include Machine Learning, AI and data visualization. We will also survey cost and budget data sources which are publicly available and ripe for analysis using data science methods.

Keywords: Artificial Intelligence, Data Science, Machine Learning, NLP

TI03 – McKeel – AI and Cost Estimation – ppt

Software Estimation Using Functional Size Derived From User Stories
Technical Innovations Track (TI04)
Karen McRitchie
Esteban Sanchez
Alton Ng

Natural language processing and ISO standard functional size metrics offer an opportunity for an improved approach to sizing and estimation directly from user stories. This presentation will offer an overview of how functional size generated by ScopeMaster can be used for directly for estimation in SEER-SEM. This approach uses either Cosmic or Simple Function Points and considers the uncertainty around sized, ambiguous and potentially missing requirements.

Keywords: Agile, Functional Requirements, Story Points, Sizing

TI04 – McRitchie – Software Estimation Using Functional Size – ppt

Software Maintenance and Software Obsolescence: A Tale of Two Cost Drivers
Technical Innovations Track (TI06)
Sanathanan Rajagopal

Software Obsolescence is one of the key issues in the modern warfare technologies especially in the cyber space where vulnerabilities of obsolete software are exploited. Understanding these vulnerabilities, predicting, managing them cost effectively are now key priorities of any key defence contracts. In this original research paper the author looks at the relationship of software maintenance and its key cost drivers with software obsolescence. Author will further explore the appropriate mitigation strategy and resolution approach for managing software obsolescence while providing an insight o

Keywords: Cost Management, Parametrics, Software, Software Maintenance, Software Obsolescence

TI06 – Rajagopal – Software Maintenance and Obsolescence – ppt

Wouldn’t It Be Nice to Use All of Your Data Points: An Introduction to NICE
Technical Innovations Track (TI07)
Steven Read

*2023 Best Paper: Processes & Best Practices/Technical Innovations Category

The difficulty of having non-homogeneous data when fitting a CER can be addressed using data imputation. A Non-linear Iterative Constrained Estimator (NICE) is an approach to both data imputation and CER fitting that considers the whole model and uses the WBS relationships to recalibrate systems of equations for a better fit. This presentation will explain what NICE is, how it works, and begin to explore what problems this method can solve.

Keywords: Data-Driven, Methods, Modeling

TI07 – Read – Wouldn’t It Be Nice – ppt

TI07 – Read – Wouldn’t It Be Nice – Paper

VEGA: Shining Light on the Battles of Data Collection and Management
Technical Innovations Track (TI08)
Abby Schendt
Omar Akbik
Raymond Vera

Every cost organization battles both the collection and management of cost, schedule, technical, and programmatic data. This paper describes how the Office of Cost Estimating and Program Evaluation (CEPE) at NNSA has built and matured this capability since the office’s inception in 2015. This paper will present seven years of lessons-learned that will pay dividends for any cost organization looking to optimize how it collects and manages mission critical data.

Keywords: Data Collection, Data-Driven, Government, Database, Data Storage, Policy, Data Structure, Data

TI08 – Schendt – VEGA – ppt

TI08 – Schendt – VEGA – paper

CEBoK-S Training Track

CEBOK, CEBoK-S and Certification Exams Overview
CEBoK-S Training Track (CEBoKIntro)

An introductory session to outline ICEAA’s our knowledge base and product line. Our instructors will provide an overview of the ICEAA certification exams, how to prepare, and answer your questions.

CEBoK-S Lesson 0: Overview & Curriculum
CEBoK-S Training Track (CEBoK00)
Kevin Cincotta
Jennifer Kirchhoffer

This short lesson outlines the goals and objectives for CEBoK-S and where it fits in the context of ICEAA’s product line. The lesson provides a high-level roadmap for the CEBoK-S curriculum and gives acknowledgements to the contributing subject matter experts and industry leaders who were involved in CEBoK-S v1.0 development.

CEBoK-S Lesson 1: Introduction to Software Cost Estimating
CEBoK-S Training Track (CEBoK01)
Bob Hunt

This lesson explains the importance of software cost and schedule estimating for CEBoK-S audiences, and includes historical contexts, examples of software program failures and success, and highlights what students can gain from CEBoK-S. Note that CEBoK-S is intended to supplement the audience’s understanding of the cost estimating tools and techniques covered in CEBoK®, introducing the differences between software and hardware estimating and what is included in systems vs. software cost estimating.

CEBoK-S Lesson 2: Software Paradigms
CEBoK-S Training Track (CEBoK02)
Eric van der Vliet

This lesson introduces the two major paradigms involved in custom software development, predictive and Agile, and presents a variety of software development methods and where they differ from perspective of software cost estimation of custom software development efforts. Included are highlights of waterfall, iterative, spiral, Agile, Scrum, SAFe, DAD, and other approaches to give the software cost estimator a basic understanding of the activities and cost drivers involved.

CEBoK-S Lesson 3: CEBoK-S 5-Step Estimating Process
CEBoK-S Training Track (CEBoK03)
Jennifer Kirchhoffer

This lesson outlines the major considerations of our CEBoK-S Five-step Estimating Process for custom software development including the pre-requisite data, cost drivers, and considerations involved in the software development process. We will identify and discuss types of estimates for custom software development efforts and present the CEBoK-S 5-step software estimating process.

CEBoK-S Lesson 4: Estimating Custom Software Development
CEBoK-S Training Track (CEBoK04)
David H. Brown
Kevin Cincotta

Lesson 4 outlines where we use relevant data & custom-built software Cost & Schedule Estimating Relationships (CER & SER) to estimate software development effort, cost and schedule duration. We will present three ways to create the software estimate using estimating techniques: an overview of estimating approaches, how to create a software development estimate based on CEBoK-S Five-step Estimating Process and four estimating approaches, developing a schedule estimate, time-phasing the estimate, cross-checking the estimate, and reviewing estimates prepared by others.

CEBoK-S Lesson 5: Software Sustainment
CEBoK-S Training Track (CEBoK05)
Carol Dekkers

Software sustainment is a definable, robust term, used to describe the second stage of the software life cycle, in place of the term Software Operations and Support (O&S). The first stage, called the investment stage, has been covered in other lessons. Software sustainment is a parent term for many types of software-related work as we’ll outline in this lesson. Software maintenance, in the context of CEBoK-S, is defined to be a child component within software sustainment.

CEBoK-S Lesson 6: Estimating Procured Solutions
CEBoK-S Training Track (CEBoK06)
Arlene F. Minkiewicz

Identify key characteristics of cost estimating considerations for, and differences between Commercial Off-the-Shelf (COTS) software and Enterprise Resource Planning (ERP) systems among other procured software solutions. This lesson will provide an introduction to procured software solutions, describe COTS software and the major types of procured software solutions, and cost drivers and estimating approach(es) for each. We will consider cost growth and sustainment in procured software solutions.

CEBoK-S Lesson X: Software Size
CEBoK-S Training Track (CEBoKX)
Carol Dekkers

This lesson focuses on the various units of measure for software size, options and methods for their measurement, and how to select an appropriate unit of measure suited to your estimate. We also discuss conversions and rules of thumb, and then finally how to ensure whatever size estimate you are given as the software cost estimator can be validated with a checklist of questions.

CEBoK-S Lesson Y: Productivity
CEBoK-S Training Track (CEBoKY)
David H. Brown
Kevin Cincotta

This lesson outlines the complex topic of productivity, which is often used interchangeably with several other terms including throughput, delivery rate and even velocity (in agile software development.) We discuss productivity as a concept and in general, before delving into what constitutes productivity, how to measure it, and the relationship between size and productivity.

CEBoK-S Lesson Z: Commercial Estimating Models
CEBoK-S Training Track (CEBoKZ)
Karen McRitchie

This lesson highlights how various commercial estimating models provide support to our CEBoK-S Five-Step Estimating Process, and how they support creating the custom software development (Lesson 4), procured software solutions (Lesson 6), and software sustainment (Lesson 5) estimates. We will use these models to cross-check out our Midnight and FFS examples from Lesson 4, using the COCOMO II Web Tool, Galorath Inc;s SEER-SEM®, Unison’s TruePlanning® for Software, Quantitative Software Management (QSM)’s Slim Estimate®and additional models including Namcook Consulting LLC’s Software Risk Master (SRM) and COCOMO II-based automated estimating tools.

2023 Workshop Breakout Sessions

Analytical Methods Track

Data Science Track

IT& Cloud Computing Track

Modeling & Case Studies Track

Processes & Best Practices Track

Software & Agile Track

Technical Innovations Track

Trending Topics Track

CEBoK-S Training Track

Upcoming Events

WCA Chapter Virtual Luncheon: Well, That Escalated Quickly – A Novel Approach to Forecasting Escalation

2025 Professional Development & Training Workshop