2024 Workshop Breakout Sessions Browse by track:Analytical MethodsData Science & Machine LearningManagement, EVM & RiskModelingProcesses & Best PracticesSoft Skills & Trending TopicsSoftwareStrategyCEBoK® Training Track Analytical Methods Track Data-Driven Lifecycle Analysis to Optimize Cost, Risk, and Sustainability Analytical Methods Track (ANM01) George Bayer Brian Carroll Many government infrastructure investments adhere to a standard lifecyle to estimate program cost, plan replacement timing, and compare business cases to one another in a cost-benefit analysis. What if those lifecycle replacement timelines are inconsistent with system sustainability and are not cost-effective? Some infrastructure systems which are replaced according to an end-of-life schedule can be sustained more cost-effectively for longer periods of time via preventative maintenance. Our team examined multiple infrastructure program replacement timelines and analyzed operational effectiveness, cost/risk trade-offs, system redundancy, and sustainability, and we recommended lifecycle adjustments based on those considerations. We reduced overall program cost by extending replacement timelines, eliminating system redundancy without compromising sustainability, and reprioritizing maintenance portfolios on critical backlogs. We document a comprehensive process to customize program lifecycles to optimize cost, risk, and sustainability. Keywords: Business Case, Cost-Benefit Analysis, Lifecycle, Statistics, Data Analysis, Critical Thinking, Cost Avoidance, Regression, Forecasting, Process Analysis, Monetization ANM01-Bayer-Data-Driven-Lifecycle-Analysis-PPT ANM01-Bayer-Data-Driven-Lifecycle-Analysis-Paper Triage the Sub-Projects: Calculating and Applying Portfolio Contingency Analytical Methods Track (ANM02) Stephen Koellner Nick Peeples Risk-adjusted cost estimates are needed to understand the potential range of actual costs through execution. Cost risk analysis produces uncertainty distributions which can be used to calculate an expected cost as well as contingency, which can be thought of as the difference between expected cost and a higher confidence level chosen for planning purposes. In a portfolio of projects, allocating uncertainty at the portfolio level will result in a different risk-adjusted cost than applying the same allocation at the project level, and so it is unclear whether a portfolio should allocate and manage risk-informed contingency at the portfolio or project level. This topic will explore methods for calculating portfolio contingency, using a tangible example to demonstrate. Keywords: Budgeting, Cost Management, Decision Analysis, Program Management, Project Controls, Uncertainty ANM02-Koellner-Triage-the-Subprojects-Paper ANM02-Koellner-Triage-the-Subprojects-PPT Things Forgotten Since College – Foundational Statistics Analytical Methods (ANM03) Jordan Harlacher Kyle Davis Statistical analysis is one of the foundations of cost estimating, but fundamentals are easy to overlook. This presentation will help ensure that is not the case for your next estimate as we will discuss how the data collection and organization processes can form the basis for your estimate. Once the relevant data has been collected and organized, the real fun begins, as the central tendencies and variability of the data can now be examined. The central tendencies and variability can be used to determine the most applicable distribution and assess the probability of different events occurring. We will examine the best ways to visualize different data sets, using charts and graphs to convey the information clearly to stakeholders, as visualizing the data can help inform relationships between variables. Finally, we will touch on key statistics to look for in your regression analysis to ensure a meaningful relationship is defined. Keywords: Data Collection, Regression, Statistics ANM03-Harlacher-Things-Forgotten-Since-College-PPT Stretching Purchasing Power through Improved Escalation Methods Analytical Methods Track (ANM04) Amanda Schwark Matthew Siiro Shahriar Rayhan Carson Lo Escalation methods ensure cost estimates adapt to economic changes and facilitate accuracy and reliability. The NNSA chartered the Programmatic Recapitalization Working Group (PRWG) to track mission-critical equipment directly supporting weapons activities across the NSE. The PRWG maintains a comprehensive database of equipment above the NNSA capital acquisition threshold of $500,000. The previous escalation methodology for equipment purchase price was limited to using a single equipment inflation index. Additional fidelity in price projections can be achieved by leveraging empirical price data and published indices to derive escalation rates specific to various equipment categories. This paper explores our approach to improving upon the previous escalation methodology to better inform planning and programming decisions. This approach can be leveraged when one broad escalation index is used to predict costs for many significantly differing data elements. Keywords: Data-Driven, Regression, Escalation ANM04-Schwark-Stretching-Purchasing-Power-Paper ANM04-Schwark-Stretching-Purchasing-Power-PPT Spacecraft Design to a Cost Target Analytical Methods Track (ANM05) Ryan Sukley Perfect performance of every system is critical for space missions. Identifying capable designs is a challenging process, and one that often comes at the expense of exceeding cost targets. The Cost as an Independent Variable (CAIV) approach helps mitigate this issue by treating cost as a primary consideration in the design or procurement of systems. Establishing a fixed cost target sets a ceiling for the cost versus performance trade-off and, in the case of NASA’s in-house spacecraft, enables more cost-conscious decision making. This paper examines the application of CAIV to identify upper bounds for parameters (mass, power, quantity, etc.) early in the process of designing a spacecraft that satisfies mission requirements. It describes the process of developing, maintaining, and explaining the limitations of this capability, and addresses potential applications of the approach to other commodities. Keywords: Cost Management, Data-Driven, Early Cost, Parametrics, Space ANM05-Sukley-Spacecraft-Design-Cost-Target-Paper ANM05-Sukley-Spacecraft-Design-Cost-Target-PPT Early-Stage Cost Growth CER Development Analytical Methods Track (ANM06) Gabriel Sandler Haley Harrison Capital acquisition projects at the National Nuclear Security Administration (NNSA) have experienced significant early-stage cost estimate growth, driven in part by early optimism and missed scope. To account for these potential scope changes, NNSA’s Office of Programming, Analysis, and Evaluation (PA&E) developed a cost estimating relationship (CER) for construction projects which relates the actual total project cost (TPC) to its early-stage scope estimate. This methodology differs from usual CERs which model actual cost as a function of actual scope, but reflects the scope uncertainty NNSA projects have at early stages. Three cost drivers (gross square footage, hazard category, and equipment complexity) were selected as the variables to solve for the TPC. The results of the CER were compared to another PA&E CER built with actual scope and actual costs so that early-stage cost estimate growth at the NNSA for various types of capital acquisition projects could be quantified. Keywords: none provided ANM06-Sandler-Early-Stage-CER-Development-PPT Market Dimensional Expansion, Collapse, Costs, and Viability Analytical Methods Track (ANM07) Douglas K. Howarth *2024 Best Paper: Analytical Methods Category -and- Best Paper Overall Most government programs set out with cost caps and minimum force requirements. Commercial projects usually begin with a budget, sales targets, and specifications. All too often, in both cases, producers and customers give little thought to the changing market structures they face. When it comes to Demand, markets self-organize to form up to four boundaries each, including 1) Upper (price-limited), 2) Outer (saturation-limited), 3) Inner (efficiency-limited), and 4) Lower (margin-limited) Demand Frontiers. When new market segments appear as different product forms with enhanced functionality over existing options, as the new markets grow, the product groupings they replace may contract across one or more Demand Frontiers. This paper examines preparing for these inevitable eventualities in an N-dimensional framework. Keywords: Functional Requirements, Modeling, Viability, Multidimensional ANM07-Howarth-Market-Dimensional-Expansion-Paper ANM07-Howarth-Market-Dimensional-Expansion-PPT Comparison of UMP in the Great Recession and the Covid-19 Recession Analytical Methods Track (ANM08) Nathan Gallacher This piece aims to produce a review of the Unconventional Monetary Policy (UMP) used in both the Great Recession 2007-09 and the COVID-19 Recession, then compare the two recessions to show how unconventional monetary policy changed, differences in tools used by the Bank of England and the size of the tools put in place. Notably, tools such as quantitative easing see use in both recessions suggesting similarities in the aims of the Bank of England during both recessions. The main results show a significant increase in the use of unconventional monetary policy from the Great Recession to the COVID-19 Recession. At the same time, inflation outcomes were worse during the COVID-19 Recession. This suggests that the greater reaction by the BoE in the use of UMP towards the COVID-19 Recession may not have been as effective in controlling inflation compared to the Great Recession. Keywords: Decision Analysis, Government, Microsoft Excel, Modeling, Economics, Monetary policy, Unconventional monetary policy, policy analysis ANM08-Gallacher-Great-Recession-Covid-Recession-Paper ANM08-Gallacher-Great-Recession-Covid-Recession-PPT Data Science & Machine Learning Track Explosive Analysis: Using Data to Hold Warfare Centers Accountable Data Science & Machine Learning Track (DML01) Ryan Webster Robel Semunegus Taylor Fountain The Joint Service Explosive Ordnance Procedure Publications program creates and maintains critical documents for the safe handling of ordnances. This effort is managed by Naval Warfare Centers. Historically, senior leadership has funded these efforts without the ability to evaluate reasonableness of annual funding requests. Augur has recently obtained publications system data, resulting in valuable analysis of historical efforts. This data is being leveraged to develop a planning calculator, capable of estimating ranges of labor hours based on ordnance type, country of origin, and other complexity drivers derived through regression analysis and other visualization techniques. This tool and the accompanying insights will enable senior leadership to negotiate with warfare centers and more easily measure performance. Keywords: Data Science, Parametric, Performance Management, Cost Tools, Regression DML01-Webster-Explosive-Analysis-Paper DML01-Webster-Explosive-Analysis-PPT Maximizing Analysis of Minimalized Datasets Data Science & Machine Learning Track (DML02) Taylor Fountain Obai Kamara *2024 Best Paper: Data Science & Machine Learning Category Many techniques exist to determine parametric relationships within large datasets. While cost estimation relies heavily on identifying such relationships, a data-scarce environment, driven by factors such as vendor proprietary restrictions, security concerns, and the uncertainty of emergent technologies, is a common barrier in implementing these techniques. This topic will evaluate common methods to analyze minimalized datasets for developing defendable cost estimates, such as complexity factors and 3-point distribution fitting, and demonstrate the statistical impacts of their underlying assumptions. Keywords: Data-Driven, Methods, Monte Carlo, Parametrics, Regression, Uncertainty, Data Science DML02-Fountain-Maximizing-Analysis-Minimalized-Datasets-Paper DML02-Fountain-Maximizing-Analysis-Minimalized-Datasets-PPT Labor Mapping in Parametric Estimates Data Science & Machine Learning Track (DML03) David Ferland Tristan Judd Contractors and Original Equipment Manufacturers (OEM) alike often struggle applying specific resources or labor categories to their parametric estimates. Many parametric modeling techniques produce hours by generic resources that still need to be translated into labor resources that have rates and other attributes before they can be useful for analysis. I will outline a tool development framework that fills this gap and allows the cost estimates to stay in-sync with downstream tools like ProPricer that may compile the final estimate. This case study uses TruePlanning® as an input to the pipeline but can be applicable to most parametric sources. In cases where Basis-of-Estimates (BOEs; as opposed to Realistic Cost Estimates or RCEs) using proposed resource hours are still being required to justify parametric estimates, the traceability and justification of these pipelines is also an important consideration. Keywords: Data Collection, Labor, Parametrics DML03-Ferland-Labor-Mapping-Parametric-PPT Data Cleaning in Python for Beginners Data Science & Machine Learning Track (DML04) Alexis Somers As cost estimators, we collect large amounts of data from many sources, and it’s often messy. Cleaning and organizing the data often requires time-consuming manual effort before proper analysis can begin. Using Python to clean and manipulate data is one of the easiest ways to save time and maximize efficiency when working on cost or other data analyses. As a free, beginner-friendly, and versatile tool, Python is an excellent choice for processing and analyzing data. This session will cover how to get started using Python to create simple scripts that produce clean, organized data. We will use the pandas and NumPy libraries to clean datasets by correcting errors, reformatting data, handling missing values, adjusting for outliers, and more. The ability to create simple Python scripts can improve the quality of your cost estimates and other deliverables by improving accuracy, efficiency, and saving time. Keywords: Cost/Benefit Analysis, Data Collection, Data-Driven, Python, Data Cleaning, Automation DML04-Sommers-Data-Cleaning-in-Python-for-Beginners-PPT Going Beyond Count-Based Methodologies with Semantic Vector Embeddings Data Science & Machine Learning Track (DML05) Trevor Lax David Ferland Machine Learning (ML) is a topic of persistent interest and a frequent buzz word because of the astounding capabilities it has shown across disparate fields. However, the complexity of ML combined with the overwhelming number of options can lead to decision fatigue and reduced understanding in new users. While much attention is duly focused on the data and machine, occasionally the basic components of ML, such as input data type, are not properly interrogated. Indeed, a frequently used Natural Language Processing method, Term Frequency – Inverse Document Frequency (TF-IDF), simply uses counts, which cannot encode syntactic or semantic information. An alternative to TF-IDF, Word-2-Vector, creates vector embeddings of the words in a corpus, instead of relying on sentence-level counts, and attempts to encode Semantic information. Word-2-Vector has its own limitations, such as the need for a large corpus, however, it can allow for better performance and greatly improved flexibility. Keywords: Data-Driven, Parametrics, Variables, Machine Learning DML05-Lax-Analyzing-Vector-embeddings-PPT Automation and Process Improvement in Cost Estimating Data Science & Machine Learning Track (DML06) Anil Divvela In the last couple decades, there has been a wave of innovation across all Industries to streamline data analysis processes and cut costs with the introduction of data science tools. I will demo simple Python use cases that will automate time consuming redundant tasks that every Cost Estimator hates! Keywords: Python, Automation, Process Improvement DML06-Divvela-Automation-Process-Improvement-Cost-Estimating-PPT AI and Machine Learning/Data Science Tools for Cost Analysis Data Science & Machine Learning Track (DML07) Daniel Harper Kevin McKeel AI and Machine Learning/Data Science Tools such as Chat GPT have taken on an expanded presence in Cost Analysis. E.g., NLP is used to automate functional software sizing in commercial models. Large Language Models (LLM) may even have applications for cost and acquisition professionals. We will present an overview of modern usages of data science, to include Machine Learning, AI and data visualization. We will also provide several use cases for applying these tools in cost estimation. Keywords: Artificial Intelligence, Data Science, Machine Learning, NLP DML07-Harper-AI-Cost-Estimation-PPT Costing Web App Development for Operations Research Data Science & Machine Learning Track (DML08) Kyle Ferris Eric J. Hagee Commercial-off-the-shelf (COTS) web application development platforms empower analysts to leverage low-code environments to build comprehensive business tools. Therefore, understanding the lifecycle cost requirements to design, develop, deploy and maintain low-code web applications as both analytical and decision support tools for stakeholders is of interest to the cost community. We define web application lifecycle requirements as analogous to an overarching Data Operations Stack. The Data Operations Stack is a conceptual framework that describes data operations as a set of hierarchical requirements, from base-level IT infrastructure and tools to high-level business products. With this framework in mind, we describe web application lifecycle requirements through successive levels of the Data Operations Stack, elucidating the required personnel, tools, and capabilities integrated into each level. Finally, we discuss how an understanding of interconnected dependencies across the Data Operations Stack can be used to develop defensible cost estimates and manage resources for web application lifecycle requirements. Keywords: Communication, Data-Driven, Data Science, Web Applications, Web Apps, R, Shiny, Programming Languages DML08-Ferris-Costing-Web-App-Operations-Research-PPT From a Man-Month to an AI-Minute, Myth or Reality? Data Science & Machine Learning Track (DML10) Colin Hammond In this session I will share some of our discoveries of using AI over the last five years that can help software cost estimators and our thoughts on how AI will be changing software development costs in the coming years. Back in 1975 Fred Brooks discussed observations of software engineering, many of which are counter-intuitive in a book entitled The Mythical Man Month, we pay homage to his book title in this talk as we share some observations and quantifications of how AI is helping to improve early software estimation. I will also share our predictions on areas where AI will help accelerate software development and impact on software costs over the next few years. Keywords: AI, NLP, cost estimation, software estimation DML10-Hammond-Man-Month-AI-Minute-PPT Implications of Generative AI (Artificial Intelligence) in Software Engineering Data Science & Machine Learning Track (DML11) Arlene F. Minkiewicz Generative AI is positioned to revolutionize software development, with potential far reaching implications for productivity. Generative AI applications leverage Large Language Models to understand language, imagery and code, then use what they learned to generate content; answering questions, organizing multimodal information, and writing text and code snippets. A McKinsey report from 2023 reports that the software development landscape is quickly changing as Generative AI applications such as ChatGPT and Github Copilot have the potential to enable software engineers to complete development tasks; achieving as much as 2x productivity over traditional development practices. Activities such as inception and planning, system design, coding, testing, and maintenance can all be aided through applications of Generative AI. This paper will include an introduction to Generative AI in the software engineering context. Following will be a discussion of productivity impacts and guidance for incorporating them into a software estimates. Keywords: Software, Machine Learning, Generative AI, ChatGPT DML11-Minkiewicz-Impact-of-Generative-AI-Paper DML11-Minkiewicz-Impact-of-Generative-AI-PPT Distribution Free Uncertainty for CERs Data Science & Machine Learning Track (DML12) William King Shaun Irvin For this presentation we intend to introduce and demonstrate the application of conformal prediction as a tool to specify prediction intervals for any machine learning algorithm. Conformal prediction intervals offer rigorous statistical coverage guarantees without distributional assumptions and only require the exchangeability of data (a weaker assumption than independence). Moreover, generating these prediction intervals is an easy consequence of retaining the sub-models trained during k-fold cross-validation. Specifically, we intend to summarize the “CV+ for K-fold cross-validation” method (and its locally weighted variant) from Predictive Inference with the Jackknife+ (Barber, Candes, Ramdas, Tibshirani, 2021, The Annuals of Statistics), and show how conformal prediction enables distribution free uncertainty for CERs. Additionally, we plan to discuss how this technique can be applied to direct human-in-the-loop intervention when applying machine learning models. Keywords: Methods, Modeling, Regression, Risk, Statistics, Uncertainty, Conformal Prediction DML12-King-Distribution-Free-Uncertainty-PPT Industry Leaders’ Insights: Enhance Efficiency and Simplify Your Work Using AI Data Science & Machine Learning Track (DML13) Karen Richey Mislick Greg Mislick The modern workplace is increasingly influenced by leaders who recognize the transformative power of data analytics and AI. This presentation delves into the practical experiences and insights gleaned from industry frontrunners effectively utilizing these technologies. These leaders have not only achieved significant operational efficiencies but have also mastered the art of simplification in complex business processes. Their lessons underline the importance of strategic integration, the value of data-driven decision-making, and the transformative potential of AI-driven automation. Attendees will gain a comprehensive understanding of how top enterprises are reducing costs, streamlining operations, and fostering innovation. Drawing from real-world case studies, this presentation aims to encourage cost analysts to tap into the immense potential of data analytics and AI, turning insights into actionable strategies for enhanced work efficiency. Keywords: AI, LLMs, machine learning, ChatGPT4 DML13-Mislick-Industry-Leaders-Insights-PPT Generative AI for Government Data Science & Machine Learning Track (DML14) Conner Lawston ‘ChatGPT’ has been making massive waves across the world in the last year! This presentation gives an introduction to several ‘Generative AI’ models, and how they can create new images, code, data, and text, seemingly out of thin air. We will look at the process of how to build these models, including their training dataset sizes and costs. Examples will be shown of how to use ChatGPT to generate python code for you, as well as R, and PowerBI. After the general overview, specific examples of applications to Government will be shown (including acqbot- an AI tool for generating proposals). There will also be a demo of the ‘GURU’ bot, which was trained on the Federal Acquisitions Regulation (FAR) pdf, and can answer questions about PPBE, EVM, and Acquisition questions. We will summarize the pros, cons, and potential risks of Generative AI, as well as the future outlook to come. DML14-Lawston-Generative-AI-for-Government-PPT Management, EVM & Risk Track The Cost-Risk Uncertainty Determination (CRED) Model – A New Approach Management, EVM & Risk Track (MER01) Cheryl L. Jones Robert Charette, PhD Bradford Clark, PhD Sariyu Marfo The objective of this model is to improve the credibility of and trust in a cost estimate by: 1) Identifying, characterizing, and accounting for different cost performance factors that may be sources of risk/uncertainty that can result in creating material impacts on a software sustainment and maintenance cost estimate. 2) This approach makes visible the “knowledge gap” (if any) between “what should be known” and “what is known” about the system under assessment – this “gap” is an input used to assess a range of uncertainty associated with the estimate. 3) It also fully documents the key program issues and related performance factors that may influence the cost estimate and why. While this presentation focuses on the software domain, it is easily adaptable to other domains. Keywords: Modeling, Program Management, Risk, Software, Uncertainty MER01-Jones-CRED-Model-Paper MER01-Jones-CRED-Model-PPT Schedule Risk at Early Acquisition Management, EVM & Risk Track (MER02) Gabriella Magasic Sam Kitchin It can be difficult to construct a realistic schedule early in the acquisition lifecycle due to the limited certainty of requirements, design decisions, and other key elements of program planning. Understanding risk and uncertainty in a schedule is essential, and the GAO Scheduling Guide includes “Conducting a Schedule Risk Analysis” as one of the 10 Best Practices. A Schedule Risk Analysis (SRA) can provide quantitative insight into potential areas of delay along with associated cost impacts. However, a well-formed SRA requires clear input and structured analysis of risk events and uncertainty. In this presentation, we will discuss how to address schedule risk in low maturity projects by investigating different risk modeling techniques, reviewing existing guidance on schedule risk, and analyzing how uncertainty analysis must be interpreted and applied early in the project lifecycle. Keywords: Program Management, Risk, Scheduling, Uncertainty, Analysis MER02-Magasic-Schedule-Risk-Early-Acquisition-Paper MER02-Magasic-Schedule-Risk-Early-Acquisition-PPT Cost Estimation for Project Control Management, EVM & Risk Track (MER03) Ed Spriggs Project control in software development is a critical responsibility of program managers and contracting officers. And although the job is a difficult one for most analysts, the inability to measure and control what is being created and tested can result in loss of stakeholder confidence and, in the worst case, a cancelled project/program. What got us here? You guessed it – agile development. The adoption of agile means less defined up-front scope and little to no requirements documentation. While that flexibility allows for more development freedom it creates more variability in the features and functionality of the delivered product. This paper will describe the best new and existing practices for forecasting capabilities (features) that can be delivered within a certain timeframe given the fixed parameters of cost, schedule, and development team size. We will explore innovative techniques to measure software features, even in the absence of requirements, using function points and story points among others. Keywords: Budgeting, Cost Management, Cost/Benefit Analysis, EVMFunction Points, Functional Requirements, Performance Management, Scheduling MER03-Spriggs-Cost-Estimation-Project-Control-PPT Advancing EVM with a Modernized Framework Management, EVM & Risk Track (MER04) Aaron Everly Corey Maples Scott Campbell DoD’s FY24 procurement budget is the largest in history. The cornerstone of this budget is the procurement of complex, technologically advanced systems. DoD programs require new technologies to meet end-user requirements; however, the challenges inherent in new technology often translate to significant cost growth. PMs utilize EVM analysis to make informed decisions and mitigate contract cost growth. The IPMDAR exemplifies DoD’s recognition of the need for meaningful data by requiring a modernized data schema (machine-readable format providing near real-time cost performance). Likewise, Technomics implements a modern approach to EVM using data analytics software and BI tools applied through a framework that incorporates a comprehensive view of EVM. This paper describes Technomics’ EVM Framework (EVMS Surveillance, Contract Startup, Data Aggregation, EV Analysis, and Program Management), which implements modern tools to not only reduce monthly reporting tasks but also perform powerful EV analysis that enables programmatic decisions. Keywords: Data-Driven, Government, IPM, Performance Management, Program Management, EVM MER04-Everly-Advancing-EVM-Modernized-Framework-Paper MER04-Everly-Advancing-EVM-Modernized-Framework-PPT EVM Reviews – Surveillance Reviews vs. IBRs Management, EVM & Risk Track (MER05) Sam Kitchin Greg Smith Successful Earned Value Management (EVM) implementation requires an effective Earned Value Management System (EVMS) and a well-planned performance measurement baseline. Meaningful insight into project performance can only be achieved with this combination of a compliant system with the active planning and management of project execution. A critical method to evaluate adherence to EVM best practices is to conduct reviews. Compliance reviews and surveillance reviews are used to evaluate the sufficiency of the EVMS, while integrated baseline reviews are used to assess the reasonableness of a project baseline. This presentation will compare and contrast these two types of review, demonstrating how and why they differ. Key terminology, stakeholders, artifacts, timeline, and intended results will be discussed. Real life examples may be used. Keywords: EVMFunction Points, Government, Performance Management, Program Management, Project Controls, Earned Value Management MER05-Kitchin-EVM-Surveillance-Reviews-PPT Advanced EVM Analysis using Time Series Forecasts Management, EVM & Risk Track (MER06) Anna B. Peters Mark W. Hodgins *2024 Best Paper: Management, Risk & EVM Category The recent digitization of contractor EVM data affords cost analysts a newfound ability to execute robust statistical and data science techniques that better predict total project cost and schedule realism. Time series analysis, a well-established method in private sector finance, is one such method. Auto regressive integrated moving average (ARIMA) models may capture the persistence and patterns in EVM data, as measured by CPI, SPI, and schedule execution metrics (SEMs). As a second option, macroeconomic regression models can measure the relationship between contract performance and external considerations, like unemployment and inflation, over time. Both techniques, moreover, may forecast future changes in EVM variables interest, like IEAC. This presentation will discuss how these types of time series models and forecasts are employed on real acquisition programs and their associated IPMDAR data using Python based tools to raise program analysts’ alertness to emergent acquisition risks and opportunities. Keywords: Cost Management, Methods, Regression, Statistics, EVM, Time Series Modeling, ARIMA, Vector Auto Regression MER06-Peters-Advanced-EVM-Time-Series-Forecasts-Paper MER06-Peters-Advanced-EVM-Time-Series-Forecasts-PPT Deriving Total Project Costs from WBS Elements’ Probability Distributions Management, EVM & Risk Track (MER07) Prof. Dr. Rainald Kasprik Studies on possible cost variances in major acquisition projects are focusing on total project costs in order to come to plausible project budgets with a confidence level of 80%. Different lognormal probability distributions had been worked out representing different states of uncertainty. However, these models cannot be applied when using risk management software for deriving the total project costs based on cost probability distributions for WBS elements. Due to a limited processing capacity, risk management software demands a division of the underlying probability distributions into intervals A simple discretization of the models developed to date is not possible, as these models contain unrealistic cost growth factors. Based on simulation studies, three lognormal probability distributions are presented that meet these challenges. Finally, some practical hints are given on the minimum number of intervals which still represents the curvature of a probability distribution and on how to interpret the joint CDF’s not-defined areas. Keywords: Budgeting, Modeling, Uncertainty, Discretization MER07-Kasprik-Total-Project-Costs-from-WBS-Paper MER07-Kasprik-Total-Project-Costs-from-WBS-PPT Modeling Track Cascading Effects – Performance Impacts of Fragile Tasks Modeling Track (MOD01) Tommie (Troy) Miller Joshua Hamilton The growing popularity of Joint Cost & Schedule Analysis has highlighted the need for quality Schedule Risk Assessments (SRA). Modeling schedule risk and uncertainty requires an understanding of schedule networks. Network Analytics (NA) has been furthered in recent years due to research in fields such as social networks, IT networks, and transportation networks. Key aspects of these advancements can be used in SRAs to improve our understanding of schedule risk and mature our modeling techniques. For example, epidemiologists study the propagation of diseases through a community. The techniques used to model this phenomenon may be applicable to SRAs to model the propagation of task slips through schedules. This presentation integrates classical concerns in schedule analytics, principally Merge Bias, with NA processes, such as node centrality measures and edge properties, to uniquely identify fragile tasks and illustrate how delays in these tasks cascade through a schedule and ultimately affect program execution. Keywords: Decision Analysis, Statistics, Uncertainty, Variables, Schedule, Task, Merge Bias MOD01-Miller-Cascading-Effects-Paper MOD01-Miller-Cascading-Effects-PPT Data-Driven Constellation Architecture Design Using Integrated Models Modeling Track (MOD02) W. Allen Wautlet David Brown Greg Thanavaro The modern space mission landscape requires consideration of numerous trade variables to deliver optimal mission performance at low cost. Academic methods exist to address such challenges, however, practical deployment of these methods to constellation mission design remains uncommon. This paper presents a practical space mission constellation architecture approach that employs proven statistical, data science, and machine learning techniques on the products of an integrated cost and engineering modeling framework. When deployed at the early stages of constellation development, this integrated modeling framework and analysis approach provides stakeholders insight into key design parameters that drive mission performance and cost sensitivity. Furthermore, it enables the uncovering of promising design regions in large trade spaces that can be further examined and refined by technical subject matter experts. This approach leads to better decision making earlier in the acquisition timeline and increases the efficiency of design cycles. Keywords: Data-Driven, Modeling, Parametrics, Mission Modeling, Bus Modeling, Cost Modeling MOD02-Wautlet-Data-Driven-Constellation-Architecture-Paper MOD02-Wautlet-Data-Driven-Constellation-Architecture-PPT Mission Operations Cost Estimation Tool (MOCET) 2024 Status Modeling Track (MOD03) Marc Hayhurst Roshni Patel Cindy Daniels Lissa Jordin The Mission Operations Cost Estimation Tool (MOCET) is a model developed by The Aerospace Corporation in partnership with NASA’s Science Office for Mission Assessments (SOMA). MOCET provides the capability to generate cost estimates for the operational, or Phase E, portion of full NASA space science missions. MOCET is a widely accepted model in the NASA community used in full mission Announcement of Opportunity competitions since 2015. MOCET was awarded NASA Cost and Schedule Team award in 2017 and honorable mention in the 2021 NASA Software of the Year competition. The cost estimating relationships and documentation have been implemented as a standalone Excel tool that is available within NASA and publicly through software.nasa.gov. Extended mission and Level 2 work breakdown structure costing capabilities are continuing to be developed and a status will be presented. Keywords: none provided MOD03-Hayhurst-MOCET-2024-PPT A CASE for Estimate Analytics at the Enterprise Level Modeling Track (MOD04) Josh Angeo Miguel Aceves Are our estimates improving over time? What did this cost 2 years ago? When was the last time we reviewed this estimate? These questions, amongst many others, are why SSC FMC developed the Cost Analytics for SSC Estimates (CASE) tool. CASE includes over 175 cost estimates, 60 programs, and goes back as far as 2017. The tool creates comprehensive dashboards capable of analyzing programs individually and in aggregate. CASE utilizes various data sources and performs extensive data pre-processing to ready the data for Power Bi. Data pre-processing steps utilize python, DAX, and Power Query. Estimate data comes from a combination of POST reports, PDFs, and spreadsheets. Custom meta data tables were developed to enable parsing and other functions. Lastly, data sources comprising of program actuals have recently been integrated. All of this results in a new found capability to evaluate estimates using analytics. Keywords: Cost Management, Data-Driven, Government, Microsoft Excel, Program Management, Story Points, Power BI MOD04-Angeo-CASE-Estimate-Analytics-PPT Modeling Electronic/IT System Deployment Projects Modeling Track (MOD05) F. Gurney Thompson III Ben Robinson This presentation will discuss the development and application of cost models for electronic and IT system deployment projects. The deployment projects include various technical studies and preparation activities, site survey visits, and comprehensive installation efforts across many sites. The models consider size drivers such as the amount of hardware and software systems to be installed, number of sites, scope of activities, and number of different system configurations. Project complexity can be adjusted for many system/technology intricacies and site conditions. The models have been applied successfully, with validation against actuals, in estimating deployment costs for communication network deployment projects such as data centers, air traffic control towers, and military vehicle/ship/aircraft communication systems. Additionally, these models have been applied to weapon system and train signaling equipment deployments, with model validation relying on expert judgment. This presentation outlines the model’s development, stru Keywords: none provided MOD05-Thompson-Cost-Estimation-Smart-Cities-PPT Recipe: Homemade Pizza (or Facility Estimate) Modeling Track (MOD06) Kristen Marquette Caitlin Burke Olivia Lindsey Have you ever wanted to “”wow”” your guests with a homemade pizza, but didn’t know where to start? This is how we felt when beginning our facilities estimates. This presentation will break down both recipes step by step, leaving everyone satisfied and writing rave reviews. Just as you need delicious dough, sauce, and toppings for great pizza, you need detailed structural, material, and recurring scope requirements for a great facilities estimate. We will take you through our experience with data collection spanning multiple facilities and serve up comprehensive methodologies with high fidelity. If you don’t have time to create a homemade pizza or perform your own detailed facilities analysis, you can leverage the tools and methodologies provided (as to-go slices), to build your own facilities estimate based on your specific program requirements. Keywords: none provided MOD06-Marquette-Homemade-Facility-Estimate-PPT Well, That Escalated Quickly – A Novel Approach to Forecasting Escalation Modeling Track (MOD07) Sean Wells Kaitlyn Hagy Escalation rates are an important part of estimates and as such the provenance and derivation of indices should be regularly scrutinized, yet are rarely contemplated. This paper will compare a commonly used black-box escalation resource, IHS Global Insight, to a traceable, simplified forecasting method to determine if a purely mathematical model delivers an improved level of forecasting accuracy. Our model relies on a curated set of Bureau of Labor Statistics (BLS) indices to develop a moving average forecast. With access to over 15 years of IHS forecasts dating back to 2006, spanning 800+ indices, this study has the unique opportunity to quantify the accuracy of IHS and moving average forecasts against historical BLS indices. Our paper will establish and explore various measures of forecast accuracy for use in creating defensible estimates. The goal is to provide a quick, transparent, and flexible way to develop tailored escalation projections without sacrificing accuracy. Keywords: Data-Driven, Methods MOD07-Wells-Escalated-Quickly-PPT MOD07-Wells-Escalated-Quickly-Paper Comparative Analysis of NASA Cost Estimation Methods Modeling Track (MOD08) Camille Holly NASA policy and customer expectations dictate use of various cost estimating tools depending on milestone and program maturity, regardless of level of effort or accuracy of results. This paper presents a case study of the tradeoffs of modeling the cost of an unmanned space mission using different NASA-approved parametric tools. The comparison addresses subsystem and component-level cost estimates, providing invaluable insight into the granularity of cost modeling for complex space missions and differences in results associated with more or less granular estimates. The study offers perspective on the challenges and opportunities associated with parametric cost modeling methodologies due to the varying levels of input detail, and of effort, needed to complete an estimate. It also aims to provide practical insights on the number and types of subjective decisions made when modeling costs using different approaches, and the impacts that these choices have on cost results. Keywords: Subsystem, Component-level, Parametric tools, Cost modeling, Case study, NASA, Cost estimates MOD08-Holly-Comparative-NASA-Estimation-Methods-Paper MOD08-Holly-Comparative-NASA-Estimation-Methods-PPT The Nuclear Option: Avoiding Critical Delays with Advanced Constraints Analysis Modeling Track (MOD09) Hannah Hoag Lee *2024 Best Paper: Modeling Category NNSA construction projects are often subject to funding constraints. The ripple effect of funding shortfalls can be severe; projects are forced into suboptimal execution profiles that produce costly schedule slips with drastic mission implications. This experience is not unique to NNSA construction projects. Funding constraints occur in most government sectors, negatively impacting many types of projects’ progression, schedule, and mission. However, since inadequate funding is often unavoidable, it is imperative to use a data-driven methodology to predict schedule deviations and calculate ideal cost phasing to mitigate additional or unanticipated implications on project timeline. This paper demonstrates how a constrained phasing model uses historic project cost and schedule data to estimate a new project timeline based on a constrained funding profile. It also reveals how the model re-phases costs for the remainder of the project duration to generate a viable execution plan. Keywords: Data-Driven, Scheduling, Statistics, Phasing, Weibull, Funding constraints MOD09-Lee-The-Nuclear-Option-Paper MOD09-Lee-The-Nuclear-Option-PPT Costing a Ballistic Schedule Modeling Track (MOD10) Rob Carlos Kaden Howell Join us to explore an imminent solution addressing recurring concerns in the DoD involving cost overruns and schedule delays resulting from program practices and schedule dynamics. We’ll address the power of Integrated Cost & Schedule Risk Analysis (ICRSA) & Joint Confidence Level (JCL) assessment from a DoD program office perspective, emphasizing its practicality. Such outputs yield more reasonable and quantifiable estimates by incorporating cost & schedule risk and uncertainty. We’ll present a case study involving a DoD ACAT IB program, discussing the lessons learned during ICSRA implementation and JCL attainment. Our presentation illustrates the impact of ICSRA and JCL, facilitating improved forecasting, early risk identification, trade space analysis, and informed decision-making. The primary objective is to provide real world insight based on lessons learned, quantitative analysis, and creative problem solving on the efficacy, utility, and power of the ICRSA and JCL. Keywords: Agile, Data-Driven, Government, Monte Carlo, Risk, Scheduling, Uncertainty, Integrated Cost & Schedule, Joint Confidence Levels, Integrating Schedule & Cost Risk Analysis, Risk Based Integrated Cost and Schedule Analysis, Simplified Cost and Schedule Risk Analysis, Cost Overruns, Schedule Delays MOD10-Carlos-Costing-Ballistic-Schedule-Paper MOD10-Carlos-Costing-Ballistic-Schedule-PPT Flavors of Commonality: Learning in a Multiple Variant Environment Modeling Track (MOD11) Brent M. Johnstone Commonality – the reuse of parts, designs and tools across multiple aircraft models — is a popular strategy to reduce program costs in commercial and military applications. But its use poses unique challenges to learning curve practitioners. This paper examines five approaches to estimating multiple variant programs using different learning curve techniques. A notional dataset is created, and the accuracy of each method is measured to highlight the advantages and disadvantages of each. This presentation should be of interest to anyone doing learning curve analysis in their cost estimates. Keywords: Labor, Learning Curves, Manufacturing, Methods, Modeling, Commonality MOD11-Johnstone-Flavors-of-Commonality-Paper MOD11-Johnstone-Flavors-of-Commonality-PPT Processes & Best Practices Track Scrutinizing an Organization’s Project Planning Performance Processes & Best Practices Track (PBP01) Sergey Kozin Explore the intriguing world of Project Planning within a typical sustainment organization, spanning nearly a decade worth of estimation and execution data for dozens of special projects as PMs, Engineers, and Estimators desperately fight to defend their budgets and keep the system operating. Did we prioritize having thoroughly developed requirements definitions or wait till the 11th hour to establish them? Was schedule and scope realistic or heavily reliant on optimism as a primary methodology? Did we find ourselves broken and send up a signal flare or accept the shackles of a constrained budget? It is accepted that no plan or estimate is perfect, but rarely do we scrutinize and quantify the errors of our ways to encourage improvements within the process. Join this thought-provoking expedition, as we use metrics to judge the performance of planning practices, seeking insights and wisdom for the projects that lie ahead. Keywords: Data-Driven, Performance Management, Program Management, Project Controls PBP01-Kozin-Scrutinizing-Project-Planning-PPT Mission Class in Unmanned Space Estimating Processes & Best Practices Track (PBP02) John Swaren Bryan Howe Vivian Tang, PhD The cost engineering community needs consistent guidelines in addressing mission assurance processes for a given space vehicle mission risk class (A, B, C, or D) based on programmatic constraints and mission needs. This presentation reviews current considerations and research. Current best practice recommendations typically relate Mission Class to an operational environment specification that conveys quality information based upon requirements. Specific end item maintenance accessibility, reliability, structuring, testing and documentation requirements are typically driven by mission operating environment. More user-definition is needed for factoring in parts quality, test-sampling, orbit-ranges and mission duration. Mission Class estimating needs to tailor component-level Part Quality as well as affect “informed” higher-level assembly and system charges. Modeling operational environment should reflect specification flow-down, validation and documentation and modification/ integration of subcontracted material items. Keywords: Unmanned Space, Mission Class PBP02-Swaren-Mission-Class-Unmanned-Space-PPT GAO Cost Guide and 10 Years of Cost Estimate Assessments Processes & Best Practices Track (PBP03) Ben Wilder Jennifer Leotta Since the Government Accountability Office (GAO) Cost Guide was released as an exposure draft in 2007, GAO has used it as criteria to review and assess agency’s cost estimates. This presentation will look at a 10 year period (FY13-FY23) to see (1) if there are any consistent gaps in agency performance of the four characteristics of a reliable cost estimate and (2) if there has been any improvement in scores over the course of the 10 year period. Keywords: Cost Management, Life Cycle, Performance Management, Project Controls PBP03-Wilder-GAO-Guide-10Years-PPT GAO Agile Assessment Guide: Best Practices in an Agile Environment Processes & Best Practices Track (PBP04) Jennifer Leotta In 2020, the Government Accountability Office (GAO) released the Agile Assessment Guide as an exposure draft. After an open comment period, vetting comments, updating text, and applying best practices during audits, we have recently issued the updated Agile Guide. This presentation will provide an overview of the best practices discussed in the Guide and then take a deeper dive into Chapter 7; using program controls such as using a WBS in an Agile environment and what we have found in recent audits for programs using cost estimating, scheduling, and EVM best practices in an Agile environment. Keywords: Agile, Performance Management, Program Management, Project Controls, Scheduling, Software PBP04-Leotta-GAO-Agile-Assessment-Guide-PPT A Series of Unfortunate Slides Processes & Best Practices Track (PBP05) Shannon Cardoza James J Monopoli Embark on a journey through the realm of impactful presentations, where we unravel the secrets to captivating briefings. Picture this: a vivid showcase of real-life blunders that often muddy the waters of comprehension and engagement—slides lacking labels, drowning in excessive words, or confounded by chaotic transitions. Join us as we delve into the essence of strategic naming and purposeful design to craft presentations that captivate and inform. Witness the transformation with us as we reveal the magic of visuals and charts, drawn from successful briefings to Cost Chiefs, PEOs, and beyond. You’ll discover how to master the art of avoiding tricky questions by leveraging compelling visuals and enhancing your soundtrack for seamless narrative flow. Moreover, we’ll shed light on how these skills not only save valuable time and resources but become a cornerstone for professional growth — empowering you to conquer larger audiences with clarity and confidence. Keywords: Communication, Data-Driven, Decision Analysis, Visual Story Telling PBP05-Cardoza-Series-Unfortunate-Slides-PPT The ABC’s of Contract Proposal Pricing Evaluation& Cost Analysis Processes & Best Practices Track (PBP06) Christopher Svehlak Almost every nook and cranny of the Government relies on contracts for services, support and, well, “stuff.” As a cost estimator (especially a certified one), you may not know that you probably already have the requisite base of knowledge, skills, abilities and Excel-spreadsheet-jockey talent to learn and do pricing evaluation and cost analysis of contract proposals. This presentation offers you the “what-for,” the “why,” and the “how-to-perform” to potentially add this tool to your arsenal. It will distinguish between price evaluation/analysis and cost analysis, their purposes, when each is needed, and explain cost realism and reasonableness. Then comes the nitty-gritty — how to perform a pricing evaluation and cost analysis on contract proposals. The goal: you leave with a better understanding & appreciation of the process … and perhaps even consider offering your services to the contracting department. Keywords: Price Analysis, Price Evaluation, Cost Analysis, Contracting PBP06-Svehlak-ABCs-Contract-Proposal-Pricing-PPT The Complex Database Design Tool Belt Processes & Best Practices Track (PBP07) Jamie Bouwens Tristan Judd David Ferland The process of designing a Dimensional Database (DDB) for complex and evolving data types can be difficult for those who have never made one before. A case study is used to demonstrate how to turn an unsustainable method of data management into a DDB using two Six Sigma methodologies, Define-Measure-Analyze-Improve-Control (DMAIC) and Define-Measure-Analyze-Design-Verify (DMADV). DMADV is a preventative method used to create a process from scratch. While DMAIC is a reactive method that is used to improve an existing process. We walk through answering questions such as: What is a fact in a varied, complex, and evolving data set? How do you visualize these fact tables and dimensions? How do you track time phased data? We illustrate that these techniques are cost saving, because they reduce rework, and, most importantly, enable individuals without extensive prior experience to successfully implement an operable DDB. Keywords: Data-Driven, IT, Methods, Six Sigma PBP07-Bouwens-Complex-Database-Toolbelt-PPT Context-Responsive Cost Evaluation: Dynamic Approach to Cost Estimate Reviews Processes & Best Practices Track (PBP08) Brittany Holmes Christan Yarbrough Annually, MDA’s Cost Estimating Directorate self-audits program cost estimates and provides each program a score ranging from 0.0 to 4.0 based on questions extracted from GAO’s Cost Estimating and Assessment Guide. The questionnaire does not consider details about the program being estimated, so it assumes that an immature program can achieve the same score as an established program. This project suggests a new approach to assess estimates that takes into account the life cycle and known details of the program. This presentation compares the two scoring mechanisms, details the variables used to define optimal scores, and defines how to implement the new scoring mechanism. Keywords: Methods, Operations PBP08-Holmes-Context-Responsive-Cost-Evaluation-PPT Space Fence: A Cost Analysis Success Story Processes & Best Practices Track (PBP09) Rick Garcia In 2007, the AFCAA was tasked with providing a Non-Advocate Cost Assessment (NACA) for the Space Fence program, among others. As a central cost estimator on that team, this analysis will describe the research and actions taken by the team to develop cost and schedule estimates that represented an unbiased view of the program’s most likely cost under normal conditions. The resultant cost and schedule estimates were within 5% of the eventual program actuals. This analysis will also describe the partners used for the critical independent technical assessment, as well as the successful interactions with the industry leading companies that were competing for the contract. Lastly, this analysis will outline how our team developed a Ground Radar specific expenditure phasing model based explicitly on historical ground radar programs. Keywords: Cost analysis, Methods PBP09-Garcia-Space-Fence-PPT Soft Skills & Trending Topics Track Convincing Leaders of the Value of COTS Tools for Quick Assessments Soft Skills & Trending Topics (SST02) Karen Mourikas Denise Nelson Terrance Evans Multiple COTS tools and industry databases exist in our profession. But many organizations prefer to develop their own tools based on their own historical data, which then better represents their own environment. However, often the effort to develop these tools can be time-consuming. What happens when decision makers need answers immediately and there isn’t enough time to collect and analyze their own data? One approach employs COTS tools and their underlying industry data. But cost analysts often need to convince decision makers of the validity of using COTS tools. This presentation describes several use cases in which program decision makers needed information right away, issues facing the decision makers, how the cost analysis team convinced program leaders of the validity of using COTS tools, including their pros & cons, as well as surprising insights that emerged, ultimately enabling decision makers to determine feasible paths forward. Keywords: Communication, Decision Analysis, Modeling, Performance Management, COTS, Industry Data SST02-Mourikas-Convincing-Value-COTS-PPT Priceless Culture: Crafting a Culture for the Future of Work Soft Skills & Trending Topics (SST03) Cassidy Shevlin Wyatt Beaman Priceless Culture: Crafting a Culture for the Future of Work delves into the intricate web of elements that constitute a thriving organizational culture. At its foundation lies effective leadership, setting the tone for a space where core values are not just stated but lived out daily. A unified purpose drives every team member, fostering genuine accountability across all levels. Essential to this mosaic is effective communication, ensuring that everyone is not only heard but also understood. Furthermore, the culture is enriched when leadership embraces vulnerability, showing authenticity and encouraging openness. Intertwined with all these is the spirit of gratitude, acknowledging every contribution, big or small. In an era where workplaces are rapidly evolving, crafting such a priceless culture is not merely beneficial—it’s imperative for the future of work. Keywords: Communication Leadership, Culture, Employee Retention SST03-Shevlin-Priceless-Culture-Future-of-Work-PPT Equity and Environmental Justice in Early-Stage NNSA Planning Soft Skills & Trending Topics (SST04) Haley Harrison Zachary Matheson Recent executive orders (EO13985, EO13990, EO14008) directed federal agencies to prioritize environmental justice and reduce systemic barriers affecting minority and underserved groups. As an organization specializing in decision support for early-stage planning, the Office of Programming, Analysis, and Evaluation has developed a framework for incorporating quantifiable factors as proxies for equity and environmental justice-related factors in analyses of alternatives and early-stage planning studies. This framework will be used to inform decision-makers about potential project impacts from an equity and environmental justice-focused lens. An equity and environmental justice-informed approach to planning within the NNSA can minimize the incidence of negative environmental and health outcomes, maximize the number of opportunities available to historically marginalized groups, and contribute to greater trust of the NNSA mission within minoritized communities improving equity. SST04-Harrison-Equity-Environmental-Justice-NNSA-PPT Advancing the Art of Cyber Cost Estimating Soft Skills & Trending Topics (SST05) Austin MacDougall William Gellatly Jessica Kleinman The growth in quantity and intensity of cybersecurity threats has led to new cyber best practices, such as Zero Trust and Secure by Design. These practices present challenges when developing cost estimates for the development and maintenance of information systems. This paper examines how these topics and other new cyber trends influence costs. It evaluates the cost implications in both the design (incorporating cyber requirements into new system development) and sustainment (cyber support for existing systems) phases. This research also examines existing cyber frameworks and relates them into a cost element structure to drive data collection and methodology development. Finally, this paper translates cyber cost estimating lessons learned into recommended content improvements to the technical baseline documentation upon which cost estimators rely. Standard treatment of cyber in technical baselines should facilitate much needed consistency in the composition of cyber cost estimates. Keywords: IT, Software, Cybersecurity SST05-MacDougall-Advancing-Cyber-Cost-Estimating-Paper SST05-MacDougall-Advancing-Cyber-Cost-Estimating-PPT Mind the Gap: Bridging the Generational Divide Soft Skills & Trending Topics (SST06) Jennifer Aguirre Annie Bagay Shannon Cardoza Do you ever feel you’re speaking a different language than your peers? Ever struggle relating to your IPT as they talk about recent college experiences or upcoming retirement plans? Join us as we explore various ways each generation sees the world, whether through their own eyes or through a high-res smartphone camera. Let’s bridge that gap to reap the full benefits of working in a multi-generational environment enabling effective connections between cost and IPT members. With each wave of people comes new ideas, perspectives, communication styles, and workplace preferences. This diversity can be challenging to navigate and when not properly managed can cause miscommunication, feelings of exclusion, disconnected goals, and failed tech baselines. When harnessed properly, it can be the superpower enabling success within team cohesion, gathering cost inputs, and delivering estimate packages. Whatever stage of life you’re in, come with us on a journey of self-discovery in the workplace! Keywords: Communication, Soft-Skills SST06-Aguirre-Mind-the-Gap-PPT ChatGPT: Friend or Foe – Meet Your New EN SME Soft Skills & Trending Topics (SST07) Patrick Casey *2024 Best Paper: Soft Skills & Strategy Category ChatGPT Friend or Foe is an insightful exploration into the capabilities and nuances of ChatGPT. Delving deep into the genesis of this AI model the presentation tracks its evolution from inception by OpenAI to its fourth iteration. Patrick Casey, a Senior Cost Analyst at Quantech Services, candidly shares his experiences with the tool, highlighting its transformative power in various use cases for cost analysts ranging from WBS considerations to innovative recipe creations. While celebrating its prowess the presentation does not shy away from addressing its limitations and security concerns, urging a cautious approach. As a grand finale, attendees are treated to an entirely AI-generated TV commercial. This engaging journey demystifies ChatGPT offering both appreciation and critical insight into this modern marvel. Beyond mere technology the presentation invites audiences to consider the impact of AI in our lives challenging us to harness its potential responsibly. Keywords: Data Collection, Functional Requirements, Life Cycle, Methods, AI, Large Language Models, Use Cases SST07-Casey-ChatGPT-Friend-Foe-Paper SST07-Casey-ChatGPT-Friend-Foe-PPT Economics of Digital Twins in Aerospace and Defense Soft Skills & Trending Topics (SST08) Patrick K. Malone Defense and Aerospace systems engineering is transforming from a document to a digital model framework, leveraging low-cost multidisciplinary modeling, analysis and simulation tools. With these methods, engineers can specify, analyze, design and verify systems. Digital Twins enable the approach, they are digital or virtual replications of physical products and processes allowing increased speed to market and performance evaluation at reduced costs. Not straightforward is return on investment evaluation when generating cost to develop digital twins. This paper looks at development of DT architectures, capabilities and resulting life cycle cost estimates. Factors impacting DT development costs are model fidelity, design features, analytical tools, integration difficulty, scalability, and programming languages. Concepts are grouped providing practitioners tools and methods to apply digital twin concepts to recommended solutions that maintain positive ROIs and identify cost drivers. Keywords: Cost Management, Decision Analysis, Program Management, System Engineering Digital Twins SST08-Malone-Economics-Digital-Twins-Paper SST08-Malone-Economics-Digital-Twins-PPT Software Track Cloud Estimating in the 21st Century – Okay, well in 2023! Software Track (SWR02) Chris Price Cloud deployments represent a fast-paced technology. The ability to produce quality cost estimates for Cloud Deployments is challenging. In the current state, cost estimates must be able to address Kubernetes Orchestraters, Containers, IaaS and PaaS. Cybersecurity is also key to cloud deployments and modern development processes include working in a DevSecOps environment using Agile software development approaches. This presentation will discuss all these challenges and describe ways to perform quality cost estimates for cloud deployments. Keywords: Data-Driven, Decision Analysis, Early Cost, IT, Parametrics, Cloud, Containers, Kubernetes, Orchestrators, IaaS, PaaS, DevSecOps SWR02-Price-Cloud-Estimating-21st-Century-PPT Simplified Software Estimation (SiSE) – Updated on Advancements and Trends Software Track (SWR03) Curtis Chase Carol Dekkers In 2019, representatives from the DHS Cost Analysis Division (CAD) presented early research findings for their Simplified Software Estimating (SiSE) approach at the ICEAA annual professional development workshop. Since then, further advancements ensued facilitated by the IFPUG Simple Function Point Method (SFP), revisions and expansion of the DHS CAD verb keyword lexicon, the addition of requirements risk and uncertainty considerations, and a full guidebook supporting the method. The addition of uncertainty reduces the risk associated with requirements and verb keyword ambiguities. It also gives the estimator the flexibility to create min/max/most likely estimates for requirements that are simply vague at this early requirements stage. As such, the sizing results take into account uncertainties related to different document writers, style, and verb interpretations. This presentation outlines some of the key findings, ongoing research, and (re-)introduces the SiSE approach, offering a more streamlined and accessible process. Keywords: Early Cost, Functional Requirements, Risk, Software, Uncertainty SWR03-Chase-Simplified-SiSE-Trends-Paper SWR03-Chase-Simplified-SiSE-Trends-PPT Agile Software Development Cost Estimating Software Track (SWR04) Jim Golden This presentation will discuss agile software development cost estimating in the multi-year planning cycle. Agile software development programs focus on near term workload and activities with only limited future planning cycles identified. Future cycles are only activated as their start date nears. Any model-based cost estimate or predictive analysis for the budget needs to be flexible, responsive, and adaptive to the daily dynamics of Agile software development program planning and execution. As a cost estimator, integrating with the IPT for a particular program or project has always been a critical factor in understanding requirements, gathering data, and producing a quality estimate. With agile processes being adopted more frequently across software development organizations, cost estimators and program offices are challenged even further to work closely with developers to continuously update cost estimates. Agile sprint results reveal progress of development, and subsequently could affect the cost estimate and budget requests. Keywords: None provided SWR04-Golden-Agile-Software-Development-Paper SWR04-Golden-Agile-Software-Development-PPT How to Choose a Database Storage Model Software Track (SWR05) Tristan Judd Jamie Bouwens David Ferland To design and implement a database solution, teams must conceptually understand how data is formatted in storage. We compare traditional ways of storing data in Excel or CSV formats with that of a scalable SQL format. Within a SQL database, data is typically stored in either a relational or dimensional format and we will explain these formats for novices with examples. Relational may be easier to implement but is less powerful than a dimensional format. We take you through the process of analyzing the types of data used in a team, and how that would be reflected in a dimensional format. The ability to query efficiently, linkage to popular business intelligence techniques, and scalable structure make dimensional databases the preferred option for structured data storage. Keywords: Data-Driven, Data Storage SWR05-Judd-Choose-Database-Storage-Module-PPT Measuring Software Development Efficiency in an Agile Environment Software Track (SWR06) Benjamin Truskin Aubrey Dial Peter Braxton Ken Rhodes Agile software development practices, while designed to deliver value sooner and accommodate changing requirements, are not intended to mitigate cost growth. Nevertheless, Program Managers must navigate this paradigm and control risk while ensuring stakeholder requirements are fully met. Traditional metrics used to measure growth (e.g., SLOC counts, productivity factors, requirements sell-off) are likely unavailable in Agile projects and while recent DoD policy recognizes the need for metrics, agile metrics are not standardized and using them for independent estimation is uncommon. This paper discusses real-world experience balancing leadership’s goals for independent analysis with the realities of an Agile environment. It will show the value of utilizing program-specific metrics and calculating useful measures such as Change Traffic and Feature (in)efficiency for producing defensible estimates, enabling better program outcomes, and providing insights for others to use themselves. Keywords: Agile, Data-Driven, Government, Modeling, Performance Management, Program Management, Software SWR06-Truskin-Measureing-SW-Efficiency-in-Agile-Paper SWR06-Truskin-Measureing-SW-Efficiency-in-Agile-PPT A Software Sizing Toolkit – Choosing the Right Approach(es) for Your P Software Track (SWR07) Carol Dekkers Dan French You’ve probably heard of source lines of code (SLOC) and function points as choices for software size, but what about RICEFW, t-shirt sizing, story points and Simple Function Points? Like the old adage “If all you have is a hammer, everything looks like a nail” – the most appropriate software sizing approach for your cost estimate may include multiple sizing methods. This presentation outlines the various units of measure available and outlines how and when each approach is most suitable. It’s a primer for cost estimators new to software intensive systems and who need to understand what are the options available when estimating software projects. Keywords: Data-Driven, Early Cost, Functional Requirements, Software Size SWR07-Dekkers-Software-Sizing-Toolkit-PPT Unlocking Untapped Software Metrics Potential with Jira’s RESTful API Software Track (SWR08) Blaze Smallwood Many software projects manage their efforts in Application Lifecycle Management (ALM) tools, like Jira, and these tools can capture a rich set of data, which can be a treasure trove for a cost or project analyst. However, many analysts limit themselves by simply exporting flat lists of records from the tool’s User Interface (UI), which ignores valuable data captured in the system that can further enhance various analyses. This paper will focus on Jira and explain how an analyst can access several interesting additional data sets from its RESTful Application Programming Interface (API) with appropriately structured Uniform Resource Identifiers (URI). This paper will also cover how an analyst can use Java or Python programming to parse the JSON data returned from the API and produce simple but powerful data formats that can inform metrics dashboards or cost analyses. Keywords: Agile, Data Collection, Software, Jira SWR08-Smallwood-Unlocking-Untapped-SW-Metrics-PPT Risky Business: Navigating the World of Software Productivity Software Track (SWR09) Dave Brown Kevin Cincotta Size and productivity are commonly cited as the two major software development cost drivers. Logic dictates that the two are related and inversely correlated. But what is the probabilistic range of uncertainty for productivity, given a software size? What is meant by “an 80% confidence level for productivity”? Cost analysts often quantify uncertainty with an S-Curve; why can’t this be done for productivity directly? We use International Software Benchmarking Standards Group (ISBSG) data to estimate the distribution of productivity directly and provide closed-form formulas for the fitted distribution(s). We find that productivity (and, with certain assumptions, cost) can be estimated with an S-Curve directly, using built-in Excel formulas, with no need for Monte Carlo simulation. This result has significant implications for almost any software development cost estimate, and is particularly relevant to agile development efforts where time-boxed effort is generally fixed. Keywords: Agile, Risk, Software, Uncertainty, Productivity, Probability Distribution, S-Curve SWR09-Brown-Risky-Business-Software-Productivity-Paper SWR09-Brown-Risky-Business-Software-Productivity-PPT Sizing Agile Software Development Programs Software Track (SWR10) Bob Hunt Heather Meylemans Denton Tarbet Chad Lucas Rainey Southworth Size is a critical element in software cost estimation. As Agile has become more prevalent, the use of lines of code as a software size metric for software estimation has become less accepted. This presentation will discus and compare sizing alternatives including “tee shirt” sizing and functional size alternatives on large Federal Programs. The presentation will provide some emerging metrics for assessing size.. Since many automated models convert functional size to physical size, the presentation will address techniques to accomplish “backfiring”. The presentation will address the use of Natural Language Processing and models such as Cadence and ScopeMaster. And, the presentation will discuss models such as COCOMO III that directly convert functional size to hours. Keywords: Agile Software, Software Sizing SWR10-Hunt-Sizing-Agile-Software-Development-Paper SWR10-Hunt-Sizing-Agile-Software-Development-PPT Why Care About CEBoK-S if we Don’t Build Software? Software Track (SWR11) Carol Dekkers Given the increase in software-intensive programs today, it should come as no surprise to experienced cost estimators that even minor software development can render a program overbudget and behind schedule. This presentation outlines the key differences in cost estimating approaches from traditional industries (hardware, facilities, systems) versus software development, and why CEBoK-S knowledge is critical for today’s cost estimators. Given that close to 60% of software projects are deemed failures (overbudget and/or late), with little improvement despite modern technologies, understanding the basics of software cost estimating can provide a competitive advantage for anyone involved in estimating programs for which software development is a component. SWR11-Dekkers-Why-Care-About-CEBoK-S-PPT Strategy Track From “Plan and Pray” to “Sense and Respond”: War Gaming Defense Acquisition Strategy Track (STY01) Alex Wekluk Brian Flynn Ben Bergen “The most dangerous phrase in the language is, ‘We’ve always done it this way.'” – Rear Admiral Grace Hopper. The need for flexible and rapid solutions in the face of emerging threats warrants a radical reset in defense acquisition. NATO’s canonical post-World War II plan-acquire-pray acquisition processes lack the agility to meet a generational change in what military historian John Keegan calls the face of battle. A new paradigm is urgently needed to meet the exigencies of modern warfare with the adaptability of the best business firms: innovating and reacting at the speed of competition. This paper provides an innovative risk-driven framework for an Acquisition War Game that laser-focuses on key metrics such as scalability, logistical footprint, time-to-contract, and fungibility – to support today’s battles and near-peer competition with our enemies. This new Acquisition War Game strategy senses and responds rather than plans and prays, meeting reality head-on in an ever-changing battlespace. Keywords: Cost/Benefit Analysis, Data-Driven, Decision Analysis, DOD/MOD, Methods, Modeling, War Gaming STY01-Wekluk-Wargaming-Defense-Acquisition-Paper STY01-Wekluk-Wargaming-Defense-Acquisition-PPT FP&A: Can We Disrupt Traditional Government Cost Estimating? Strategy (STY02) Christopher Metz There is tremendous value potential in the cost estimates built today across Government under the guidance of GAO’s best practices, but perhaps with varying realization. “Cost Teams” and “Cost Estimators” are sometimes viewed as simple calculators of FTEs times labor rate in the minds of those who do not understand where a cost estimate goes and how its value can increase the chance of mission success. At our relatively new DoD Agency, we set out to find the industry equivalent to “Cost Estimating” and found “Financial Planning & Analysis (FP&A).” As we stand-up this competency we have been gathering and integrating best practices from industry’s “FP&A” and Government’s “Cost Estimating” along with novel ideas and contracting vehicles to disrupt the cost estimating field to better operationalize our cost estimates, steward taxpayers’ dollars, and meet the mission. Keywords: Government, Operations, Cost Estimating, Financial Planning and Analysis, FP&A STY02-Metz-FPA-Disrupting-Government-Estimating-PPT Portfolio Analysis Made Effective and Simple Strategy (STY03) Brandon Schwark Alan Karickhoff Effective portfolio analysis strategies rely on robust recognition of resource constraints, competing priorities, interdependencies, and executability. They transform complexity into simplicity. Our strategy details a flexible, efficient, and analytically rigorous evaluative framework that integrates complex sets of interconnected analyses to assist leadership with data-driven resource allocation. The framework offers solutions in data cleaning, optimization algorithms, and visualization tools that enable stakeholders to effectively navigate complicated portfolio landscapes. Applicability of the framework is demonstrated through a use case that details a facility construction portfolio expected to grow aggressively in the coming years. This paper addresses the complex and often conflicting portfolio objectives mentioned above and outlines their corresponding solutions. Keywords: Data-Driven, Decision Analysis, Government, Infrastructure, Modeling, Statistics, Portfolio Analysis; Portfolio Optimization; Resource Allocation STY03-Schwark-Portfolio-Analysis-Made-Simple-Paper STY03-Schwark-Portfolio-Analysis-Made-Simple-PPT Parametric Construction Estimating: Consistency through Turbulence Strategy (STY05) Cortney Collins Margaret Melchor Not all estimates are created equal, but all are necessary. How is construction estimating different from DoD estimates? They both predict costs based on agreed-upon requirements; they both use historical information to develop parametrics; and they both exist as living documents, updated as new information becomes available, and delivered to the customer to assist with budgeting and purchasing. So – how are they different? This paper will highlight some of the major disparities – from how inflation is handled, to validity of pricing, to how the current economy factors into the estimate. We will also explore how materials could be affected by pandemics and natural disasters (hurricanes, earthquakes, etc.). All of these events can impact the prices of lumber and steel – which in turn, can have estimators scrambling to update the models. Keywords: Data Collection, Government, Methods, Microsoft Excel, Parametrics, Construction STY05-Collins-Parametric-Construction-Estimating-PPT Leveraging Cost Estimating Techniques in Price to Win Analysis Strategy (STY06) Darren Kreitler Leveraging cost estimating techniques is pivotal in “Price to Win” (PTW) analysis for competitive bidding. This session delves into various techniques, from analogy-based to parametric and bottom-up estimating. By integrating these methods with PTW analysis, organizations can strategize optimally, balancing profitability with competitive pricing. Real-world applications underscore the benefits of this synergy, emphasizing the role of accurate cost prediction in securing contracts and ensuring sustainability in today’s dynamic markets. Keywords: Price-to-Win, PTW, Price Strategy, Pricing STY06-Kreitler-Leveraging-Techniques-PTW-Paper STY06-Kreitler-Leveraging-Techniques-PTW-PPT CEBoK® Training Track Complimentary access to CEBoK®2.0 is available for all current ICEAA members. Log on to your ICEAA profile to view CEBoK® materials. Download the Certification Program Overview presentation (PDF) Certification Program Overview CEBoK® Training Track (CEB00) Kevin Cincotta, CCEA®, SCEC Jennifer Kirchhoffer, CCEA®, SCEC This interactive session introduces the ICEAA certifications – Certified Cost Estimator/Analyst (CCEA®), Professional Cost Estimator/Analyst (PCEA®), and Software Cost Estimating Certification (SCEC). It covers eligibility and certification requirements, examination topics, relationships to the Cost Estimating Body of Knowledge (CEBoK®), the online exam format, and recertification requirements. Great opportunity to talk with Certification Principal (Kevin Cincotta) and VP of Professional Development (Jen Kirchhoffer) and get your questions answered. Cost Estimating Basics, Costing Techniques, and Parametric Estimating CEBoK® Training Track (CEB01) Bill Barfield, CCEA® The Basics & Techniques session introduces an overview of cost estimating and analysis and the reasons for doing cost estimates, as well as four essential cost estimating techniques most often used to develop realistic and credible estimates. Additionally, we will review cost estimating products and related topics such as schedule and operations and support estimating, providing the background information and fundamental knowledge from CEBoK® Modules 1-3. Data Collection and Normalization CEBoK® Training Track (CEB04) Markie Harris Katiana (Kat) Lemmons This session covers the Core Knowledge section of CEBoK® Module 4: Data Collection. All estimating techniques and cost estimating models require credible data before they can be used effectively. In this module we will discuss the various types of data, processes needed to collect and analyze the data used in parametric applications, as well as data types, sources, and adjustment techniques. Inflation and Index Numbers CEBoK® Training Track (CEB05) Peter Braxton, CCEA® Bob Hunt This session covers the Core Knowledge section of Inflation and Index Numbers (CEBoK® Module 5). Proper inflation analysis is essential to the success of any cost estimate or economic analysis. Calculating inflation correctly and understanding the fundamental concepts will enable you to produce cost estimates that are timely, accurate, and credible to support your program’s lifecycle needs. It will also empower you to communicate with key stakeholders on the need to adjust your financial estimates based on changes in the economy. Basic Data Analysis Principles and Probability and Statistics CEBoK® Training Track (CEB06) Kimberly Roye, CCEA® This session discusses the analytical steps to take after obtaining a set of cost data and covers techniques for displaying and analyzing data graphically and statistical and graphical analysis of univariate and bivariate data sets (CEBoK® Modules 6 & 10). Other topics include measures of central tendency and dispersion and important probability distributions. We also introduce the concept of a random variable; Monte Carlo simulation; and the differences between the normal and lognormal distributions. Finally, we discuss hypothesis testing. Learning Curve Analysis CEBoK® Training Track (CEB07) Troy Miller, CCEA® This is a training track presentation of the CEBoK® Module 7 (Learning Curves) will cover the key ideas, analytical constructs, and applications of the module. Beyond the theoretical information, we will present the study questions for Module 7 with steps required to solve the problems using only a calculator as is required on the certification exam. Regression Analysis CEBoK® Training Track (CEB08) Dave Brown, CCEA®, SCEC Kevin Cincotta, CCEA®, SCEC This course introduces the basic concepts of regression and provides a demonstration of a simple linear ordinary least squares model (CEBoK® Module 8). This session focuses on the basics required to build and evaluate a simple linear model such as a Cost Estimating Relationship (CER). Key concepts include correlation, minimizing error, homoscedasticity, statistical significance, goodness of fit, confidence intervals, uncertainty, and analysis of variance. The better you understand these concepts, the better you will be able to make inferences about cost data and employ more complicated regression techniques. Cost and Schedule Risk Analysis CEBoK® Training Track (CEB09) Mel Etheridge, CCEA® This session will provide motivation for the need for risk analysis and introduce the basic types and uses of risk (CEBoK® Module 9). It will focus on the practical execution of the general risk analysis process: develop a point estimate; identify the risk areas in the point estimate; determine uncertainty around the point estimate; apply correlation between uncertainty distributions; run the Monte Carlo simulation; assess the reasonableness of results; calculate, allocate, and phase risk dollars; and show the results. Manufacturing Cost Estimating CEBoK® Training Track (CEB11) Pat Malone, CCEA® The goal of the Manufacturing Cost Estimating module (CEBoK® Module 11) is to arm the student with a set of techniques used to address issues unique to estimating in the manufacturing environment. It will be our objective in this module to raise a few of the most common general issues, considerations and concerns the estimator must be aware of in a typical major manufacturing environment and to provide techniques for addressing them. Depending on time and interest of attendees, example problems can be worked as exam preparation. Software Cost Estimating Using CEBoK-S CEBoK® Training Track (CEB12) Carol Dekkers, SCEC This session covers the core knowledge of Software Cost Estimating using CEBoK-S (all PCEA/CCEA testable topics are included). It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides an introduction to the basics of the software development and maintenance processes and how to estimate the related effort. The key ideas of Software Cost Estimating include the cost drivers of size, complexity, and capability. In the sizing area, we’ll focus on understanding the physical size, functional size, relative effort measures (agile software development), and RICE(FW)1 objects. We’ll also discuss the primary software development paradigms – Predictive (waterfall), Predictive with modification (incremental, evolutionary, and spiral methods), Agile (iterative, scrum, SAFe), and Hybrid – and how to model them from a cost estimating perspective. Economic Analysis CEBoK® Training Track (CEB13) Kellie Benefiel, CCEA® This session covers the Core Knowledge section of Module 13 Economic Analysis of CEBoK®. It will be of particular interest to anyone studying for the ICEAA certification exam. The session provides a practitioner’s perspective for conducting an economic analysis (EA) by reviewing EA concepts, terminology, variables and measures-of-merit. By accounting for monetized costs, monetized benefits, opportunity costs and time-value-of-money (“discounting”), an EA enables one to calculate economic measures-of-merit. Contract Pricing CEBoK® Training Track (CEB14) Chris Svehlak, CCEA® This session explores the basics of contract pricing (CEBoK® Module 14). We explore various contract types and the factors and considerations related to choosing a contract type. We also explore fee, shared risk, cost-price proposal preparation, the makeup of a good Basis of Estimate (BOE), and evaluation efforts. This session also provides an introduction to cost management. Some methods discussed include Total Ownership Cost (TOC), Cost As an Independent Variable (CAIV), Target Costing, and Activity Based Costing (ABC). Earned Value Management and Cost Management CEBoK® Training Track (CEB15) James Freeman, CCEA® This session will provide an introduction to the basic concepts of earned value management (CEBoK® Modules 15 & 16), with a focus on implementation, governance, and practical application in support of a project or program. Specific topics will include basic EVM components and data elements, as well as standard earned value analysis techniques. We will use practice problems throughout the presentation to demonstrate and reinforce the basic principles of EVM. Introduction to Design to Cost CEBoK® Training Track (CEB17) Dan Kennedy This session provides an introduction to Design to Cost (DTC), including an historical overview of DTC and other related concepts such as CAIV and Should-cost. It covers what DTC is, when it is best applied, how to implement it, and why it is important. We delve into the DTC process, focusing on several key steps in the process. Various examples of DTC in practice are discussed, as well as challenges and best practices related to DTC.
