Distance Learning Series

Using Predictive Analytics and Open Source Data to Estimate Cloud Infrastructure Costs for Government IT Systems
Paper Download
Cara Cuiule
Amanda Ferraro
Daniel Harper
Richard Mabe

Establishing federal budgets for cloud infrastructure costs prior to selecting a cloud provider requires vendor agnostic cost estimating methods. These methods need to reflect the correlation between rates for a variety of infrastructure instances across all viable cloud service providers. This paper describes research and validation leading to CERs based on over 28,000 virtual machine and storage instances. The predictive analytic approaches presented in this paper can provide valid and verifiable vendor agnostic estimates.

Cara Cuiule is a Cost Research Analyst for PRICE Systems, L.L.C. Her interests are in software cost estimation, automated data collection, and improving current parametric models. She received her Bachelor’s degree in Mathematics from Stockton University in Galloway, NJ.

Amanda Ferraro is a Cost Research Analyst for PRICE Systems, L.L.C. She received her Bachelor’s degree in Mathematics from Kutztown University in Kutztown, PA. Her interests include ordnance cost estimation and automated data collection.

Mr. Harper is a CCEA® and has been an ICEAA member for over 15 years. He has estimated Hardware and Software/IT during all Acquisition and Support Phases and has a passion for knowledge sharing and knowledge management and was awarded the annual MITRE Presidents Knowledge Management award in February 2014. He has a Finance degree from Virginia Tech as well as a Masters in Technology Management from GMU.

Mr. Mabe has over 25 years of cost estimating experience as a contractor (Tecolote, Quantech) and as a government employee, focused on IT, cyber and networking systems in the DOD. As the Tech Advisor for the IT and Electronics division of AFCAA, he led significant research into Cloud estimating methods and approaches, which he is continuing at PRICE Systems. He has a BS in Geology from Boise State and an MS in Logistics from AFIT.

Storytelling for Cost Estimators
Christina Snyder
Paper Download
Slides Download

As estimators, we advocate the importance of good data; but without context, estimates and analyses are just numbers. To give power to our work, we need to effectively pair good estimating with good communication. There is no existing best practice guidance for estimators on how to create a compelling narrative to accompany analysis. By leveraging a storytelling structure, we can inspire action, communicate our findings in a way that resonates, and ultimately become more effective.

Always being passionate about math and logic, it was a natural fit when Mrs. Christina Snyder, CCEA® became a cost estimator in 2005. Since that time, she has supported multiple DOD programs with various aspects of decision analysis, cost estimating, project management, and strategic planning. She holds a B.S. in Applied Computational Mathematics from Virginia Tech.

The Fact That Your Project is Agile is Not (Necessarily) a Cost Driver
Arlene F. Minkiewicz
Paper Download
Slides Download

All true agile projects follow the same philosophy, they do not all apply the same set of practices, tools or processes. Agile projects are value driven, thus subject to change. There are, however, business and contractual requirements for up front estimates creating a conundrum. This paper discusses a methodology and rules of thumb for estimating agile projects, based analysis of publicly available datasets, that provides value to stakeholders and aligns with Earned Value Management requirements.

Arlene F. Minkiewicz is the Chief Scientist at PRICE Systems, LLC with over 35 years of experience at PRICE building cost models. She leads the cost research activity for TruePlanning, the suite of cost estimating products that PRICE provides. She is a software measurement expert dedicated to finding creative solutions focused on helping make software development professionals successful. She is widely published and speaks frequently on software related topics.

Are you Smarter than an Algorithm?
Andy Prince
Paper Download
Slides Download

Cost analysts rely on mathematical algorithms, experience, and subjective assessments to develop cost estimates. However, these analysts often disagree over what is more important: statistically derived algorithms; or experience and judgment. To try to answer this question cost estimating professionals were surveyed for their expert judgment on the complexity and new design values for 15 NASA science missions. The results may or may not be surprising, but will surely be interesting.

Andy Prince (CCEA®) is the manager of the Engineering Cost Office at NASA’s Marshall Space Flight Center. He has 35 years’ experience in cost model development and cost estimating and analysis. He is a past ISPA Board Chair and ICEAA Co-Chair. In 2019 Andy received the Frank Freiman Lifetime Achievement Award.

A 3 Market, 10 Dimension Trade
Doug K. Howarth
Slides Download
Paper Download

Any person, company, or government working across three or more related markets decides how to divide the costs between them. Often decision makers give little thought as to how those resource splits need to work when working in conjunction to a common goal. Using the example of the Prompt Global Strike (PGS) initiative, this paper studies ways to optimize costs in three connected markets (air-to-surface missiles, bombers, tanker aircraft) across ten dimensions.

Doug Howarth discovered and named the field of Multidimensional Economics. Along with two of his MEE Inc colleagues, the United States Patent Office awarded him US Patent 10,402,838 for Multivariable Regression Analysis, the world’s first software designed to deconstruct markets into four dimensions. Author of 13 peer-reviewed papers issued on four continents, he is the recipient of two best Track: paper awards from ISPA/ICEAA. Investor’s Business Daily and Authority magazine wrote featured articles about him.

Lessons Learned Implementing EVM on Government-led Delivery Efforts
Joshua Teitelbaum
Slides Download

Implementing Earned Value Management on projects where a Government entity serves as the Lead Systems Integrator presents unique challenges and opportunities when compared to typical EVM applications on industry vendor contracts. This paper will cover lessons learned and best practices for implementing EVM on Government-led integration projects based on field experience from a team that has helped the Government with several such efforts. This will include a description of the methods and tools the team used to baseline projects, gather data from performers, and report status to stakeholders.

Joshua Teitelbaum is a Senior Consultant at Booz Allen Hamilton. Integrated Project Controls Analyst. BS in Management Science from UCSD. PCEA® certified (CCEA® eligible July 2020).

The costverse for the FlexFile: Enabling Powerful Analysis in R
Benjamin Berkman
Justin Cooper
Slides Download
Paper Download

The Cost and Hour Report (“FlexFile”) is a new Contractor Cost Data Reporting (CCDR) format that promises to change the world of Department of Defense (DoD) cost analysis by delivering significantly more granular cost and hour data than its predecessor, the DD 1921 series of reports. The volume of the FlexFile requires a more thoughtful approach to importing, wrangling, transforming, and ultimately communicating data than Microsoft Excel (Excel) may offer. This paper introduces three R packages that help the analyst exploit the FlexFile to its fullest extent.

Ben Berkman is a lead analyst at Technomics. He has supported OSD CAPE’s Defense Cost and Resource Center since 2017. A lead of CAPE’s FlexFile initiative, he is heavily involved in building workflows in R to support the FlexFile submission process. He was part of the CADE team to win this year’s ICEAA Educator of the Year award. He is an ICEAA certified analyst and holds a BS in Economics from Penn State University (2017).

Justin Cooper is a senior associate at Technomics. In addition to supporting OSD CAPE’s Defense Cost and Resource Center since 2018, he has also been involved in supporting CAPE O&S analytics tools working in applications such as R, Excel, and Tableau. He was named this year’s ICEAA Junior Analyst of the Year. He holds a BS in Economics from Penn State University (2018).

Advanced Data Analytics for Maintenance & Repair Reporting
Paul Hardin
Alexander LoRusso
Tyler Staffin
Paper Download
Slides Download

The 1921-M/R (Maintenance & Repair Parts Data Report) is the DoD system for collecting actual maintenance event and repair part data in the Cost and Software Data Reporting (CSDR) system. This paper will employ the R Shiny package, which is used for the construction of interactive web applications, to demonstrate the analytical value of -M/R data. Additionally, this paper will explore the mechanics of the R Shiny framework within the environment of advanced data analytics.

Paul Hardin’s principal experience is methodology design and model development, including application of non-linear optimization techniques. He has over 25 years’ experience leading independent cost estimates, conducting cost research initiatives and developing innovative cost estimating methods and visual data analysis tools. His cost research efforts include high risk non-recurring acquisition cost (e.g., design engineering, software development and integration) and operating and support (O&S) costs. He has also led the development of many cost estimating tools.

Alex LoRusso is a recent graduate from Penn State University where he received a bachelor’s degree in Industrial Engineering. Since July, he has been working at Technomics in Arlington, Virginia as an Associate. He has experience in cost data analytics and supporting the development of estimates for naval programs. He hopes to grow his involvement in the cost estimating community through future efforts as an ICEAA member.

Tyler Staffin is a Mathematics (B.S.) graduate of Penn State University. Since then, he has been working at Technomics in Arlington, Virginia in Data Analytics and Cost Estimating. He has been introduced to the DoD Cost Analysis community through the lens of EVM and Data Management, and has been involved in ICEAA-related research efforts over the past year. Outside of the office, Tyler enjoys working out and exploring his passion for learning new programming skills.

Improving Software Estimating Relations for Army Software Sustainment Data
Cheryl Jones
Bradford K. Clark
James Doswell
Paper Download
Slides Download

New approaches were employed to improve Army software sustainment cost estimation: causal analysis and annualization of release data. Causal analysis examines the cause/effect relationships between factors that indicate which CERs should be derived. Converting multi-year data to annualized values has improved CERs. This presentation shows what was discovered using causal analysis and the resulting improved CERs.

Ms. Jones, from US Army CCDC-AC, is a technical lead for the Army Software Sustainment Cost Estimation initiative. The objective of this project is to provide the Department of the Army with the ability to accurately estimate, budget, allocate, and justify the software maintenance resources required to meet evolving mission and service affordability requirements across the system life-cycle.

Dr. Brad Clark is Vice-President of Software Metrics Inc. – a Virginia based consulting company. His area of expertise is in software cost and schedule data collection, analysis and parametric modeling. Dr. Clark received a Ph.D. in Computer Science in 1997 from the University of Southern California. He co-authored a book with Barry Boehm and others on “Software Cost Estimation with COCOMO II.” Brad is a former US Navy A-6 Intruder pilot.

Mr. Doswell is a software cost estimator for the US Army DASA-CE, developing software cost estimates. He is a technical co-lead for the Army Software Sustainment Cost Estimation initiative. The objective of this project is to provide the Department of the Army with the ability to accurately estimate, budget, allocate, and justify the software maintenance resources required to meet evolving mission and service affordability requirements across the system life-cycle.

Assuring Credibility in the Cost Estimate: Part II
Hank Apgar
Paper Download
Slides Download

This presentation updates the original, presented at the 2016 ICEAA International Workshop (Bristol), which traced the maturation of cost estimating attributes and focused on cost credibility. Evidence is provided in the words of government and industry executives, estimating and engineering handbooks, professional journals, and government auditing manuals. This update incorporates the impact of popular cost drivers such as system maturity and cost growth. This presentation concludes with guidance for the estimating professional.

Henry Apgar is an ICEAA lifetime member and Certified Cost Estimator/Analyst (CCEA). He is co-founder of ISPA and has earned its Lifetime Achievement (Freiman) Award. Hank is a Cost Analyst for MCR where he develops parametric cost estimating models and prepares independent cost estimates for international clients. He has a BS degree in electrical engineering and an MBA. Hank authored the Cost Estimating Chapter for the Space Mission Engineering Handbook and he is an Associate Fellow of the AIAA.

Costing Out an Air Force Software Factory
Stephanie Quintal
Caitlin Burke
Kristen Marquette
Slides Download

The novel concept of standing up a software factory has left even the most seasoned cost estimator scrambling for guidance. Presented by the Kessel Run cost team, this presentation will provide insights on staffing, physical locale and other hidden stand-up costs. We will discuss real world actuals on team sizing, skill mix, and phasing methods as well as labor rate analysis and acquisition support. Lastly, a template for a generic software factory will be provided.

Stephanie Quintal is the Deputy Cost Chief for Kessel Run at Hanscom Air Force Base in Massachusetts and cost estimator for the WingOps branch in DET-12. Stephanie started her operations research role in March 2016 after receiving a BS in mathematics and a MS in computational mathematics at the University of Massachusetts – Lowell. Her cost estimating expertise has contributed to agile cost estimating for new efforts, revolutionizing the way the Air Force develops software.

Caitlin Burke is the lead cost estimator for the Command and Control Common Platform, Det-12, Hanscom AFB, Massachusetts. She has been with the Air Force since April of 2019, prior to which she was a Professor of Mathematics at the University of Massachusetts Lowell. Caitlin holds both a BA and a MS in the field, with a concentration on Applied and Computational Mathematics. She is a member of the Pi Mu Epsilon Honors Society.

Kristen Marquette is the lead estimator for Data Services, Det-12, Hanscom AFB, Massachusetts. She joined the Air Force with Quantech Services in July 2019. She graduated from Simmons University with honors magnum cum laude with a BA in Economics and International Relations. In addition to her AF work, Ms. Marquette has published an econometric model which she presented with her advising professor at the Eastern Economic Conference in February, 2020.

Diversity in Software Estimation Approaches – Perceptions to Preferences
Slides Download
Shashank Patil
Ria Bakhtiani

For strong foundation of Software Estimates, diversity in analytical abilities & preferences of winning teams are the key factors those must be effectively addressed. In this paper we talk about how perceptions and preferences play an important role at very early stages of a bid / pursuits lifecycle. This is an attempt to study and present various preferred approaches adopted by winning teams and their usefulness on critical parameters of Software Estimations; viz: Turnaround Time, Accuracy, Repeatability and Reproducibility. The views in this paper are based on our internal surveys carried out with SMEs and Engagement Managers at various hierarchical levels.

Shashank Patil is an Engineering Graduate with 20 years in Software IT industry, Shashank has played role at all levels in Software project life cycle. Currently he is leading the Global Estimation group within Capgemini which is one of the top 5 IT consulting firms. As Design Authority of Estimation group, he has introduced revolutionized tools and processes for software estimations & the way winning teams are estimating software projects in Capgemini. This innovative tool enables leveraging Capgemini’s rich experience in Software projects delivery. Shashank is based in Mumbai and pursues Yoga & Spirituality.

Navigating the Minefield – Estimating Before Requirements are Complete
Carol Dekkers

While cost estimation is challenging in hardware and manufacturing projects, software cost estimators face unprecedented obstacles when they are asked to estimate software development projects. Not only are these projects rife with uncertainty, requirements often shift, technologies change direction, and customers shift priorities – and that’s when they are based on solid software requirements!  But, what happens when you need to create an estimate before requirements are complete?  Learn how software cost estimators in leading organizations are responding to an ever-changing software development landscape and delivering estimates, that enable project success.

Carol Dekkers, PMP, CFPS (Fellow), P.Eng, CSM, is CEO of Quality Plus Technologies, Inc. and is the lead author of the forthcoming ICEAA Software Estimating Body of Knowledge (SCEBoK).  She specializes in working with companies who want to succeed with software measurement and functional size measurement. She is a co-author of several textbooks (The IT Measurement Compendium: Estimating and Benchmarking Success with Functional Size Measurement; Program management toolkit for software and systems development; Practical Software Project Estimation; and others,) and is a frequent speaker worldwide on estimating, measurement, ISO standards, project management, communication and quality. Ms Dekkers is a Past-president and former Director of Communications and Marketing for the International Function Point Users Group (IFPUG). She is also a U.S. expert on the USNB delegation to ISO/IEC SC7 software engineering standards.

Software Estimating: Is the Problem Solved? Some Myths and Facts Sanathanan Rajagopal Slides Download

Software Estimating is often seen as one of the most difficult domain and an extremely difficult to estimate. However every project (S) now have software and dependency and complexity of which is growing exponentially. In this paper author will explore why people thinks why software estimating is a big issue and will try to break some myths and explore some facts about software estimating.

Video Link

Sanath is a highly capable and internationally recognised Cost Engineer/Cost Estimator specialising in Software Estimating, Software Obsolescence Management, Software Obsolescence Cost Estimating and Parametric Estimating. Sanath is an ISPA (International Society of Parametric Association) certified parametric estimator who worked in various high profile Defence projects in ISS/ISTAR, Air, Maritime and Land domains, providing costing services at all stages of the procurement cycle in setting budgets, historic trends and Independent Cost Estimates assurances for government business cases.

Is this Schedule Credible?
Jonathan Shriqui
Slides Download

A schedule is the life and breathe of any project. As such, it should be constructed with rigor and discipline. But how does one differentiate a credible schedule from a poorly constructed one without being a SME? The US Defense Contract Management Agency (DCMA) has created a scheduling test to objectively determine a schedule’s integrity. In this presentation we will review the key concepts, pros & cons and limitation of this scheduling test. Please join me and come find out if your project schedules are transparent or is there more to them than meets the eye!

Jonathan Shriqui has over a decade of program management experience acquired from General Electric, Lockheed Martin and BMT Canada on large scale procurement programs. From 2016 to 2019, he provided EVM and project controls insights to the Department of National Defense (DND) for acquisition projects under the National Shipbuilding Strategy, the largest and most complex procurement program in Canadian history. In 2019, he joined DND’s Project Management Support Organization where he provides project control guidance and support.

Advanced Estimating Methodologies for Conceptual Stage Development
Chuck Alexander

This research paper presents statistical techniques and cost analysis that significantly enhance legacy technology development estimating methodologies. Techniques leveraging independent variables that reflect a comprehensive set of cost drivers relating to technology scale, complexity, type, maturity, and development difficulty are presented. Highly tailored solutions including uncertainty are produced that vastly expand and refine earlier development estimating models. General R&D framework relating key milestones, TRLs and cost benchmarks is constructed and woven into an integrated solution.

Chuck Alexander is a Principal at JHU/APL with over 35 years of professional experience involving engineering, economic & cost analysis and technology & business operations management across a range of technologies, applications, system and operational environments. Mr. Alexander is a licensed Professional Engineer (P.E.), CPA, and holds a BS in Mechanical Engineering from Clarkson University and an MBA in Finance from the William & Mary Mason School of Business.