Posted by Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE)
Methods and Models II Track
M2-8_Presentation_QuantifyingUncertaintyInEarlyLifecycleCostEstimation_Stoddard
Abstract:
Early lifecycle cost estimation continues to grow in importance within both industry and the U.S. Government. In the commercial arena, shrinking business and product development cycles demand more rapid and early cost estimates. Within the U.S. Defense Department, cost estimates are now mandated prior to Milestone A approval within the DoD Acquisition Lifecycle. One of the primary challenges with early lifecycle cost estimation remains the incongruence of input data required by existing cost estimation models and the data available early in the lifecycle. Recent research into capability-based cost estimation offers improvements but ignores much of what may be viewed as program execution change drivers. These program execution change drivers embody the root causes of changes in program performance leading to changes in the cost of the program. As a result, our team developed the QUELCE method as an innovative approach to elicit domain expert knowledge about the possible future scenarios of change driver behavior for a given program. This elicitation begins with use of a modern scenario planning workshop with discussion facilitated using a dependency structure matrix. The dependency structure matrix prompts the experts to consider the applicable program execution change drivers and their cascading effects among each other. The results of the program dependency structure matrix propel the development of a Bayesian Belief Network (BBN) model which predicts key outcome factors in the form of probability distributions. The outcome factors are then statistically associated with input factors of existing cost estimation models, and through the use of Monte Carlo simulation, produce cost estimate distributions.
We believe that the QUELCE method offers a number of benefits including: 1) the ability to dynamically account for program-unique change drivers representing the major share of the program uncertainty, 2) the ability to quickly update the cost estimate throughout time as knowledge of the change drivers’ behavior improves, 3) the ability to connect to and leverage existing cost estimation models, tools and cost estimating relationships, 4) the ability to quantify uncertainty, from both objective data and expert belief, for creating a cost estimate distribution, 5) the ability for stakeholders to attach confidence levels to different levels of cost, and 6) a defined method to unite cost analysts and subject matter experts in developing a credible and defendable cost estimate.
This paper and presentation will summarize the QUELCE method, offer simple exercises to enable the audience to grasp various steps of the method, and discuss the latest lessons learned from both industry and government pilots. Lastly, attendees will learn how to participate in the on-going research and/or pilot of the QUELCE method in their organization.
Author(s):
Robert Stoddard
Software Engineering Institute, Carnegie Mellon University
Robert Stoddard is a Principal Engineer at the Software Engineering Institute (SEI). Robert earned a BS in Business, an MS in Systems Management and is a certified Motorola Six Sigma Master Black Belt. He delivers measurement courses in public and client offerings and provides measurement consulting to external clients.
Robert W. Ferguson
Software Engineering Institute, Carnegie Mellon University
Robert Ferguson is a Senior Member of the Technical Staff at the SEI. He works primarily on software measurement and estimation. He spent 30 years in industry as a software developer and project manager before coming to the SEI. His experience includes applications in real-time flight controls, manufacturing control systems, large databases, and systems integration projects. He has also frequently led process improvement teams. Mr. Ferguson is a Senior Member of IEEE and has a Project Management Professional (PMP) certification from the Project Management Institute (PMI).
Dennis Goldenson
Software Engineering Institute, Carnegie Mellon University
Dennis Goldenson joined the SEI in 1990 after teaching at Carnegie Mellon University since 1982. An ACM and IEEE senior member, his work on measurement and analytical methods has focused on modeling performance and quality outcomes of software intensive systems. Recent work in addition to QUELCE and calibration of expert judgment includes systems engineering effectiveness, requirements engineering, empirical evaluation of software architecture, and statistical methods to ensure data quality. Related interests are in voice of customer methods, tools to support collaborative processes, the quantitative analysis of textual information, experimental design, survey research methods, and the visual display of quantitative information.
James McCurley
Software Engineering Institute, Carnegie Mellon University
Jim McCurley is a Senior Member of the Technical Staff at the Software Engineering Institute (SEI). During his 15 years at the SEI, his areas of expertise have included data analysis, statistical modeling, and empirical research methods. For the last several years, he has worked with various DoD agencies involved with the acquisition of large scale systems. From 1999-2005, Jim also worked as a member of the Technical Analysis Team for the CERT Analysis Center
Dave Zubrow
Software Engineering Institute, Carnegie Mellon University
Dave Zubrow is the Chief Scientist for the Software Engineering Process Management (SEPM) program where he is responsible for formulating research strategy, guiding the development of proposals, and representing the program’s research activities and interests. Dave also is the manager of the Software Engineering Measurement and Analysis initiative at the SEI, which focuses on empirical research and the developmentand application of quantitative techniques to software engineering problems.