Posted by On General Purpose Model Credibility
Parametrics Track
PA-11_Presentation_OnGeneralPurposeModelCredibility_Lee
PA-11_Paper_OnGeneralPurposeModelCredibility_Lee
Abstract:
In the aero/defense community and elsewhere, cost estimation of proposed large scale hardware projects by prospective bidders or sponsors is often done using general purpose parametric models, i.e., models based on statistical analysis of a collection of historical data. Of interest in this paper are the parametric models that provide estimates of costs of hardware development and production for a wide variety of hardware, especially hardware that may be used in many situations and environments.
Such models are both challenging and expensive to construct, and few organizations attempt them. The usual arrangement is that the organizations benefitting from such models will license them from companies that specialize in model building, such as Galorath Incorporated and its competitors.
Historically, parametric models go back at least to the ancient Greeks and Romans, but the level of sophistication of these models has increased exponentially in the last 60 or so years, when it was first possible to install a complex parametric model on a mainframe computer, and then a few years later on a personal computer.
In the early years, some cost analysts rejected parametric models on grounds that they were “black boxes” of unknown content that could output anything at all just by “spinning the dials”. That fear gradually dissipated when 1) it was realized that any estimating process can be corrupted by mindless dial spinning, and the phrase “garbage in, garbage out” was born, and 2) parametric models were seen as providing less subjective, more consistently accurate results than competing processes, especially in early project phases when the most significant decisions are being made.
Today, many sponsors and contractors have analysts who are highly proficient in use of these models. They tend to put a high level of trust in the model outputs, although they may (and should) want to run cross-checks using other methods. Still, some general purpose model users, especially new users, are concerned about use of parametric models, and often direct questions to model builders, seeking one kind of assurance or another.
In this paper, we examine the nature of a general purpose hardware model, and discuss the process of building a credible one. In the course of this discussion we propose useful questions that a concerned model user should ask the developer in order to get the most confidence in the model results. We also look briefly at a few questions that we have been asked that are not particularly helpful, and we explain why.
Author(s):
Wendy Lee
Galorath, Inc.
Ms. Wendy Lee is a Systems Engineer/ Cost Analyst with Galorath Incorporated. Her primary duties are to translate new requirements for cost models into design specification, develops test methodology to validate and evaluate results. She primarily works with SEER-H, SEER-IC, and SEER-EOS models developed by Galorath Incorporated. Ms Lee holds the BS degree in Mechanical Engineering from UCLA and has had over 4 years of experience in developing parametric cost estimating models.
Evin Stump
Galorath Incorporated
Mr. Stump is a Senior Systems Engineer at Galorath Incorporated. He has worked in the aerospace/defense industry since 1954 as an environmental test engineer, field test engineer, design engineer, project engineer, small business owner, consulting engineer, cost engineer, and systems engineer. His primary current interest is hardware cost model design.
Mr. Stump holds the BS in Mechanical Engineering from Loyola University of Los Angeles and the MS in Operations Research and Industrial Engineering from the University of Texas at Austin. He also holds the Professional Designation in Government Contract Management from UCLA Extension and the National Contract Management Association. He is a member of Alpha Sigma Nu Jesuit honor society.