Posted by New Research in General Error Regression Model (GERM) Significance Testing
Methods Track
Downloadable Files:
Abstract:
A limited but significant literature exists with respect to significance testing in General Error Regression Models (GERM). GERM refers to cost estimating relationship (CER) development in the absence of specific functional form assumptions and distributional assumptions about the error term. It is sometimes called a “distribution-free” method of CER development. Within the cost estimating community, GERM was first popularized by Book and Young (1997). More recently, Anderson (2008) proposed A Distribution-Free Measure of the Significance of CER Regression Fit Parameters Established Using General Error Regression Methods, which won best paper in its track at the 2008 SCEA Conference.
This paper puts forth new research on the subject, with specific conclusions that may be at odds with previously published research. First, we show that a commonly held significance test for GERM does not hold in the general case, nor even in the specific case of classical linear regression. In other words, the inferences drawn from classical t-tests and p-values cannot be replicated using the previously published method. Second, we argue for the impossibility of having a provable replication more generally. Significance, by its nature, is measured relative to a distribution, so a provable, closed-form, “distribution-free” measure of significance is not possible, unless certain assumptions (which may be contrary to GERM ground rules) are made.
Under those constraints, we offer an alternative method for assessing GERM significance, relying upon an existing body of statistical literature dealing with nonlinear regression. The proposed method cannot be proven true (for the reasons argued above), but (we contend) can also not be disproven. We show examples in which the alternative method replicates the results of classical t-tests and p-values more accurately than the previously published method.
Author(s):
Kevin Cincotta
Kevin Cincotta joined Technomics in December, 2009. As a senior analyst with over 13 years of experience, Kevin has worked on or led myriad projects within the Department of Defense and civil agencies. His primary areas of expertise are cost analysis, database creation and management, and statistics. His recent clients have included the Coast Guard Research and Development Center (RDC), General Services Administration (GSA), Air Force Cost Analysis Agency (AFCAA), and Office of the Secretary of Defense, Capability Assessment and Program Evaluation (OSD CAPE). Currently, Kevin assists the Defense Acquisition University (DAU) in enhancing the curriculum by which it trains and certifies acquisition professionals. He will supports the Government Accountability Office (GAO) in performing cost and earned value management (EVM) assessments.
From 2001 to 2003, Kevin served as a Senior Cost Analyst at MCR, LLC. He worked closely with government clients at the Missile Defense Agency (MDA) to develop a radar cost model, which was presented by MCR at the 2004 Society of Cost Estimating and Analysis (SCEA) conference.
Kevin also led several cost analysis-related tasks at the New Vectors (formerly Vector Research, Incorporated and the Altarum Institute) from 1997 to 2001. Notable systems on which he had a major role in creating life cycle cost estimates (LCCEs) and/or business case analyses (BCAs) include the Standard Procurement System (SPS), the Defense Occupational Health Readiness System (DOHRS), and the Simplified Tax and Wage Reporting System (STAWRS).
Kevin is a frequent presenter at both the Department of Defense Cost Analysis Symposium (DODCAS) and SCEA conferences. He is a (SCEA)-Certified Cost Estimator/Analyst (C/CEA). Known by some as “The Nature Boy,” he serves on the committees that create and review exam questions for the updated C/CEA exam, and has created several model questions for the new exam. He holds a master’s degree in economics and philosophy from the London School of Economics and Political Science, and a bachelor’s in the same fields from the University of Virginia.
Andrew Busick
LMI
Andrew Busick is a Research Fellow at LMI, formerly the Logistics Management Institute. His primary areas of expertise are cost analysis, cost estimation, and modeling and simulation. Mr. Busick has supported several cost analysis tasks for the Departments of the Army, Air Force and Coast Guard since joining LMI in October, 2007. He holds a bachelor’s degree in economics and mathematics from the University of Virginia.