Posted by Exploring Uncertainty and Risk
Kurt Brunner, Leidos
Is it a “risk”? Or are you “uncertain”? Either way, it’s definitely NOT risky to join the ICEAA Southern California Chapter for a webinar on Cost Risk and Uncertainty on April 12 at 11:30 Pacific.
You’ll learn the difference between Risk and Uncertainty, why you want to perform Cost Risk or Uncertainty Analysis, how to quantify the unknowns, relation to the Risk “Cube”, “S” Curves and Confidence Levels, and more!
In addition, the SoCal Chapter will share news of Chapter activities and plans. And, as usual, you’ll have a chance to win the SoCal ICEAA Chapter Challenge Coin!
Speaker: Kurt Brunner, Leidos
Kurt Brunner has over 45 years of aerospace and electronics cost and resource estimating experience supporting contractors and government agencies. Kurt has developed several estimating models and is a recognized authority on Improvement Curves and Uncertainty, having contributed to the Air Force Cost Risk Handbook. He holds a CCEA-P credential, and is a founding member of ICEAA, as well as having previously been a SoCal chapter president. In 2010 ICEAA’s predecessor organization, ISPA, recognized his achievements with the Keith Burbridge service award. He has a degree in Anthropology from San Diego State University. Kurt is now semi-retired and is a Consulting Employee in the role of the Senior Principal Parametric Cost Estimator for Leidos. In his spare time, he races vintage motorcycles and tends to his Dalmatian plantation in the Mojave Desert.
Slides Download (PDF)
The ICEAA SoCal Chapter held their FIRST in-person Workshop since 2019 on November 30, 2022. Our program feature dnetworking, trivia games, chapter news, and an impressive lineup of presentations:
Mr. Raj Palejwala
Chief, Cost Estimating and Earned Value Division, Space Systems Command
Keynote Address
Dr. Danny Polidi
Raytheon Technologies
Linking System Cost Model to System Optimization Using A Cost Sensitivity Algorithm
Steve Sterk, CPP
NASA
Artemis Future Missions
Patrick Malone, Systems Planning and Analysis (SPA); Henry Apgar, retired; and William King, Systems Planning and Analysis (SPA);
Cost Estimating Maturity and a Vision for the Future
SoCal Chapter DTC Team: Hank Apgar, retired; Dr. Lisa Colabella, The Rand Corporation; Justin Knowles, Northrop Grumman; Karen Mourikas, The Boeing Company
Design to Cost Course Preview and Feedback Session
Hosted by Tecolote Research, Inc.
2120 E Grand Ave, Suite 200, El Segundo, CA 902452120 E Grand Ave, Suite 200, El Segundo, CA 90245
Download the workshop agenda (PDF)
2021 SoCal Chapter Board of Directors
Past Meetings
Digital Engineering Implications on Weapon System Costs
Join the ICEAA Southern California Chapter for an informative webinar, plus we’ll be discussing Chapter election, the DTC course, and our FACE-TO-FACE December meeting!
Defense programs have been implementing digital engineering (DE) throughout the acquisition lifecycle to develop, manufacture, and sustain their platforms. This presentation will focus on the impact of DE implementation on cost estimates, suggesting key considerations.
Speaker: Brittany Clayton
Brittany Clayton is an Operations Researcher at the RAND Corporation. She supports projects focused on lifecycle cost estimating and defense acquisition policy. She has a master’s degree in Operations Research from George Mason University and a bachelor’s degree in mathematics from Washington & Jefferson College.
Leveraging Multi-Monitor Run-Time Assurance
The business challenge of fielding a safety system in a military acquisition process centered around fighting capability. The presentation will cover how an advanced safety system was advocated in a system and amongst decision makers that have been trained to manage attrition and losses in force structure. It was ultimately the business case that made the difference in the Air Force accepting an advanced safety technology over other systems geared towards increasing fighting power.
Speaker: Mark Skoog
Former Principal Investigator for Autonomy
NASA / Armstrong Flight Research Center
Mark Skoog recently retired from NASA’s Armstrong Flight Research Center as their Principal Investigator for Autonomy. He recently finished leading a collaborative effort between NASA, the FAA, DoD and Industry to develop certification methods for autonomous vehicles. He graduated from California Polytechnic State University at San Luis Obispo. Over the past 41 years he has supported numerous NASA and the Air Force fighter and UAV research efforts as well as test and evaluation of the F-16, B-2 and Global Hawk. Focus areas have included the integration of digital flight controls and avionics with high authority autopilots to automatically accomplish all phases of fighter combat missions. Much of his career has been dedicated to the development of full vehicle autonomy and automatic collision avoidance systems, for both ground and air. He has recently established the company Innovative Autonomy, a research lab dedicated to the development and advancement of autonomous systems.
Southern California Chapter Virtual Workshop
Date: December 1, 2021
Get to the Point. What’s the Deal with Different Function Points Methodologies PDF Presentation
Get to the Point. What’s the Deal with Different Function Points Methodologies Dissertation
ICEAA Paper
GAO Best Practice Guide PDF Presentation
Join the ICEAA Southern California Chapter for our last virtual luncheon webinar of the year. We will be soliciting your inputs for activities and events for the chapter in 2022.
Our 90 minute program includes the following speakers:
- Anandi Hira from Tecolote Research Inc. who specializes in cost estimation methods for software development programs will present Get to the Point. What’s the Deal with Different Function Points Methodologies.
- Abstract: Since the development of IFPUG Function Points (FPs), many variants have emerged to simplify software sizing and improve cost estimation. Which of these sizing techniques should one use? To understand differences among methods, we compare IFPUG FPs, COSMIC FPs, and Simple FPs for effort estimation on University of Southern California (USC)’s dataset of Unified Code Count (UCC) enhancements. Additionally, we investigate automated tools to understand the correlation between Objective FPs and the other FPs methods.
- Brian Bothwell from the Government Accountability Office, which supports assessments of cost estimating, scheduling, EVM, AOAs, and technology readiness, will present GAO Best Practice Guides.
- Abstract: This presentation will cover GAO’s responsibilities, introduce the best practice guides GAO uses and briefly describe their development, and discuss how the guides’ criteria are used in audits through the presentation of 2 or 3 case studies.
Bugs, Bugs, and More Bugs
August 19, 2021
What do “bugs” and ICEAA have in common? To find out, join the ICEAA SoCal Chapter for our Summer 2021 Webinar on August 19 from 11:30 to 1 Pacific titled “Bugs, Bugs, and More Bugs!”
We have invited two guest speakers from different industries who will be sharing their research on the costs of different types of “bugs”:
Bug Bounty Program Pricing by Bryce Downing, Quantitative Research Assistant at the RAND Corporation
A Cost-benefit Analysis of Fiscal Stimulus Packages for Mitigating Covid-19 Pandemic by Dr. Wali Mondal, Professor of Economics from National University
General Webinar Slides
Bug Bounty Program Pricing by Bryce Downing
A Cost-benefit Analysis of Fiscal Stimulus Packages for Mitigating Covid-19 Pandemic by Dr. Wali Mondal
Machine Learning in Space!
April 8, 2021
The ICEAA Southern California Chapter and the ICEAA Machine Learning Working Group have teamed up to offer three presentations on Machine Learning in Space:
• Acquisition Complexity (ACS) Using Principal Component Analysis (PCA)
Mike Finnegan, The Boeing Company
• Proposing a Space-Based Missile Defense Program: Examining Engineering Complexities and Cost Feasibility
Sirina Nabhan and Matthew Ramirez, NASA Jet Propulsion Laboratory
• 2021 Plans for the Machine Learning Working Group
ML Group Steering Committee: Bryan Anderson, Dan Harper, Adam James, Karen Mourikas, Christian Smart
Attending this webinar counts as 1 hour of training or 0.1 points towards your CCEA recertification.
Past ICEAA Southern California Chapter Workshops
December 11, 2019 SoCal Workshop
Hosted by Galorath Incorporated
100 North Pacific Coast Highway, Building 100, 1st Floor – The Great Room, El Segundo, CA 90245
Attending this Workshop, whether virtually or in-person, earns 0.7 points towards CCEA recertification.
Keynote Dr. Daniel Galorath, Galorath Inc.
Update on the Masters in Cost Estimating at NPS Karen Richey, Naval Postgraduate School
Self-Organizing Markets and Time Doug Howarth, CEO MEE, Inc.
Cost Engineering in a Market-based Acquisition Process Dr. Darin Hoffman, Director, Systems Integration, Galorath Incorporated
Cost and Market Modeling for Lunar Mining and Drilling Brad Blair, New Space Analytics
Cybersecurity Metrics Arlene Minkiewicz, Chief Scientist, PRICE Systems, LLC
Cost Engineering is an Integral Element of Model Based Systems Engineering (MBSE) and the Digital Enterprise (DE) Dan Cota, Director, Systems Integration, Galorath Incorporated
September 10, 2019 SoCal Workshop
Hosted by The Jet Propulsion Laboratory/Caltech
4800 Oak Grove Drive, Pasadena CA, 91109-8099
Insight: Mission to Mars Dr. W. Bruce Banerdt, JPL
Business Case Analysis of Lunar Thermal Mining Dr. Robert Shishko, Systems Engineer, JPL
Production System Cost Modeling within a Model-Based Systems Engineering (MBSE) Environment Dan Kennedy, Director of Engineering Services, Galorath Incorporated; and Karen Mourikas, Associate Tech Fellow, Systems Optimization & Affordability, The Boeing Company
Developing Machine Learning Methods for Schedule Summarization Daniel Newkirk, PhD, Tecolote Research, Inc
Responsive Cost Estimating Rick Garcia, Technical Director, MCR Federal, LLC
Estimating Composite Manufacturing Costs Through Top-Down Methods Grady Noll, Cost Research Analyst, PRICE Systems, LLC
March 20, 2019 SoCal Workshop
Hosted by MCR
Cambria Hotel – LAX 199 Continental Blvd, El Segundo CA, 92045
Case Studies of Machine Learning Techniques Applied to Cost Research Michael Schiavoni, Senior Analyst, Tecolote Research
Language Factors in SEER for Software Karen McRitchie, Vice President Product Development, Galorath, Inc.
A Program Manager’s Guide to Reliable Subcontractor Reporting Pat Malone & Garth Edwards, MCR
Legacy of Parametric Estimating Hank Apgar, Technical Director, MCR
Gamify Your Program: Unite Your Team, Motivate (the Right) Performance Nathan Eskue, Manager, Program Material Control; Northrop Grumman Innovation Systems, Launch Vehicles Division
X-Plane Schedule Impacts Steve Sterk, Cost Estimation & Analysis Officer, NASA Armstrong Flight Research Center
Methodology to Access Cost and Schedule Impact Using System And Technology Readiness Level (SRL/TRL) Nate Sirirojvisuth, Cost Research Analyst, PRICE Systems, LLC
December 12, 2018 SoCal Workshop
Hosted by Leidos
4161 Campus Point Court San Diego, CA 92121
Building E – Room E2660
A Robot Brain Might Be the Best Forecasting Tool Possible Nathan Eskue, Northrop Grumman Innovation Systems, Launch Vehicles Division
CER Issues and Solitions Kurt Brunner, Leidos Inc. Senior Principal Parametric Cost Estimator
Electrical Engineering Evolution Dr. Sanrul Ula, University of California – Riverside, Winston Chung Global Energy Center Chair
SEER – Space in the 21st Century Eric Sick, Galorath Inc, Engineering Cost Consultant
Hidden Software Costs – How to Spot Them Jay McGuerty, Leidos Inc, Cyber Security
September 26, 2018 SoCal Workshop
Hosted by Northrop Grumman Aerospace Systems
2114 Marine Ave., Redondo Beach, CA. 90278
East of Aviation Blvd on Marine Ave – Presentation Center (APC) after Bldg. E2 Badge Visit
Machine Learning & Non-Parametric Methods of Cost Analysis, Karen Mourikas, The Boeing Company, Operations Analysis Affordability & Systems Optimization
Would a Cost Growth Factor Help Alleviate Continuing Cost Overruns? Kurt Brunner, KB Enterprises
Cost Estimate Credibility, Hank Apgar, MCR Solutions LLC
Cyber Total Cost of Ownership, Rich Mabe, PRICE Systems, LLC
Demand, Recurring Costs, and Profitability, Doug Howarth, MEE Inc.
March 7, 2018 SoCal Workshop
Hosted by NASA Armstrong Flight Research Center
2825 East Ave P, Palmdale, CA. 93550
Hangar / Bldg. 703, 2nd Floor, Conference Room 211
December 13, 2017 SoCal Workshop
Hosted by Boeing Satellite Systems
September 20, 2017 SoCal Workshop
Hosted by Raytheon Space and Airborne Systems (SAS)
3D Printing – Technology, Applications and Selection – Dr. Rafiqul Noorani, Loyola Marymount University
Improvement Curves – Beyond the Basics – Kurt Brunner, Leidos
Early Cost Risk and Confidence Determination with Minimal Project Information – Stu Swalgen & Dr. Zhaofeng Huang, Aerojet Rocketdyne
Machine Learning Approach to Cost Analysis – Karen Mourikas, The Boeing Company
Deployment Cost Estimation for IT/Electronic Systems – Gurney Thompson, PRICE Systems LLC
March 27, 2017 Workshop
March 27, 2017 ICEAA SoCal Workshop Agenda
Digitalizing Business Models – Aditya Satsangi & Giovanni Martinez, Nitai Partners
Effectiveness Formula: Key to Improved Software Development Productivity – Randall Jensen, Software Acquisition Consultant
General Degrees of Freedom (GDF) [Best Paper Award for Methods and Models Track at 2015 Conference] – Shu-Ping Hu, CCEA-P , Chief Statistician, Tecolote Research Inc.
Price-To-Win in 2017 – Kent Joris, Manager of Competitive Assessment/Price-to-Win, Northrop Grumman Technology Services
Organizational Cost Estimation (paper) – David Bloom, LEngineering Cost Estimation Subject Matter Expert, Raytheon Co., Space & Airborne Systems (SAS)
Organizational Cost Estimation (presentation) – David Bloom, LEngineering Cost Estimation Subject Matter Expert, Raytheon Co., Space & Airborne Systems (SAS)
Improvement Curves: Beyond the Basics – Kurt Brunner, CCEA-P , Leidos Senior Principal Position to Win Strategist / Parametric Cost Estimator / ICEAA SoCal Chapter & Region 7 Director
December 14, 2016 Workshop
December 14, 2016 ICEAA SoCal Workshop Agenda
Keynote Address: Welcome to Fort MacArthur – John Karns, Space and Missile Systems Center Financial Management & Comptroller Directorate
Estimating Life Cycle Costs at the Skunk Works® – Wayne Wright, Lockheed Martin Aeronautics Company
The Signal and the Noise in Cost Estimating – Christian Smart, Ph.D., Missile Defense Agency
Things Change: Immediate Versus Ultimate Aiming – Doug Howarth, MEE Inc.
NASA’s X-Plane Database and Parametric Cost Model V2.0 – Anthony Olguin, NASA Armstrong Flight Research Center
Estimation Bias: Why Can’t People Estimate? – Dan Galorath, Galorath Incorporated
Training Session: Cost Management – Kent Joris, Northrop Grumman Technology Services
September 7, 2016 ICEAA SoCal/ISBSG Workshop Agenda (PDF)
March 9, 2016 Workshop Agenda (PDF)
December 16, 2015 Workshop Agenda (PDF)
Download the September 9 Workshop Agenda (PDF)
Workshop Presentations:
Kathy Kha, Galorath Federal, Heuristics for a Space Technology Cost Estimation Model
Dr. Sadrul Ula, Winston Chung Global Energy Center (University of California – Riverside), Technical and Economic Challenges of Integrating Renewable Energy, Electric Vehicle Charging and Battery Energy Storage in a Modern Grid
David Graham, Independent Consultant, Salient Federal Solutions, Two Complementary EVM Cost-Risk Models – Part 2
Lorrie Davis, The Aerospace Corporation, How Long Does It Take to Develop and Launch Government Satellite Systems?
Wayne Wright, Lockheed Martin Aeronautics, Estimating Prototype Air Vehicle Development Costs at the Skunk Works® — The Sequel
Joe Bauer, PRICE Systems, LLC, and Pat Malone, MCR LLC, Cost Estimating Challenges in Additive Manufacturing
March 18, 2015 Workshop Presentations (at Fort MacArthur in San Pedro, California).
View Agenda (PDF)
Workshop Presentations:
Christian Smart, Ph.D., Director of Cost Estimating and Analysis, Missile Defense Agency, Bayesian Parametrics: Developing a CER with Limited Data and Even Without Data
Bob Hunt, President, Galorath Federal, Cost Estimation using Story Points
Tom Harwick, Engineering Specialist in Concept and Preliminary Design Cost Models, Northrop Grumman Multi-Discipline Design Analysis & Optimization 2014
Bob Becker, V.P. Business Development, PRICE Systems LLC, Cost Management Implementation at U.S. Army PRO STRI
David Graham, Independent Consultant, Salient Federal Solutions, Two Complementary EVM Cost-Risk Tools
December 17, 2014 Workshop Presentations (at Lockheed Martin in El Segundo)
View Agenda (PDF)
Workshop Presentations:
Edward L. Burnett; LM Senior Fellow – Skunk Works; Lockheed Martin Corporation; The Skunk Works® Bringing the Future Forward
Joe Hamaker, Ph.D.; Senior Cost Analyst, Galorath Inc.; Dr. Strangelove Or: How I Learned To Stop Worrying and Love Parametrics”
Christopher Rush, Ph.D.; Product Lead Developer, Galorath Inc.; Continuous Fiber Composite Part Cost vs Production Volume By Manufacturing Process and Material”
Randy Jensen, Ph.D.; Software Acquisition Consultant; Effectiveness Formula: Key To Improved Software Development Productivity
Melissa Winter; PRICE Systems, LLC; Data Analysis in Support of Defensible Estimates
September 17, 2014 Workshop Presentations (at Boeing in Huntington Beach)
View Agenda (PDF)
Paul Geery, Director, Boeing C3 (Command Control Communications) Solutions; “Huntington Beach Site Overview” (PDF)
Bill Seeman, Chief, Space Division, Air Force Cost Analysis Agency; “The Five Pillars of Credible Cost Analysis” (PDF)
Omar Mahmoud, Lead Associate, and Blaze Smallwood, Associate, Booz/Allen/Hamilton; “The Agile Project Management Tool – Effectively Managing the Three Dimensions of an Agile Project: Cost, Schedule, and Scope” (PDF)
David Bloom and Danny Polidi, Raytheon Space and Airborne Systems; “Building a Complex Hardware Cost Model for Antennas” (PDF)
Arlene Minkiewicz, Chief Scientist, and Shawn Hayes, Independent Consultant, PRICE Systems LLC; “Space Missions Cost Estimation Model” (PDF)
Kathy Kha, Consultant, Galorath, Inc.; “Bottoms Up Estimating of NASA Instruments Using Technical Parameters“(PDF)
March 19, 2014 Workshop Presentations (at Galorath in El Segundo)
View Agenda (PDF)
Dr. Daniel A. Nussbaum, Chair of the Naval Postgraduate School Energy Academic Group; “The Nexus Between Acquisition and Energy Studies at the Naval Postgraduate School” (PDF)
Dan Galorath, President, Galorath Inc; “Estimates Without Bias” (PDF)
Bob Hunt, Vice President, Services, Galorath Inc.; “The Cost of Software Testing and the ROI” (PDF)
Dr. Randall Jensen, Software Acquisition Consultant; “The Effectiveness Formula: A Quantitative Measure of Organizational Capability” (PDF)
Anthony Shao & Elizabeth Koltz, PhD Candidates at USC (University of Southern California) James Wertz, Technical Advisor; “Performance-Based Cost Modeling – Quantifying the Cost Reduction Potential of Small Satellites” (PDF)
May 21, 2020 Hypersonic Cost Study
RAND Corporation
Guidance on Hypersonic Cost Questions
Dear ICEAA Members:
A RAND Corporation team has provided a hypersonic cost survey at this link https://www.surveymonkey.com/r/PW6CF27 and invites your participation in the survey –regardless of your level of familiarity with hypersonic systems. This is an exercise in the “wisdom of crowds,” and the team will provide a summary of responses upon receiving a sufficient number. The survey is anonymous, and the summary will include aggregate statistics.
For guidance, we are posting a short set of slides containing an introduction to hypersonics and a more detailed introduction to the survey. Please feel free to look at those before responding to the questions at the link. We hope a crowd of ICEAA members will provide input!
Thanks in advance for your participation!
Best wishes,
Lisa Colabella, Ph.D. and Éder Sousa, Ph.D.
RAND Corporation
April 8 & 15, 2020 Webinars hosted by RAND
The ICEAA Southern California Chapter will be holding a series of webinars hosted by RAND Corporation on April 8 & 15, 2020.
These webinars discuss RAND’s studies and methods for gathering quantitative cost data. Please note that you have to register separately for each webinar. The event is open to all (including non-ICEAA members) and is free of charge.
April 8: Wisdom of the Crowds: Hypersonic Cost Virtual Workshop
Time: April 8, 2020: 11am – 12pm PST.
Speakers: Steve Serk, Lisa Colabella, Éder Sousa, Lieutenant Colonel Paul L. Muller
Abstract: In wicked hard problems, it can be difficult to get quantitative cost data. One method is to collect wisdom from subject matter experts, thus distilling expertise into a quantitative form. In this virtual workshop, we will discuss the wisdom of crowds and how it can be used to help inform cost modelling. In line with the concept of crowdsourcing, we will then draw on the wisdom of ICEAA members to inform hypersonic weapon system cost estimating. More specifically, we will provide a brief introduction to a RAND study on hypersonic weapon system costs and then will ask workshop participants a set of questions about such costs. We will reconvene a week later (via Webinar 2) to share the results.
Speaker Bios:
Steve Sterk is a Cost Engineer and Program Analysts serving NASA at Armstrong Flight Research Center. He leads the Research Center in developing credible cost estimates for both NASA Programs and Projects, as well as all Reimbursable Projects from either: Other Government Agencies (OGA’s), Department of Defense (DoD) or from Industry. He has just completed training in Advanced Program Management and Advanced Systems Engineering, led by Mark Wilson, CEO at Strategy Bridge. In 2017, Steve received, NASA’s Silver Medal Award from NASA (acting) Administrator, Mr. Robert Lightfoot for developing, NASA’s X-Plane Parametric Cost Model. Steve received “Best Paper and Presentation” at the International Cost Estimation and Analyst Association (ICEAA) workshop held in Bristol, England. Prior to his seventeen years of civil service duty, he worked at Boeing’s Space Shuttle Facility in Palmdale, he worked on Space Shuttle Main Engines at Rocketdyne, Canoga Park, as a Test Engineer for Rockwell International, certified/DD250 100 B-1B Bombers to the Air Force, for 19 years. He started his Career in 1978 as a Co-Op at Edwards Air Force Base (EAFB) reading on the bloom of the F-1 Apollo Rocket Engine at Air Force Research Lab (AFRL). He has a Master’s degree from Pepperdine University and Computer Science Degree from the University of LaVern
Lisa Pelled Colabella is a Senior Operations Researcher at RAND. She has led projects on Army logistics topics (fleet management, sustainment data quality, sustainment performance metrics, global equipping strategy, and force generation) as well as projects on Air Force system acquisition costs and business intelligence. In addition, she has served as a team member contributing to a variety of other projects on Army logistics, personnel, and training; Air Force acquisition and workforce management; and homeland security technology and staffing. Lisa joined RAND in 2000 but spent three years (2011-2014) at the Air Force Space and Missile Systems Center (SMC), where she helped develop space system cost estimating guidance and conducted research on cost estimating techniques. Prior to joining RAND in 2000, Lisa was an assistant professor at the University of Southern California, where she taught courses and conducted studies in the field of organizational behavior. Lisa holds a Ph.D. in Industrial Engineering from Stanford University, an M.S. in Operations Research from Stanford University, and a B.S. in Systems Engineering from the University of Virginia.
Éder Sousa joined RAND Corporation as an Associate Engineer in Santa Monica on January 2019. His research focuses on space related topics, Air Force acquisition costs, modeling and simulations, US Coast Guard force planning, and some topics in AI and machine learning. Recent work has focused on hypersonic and Air Force engines acquisition costs, assessment of commercial spaceports, and commercial launch market analysis. Prior to joining RAND, he was a computational scientist for the Air Force Research Laboratory (Rocket lab) at Edwards AFB where he developed modeling and simulations software for advanced plasma propulsion concepts. Éder holds a PhD in Aeronautics and Astronautics from the University of Washington, and B.S and M.S. in Aerospace Engineering from Embry-Riddle Aeronautical University.
Lieutenant Colonel Paul L. Muller currently serves as the Materiel Leader, Advanced Extremely High Frequency Space Segment Branch, Space and Missile Systems Center, Los Angeles AFB, California. He directs the space vehicle design, production, integration, and test for the 14 billion dollar constellation delivering nuclear command and control to the President of the United States. Lieutenant Colonel Muller entered the Air Force in 2001 as a Distinguished Graduate of the Reserve Officer Training Corps program at Worcester Polytechnic Institute. Throughout his career, he has served in a variety of positions at the Lab, Center, and Headquarters Air Force levels. Prior to his current position, he led a Congressionally-directed commercial satellite acquisition program. Lieutenant Colonel Muller also deployed to lead a joint US-Afghan team executing Central Command’s number two training program for building partner capacity in Afghanistan.
April 15: How to Use Expert Elicitation for Cost Estimation
Time: April 15, 2020: 11am – 12pm PST.
Speakers: Kelly Klima
Abstract: In Webinar 1, we learned about the Wisdom of Crowds. In Webinar 2, we share a particular method to obtain information from experts called expert elicitation. We will discuss what expert elicitation is and is not, as well as when and how it can be employed for use in cost estimation. We will then share the results of the Webinar 1 virtual workshop activity.
Speaker Bio:
Kelly Klima is a research engineer at the RAND Corporation with over ten years of experience in quantitative and qualitative decision analysis for risk reduction. She is also an affiliate faculty member at the Pardee RAND Graduate School. Recent work has focused on urban adaptation to reduce natural hazard or cyber risks. This includes theory and applications of (1) hazard and vulnerability assessments of hazards, and (2) behavioral and economic decision-making leading to plausible hazard mitigation and emergency management solutions. Klima’s research work supports community resilience for current heat and flooding throughout the world, and has been applied in the City of Pittsburgh, City of St. Paul, counties in New Jersey, Washington, D.C., and New York City. Other areas of expertise include decisionmaking under uncertainty, energy and the environment, and vulnerable populations. Klima has published several journal articles and served on the American Geophysical Union (AGU) Board of Directors and the Steering Committee for the National Adaptation Forum. Klima also serves as an adjunct assistant professor at Carnegie Mellon University and a summer associate mentor. She holds a CFM from the Association of State Floodplain Managers and a CCEA from the International Cost Estimating and Analysis Association. She received her Ph.D. in engineering and public policy from Carnegie Mellon University.
March 18, 2020 SoCal Workshop – CANCELED
Due to COVID-19 and recommendations from the CDC and WHO, our workshop scheduled for Wednesday, 18 March 2020 has been cancelled.
This is in keeping with the direction of our supporting company sponsors to not hold any large meetings. As of now we plan to hold our next scheduled workshop in September.
RAND would still like to provide the presentation, survey, and results of their study on Cost Overrun that they had planned for the workshop. They will be sending out a survey link and virtual meeting information soon to make this presentation available virtually. RAND and ICEAA hope that you will participate and support their work.
Thanks for your understanding.
Southern California Chapter 2021 Board of Directors
| Name | Position | Company | |
|---|---|---|---|
| Mourikas, Karen | SoCalPres@iceaaonline.org | President | The Boeing Company |
| Bloom, David | SoCalVP@iceaaonline.org | Vice President | Raytheon Intelligence & Space |
| Winter, Melissa | SoCalTreasurer@iceaaonline.org | Treasurer | Northrop Grumman Aeronautics Systems |
| Leighton, Nicole | SoCalSecretary@iceaaonline.org | Secretary | Tecolote Research, Inc |
| Hoffman, Darrin | SoCalAtLarge@iceaaonline.org | At Large | Raytheon Intelligence & Space |
| Knowles, Justin | SoCalAtLarge@iceaaonline.org | At Large | Northrop Grumman Aerospace Systems |
| Colabella. Lisa | SoCalAtLarge@iceaaonline.org | At Large | RAND Corporation |
| Pfeifer, Mike | SoCalAtLarge@iceaaonline.org | At Large | Tecolote Research, Inc |
| Apgar, Henry | SoCalAppointed@iceaaonline.org | At Large | MCR LLC |
2021-2022 SoCal Board
President
Karen Mourikas, The Boeing Company
Karen Mourikas is a Technical Fellow at The Boeing Company specializing in Operations Analysis, Affordability Analysis, and Systems Optimization. Her current work includes Production Systems Cost modeling and analysis; Model-based Systems Engineering (MBSE) incorporating cost and affordability analyses; Product Optimal-cost analyses, involving machine learning and natural language processing; Life Cycle Cost analysis; and Affordability Trade Studies.
Karen has MS degrees in Applied Math and in Operations Research Engineering from the University of Southern California.
Karen is a life-time member of ICEAA. She has presented at several ICEAA and ISPA/SCEA conferences and was the recipient of the ICEAA 2018 Technical Achievement of the Year Award. In addition, she is an active member of the SoCal ICEAA board, and has been the Secretary for the past two years.
Karen is married to a Math Professor and has two college-aged sons who are not interested in math! One is studying history; the other preparing to be a pilot.
Vice President
David Bloom, Raytheon Intelligence & Space
Mr. David Bloom graduated from the University of California at Davis in 1983, Mr. Bloom has worked for the Naval Weapons Center, Lockheed Martin, Lawrence Livermore National Labs and since 2008, for Raytheon Intelligence and Space (RIS). Mr. Bloom is currently the Cost Estimation Subject Matter Expert (SME) for the Hardware Engineering Center (HWEC), a Space “should-cost” Estimation Expert and manages the repair and maintenance budget of RIS. He has developed parametric models for all RIS Electronics Products and helped transform the culture of the organization in the use of parametric bidding methodologies. Mr. Bloom has patents and software copyrights along with a number of publications ranging from electromagnetic boundary value problems to cost estimation. In 2006, he won the International R&D 100 Award for innovating a cost effective Gigapixel Camera for persistent surveillance applications.
Treasurer
Melissa Winter, Northrop Grumman Aeronautics Systems
Melissa Winter has worked in the Aerospace community for 17 years, with Service at Northrop Grumman, Raytheon, and PRICE Systems. She spent 5 years at Northrop Grumman Space Technology, where she supported the Pricing and Cost Estimating Community, as well as system engineering. She spent 3 years at Raytheon SAS as a Cost Engineer, developing cost models for Air, Space and Land Based Systems. She then spent over 7 years as a Solutions Consultant at PRICE Systems, where she trained estimators all over the country in Cost Estimation Principles and use of the TruePlanning tool. Currently, she works as an Independent Cost Evaluation Lead with Northrop Grumman Aeronautics Systems Sector.
Melissa has been on the ICEAA SoCal BOD for 6 years, and has acted as Secretary for two terms and Treasurer for one term.
Secretary
Nicole Leighton, Tecolote Research, Inc
Nicole Leighton is a Technical Manager at Tecolote Research. She has over 16 years of experience in the aerospace and defense industry with roles in program management, data analytics, portfolio analysis, financial forecasting, pricing, proposal development and cost analysis for various DoD programs. She began her career at Northrop Grumman in 2004 developing proposal estimates and providing in-depth cost and schedule analysis on DoD program trends. Nicole then started working at Tecolote Research in 2008 where she was a cost analyst for the Air Force Launch Vehicle program performing cost estimating, business case analyses, cost management and financial modeling in support of budgetary negotiations and program execution. Nicole now oversees the cost and earned value support efforts to SMC’s Financial Management Division who is responsible for cost estimates on all Space and Missile Systems Center (SMC)’s major acquisition programs.
Nicole has a BS in Statistics from Cal Poly, San Luis Obispo.
Board Members at Large
Darrin Hoffman, Raytheon Intelligence & Space
Dr. Hoffman received his PhD from Northwestern University in Optical-Electronics with a focus on III-V semiconductor materials for infrared imaging. He has worked at Los Alamos National Laboratory and he is currently a Senior Principal Systems Engineer for Raytheon Intelligence and Space (RIS).
Dr. Hoffman is currently a technical staff member for the Specialty Engineering part of the System Engineering Directorate. He is RIS’s subject matter expert on the development and use of parametric cost models for Raytheon’s products that include airborne subsystems, space vehicles with particular attention to imaging payloads, and numerous other product lines. Dr. Hoffman’s goal is to provide guidance in the product development life cycle to ensure that cost is always used in any trade study. He has supported several hundred proposals and programs during his Raytheon tenure with a focus on cost realism, improving proposal efficiencies, design to cost efforts, and support of overall program strategy. In addition to Dr. Hoffman’s expertise on parametric modeling and its tools, he has published over 30 peer reviewed papers and has presented his results at several conferences, including methods to develop infrared imaging sensors, physical material characteristics versus optical and electrical properties, methods to improve proposal development, and modeling of space payloads costs.
While Dr. Hoffman has always tried to find balance between his technical expertise, including his cost engineering knowledge, his goal has always been to help others by finding the balance in technical offerings for the warfighter. Dr. Hoffman recently completed his MBA at Pepperdine University so that he can increase his skills and better understand how technical challenges need to be balance with corporate and fiscal responsibility.
Justin Knowles, Northrop Grumman Aerospace Systems
Justin Knowles is currently the Northrop Grumman Aerospace Systems (NGAS) Sector Manager for Strategic Decision Analysis (SDA). He has 20+ years of DoD Cost Estimating, Business Management, and Position-To-Win (PTW) experience. Prior to his current responsibilities Justin was a Technical Fellow and the West Coast Cost Estimating manager for TASC, Inc., and the Business Manager for a $500M+ ACAT-I production program. He has extensive experience in parametric estimating in the Aerospace industry having provided cost estimates on multiple prototype, production, and upgrade aircraft programs, both manned and unmanned platforms. He has used that experience to predict sales of albums in the music industry. In addition to his cost estimating work, Justin has supported multiple Independent Cost Estimating (ICE) efforts for both contractors and DoD agencies. He is a Certified Cost Estimator / Analyst (CCEA), is a Lifetime Member of ICEAA, has been a regular instructor for CEBoK classes at the annual ICEAA conventions and is currently serving on the Southern California Chapter Board of Directors.
Justin completed his undergraduate degree in Finance / International Finance / Business Management at Brigham Young University and received his MBA in Finance at Pepperdine University. He has also worked as a Cost Estimator and Financial Engineer for Lockheed Martin Skunk Works and was a Finance Manager at Warner Music Group.
In his spare time, Justin enjoys spending time with his family, exploring new ways to invest, and road cycling.
Lisa Colabella, RAND Corporation
Dr. Lisa Pelled Colabella is a member of ICEAA and a Senior Operations Researcher at the RAND Corporation, where she has led a variety of projects on cost estimating (e.g., hypersonic weapon system cost estimating, acquisition business analysis, estimating operating and support costs of weapon systems). She has also contributed to many other projects calling for cost estimates (e.g., studies of equipment recapitalization, global equipping strategy, and tailored/lean equipping).
Although Lisa began working at the RAND Corporation in 2000 and has spent much of the past two decades there, for three years (2011-2014) she served as an operations/cost researcher in the Space and Missile Systems Center (SMC) at Los Angeles Air Force Base. At SMC she worked in the Financial Management and Comptroller Directorate; earned a Level II APDP certification in Business-Cost Estimating; developed guidance documents to facilitate aspects of cost estimating (e.g., should-cost estimating, space system operating & support cost estimating, business case analyses); and contributed to cost estimates of space systems.
Lisa holds a B.S. in Systems Engineering from the University of Virginia, an M.S. in Operations Research from Stanford University, and a Ph.D. in Industrial Engineering from Stanford University. In her spare time, she enjoys playing some piano and flute; hanging out with her family; and attending her teens’ basketball, volleyball, and tennis games.
Mike Pfeifer, Tecolote Research, Inc
Mike Pfeifer is a Senior Technical Manager at Tecolote Research. He began his career at Tecolote in 1996 supporting numerous space and launch vehicle estimating efforts for the Air Force’s Space and Missile Systems Center (SMC) and NASA. He developed estimates for the Titan II and Titan IV launch vehicles and performed cost comparisons to future systems that supported the acquisition of the Evolved Expendable Launch Vehicle (EELV) program. He supported the development of the Unmanned Space Vehicle Cost Model (USCM) and oversaw the release of both the 9th and 10th editions of USCM. He supported numerous space system data collection efforts and was instrumental to the addition of sensor payloads into the USCM database. Mike has helped grow the USCM database to include over 130 combined spacecraft and payloads. Mike has led support efforts to SMC’s Financial Management Division who is responsible for cost estimates on all of SMC’s major acquisition programs. He worked collaboratively with programs across SMC to develop Single Best Estimates and Service Cost Positions that are used to establish program budgets.
Mike currently oversees Tecolote’s SMC support contracts spanning Cost Estimating, Cost Research, Earned Value, Budget Support, and Acquisition Management for SMC’s Atlas Corps, Development Corps, and Portfolio Architect. Mike has a BA in Applied Mathematics from the University of California at Berkeley.
Henry Apgar, MCR
Henry Apgar is an ICEAA lifetime member, frequent presenter and trainer, and Certified Cost Estimator/Analyst (CCEA). He was co-founder of the International Society of Parametric Analysts (predecessor of ICEAA) and served as its President, Chairman of the Board, and occasional Conference Chair; he has been honored with its Lifetime Achievement (Freiman) Award.
Hank is a Technical Director for MCR where he develops parametric cost estimating models and prepares independent cost estimates for international clients. He previously worked for IBM, Bunker-Ramo, and the Aerospace Corporation; he also served in the United States Air Force.
As an elected Board Member, Hank seeks to expand the subject scope and participation diversity in our local workshops.
Hank has a BS degree in electrical engineering and an MBA. He authored the Cost Estimating Chapter for the Space Mission Engineering Handbook (formerly SMAD) and he is an elected Associate Fellow of the AIAA. In his spare time, he serves as a docent at the Ronald Reagan Presidential Library in Simi Valley.