NASA's Cost Estimating Process Is Flawed. Duh. Who Knew?
OIG announces an audit assessing the effectiveness of NASA’s cost estimating and reporting practices for large, multi-mission programs such as those supporting the Artemis program.
— NASA OIG (@NASAOIG) October 8, 2020
Keith’s note: This audit is stating the obvious. Its nice to see that OIG has finally decided to audit “NASA’s cost estimating and reporting practices for large, multi-mission programs” given that all of its audits of “NASA’s cost estimating and reporting practices for large, multi-mission programs” have demonstrated that these practices are woefully broken and have been for decades. I guess it is better late than never to start studying the core cause whereby all of these NASA programs are over cost and behind schedule year after year after year.
It works as designed (but not as reported by NASA). It is always initial proposed cost + all management reserve.
Sometimes it takes them a few extra years to burn all that money.
Having seen it up close I can tell you the contractor knows what bid is desired and what the reserve will be. Most of the time (on major programs) a winner is known before the 1st proposal is received.
Speaking as a “cost engineer” or “engineering economist” I believe that NASA is now on the right track using a “fat” or “long tail” distribution is both appropriate and well documented.
The major problem remains (as Prof Bent Flyvbjerg’s research has shown) the two biggest problems are “OPTIMISM BIAS” and “STRATEGIC MISREPREENTATION” (lying)
BR,
Dr. PDG, Jakarta, Indonesia
100% agree – two biggest problems are “OPTIMISM BIAS” and “STRATEGIC MISREPREENTATION” (lying)
For 10 years, Glenn Butts, NASA’s head estimator hsa long railed at NASA’s cost and duration estimating practices and their latest Cost Estimating Handbook https://www.nasa.gov/pdf/26… IF FOLLOWED is consistent with the GAO’s Best Tested and proven practices. https://www.gao.gov/new.ite…
Here is the link to Butt’s 2009/2010 recommendations. http://www.build-project-ma…
We need to SUPPORT Glenn Butts, Prof Bent Flyvbjerg and others like them.
BR,
Dr. PDG, Jakarta https://uploads.disquscdn.c…
Estimating lifecycle costs for some of humanity’s most complex endeavors, especially in the political and budgetary environment NASA must live in, is not exactly easy. I’d challenge people to identify any analogous cost estimating challenge that performs significantly better (so, equivalent complexity, duration, budget caps and budget phasing constraints)
Cost estimating not hard. I do it every day. Its performance and accountability that’s hard.
Good examples:
C-17 back in 1987 – McDonnell Douglas (MD) had a fix priced contract for an N number of aircraft + development. Very bad management left the program in overrun and with significant quality issues. I will never forget seeing pillows being extracted from a wing tank. MD expected the USAF to just send more money regardless (the USAF needed a replacement for the C-141). The response, “no and give me my airplanes.” That almost bankrupted MD. After major changes in MD leadership, a true deep clean up of the production line, and some other needed fixes the USAF got its original batch of airplanes (and to be fair later on the Gov provided about 400 million in advanced payments to keep the company going.) Today the C-17 is awesome and part of the USAF, RAF, RAAF, etc..
Or look at James Webb today. By the time all is paid for it’s going to be over 13.5 billion dollars for one spacecraft. The ride to orbit is free. The fine guidance telescope (thank you Canada) is free. JWST was never costed that high. Over the many years, the design has not changed, the requirements are the same, no ground breaking new technology involved. What happened? Contractor/government mismanagement and an open NASA wallet are the problem.
The congress should prevent civil servants from joining any aerospace company for say 3-5 years after retirement. Do that and you can empower government workers to hold contractors accountable. They would have nothing to lose. By the time they became eligible to work original contractor leadership would have changed.
Performance and accountability that’s hard.
“Its performance and accountability that’s hard.”
But isn’t it performance and accountability that is being budgeted/ estimated?
That’s the tangle I get into…at what point are actual history folded into projections.
“Over the many years, the design has not changed, the requirements are the same, no ground breaking new technology involved.”
At the time it was announced I believe there were ten technologies at a TRL of 1 – 3 needed to be created and tested. At that time I said there was no way they would increase the TRL within the proposed funding level and time schedule.
JWST was originally conceived in 1996 as a $1 billion telescope. In 2005 is was about $3 Billion. How do you get to $13 billion? Even if you costed in the development of the wheel forward it would not be this much. The USS John F. Kennedy an aircraft carrier under construction is $3.58 billion. $13 billion for one spacecraft? Something is very wrong.
Actually it was 500 mil when it was first proposed.
https://wayback.archive-it….
Fixed price high rate production on an aircraft that has already been designed and manufactured previously is in NO WAY comparable to a unique, one of a kind, state of the art spacecraft that involves multiple new technologies and is a significant leap in complexity, performance, and operating regime. Comparing a C-17 to JWST is like comparing a rowboat to a destroyer.
Saying “no groundbreaking new technology is involved” displays a huge lack of knowledge about what JWST is, and how it works.
I worked on JWST for 3 years. Did you?
That being the case, can help explain the cost increases? In general terms, perhaps?
The aircraft carrier comparison is frankly shocking.
Wait! There’s a process?
It’s probably time for an audit to provide a brief history of NASA cost estimating, a lot of changes having occurred here around Constellation, none for making the field better. In short, it’s difficult to get good answers for costs when the people paying you are clear they don’t want good answers. They want the answer that helps them persist, sell the program, maintain a lie, and not rock the boat. Programs make it clear when the call comes, they only want the answer that never sees the light of day, and if it does, just tows the party line.
At the root of the matter, dishonesty, shooting the messenger, and obliviousness on the part of NASA management all the way to the
top are now the standard and corrupt operating procedure where once there was at least some semblance of interest, some sense the task no matter how uncomfortable was inevitable, even if eventually ignored. Many NASA managers at least pretended to care about costs 20 years ago. This opened a window of opportunity for cost estimators to do some good. Ten years ago even that pretending stopped.
I’ve been in the battles, seeing how independent internal cost
estimates transformed into little more than programs looking for anyone who might tell them what they wanted to hear. For everyone else, if your reputation was to approach a cost estimate with honesty, data and experience to back it all up, the phone stopped ringing about 2010-ish. Gerst personally made sure of this, seeing Constellation as falling prey to its own cost estimates, the programs own numbers used to kill it. Constellation dug its own grave through sheer incompetence. Gerst decided the strategy for the next program was to hide the shovels. Problem fixed.
Eventually this is how you have such Orwellian logic like a NASA
Associate Administrator Gerst telling Berger over at Ars that NASA can’t estimate costs because it wanted to reduce costs (“done by design to lower NASA’s expenditures”). Ignorance is now a feature!
Oh I could go on and write a book about the shenanigans of cost estimating in NASA since the 1990’s, so many offices disbanded, careers side-tracked, the way just the sycophants are mostly left, happily saying at conferences that’s just the reality and you have to be tough-skinned. For what, to assist programs in lying, which obviously you must do as otherwise you would not have a job? Who would have thought, saying “yes”, and “what number, whatever number you want of course” is now a NASA profession!