GAO on Earned Value Management at NASA
NASA: Earned Value Management Implementation across Major Spaceflight Projects Is Uneven
“The National Aeronautics and Space Administration’s (NASA) 10 major spaceflight projects discussed in this report have not yet fully implemented earned value management (EVM). As a result, NASA is not taking full advantage of opportunities to use an important tool that could help reduce acquisition risk. GAO assessed the 10 projects against three fundamental EVM practices that, according to GAO’s best practices cost guide, are necessary for maintaining a reliable EVM system. GAO found shortfalls in two of three fundamental practices. Specifically, we found that More than half of the projects did not use an EVM system that was fully certified as compliant with the industry EVM standard.”
Too much attention to science and technical problems, eh? Not taking advantage of the perfect tool? Sounds like Microsoft talking about the Office Ribbon. Earned Value and other accounting tricks become the way for the perpetually growing army of second guessers to swoop in as soon as the CPI and SPI veer even a fraction of a percent off of 1.00 – a baseline which is usually unrealistic by program start. Trivial problems become crises that require ‘help’ available from any number of organizations set up to provide it. And ensure that it is always needed. In short order the hapless program managers and technical leads become bystanders as progress stops, costs grow, and the indices get progressively worse.
If the GAO recommended more oversight in exuberant cost projections before program award, it would be closer to the mark in cost control. It might even have done so…
I agree. Simply complying with existing reporting requirements can easily absorb a large fraction of the budget. These reports don’t get you any closer to your goals.
Spoken like a true NASA Technocrat that casts blame on designs running counter to the “give me my udget” culture. I bet the author of this comment is responsible for some of the best run financially solvent projects in NASA history. Right. There’s a reason GAO exists. And it isn’t because EVM sucks!
Nope. Spoken as someone who has experienced EVM on both sides of the customer/supplier interface. Chopping work into tiny little packages in a manner acceptable to EVM. Coming up with milestones and measurement methods that EVM can handle. Forcing tech leads to be Cost Account Managers on top of their main jobs or, worse, paying a non-technical person to sit in that role and have to go to the tech person for the inevitable reporting. Having to provide endless types of backup data in a way that EVM will accept though it may not fit the type of effort. Distilling difficult problems and tradeoffs in engineering development to infopellets for fast-breeding accounting rabbits. In short, organizing one-or-few-of-a-kind R and/or D tasks into a form acceptable to an accounting method instead of the other way around. And explaining the inevitable variances month after month. It may work for production programs where the R&D is done and the risk reduced. But, it doesn’t really. The DoD hasn’t been able to deliver a product line of bombs, aircraft, missiles, or whatnot to cost or schedule either despite decades of mandated EVM. Like TQM, Lean, Six Sigma, ERP, or any fad du jour, it is popular with career managers and now is one of THE primary measures of whether to kill a program or let it continue.
I am glad the GAO exists. When it becomes a vehicle for pushing a questionable tool as a gold standard surefire way of solving a problem when it doesn’t do any such thing, I just die a little more on the inside.
Do you by any chance work for a small business that provides systemengineeringandprogrammanagementsolutions(tm) to government?
While well-intentioned, management tools like EVM are part of the problem. There are way too many second-guessers who’s only product is a process…often hurting more than helping.
So, why not try something different? like being realistic about what can be accomplished and building real trust and give them the responsibility, authority and hold people accountable. What do we have to lose? Couldn’t hurt.
Give the following article a read…it focuses on DoD, but the fundamentals are the same for NASA…
http://www.dau.mil/pubscats…
Thank you. That was a terrific article.
AMG40,
While there are important ideas in the above document that we all need to be aware of, it and others like it seem to kill their own credibility through the use of invalid comparisons. Examples:
1. “We are currently paying eight times the cost per pound for fighter aircraft that we did in the 1940s“
Considering the magnitude of the increases in complexity and capability over the decades involved, “cost per pound” is a meaningless metric.
2. “started our intercontinental ballistic missile program in 1955, and “developed three generations of systems … in a mere seven years“
Comparing fighter development (which had been the topic prior to this sentence) to ICBM development is, again, a meaningless metric, because of the differences in complexity and capability.
Unless all comparisons are apples and apples they do nothing to substantiate an argument. I agree with you that “management tools like EVM are part of the problem.” Trying to apply a generalized process to a specific industry where the shoe doesn’t really fit makes for more work and less results. Management/reporting systems need to be precisely tailored to a specific industry.
Steve
To my reading those comments were just lead-in quotes from other people who had complained years before to show the long-standing nature of the problem.
EVM is critically dependent on the statistical reliability of the initial project estimates. And, its developers therefore never expected it to be applied to research and development. It must be used with extreme caution when there are more than a few new elements or when integrating known elements in new ways (pretty much most NASA projects).
IMHO, we would be better off spending some depth study about why initial estimates are so off and trying out diffrrent ideas about how to improve them than requiring EVM (from someone with a PMP).
I have found this to be very true and agree with your study idea. Good PMs can and do use EVM and other tools wisely – that’s what makes them good. Unfortunately, as soon as these concepts become products and services, they become an end unto themselves. The assertion quickly becomes that the tools make a PM good and are applicable everywhere. They don’t and they aren’t.
Great idea! We could start by figuring out why the shuttle was ten (10) times as expensive to operate as was predicted, using the magic of systems engineering. No one seems to want to ask.
The Shuttle was nearly 100 times more expensive than predicted. I have a GAO report from the early ’80s that criticizes the program saying it would not be $7 million per flight as promised, but could end up costing more than $25 million per flight, boy did they have it wrong. I can think of 3 reasons why it ended up costing between $500 million to $1 billion per flight; 1) the design was on the hairy edge of what was technically poosible, not allowing for any operational effiiency during design with performance always being tbe driver, 2) it quickly became a jobs program used by management to justify 10 times the number of people actually needed in order to build empires, 3) the program was required to fund nearly 85% of the Agency’s infrastructure so there was nsver any money to upgrade the system for operational effiiciency.
I agree on all your points, though there was some inflation. But the actual predictions of manhours for processing were way off, too, despite the use of “management tools”. All these strategies have a dirty littleMany steps in maintenance and fabrication were much more time-consuming and expensive than anticipated. Part of the problem was the decision to go directly from plans to production without getting any actual flight experience at the prototype level as was traditional in aviation. This shortcut approach worked, pretty much, with Apollo, but we were lucky. When you are pushing the state of the art there is a lot you don’t learn until you fly.
EVM is a tool, a very powerful and well-proven tool, that informs a PM on how the project is performing against the plan. Only an idiot would not want that, or maybe a NASA PM. They hate EVM because it shows how badly their project is performing. Now, understand that this is a result of the fact that NASA PM’s never have enough money to do the job from the get go. Not even close to having enough money to do the job. Part of this is due to a lack of funding, but more importantly, it is largely due to the fact that they have to pay for the existing bloated, incompetent workforce at NASA. Projects are sucked dry by the Agency’s infrastructure in both personnel and facilities. But don’t blame EVM.
All meaningful research and development involves uncertainty. The program manager has to continually deal with unexpected events, both problems and opportunities, and be capable of making both technical and business decisions on the fly.
DVM is just the latest in a long stream of management schemes that were introduced with similar fanfare. DVM seems to be designed primarily to let administrative managers who don’t really understand the technical and cost issues force their technical people to present everything to them in predigested pieces that will let the manager make simplistic decisions in a totally defensible way that will protect his career whether the program itself succeeds or fails. This and all the other reporting requirements don’t generate any useful understanding and consume a surprising fraction of the total resources of the people on the front lines. I personally have seen management meetings at which more time was spend discussing why burn rates were not exactly matching what had been predicted a year ago to the man-hour than what the projects were actually doing to provide practical return to the taxpayers.The problem also exists in contractor management, which in many cases places a high priority on getting the maximum profits out of taxpayers and always chooses profitability over mission success, because to them profitability IS mission success. Don’t expect costs to go down if your organizational goal is actually to make them go up.There are managers in both NASA and industry who combine instinctive understanding of both cost and technology with a drive to make the program work at a reasonable cost. I would put both Wayne Hale and Elon Musk in this group. But they are not common. And they don’t depend on the management fad du jour.
Abuse and misuse of the tool does not make the tool useless. If you prroperly have the work defined and the funding codes structured, EVM is practically free. This is the problem. PMs, especially at NASA, don’t know how to define a WBS properly and don’t know how to track task completion. If you can’t do these things, you will always be lost and no tool will help. And any attempt to use any tool will take tremendous resources and provide little help. This is how it is at NASA. Tons of people trying to track and make sense of absolute mayhem. So don’t blame EVM, it is a proven useful method in the real world.
nasa817,
You’ve hit on what, to me, has always been the weak link in any PM scheme — the human element. The theoretically simple job of people reporting their progress and the PM then recording that progress into a “system” almost always manages to fail for any complex program.
A PM “system” returns a report based on the progress data input to it, but that data is almost always skewed, sometimes through a human tendency to try to make things look better than they are (somehow I’ll catch up next week) and sometimes because accurate progress reporting is a skill — a skill in which technical people are almost never given any sort of education or training. It’s assumed to be so simple that anyone can do it from birth, but results show that’s not true.
Also, the inaccurate progress reporting gets further skewed as you move up and down the chain of command, and as you move through time. I’ve worked for or with several companies who brought in consultants to teach their engineers program management (Pert, Gantt, etc.), but not once have I seen anybody teach or ask about how to effectively do progress reporting, which, I maintain, is where most of the PM problems originate. If I’m right, then the engineers, the PM, and any tools used are all equally “to blame.” The enormous deviations in program actuals all grow cumulatively from inaccurate week to week reporting data.
Steve
dogstar29 : “DVM seems to be designed primarily to let administrative managers who don’t really understand the technical and cost issues force their technical people to present everything to them in predigested pieces that will let the manager make simplistic decisions in a totally defensible way”
I agree with this. The successful managers I’ve seen stay close to the details and know what’s going on. They are able to use EVM when and where appropriate. There are others who want layers between themselves and the work and manage through reports and interrogations. They inevitably get wrong or misleading information with predictable results.
nasa817:
“Abuse and misuse of the tool does not make the tool useless. If you prroperly have the work defined and the funding codes structured, EVM is practically free.”
This has run absolutely counter to my experience. EVM and those who require it specify a particular way to structure and report on work down to how long tasks are allowed to be to pass audit.
Many types of projects do not fit the requirements but have to change their structure to meet the mandate. When a tool does not fit, it does not make sense to double down and force it to be used more.
It is very possible to game the system and it is often done. EVM can be strongly influenced by number of milestones, milestone density, and how the milestones tie into the critical path. It is very possible to have CPI and SPI well over 1.00 and still be in trouble. It is possible to be under 1.00 and not be dead. It all depends on the managers and performers focussing on real situations and using the numbers as intended.
While it is theoretically possible (however unlikely) that external outfits can come in and train people in EVM use, reporting, and such, it does not help cost, schedule, or bang for buck. It just keeps a cottage industry of consultants well fed.
This all reeks of the standardized testing problem. Curricula get reworked to ensure that the test scores are high but the toll on actual people and society is immense.
I don’t “blame EVM”, but no tool should be relied on in
conditions that violate core assumptions about its use. Let’s peel this
onion a bit more and ask why so many NASA projects have such poor
estimates or inadequate WBS. NASA PMs pretty much without exception are engineers or scientists, many of whom have waited years for a chance to lead a space-related program and who have undoubtedly depth knowledge of their field. While they prefer other tasks to some of the PM work, they can’t be ignorant about the need to actually plan for the right work to be done…so why the systemic pattern of poor definition (description, estimates) of the work?
I don’t think laziness or aversion is sufficient. and I’ve got anecdotal evidence that there may be organizational factors at work — meaning factors that may be difficult for individual PMs to overcome, even with the best of intentions. Some or perhaps even most of this resistance to starting out with a realistic project estimate and WBS may derive from being an organization where politicians play around not just in the authorization of projects, but in their definition. I have at at least one former PM tell me that there is no way in h**l that this project would have gotten funded if his true estimate had been brought forward…
Other input from people who help stand up new projects reflects what in other cases might be an admirable “bias for action” to get on with doing the stuff that matters as in “a perfect WBS never built a satellite.”
There is not space or format here to list things that are creating an environment in which even projects for which EVM is a good fit (e.g., ones with out much “unknown” or “new” or significant integration challenges) would have a difficult time being framed properly for EVM application.
What I hear from thoughtful NASA folks about this is that they see the cultural-organizational-political nature of this as something beyond their personal experience, understanding, or impact…and they just want to do rocket engineering…so they do what their colleagues have done and “do what a long time friend calls “budget theatre.”
I am not an engineer, but an organizational researcher — so I think the problem can be studied and changes made (although none are likely to be easy or comprehensive). But GAO’s approach is akin to hammering everything because that’s the tool they’ve got and blaming the screw for being difficult.
Lika2know:
“NASA PMs pretty much without exception are engineers or scientists, many of whom have waited years for a chance to lead a space-related program and who have undoubtedly depth knowledge of their field..[deletia]…so why the systemic pattern of poor definition (description, estimates) of the work?”
My experience is that both the customer and the supplier(s) play “price-to-win”. The former to get the procurement authorized and the latter when contracts/subcontracts are let. Additionally, the people who bring in the work are seldom the people who have to execute to cost and schedule both of which can be laid-in before ATP.
In industry, scheduling, pricing, cost-estimating, quality, and accounting are well outside the control of the Program Manager. These are processes held at the division level or higher. The PM, if s/he is strong, may be able to get the most onerous/least applicable methods formally waived. Changing processes is nearly impossible and ignoring processes gets one fired.
Why technical people as PMs?:
My experience is that keeping close watch on schedule and risk (vs. EVM or other tools for tools sake) help bring in tough jobs of all kinds – R&D or production – on time and on budget or close to them. Technical people are usually the savvy about schedule and risk because they’ve spent the most time with the item(s). They are often overruled to win (the true estimate story). It is still possible to make good things happen if a technical lead/manager can look at problems and make decisions without having to spawn a geometric series of working groups, tiger teams, or committees to follow a process. That’s where having technical chops in the leadership pays off – teams keep moving. A side effect is that morale is higher and people feel comfortable with surfacing problems if they have confidence in the leadership to solving problems vs. endless fact finding or messenger shooting. Taken together, that comes out in snappy schedule meetings, smooth project reviews, and normal blood pressure. The EVM metrics will reflect health accurately in that case.
When projects are overcommitted and underresourced, they will fall behind and cost more. EV will reflect that of course; “You will need x more dollars and y more people to finish.” But those will be the dollars and people cut to win the business.
I also accept that there are consultants who will offer courses in schedule and risk management, just as in EVM, Process control, CMMI, etc. I don’t equate good schedule and risk use as knowing how to wrangle GANTT charts in Project but tool-independent ways of knowing what’s being done, what’s coming up, and how to deal with the unexpected.