Why Is NASA Is Stuck Paying For Northrop Grumman Errors?
According to #JWST IRB chair Tom Young simple fixes that weren't implemented by @NorthropGrumman ended up costing #NASA $600 million. Delays now cost $1 million/day. One would hope that @NASA is examining ways to get Northrop Grumman to refund the cost of fixing their mistakes. pic.twitter.com/pRWrDdOVrJ
— NASA Watch (@NASAWatch) June 28, 2018
Heads up @NorthropGrumman There is a precedent for @NASA to follow for seeking reimbursement for contractor errors in the construction of #NASA Space telescopes: "Hubble Space Telescope Makers Pay for Costly Mirror Mistake" Nov 1993, Physics Today https://t.co/kmVpSA0kV2 pic.twitter.com/YHRLspqBFJ
— NASA Watch (@NASAWatch) June 28, 2018
Gee whiz, Keith. Don’t you read about the team spirit? That everyone is responsible for the error?
Maddening.
Did anyone else curse under their breath when they read: “The error should have been detected by the inspector, who did not inspect, but relied on the technician’s word that he had done the wiring correctly”?
What’s the point of QA, when they don’t actually assure any quality?
We want names.
I’ll leave the liability issues to the lawyers. I’m sure Northrop Grumman had plenty of them on staff when the contracts were written…
But I missed something the first time I read the report: “To address a risk that fasteners for sunshield membrane covers might snag the membrane, the fastening lock nuts were tightened only to be flush with their bolts…”
That explains why there was something rattling around after vibrational testing, which isn’t good. But I wish the report had gone into a little more detail, because this raises some other red flags in my mind.
That’s a serious violation of the “test as you fly” principle. Tests are supposed to be done in a flight-like configuration, to the greatest extent possible. Of course, that’s never entirely possible (e.g. mechanical deployments are tested in gravity, which isn’t flight-like.) But the exceptions really should be deliberate and considered. It’s definitely something you want more than one pair of eyes looking at. I hope that was true in this case. But I’ve also noticed that formal processes can get cumbersome enough that people sometimes avoid them.
The other concern is the perceived risk. If, when tightened down to their designed and flight state, the lock nuts might be a risk of snagging something, that’s not something that should be side-stepped by testing in a non-flight-like manner. Maybe someone thought about, decided it was a very small risk, decided taking a very small risk more than once was a bad idea. I could understand that. But, again, that’s the sort of decision that needs some deliberation and documentation. It isn’t clear if that was the case.
why do you think they tested a non-conforming specimen ? I read that to mean they intentionally disregarded common practice (to have 2-3 threads showing on the outside of the nut) to optimise a different design property … instead of other fixes, possibly domed a/nuts, although there a many design considerations even for such a seemingly simple issue ?
I have no idea about the lock nut, since the report doesn’t go into much detail. But there are good reasons for testing a non-flight configuration.
The one I’m most familiar with involves instruments using high voltages. They often have a risk of arcing if you turn the voltages up in a poor vacuum. 1e-6 Torr is a common limit. In spacecraft level thermal vacuum tests, that can be impractical. With the whole spacecraft involved, outgassing and pumping down to that hard a vacuum could take weeks. It isn’t uncommon to do the spacecraft level tests with the voltages down (even though that means you don’t have all the electronics producing heat in an entirely flight-like way) and supplement the spacecraft-level tests with stand-along tests and modeling. That’s not a problem, but it isn’t something you’d want to do without some careful thinking.
The fact that some of the JWST testing wasn’t flight-like doesn’t worry me too much. But, brief as it is, the description of the lock nut business makes me wonder how carefully they were thinking things through.
Hire a couple lobbyists, drop a few thousand into congressional coffers and the contracts get written to avoid responsibility… just another day at the office on capital hill.
My worry is that similar things are happening with commercial crew and will cost astronaut lives. http://Www.robert-w-jones.com
Evidence?
2 falcon 9 explosions
So you don’t think SpaceX is up to transporting astronauts, that their hardware is unsafe?
I probably shouldn’t feed the troll, but I’m tempted to say something about SLS/Orion. Flying astronauts on the second or third flight of a new launch vehicle? Or the first flight of the Exploration Upper Stage?
Isn’t that more concerning than two failures due to solved problems out of 57 flights? Or, if you want to stick the same configuration, no flight failures in the full thrust version (block 3 to 5, by some reports.) If you count the pre-launch Amos-6 failure, that was the first full thrust version, and SpaceX is a perfect record for twelve flights of the Block 4 and one flight of the Block 5.
I’m afraid I get a completely different impression about NASA oversight, JSWT and commercial crew versus SLS/Orion. JSWT proves that, even with a large amount of NASA oversight, you can have problems. Potentially serious ones. Therefore, the idea that SLS/Orion is safe because of NASA oversight (i.e. safe enough to fly astronauts on without lengthly unmanned experience) is very questionable. That oversight doesn’t seem to be a magic trick which makes things fine for SLS/Orion and not for commercial crew.
I am also concerned with the small number of CST-100 flight tests and the use of solid rocket boosters on Atlas 5 and much much more.
I notice that over the last day or two, suddenly this is all the contractor’s fault. I would recommend reserving judgment on that point – the chief thing one can conclude with certainty from such sudden public finger-pointing is that someone within the project is finally convinced that it’s in serious trouble.
It is my understanding that with such high-profile NASA contracts that NASA tends to be intimately involved in the details at all levels. Obvious questions occur:
– Who decided not to tighten those nuts down to reduce snagging potential? It strikes me as extremely unlikely the contractor made such a critical tradeoff decision with no NASA input.
– Was NASA at the table in the meeting where they decided on solvent X to clean those valves? (There’s just about always a meeting. Such projects tend to spend the majority of their personnel-hours in meetings.)
– Who did the inspector who signed off on the wiring harness without testing work for? N-G or NASA?
Mind, I wouldn’t be at all surprised if some of the JWST problems do turn out to be primarily the contractor’s fault.
I would be utterly gobsmacked however if ALL the problems are solely the fault of the contractor – or even a majority of them. Such projects tend to be the complete opposite of hands-off don’t-joggle-their-elbows project management. I expect that the NASA project management team has been deeply involved at all levels. “Not NASA’s fault” should require a great deal of proving here before anyone buys off on it.
The report is missing some details, so it’s hard to say who is to blame. For example, with the solvent and valve issue, that meeting you mention probably had other things on the agenda. That would likely come up in a review and approval of an assembly procedure or checklist. If there are enough items on that list, it’s not hard for something to slip through the cracks. Did it occur to anyone that they shouldn’t use whatever solvents they normally use? Or, if the part could be damaged by the wrong solvent, why should they need to contact the vendor? Shouldn’t that be the sort of information the vendor should provide without being asked?
But the careful and extensive oversight can be a problem in and of itself. Every single problem doesn’t end up on the agenda for all those meetings. At some level, people do make individual decisions about whether or not a problem is serious enough to be worth reporting. Unfortunately, reporting a problem can trigger a sizable amount of extra effort, meetings and paperwork. The more extensive and through you try to make the oversight, the more you create a tendency (conscious or unconscious) for people to say, “that’s not a big deal; let’s not get everyone worried over nothing.” And that can let serious things slip through.
The problems with the prime contractor (NG) are far deeper than the improperly tightened bolts. Anyone who was paying attention will notice that there was no actual design for the spacecraft beyond some notional powerpoint slides for far too far along in the project’s lifecycle. And this fundamentally is what blew up the cost by creating years of delays. Now, why was this? Shouldn’t NASA have noticed? Yes, but every time this was brought up, NG swore up and down that they had built space-deployable platforms like this, and it would be no big deal. What platforms, you ask? “We could tell you, but then we would have to kill you. Wink, wink, nudge, nudge.” There was a clear misdirection to believe that NG had built classified spacecraft like this before, and there was certainly precedent for this sort of thing looking back to the history of the development of the underlying platform for HST. But it turned out to be BS, in the end.
I’m not sure I’d go that far. It’s pretty clear _someone_ has been making large deployable structures for classified satellites, and Northrop Grumman is a likely suspect. In other contexts (e.g. the Hubble solar arrays which were replaced in the first servicing mission) the problem isn’t a lack of related experience on classified projects. It’s the fact that those projects are classified.
Even if someone at Northrop Grumman had built a 18-segment mirror for a classified project, they could not simply give the blueprints to the people working on JWST. Even if the same people were involved, they couldn’t talk about or reference those blueprints. The classification of a project makes any technology transfer to unclassified work extremely difficult (if not impossible.) That’s a known problem with defense contractors claiming heritage. It sounds good when you’re trying to get a contract, but even if the heritage is real, being able to use it is a different matter.
By the way, it’s typical for spacecraft designs to be largely notional until the preliminary design review. That’s because you don’t want to waste time and money on detailed blueprints of a system the review board may ask you to change in major ways. It’s at the critical design review when you’re expected to have drawings you can start building to.
The mirror isn’t the deployable structure in question, that part is comparatively easy. In fact, it’s substantially simpler than the prototype of the Keck mirrors, which require much more constant phasing due to the constantly changing gravity vector. It’s the deployable sunshade, without which the mission doesn’t work, and it’s been obvious for at least 5 years that NG didn’t have the slightest clue how to build it. Since I specifically spent 30 years working on infrared mid and far-ir space telescopes, I’m pretty sure I know far too much about them and the unending debacle JWST has been. JWST’s telescope assembly and instrument package were well on their way to completion before a credible design for the spacecraft actually emerged. And that has huge implications for the whole thing in terms of thermal management. Hell, just look at the amount of plumbing required for the MIRI cryocooler. Any discussion of “well, we didn’t do X because of the CDR”, etc is part of the whole history of JWST moving the goalposts. In the end we got a mission that was much less capable for nearly an order of magnitude more money.
I know the main concern is the Sun shield. I’m just a bit concerned about the mirrors as well. I don’t think it’s easier than Keck. JWST needs precision alignment without anyone touching it. I know the mirrors are articulated, but that’s for fine adjustments. The deployment needs to get them very close or there would be a big problem.
I definitely see what you mean about the development schedule. You aren’t describing vague designs at PDR, you’re describing the instrument/telescope and spacecraft/Sun shield designs being badly out of synch. That’s not good; I saw the same thing with Cassini (operability and IR instrument thermal requirements were one of the problems.)
If the instrument development is too far ahead of the spacecraft, it’s easy to make problems someone else’s problem. Just say, “I have a fine design which satisfies all the requirements for my instrument. I just need to require the spacecraft to provide X, Y and Z.” If the spacecraft development isn’t far enough along, then they aren’t in a position to say, “Y is either physically impossible or likely to double the cost of the mission. Can you live with 85% of Y?” The requirements just get levied and frozen in, and there isn’t space for any feedback about the implications to the mission’s cost or schedule.
By the way, as a planetary scientists, what really made me start worrying about JWST was learning it can’t observe planets at opposition. The thermal design limits observations to 90 deg. from the Sun (give or take 10 deg. if memory serves.) After thinking about it, I can understand why. But it made me realize this was a thermal problem which produced a dog’s breakfast.
FWIW, Lockeed-Martin was doing a similar tapdance twenty-five years ago regarding X-33. “We’ve got lots of experience at this sort of thing, but we’d have to kill you if we told you about it, nudge nudge wink wink.”
Whether L-M actually had any such corporate experience is still unknown – but moot. Most of the people L-M actually put on the X-33 development after they won clearly did NOT have “black” experience with anything similar, else they’d not have botched so many essential project fundamentals.
Then again, in both cases they had the benefit of a crack NASA program management team attempting something NASA had certainly never done before.