Boeing Dropped The Ball Again – And NASA Let Them
Boeing didn’t perform full end-to-end test of its astronaut capsule before troubled mission, ‘surprising’ NASA safety panel, Orlando Sentinel
“Boeing and NASA officials are expected to release the results of an independent investigation into the set of issues that occurred during Boeing’s late December test of Starliner, its astronaut crew capsule, within the next week. But speaking to the Orlando Sentinel, members of NASA’s safety advisory panel expanded on some of the testing decisions Boeing made that drew questions about whether Starliner was ready to fly. Critically, the panel learned early this month that Boeing did not perform a full, end-to-end integrated test of Starliner in a Systems Integration Lab with ULA’s Atlas V rocket. The test typically shows how all the software systems during each component of the mission would have responded with each other through every maneuver — and it could potentially have caught the issues Boeing later experienced in the mission.”
– Boeing Really Needs To Get Their Software Fixed, earlier post
– ASAP: Boeing Starliner Software Issue Potentially “Catastrophic”, earlier post
“Boeing did not perform a full, end-to-end integrated test of Starliner in a Systems Integration Lab with ULA’s Atlas V rocket” – This is inexcusable.
Not necessarily. Full, end-to-end tests are one way of verifying things work as intended. But they aren’t the only way to do it. I couldn’t say if SpaceX did that sort of testing on Dragon 2. But the idea that Boeing didn’t do end-to-end test _or_ something else capable of identifying the problems, that’s something I’d have trouble forgiving.
Agreed. I’d like more information on exactly what Boeing did test. Specifically, I’d like the investigation to find out why Boeing’s testing didn’t find these multiple issues.
And see what SpaceX tested. Just because they made one complete space flight doesn’t mean they did all of the testing.
All indications are Space X did a more thorough testing than Boeing for less cost and in a quicker time. There appear to be two standards.
Whole heartily agree. When you are looking to save money the QC dept. is one of the first areas that get targeted. When this happen to me I immediately started looking for another job. Within 18 mo. the company was out of business. This won’t put Boeing out of business but sure is a big hit on their credibility.
It is inexcusable.
Boeing cannot compete with Space X in time, reliability, or cost. They are in the race only because there is not a level playing field.
I hope NASA holds their feet to the fire. Boeing is not allowed to drop out of this contract.
Is there a cancellation clause? Repaying everything they’ve received to date plus a 15% annoyance fee should be acceptable.
There is always a cancellation clause. I don’t know what it is, and it isn’t unusual for contracts to be considered proprietary information (so we may never know.) I have trouble believing Boeing would sign anything as harsh as paying back 115% of all the money they’ve spent. I hope NASA wouldn’t sign anything that was just a minor fee and a slap on the wrist if Boeing cancels. But sometimes the government has to pay extra, as closing costs, when they decide to cancel. Even if it’s due to contractor underperformance. That was an issue when the Department of Defense’s failed experiment with fixed price contracts. It really boils down to who employs the best contract lawyers; Boeing or NASA?
I feel Boeing should drop out because they can not complete on cost and their technology is inferior. NASA should encourage them to throw in the towel.
I don’t know what the contract details are. Usually, the government (taxpayers) ends up screwed. However, given the extensive spread of Boeing all over military and space assets contracts, I think the US government could apply pressure to Boeing to stay in the contract and FIX IT. No?
Maybe. Sometimes, it has gone the other way. What if the Air Force says, “We really need those KC-46s. Don’t do anything that would make Boeing file for bankruptcy.” That’s the problem with having a small number of contractors.You can’t afford to hurt one for screwing up without also hurting yourself.
I see no way NASA allows Boeing to continue without repeating test #1. OFT-1 was indeed a Cluster Flight… Reasonable & prudent engineers would have performed EtoE testing at least for the very first flight. Now, they get to do it over again. Embarrassing.
Yes, and NASA shouldn’t give Boeing one single dime to do it, since Boeing dropped the ball on it. NASA shouldn’t reward Boeing cutting corners by giving them more money to fix it.
Yea… I guess we now know what a flight of Starliner costs to Boeing. $410 Million? (the amount the set aside to refly) =)
It looks that way, or at least a long delay before flying with astronauts. And the rumor mill has noticed the training schedule for the two astronauts on SpaceX DM-2. It looks like they are getting ready for a long stay on ISS as opposed to a quick visit. Nothing has been announced, but that is not a vote of confidence in Starliner flying astronauts any time soon.
At the risk of being called Capt. Obvious, =)
SpaceX is clear ahead in this race. I hope the rumors are true and they make the next flight a “work flight” vs a test flight for ISS. On the other hand, instead of trying to fight it, Boeing may realize it is easier (and much safer for their sake) to re-fly OFT 1 and eating the cost to make sure all the i’s and t’s are dotted and crossed. They cannot afford to keep screwing up. Time to turn the ship around, Bob.
Garrett Reisman, ex-astronaut, ex-SpaceX, confirmed this on Feb 23: https://twitter.com/astro_g…
Keith, please consider omitting the GIFs in future posts; they are distracting.
I like the Data Gif – it’s amusing and appropriate.
Given the show of emotion I am guessing it’s actually Lore not Data. ?
You’re right!
I’m not happy with the play speed, but that’s probably my browser not NASA Watch. As long is he doesn’t make the flashing light mistake (which some television and cartoon companies in Japan did) I’m fine with the animations.
I guess we aren’t paying them enough wasted tax dollars…
I saw the Sentinel article and (if true) I now see one of the principal reasons OFT-1 was not entirely successful. Note: it doesn’t explain the problems upstream, and a comprehensive list of everything that went wrong on that mission hasn’t been revealed (nor will it likely ever be).
Of greater interest is why a former member of the safety panel would go public. Is Boeing putting pressure on NASA to allow OFT-2 to proceed despite the outcome of OFT-1? Is NASA seriously considering it?
Sitting here reminiscing about the long hours spent poring over SAIL data, wondering WTF are these people doing now?
Congress still doesn’t care – Boeing is living off their legacy now. Do not be surprised if Boeing carves out extra funds in the upcoming budget to “speed up” Starliner efforts.
Does anyone think that Boeing was able to sneak Starliner onto the launch pad without that end-to-end integrated test completed and with NASA not knowing what was going on? If that test was omitted, NASA signed off on that. Those system integration tests are lengthy, thorough, take weeks to perform, and involve dozens of engineers and managers on both the spacecraft and on the launch vehicle.
NASA has signed off on a lack of end-to-end tests before. So have other space agencies. As you note, they are really difficult to do. So it isn’t unknown for people to agree that acceptable alternatives are fine. And sometimes that doesn’t end well. That was part of the problem with Ariane 501 and Mars Polar Lander, to mention a couple.
Good to know. Thanks.
I do! Long ago, in a location on this planet, a contractor was prepping a launch vehicle to send a large communication satellite into orbit. A previous launch of two smaller satellites had been successful, so it was assumed “all was well”! When two satellites were launched, the top one was wired to firing circuit “A” and the bottom satellite to firing circuit “B”. No problem. But when a single large satellite was to be launched, the firing circuit for the separation ordnance was supposed to be connected to “B”. But the contractor’s electrical troops specified “A”, and the “chicken checkers” looking at the drawings, didn’t catch it! I don’t recall there being any systems engineers in the loop. The contractor’s launch personnel were concerned and recommended a flight simulation check before ordnance was connected. But management said, “NO! It will cost too much money!” (A detector circuit could have been kludged up with parts from Radio Shack.)
So, when the flight took place, the satellite failed to separate from the upper stage, putting the $450M satellite in a useless orbit., One of the Space Shuttles went up and got it, but the insurance company paid off and it was sold to another outfit!
“He screwed up, and QC swore to it…at the direction of management!”
Let Boeing continue on the path it has taken with Starliner and 737MAX, and see who will be willing to fly on that thing!
Thanks for your input. Failure to make single electrical connections is one thing. Failure to do a complete end-to-end system integration test is something quite different. The first is a problem involving a few technicians on the launch pad. The latter involves many engineers and managers and takes weeks or months.
The point was that the techs at the Cape wanted to double-check continuity of the circuits…and management said, NO! I wonder how many of Boeing’s engineers recommended an end-to-end check and were told by management to “be a mushroom”, i.e., sit at your desk or computer, keep your head down, and…SHUT UP! There is never time nor money to do the right thing, Only time and money to do it over, the right way, hopefully before somebody gets killed! Or as the late Walt Williams said, “You don’t get medals for on-time failures!”
Let’s not forget what commercial crew (and cargo) are all about. NASA isn’t supposed to be looking over the service providers’ shoulders and monitoring their every step. Part of the concept is that it’s Boeing’s job to get it right not NASA’s. Now it’s Boeing’s job to fix the problem. I just hope NASA learned something from the Department of Defense attempts as firm fixed price contracts. Those contracts need some really serious penalty clauses, so the contractor can’t simply bail out. (Is there such a think as writing in a lead parachute?)
There’s really nothing to worry about in a firm-fixed-price contract that is “pay on delivery”. But ‘commercial’ crew has strayed very, very far away from that philosophy.
Should I be shocked (dismayed?) that they failed to learn this basic lesson from Hubble? There are no shortcuts and you are never as perfect as you think you are. https://ntrs.nasa.gov/archi…
Anyone want to take bets that Boeing will use political clout and ride roughshod over NASA to block a re-fly of the Starliner? In fact; don’t be surprised if it is suddenly announced that Boeing gets the big prize of flying the first U.S. crew from KSC again…
I think I’ll take that bet. Both congressmen and NASA senior managers know how bad it would look is an astronaut died in flight. I don’t think even Boeing has enough pull to twist arms that far. No matter how much lobbying they do, people aren’t going up on Starliner until Boeing has spent some serious time solving the evident problems. And NASA can’t wait for that, when they will shortly be down to just one astronaut on ISS.
I would be more than happy to be wrong, but we’ll see. The Lobbying Force is Strong with that one (Boeing).
Uh, apropos of nothing in particular, has anybody seen anything of a flying Tesla with a manikin crewmember lately? (I won’t say “dummy” because I think most of those are still here on Earth!)
I’ll take that bet. First, a full analysis of the flight is probably not complete. Second, Boeing is undergoing the rare pleasure of a full audit of the Starliner software as well as the change control processes; that’s going to take months to complete. Third, that company needs a success very badly, and their management knows it; right now risk aversion is probably endemic in the ranks.
I’ve always thought SpaceX was way ahead of Boeing in the Commercial Crew contract. Their experience with Commercial Cargo taught them lessons that Boeing is just now learning.
Ha, Ha. “the rare pleasure of a full audit…” Yeah, those tend to find much more than originally intended. I would not be surprised it the story gets worse…. but I would also be happy to be proven wrong on this one. I want to see that flight repeated, as soon as Boeing (& NASA) is ready.
I’d take that bet too. Boeing has lost NASA’s confidence. I believe they will not be left off the hook. Human spaceflight is just too critical at this point. The original press conference where they said, “if we had astronauts on board, they would have fix it” was later proven to be premature & not true. Get that software checked and validated, then fly OFT-1 again.
Some additional issues that come from all this.
1. NASA Independent Verification and Validation (IV&V) was useless and did not find any of the problems or raise a concern that end to end testing was not done. That’s why they exist, to find software errors and verify the system is being built the right way.
2. NASA S&MA did not find the problems despite the contractor being paid to find the issues.
No one looked at Boeing’s code or requirements/processes/test plan with any degree of competence (maybe Boeing wanted it that way.)
Boeing skipped on testing for money. This happens when accounts take over an engineering company. NASA should demand and Boeing needs to take the Flight Software ( code) to a third party (Not NASA S&MA or NASA IV&V) and at a minimum have static code analysis done. MDA has the ability to do it and do it quickly. No joke. This could be done in a few months. Without a static code analysis (and dynamic if they provide everything) no one should fly on CTS-100. Not even the crash test dummy.