This is not a NASA Website. You might learn something. It's YOUR space agency. Get involved. Take it back. Make it work - for YOU.
Commercialization

ASAP: Boeing Starliner Software Issue Potentially "Catastrophic"

By Keith Cowing
NASA Watch
February 6, 2020
Filed under

https://media2.spaceref.com/news/2020/data.gifNASA, Boeing to Provide Update on Starliner Orbital Flight Test Reviews
“NASA and Boeing will host a media teleconference at 3:30 p.m. EST Friday, Feb. 7, to discuss the status of the joint independent review team investigation into the primary issues detected during the company’s uncrewed Orbital Flight Test in December as part of NASA’s Commercial Crew Program.”
NASA Safety Panel: Second Starliner OFT Software Error COuld Have Been :Catastrophic”, Space Policy Online
“In an emailed statement to SpacePolicyOnline.com this evening, Boeing said it accepts and appreciates the recommendations of the IRT as well as suggestions from ASAP: “They are invaluable to the Commercial Crew Program and we will work with NASA to comprehensively apply their recommendations.”
Starliner faced “catastrophic” failure before software bug found, Ars Technica
“At Thursday’s meeting, Hill revealed the second issue related to software and thruster performance publicly for the first time. However, as part of reporting on a story about Starliner software and thruster issues three weeks ago, a source told Ars about this particular problem. According to the source, Boeing patched a software code error just two hours before the vehicle reentered Earth’s atmosphere. Had the error not been caught, the source said, proper thrusters would not open during the reentry process, and the vehicle would have been lost.”
Keith’s note: And of course there are all of the SLS software issues that have plagued the Boeing and NASA MSFC folks:
SLS Upper Stage Changes While Software Problems Linger, earlier post
SLS Software Problems Continue at MSFC, earlier post
This Is How NASA Covers Up SLS Software Safety Issues (Update), earlier post
MSFC To Safety Contractor: Just Ignore Those SLS Software Issues, earlier post
SLS Flight Software Safety Issues Continue at MSFC, earlier post
SLS Flight Software Safety Issues at MSFC (Update), earlier post
Previous SLS postings
But wait, there’s more in other parts of Boeing:
Boeing Finds New Software Problem With Scandal-Plagued 737 Max Plane, Gizmodo
“During flight testing of the 737 MAX’s updated software, an indicator light associated with the stabilizer trim system illuminated in the flight deck,” a Boeing spokesperson told Gizmodo via email. “We determined that the illumination of this light was caused by differences in input data between the flight control computers (FCC). This is a result of the FCC cross compare redundancy software update issued in June 2019.”

NASA Watch founder, Explorers Club Fellow, ex-NASA, Away Teams, Journalist, Space & Astrobiology, Lapsed climber.

42 responses to “ASAP: Boeing Starliner Software Issue Potentially "Catastrophic"”

  1. Bill Housley says:
    0
    0

    This is very discouraging. It pains me to say this, I like Boeing, but it points to the possibility of a systematic infrastructure problem rather than just isolated mistakes. This should prompt a review of ALL of the code and a fresh uncrewed test flight before ANY humans fly on Starliner.

    I wouldn’t mind NASA paying them a little bit more money (i.e. buying another flight) to make this happen.

    • ed2291 says:
      0
      0


      I wouldn’t mind NASA paying them a little bit more money (i.e. buying another flight) to make this happen.” I would mind. NASA has set up two standards. Boeing has the lesser standard while being paid much more. Space X has the greater standard while being paid much less.

      • fcrary says:
        0
        0

        With one exception NASA’s standards have been consistent. They have expected the companies to do what they proposed to do. SpaceX proposed to do more tests at a lower cost. Boeing proposed to do fewer tests using a process NASA was familiar and comfortable with but at a higher cost. Since the selections, that’s what NASA has gotten. In hindsight that probably means the AO and the selection process was flawed. But NASA has gotten what it paid for. An expensive lemon and a good, less expensive spacecraft which might not have worked out as well as it has.

        • ed2291 says:
          0
          0

          What about the bonus Boeing got that was not contracted for? What about the criticism of Space X alone while Boeing skated? The bigger question is if Space X keeps its obligations and Boeing does not, why should Boeing get over 50% of the business going forward?

          • fcrary says:
            0
            0

            Ok. Two or three times. The extra $280 million was definitely not consistent treatment. I don’t think the criticism was about favoring Boeing; it was about not liking novel development approaches. But I could see how it sounds like favoritism.

            In terms of the number of launches, I don’t think the contracts say how many Boeing will get. Both companies will get at least six. Assuming ISS operates until 2030, and there are two rotations per year, that makes something like 20 flights up for grabs. I’m not sure how many the Russians can manage, or how many they want to fly. I doubt it’s one per year.

            NASA has made a point of awarding flights based on the vehicles’ different capabilities and NASA’s needs for each flight. I can easily imagine Starliner getting the contractual minimum of six and Dragon 2 getting eight or nine.

      • Paul Gillett says:
        0
        0

        Boeing has already announced that the company booked a $410 million charge last quarter in case NASA requires another unmanned flight.

        https://www.cnbc.com/2020/0

        Given the latest revelations, does anyone really believe that the question of a second test mission would still be undecided if it had been a SpaceX flight failure?

      • Bill Housley says:
        0
        0

        The essential point of Commercial Crew is competition. It is a valuable and I think critical part of the program for triggering a space economy. I think that’s worth another couple hundred million to get it right.

        I’ve witnessed dinosaurs reborn. It can happen to Boeing. Yes, I’ve criticized them, both here and elsewhere, and I think they deserve that criticism… especially over this newly revealed problem. But in the end I want a reborn Boeing Space, flying the Starliner. I do not want them to just take their ball and go home. That would be the larger waste of both money and opportunity.

  2. strangeluck says:
    0
    0

    Link to the source?

  3. ThomasLMatula says:
    0
    0

    It seems everything Boeing does these days, the B737 Max, Comsats Starliner, KC-46, etc. is suffering from a combination of poor design or lack of quality control. The last time I recall a firm melting down this way it was PennCentral, and we know how that turned out. I hope the new CEO is able to get it under controlled and fixed.

    • fcrary says:
      0
      0

      To be fair, Spaceway-1 was launched in 2005, and had a 12 year design lifetime. It’s in trouble because the warranty ran out three years ago, which isn’t exactly shocking. And the Boeing 702 is a fine line of communications satellites, and a design they inherited from Hughes when they bought up their Satellite Systems division. That’s just bad luck, especially coming on top of so many big mistakes which Boeing can fairly be blamed for.

      • ThomasLMatula says:
        0
        0

        Actually I was referring to Intelsat-29e which was only 3 years old when it failed.

        https://spacenews.com/intel

        Intelsat-29e declared a total loss
        by Caleb Henry — April 18, 2019

        “Intelsat-29e suffered a fuel leak April 7 after just three years in geostationary orbit. Most geostationary communications satellites last 15 years if not longer.”

        • fcrary says:
          0
          0

          Yes, but… The 702 really is a good spacecraft with a long track record of reliability. It’s probably up to the 90% success rate NASA gets from major, robotic planetary missions. I don’t think that failure is a huge black mark. Something did go wrong in a very explosive way. But we should expect that to happen on an occasional basis. I wouldn’t put that in the same class as three serious issues on a test flight which was supposed to go perfectly or two crashes within months due to bad avionics and a lack of pilot training. Boeing is definitely in the middle of some serious problems, but I’d call there record for communications satellites par for the course. Things like the Starliner and 737 MAX issues are much more concerning to me.

          • ThomasLMatula says:
            0
            0

            True, but also remember the B737 overall has a good record, it’s the latest version that ruined it. I suspect given the low production rate of comsats each is highly customized and so there is the possibility that the problems that are impacting other areas of Boeing may be creeping in.

    • mfwright says:
      0
      0

      It seems Boeing is really taking a beating on the public awareness area, here’s another doozy:
      “What’s wrong with Boeing” by Mentour Pilot,
      https://www.youtube.com/wat

    • John Thomas says:
      0
      0

      Not everything is as it seems. The news media harps on a few major issues but does not put things in context. Boeing makes a lot of different aircraft and spacecraft. It’s always best to examine things analytically.

      • fcrary says:
        0
        0

        Analytically three usually isn’t a statistical significant number. But three serious bugs on the first flight of Starliner is quite serious. The requirement is for less than a one in 270 risk of a fatal accident on flights carrying astronauts. Analytically and statistically, Boeing’s flight software is a long way from demonstrating that it satisfies that requirement.

  4. Jeff2Space says:
    0
    0

    Time to get all up in Boeing’s business and investigate its culture like NASA did to SpaceX after the infamous Elon Musk weed smoking incident.

    • fcrary says:
      0
      0

      I’m not sure if that would help. That sort of investigation by an agency like NASA has a lot to do with whether or not the company has rules and follows them. These software issues don’t feel like things that would help. Do you have a rule about doing a full code walk-through? Do you follow it? Did you write a software requirements document? Was it reviewed and if so by how many reviewers? Etc.

      The answers can all be yes, but that doesn’t necessarily tell you much. It can leave out things like who was at the code walk-though, software engineers with similar experience and in a similar programing language or likely users who can’t code at all. I’ve seen reviews for flight hardware which were very carefully choreographed. Just enough information to make the review panel think they were being told enough, without actually letting them look under the hood. But the rules say the review must be held, and they did hold the review.

    • John Thomas says:
      0
      0

      Boeing is a much bigger company than SpaceX with 150,000 employees compared to SpaceX 7000 employees and makes many different things.

    • richard_schumacher says:
      0
      0

      Not NASA; Congress. Boeing has been messing up for many years now on the KC-46 tanker and 737MAX while suckling at the public teat.

  5. TheRadicalModerate says:
    0
    0

    “@Boeing needs to get its software shop in order.”

    Any company that set its software development operation up in the 80’s or 90’s is now the middle of a crisis. The linear “spec, then review, then design, then review, then develop, then test, then accept” model that both NASA and the military are so fond of simply isn’t how modern software is developed any more. Engineers aren’t trained on it because it’s vastly less efficient–and more error-prone–than iterative development is.

    At the same time, converting from one model to another is so painful that committing to doing so is a bet-the-company decision, especially when NASA and USAF contracting and acceptance are still so firmly rooted in the old model.

    This is the dominant reason why SpaceX and other New Space companies, who set themselves up as iterative design shops from the git-go, are eating Old Space’s lunch. That’s bad enough, but if we’re not careful, it’ll also be the reason why China winds up eating the US’s national security lunch as well.

    This is a major national crisis and likely requires a lot of defense and public policy support to resolve. Much though I’d enjoy watching Boeing Defense and Space shrivel up and die, it’s in the national interest to give them and the other incumbents the (urp!… ulp!…) financial incentives to get dragged kicking and screaming into the 21st century.

    The same goes for the DoD and NASA.

    • fcrary says:
      0
      0

      True, but for NASA and NASA contractors, it’s even worse. The iterative process does mean lots of early bugs and failures. The companies involved have to accept that. Actually, the customers have to accept that. But major NASA missions are few and far between, and anything involving the lives of astronauts gets enormous press when things go wrong. This might sound harsh, but if Tesla’s autodrive software caused a fatal accident, it would make the news but it wouldn’t stay on the front page for more than a few days. People get killed in car accidents all the time. If SpaceX or Boeing flight software kills an astronaut, that’s going to be in the news and in congressional committee grillings for months. So there is a lot of pressure to get it right from the start and not have the early bugs and failures the iterative approach involves.

      • Sam S says:
        0
        0

        The trick is to fail lots of times before you put people on it. That means test firings of prototypes, with acceptance that some of them will go boom and burn tens of millions of dollars each, and being able to look the Senate Committee in the eye and say “we are blowing expensive things up now so we don’t kill people later.”

        As far as the one-off programs like Juno, Curiosity, etc., maybe we can start approaching something like a “standard chassis” for rovers and orbiters with standardized sensor interfaces, etc. That could allow development to focus on the new instrumentation. It may be a little early for that kind of approach though, since e.g. Jupiter requires more radiation-tolerance, Saturn requires RTG power, etc.

        • fcrary says:
          0
          0

          For human spaceflight, yes, you could do lots of tests (and failures) before the first flight with an astronaut. Commercial Crew isn’t doing that, and neither is Orion/SLS. Those launches and capsules are _expensive._ To be honest, maybe we shouldn’t complain about Boeing’s buggy software on Starliner. Finding bugs in test flights is what this iterative approach is about. A better criticism might be that they didn’t do that on purpose. They thought or claimed they had a process that would get it right from the start. And that their test program has astronauts onboard the second flight.

          Cost and flight opportunities are issues for the robotic missions. A standardized bus is going to be suboptimal for any particular mission. It might be good enough, but when these are once-in-a-career opportunities, getting people to settle on “good enough” instead of a custom design is hard. The same goes for the instruments. These days, at least the hardware interfaces have converged to a small number of standards. The data comes out of the instruments in CCSDS packets. But the subsystems have custom software, and that means getting them to talk to each other over a standardized interface is still exactly standard.

    • William Bormann says:
      0
      0

      Remember Feynman’s observations about software development and validation for the space shuttle? And what management thought about it? Does this sound like Boeing?

      From Appendix F of the Rogers Commision Report:

      “To be sure, there have been recent suggestions by management to curtail such elaborate and expensive tests as being unnecessary at this late date in Shuttle history. This must be resisted for it does not appreciate the mutual subtle influences, and sources of error generated by even small changes of one part of a program on another.”

    • gearbox123 says:
      0
      0

      People who insist that “Agile will solve everything!” are no different than people who said “Databases will solve everything!” 50 years ago.

      The underlying problems are lack of attention to detail, featherbedding, and job-guarding by project managers.

      • TheRadicalModerate says:
        0
        0

        I don’t think I said “Agile” (although I was thinking it). Iterative design isn’t a software module or a design pattern–it’s a methodology. And as far as I’m concerned, it’s a superior methodology for almost any kind of problem.

        Also: from the number of MVC design patterns floating around out there, to say nothing of the number of frameworks where MVC is baked in, to a first approximation, databases did solve everything. At the very least, formal data models solved everything.

  6. DJE51 says:
    0
    0

    NASA’s ASAP (Aerospace Safety Advisory Panel) seems to be recommending a review of how Boeing approves things (at least as I understand their recommendation). Since the approval process is being questioned, I would think that the approvals that have been granted should automatically be questioned as well. And given the problems with first, the pad abort test (parachute attachment issue), and then the uncrewed test mission (Orbital Flight Test, or OFT), and now this revelation, I would think that NASA needs to be prudent, not get sucked into “go fever”, take a step back and demand that Boeing go back a few steps and start a new track of approvals. Meanwhile, the whole reason that NASA incorporated two commercial crew providers was this exact reason, to have one ready if the other one got into trouble. So, it would validate NASA’s two provider strategy, and it seems that SpaceX is ready with all of its test missions completed satisfactorily.

  7. Winner says:
    0
    0

    SpaceX ought to be the subcontractor for Boeing’s Starliner software, since it appears SpaceX knows how to do it properly.

    • fcrary says:
      0
      0

      As bad as things look now, that would be worse. Real-time software is very dependent on how the hardware it controls works. The software developers need to know all about the hardware. Subcontracting that means, at worst, proprietary information the software people can’t see, and at best the miscommunications you get when the hardware and software people are a thousand or so kilometers apart. And, when it doesn’t work, you the hardware and software people pointing fingers at each other and saying, “It’s not my fault; it must be his part that didn’t work.” I’ve seen the results when the institutions involved are on friendly terms and collaborating on a scientific instrument. It isn’t pretty. I don’t want to see what would happen in the two were competitors for big contracts.

    • SheBlindedMeWithScience says:
      0
      0

      Bite your tongue! Boeing is awful. SpaceX should have nothing to do with Boeing!

  8. Jack says:
    0
    0

    Sounds like the managers for software development at Boeing are of the type that think if your not constantly tickling the keyboard you are not working and that testing is not as important ans tickling the keyboard.

  9. Johnhouboltsmyspiritanimal says:
    0
    0

    Keith didn’t you have a bunch of articles about contractors reporting all sorts of Boeing SLS software shenanigans that management was glossing over. What are the chances the green run flushes the issues out? Or does the Artemis I test flight end in a spectacular fireball?

  10. Richard H. Shores says:
    0
    0

    The Friday media teleconference should be interesting to see if ASAP’s recommendations for more detailed work will be implemented.

  11. spacecadette says:
    0
    0

    From what I’ve read (mostly shortly after the mission), the problem was that the spacecraft determined when it was at the ISS by consulting a timer that had been initialized with the wrong time. If this is true, it’s astonishing that the software decided that it was in the vicinity of ISS simply by checking the time since launch. “Well, It’s been X long since launch, so I must be at the ISS now.” It’s standard practice for software to check a set of “preconditions” which must be satisfied before any critical actions are taken. If it’s true that such a critical event was based only on what time it is, there’s a serious problem with how the software was specified and written, warranting a thorough review of all of the flight software. That will take time and will require serious involvement and oversight of the process by NASA.

  12. Bad Horse says:
    0
    0

    When you consider how basic the software errors are and that NASA’s Independent Verification and Validation facility and NASA S&MA contractors reviewed the code and testing, it becomes clear someone did not want errors to be found and worse everyone went along. This is fundamental cultural problem. If your checkers don’t care, what’s the point of having them on contract. NASA should require Boeing to take the code to an independent non NASA IV&V organization (like MDA, Army aviation in Huntsville or US Navy). You have to deft, dumb and blind to fly with that software. Or just not care.

  13. Dewey Vanderhoff says:
    0
    0

    Lest we not ignore Lockheed Martin in this discourse…

  14. Michael Spencer says:
    0
    0

    Two very high profile cases, for sure.

    But folks this looks like jumping to a huge conclusion with n=2. Boeing is a huge company, after all.

    Let the dust settle.