This is not a NASA Website. You might learn something. It's YOUR space agency. Get involved. Take it back. Make it work - for YOU.
Exploration

Significant Command and Control System Problems for SLS and Orion

By Keith Cowing
NASA Watch
March 28, 2016
Filed under , , ,
Significant Command and Control System Problems for SLS and Orion

NASA OIG Audit of the Spaceport Command and Control System for SLS and Orion
“The SCCS development effort has significantly exceeded initial cost and schedule estimates. Compared to fiscal year 2012 projections, development costs have increased approximately 77 percent to $207.4 million and the release of a fully operational version has slipped by 14 months from July 2016 to September 2017. In addition, several planned capabilities have been deferred because of cost and timing pressures, including the ability to automatically detect the root cause of specific equipment and system failures. Without this information, it will be more difficult for controllers and engineers to quickly diagnose and resolve issues. Although NASA officials believe the SCCS will operate safely without these capabilities, they acknowledge the reduced capability could affect the ability to react to unexpected issues during launch operations and potentially impact the launch schedule for the combined SLS-Orion system.”
Keith’s note: Typical NASA double talk. They design a system to do a bunch of things. They claim that all of the program’s requirements are necessary for safety and reliability and worth the large cost. And oh yes, NASA can do it much better in-house rather than use existing commercial solutions since NASA’s requirements are one of a kind. Then the costs dramatically increase and implementation delays move to the right. Then the OIG steps in an points out the problems.
Then NASA says ‘Oh, [INSERT ANY PROGRAM NAME] will still work without all the stuff we wanted to do. But some things won’t work. But it is still safe to use it. But we need more money to fix the things that we don’t really need but want to have because maybe it is not totally safe to use it after all – but we’ll still use it without those functions because we have no alternative. And oh, by the way: stop bothering us: we know what we are doing.’ But wait – there’s more: NASA can save its own bacon by slipping SLS/Orion flights further to the right such that the SCCS now has more time to get things right since there’s no actual missions for it worry about.
The more things change …

NASA Watch founder, Explorers Club Fellow, ex-NASA, Away Teams, Journalist, Space & Astrobiology, Lapsed climber.

29 responses to “Significant Command and Control System Problems for SLS and Orion”

  1. Neil.Verea says:
    0
    0

    Taking planned designed capabilities out costs more than leaving them in.(they almost always make it back in with a Life cycle cost significantly greater then leaving them in). Its not intuitive and is a bit convoluted but their costs just sky rocketed (all pun intended)

  2. RocketScientist327 says:
    0
    0

    Never saw this coming… :eyeroll:

    • Robert van de Walle says:
      0
      0

      The SLS and Orion programs are functioning as intended. Money is being spent hand-over-fist in some congressional districts.

  3. Chris Winter says:
    0
    0

    In addition, several planned capabilities have been deferred because of cost and timing pressures, including the ability to automatically detect the root cause of specific equipment and system failures.

    Quote without comment:

    https://www.youtube.com/wat

    • Daniel Woodard says:
      0
      0

      The 1973 VW had a centralized sensor connection plug, as most cars do today. The “computer” was a device to print out the sensor readings. There was no system to find the,”root cause” .

      • Chris Winter says:
        0
        0

        OK, If in the VW example the low battery water level was not the “root cause”, how much additional expense would have been required to find the “root cause”?

        Granted that a rocket (e.g. SLS) is more complex than a Volkswagen, how much harder would it have been to build a system that would have isolated that “root cause?”

        • Daniel Woodard says:
          0
          0

          The low battery water level was not the root cause of a failure because no failure had occurred. It was simply an abnormal reading on the list of sensor readings. The best response to such a problem is to redesign the system to eliminate the problem at the source. Batteries were redesigned to incorporate catalysts to allow evolved hydrogen and oxygen to recombine. Battery water level sensors are no longer needed.

        • Jeff2Space says:
          0
          0

          A better example might be a temperature sensor in a turbopump. A a programmer, without much mechanical engineering experience, who is tasked with writing the code to identify the root cause
          would likely put in a print statement which says “turbopump temperature exceeds upper limit: root cause is turbopump failure”. But, is that truly a root cause? I’d argue no. Even if one assumes it’s a bearing failure, what caused it? Was it a manufacturing defect? Was it foreign object debris that came from the upstream plumbing? How could a hard coded piece of software ever identify a root cause for something like this?

          In the real world, the root cause would not be known until the pump, and upstream plumbing, is torn down and inspected.

  4. Brian Thorn says:
    0
    0

    To play devil’s advocate, ask the Delta III or Ariane 5 teams how well using an off-the-shelf solution worked the first time.

    • Jeff2Space says:
      0
      0

      True, but NASA quite often times doesn’t do a good job of separating requirements out into categories like “must” and “desired”. If this separation isn’t done properly during the requirements phase, during analysis and design requirements which ought to be delivered separately can become inextricably co-mingled. This drives up cost when something that should have been a “desired” from the beginning gets dropped from a milestone “at the last minute”. This can lead to hacks instead of properly designed and implemented code.

      • fcrary says:
        0
        0

        There is problem with listing “desired” rather than “required” capabilities. I’ve seen several managers who feel doing something which is just “desired” is a waste of time and money. If it’s necessary, you must do it; if it is not necessary, you should not waste time on it. This is the same mindset which, given a requirement for 1-m resolution of a planet’s surface, would say 1.05-m resolution is a failure (does not satisfy requirement) but 0.95-m resolution is wasting resources on exceeding requirements. I’m afraid people like that are out there, and it pushes others to call a “desire” a “requirement.”

        • Jeff2Space says:
          0
          0

          In a perfect world, engineers should not “wasting” time on “desired” items until the “required” items are delivered. In the real world, if someone does not have any “required” work to do, spending time on a “desired” item is preferable to surfing the Internet, or commenting on NASA Watch. 😉

          But my bigger point is that requirements should be kept separate so that projects can be deemed “complete” with some minimal set of “required” functionality. For SLS that would be the ability to fly a mission safely (not kill a crew) with a reasonable chance of mission success.

          • fcrary says:
            0
            0

            For planetary (unmanned) missions, that’s usually called to “performance floor”. Some requirements are attached to the baseline mission, and can be dropped if the project is descoped. A project which can’t meet performance floor requirements is supposed to get cancelled.

        • Daniel Woodard says:
          0
          0

          The division into requirements and desired capabilities is artificial and not very helpful. Every capability costs something, and the engineer-in-chief (the title of the director of NACA) had to consider them all and decide on the best way to control cost and maximize value.

    • SpaceMunkie says:
      0
      0

      or SpaceX

  5. Daniel Woodard says:
    0
    0

    An aerospace vehicle is an integrated whole, and the great ones were always the creation of a single visionary, someone like John Boyd (F-16), Ed Heinemann (A-4), Kelly Johnson (C-130, F-104, U-2, F-117, SR-71), or Werner von Braun (Saturn V). If requirements are considered in the abstract by a committee with the assumption that anything that is “required” by management can be incorporated into the design, weight, cost, performance and reliability will suffer.

    In this case the idea that the system is required to automatically identify the “root cause” of a failure boggles the mind. Design analysis only considers failure modes that can be anticipated, and most catastrophic launch vehicle failures are the result of failure modes the were unanticipated in the design process. It is more effective to keep the system simple, test it until it fails, and change the design to eliminate failures at the source.

    • SpaceMunkie says:
      0
      0

      but all those great men were very competent ENGINEERS put into management positions, rather than managers put into engineering positions

      • Daniel Woodard says:
        0
        0

        The title of the director of NASA is “NASA Administrator”
        The title of the director of NASA’s predecessor organization, the NACA, was “Engineer in Chief”
        http://history.nasa.gov/SP-

        • Michael Spencer says:
          0
          0

          Good point. But the mindset changed. Perhaps this happened when STS was declared operational.

          • Jeff2Space says:
            0
            0

            And as we all know, STS may have been declared operational, but the reality was that there were many problems with the system both in terms of engineering (e.g. differential steering via braking wasn’t cutting it, o-rings in the SRBs were getting blow-by, and etc.) and in terms of program management (lack of spares meant that cannibalization of parts from other orbiters to meet “schedules” was becoming routine). And as we all know, safety took a back seat to ramping up the flight rate. All of these issues were swept under the rug until the Challenger disaster forced a good look at the entire program.

          • Daniel Woodard says:
            0
            0

            I believe the mindset changed when we began the race to the moon. Prior to that our goal was to provide practical benefits for America.

  6. Neal Aldin says:
    0
    0

    If you take a look at the kinds of things Orion started out to be able to do-7 person crew, land landings, re-usability…..really it meets no requirements. The requirements were never vetted, never appropriately reviewed or approved; they never had a proper preliminary design to vet their requirements against. So the idea that any of its systems fails to function as intended does not come as a surprise.

    Next program-and I am speaking to the Griffins, Boldens and Gerstenmaiers, and others who held similar positions over the last decade, find some people who had some proven experience and success and competence and put them in charge. I know most of the people who ran this program and there was not even a single one who had any pertinent prior experience. One guy had designed some subsystems on the F-15 and he was the first one to raise his voice in protest and the first to be canned.

    In fact, as Daniel Woodard said, the most successful air and space projects were the vision of A visionary. NASA’s visionaries today are few and far between and they are not developed and not recognized. Instead they keep putting astronauts and flight directors rs in charge and those folks are great at following directions, but their vision and design skills are sorely lacking. In fact those attributes are exactly the kinds of attributes that are frowned upon in those positions. Instead, Constellation, Orion, and its boosters were the result of a debating society in which no one had a clue what they were even trying to do.

  7. Rich_Palermo says:
    0
    0

    NASA is very good at waiving itself and selected suppliers.

  8. numbers_guy101 says:
    0
    0

    I’m glad the IG walked through a little history here. That’s a nicely done report. They could have added – that the 80’s effort at a replacement, “CORE” on pp 4, which resulted in nothing, would be a near $750M loss in today’s dollars. Probably close to a billion dollar waste by the time related costs are included.

    I would add one observation for the IG – from my own experience, as I was there for the start of the 90’s CLCS effort. The IG reviewers should note that you can’t IMPROVE on a system when improving the system is not really, truly what the project is about. This was my experience then with CLCS. “Improvement” is a word used to SELL the project or to keep it’s funding flowing. “Improvement” is not for defining what the project does or what it’s about. You can’t expect results that are not real goals, when “improvement” is just a PowerPoint and management buzzword.

    This goes for KSC ground systems like this as well as things like SLS. In practice no one builds “improvement” into the actions and measures and tangible matters of these cost-plus mindset projects. Back in that CLCS effort I was stupid enough to ask how many people LESS we would need to launch the Shuttle? How much less would we spend a year versus current? What was our goal for how many LESS? The looks I got. It was that – “I must not understand what the project is about, you idiot” – kind of look.

    And here we are. Hey maybe I’ll get asked to help for that item – “commission an independent assessment”. I’d have fun just seeing people’s faces when I walk in the room…I think they know where I stand on this.

    • Daniel Woodard says:
      0
      0

      I remember CLICS. It seemed fairly simple; a network of PCs and servers to control the countdown, much as SpaceX uses. I don’t know why it became so expensive, but if we could understand why this happened, i might have lessons for other systems.

      That’s why I feel the IG may be missing the point. They are suggesting that the system is not meeting its requirements. They do not appear to be questioning whether the requirements are correct. Or am I missing something here?

  9. Golfball1 says:
    0
    0

    After a 4 year hiatus from KSC, I worked on the SLS software for 6 weeks. That’s all I could handle. The same mindset that shackles creativity and the extreme lack of accountability for management is as strong as ever. The end of the Shuttle should have cleared this all out. But these managers and the archaic approach to software is still there and growing. I will be very surprised if SLS ever becomes a flight program. My guess is that once Commercial Crew gets flying, SLS will be canceled and the multiple launch scenario will be revisited for future missions.

  10. SpaceMunkie says:
    0
    0

    all these problems are caused by uneducated incompetent managers that have the “foresight to stick their noses into the design” and tell the engineers how to do their job
    the “why can’t we do it the way shuttle did it” and “that is the way we did it on shuttle” resurfaces yet again

  11. SJG_2010 says:
    0
    0

    Funny, but I will BET that the Open Source COSMOS system could be used to launch a rocket (Yes even a very complex one). http://cosmosrb.com/ It is already used for spacecraft test and even some on-orbit operations.

  12. Tritium3H says:
    0
    0

    Maybe they should just hire the programmers from the Kerbal Space Program spaceflight simulator.