SLS Flight Software Safety Issues at MSFC (Update)

Keith's 29 September update: Sources report that a substantial portion of the contractor staff working for the SLS safety contractor at NASA MSFC QD34 want out and are asking for reassignment to other programs. Many are openly looking for new jobs elsewhere. The prime contractor has been told by NASA MSFC management that if anyone leaves SLS safety support without permission or by other than NASA-directed termination that the incumbent contractor risks not receiving consideration during the contract re-competition next year. SLS safety risks under development are being deleted. People are scared to come forward with issues. SLS management was at Michoud and Stennis for an AOA yesterday and today. This was reportedly a topic for discussion.

Keith's 26 September update: Shortly after this original posting on 9 September Andrew (Andy) Gamble was summoned to MSFC Center Director Todd May's office to talk about QD34 issues. During subsequent closed door meetings MSFC management decided that they did not have any software problems - when in fact they do.

A few days later Steven Pearson, Deputy Director of NASA MSFC's Safety & Mission Assurance Directorate, suddenly announced his retirement after 37.5 years at NASA. MSFC SM&A hired some outside experts in the area of software safety and V&V and asked them to determine if the MSFC QD34 contractor or NASA QD34 was right with regard to the issues under dispute. The outside experts completed their review and agreed 100% with the positions taken by the contractor.

Meanwhile, contractor employees working for QD34 who have surfaced the issues I reported earlier have suddenly found their funding yanked. Moreover, employees who are leaving or thinking of leaving as a result of raising concerns - are being specifically blackballed - by name - by MSFC management. Potential employers are being told by NASA MSFC that they risk wining new contracts - or losing existing contracts - if they hire these individuals.

According to an internal MSFC memo previously cited "Andy and George stated that Software V&V is really just a function of Software Quality Engineering. (Really, our QD34 customers seemed not to have a clue of what comprises Software V&V, or even Software Assurance or Software Quality for that matter.) False - NASA/MSFC procedural requirements, standards, guidebooks/handbooks, etc. clearly speak to the contrary. Just because Software Assurance isn't organized on SLSP (or at MSFC) with people explicitly assigned to a Software V&V group doesn't mean we don't do Software V&V tasks (separately from Software Quality tasks)."

Most troubling of all, the internal assumption at MSFC is that the first SLS flight will have a built-in risk of failure of around 8%. This risk is being "baked in" to the design of SLS in part due to decisions being made at MSFC about software and avionics - decisions that are being made so as to not surface troublesome issues that no one wants to deal with. One can imagine that safety folks at MSFC are nervous.

This is no way to build a rocket folks.

Keith's 9 September note: NASA's SLS program has been experiencing budgetary and scheduling issues for years as noted in multiple GAO and OIG reports. The program has also had problems with technical issus such as software and avionics. Multiple sources report that one of the places experiencing significant problems is QD34 at NASA MSFC - where significant SLS flight software safety issues are mounting - issues that no one else is hearing about. The Branch Chief for QD34 is Andrew Gamble. NASA MSFC management - and perhaps the NASA OIG might - want to pay that organization a visit.

Keith's 9 September update: According to an internal memo there is a "lack of understanding by QD34 of the intent of NPR 7150.2B (A or B) [NASA Software Engineering Requirements] and of CMMI [Capability Maturity Model Integration] when it comes to a higher level of process, procedure, etc. rigor expected for Class A software than for Class C" and that actions by NASA MSFC QD34 management represent "a direct attempt by our QD34 customers to intentionally minimalize the differences to avoid making matters complicated and having to make any corresponding changes for SLSP at this time."

SLS Avionics: The Brain Without a Body, NASA

"Ultimately the avionics boxes and software have to work perfectly. But how can you be sure without putting it on the world's largest rocket and seeing how it works? That's the focus of the Integrated Avionics Test Facility - or IATF - at NASA's Marshall Space Flight Center."

Earlier SLS/Orion posts

  • submit to reddit


Terran Orbital - Extraordinary Capabilities to Support Any Mission
Kepler Communications - Aether - Unbound Connectivity to your on-orbit assets

Join our mailing list

Monthly Archives

About this Entry

This page contains a single entry by Keith Cowing published on September 9, 2016 10:08 AM.

OSIRIS-REx Has Left Earth For A While was the previous entry in this blog.

That Alternate Reality Without SLS is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.