NASA Says Starliner's Problems Were A "High Visibility Close Call"
Keith’s note: NASA and Boeing held a media briefing today about the report of the NASA Internal Review Team (IRT) report on the various problems with the recent Starliner Orbital Flight Test (OFT). In summary the IRT found 61 things – recommendations – action items – problems – call them what you will – that need to be attended to by Boeing. We really do not know what they are since NASA and Boeing have not released them. But maybe they will.
According to NASA HEOMD AA Doug Loverro this was a close all with the possibility that the Starliner could have been lost at the beginning of the mission or at the end of its mission. Since this is a Boeing project with significant NASA insight Loverro has started the process with the NASA Safety office to set up an organizational root assessment of all Boeing and NASA actions leading up to this mission.
When Jim Chilton, senior vice president at Boeing Space and Launch initially spoke he was, in essence, saying that he wanted to thank NASA IRT for helping Boeing to find these 61 issues that Boeing was unable to find prior to launch – despite the biliions spent on Starliner – and despite all the help from NASA.
Doug Loverro said that he was designating this whole Starliner thing as a “High Visibility Close Call” (HVCC) which is NASAese for setting up an internal NASA process that includes all involved plus NASA Safety to find out what went wrong. Loverro noted that Boeing had “graciously” agreed to support this team. Uh huh. Nice of them to be gracious about it.
I asked a question about these 61 technical issues and Chilton started to get into semantics as to whether they were “61 technical problems” since many of these things mapped against the same problems. And then NASA hung up on me before I could hear the rest of my question. It took a while to be able to reconnect to the telecon. Maybe its the snarky questions I ask. Then again Jeff Foust from Space News got thrown out of the question queue. Houston we have a problem …
Listening to Doug Loverro talk he got into many fundamental aspects of how to manage a large aerospace program that speaks to experience gained from a 40 year career doing just that. Although he was trying to be positive about this it is clear that he is aware of pervasive Boeing/ NASA Starliner problems and that some structured adult supervision is required.
Everyone on the call wanted to know if there would be another OFT flight without a crew or with a crew. Loverro explained that the initial requirement for crew transport was to show NASA that the vehicle could safely deliver a crew to the ISS. Boeing opted for an actual docking to prove this requirement and NASA wrote it into the contract. Whether an un-crewed OFT re-flight is needed to do this or whether a crew can fly next time and make up for missed requirements is still TBD – and NASA was not showing its cards on this. Chilton later said that Boeing would re-fly the OFT if need be, but we do not know who’d pay for this extra mission.
Between my first question and my re-asking of that question an hour later I inquired of Boeing that since Boeing had all of these undiscovered problems prior to launch – of what they thought was a perfectly good spacecraft = one would assume that a more complex Boeing space vehicle such as the SLS would now require even more time to double check. Chilton said that there was output from the Starliner IRT effort and that it was being sent around their company for everyone to analyze to see if it applied to their programs. But other than that he did not get into specifics.
Loverro added that the pacing items on SLS were not software but instead were the green test (engine firing) and that other than a ground system issue at Stennis, no software needed to be finished. He also noted that the software team at IV&V was looking to all of this as well.
Of course Boeing and NASA originally thought that the Starliner’s software was good to go – so its a little curious that no one is overly concerned that there are more undiscovered things lurking in otherwise certified SLS software given all of the software issues the Boeing SLS people have experienced at MSFC. And again, SLS is a much larger, energetic, and complex spacecraft than Starliner – one upon which a crewed vehicle will eventually fly.
As to whether the way that NASA Has bought commercial services is the issue Loverro commented that he has had great successes with fixed cost and cost-plus contracts as well as failures from both types of contracts. So the contracting mechanism is not the issue.. Instead Loverro thinks that this is a managerial issue – hence the convening of the follow-on internal assessment of how NASA and Boeing did what they did on the Starliner flight. No timeline was given so one would assume that no Starliners are going to fly until this effort is completed. So .. barring any unforeseen problems SpaceX looks to be poised to win the flag that awaits them on board the ISS.
When asked if any more Soyuz seats are being bought Loverro said that they are talking to Russia and the plan is to buy one more seat. The telecon closed with Loverro saying the obligatory “we won’t fly unless its safe … stuff” and the operator ended the teleconference. I was busy typing and did not hang up right away. Other reporters did hang up. Then the NASA guy came back and said that Jim Chilton had some closing thoughts and he had an important one – that if NASA wants Boeing to re-fly OFT then they will. Nice of NASA to tell reporters that the telecon was over before Chilton had a chance to say this.
If I sound a little impatient with NASA’s teleconferencing system – I am. I did live video webcasts from Everest Base Camp for a month in 2009 using gear I carried on my back with far fewer technical problems than NASA has with a simple dial-in system. Seriously NASA, fix it. And get some new on-hold music too while you are at it.
“….that if NASA wants Boeing to re-fly OFT then they will.”
And if they do will Boeing pay for it?
No one wanted to talk about who’d pay for a OFT reflight.
Keith – one way or another the taxpayers will pay for it.
How long can Boeing hide their incompetence and corruption? Someday they may find an administration and NASA which is not so sympathetic.
What NASA and the administration can do is limited by federal law. What we need is for Congress to add some clear language to the Federal Acquisition Regulation. Something about tracking a company’s performance on past contracts and use that in evaluating their proposals and bids. I’m not holding my breath until Congress does that.
Past Performance is normally part of the evaluation process. https://www.acquisition.gov…
Unfortunately, that section is on tracking performance. At the beginning, it does say past performance, “is relevant information” for future selections, but that’s about it.
Worse, if you scroll down to the table on what qualifies for what rating, the poor ratings are defined in terms of “contractual requirements” and formally reported “contractual deficiencies.” Depending on the language of the contract, that leaves out a whole lot of things which most people would consider “poor performance.”
Delivering something late and missing a deadline might qualify. But contacting the appropriate government official in advance, asking for an extension, and having him change the deadline would not qualify as poor performance. Delivering a product which did not satisfy a requirement might not be poor performance of the appropriate person signed a waiver. Going over budget on a cost-plus contract isn’t poor performance if the extra costs are approved. On the other hand, failing to file all the accounting paperwork on schedule could easily be poor performance.
If a company’s lawyers and accountants get all the details right, the company can do things most people would consider poor performance, without getting any official black marks for it. Companies with extensive experience with government contracts have lots of well-paid lawyers and accountants for exactly this reason.
FWIW, the GAO did look into the CCtCap award and found the award “recognized Boeing’s higher price but also considered Boeing’s proposal to be the strongest of all three proposals in terms of technical approach, management approach and past performance.” As you rightly point out though, “past performance” is subject to perception and may be obscured by a carefully constructed legal snowstorm.
When it comes to the poor teleconferencing, look at the bright side of COVID-19 (even if it’s a tiny one.) Today, NASA is holding a “telework day” to “test [their] capabilities, resources, and preparedness for large-scale teleworking.” With luck, that will make a whole lot of people realize how dysfunctional those capabilities are. With more luck, some of those people will have the authority to get it fixed.
Yea this is prudent. Maybe it takes a pandemic for NASA to get the whole telecon thing down.
That is, unfortunately, all to common with funding for infrastructure. And especially maintenance of infrastructure. It shouldn’t take a hurricane and a flooded city to make people realize levees are important if part of a city is below water level. But it did. It shouldn’t take a dramatic collapse to convince people that bridges really ought to be built and maintained so that they won’t collapse. And telecommunication infrastructure is important, and it shouldn’t require a possible pandemic and work from home orders to make people realize that.
I would not be surprised if NASA is locked into a particular teleconferencing system by contract, either a NASA one or a GSA one.
The planetary science assessment groups (OPAG, MEPAG, VEXAG, SBAG, etc.), they use different systems and providers, but the list of options is limited. So do planetary missions. I’ve seen teleconferencing support from LPI, NRESS (the contractor who also supports things like grant proposal reviews), JPL and APL. Some are better than others. I think it is a contractual thing. It’s much easier to add tasks and funding to an existing contract than issue a whole new contract.
what does “HVCC” mean ?
that they nearly had a smoking crater with the world’s TV cameras recording it all ?
Read the article.
Well, at least the OFT-1 entry, descent and landing worked, so they weren’t at risk of smoking crater images showing up on YouTube. So there wasn’t a close call in that respect. But I would call something like that high visibility. At the same time, I’ll admit I’m not too happy with the term. I’m more worried about _low_ visibility close calls. Those are the ones that don’t get attention and that means a serious problems may not get fixed. That happened twice during the Shuttle program.
High Visibility Close Call
It was interesting hearing Doug Loverro give his reasons for classifying this as HVCC. He seemed to imply that it could be argued either way, but he said it seemed to him to meet the spirit of the classification. But he also said that giving it this classification automatically sets wheels in motion and creates a requirement for an organizational review of processes and root cause, and he made it clear that he felt that in this case this was warranted.
But even more interesting is when he mentioned that on the wall in his office is a list of all incidents that have been formally classified, which includes higher level classifications such as mishaps, loss of mission and loss of crew. But the list also includes high visibility close calls. He said designating this as HVCC essentially puts it into the record books (my paraphrase) so that it will be studied from now on.
In spite of all the politeness and the “working together” slogans, the fact that Doug Loverro made these particular statements during the teleconference seems in my opinion to send a pretty clear message to Boeing that just fixing the items on the list isn’t good enough, that real change is needed. But the HVCC classification also requires a review of NASA processes, so in a subtle way this may have been a message to not just Boeing.
“Chilton later said that Boeing would re-fly the OFT if need be, but we do not know who’d pay for this extra mission”
I thought this was fixed price and that the company had to pay their own way.
I thought this was fixed price
That was the original plan but they changed it to cost plus because NASA thought it was important to have oversight.
That worked out well.
Since when? There was a very dubious $300 million added to the contract, for Boeing to provide additional services (availability and lead time) which weren’t in the original contract. But that’s not changing the contract to cost plus. I don’t even know if it’s legal to make such a change. They’d probably have to cancel the fixed price and issue a new, cost plus one. That’s a massive amount of paperwork, and I haven’t never heard anything about such a change.
I don’t think so. Switching contract type after award. I’d double check that one.
A slightly off-topic question for those in the know.
Are we paying the Russians to send Canadian and Japanese astronauts to the ISS?