The Aerospace Safety Advisory Panel is Confused
Aerospace Safety Advisory Panel Releases Annual Report
“This report is based on the panel’s 2012 fact-finding and quarterly public meetings; center visits and meetings; direct observations of NASA operations and decision-making; discussions with NASA management, employees, and contractors; and the panel members’ past experiences. The report highlights issues that could have an impact on safety.”
2012 ASAP Report
“In FY13, we predict this planning-funding disconnect will again drive a change to acquisition strategy, schedule, and/or safety risk. The ASAP is concerned that some will champion an approach that is a current option contained in the Commercial Crew Integrated Capability (CCiCap) agreement. There is risk this optional, orbital flight-test demonstration with a non-NASA crew could yield two standards of safety–one reflecting NASA requirements, and one with a higher risk set of commercial requirements. It also raises questions of who acts as certification authority and what differentiates public from private accountability. Separating the level of safety demanded in the system from the unique and hard-earned knowledge that NASA possesses introduces new risks and unique challenges to the normal precepts of public safety and mission responsibility. We are concerned that NASA’s CCiCap 2014 “Option” prematurely signals tacit acceptance of this commercial requirements approach absent serious consideration by all the stakeholders on whether this higher level of risk is in fact in concert with national objectives.”
Keith’s note: It is exceptionally odd that the ASAP gets all hot and bothered about certifying American-produced commercial crew spacecraft when the ASAP all too willingly said it was OK to fly Americans on Russian Soyuz spacecraft – spacecraft which have never been given the same level of formal safety certification by NASA – i.e. the certification that the ASAP apparently wants for domestically produced commercial spacecraft. A number of years ago, at a time when Americans living on Mir were exposed to repeated accidents, I asked (then) NASA Deputy Administrator Fred Gregory in a public setting if Russian spacecraft meet or exceed NASA safety requirements. Gregry said “clearly they do not”. This question and response was subsequently referenced in a congressional hearing.
It is also a bit odd that the ASAP was perfectly happy with NASA’s plan to fly crews on Orion/Ares 1 flight after only one unmanned test. The same (apparently) goes for the current plan for Orion/SLS. The ASAP’s credibility suffers when they pursue contradictory and inconsistent paths such as this.
Folks:
Find out who owns the folks on this board and you’ll have your answer why they’re dissing commercial crew.
The fact that the next ‘awards’ in the spring of ’14 are FAR contracts (just like the good ol’ days) says a lot about where they are coming from.
tinker
Tinker: Check out the link below for their members. I do not see any significant membership by what some may call ‘Old Space’. Most of the members have past government links or a few with commercial links. I do not see your reasoning here for this comment. http://oiir.hq.nasa.gov/asa…
My experience with the NASA ASAP panel in the past has found them to be a bit conservative, but that is their job. They are advisors only. In the end, it is the NASA administrations job to make to call.
In terms of FAR contracts, if companies like SpaceX want to compete for government business, like EELV – then it only makes sense for them to comply with FAR requirements.
As for FAR versus SAAs, a key issue is cost. FAR compliance adds significantly to the cost of a procurement without necessarily improving the end product.
NASA’s procurement strategy already is broken, and NASA knows it, which may be one reason that NASA was less enthusiastic about FAR than Congress (trailing clouds of lobbyists).See the August 2011 Falcon 9/NAFCOM procurement study to see just how far out of whack FAR procurements can be.
The comparison with Ares/SLS with Orion is spot on.
ASAP just shredded their …If there was any left to begin with.
This report assumes without justification that anything other than traditional NASA safety rules automatically gives “a higher level of risk”. The report doesn’t even consider the possibility that there are other ways to get equal or greater safety. It seems to be the work of people with limited critical reasoning skills.
Let’s look at some of NASA’s decisions versus SpaceX decisions. With Ares I, NASA was going to put people on a new launch vehicle after only one test flight, as Kieth noted. The same is true of SLS. SpaceX won’t be putting people on Falcon 9 or Dragon until both have made quite a few flights to orbit without anyone on board. There’s a high-level strategic decision by SpaceX that leads to much more safety by leveraging a high flight rate of a cheaper system that can be used for un-crewed as well as crewed flights. That doesn’t automatically mean every decision made by SpaceX is going to lead to better safety, but it is suggestive. It demands, at the very least, an open mind about whether NASA is really all that safe compared to commercial decision-making, an openness which is clearly absent in this report by ASAP.
Through COTS, CCDev2, and CCiCap, NASA has been closely involved in reviewing the designs of its commercial partners. There have been dozens of very detailed review meetings, and dozens more are planned. And NASA’s commercial crew office hasn’t been ignoring traditional NASA safety standards — they have simply adopted a policy of flexibility that presents the traditional safety rules to the commercial partners but allows the partners to propose alternative ways to achieve the same or greater safety. NASA must still be convinced the alternatives do so.
Apparently, that kind of open-mindedness is too much for the small-minded people on ASAP.
I agree. The NASA approach to safety is based on probablistic design risk analysis. It is only as accurate as our ability to anticipate every failure mode and assign it a precise probability based on a paper design. It pushes the designer to make every system redundant, adding weight and cost which often serve no purpose. This also explains why NASA feels “safer” putting a crew on the second SLS launch than on the tenth Falcon. But the NASA strategy is based on a false premise. In reality most launch vehicle losses are due to failure modes that are not anticipated anywhere in the design process and can be discovered only by repeated testing in actual flight. The solution is not redundancy, but correction of the original flaw in the design. In fact, the O-ring that failed in Challenger was redundant, and thus, by NASA methods, infallible. Thus for launch vehicles at least, the approach taken by SpaceX and ULA under SAAs for the Falcon and Atlas, to fly at least ten missions (and correct any design flaws that are revealed) before putting a crew on a launch vehicle is actually considerably safer, and the ASAP premise the “NASA quality control” strategy leads to safety and industry strategies do not, does not appear consistent with the historical record.
Another example of poor reasoning skills by the ASAP: “The ASAP strongly believes that a cost type contract is appropriate for Phase 2. Fixed-price type contracts are appropriate for low-risk undertakings where the requirements are clearly understood by both the government and the contractor. Much of Phase 2 is neither, and we believe both schedule and safety would be at risk in a fixed-price environment because of the relative inability to defer or apply resources to problem areas that will inevitably develop.” (Page 6)
First of all this part is poorly worded because the term “risk” is used both to talk about safety and to talk about cost/schedule overruns. But it’s clear that “risk” in the phrase “low-risk undertakings” can only mean cost/schedule risk, because that is what a cost-plus contract can directly address, by adding to the schedule and price when unknown factors come up.
There’s no reason fixed-price contracts can’t be used for any level of risk. The private sector has a huge amount of experience with determining the appropriate pricing for risk. It’s up to the commercial partner to take into account risk when determining the price they are will to take for the work. If there are more complications than expected, the contractor has to eat the cost overruns. If the contractor has fewer complications than expected, the company gets more profit.
Fixed price contracts force the companies to be realistic about the risks and set the price appropriately, not under-bid, as is the norm in the world of cost-plus contracting.
The private sector has to make these kinds of judgements about risk all the time when deciding what internal projects to proceed with. Companies that can’t do it well quickly go out of business, and their resources are re-allocated to those organizations that are better at judging risk.
By claiming that fixed-price contracts are only appropriate for “low-risk undertakings”, the ASAP shows a profound misunderstanding of how business works.
Steve:
Under these onerous conditions, I think that SpaceX should not even apply for the next contract and do so loudly. Let the taxpayers know that their government is acting in bad faith, from the lawmakers to the committees they stack with their cronies. SpaceX doesn’t need a penny more from the government to develop a crewed version of Dragon. They will receive hundreds of millions from the last contract to finish development of a spacecraft that was always meant to carry crew.
NASA has succeeded in giving SpaceX a much needed lift up in development funding and the technological ‘knowledge trust’ that taxpayers supported over the decades. They did this is spite of a hostile environment imposed on them by said lawmakers and committees. As end runs go, it was brilliant.
But you could hear the frustration in Mr. Mango’s voice in the Commercial Crew update yesterday as he put forth his case by showing how many states were participating in the program, trying to put it in language that lawmakers could understand. I read it as “Look how well us and our commercial partners have done with so little. Our way does work and is working!”
SpaceX will do fine with cargo contracts from NASA. It’ll be pretty hard for the American people to swallow SpaceX having a perfectly good crewed Dragon on orbit, fully tested, and not have it allowed to visit the ISS.
Instead of ranting at you, maybe we should be ranting at this committee (or is listening to the public not in their mandate).
tinker
Chris – I am not sure what experience you have managing fixed proce contracts, but I can tell you your comment..
“By claiming that fixed-price contracts are only appropriate for “low-risk undertakings”, the ASAP shows a profound misunderstanding of how business works.”
.. is incorrect. I have managed quite a few fixed price contracts and the supplier does not ‘eat the overruns’. As is often the case, especially in a development phase, the supplier lists all of the new/changed requirements and they will submit a change order to get paid. I have rarely seen a development program executed in FFP (Firm Fixed Price) that does not have a series of change orders along with it.
So, I would say, based on my experience the ASAP is correct that FFP is not a proper contract method for a early phase development program.
If the buyer ends up with a bunch of change orders after original contract is signed, that’s an indication that the buyer didn’t do a good job of specifying what was required in the initial contract, most likely because the buyer tried to micromanage the details too much.
The fact that there have been some programs in the past that were “fixed price” but with lots of change orders leading to additional costs doesn’t prove that that’s the way it always has to be, even with high-risk, high-unknown programs.
COTS managed to successfully get commercial cargo service to the ISS developed with zero change orders, and zero cost overruns. And there were definitely significant risks and unknowns involved at program inception, and rigorous safety standards to uphold, since the spacecraft developed under the program had to dock with the ISS and become a part of it for a couple of weeks at a time.
ASAP has lost all credibility