SpaceX Releases Crew Dragon Explosion Statement
SpaceX In-Flight Abort Static Fire Test Anomaly Investigation Statement
“Initial data reviews indicated that the anomaly occurred approximately 100 milliseconds prior to ignition of Crew Dragon’s eight SuperDraco thrusters and during pressurization of the vehicle’s propulsion systems. Evidence shows that a leaking component allowed liquid oxidizer – nitrogen tetroxide (NTO) – to enter high-pressure helium tubes during ground processing. A slug of this NTO was driven through a helium check valve at high speed during rapid initialization of the launch escape system, resulting in structural failure within the check valve. The failure of the titanium component in a high-pressure NTO environment was sufficient to cause ignition of the check valve and led to an explosion.”
so, this is an easy fix as I understand it. Good news.
Titanium has a long history (back to its use in the SR-71) of not reacting well to various chemicals. As I recall, using a Pentel pen on titanium caused the titanium to fail, as the ink had an effect similar to acid. Cadmium plated tools also caused bolts and nuts to fail. So it seems the statement “It is worth noting that the reaction between titanium and NTO at high pressure was not expected. ” is a bit disingenuous.
Have the other titanium components in the spacecraft might have “unexpected” reactions with other components?
They also noticed that spot welds failed if they were done in the summer rather than the winter. Turns out Burbank increased the amount of chlorine in the city water during the summer to reduce algae.
I suppose their statement could be taken two ways, I took it to mean that prior to the incident they didn’t think the check valve would have ignited in the presence of NTO in that situation. But I suppose it’s possible they simply meant that they never anticipated the scenario of NTO not only leaking into the helium line but then becoming essentially a bullet fired into the check valve.
Especially since this was not normal corrosion this was an instantaneous ignition presumably caused by or at least exacerbated by the heat created by the high speed impact of the slug of 1.4 density NTO impacting the check valve and destroying it, and igniting it at the same time. Their investigation seemed to focus on testing the flammability of the titanium in the check valve in that scenario, probably as a way to confirm the theory of what ignited the check valve.
They did however use the term “reaction” so it’s possible that it was caused by the titanium being heated in the presence of this particular oxidizer. Coincidentally the oxidizer also happened to supply the heat by being slammed into the check valve at high speed and breaking it apart (or whatever was meant by “structural failure”). Although the statement doesn’t mention heat only high pressure, so it’s conjecture on my part that heat had a role. I did some quick reading on titanium and it seems it becomes more flammable if a “fresh” non-oxidized surface is exposed to oxygen, such as occurs when titanium is scratched or cracked. So maybe exposure of fresh titanium to NTO would have a similar effect, especially in a heated environment at high pressure.
I would say you are correct that when the pressure increased that the NTO absorbed some of that heat which will probably make it more reactive with the titanium, but IMHO the real problem is the check valve failed to prevent the NTO from getting into the helium line in the first place. That’s the point of having the valve there. As I noted above, this was a single point failure that would have killed a crew on the spacecraft. I think that is unacceptable as it should have been considered in a design review. It’s a basic question that needs to be asked “What happens if the check valve fails to prevent some NTO from backing up into the helium line and then we need to do an abort? What problems could this cause?”
Since we weren’t there, it’s possible it was asked and was decided it was not a problem based on engineering experience. If this system had not ever been used before, then it probably would have been prudent to run a test. Maybe that would have demonstrated a problem, or maybe not because you might need to get all the parameters perfect to cause an explosion.
But at the least the proper process would have been followed. That is what makes this incident appear to me to be much more serious than it’s been reported.
I don’t think there is any evidence that they didn’t consider the possibility of NTO getting into the line or that they didn’t run what-if scenarios for that. What they would most likely be looking for is whether this could lead to NTO coming into contact with something that it shouldn’t.
It is also possible that they considered what would happen when they pressurized in that situation, but didn’t foresee any problems with that either.
What they didn’t think of is the possibility that the NTO could cause the check valve to ignite, or again they may have thought of it but concluded that it would not happen. In hindsight that could have possibly been caught through testing, but maybe not as it may have required multiple tests with varying amounts of NTO to recreate the problem. I not trying to indulge them with an unlimited amount of benefit of the doubt, but I don’t think we have evidence that this was the result of carelessness or holes in their processes.
I believe NTO is an incompressible fluid, while helium in the lines would be a compressible gas. That almost certainly would alter the startup pressure transient.
When it comes to a “structural failure” and exposed titanium, it might not take much to put holes in that titanium oxide layer. This came up with one of the instruments on Cassini.
The mass spectrometer had a titanium collection chamber, and that was assumed to be chemically inert due to the oxide layer. During passes through the Enceladus plume, they got some odd results and eventually tracked it down to bare titanium. Nanometer sized dust particles in the plume (really; as in a thousandth the size of a common smoke particle) could fly in at orbital speed (5-15 km/s) and put microscopic pits in the titanium oxide layer. That exposed enough metallic titanium to act as a gettering pump and selectively remove certain gases. Fortunately, it was a short-term effect, and they could correct for it once they knew what was happening. But it’s evidence that titanium oxide layers are pretty thin and easy to ding up.
That’s not entirely the case. As I understand it, titanium parts are often considered very inert, chemically _because_ titanium is quite reactive. I know that sounds very counterintuitive, but the fact is that a titanium part will react with air and end up with a very inert, titanium oxide surface layer. That happens quickly and afterwards, that surface layer prevents the underlying titanium metal from reacting with anything the exterior of the part is exposed it. Now, if you scratch, ding or break up the part, that’s a different matter. That exposes the underlying metal, and that could be bad. Doing that with a pressure pulse from a strong oxidizer, and it could be very bad. But, from the sound of it, that check valve should never been subject to such a pressure pulse and NTO should never have been in the gas mix. So that would be an unexpected situation exposing metallic titanium and simultaneously exposing the titanium to a strong oxidizer which the part was never expected to be exposed to. I think it’s fair for SpaceX to say they did not expect that, and also that they didn’t have any reason to expect that.
That’s ok up to a point, but in engineering a critical system like this you need to chase down the “what if” something happens. Basically if the check valve fails to operate properly or not fast enough and some NTO makes it past that, what happens then. I don’t think that’s a problem that could not be anticipated in a design review. This is an accident caused by a single point failure that would have killed a crew if they were in the spacecraft.
I’m actually a bit surprised at the way that both Spacex and NASA talk about this. It has kind of been ‘ho-hum’, just a small problem that’s easily corrected. As a program manager, I’d start by reviewing other systems for potential single point failures that would jeopardize the crew. I thought the rules in the space game were that a single point failure could cause a mission abort, but not put the crew in hazard.
You are exactly right. What ever happened to redundancy and fault tolerance? Fault tolerance appears to be lacking in this system design. In a human-rated system a single fault should not cause a catastrophic event. And, like you said, what other single point failures leading to a catastrophic event are lurking in other systems on the spacecraft. I would also question the completeness and accuracy of their hazard analyses.
I don’t think this is something NASA would regard as a single point failure. It’s more like one failure (a leak) increasing the odds of a second failure (pressure transient damaging a valve), and those two failures in combination exposed a fresh (unoxidized) titanium surface to high pressure NTO (boom, due to conditions which should never happened in the first place.) That a chain of events, not a single one.
A chain of possible events isn’t something that’s usually considered when eliminating single point failures. Maybe it should be, because that’s usually what causes most serious incidents and accidents. The excuse is that, once you start considering how one event might make another more likely, the number of possibilities goes through the roof. The odds of even thinking of all of them, let alone mitigating them, just isn’t practical. In aviation, that’s what flight tests are about. Discovering problems through experience, not trying to design them out from first principals.
So you subscribe to the fly-fix-fly approach to system development. I prefer that hazards be found during the design phase and then designed out of the system before someone risks their life flying the vehicle. BTW I worked 33 years in flight test as a safety engineer, no injuries or fatal accidents on my watch.
Do you actually believe it’s possible to identify and mitigate every possible hazard during the design phase? Without infinite time and funding? Of course you want to catch as many potential problems as possible, but the idea that you’d catch everything isn’t realistic.
The question isn’t about fatal accidents or injuries. In the case of the Dragon 2 incident, there were no injuries. But it was a test, and tests of new designs are supposed to identify previously unidentified problems (among other things.) In your career in flight testing, are you saying there were never any accidents? I mean ones that didn’t injure anyone.
And, as I’d think you’d understand, that “fly-fix-fly” approach isn’t a matter of designing hazards out of the system before someone’s life is at risk. It’s about conducting the tests so that you find the problems without risking someone’s life.
I don’t think you’re using “single point failure” correctly. Or perhaps you are, and most people in the field are not. I’ve seen it refer to a single, isolated event. That makes this three failures.
First, something (and not necessarily the check valve) allowed NTO into the helium lines. That event was not the cause of an explosion. Since it happened in ground processing, that NTO was there for quite some time, and nothing happened.
Second, a contaminant in the helium lines cause a pressure transient during startup and damaged a check valve. By itself, a damaged check valve from a startup transient isn’t a problem. If it doesn’t block the flow of helium (fails open), there isn’t a problem. If it does, then the vehicle has eight SuperDracos and survive with one engine out.
Third, a titanium part was exposed to high pressure NTO. I don’t think that’s a problem in and of itself, since the part ought to have an oxide layer separating the NTO and the metallic titanium. If the part were damaged, it would be, but that implies a second event (failure) which damaged the part.
In the failure analysis I’ve seen, people track down the obvious, certain and immediate consequences of a failure. But looking at every way one things could, possibly, lead to another is beyond the usual practice of eliminating single point failures.
Well there you go. Glad they’ve finally released a statement on the root cause. Now they’ll have to address all of the possible ways that nitrogen tetroxide could have gotten into the high pressure helium lines. Faulty check valve? Faulty procedure?
SpaceX just released a more detailed statement. They’re going to switch to burst disks to eliminate the possibility of leaks, or this particular check valve anomaly.
Titanium in a high-pressure NTO environment is impact sensitive, but the explosion is not self-propagating. There’s a 1961 NASA technical report on it that I just linked in a comment at Transterrestrial Musings.
If you are referring to the helium check valve they didn’t say it was necessarily a suspect although they did make the interesting statement that “the reaction between titanium and NTO at high pressure was not expected”. So I suppose the check valve isn’t ruled out. However it becomes a moot point since they will no longer use a check valve and instead use burst disks. Although I don’t know if that means they don’t plan to hotfire test the SuperDracos anymore or else they accept that after a hotfire test they have to replace the burst disks.
As for the source of the leak the statement only refers to “a leaking component” without stating if they have identified the specific component. Either way I’m sure they (and NASA) will want to take this opportunity to check for any other possible leak sources. Although it sounds like there’s going to be some plumbing rerouted anyway.
I suppose it’s possible that some component other than the check valve leaked NTO into the helium line. But, since the check valve was there to prevent NTO from flowing backwards into the helium line, that would be the likely suspect, IMHO. But obviously I don’t know for sure since I don’t work for SpaceX.
Similarly, it’s not crystal clear to me exactly what Boeing got wrong with the design of their abort system plumbing either. They too gave a general statement on the cause without going into the engineering details.
I think the check valves were only controlling the flow of helium, this was apparently a helium pressurization line and I’m pretty sure NTO was never supposed to be in those lines. They said the leak of NTO into the helium line occurred during ground processing. They are solving it by rerouting plumbing so that NTO cannot accidentally enter the pressurization line, and for good measure they are also replacing the helium line check valves with burst discs which I would take a guess are not made of titanium but I don’t know. I wonder if burst disks are preferred also as being a better seal, to help keep the wrong things from getting into the wrong places. Although it does mean a part that has to be replaced after a hot fire test. Or maybe they are not planning to hot fire the SuperDracos on every capsule.
I don’t know either but I’m not sure the check valve is the most likely suspect. I think most pressure-feed tanks have some sort of bladder, to keep the fuel or oxidant and the pressurant from mixing. I don’t think that’s a universal practice, but it’s fairly common. Also the leak happened during ground processing, when, presumably, there wouldn’t be much pressure across the valve. On top of that, the valve just seems to obvious. Obvious problems tend to get attention from engineers. I’m used to failures in things that no one expected to be a problem, so no one really put in time and effort to make sure.
The announcement also confirmed that the original Demo-2 capsule will now be used for the abort test and the planned 1st operational crew capsule will fly Demo-2.
This is exactly what I predicted the day after the event: “I would guess that the process of recycling the system allowed one of the two (fuel or oxidizer) to migrate to somewhere that is was not supposed to”. I mentioned that we ALMOST NEVER pressurize these systems to operational pressure with fuel/oxidizer in them then RECYLE the system and de-press/re-press.
So they keep the tanks at full pressure (i.e. higher than the combustion chamber during firing) between firings? Say between the pre-launch tests and multiple restarts in flight? That wasn’t my understanding. It would mean a large overpressure on all of the plumbing for long periods. That sounds like a good was to guarantee a leak.
“Say between the pre-launch tests and multiple restarts in flight”
As I mentioned before: In processing of almost a dozen space vehicles with propulsion systems: they were fired a total of ZERO TIMES before launch. The thrusters are verified to work in vacuum at the subassembly level, then assembled into a complete system and pressure tested but NEVER FIRED before flight. I have also always had contingency procedures to offload the fuel & oxidizer after initial load and pressurization, but NEVER had to exercise this contingency. SO in about a dozen spacecraft programs we have pressurized the system with fuel and oxidizer in the system EXACTLY ONCE then launched it.