SpaceX Releases Preliminary Analysis – Strut Failure Likely Cause [With Audio]
![SpaceX Releases Preliminary Analysis – Strut Failure Likely Cause [With Audio]](https://media2.spaceref.com/news/2015/crs-7.jpg)
SpaceX – CRS-7 Investigation Update
“From the first indication of an issue to loss of all telemetry was just 0.893 seconds. Over the last few weeks, engineering teams have spent thousands of hours going through the painstaking process of matching up data across rocket systems down to the millisecond to understand that final 0.893 seconds prior to loss of telemetry.
At this time, the investigation remains ongoing, as SpaceX and the investigation team continue analyzing significant amounts of data and conducting additional testing that must be completed in order to fully validate these conclusions. However, given the currently available data, we believe we have identified a potential cause.”
Marc’s note: Today Elon Musk of SpaceX stressed that the substance of the media briefing was preliminary analysis and not a definitive result.
Having said the likely cause of the failed Falcon 9 launch was a failed strut that broke free in the second stage liquid oxygen tank that was holding down a helium tank.
At approximately 3.2 g, the strut holding down the tank snapped. There was no evidence of damage prior to launch from close-out photos. The struts are not made in-house. The supplier was not named. Musk said that they were able to replicate failure with 1000’s of struts and they found a few that did not meet specifications.
In future SpaceX will test each strut individually in-house to certify them and will most likely move to a different material and supplier.
Addressing the issue is pretty straightforward and may only delay the next launch a few months. However, Musk indicated that the delay would be costly to SpaceX, perhaps in the 100’s of millions.
Musk also noted that if the Dragon capsule, version 1, had updated software, which will be on Dragon V2, that they could have deployed the chutes and saved the capsule. The next Dragon launch, also a version 1, will have the updated software.
The Falcon Heavy which was to have launched in Q1 next year will likely slip to Q2 with Musk saying, perhaps April.
SpaceX briefed its customers last week and all are still showing their support. He also indicated that the failure will not affect the commercial crew timeline.
Musk indicated a statement would be issued after the conference call and we’ll post it as soon as it’s available.
Listen to the conference call:
It just goes to show you that no amount of ground testing is really enough. All it takes is something like a microscopic stress fracture or crystallisation of the alloy in a way that’s fragile under certain harmonics. Combine that with the right harmonic vibration and g-forces and BOOM!
Space is hard.
It might be possible to have tested the struts or samples from the strut lot on the ground and caught the problem. One of the things that can drive the cost up. Musk admitted it will cost more, but they will eat the cost (at least for the currently signed contracts).
NB: This is my experience from the defense industry quite a few years ago. We used to buy aerospace components, but things may be different in the space industry or have changed over time.
The normal procedure for testing big batch components like that was to do sampling. Pick some number, based on the batch size, and test them to the specified limits For example, for bomb fuzes, put them in bombs, drop them and see how many go bang. If too many fail – which could be one depending on the intended usage – reject the entire batch and don’t accept it back until the manufacturer can show they’ve reworked all of them acceptably.
That’s called single sampling. You can also do double sampling, where you test a smaller fraction – but more than half of what you’d take for single sampling – of the batch. If too many fail, reject the batch. If few enough fail, accept the batch. If an intermediate number fail, pull as many again from the batch and test those as single sampling, but with fewer allowed to fail than in plain single sampling. This allows suppliers with very high confidence in their product to lose fewer units to testing and penalises overconfident suppliers.
I suspect this won’t work here because each strut might be made independently of each other. Sounds like they are going to test each strut which means they won’t want to test it such that it weakens it significantly.
The struts are made to a single spec, although dimensions might vary.
I suspect that they will test all (including flight hardware) to something like say 150% of expected load, a smaller sampling to max spec and a few to failure.
They had to test thousands of these struts before they had one fail way below spec. Apparently they have flown hundreds of these struts on Falcon 9 flights before now and had never had a problem. Musk seems to have implied they will ground test these struts from now on and only fly the ones that pass.
I did not see any data on the actual failure rate. But the fact is that no part, for a car, plane, or rocket, would be acceptable if any parts fail at only 20% of the expected load at failure. “Quality control” as Arthur Deming formulated it, doesn’t mean testing parts and discarding the bad ones. It means controlling the manufacturing process so that each parameter falls within a specified normal distribution. A factor of five difference in failure load cannot occur due to random variation and means that the manufacturing process is not under control.
Given that only a few out of thousands failed below spec, the one that failed at 20% is definitely an outlier. Implies a failure mode that just doesn’t happen 99.95% of the time, very difficult to find something like that with normal batch inspections.
Uh, you might want to read that statement a little more closely. They replicated the failure on more ground struts. As written above, they either:
a. Replicated the failure and thousands of failures occurred, or
b. Replicated the failure, and tested thousands, with some failures. or
c. Replicated the failure and found a few that did not meet specification.
It is both B and C.
Space is expensive.
So it was the helium tank and will be easy to fix. A support strut. He also said that Dragon will get software to self abort and save the cargo. Should have done that as people suggested here years ago. Maybe he does not read NASA Watch.
More likely it wasn’t done because the odds of a cargo capsule surviving a launch vehicle failure are very low.
SpaceX only just had the first test flight of a Dragon equipped with an escape system, after some years of development. Without an active escape system to create a more controlled situation it seems impossible to make a decision system smart enough to determine the right moment to deploy parachutes (“Are we clear of the debris cloud? Is the wind right for a safe landing? Do we still have enough altitude? How about now?”).
It’s not necessarily clear yet that the Dragon could have survived this or a similar accident even if the parachute had deployed. Not enough is known about the separation and the condition of the capsule after separation to say that it could have survived. Without doubt, the most expensive payload item, the IDA, could no have survived since it was in the trunk anyway. While it may ultimately be fair game, when more of the story is known, to critique the system that allowed whatever it was that was wrong with the strut to slip through, I don’t see any reason for recrimination about not having a parachute activation mode for this case.
Keep in mind that if an automatic parachute deployment mode for off-nominal launch conditions is added, then that itself would become an item susceptible to failure that could potentially falsely trigger and thus cause failure of another wise successful mission. Good systems engineering would require rigorous assessment of the relative probabilities of the various competing failure and recovery modes to determine whether it’s really a good idea.
I suspect the parachute deploy would be ground initiated but you’re correct that anything you had figures into the failure equation. You could still have the circuit that deploys the chute false activate. They would likely have to give an arm command prior to the deploy command.
They said they had contact with the Dragon until it dropped below the horizon.
I suspect that the Dragon, Falcon 9 Upper Stage and First Stage need putting on a vibration table to see what breaks lose. Fully fuelled, just in case there are temperature problems that room temperature water would not find.
A vibration table that can hold a 224 foot tall fully assembled and fueled rocket? and subject it to flight-like stresses of 3 g or more?
I think a tabletop apparatus that can test the necessary components would be a little bit more of a sane suggestion.
Vibration testing of the struts as parts may have been done. However the strut is unlikely to have had a helium tank fastened to it.
After 17 successful flights we are looking for unusual cases so simple tests are unlikely to detect the problem.
There’s a good chance that not all simple design errors have been found yet. 17 flights would be about one month into a proper test program for a commercial aircraft.
Yet another reason that reusability is key: it enables thorough testing without bankrupting the vehicle supplier.
Amen!
Hold-down test firing of the completed second stage. Yes, I expect they would have to remove the high-expansion section of the nozzle, or temporarily replace it with a low-expansion section.
[Edit] But it sounds like such a test under only 1 g of acceleration would not have found this problem.
A full length hold down test is one way of finding out what happens to a second stage. Upper stages are normally destroyed when they hit the ground.
How would that simulate 3.2g in an unfired second stage during flight?
Fil the LOX tank with mercury first 🙂
It wouldn’t, which is why we need re-useable second stages as well. It could find several kinds of assembly errors.
They did put it on a vibration table. It just happened that the vibration table was flying through the atmosphere and the Dragon had a full load of usable cargo. That’s how you do a full functional test of a fully loaded vehicle – by flying it.
https://screen.yahoo.com/th…
Just fix it!!
No complete description of the component or testing procedure is given, but one possibility is that the strut was made of an alloy that became brittle, at least in some areas, when cooled by liquid oxygen. The best test for this is impact toughness. Problems as subtle as an error in heat treatment temperature can reduce cryogenic impact toughness without an obvious change in appearance. Anotehr possibility is that at cryogenic temperatures a scratch or material imperfection acted as a starting point for brittle cracking.
The component must be manufactured in a consistent manner, in terms of composition, microstructure, materials properties, dimensions, surface finish, hardness, impact resistance, yield point, elongation, and ultimate strength. If this is the case, each parameter will vary only in a consistent, normally distributed manner. the manufacturer should be able to provide these records of the variations in component parameters, which reveals whether the manufacturing process is under adequate control.
Several times Musk also referred to a bolt. I never got firm conclusion as to whether they suspect the strut failed or the bolt or what they actually tested. “Thousands” of struts per launch vehicle seems pretty high but thousands of bolts holding struts seems more likely. And bolts can be easily damaged if they are over torqued with no visible signs of damage in closeout photos. And of course if the bolt were defective to start, it would multiply the effect of over-torquing. I wonder if they have documentation on the torque for every bolt.
Couldn’t find the old comment in NASA watch archive because it was before Disqus.
But I had joked “How many NASA engineers does it take to torque bolt”
And a guy replied with a remarkable answer that detailed how crazy expensive proceedures had become.
Right, but knowing Musk, he would take a digital readout torque wrench with barcoded bolts and automate the whole thing. You wouldn’t need one guy doing the torquing and 3 guys there to record it.
I actually think I remember that comment lol.
Here’s the “How many NASA engineers” story…
http://www.transterrestrial…
I’ve read things like that before, but this was impressive.
It’s done that way because people have learned the hard way that if you don’t do it that way, then people die and expensive stuff blows up. It’s done pretty much the same way in the airplane business too, for the same reasons. It’s even more rigorous in the nuclear power business.
Keep in mind that just because you can buy them at your neighborhood hardware store does not mean bolts are simple. Bolts are very not simple. When used in very high performance systems, bolts are extremely not simple.
So I know that I am just revealing my materials science ignorance… but why would you make a strut operating at LO2 temperature out of steel? Aren’t there other metals which don’t suffer embrittlement from low temp?
Dear lord, surely they didn’t make that mistake.
???
The mistake of using in a cryogenic environment material which embrittles at cryogenic temperature.
Obviously becomes an issue with re-use I would think? How many times can you chill them till they fail?
A correct design correctly made should be re-useable a large number of times.
Surely the specification considered that. They may not be talking though.
I think I heard they’re thinking of replacement struts made of Inconel.
Interesting, it looks like they already print the superdraco combustion chamber out of Inconel in-house.
I wonder how many components like this are single points of failure?
Probably hundreds of structural components, Struts, welds in the tanks, etc. Usually structural single points of failure are deemed “non credible” by design. It’s hard to make a redundnat tank vs. a redundant flight computer.
Funny thing is, I don’t recall ever hearing of a similar failure before.
Parts break off. Look at skylab for example or GSLV in 2010
I t looks like they will have to scrap the stages they have built and build new ones. At least they can use the engines. H e said there are a hundred or so on each flight. I doubt there is any way to test in place. It sounds like a pull test found the ones that failed. Failed at 2500lbs, should have taken at least 10,000lbs.
No, they don’t. They need to replace the struts, that’s it.
Why not just replace the struts S13?
They might,but there are a hundred in many different places. They could cut them off with a plasma cutter and weld in new ones, then x-ray. They might not be assessable. If a stage starts out as a tube, struts may be welded in first and something covers them up. I sure hope they can. They have stages in different points. If no struts are installed, no problem. From finished back there should be an access port they can go through and bolt in new ones. They must be bolted. A little hard to weld steel to Al.
If they’re bolted in, it should be a relatively simple task to install new struts and bolts for all of the helium tanks.
In the conference call, Elon Musk specifically mentioned the struts were bolted on. It should be pretty easy to replace them all.
See the video and you will see that they are access able.
Looks like a very easy http://gizmodo.com/the-spac…
Two observations: Firstly, if one strut failed under purely nominal circumstances, have other struts failed but not actually caused a catastrophic failure? If you fly hundreds of struts and one fails, and then you test thousands and one fails, then we’re in the same ball-park in terms of the statistics. Secondly, this demonstrates absolutely the value of SpaceX’s first stage recovery attempts – I’d wager that the first one back now gets torn down to individual components rather than being re-flown.
Yeah, that’s way smarter than it sounds. If it’s so oversized (x10 expected load), doubling up on smaller ones will give you a single-point safe-fail structure, while only doubling the failure points, in the end dropping the likelihood of structural failure if the quality control is any good.
In fact, I’m actually surprised it’s not already done like that, if it’s a catastrophic point of failure that leads to a helium bottle blowing up inside the O2 tank if it fails, and the solution doesn’t seem prohibitive in terms of mass. Structural margin doesn’t protect you from everything!
Strut out capability.
Seriously, lol. Was this strut just a piece of metal? Pretty inexcusable for one to fail that badly.
I’m just an old construction guy,
A Stud wall with many studs is much safer if you have a bad stud
Distributive load
Than a post and beam structure were each beam is critical to the support of the structure.
Point loads
Common sense
Share the load
Adding mass to a rocket is much more of a problem than adding mass to a house. Just adding extra strength can easily cut into the performance too much. If all of the structure were made redundant, the rocket probably wouldn’t get off the ground.
How many times do you double up? what is the “sweet spot” in oversizing vs. quantity? This was not the only strut they had. Say they had 10 already. Do you go to 20? Why not 40? At some point you cal it quits. It is easy to say they should have had more struts without knowing the tltal design trades. I also openly admit that I have no idea whether for this application a x10 margin is adequate, too smal, or too large.
Depends on your safety rating for that particular structure. In planes for example (which, admittedly, are a very different business), no part can be less than safe-fail (i.e: the plane can fly if it breaks). The crucial ones (lose it, lose the airplane) need redundancy, where feasible, so if one fails the others save your bacon. Only where it is completely infeasible to add redundancy (say, frontal landing gear) do we resort to high structural margins (3-5 times the design load).
My point is maybe the helium tank support structure (as a whole) should perhaps have been designed as a safe-fail structure where any of the individual components can fail without the structure as a whole to fail, instead of trusting on wide design margins. Hopefully, if I’m right and the mass penalty for that isn’t too big, that’s exactly the kind of fix SpaceX will implement.
Makes sense, good description. They could have more redundancy in this case, rather than just building to high structural margins.
Performance margins on launch vehicles don’t permit you the mass allowances to have high structural margins, which is why you must have rigorous quality control on your engineering and manufacturing. And that is why the rocket business is expensive.
Yup, I know how tight they shave the margins. Which is why I opened my eyes a lot at “rated for ten times the expected maximum load”. That takes mass! Sometimes the culture of an industry is so ingrained things that seem obvious in retrospect don’t get found until someone with no experience looks at it and goes “why do you guys do that?”.
And of course maybe I am missing something about this particular structure that made redundancy totally not practical. I would have to run the structural analysis on the stage structure, and maybe have a talk with the thermal guys, and all that stuff is confidential and well away from the public (and the competitors!) eyes.
But from the comfy confines of my chair, these are my humble two cents! 🙂 I hope they where at least thought-provoking and someone learned something in the discussion.
Anyone that knows anything about He bottles on that system knows that they are CF composite, and composite does not fail like malleable metal but like a ceramic, The slow increase of pressure in the O2 tank does not support the conclusion of failed strut.
The He tank didn’t fail at all, the strut was made out of a steel alloy.
The tank came loose and broke a feed line. Musk says buoyancy, 3.2g would make it heavy and it pull down on the line. Doesn’t matter, The line broke. At least the tank did not burst, might be a bigger fix.
Was it a slow increase of pressure in the O2 tank? Where did you get that? They are talking less than a second. The first stage kept on pushing for around 8 seconds after that in the video, and then the whole thing finally went poof. I think it was Dragon that is the dark spot making its way upward off the scene and it must have separated within a second or two of the original failure judging by its movement.
The initial reports and data from SpaceX said that the O2 tank pressure was increasing since before the start.
Source????
I think it was in one of the first reports.
SpaceX:
“we expect to return to flight this fall and fly all the customers we intended to fly in 2015 by end of year.”
Falcon 9/Jason 3
Falcon 9/SES 9
Falcon 9/Orbcomm OG2
Falcon 9/SpaceX CRS 8
Falcon 9 • JCSAT 14
Falcon 9 • Eutelsat 117 West B & ABS 2A
Falcon 9 • SpaceX CRS 9
wow
I’m going to guess they haven’t stopped Falcon 9 production and have a batch or two of rockets that are nearly ready to fly (pending strut changeout). Keep in mind, if they have LC-39A ready to go by September, then they would have two pads to work with. Assuming RTF launches from SLC-40 on Sept. 15, there’s then 107 days left in the year. Jason 3 is set to go up from Vandy and can be launched more or less anytime. That leaves 5 more flights from the East coast. If only SLC-40 is available, that means a pad turnaround of 21 days per launch. If LC-39A can take two flights, then you have a pad turnaround time of 35 days for SLC-40 (or the other way around).
So it’s ambitious, but possible; even with only one launch pad.
However, factor in delays, payload, weather, and rocket related, and I’d say 3 or 4 more launches this year is a more realistic number.
Unless they have to really gut rockets which are essentially complete they should be able to fly that manifest – in principle. But it would be a flurry and chaos beyond amazing.
I agree that its likely to be whittled down.
If SpaceX lauches a flight or two by the end of the year I’ll call that a huge success.
As some have said it might well be difficult to get all the flights off the pad by the end of the year but it’s the right attitude to have and shows exactly why I think Space X can achieve the sort of things NASA can only ever dream of today.
After hearing their launch plan I was inspired.
If your goal is to die on Mars you can’t lag around, not making any money, not flying rockets.
I had joked about the Saturday night lift skit where the guy says just FIX IT!!!
SpaceX are lucky the the fix is so simple.
I recall many comments saying that when they blow one up, they will be driven out of business, Hardly 🙂
Seems to me they will just get better.
Can’t wait to see them land one on their Barge this fall!!!
Fixing the bad struts may turn out to be simple. Fixing whatever allowed the bad struts to fly in the first place may not be so simple.
1st return to flight launch, 2nd week in Sept.
2nd launch first week in Oct.
3rd launch third week in Oct.
4th launch 2nd week in Nov.
5th launch 4th week in Nov.
6th launch 2nd week in Dec.
7th launch 4th week in Dec.
and then, after that, on Dec 31 the entire SpaceX staff enters rehab.:-)
Might be more like 1st launch in Oct, 2nd in Dec. Done for 2015.
So their quality control is not up to snuff? This problem is more than just a strut, I suspect.
No, it’s likely their inspection processes adhered to normal aerospace industry standards, which is to test a percentage of a batch of parts (normally 10% or 15%), and if that percentage passes testing, declare the whole lot good and send it to the customer. The customer usually does the same thing after receiving a shipment of parts. This means that each individual item isn’t tested – this would be time consuming and probably also prohibitive by cost. Of course, if you don’t mind paying extra, you can have extra testing.
And keep in mind this failure mode is 1 in 1,000 or higher, possibly above 1 in 2,000. Elon Musk said they had to test “thousands” to find one that failed as far below spec as the one in the rocket did.
Makes me wonder if a bolt failed not a strut???
Elon’s comments in the conference call are a bit ambiguous. It kinda sounds like both bolts and struts failed on the ground and that the bolts and the struts are made by the same supplier and are made of the same metal. Perhaps the bolts are considered part of the strut assembly. I think it’s likely there’s no way to tell if the bolt or the strut failed from the rocket’s telemetry data, my guess is the effects that the accelerometers detected would be about the same either way. Prudence dictates they replace both anyway. Beefier struts and bolts made with a higher quality material, if necessary, from a different supplier.
Maybe you have a different source. That is NOT what is in this article. “Musk said that they were able to replicate failure with 1000’s of struts and they found a few that did not meet specifications.”
I’m sorry, I think you just repeated what I said.
Elon Musk said the struts were designed and certified to handle up to 10,000 pounds-force. SpaceX tested thousands of struts and had “a few” fail around 6,000 pounds-force, and then finally had one fail at 2,000 pounds-force like the one on the rocket did.
It is amazing to me that thousands struts needed to be tested. The probability of failure could have been calculated with fewer tests by determining the mean and standard deviation.
If the failure only happens in one in every 2000 struts, how could fewer tests have ever caught it?
Most flight structures are qualified for flight using few tests. Testing thousands of components is usually unaffordable. A statistical test provides enough data to show that requirements are met. If yes then the obvious question is then are the requirements adequate.
Requiring a structure to handle 10,000 pounds-force when it only experiences 3,500 pounds-force in flight should have been quite adequate.
After some consideration, I see what you mean. That sentence is rather poorly worded. I recommend you read the transcript to the conference call. I will post the link here:
http://shitelonsays.com/tra…
A picture is worth a thousand words
http://gizmodo.com/the-spac…
A little help from @John_Gardi Tinker for the rest of us.
It might be the falcon9 v1 flight using the older Merlin 1C engines. There was a shutdown of an engine and the sudden pressure drop caused shrouding to blow out with a flaring.
The engine itself didn’t explode.
https://www.youtube.com/wat…
Looks like SpaceX’s Texas launch Pad is almost ready for action?
My mistake this picture is from McGregor. I thought it was a construction picture from Brownsville.
Does anyone else think it was odd that they had “1000s” of these struts available for test? The only way I can think this is possible is if it was an off the shelf part, or if in fact it was a bolt that was part of the assembly. It’s also possible it was a flange, or Mr. Musk misspoke.
Off-the-cuff calculations show there’d be about 120-160 of these struts installed in each rocket. If they had about a year’s worth of these struts on hand for rocket assembly, that easily gets up to the “thousands” range.