This is not a NASA Website. You might learn something. It's YOUR space agency. Get involved. Take it back. Make it work - for YOU.
Commercialization

SpaceX Identifies Possible Cause in Pad Explosion

By Marc Boucher
NASA Watch
September 23, 2016
Filed under ,
SpaceX Identifies Possible Cause in Pad Explosion

SpaceX AMOS-6 Anomaly Update 23 September 2016, SpaceX
At this stage of the investigation, preliminary review of the data and debris suggests that a large breach in the cryogenic helium system of the second stage liquid oxygen tank took place. All plausible causes are being tracked in an extensive fault tree and carefully investigated. Through the fault tree and data review process, we have exonerated any connection with last year’s CRS-7 mishap.
… Pending the results of the investigation, we anticipate returning to flight as early as the November timeframe.

SpaceRef co-founder, entrepreneur, writer, podcaster, nature lover and deep thinker.

53 responses to “SpaceX Identifies Possible Cause in Pad Explosion”

  1. Daniel Woodard says:
    0
    0

    If you look at the initial fireball you can see the LOX vapor plume running clockwise from the 7 o’clock position around 180 degrees to the 1 o’clock position. This puts the site of the explosion at a location where there is LOX above, LOX below and to the left, but no LOX to the right. This puts it inside the LOX tank on the right hand sidewall, the location of one of the helium COPVs. Possibly the thermal stresses induced by the LOX contacting the filament windings on the outside of the tank induced critical stress in the outer composite layers. This might be prevented by a modest layer of thermal insulation around the COPV. It can be seen here: http://www.douradosinforma….

    Some COPVs are definitely burst-before-leak
    https://www.youtube.com/wat

    • John_AnotherContractor says:
      0
      0

      Looking at their suppliers website they use a linerless He Tank.
      (Edit: sorry, that may have a composite bladder. It’s not a metal liner, though.)

      Has NASA used that on any other rockets? I’m seriously doubting a November RTF if ANY portion of that system failed. I was really hoping it was a ground op issue.

      If the He tank blew from Over pressurizing, such as a stuck fill valve, the data would have shown it and they would have pointed to it in the release. If not, then design, manufacturing, or assembly flaw. They needed a smoking gun, but I don’t think they are going to find it soon enough.

      • ThomasLMatula says:
        0
        0

        But this is the first time it occurred which may indicate a quality control issue. Yes, it would be good to redesign so it is more robust, but better inspection could be an interim solution.

        • Daniel Woodard says:
          0
          0

          A similar explosion happened at their original supplier during testing, though they since shifted to inhouse manufacturing. These tanks have been referred to as COPVs although I agree there is nothing specific to suggest they are lined. But the helium probably isn’t as much of a problem as the LOX outside the tank. There is clearly no external layer between the filaments and the LOX.

          Rare but unpredictable COPV failures have happened often enough that late in the program NASA restricted access to the area around the Shuttle after the COPVs were pressurized.

          • Charlie X Murphy says:
            0
            0

            How would a non lined composite retain He?Hence they have a liner. Plus SpaceX calls them COPV’s,

          • Spaceronin says:
            0
            0

            They are all lined indeed. The thing that drives the liner design is manufacturability. In theory a minimal thickness in the metal liner is all that is necessary. Its job is to contain the gas after all and not react the load. However practical liner handling during build limits this to a thickness that carries of order of 10% of the load for metal lined COPVs. Not sure about the plastic ones haven’t got to play with them yet but doubt they would be too good in cryo. Was this a trapped gas in the overwrap issue? I haven’t used these babies myself in cryo but can imagine any gases getting into the overwrap in a cryo tank could play havoc……….

            COPV rupture can release fragments with the muzzle velocity of an M-16. The pressure front itself will significantly raise the local temperature through adiabatic compression, above ignition threshold. If there is any mixing of combustables then it is likely to ignite, no need for sparks or static.. Although they are equally likely.

          • Daniel Woodard says:
            0
            0

            Metallic tanks of alloys such as Aluminum-lithium are also a possibility. They are somewhat more tolerant to minor damage.

        • Daniel Woodard says:
          0
          0

          Neither 100% inspection, nor increased load factors, nor NASA-type quality assurance (i.e. monitoring of procedures) nor even 100% testing to the specification are directly related to quality control, as conceived by W, Edwards Deming. Quality control requires keeping the manufactuing process itself under control, and one can only assure this with COPVs by measuring the actual variation in product quality by selecting a random sample of the parts and subjecting them to testing to the actual point of failure.

          If the manufacturing process is under control, the point of failure (in this case burst pressure) will vary only slightly throughout an entire production run and the points of failure must fall within a normal distribution. Anything else means the process is not well controlled even if every part meets the spec. One hundred parts might work and the next one may fail in service.

          This is particularly true with composite pressure vessels, since effective nondestructive inspection of such a complex internal structure is impossible and hidden defects may propagate during testing but only cause failure when in service.

          • ThomasLMatula says:
            0
            0

            This is one of the problems with an industry with low volume production like launch vehicles, the statistical based process that TQM are based on don’t properly apply because the production runs are just too small. Keep in mind that Dr. Deming’s work was based on his experience with Dr. Shewhart and focused on mass production where such sampling has validity. You would have to produce dozens of tanks to use those methods properly, an excessive cost in low volume production.

          • Daniel Woodard says:
            0
            0

            A valid point, but I would have thought that they do produce dozens of tanks, since each vehicle has three or more in both the first and second stage …

            I thought Deming’s work was called “statistical process control”., I remember TQM as more of a management fad.

          • ThomasLMatula says:
            0
            0

            It was Dr. Walter Shewhart who developed Statistical Process Control at Bell Labs in the 1920’s. Dr. Deming worked for him and became an advocate for it. It was a key element of his 14 Point System that become the basis of the TQM movement in the late 1980’s.

            The problem is the batches are likely too small to be used for SPC. You would probably have to destroy a third to a half in testing to get any valid number. That is the problem with very small production runs.

            Instead you need to do what Boeing did with the B-52, KC-135 and B707 and just over-engineer them. That is why those aircraft are still flying today. The DC-3 is another example. Yes, it adds weight, there is no free lunch, but it makes the design more robust.

          • Spaceronin says:
            0
            0

            Went trough ye olde DFSS black belt course back in the day…. Company policy. Interesting; but the statistical tools were not much use for small batch production. The confidence levels were to low to action. 100% inspection was the norm so it was not like we could save anything anyway. Over engineering is par for the course in the space sector. Of course the definition of over engineering is where the Devil is. MOS 1.05 is over engineered is it not? Presuming you have your load cases locked.

          • Daniel Woodard says:
            0
            0

            100% inspection is not necessarily as effective as testing a sample to failure. 100% inspection is by necessity nondestructive and NDT is of limited reliability with a structure as complex as a COPV. More critically, NDT does not provide an actual point of failure so it does not rule out variations in the manufacturing process. It isn’t clear whether the COPVs in the first and second stages are the same, but if they are than the production rate must be at least 50-60 per year. Six units selected at random and tested to failure per year would provide a reasonable estimate of the standard deviation of burst pressure and some assurance that the manufacturing process is under control.

      • Terry Stetler says:
        0
        0

        What supplier are they using? Type 4 or 5?

    • Tritium3H says:
      0
      0

      Hi Daniel, thanks for posting that video, and your analysis. I am still a bit unclear, however, as to how the rupture of composite cryo helium tank would cause the LOX explosion. Is there enough thermal energy in the rupture of a composite COPV to ignite the LOX? What exactly is the energetic trigger event that causes the combustion of the LOX? Thanks in advance!
      Cheers, John.

      • John_AnotherContractor says:
        0
        0

        The theory we are punting around here is the He tank failed in a catastrophic way as seen in the above video. This would have then ruptured the Lox tank, leading to the explosion.
        The He serves two purposes: It maintains a constant pressure in the Lox tank, refilling the volume as the Lox is burnt off. This gives structural stiffness to the tank, allowing it to be lighter overall. The constant pressure in the tank also gives constant pressure to the Engines pumps, a very good thing.

        • fcrary says:
          0
          0

          Rupture the LOx tank and produce some sparks in the process. The question was about the source of an “energetic trigger event” to start combustion. In a very oxygen rich environment, the event doesn’t have to be very energetic; a spark from metal hitting metal would probably do.

          • John Thomas says:
            0
            0

            You still need fuel. Perhaps Aluminum was the fuel or maybe more likely a piece of a COPV was propelled through the bottom of the LOX tank into the fuel tank.

          • Daniel Woodard says:
            0
            0

            https://www.physicsforums.c
            The energy of a compressed gas can be a tricky thing but as a very rough approximation E=P*V*ln(P1/P2) so for 100 liters of gas at 350 bar the energy released is about 210 MJ or about 40 kilograms of TNT.

            Again this isn’t by any means exact, just showing that there is easily enough energy in such a tank if released instantly by a catastrophic rupture for an explosion that could create the small initial fireball and ignite the released propellants from the second stage.

          • Tritium3H says:
            0
            0

            Understood, thanks for the info. Wow, I had no idea that the COPV tanks would be storing the helium at around 5,000 psiG. For some reason, I was (incorrectly) thinking it would have been much lower, and as such, I was wondering if a catastrophic rupture event would be an “energetic” enough to set off the LOX. However, given the pressures you mentioned, a COPV rupture would be like setting off a bomb inside of the LOX tank. Yikes.

          • Saturn1300 says:
            0
            0

            Or the over pressure blew out the common bulkhead, over pressure the kerosene tank, blowing it out. There was a large cloud of vapor ignited. The cloud could have had static electricity.

          • Shaw_Bob says:
            0
            0

            I’ve seen non-explosive demolition work on metal structures where the quantity of white-hot sparking as they collapsed was amazing. I can easily imagine a rupturing tank initiating an explosion.

  2. MountainHighAstro says:
    0
    0

    who knew Musk’s greatest enemy would be element number 2?

  3. John Thomas says:
    0
    0

    While they don’t believe this accident and the one last year are related, once they determine the cause they’ll need to review last year’s accident to verify the latest failure mode did not cause that accident.

    Also, I doubt they’ll fly this year. I’m guessing 6 to 12 months, longer if something has to be significantly redesigned.

    • fcrary says:
      0
      0

      That’s not at all clear. NASA would probably go back and reassess last year’s accident investigation. But it isn’t obvious why that should be necessary, or that SpaceX will do so.

      I can even imagine them launching without a permanent fix to the current problem. That is, If the problem can be avoided by lots of additional inspection and attention, it might make sense to go ahead and do it that way, while a permanent solution is in the works.

  4. Saturn1300 says:
    0
    0

    A solution would be to use the tanks or supplier of the Atlas tanks. 100% reliable. Bruno said he would be happy to help.
    Dragon 2 will have to be on internal power. Abort switch guards off. Finger on abort switch and only watching the abort light. An automatic would have to be good enough not to make a mistake. Or my favorite a SRB.

    • Charlie X Murphy says:
      0
      0

      SRB would be worse

  5. Neil.Verea says:
    0
    0

    While 2 accidents do not represent a trend and even though this latest accident is unrelated to the strut failure in last year’s loss from a first order view, the cause of this may be in their development process. They do pride themselves on simplicity and streamlining, could that be the cause? Could it be the very approach that allows them to access space at low cost (i.e. not testing as much as NASA does, not having all the expensive checks and balances that NASA has) that is the root cause? There is a risk trade going on here that allows Space X to fly at a significantly low cost vs NASA launch systems. These risks cannot be discovered operationally once Humans begin to fly, even for satellite launches the insurance rates may make it prohibitive to fly on SpaceX launchers if this continues. SpaceX has a slightly over 10% failure rate over the life of the Falcon 9, and ~18% failure rate over the last 11 Falcon 9s.
    They have broken through ceilings no one has dared to test before and have made bold strides in pushing the envelop for access to space. They have made it more exciting and created a buzz never seen before, however reality is reality and how they recover from this accident and hopefully move forward toward Commercial Crew Launches will be their greatest test.

    Elon has already accomplished what the “Wright Brothers” accomplished for heavier than air flight, the questions is will he go the way the WBs did in the long run? I hope not, but he is again in an uncharted area and how he comes out will set standards.

    I wish them the greatest of success.

    • fcrary says:
      0
      0

      It isn’t clear that the SpaceX approach is higher risk than the NASA (or traditional aerospace) approach. The failure rate for the Falcon 9 isn’t higher than other launch vehicles in their first couple dozen launches. I know that’s no consolation to the customers, and doesn’t help sales. But it could mean the traditional approach doesn’t really reduce risk; it’s just that the the launch vehicles involved have had plenty of experience working out the bugs.

      • Neil.Verea says:
        0
        0

        It is early to tell, only time will tell, however time is a premium in this business and SpaceX can’t wait to long while paying a standing army. This current stand down is expensive in a number of ways. What other “launch vehicles” are you using as comparison?

        • Odyssey2020 says:
          0
          0

          Yes, it’s too early to tell. F9 has had 2 failures in a little over a year but we might be sitting here 2 years from now with 10 or more successful launches, who knows?

          To me, every time SpaceX launches a successful mission is a feel good event. These 2 failures are a bummer but they make me think “yeah, this was almost certain to happen no matter what”. It’s almost more exciting to see Elon get back on his feet and fight on..

        • fcrary says:
          0
          0

          Ariane 5, for example, had two failures and two partial failures in its first 29 launch attempts. That’s slightly worse than the Falcon 9, especially since the one Falcon 9 partial failure was very partial (100% success for the primary payload and putting the secondary payload in the wrong orbit.) Interestingly, the Ariane 5 failures were on launch attempts one and 14, while the Falcon 9 failures were on numbers 19 and 29. I’m not sure what that means about teething pains and learning curves.

          The Space Shuttle had one failure in the first 29 launch attempts (with the failure on number 25), which is not significantly different from the Falcon 9 in a statistical sense.

          I was also thinking of the early Atlas, Thor/Delta and Titan launch vehicles.

      • Bernardo de la Paz says:
        0
        0

        The Falcon 9 failure rate is massively higher than the failure rate for the EELV’s (Atlas V and Delta IV) in their first couple dozen launches or in fact all of their launches to date, given that they haven’t had any failures. (Other than the partial failure of the first Delta IV Heavy demo flight in which a slightly early engine shut down left the boilerplate demo payload in a lower than intended orbit.) Given that EELV is arguably the closest peer to Falcon 9, that’s not a very inconsequential point of comparison. That is a pretty solid argument that the SpaceX approach is higher risk than their contemporaries. With 126 missions between them (127 if the recent pad failure counts as a mission), there is a pretty clear trend that EELV costs more but Falcon fails more. (Even worse if the Falcon 1 missions are included.) Customers’ choice.

        • fcrary says:
          0
          0

          I wasn’t including the EELVs, since they aren’t new designs. The first “E” in EELV stands for “Evolved.” The Atlas V design is heavily based on the Atlas III (no idea why they skipped IV), which in turn was based on the Atlas II, and so on, going back half a century. The same is true of the Delta IV. The original Atlas launch vehicle, Mercury-Atlas, had two launch failures in nine attempts, which is quite a bit worse than the Falcon 9. (And the first manned launch was on the fifth attempt, when the record was two failures in four attempts.) The original Thor-Delta’s record was a bit better, but still not impressive.

          My point is that producing a reliable system always involves an iterative process of design/test/use/identify problems/fix problems. The internal details of that process are different for SpaceX and for NASA/ULA/LMA/etc.

          Traditional launch vehicles like the Delta IV or Atlas V have also the result of many, many iterations of that process over half a century. There reliability is either the result of details of the process, the number of iterations, or both. We can’t really say which.

          The Falcon 9 uses a different process, but not necessarily an inferior one. They haven’t had nearly the number of iterations (i.e. flight experience.) In half a century, their process may have evolved the vehicle into something with the same reliability as a Delta IV or an Atlas V. We just don’t know.

          Of course, that’s no consolation to the customer’s whose payloads are lost. I’m talking about the design/accident investigation/return-to-flight process, not the cost-benefit decision of cheap versus reliable launch services.

          • Neil.Verea says:
            0
            0

            You can never really compare Atlas to Space X as its apples and oranges with all sorts of caveats. That said there is a factor that should also be considered, the one of heritage and leveraging off others failures. It could be argued that Falcon 9 is based on all that has been learned over the last 50 years of orbital launches, including manufacturing processes and materials development, while Atlas did not have that same advantage as it was in “discovery” mode.

          • fcrary says:
            0
            0

            And the Atlas V, with its spotless record, was designed with far more heritage and leverage from past work than the Falcon. Much of that heritage and leverage is proprietary information and unavailable to new launch vehicle providers.

          • Bernardo de la Paz says:
            0
            0

            I see that you were fooled by the first E in EELV every bit as badly as the Air Force intended. While it officially stands for “Evolved”, there is nothing evolved about either Atlas V or Delta IV. Both were completely new vehicles from the ground up. In fact, not just the vehicles, but the complete launch and manufacturing infrastructure. They aren’t even built in the same factories or even the same cities as their heritage namesakes and aren’t even shipped the same way. Furthermore, given the takeovers of GD Convair by LM and Douglas by MD and then Boeing, and ultimately the ULA joint venture, neither EELV was developed, manufactured, or operated by the same companies as their namesakes, nor by the same engineering teams given that EELV started generations after the original Atlas and Thor. Other than using the names, the only heritage components that come to mind are the use of elements of the payload fairing and upper stage tank structure from late model Atlas II’s on the early model Atlas V’s and the use of evolved versions of the RL-10 engine on both EELV’s. None of that reasonable constitutes “heavily based on” the design of the namesake vehicles. Certainly much less so than the heritage of the Falcon 9 from the Falcon 1 for example.

            The point remains that both EELV rockets are all new systems that have demonstrated essentially perfect reliability right from the very beginning, in contradiction with your originally statement that the Falcon 9 failure rate is not out of family with the failure rate of comparable systems at a similar level of maturity. I will grant you that the Falcon 9 failure rate is not worse than the other current American launch vehicle provider – Orbital, but, other than Antares, they generally address a different market segment.

    • fcrary says:
      0
      0

      How do two failures out of 29 attempts add up to “slightly over 10℅”? I think you are counting the very secondary payload they delivered to a lower than planned orbit, as a full mission failure. Since the customer didn’t complain and the primary payload ended up exactly where it was supposed to, this seems odd. Also, why talk about the rate over the last 11 launch attempts? Why not the last 10 or 15? The only thing special about 11 is that window gives the highest failure rate.

      • Neil.Verea says:
        0
        0

        Its considered a failure when you don’t achieve the orbit that was intended due to Launch vehicle failure. On looking at the last 11, precisely what you said its “special”, its how you determine pattern, but I’m sure you knew that.

        • fcrary says:
          0
          0

          The Falcon 9 CRS-1launch is considered a partial failure, not a failure. It delivered the primary payload (the Dragon ISS supply vehicle) to exactly the correct place and orbit. It did not get a minor, secondary payload to the intended orbit _because_ the primary payload customer (NASA) had unusually strict requirements on propellent margin and delivery accuracy. There is a very large grey area when it comes to partial launch failures. That term has been used to describe putting the primary payload in an incorrect and useless orbit. That Falcon launch was clearly not a 100℅ success, and no one has claimed it was. But it also was not the 0% success you implied. Personally, I’d call it over 75%. Unless you are deliberately trying to invent a high failure rate, to win debating point, I do not see why you would count this as a complete failure.

          In addition, I do not see any validity in talking about the failure rate for the last 11 Falcon launches. Picking the number which maximally exaggerates the statistics is not an accepted or accurate way to “determine pattern.” It is simply a way to twist statistics to justify a position. A valid and honest method would be to look at the last 10 and last 12 launch attempts, and if 11 gave a very different result, dismiss that as a statistical fluke.

  6. Bernardo de la Paz says:
    0
    0

    I am not understanding how “a large breach in the cryogenic helium system of the second stage liquid oxygen tank” can be “exonerated [from] any connection with last year’s CRS-7 mishap” in which “the helium system integrity was breached” “in the upper stage liquid oxygen tank” (http://www.spacex.com/news/….

    While it is certainly plausible based on publicly available information that the two failures may be due to different root causes and detailed chains of events, they certainly seem to be connected in the sense that both failures occurred in the same system. That in itself is potentially troubling given that this system was presumably a primary focus of the complete design scrub following the CRS-7 failure.

    • fcrary says:
      0
      0

      I guess I missed something. What “complete design scrub following the CRS-7 failure”? I didn’t see any internal details of the accident investigation’s work, just the results. A complete design scrub of the entire vehicle is something NASA would have done (and what it did after the Challenger and Columbia accidents) and probably what ULA would do. On the other hand, such a complete design scrub of an entire vehicle would be inconceivable in either civil or military aviation. I don’t think we know what SpaceX internal practices are like.

      • Bernardo de la Paz says:
        0
        0

        Complete scrubs after major failures are not uncommon – see Apollo and Shuttle. It happens in aviation too – the de Haviland Comet comes to mind. My recollection is that SpaceX said they did a complete scrub after the CRS-7 failure, but regardless, the point is they most certainly did a design scrub of the system that caused the failure, namely the upper stage LOX tank helium pressurization system. Which is the relevant point here – both catastrophic failures of Falcon 9 involved the same system, which is not necessarily an insignificant coincidence.

    • Hug Doug ✓ᵛᵉʳᶦᶠᶦᵉᵈ says:
      0
      0

      i.e. the cause was not a failed strut.

  7. Jeff Havens says:
    0
    0

    Curiosity question — how much difference is there in the pressurization systems between the first and second stages? Will a potential redesign of the He system in the second stage also necessitate a redesign of the first stage system, even though there have not been any (reported) issues there?

  8. Daniel Woodard says:
    0
    0

    On the Shuttle the LOX tank was pressurized with gaseous oxygen tapped from the engines. Although oxygen itself is heavier than helium, eliminating some or all of the COPVs might save more weight.

    “Pressurization in the tank is maintained by gaseous oxygen tapped from the three main engines and supplied to the liquid oxygen tank through the orbiter/external tank gaseous oxygen umbilical.”
    http://space.stackexchange….

    • HyperJ says:
      0
      0

      That works for the LOX tank, but is a bad idea for the RP tank. 🙂

      This is also another reason for SpaceX choosing Methane for their next vehicles, which can be autogenously pressurized without the need for Helium. No more COPVs will be needed at all.

      • Daniel Woodard says:
        0
        0

        So how will the LOX be pressurized?

        • HyperJ says:
          0
          0

          By gaseous oxygen, of course. And liquid Methane by gaseous methane.

          My point was merely that pressurizing RP with oxygen was an accident waiting to happen, but you didn’t intend that, I assume.

          • Daniel Woodard says:
            0
            0

            No, and that is a good point, although some launch vehicles apparently use the gas generator (i.e. turbopump) exhaust to pressurize the RP-1 tank.