SLS Green Run Test Update: We Don't Know What Happened
Keith’s note: In summary NASA is not sure how long the engines fired. Seriously – they said that they do not know. They saw a flash near engine 4 and moments later the rocket commanded itself to shut down. They do not know the cause of the shutdown, nor whether the test needs to be run again nor whether they can ship the rocket to KSC. They do not know if a flight in 2021 is possible.
|
|
|
|
The only thing the SLS is good for is sucking money like a black hole.
As we said down on the (MI) farm, “This dawg don’t hunt…”
I only watched the test a few times & I saw it on the screen, the engines fired for 67.7 seconds. They said they wanted 250 sec minimum, but the mission is > 8 minutes. Was Full Duration a requirement? May NASA waive it?
Before the test, they said that roughly 250 seconds was the minimum time for a successful test, and that eight minutes was the goal. So the full duration was not a requirement. If the hot fire test had been ended after 500 seconds (five minutes) they could honestly have called it a success and moved one. That wouldn’t have been spin control, since it would have matched their stated, pre-launch criteria for a successful test.
But in reality, the test was terminated well before that 250 second duration. So the first SLS core clearly did not pass that acceptance test. Technically, the right people within NASA could waive the requirement for a successful Green Test hot fire. But that would be insane. I strongly suspect that they will spend a few weeks, or possibly a couple of months, studying the data and deciding what to do, and then they’ll officially announce that another test is necessary.
Short version: just another in a long series of Charlie Foxtrots
When it comes to launch vehicles, I’m going to reserve that term for the Ariane 501 launch failure. That was the first launch of an Ariane 5, and destroyed ESA’s Cluster mission (a four-spacecraft scientific mission, which was later rebuilt and flown as Cluster 2.)
The test fire stopped just as they were starting the first set of engine nozzle movements to simulate a flight profile. I never noticed any movement of the engines. There was supposed to be another set of movements towards the end of the burn, when the vehicle would be almost empty of propellants. Neither one was achieved, so. Retest.
Shutting down before the TVC test is more significant than any shortfall in seconds. SLS doesn’t use STS-style hydrazine powered APUs to vector the nozzles, but instead uses CAPUs powered by hot hydrogen gas tapped off the RS-25 low-pressure fuel turbopump turbine discharge, and they need to test that during engine operation.
They have to know what happened and why. The software would have returned error codes. The stage automatically shutdown so the conditions that cause the shutdown are known. Modes and states (and error codes). Why a hardware component failed can take a little time to figure out, but what failed is known. All of this is captured in the instrumentation (ground, flight and development). If it was a software error (and someone did predict a shutdown would occur at about 83 seconds a few years ago based on software issues), then that is very, very bad and SLS development is in far worse shape than anyone knew.
The data a spacecraft typically produces is no where that specific. Fault protection can be triggered by things like a temperature being too high, or a voltage being too low. They would know that. If not instantly, at least within an hour of inspecting the data. But that wouldn’t tell them _why_ that value was out of the acceptable range. Figuring that out could involve partially disassembling the system, going through other data for values which were within an acceptable range but changed immediately prior to the event, etc. That can take weeks.
It can be that specific. They know why. Fault Protection is tripped under specific conditions. Auto abort is triggered when a value is above or below a preset level. So yes, they know why it shut down, they may not yet know why a component failed (if hardware) but they do know what modes, states and reported error conditions occurred that aborted the engines firing. That all 4 shutdown is even more troubling. I sure hope that’s not a flight parameter (lose one and shut down all 4 – like the N-1 did – lose one, shut down 29 others ). If it was a software fault, that’s real bad because is indicates insufficient testing (at all levels), system engineering problems and IV&V of very little value. I for one hope it was hardware (and hardware that’s not hard to replace). If it was software, bring in someone from the outside (of both NASA and Boeing) to do detail static code analysis NOW now and save a future crew. I will give them credit for saving the stage. They live to fire another day!
Sometimes, if you are lucky, the cause for fault protection tripping can be very clear. And the conditions which tripped fault protection are pretty clear and specific. But usually the _reason_ they tripped is not at all specific or clear, and that’s what “know[ing] what happened and why” (as you originally wrote) is all about.
If a sensor reads a temperature outside the red limits, it would trigger fault protection, and the people involved would know that this is what tripped fault protection. That might not, as you assume, produce an error message saying “Sensor X reported a temperature above red limits”, but it’s not more than an hour’s work to go over all the data and see _which_ monitor hit red limits. That still leaves the question of why a particular sensor showed conditions outside the red limits. Which can, easily, take weeks to figure out.
I’ve seen this sort of thing happen when I worked on _Cassini_, and tracking down the causes of a anomaly isn’t easy. For example, the instrument I was working on once shut down one of its sensors, and we immediately (well, once the data was downlinked, we saw a problem and we had an hour or so to look at the data) knew it was due to the software to shut down when the sensor was in hard saturation (to prevent damage or excess wear on the detectors.) And within a day we knew that it shouldn’t have happened, since the detectors weren’t in hard saturation. It took a week or so to figure out that the fault protection software was looking at the signal from nine detectors, used the same threshold for all of them, and threshold shouldn’t have been the same because the detectors weren’t identical (actually, one should have had a different threshold at least five times higher than the others.) That’s just one example, and I’ve seen many. It’s rare for the proximate cause of for fault protection tripping to instantly reveal the actual problem.
And I’m not sure why you’re writing so much about software being to possible cause of the problem for the Green run (and placing blame on the assumption that it was the problem.) We’ve got no evidence to support or even suggest that. A more plausible idea, being tossed around in comments on Ars Technica, is a stuck valve in the thrust vector controls and gimbals. That’s one of the things they modified on those engines, one which caused problems in a previous, single engine test, and the problem with the Green run happened at about the time they started up that system. But even that’s speculation by people who haven’t seen and don’t have access to the actual data.
The error was detected around 45 seconds into the test but the engine did not shut down until almost 20 seconds later. I do not know of any component failure in a SSME that lets it keep firing for 20 or so seconds after detection without blowing up. A failure in the gimble (I would think) commands a shutdown right away. Or was it commended by the test team? That could explain the delay.
“The software would have returned error codes.”
Has Boeing software come under scrutiny lately for not always performing up to par?
Unless there was a potential for a RUD, why would all four engines shut down? Doesn’t this beast have engine-out capability? If that had happened during an actual flight, the capsule escape system would have activated. During an actual flight the SRB’s would still have been firing at this point, which would certainly complicate things!
Quite honestly, it will be interesting to see what the new administration will do about SLS, and whether Congress would allow cancellation, and replacement by SpaceX. In any event, 2024 isn’t going to happen regardless.
I’ll bet the test parameters were set so that any failure terminated the test. Continuing with three engines would not really make for a successful test. Further since this actual flight hardware, having a possible RUD wasn’t really an option. Can you imagine how far further back the planned launch would be set if an RUD occurred?
Agree. I don’t think it makes orbit on 3 engines, but I have never heard if that is an option. Is it? Of course, we are spoiled now, because Falcon 9 will continue flying with one engine out, maybe more. To answer your question at the end. If the first SLS item blows up, that will be The End.
(Note: I really tried to write this without reference to SX):
Where is the outrage? It appears that as a nation we can become accustomed to just about anything. Are we simply accustomed to it, shrugging our shoulders? Yep.
When I read about SLS “progress” I don’t know if I should laugh or cry.
Every piece of SLS-related hardware appears to these non-engineering eyes as massively over-engineered; my mind’s eye compares Stennis facilities with the rickety test stands in Texas, and elsewhere, favored by SX, where utilitarian is the order of the day. I see the massive build-up to the green run while recalling the dulling ordinariness of testing Merlin engines. Viewing the huge crawler at Kennedy, the mind wanders to Soyuz’ simple locomotive, or to SX’ even simpler methods.
The comparisons aren’t entirely fair, I realize, but the Big Picture is plain as day. The project gives righteous ammo to those carping about government excess. And they are right!
How can this Leftist regard government as an underutilized force for good when my friends on the right can point to the stunning excess of SLS?
I support the notion of ‘make work’ to the extent that it maintains an essential national, technical workforce. But can we not have some ground rules here? Perhaps a modicum of comparability with private efforts, as a first step? Or an actual work product? Finesse? (not likely).
And get off my lawn.