Assessment of MCC Ops During the Power Loss Event of 7/17/97

From:
Sent: Monday, July 28, 1997 1:42 PM
To: ALGATE, ANDREW F. (JSC-DA8); AUSTIN, BRYAN P. (JSC-DA); BANTLE, JEFFREY W. (JSC-DA8); BRISCOE, ALAN L. (LEE) (JSC-DA); CASTLE, ROBERT E. (BOB) (JSC-DA8); DAVIS, SALLY P. (JSC-DA); DYE, PAUL F. (JSC-DA); FERRING, MARK J. (JSC-DA); HALE, N W., JR (WAYNE) (JSC-DA8); HAM, LINDA J. (JSC-DA8); HANLEY, JEFFREY M. (JEFF) (JSC-DA8); HILL, PAUL S. (JSC-DA8); KELSO, ROBERT M. (ROB) (JSC-DA); KIRASICH, MARK A. (JSC-DA); KWIATKOWSKI, THOMAS W. (TOM) (JSC-DA); PENNINGTON, GRANVIL A. (AL) (JSC-DA8); REEVES, WILLIAM D. (JSC-DA); SHANNON, JOHN P. (JSC-DA); SHAW, CHARLES W. (CHUCK) (JSC-DA8)
Subject: MCC-M Response
The above is a report I have drafted on the MCC-M response to the power loss incident which delayed the Mir EVA. It starts with what I think was a recoverable error (unplugging a key attitude sensor), but inaction by the ground allows the situation to deteriorate into a major problem. It is not completely clear from the information available, but it appears the ground may have even aggravated the situation with an incorrect quaternion uplink.
A classic example of the ground having their heads “up and locked”…

—————————————————————–
Assessment of MCC Ops During the Power Loss Event of 7/17/97
During the night (Moscow time ) of July 17-18, 1997, an incident occurred on Mir which caused a total loss of battery charge onboard. These events were precipitated by the crew inadvertently disconnecting a key cable onboard the Mir, which was purely crew error. However, review of the air-to-ground transcripts, discussions with the Shift Flight Directors, and information gathered from systems experts indicates at least a high priobability that the ground (TsUP) contributed to the problems, or possibly even aggravated them. No definitive conclusion can be reached from the data available, given our level of insight into Mir systems and commanding, but some observations can be made.
Observations: As we arrived at TsUP on the morning of July 18 we were mat by Shift Flight Director Victor Chadrin. With all of the NASA and MOST team present, Victor began his account of the events with “Well, the crew put us in a corner again last night!”. After a description of the cable problem and the current status of Mir, he also mentioned that the previous shift in TsUP (on console during the event) “did not do a very good job” dealing with the incident. The Shift Flight Director during the incident was Ekrem Koneev, who is very well known to us through Phase 1. He has a reputation among even the Russians as being difficult to deal with, and among the Ops Leads at TsUP and RIO’s at JSC he is considered the least cooperative PRP or SRP they interface with.
Also widely known (and evidenced in Mr. Chadrin’s opening words), is the fact that the ground team has a predisposition to attribute most problems to onboard crew error. This was clearly demonstrated in the investigation of the Progress collision and is very prominent in discussions with the ground controllers. Further, Mr. Chadrin and several others quickly pointed out that the cable activities the crew were performing were not sanctioned or scheduled by the ground.
Transcript Review: The following synopses describe significant content of the air-to-ground exchanges before and after the event.
DMT 20:20 to 21:35 (before the error)
The crew tells the ground (Vladimir Soroka, a trainer) they are reviewing for the EVA by disconnecting cables per the uplinked messages (radiograms), including leaving disconnected some cables that are not needed. The ground acknowledges this report. (That indicates the ground was at least AWARE of the crew’s activity, even if it was unscheduled). The ground discusses with Vasily the markups to the EVA procedures and his plans to act as a director during the EVA, similar to our IVA role.
DMT 21:56 to 22:10
Crew immediately reports the “situation” (indicator “lit up, and immediately went out”, followed by the staement ‘we must have disconnected the wrong cable’. They describe what they see on the computer (a control mode they are not familiar with) and indicate that they think some system (backup control system ORK?) is in hot reserve. The ground tells the crew they have no TLM yet and proceeds with other discussions as though there is no urgency. The crew continues to press, asking if they should not bring up thrusters, as a backup. The ground continues to tell the crew not to worry about it, at one time telling the crew to “take it easy”. Near the end of the pass the ground reports the “gyros look good”, then notes the “tlm has some peculiarities”, but then says they will look at the tlm again next orbit.
23:33 – ??
This comm pass is hard to decipher, and the typed transcript indicate only that discussion centers on attitude, difficulty in reading the TLM, and asking at least one crewmember to be awake at the next pass (Vasily volunteered). This next pass would start at 02:42 Moscow time, which would have been very late for the crew.
02:29 – 02:48
Crew is reading parameters to the ground from onboard, and the ground does not understand what all of it is (after the crew reads some data down, TsUP responds “OK, this may be some program for constructing ‘for the M quaternion’.”). They ask the crew to turn the Elektron off, but it has already been off since 22:30. The ground again says they will need to see the TLM next pass, and a crewmember will need to stay up.
Late in the pass the crew tells they ground that speaking for themselves, they don’t understand what the machine (Mir) is doing, and the computer is not responding to inputs as expected. The ground continues discussing things to turn off. After the crew gives several examples of data they don’t understand, the ground says they will think about it some more between passes. The crew finally tells the ground “well, we have not reconnected those cables we disconnected”. (This disagrees with an earlier statement by the crew). The ground is totally surprised and asks what cable they are talking about. This yields a discussion in which TsUP finally grasps the situation onboard.
04:03 – 04:24
Ground tells crew they still can’t figure out the data, they’ll have to wait until in the morning. The ground sends up a maneuver quaternion., but the maneuver is not going to be executed until the next pass. During the pass the ground eventually sees responses to the inputs in the telemetry and responds optimistically.
Analysis of the Comm Passes: Several things are surprising in this exchange. First, although the crew tells the ground immediately that they disconnected the wrong cable and now the vehicle is acting unusually, but it never registers with anyone on the ground. It is several passes into troubleshooting the computer messages before the ground rediscovers the source of the problem, and again only because the crew tells them.
There also appears to be a total lack of concern (or regard) for the fact that the vehicle commander does not believe the vehicle is responding properly. He cannot seem to get the ground concerned, and he is repeatedly told (in effect) to relax, don’t worry about it, we’ll look at the telemtry later. I have concern this is a direct fallout of the relations between the commander and the ground. I think there is also a contributing factor in the personality of the shift flight director, resulting in a sort of worst-case combination.
Last, even though the ground has observed this identical situation as recently as March 1997, the signature was not recognized even after 3 passes of telemetry. In the March incident an angular rate sensor failed, in this case the sensor cable was disconnected, but the effect was identical. In March, the vehicle was out of comm so no ground intervention was available, and power was lost in that incident as well.
Information not present in the transcripts: The ground never did identify that there was a discrepancy between the actual and assumed attitude onboard, until the vehicle lost all power reserves due to bad pointing. The ORT backup attitude system did not capture attitude within 12 degrees as expected and the ground did not (or could not) identify this fact. The attitude quaternion that was uplinked has been determined to have been in error, but the cause for this is unclear. In any case, it was more than 6 hours from the time of the original event to the time the ground figured out that the power situation was critical. Even then, a graceful shutdown of onboard systems was not fully achieved. The gyrodynes eventually were knocked off line due to lack of power, resulting in physical damage to several of the gyrodynes. During power recovery procedures, the recharging process was delayed because the equipment had not been powered off. When the Mir attitude was close enough to capture sunlight, the still-powered equipment consumed much of the power, resulting in little or no excess power to the batteries. Further, the crew had to catch the command panels during the day passes, because control panel power was needed just to command the equipment off.
Conclusions:
The cause of the initial incident here was crew error but there appear to be several aggravating errors afterward. Observations from the available information include:
1. There appears to be an inability on the part of the TsUP, even when TLM is available, to identify even major problems, like a loss of a major attitude sensor component or loss of nav base reference.
2. The ground does not appear to give credence to an evident state of concern on the part of the mission commander. The sense of team cohesiveness between the ground and onboard crew, to which we are accustomed, is absent.
3. TsUP situational awareness is also lacking. Although they are advising the crew to power off equipment, they evidently did not understand the severity of the power deficit nor pursue the cause for it.
Unfortunately, without much more information to evaluate, it would be impossible to derive more specific conclusions. These conclusions do contain some interpretation of the events.
Remaining work/unresolved concerns:
Assess implications for the ability of the Russian ground control team to safely operate a manned spacecraft (Mir, ISS); implications for launching subsequent Long Duration Crewmembers.
Assessment of TsUP’s objectivity in this incident, the Progress collision incident, and the STS-81 dual attitude control incident. Pattern of refusing to acknowledge data that says there is a problem, or blame crew, or both.
Assess TsUP capacity to learn from past events and incorporate into their experience base (virtually identical problem to the March attitude control incident, but still did not recognize signature or improve response time).
Impact on MCC-H strategies/responses to events during joint flights.