Chandra Goes In And Out Of Safe Mode
NASA Chandra X-ray Observatory Update: Out Of Safe Mode
“The cause of Chandra’s safe mode on October 10 has now been understood and the Operations team has successfully returned the spacecraft to its normal pointing mode. The safe mode was caused by a glitch in one of Chandra’s gyroscopes resulting in a 3-second period of bad data that in turn led the on-board computer to calculate an incorrect value for the spacecraft momentum. The erroneous momentum indication then triggered the safe mode.”
Chandra X-ray Observatory Enters Safe Mode; Investigation Underway
“At approximately 13:55 GMT on October 10, 2018, NASA’s Chandra X-ray Observatory entered safe mode, where the telescope’s instruments are put into a safe configuration, critical hardware is swapped to back-up units, the spacecraft points so that the solar panels get maximum sunlight, and the mirrors point away from the Sun. Analysis of available data indicates the transition to safe mode was nominal, i.e., consistent with normal behavior for such an event. All systems functioned as expected and the scientific instruments are safe. The cause of the safe mode transition is currently under investigation, and we will post more information when it becomes available.”
Just another indicator of our aging satellite observatory system, and another example of NASA’s decline.
Yes, Hubble, the ISS and now Chandra all at risk, with a bad report on progress for the SLS, it’s been s bad week for NASA.
NASA is just lucky the Congress is campaigning and so may not take notice of it.
Chandra has already lasted longer than expected. It’s on Congress, not NASA, that there isn’t a replacement for it.
The same thing is true of Hubble, Opportunity, Curiosity and Kepler. They are all well past their design lifetime and currently in one sort of trouble on another. It isn’t fair, but that does look bad. It probably is just as well that congressmen have other things on their minds this month.
Don’t forget Spitzer, which is still chugging away 3x past it’s primary cryogenic mission, although telecom is getting increasingly difficult as it heads to the other side of the sun. NASA has a general funding issue in that these missions create funding liens that continue long after the primary mission, but which isn’t really held in reserve by HQ. I don’t think NASA can really tell Congress “we want a few hundred million to keep in our back pocket, which we _might_ use”. My experience was that we were in a perpetual senior review process to get continuing operations funding – as soon as one review ended, we had to start preparing for the next one.
With planetary missions, it’s a two-year cycle for mission extensions, and I’d say people starting to think about the next senior review a year in advance. So yes, it does seem like you’re always preparing for one. But I think NASA does put in budget requests for extended missions. If memory serves, the proposed budgets have a line within each program (e.g. Mars or Discovery) for unspecified extended mission funding. The senior reviews decide which mission gets what, but I don’t think the money isn’t coming out of thin air or delaying new missions.
I wonder if the design engineers and the PIs and everyone else are doing themselves a disservice when they estimate the life of these lovely machines- which uniformly exceed expectations?
Congressman: “You want us to fund a replacement for the Super Duper Star Spy, is that correct?”
Harried PI: “Yes, Sir, and one with more modern technology.”
Congressman: “But hasn’t SdSS already lasted 10 times longer than your estimates? Here’s an idea: Come back to us in, oh, 15 years, and then we’ll talk”.
As to WHY they exceed, that’s a different and probably more interesting question.
There’s also a difference between 95% confidence of survival (design life) and expected life (50% confidence).
My observation for why they often last longer than designed has to do with the interaction between subsystem requirements and real hardware. For example: the requirements require a reaction wheel with a 3-year lifetime. But you can’t actually order one like that: you either get one that lasts 6 months, or 10 years. So you have to get the one that lasts ten years. This is also why some subsystems vastly exceed their performance requirements.
There are good and bad sides to that. You are certainly right be development cycle. Flying the next generation of a satellite every five years or less has done impressive things for small satellites (e.g. CubeSats.) For big missions, order of magnitude improvements in the technology can take decades.
But there are advantages to long missions. Aside from the job security involved in a long-running project (and many people do prefer that), long missions allow people to learn how the spacecraft and the instruments work. The hardware is, in general, new and one of a kind. It takes some time to learn what sort of observations work well, and which are just a waste of time.
Some of the moons of Pluto were discovered using a trick which no one imagined when HST was first launched. In the case of Cassini, we were pretty brutal to the spacecraft’s reaction wheels during prime mission (we’re lucky the are no laws against cruelty to robots…) Going into the extended mission, we had to stop that because we didn’t want to break anything. But we’d learned enough that we could figure out ways to go easy on the wheels without dramatically reducing the quality or quantity of the science. (And one scientist got lots of observing time looking at the outer irregular satellites by finding a way to do so which was almost the same as idling the spacecraft.)
I think you can send up new missions, with new and better instruments, too often. If each one was a incremental upgrade or evolutionary improvement of the last mission, it might be different. But as it stands, I think it takes several years to really get most of the way up the learning curve. And then you want more years to actually benefit from that experience. That’s why I worry about JWST’s 5 year (baseline) to 10 year (maximum) lifetime. After five years, I’m afraid we’ll still be learning how to use it well.
Flight opportunities are also an issue. Under the current way of doing things, there is no assurance the next mission will have any relation to the previous one. I could argue that we learned enough from Spirit and Opportunity in the first five years, to make it worthwhile to retire them (that would be in 2009) and fly a pair of new and improved, but similar, rovers. But no one wants to do that, because the next Mars mission to be selected could easily end up being completely different (e.g. MAVEN or InSight.) Extending the Cassini mission as long as possible was definitely the right choice for studies of planetary rings. That field is now limited to Earth-based observations and will be for decades. So the nine years of extend mission on top of the four years of prime mission were definitely worth it.
They are past design lifetime because you scientists are very careful with your toys!
And just how is the Congress responsible for the cost over runs and delays for the JWST?
If you consider the President and Congress the most senior level of management, then they are responsible for making sure the managers one or two steps down are competent and doing a good job. If those slightly lower managers drop the ball and mismanage a major program, then the President and Congress are not totally blameless.
True, they should have pulled the funding for JWST years ago when it was clear the program was going to exceed it’s funding limits, instead of extending those limits and rewarding poor program management at NASA.
That might be extreme. Slapping on a cost cap, with a real, credible threat of cancelation for exceeding it, and telling the project to descope if that’s what it takes, has worked in the past.
Yes, except they are near to exceeding it. But I hope they are allowed to finish it as new telescopes always surprise and change our perception of the Universe with the discoveries they make which is why I except some great ones from the JWST.
The thing is there must be some way to make these flagship telescope projects go smoother, without the delays and cost over runs. Maybe it should be the subject of a conference or workshop so the lessons learned could be applied to new ventures.
I ought to re-read it, but there was a receint study by the National Academies on large strategic (i.e. flagship) missions. It found that they don’t go off track any more frequently than smaller missions. But I need to re-read it to see if they go further off track when they do (e.g. 200% versus 25%) and if the NAS study considered the greater impact an off track major mission has, compared to an off track small mission.
When it comes to conferences or workshops to study this sort of thing, I think the value depends on who attends (and who has funding to attend.) Getting together a bunch of people who all have similar views would not be all that productive.
True, it should be an invitation only event to provide a good mix of experience and diversity of views. And to encourage those attending to talk freely. Perhaps even start it with a good barnstorming session. Funding of course is always an issue.
Sounds like a mission for Skype or FaceTime.
After identifying projects that go off track, of particular interest would be finding similarities in the facts responsible. Was a new instrument designed and fabricated, one never used before, for instance? Is the overrun related to deployment, or operation, or?
There was a discussion hereabouts in the past few months about a related issue. The question revolved around the portion of a given project that involves well-understood technology, and the part that requires a new level of understanding.
This seems to me a useful way to look at new projects, and a useful tool in assessing current projects as they go over budget.
For instance: we know well how to make light-weight glass for telescopes (even space telescopes). Mounting the glass in cunning ways to facilitate after-launch expansion might be something less well understood.
And it’s possible my examples are poor. Nonetheless doesn’t every project have something entirely new? Or perhaps an existing technique is stretched to new limits? And if that’s the case perhaps the technological readiness of a project should allow for the disparity?
(Or, maybe it already does, in which case, as someone faros on SNL once said: “Nevermind”…)
Congress should have cancel or descoped JWST further after it was apparent the project can not meet either the budget cap or schedule. Instead Congress let the JWST budget balloons to over $10B if the International contributions is included when the mission is completed.
The JWST might be launched by 2021. If further delay than it might have to switch to another launcher after the Ariane 5 retires. Heck it might even go up with the folks from Hawthorne or Kent.
J-W-S-T
Eventually research institutions will just have to pay private launch companies to launch their own observatories since NASA can’t seem to get anything new up there with JWST devouring away the budget. Are there any that aren’t owned/operated by a government space agency?
Unfortunately, we’ve thought a lot about that. That might work for cubesats or smallsats. But the problem is in the funding model – universities can’t afford to invest sums this large where there is a non-zero chance of catastrophic loss (i.e. zero ROI). So you’ll see ground-based telescopes built by international consortiums that start to hit the billion-dollar level, which you might intuitively think is approaching space observatory budgets. But there is no chance of the ground-based telescope simply vanishing one day. No mater how bad things might go for it, you can always fix it, work on it, spend decades on it.
There have been some good orbital telescopes in the Small Explorer line. I think $100 to $200 million might be viable for a university consortium, even given the possibly of a failure or only a decade or so lifetime. But I think just about any university would _much_ rather propose to NASA and them to pay.
Are those generalized numbers exclusive of launcher? (I’m wondering what effect much cheaper boosters will have over the long term).
The cost cap for Small Explorers is about $150 million not including launch costs. Most of them launch on a Pegasus, so that’s $55 million (for the ICON, which should launch later this month.) A Pegasus can put about 400-500 kg in orbit, so that’s the size of the satellites we’re talking about.
Unfortunately, there aren’t too many new launch vehicles on the horizon for that size spacecraft. ESPA-class secondary payloads are 180 kg (300 kg for an ESPA grande adaptor.) Rocket Lab’s Electron is 180-200 kg and $6 million.
Not that you couldn’t fit a good orbital telescope into an ESPA-class spacecraft (and a $50 million plus launch cost price tag is consistent with some NASA calls for proposals.) But I pointed to the Small Explorer line because it’s got a long and generally good track record.
True, the Russians have been trying to fix the BTA-6 for decades.
That’s true. I think Cassini may have the record, with only two or three safings over a long mission (and one of those sort of delerabate, i.e. predicted in advance with an immediate recovery put together rather than avoiding the safing by stopping the on-spacecraft command sequence in advance.) But having multiple spacecraft safe within a few weeks isn’t a good thing. It’s probably a matter of coincidence or having more spacecraft deep in extended missions. But it definitely doesn’t make NASA look good.
No, it isn’t the end of the story, because public perception has a lot to do with future funding. It doesn’t matter whether or not that perception is an unfair result of a coincidence. There are plenty of companies which went bankrupt over nothing but bad luck, coincidences and the resulting negative PR. The same is true of Congress and their willingness to fund NASA missions; if it looks bad, fair or not, it can impact the budget.
That’s especially true not that most people get their news from electronic media. The news media has always paid attention to what people do and don’t read. Now it’s easier and the information is more detailed. That has a strong influence on what they cover and how heavily they cover it.
If a coincidence and a couple of high profile safings get people’s attention, stories about safings are more likely to show up in the future. A minor glitch which wouldn’t have made the front page (or, today, the first page of a Google search) suddenly does. Which draws more attention, and that sort of feedback can get out of hand. So, no, just saying it’s a coincidence isn’t the end of the story.