This is not a NASA Website. You might learn something. It's YOUR space agency. Get involved. Take it back. Make it work - for YOU.
IT/Web

SDO Website Offline Due to Hurricane That Is Not Even Here Yet (Update)

By Keith Cowing
NASA Watch
October 30, 2012
Filed under ,

Keith’s 29 Oct note: The @NASA_SDO twitter account just noted “Due to the impact of Hurricane #Sandy @NASA_Goddard the @NASA_SDO website is down. Sorry for the inconvenience We will have it up again ASAP”.
This is baffling. The most weather we have here in the DC/Baltimore area right now is steady rain. No hurricane effects are being felt yet. But none the less a GSFC website is offline? I wonder what would happen during a solar storm when the website is actually needed. Hasn’t NASA learned how to prepare for such simple contingencies i.e. placing its websites (or at least a back up mirror) in the cloud? Maybe if the SDO folks spent a little more time on routine web support and less time on their dead rubber chicken mascot this wouldn’t happen.
But wait: since NASA is incapable of having one official SDO website (due to a chronic organizational inability to adopt a simple website plan) there is another official NASA SDO website online here but it has nothing to do with this equally official SDO website here (which is offline) except that it loads images from the site that is offline right now.
Keith’s 30 Oct update: It has been 24 hours. The hurricane is gone. All of NASA’s websites seem to be working just fine – none seem to have been knocked offline – except for the SDO website at GSFC which was taken offline. The other (competeting) official NASA SDO website that relies on this downed GSFC server for images shows blank space where the “Latest SDO AIA Image” should be.
Keith’s 31 Oct update: The website is back online. What is really odd is this notice they posted:
“The SDO Website is Down Mon, 29 Oct 2012 Due to anticipated power grid problems caused by Hurricane Sandy the SDO website has been shutdown. We regret the inconvenience. The website should return tomorrow. All SDO data is sent to the ground and stored at the data centers.”
Think about this for a second. They posted this notice on 29 October on a website that no one could see on 29 or 30 October. Who did they expect to read this notice? You would have had to actually be able to see the website in order to see the notice that the website was offline. Only at NASA.

NASA Watch founder, Explorers Club Fellow, ex-NASA, Away Teams, Journalist, Space & Astrobiology, Lapsed climber.

19 responses to “SDO Website Offline Due to Hurricane That Is Not Even Here Yet (Update)”

  1. James Lundblad says:
    0
    0

    I would shut my computer systems down prior to the power failure.

    • kcowing says:
      0
      0

      No modern organization or company these days should have a single point of failure when it comes to IT and webservices. The NY Times website will be online even if the city is underwater. I wonder why ..

      • sunman42 says:
        0
        0

        Would that NASA could have that capability for things that were not (1) mission critical and (2) deemed of the highest public relations value. Thus, Hubble has redundant control centers well spaced geographically, but SDO does not. SDO does have redundant ground antennas spaced far enough apart that the same thunderstorm shouldn’t affect them both.

        The data from those antennas goes to a Joint Science and Operations Center on the west coast, and you can see current SDO AIA imagery here: http://sdowww.lmsal.com/sun… .

        We would love to have redundant capability for public-facing servers, but it ain’t currently in the budget: the cloud model doesn’t work too well for large amounts of online storage (costs double if you need a couple of Pbyte a year at each cloud location) rather than just a single server or virtual machine in more than one physical location. We are looking at cloning Helioviewer to a cloud for exactly the reasons you state.

        The machines in the room that house the SDO servers were taken down today as (1) a safety measure to prevent overheating and server/storage failure in case the HVAC went down but the facility UPS kept all the machines running, as happened post-derecho, and (2) a safety measure to prevent our sys admins from having to travel in extremely hazardous conditions to respond when the power cuts out, if it does.

        • kcowing says:
          0
          0

          This is just a bunch of excuses for a part of the agency that refuses to pay attention to how the rest of the world does IT. Another example of NASA putting its head in the sand. Why does NASA already have two independent SDO official websites – that other site costs money too and it is still online. Why have two sites with partial backup and duplicated web hosting/maintenance when you can have one site with solid back up? I’ll bet you one site would cost less and work better. But no – GSFC and HQ never cooperate on such simple things, now do they? Meanwhile the main GSFC website is still online as are lots of other projects runout of GSFC. Maybe should have lunch with the guys down the hall and see why they are still online.

          • sunman42 says:
            0
            0

            No, it’s a statement of fact. NASA, just like any other part of the government bureaucracy, turns like the Queen Mary, not on a dime. People have been proposing, and trying to build effective clouds for several years now, but not much to show for it. It’s a shame, because we clearly need cloud services for a lot of what is done in government IT, but we also don’t need it rammed down our throats when, as in the above instance, we can’t afford it, and it would be the Phase E projects who’d have to pay. As you know, their budgets are orders of magnitude smaller than those in development phases — which are always long enough ago the IT technology available in Phase E wasn’t even a dream when the design was laid down.

            And weighing the risks to human life and safety is never an excuse. It comes before p.r., it comes before “branding’ or Web presence, and it even comes before space weather awareness, which only very rarely relates to human safety in the short term. Remember, NASA has no “operational” space weather assets in the NOAA sense. We only do R&D. It’s great we can usually provide them with useful data, but its not the same as NOAA having an operational-quality data source.

          • kcowing says:
            0
            0

            You really need to talk to someone who knows how this all works – OUTSIDE of NASA. You also seem to be utterly confused as to the difference between a webserver and NASA’s monolithic mainframe mentality.

          • sunman42 says:
            0
            0

            Keith –

            We always do talk to people outside of NASA. We don’t use mainframes, and haven’t, ever. Some Webservers are different from other Webservers, though: some are simple, self-contained, and can happily live in a VM with 2 Gbyte or less of memory plus access to a few Gbyte of data. Others get enough high-bandwidth traffic to occupy a couple of dozen cores, several dozen Gbyte of memory, and require access to a databases in excess of 2 Pbyte, or at least a running window of a few hundred Tbyte of the most recent parts of that.

            Let’s just say that I have no concept of what it takes to run the content management system for your sites, and you have no concept of what it takes to run a bandwidth- and storage-limited site. Fair?

          • meekGee says:
            0
            0

            Keith – he’s making sense.
            And the public-facing side of the database is really not that important.The regular customers of the data are, but they already know where to get it.

        • kcowing says:
          0
          0

          A far as I can tell your SDO website is the *only* NASA website taken offline because of the hurricane – for whatever reason – anywhere.  You need to get some outside advice.

          • sunman42 says:
            0
            0

            We get outside advice all the time. We’re getting it now. 😉

            And no, it’s not the only site that went off the air because of the concern we have about overheating of the server room.

      • AlanL says:
        0
        0

        While servers are down SDO data available at http://sdowww.lmsal.com/sun

        • kcowing says:
          0
          0

          Yea except how are people supposed to know about that URL given that one SDO website is offline and the other site makes no mention of this link.

  2. Citizen Ken says:
    0
    0

    Not a big fan of the rubber chicken either (they stopped by ConDFW in Dallas this year to spread some sci-fact to the sc-fi fans).

    But frankly, Keith, I don’t think that you or I are their target market for the rubber chicken.  Are you familiar with the Flat Stanley phenomenon that elementary school teachers use?  I think that’s the target market, and a rubber chicken that travels is not an unreasonable choice.

    And kids are interested in their stuff.  We still have a few handouts leftover from their visit, and NSS of North Texas was distributing these at Astronomy Day at the UTA Planetarium this last weekend.  The tilt-a-picture of the solar flare is particularly impressive, especially when we point out the to-scale Earth off to the side.  A Montessori teacher requested a bunch of the Sun-observing glasses left over from the Venus transit, which came with stern admonitions to the teacher on their safe use.  We also gave out some Sun posters.

    So why don’t you cut them some slack on their EPO efforts?  At least they’re trying.

    • kcowing says:
      0
      0

      I am not going to “let up” at all. I had a teacher come up to me after the SDO dead chicken and her handler did a presentation. She told me that her young students had see another ‘presentation’ by the SDO mascot before and asked if you die when you go into space since the chicken was dead.  The teacher said that other children in her though this was the case. Not the best message to be sending to the next generation.  As for “trying” I’d rather have them use a little common sense too.

      • Citizen Ken says:
        0
        0

         Fair enough.  So what hook would you use for that audience?  What tool would you propose they use?  Surely you have something in mind?

        I use genuine fake Moon rocks as my hook for this age group, and the kids love them.  You can always tell the future geologists, because they inevitably taste the rocks as well.

        What do you use, Keith, when you’re out “inspiring” elementary school kids?

        • Citizen Ken says:
          0
          0

           P.S.  I call dibs on the ‘Roamin’ Gnomon”.

        • Citizen Ken says:
          0
          0

           How about a NASA Watch? (ba dum bum)

        • kcowing says:
          0
          0

          Using a rubberized version of dead poultry to inspire students about the sun is odd to say the least. For starters, they could use a representation that does not resemble what chickens look like just before they are rendered as food in a processing plant. As for things I have done, I have gone out of my way to post them on my websites. If you missed them then perhaps you should Google my name. FWIW I have endured frostbite, hypoxia, dysentery, parabolic flights, and centrifugation – at my own expense – to provide education and public outreach for space exploration.