OIG: NASA’s High-End Computing Is Disjointed, Oversubscribed, and Overburdened

Keith’s note: According to the NASA OIG Report Audit of NASA’s High-End Computing Capabilities: Despite a history of innovation in HEC, NASA needs a renewed commitment and sustained leadership attention to reinvigorate its HEC efforts. Without key changes, the Agency’s HEC is likely to constrain future mission priorities and goals. NASA’s HEC is not managed as a program or centralized Agency strategic service; instead, resources are managed within the Earth Science Research Program within SMD, and this organizational placement hinders NASA’s HEC efforts. One scientist within that Program is responsible for HEC capabilities at both HEC facilities, in addition to their Earth science responsibilities. This disjointed organization and management of HEC resources exacerbates several issues, including oversight, monitoring, and the foreign national accreditation access process.

Although NASA has identified HEC as a capability portfolio, we found that key guiding documents and frameworks are absent, such as a management plan—an agreement detailing how the portfolio will be managed—and a commitment agreement designed to engage all relevant stakeholders and identify HEC as a strategic activity. Furthermore, while the Office of the Chief Information Officer (OCIO) has some oversight of HEC, it is not directly engaged in HEC activities or governance. Lastly, NASA is not keeping up with technological developments and advanced research computing requirements, in part due to these organizational and funding constraints.

NASA’s HEC resources are oversubscribed and overburdened—in other words, Mission Directorates are requesting more computing time than existing capacity can provide. This scarcity drives schedule delays and often leads to NASA teams purchasing their own HEC resources to meet deadlines. For example, the Space Launch System team invests about $250,000 annually to purchase and locally manage their own HEC clusters rather than waiting for existing HEC resource availability.

Agency officials told us that, except for Goddard Space Flight Center and Stennis Space Center, there are independent HEC assets installed at almost every NASA Center. NASA also lacks a comprehensive strategy for when to use HEC assets on the premises versus when to utilize cloud computing options—or a widespread understanding of the cost implications for each choice. Stakeholders told us that while they know NASA has HEC cloud computing options, they were hesitant to use them due to unknown scheduling practices or assumed higher costs.

NASA’s decentralized HEC management also raises cybersecurity concerns. In addition to teams building their own HEC assets, OCIO-mandated cybersecurity controls are sometimes ignored or bypassed by Mission Directorates that view them as too stringent. OCIO’s limited involvement with HEC system management can also result in duplicate spending (such as for software that OCIO already licenses) and difficulty in identifying and controlling access to HEC systems that are not included in OCIO’s asset inventory tools. We also identified extensive use of NASA’s HEC assets by external and foreign national parties without adequate user activity monitoring or a review process by security personal for gaining access to HEC systems.

Finally, we found that individual Center HEC asset users are not steadily monitored, and there are no evaluations conducted to verify rights and accesses granted to international partners. Without an integrated HEC strategy and a more focused, security management approach, the Agency’s trailblazing science and technology research will continue to be unnecessarily limited by NASA’s disjointed HEC efforts.