GAO Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects, GAO
“NASA space missions are more ambitious and require the development and integration of more advanced and complex scientific instruments and vehicles than ever before. Hardware systems embedded with software challenge traditional ways of viewing and evaluating critical technology. 13 The issues include a lack of distinction among software types (newly developed, reused, and commercial-off-the-shelf), insufficient experience and knowledge when moving from the laboratory to a “relevant” environment, poor oversight during development, and inconsistent definitions of what represents new software technology. In addition, in some cases, it is no longer possible to evaluate the maturity of certain hardware technologies without their embedded software.”
“NASA introduced TRLs in the 1970s and DOD introduced TRAs in the 1990s; they have been adopted by other agencies and industry, and internationally as effective tools for facilitating understanding and increasing knowledge about the maturity of critical technologies and their readiness for integration into larger acquisition systems. Some experts, however, have argued that existing assessment tools are not well suited to addressing various areas – including software systems and systems’ integration. For example, historically, the TRL scale has not always been understood in terms of what needs to be demonstrated when it comes to software at each of the nine maturity levels, since software development did not start until the later phases of the acquisition life-cycle, such as after critical design review. New assessment tools or variations on existing tools have been developed for these areas and others.”
Glad to see GAO looking to the future, but I wonder how the real scientists and engineers around here feel about these comments, particularly the distinction between software and hardware.
It’s a good point. For hardware, the meaning of each TRL is clear (at least somewhat clear), as are the steps required to go from one level to another. For example, testing in a relevant environment, flight testing, etc. I’m not sure what the equivalent would be for software. It wouldn’t make sense to test it with the computer heated up to 40 C or cooled to -25 C. That’s not “a relevant environment” as far as software is concerned. But having some sort of standards does make sense.
JWST hardware was promised to Congress as TRL level 6 or better, but people in the program knew this was a lie for many systems. TRLs are a sad joke and just annoy the managers.
I agree with Gonzo. Another problem is that most research and development funds are for one year, are restricted to specific TRL levels, and require a commitment to increase the TRL level. That means that if you are lucky enough to get funding for a year, you are cut off unless you can get funds from a totally different source for the next year. You can’t apply for a second year of funding from the same source because 1) if you did not advance the TRL level you failed to accomplish what you claimed you would in your proposal, and 2) if you did advance the TRL level you are no longer eligible for the funding source. You have to abandon your idea, even if it is promising, and start on something else.
This happens so often it has a name in the research community; “the Valley of Death”.
The “T” in TRL speaks to the invention of ‘new technology’. This is distinct from taking existing technology , and applying it differently in a way that requires new engineering development. I put software in the ‘engineering development ‘ category. Software is software. Its not new technology. Its use in embedded systems can vary widely and possibilities are left to the imagination; however, it’s not ‘new technology development’ in that sense.
There are yearly funded programs that support maturation of technolgoy along the TRL path from 1 to 5. But when you need to jump from 5 to 6, you basically have to be embedded in a flight project that needs the technology, and has the big bucks required to jump from 5 to 6.
Sadly there are more great ideas for technology development, than there are flight projects to move them to TRL 6