NASA Economic Math: Simple Answers Are Hard – Fancy Equations Are Easy
Keith’s note: Three weeks ago I sent NASA Chief Scientist Ellen Stofan (and NASA HQ PAO) a simple question about her statement regarding NASA’s value to America’s economy i.e.”there was a report that showed that for ever $1.00 you spend on NASA you get $4.00 returned to the economy”. As I have been ranting for the past month Stofan – and the rest of NASA – refuses to answer a simple media inquiry about a public claim by NASA about returns on investments in NASA technology. Yet that has not stopped NASA from putting out a report today titled “Economic Development of Low Earth Orbit” that includes complicated fancy math to calculate what an investment in ISS R&D can expect to see emerging from that investment (page 46):
As you can see the math in my original question was much simpler than what is in this report (image of full reference). The report goes on to gush about the economic potential of space commerce with regard to Low Earth Orbit. That economic potential is most likely quite real. Alas NASA is not necessarily the best equipped to actually understand that commercial potential – much less act strategically to facilitate its development. NASA is also under some collective delusion that it actually understands “commerce” since they seem to think that “commerce” is equivalent to government spending. Just ask Bart.
In the mean time it is clear that one part of NASA is not talking to other. While one office on the 9th floor is incapable of responding to a simple question on this topic just a few feet down the hall NASA’s Deputy Administrator’s office is co-launching this report with the White House – including the fancy math that is over the head of NASA’s Chief Scientist.
NASA Cannot Answer A Simple, Basic Question on Its Value, earlier post
That NASA report on “Economic Development of Low Earth Orbit” must have been a fun read. But the equation you quoted is a wonderful example of saying something fairly pointless in great detail. I’ll assume the undefined variables are just your editing. That’s fine, and I’m not curious enough to look it up in the original document.
What I find odd is the assumption that return on investment can be described as a monotonically increasing, strictly concave (negative second derivative, I assume) function of the investment. That might give some analytic solution to the equations, but it doesn’t really reflect reality. There are ways to spend more on a project and generate a smaller return (an expensive, fantastically botched marketing campaign is an example.) Strict concavity isn’t true in cases where a slightly greater investment enables something new, and that new thing drastically increases revenue.
Solving complicated equations is something I do for a living, and I appreciate making assumptions which make equations solvable. But the trick is to make find assumptions which also reflect reality in some way.
http://images.spaceref.com/…
Now I really wish you hadn’t posted more of that report. Seeing someone use math in that way is professionally painful. As far as I can tell, the author is saying that relevant research can help companies optimize their investments and thereby increase their rate of return. I wouldn’t disagree.
But the author then uses some equations, based on doubtful assumptions, combined with some math (a few steps of which are poorly described and may involve additional assumptions and approximations) to show that the difference is some particular number (3/8 + sqrt(2)/4 versus 3/4.) The final line doesn’t even get the sign right. It should read “smaller” not “bigger”
The Economic Development of Low Earth Orbit is an interesting read. It looks more like something that would have been written around 1984 or 85, when Shuttle was just getting in gear, Centers for Commercial Development were starting, Challenger had not yet occurred. Prospects for protein crystal growth were great (in 1985)…protein crystals were going to be the raison d’être in 1983 when the first report was written by Harvard and PMMG, but it hasn’t happened in 35 years. Likewise the comparison of ISS-National Lab with National Labs like Los Alamos, Sandia, Brookhaven…those are major national facilities with thousands of people conducting hundreds of millions of dollars of applied research on a continuous basis. ISS is a couple people spending a couple hours a day in seriously compromised research facilities. with the B team handling integration. They’ve been waiting for years to get Kate Rubins in orbit just so they can have a real US PhD research scientist in orbit again-first time in several years.. I guess this is an effort to show there are prospects for research on ISS, but prospects were a lot better 10-15 years ago before the NASA managers shut down the already ongoing research program. They’ve now wasted a dozen years of prospective work. Any suggestion that this report will tell you how successful commercialization in orbit might be is bogus. Hype, in the same spirit as mission to Mars.
The excerpt from page 46 is interesting in that it assumes that NASA “knows” the expected rate of return and shows that on average companies will invest more if NASA “tells” them. But of course NASA has no idea what the probability distribution of financial returns is so the discussion is moot.
The section on X-ray crystallography references an NIH publication that is actually a high school textbook. Astoundingly in a chapter on industry interest in microgravity there is no evidence of any discussion with industry.
Years ago I was invited to give a lecture for a meeting of researchers from a major pharmaceutical firm. Naturally I jumped at the chance to talk about microgravity and X-ray crystallography. The audience was polite but later as we had lunch at Mila’s Road House the research department manager explained that “We don’t use X-ray diffraction anymore”. While I picked myself up off the floor he explained that with proprietary software and really powerful computers they had essentially solved the protein folding problem and could go directly from DNA sequences to 3-D structures, at least for the receptor sites, with computational chemistry alone. The speed, versatility and robustness of computational methods as compared to the trial and error of crystallography simply made the latter uncompetitive. Today I would take a more nuanced view; there are some cases where 0-G crystallization can improve the accuracy of structures that are already known. But this is mainly in the basic research realm, and basic research is pretty much only done with tax dollars, and tax dollars for biomedical research are extremely hard to get.
It would be interesting to know how much (if anything) GWU was paid for the study by NASA.
Yes. The first element in any industrial marketing strategy – talk to your market or more accurately listen to their needs and then work with them.
Boeing and Airbus understand this quite well.
The spherical chickens are coming home to roost.
Rather hilarious. I studied economics not aerospace engineering so reading and understanding that was easier then the rocket equation. As someone else said, they are taking VERY simple economic terms, e.g. law of diminishing returns, multiplier effect, etc and then using them to make a simple explanation a lot more complicated.
But without actual investment data, which they should have from the ISS, they are plugging in assumptions not based on historical data.
I agree, this report is just sad. They take very simple ideas and make them needless complex for no functional reason. For example in Chapter 3 they spend time constructing unnecessarily complex equations then instead of using them to create a model and testing it they just go into a discussion based on results from a poorly designed survey from only a handful of firms picked by CASIS instead of doing any proper industrial research. What was needed here was a well designed ladder interview with a sample that was large enough to be statistically valid. Its clear they had no clue on how to do industrial market research.
Looking at the bios of those writing it, it is understandable why it is so poorly done. NASA basically asked public policy economists and government policy experts to do what is an really a simple industrial marketing research study. That is like asking a geologist to design a telerobotic excavator on the theory that since they work with rocks they will know how to design a machine to move them. What they needed to do was select individuals with a solid knowledge of strategic management and industrial marketing not theoretical economics.
If you pick individuals with the wrong qualifications you are not going to get a good study. A Nobel Prize in Economics won for theoretical work doesn’t give someone either the skills or knowledge to do basic industrial market analysis. Its no wonder they went to writing complex equations instead of doing the simple industrial research that was needed. Its what they know how to do. They may be brilliant theoretical economists but NASA doesn’t need a special theory of LEO economic development, it needs to simply apply the basic principles already known about how to develop regional economies to build LEO industries.
I do see one appropriate use for this report. I will assign it to my graduate research students this fall for their assignment on reporting research results to decision makers. There is no shortage of problems for them to identify and discuss in it Thanks Keith 🙂