NASA JPL Sat On A FOIA Request From A Very Rich Smart Guy. Oops.
Two years of stonewalling: What happened when a scientist filed a public records request for NASA code, Retraction Watch
“In June 2016, I filed a FOIA request with NASA and JPL for materials related to the NEOWISE project. Both NASA and JPL immediately bounced my requests. They were “unable to process” them, they said, because “it is unclear what specific records you are requesting.” Really? One of the requested categories on my list was “Documents about WISE/NEOWISE data analysis, model fitting and details thereof, including any documents on least-squares algorithms, for example the Levenberg-Marquardt algorithm or variations thereof.” That was not specific enough? Frustrated, I hired some attorneys to revise the request into a form that met all legal requirements. My lawyers submitted very lawyerly clarification letters a few weeks later. Incredibly, NASA persisted in its claim that it could not process the requests, going so far as to “close” the cases. Among other absurdities, NASA claimed that it could only search paper files and not email. They can send men to the moon but…never mind.”
NASA Response to Recent Paper on NEOWISE Asteroid Size Results, earlier post
“Examination of the paper by members of the science community studying near-Earth objects has found several fundamental errors in Myhrvold’s approach and analysis–mistakes that an independent peer review process is designed to catch. The errors in the paper lead to results that are easily refuted, such as sizes for well-known asteroids that are significantly larger or smaller than their already-verified sizes.”
Asteroid thermal modeling in the presence of reflected sunlight with an application to WISE/NEOWISE observational data, astro-ph
Lord deliver us from amateurs and cranks who have a lot of money and free time.
He may well be wrong, but that’s not the point. The problem is that the response by NASA etc is wrong in so many ways. This sort of absurd corporate behaviour serves the cause of science very badly, and has delicate a great boost to the likes of the climate change deniers who can now point to a well-documented cover-up. That’s the issue!
I disagree. Why should a small number of scientists be forced to spend their limited time and resources responding to a crank with a grudge? Published results should be replicable, sure – but not without putting in corresponding effort. Those analysis codes take years of effort to develop and the developers should be allowed to reap the fruits of their labor. Otherwise I can promise you there would be a whole herd of people ready to pounce on a new set of instrument data and scoop the developers. Do that a couple of times and pretty soon you won’t have anyone developing new instruments or codes. (Different dynamic than open source software because the value isn’t in the software but in the science that comes from it.) By contrast, making nearly raw instrument data available after a delay is reasonable.
If Myhrvold feels so strongly about this let him develop his own code and publish results, and comparisons with occultation and radar data will shake things out. Harassing JPLers with foia requests doesn’t get more science done. It’s also extremely stressful to the researchers – none of whom has the time and resources to deal with a rich crank. Think what lawyers cost per hour.
What concerns me the most is that he’s apparently bought a couple grad students with large grants to certain faculty; I think it would be wise to be worried about the objectivity of such efforts. In this case it’s an eccentric billionaire with strange ideas about NEOs. But sometimes it’ll be a pharmaceutical or chemical conglomerate with a business interest.
Where to start…
Why are you calling Myhrvold a crank with a grudge? His two papers in _Icarus_ don’t suggest that to me. What are his “strange ideas” about NEOs? As far as I can tell, he’s questing assumptions, complaining about inadequately described methodology and showing that the uncertainties are greater than the originally claimed. That’s quite normal in scientific research.
“Published results should be replicable, sure – but not without putting in corresponding effort.” I completely disagree. That’s saying they should be reproducible in theory, but making it so difficult that no one would attempt to reproduce them in practice. Having others reproduce your results is a key part of validating them. If no one tries, how can we know if the initial authors didn’t make a mistake? At the very least, I expect the methodology and everything going into the code described in the paper. If someone can’t reproduce the results, we need to be able to figure out why.
As far as code availability, I’m afraid NASA disagrees with you. For Research and Analysis grants, they are now requiring proposals to include a data management plan, which is expected to cover any software developed as part of the NASA-funded research. The authors may be spending years writing the code, but NASA is paying. It isn’t, strictly speaking, something they are spending their own time on.
Even without that, professionally, it’s good to make a technique you develop available. Some of the most widely cited papers are those where a new technique is developed and given away. Then everyone starts using it, and citing the original paper. Saying you developed the gold standard of your field counts more than getting out another paper applying the technique to a new data set. It also lets researchers build on the results of their colleagues. Keeping the methodology secret doesn’t advance the field.
it’s true that he isn’t like most cranks that plague Astro and physics department grad student lounges with letters containing their theories proving Einstein wrong. He’s trained in physics, but seems to be espousing some shall we say unorthodox solutions for diameter, reflectivity and temperatures for certain NEOs. They are allowed by the blackbody curve (two solutions possible given the data) but seem unlikely given other lines of evidence. But he’s chosen to make this personal – accusing researchers of professional misconduct, fraud etc. I read his 2016 arxiv paper (skimmed) and it was tough going; one doesn’t usually see personal attacks interspersed with derivations.
The data – magnitudes – are available on PDS. He’s written analysis code of his own; I don’t know that I can download that. What I couldn’t find in his paper was a list of his solution parameters though to be fair I didn’t look very hard after the first few pages of personal attacks and tutorials on 1-e=a.
I am puzzled by him though. I know that if I were a billionaire I’m sure I’d devote my time, fortune and legal resources to harass a small group of scientists. Hell, if he cared so much about the science he could probably fly his own space mission. That would at least advance the field. Instead he’s chosen to make the lives of the NEOWISE folks hell: foia requests (that sucks I can tell you, and is a favorite tactic of climate denialists against climate scientists), hit pieces in the media, allegations of misconduct. Attacks aimed at disrupting their mission proposals. It’s all pretty nasty stuff. And unwarranted- researchers make approximations all the time; they don’t usually refer each other to prosecutors for violating “Laws of Nature”. And researchers make mistakes and have bugs in code all the time. Science tries to catch those errors not by hostile code review so much as by independent analyses and comparison of results.
What more does he want? A full page apology in the NYT? Universal acclaim as a better scientist? A date?
“As far as code availability, I’m afraid NASA disagrees with you. For Research and Analysis grants, they are now requiring proposals to include a data management plan, which is expected to cover any software developed as part of the NASA-funded research.”
While certain program elements explicitly require code, according to the ROSES Data Management Plan FAQ (https://science.nasa.gov/re…
“What to require for code is interesting and complicated question not addressed in the NASA plan. …We fear that as with proprietary data, there are cases where forcing an organization to release it’s code would be an unreasonable burden. For now, as with other areas that we have not specified (see #7, above), we leave it up to proposers to use their best judgment and suggest in their DMP what is appropriate given the standards of their community. “
That FAQ may or may not be out of date, although I didn’t notice any mention of software in the NRA.
They are pretty good about keeping the NSPIRES web site up to date, so if you downloaded the FAQ recently it should be current. But I wouldn’t be surprised if it changed a month of two from now. I’ve been on a number of review panels over the past years, and this policy is only about four or five years old. They keep changing it and no one is entirely sure what it means.
One key item is the wording, “leave it up to proposers to use their best judgment and suggest in their DMP what is appropriate…” It means you have to provide a good reason why you don’t plan to archive code. But that justification is reviewed, and both the review panel and/or the program officers can say your justification was inadequate. And, as a proposer, you have to guess what is and is not considered adequate.
So far, since they are still trying work the details out, they haven’t been rejecting proposals over the data management plan. But they have made selections which were conditional on fixing an inadequate DMP. (As in, “try again, and we aren’t sending the money until you get it right.”)
From what I’ve seen on review panels, a good justification for not archiving code include things like: The code isn’t portable and we have no funding to make it portable; the code isn’t documented and learning to use it typically takes a new grad student months working with someone who already knows; the code is fairly simple, fully described in published papers, and the way I write code, someone else will be better off re-writing it themselves based on the published work.
What would not be acceptable is the earlier suggestion by Ben Lane. That, once written and applied to a particular data set, the author should have a monopoly on its use on all future, related data sets. If the code development is NASA-funded, that’s no longer considered appropriate.
As you point out DMP requirements change and the one that Mai11 was written under may not include posting code. It’s true that now you pretty much have to release everything; which is a mistake in my view because of screwy incentives (it encourages quick flyby papers and discourages long-term investment in software). My suggestion tho is not what you stated. I think that if you write code to analyse data that is publicly available to derive results you put in a paper then you should be free to share that code or not. Many will share, and that’s great. Where forced sharing makes sense is if there is no practical way for other scientists to replicate the work – so that’s why the WISE survey data (not the derived diameters) is public. After all, only a billionaire could try to replicate those from scratch. Oh wait…
All of the NEOWISE data is available on PDS; people can and should use them derive their own diameters if they don’t trust the ones from the team. That’s how we replicate results.
I did read Myhrvolds latest, and I was disappointed. I didn’t see any new science – it was basically a peer reviewed rant aimed at the NEOWISE team personally. Very distasteful.
Sorry guys, but this sums it up pretty accurately.
https://xkcd.com/793/
Yeah, that pretty much nails it. For some reason my previous post comes out of order making it look like I disagree with xkcd. I don’t!
Nathan Myhrvold was Stephen Hawking’s postdoc at the University of Cambridge – and I know that’s true because Nathan supervised the first few months of my Astrophysics PhD at Cambridge.
Let’s get the analysis correct regardless of who’s doing it.
Some of the best planetary astronomy is done by amateurs. Just because they aren’t paid to do the job, it doesn’t mean they’re incompetent. In another field, until the 1992 Olympics professional athletes were excluded. In the case of astronomy, some of their equipment is extremely good and their data analysis better than some professionals I can think of. (E.g. there is one amateur in Australia who does very good work with Jupiter and is careful to point out exactly when and how his image processing could introduce artifacts.) Others might just prefer to get rich, retire early, and be able to work in a field they are interested in without having to constantly write proposals for funding. I don’t know if that’s true of Mr. Myhrvold, but it’s not correct to simply equate amateurs and cranks.
Nathan Myhrvold was Stephen Hawking’s postdoc at the University of Cambridge in the Department of Applied Mathematics and Theoretical Physcis!!!
Yes, amateurs have always had a significant role in astronomy, especially for the tedious work of generating data sets by repeated observations for a large number of years that rarely generate papers directly for academic journals. Also, goven the limited number of openings for professional astronomers, that is not a bad strategy, especially if you don’t need access to really large instruments.
Well, no, this really hasn’t been very true in years. I am a professional astronomer, and I would never trust the inconsistency in amateur-gathered datasets. And in particular, we have been in an era for at least a decade where the whole sky is under constant monitoring every night, at least to the depths that amateurs can reach.
I guess we’re going to have to disagree about that. At least in terms of planetary astronomy the only constant monitoring I know of is done by amateurs. Possibly also the asteroid searches, but I don’t know if they get the full sky every night. Recent amateur work includes discovery and monitoring of the great Saturn storm of 2010-2011 (which was embarrassing for Cassini) and impact features or flashes on Jupiter.
As far as inconsistency is concerned, some are better than others. That’s also true of professional astronomers. But by sheer number, there are many duplicated (or triplicated, or whatever) amateur observations; the comparison goes a long way towards resolving inconsistencies.
Yes, they have the ability to do the routine monitoring that the professionals lack the money to do. Besides just monitoring the Solar System the American Association of Variable Star Observers has produced a data base going back over a century, mostly by amateurs. It’s a pity some professional scientists feel the need to look down on their work.
A friend of mine (retired professional software developer) has been writing software for an astronomical society that processes occultation data. He may not be a “professional” anymore, but I assure you that the code he writes meets specifications.
Yes, much valuable science has been and is being done by amateurs who are not cranks. I did not equate amateurs with cranks; to be clearer I should have written “dilettantes and cranks”.
Huh. That just seems… weird.
As much as I have read so far of Myhrvold’s paper seems well thought out and some of his critiques of the NEOWISE analysis appear reasonable. Lack of complete descriptions of methods is a common problem in science. it’s difficult to see why the NASA program would not want to release details of the NEOWISE analysis. Classically scientific findings were considered unconfirmed until other researchers came to similar conclusions. In this case the entire study is analytical as the dataset has been released, so without the complete methods such confirmation is impossible. NASA should be happy to see outsiders examining the data from other perspectives. That’s why the policy of releasing the raw data originated. Disagreements in conclusions are common in science and should not result in stonewalling.
I’d really like to see others with more of an astrophysics background look at the original paper and Myhrvold’s critique. So far I have not been able to come to a firm conclusion.
I glanced through those papers, just enough to see the point and realize the details don’t matter. Mr. Myhrvold, for example, points out that assuming a constant near-IR emissivity and a wavelength-dependent albedo is inconsistent. It is. I don’t know if his approach is any more consistent, but that doesn’t matter. His point in the first paper is that the NEOWISE team made a somewhat dubious assumption, and if you relax that assumption the uncertainty in the derived size is larger than the NEOWISE team claimed. In fact, the situation is worse than that, since the modeling involves many other, also dubious, assumptions. E.g. shape, uniformity of surface properties, uniformity of _sub_surface properties, etc. And some of those would give systematic errors rather than random uncertainty.
I agree about the reproducibility of results, and I dislike a trend I’ve seen over the past couple decades. Journals and reviewers are more and more interested in results and seem to be pushing methodology into “supplemental online material” (which is read even less often than an actual appendix.) But sometimes there are too many details and assumptions to fully describe. In those cases, I like to see different people doing the same data analysis (in this case, converting brightness in the visible and IR into radius and albedo) with the same data but with different assumptions. That’s (in effect) a Monte Carlo analysis of the sensitivity of the results to the assumptions.
As far as the reactions of the NEOWISE team, I hope I won’t offend anyone, but… Scientists do have egos. They do have careers which depend on the quality of their past work. And many aren’t eager to have past mistakes pointed out. Some of that is tied to being proud of doing good work, which is usually considered commendable. And there is a tendency to assume amateurs should automatically be dismissed.
I was hoping you would look at the papers; your reaction is the same as mine. “Peer review” used to mean that several of your peers would actually read your paper after it was published and either confirm or critique it. Journals would publish such letters and author replies. As an occasional reviewer, I feel qualified to report that reviewers and authors both make errors.
The argument that is going on in the open press is not a scandal, it is a return to real peer review. It’s unfortunate that journals themselves are reluctant to host such discussion, though some (i.e. PLOS One) provide a comment blog for all articles.
A classic requirement for a paper was that it provide sufficient information for others to repeat and confirm your work. It does not seem that it would be difficult for the original authors to release the details of their analysis as an online supplement so others could confirm or question their conclusions.
I honestly don’t think a FIOA request is the best solution to this sort of problem. It gets into all sorts of thorny issues. Asking them for information on the use of the Levenberg-Marquardt or similar algorithm, for example, assumes the people processing the request know what it is. I do a lot of chi-square minimization, and I had to look that one up. I doubt the people at NASA who process FIOA requests are familiar with relatively specialized, technical jargon. It’s also quite true that FIOA are intended for government documents, not contractor-internal and proprietary data. I don’t think that’s 100% absolute, but it does cloud the issue.
What NASA and many professional journals have started to require in recent years is open access data. In theory, NASA won’t give you a grant unless you have a data management plan which assures anyone can reproduce your results. And many journals put similar conditions on publishing you work. That has caused all sorts of confusion, what it means under various circumstances is unclear, and currently doesn’t work too well. But it probably avoids most of the legal issues an FIOA request can get entangled in.
If you go to the retraction watch website, you can see a fair amount of the back and forth – “what exactly are you looking for” a reply with more specifity, a pile of records with big redacted areas (emails with “please see attached draft”, and the attached draft redacted) , some back and forth on whether b(5) (deliberative process) exemption applies to things like drafts of reports. There was a specific comment that “FOIA is not discovery” in one of the letters.
I see it more as standard bureaucratic “grinding slow and exceedingly fine” about stuff which happened to be a hot button, with someone with deep resources on the requesting end.
I don’t see it as an evil conspiracy to hide the data – I think in a lot of cases, there’s probably significant uncertainty about the “FOIA releasability” of databases and computer code – A lot of analysis is done with software that passes internal peer review, but is not of “publication quality” – I think this will be changing in the future as more journals and the science community at large is coming to expect it to be released. There is a strong tendency to not release software- it’s somewhat of a pain procedurally, because it might have commercial value – the aftermath of dBase and Ashton-Tate probably lingers in some minds. And it’s going to be a cultural shift from “my precious hand honed software I’ve worked on for 20 years” to “sure, you can have a copy”
Everyone who writes software that’s adhoc and specific to a narrow need is apprehensive about releasing it, because of the possibility of needing to support that software. Sure, it costs almost nothing to dump it on the public server, but what about responding to the phone calls and emails about it.
So I was literally at the institution being criticized for several decades. Let’s hit the major wrong points here:
First, the resource issue. These requests are treated like they are sent to “The NASA”, and it’s full resources will somehow be used to fill the request, as thought here were some sort of office of FOIA requests at HQ with funding to handle it. In reality no resources exist to fill it, and in fact it probably is not possible to fill it at all. In this particular case, this request is being sent to a project that never had more than a half-dozen people in the first place actually at the NASA science center responsible, and by the time the FOIA request arrives it might be half a person at best. I know this for a fact. There literally are no resources to fulfill the request, and the people who might have been able to fill it are long gone. And since many of these projects have significant components that are at non-NASA institutions (almost all astronomy missions are multiple university/industry-NASA partnerships) and aren’t paid, they have zero interest in responding, and can’t be compelled to do so. Even if some of them are still at the center, you can’t just grab them for a few months to work on this, because they aren’t being paid to do that. Cost sharing like this isn’t allowed. Someone, somewhere, has to cough up the money to do it. And just going from my own fully-encumbered salary, that’s probably at least $100k that has to be found.
Here’s a simple example, one that I dealt with more than once – a FOIA request is filed, like this one, for “software”. Well, that software is probably hundreds of individual modules comprising tens of thousands or even millions of lines wrapped in miles of wrappers, and tied into custom databases and directory trees. The software isn’t portable, and probably is barely documented, at least in the way they imagine. The people who actually wrote it probably left years ago. So we tar the whole thing together and send it, then get yelled at because this isn’t what the person who filed the request imagined.
Second, there is a tendency to mistake “the project” for “the science team” or “the scientific community” that actually uses the data produced by the project. Don’t like the way the NEOWISE science team derived parameters? Guess what? All the data they used is publicly available. Go ahead and do it yourself, publish your results, and duke it out in the journals. That’s how this actually works.
The current story is a result of the two papers on the subject Myhrvold just published in _Icarus_. But he also claims he repeatedly tried to discuss this with the authors. They wouldn’t give him the time of day. Discussing things like this at conferences is also part of how things actually work. In the case of the papers he’s critical of, it’s not the broader community. What constitutes the science team and the project is a bit vague, since it seems several of the authors were both members of the mission’s science team and project management. In any case, and I know it’s fallen out of fashion, the methodology should be reported in sufficient detail to allow results to be reproduced.
Yet another comment above says “Those analysis codes take years of effort to develop and the developers should be allowed to reap the fruits of their labor.” Is the code actually useful, in which case it must be maintained, or is it no longer even understood by anyone, in which case the analysis cannot be reproduced and future researchers will have to start over? The basis of “open source”, a change in perspective that has been accepted by a large froction of the field of information technology even though it contradicts the tradition of proprietary IP, is that the best way to preserve software is to make it publicly accessable.
Looking at the original request, I can see how it would be hard to directly respond – it has the form of a generalized discovery request in a lawsuit asking for all and any records in any form on a fairly broad array of topics. The usual FOIA requests I’ve seen on the FOIA page tend to be pretty specific (“engineering drawings of Cassini”, “report D-1234”, “Software used for ABC”, things like that) So I can see the folks who received this request going “oh my, this is going to take a while to even find the stuff” and then there’s the whole “who’s going to review it for redaction of proprietary info, etc.” it’s not like there’s a magic google engine with all of NASA or JPLs content from all repositories (email, DocuShare, Sharepoint, local disk drives, etc.) that can be searched in an effective manner. There is a generalized search capability for some of that, but there are probably tens of thousands of things that have “NEOWISE” somewhere in them.
And winnowing through that huge, enormous pile is something that requires specialized knowledge and some situational decision making. Is a summary status slide at a monthly status report responsive to:
(A tiny part of the request): “
All documents discussing the accuracy of NEOWISE diameter estimates, including:
i. How the estimates were calculated;
ii. How the estimates should be described;
iii. Different approaches to calculation; and
i.v. Policies or decisions regarding the accuracy and how NEOWISE-related papers or presentations should refer to
them.
“
probably, but is the stuff about workforce and budgets in that same report responsive? Probably not – so it needs to be redacted.
Note that emails were requested – I was once involved in a civil lawsuit where we did discovery of emails from the defendant – this was in the 1990s when not as much email was used – it was a herculean ordeal to go through all those Bates-stamped pages of emails, or alternately, an ascii dump of them all. Email threads tend to be very “stream of consciousness” with all sorts of parallel information tracks – segregating out the request responsive stuff from “I’ll be in the cafeteria at 1130” is a challenge.
The people who have the background knowledge and skill set to do this are pretty busy already – it’s a significant disruption and cost – not that this a reason not to do it – it’s a legally required thing, and besides I think that Dr. Mhyrvold deserves to get the information he’s looking for. But the FOIA process does tend to get in the way of refining the request -it’s not structured (or at least it’s not being used in this case) with a iterative refinement.. “Here’s something” “That’s not quite what I want, I was more looking for this”, “Oh, let’s see if we can find that”
it’s not a simple matter of “go to library, find document, make copy” – the librarians and records people at JPL do that all the time. – Mhryvold’s request is a big enough chore that someone’s going to need to create a task for it, assign some staff and budget, find replacements for the staff that were pulled off whatever else they were doing.