Rich Smart Guy Challenges NEOWISE Data (Update)
Asteroid thermal modeling in the presence of reflected sunlight with an application to WISE/NEOWISE observational data, arxiv.org (Revised version)
“As a result, the new analysis finds asteroid diameter and other physical properties that have large differences from published NEOWISE results, with greatly increased error estimates. NEOWISE results have a claimed ±10% accuracy for diameter estimates, but this is unsupported by any calculations and undermined by irregularities in the NEOWISE results.”
NASA Response to Recent Paper on NEOWISE Asteroid Size Results, NASA
“Examination of the paper by members of the science community studying near-Earth objects has found several fundamental errors in Myhrvold’s approach and analysis- mistakes that an independent peer review process is designed to catch. The errors in the paper lead to results that are easily refuted, such as sizes for well-known asteroids that are significantly larger or smaller than their already-verified sizes. While critique and re-examination of published results are essential to the scientific process, it is important that any paper undergo peer review by an independent journal before it can be seriously considered. This completes a necessary step to ensure science results are independently validated, reproducible, and of value to the science community.”
Billionaire technologist accuses NASA asteroid mission of bad statistics, Science
“Myhrvold retorts that he is fixing the errors, which he says are cosmetic and do not alter the thrust of his criticism. He says the NEOWISE scientists are defensive because many are involved in a proposal for a future asteroid-hunting telescope called NEOCam, one of five finalists in NASA’s Discovery program. “They’re up for this NEOCam thing and they’re afraid it looks bad. And it does look bad,” he says.”
Repurposed NEOWISE Spacecraft Observes and Discovers Asteroids, earlier post
Wow. Wonder why Science is echoing such serious charges from a paper that was posted on a web-site without review.
And why would any author want to post their paper before getting feedback from a peer review and answering objections? If he’s right, referee input would result in an even stronger paper.
Something sounds weird.
I’ve spent 10’s of hours refereeing two papers that ended up being recommended for rejection as unfixable due to systematic errors and the author just not cognitively understanding the subject.
In each case, the journal agreed with the analysis and data I provided, but each paper was then posted by the author to arxiv.org anyway, and merrily assimilated by whoever pays attention to things posted there (Science magazine?). I guess peer review is now “optional”. Great motivation to spend lots of time on it.
Curious how Myhrvold is quick to link what he’s doing to the NEOCam Discovery proposal currently in selection …
Myhrvold’s PR people reached out to me to make certain that I saw the paper – add in the lack of peer review in a refereed journal – there’s more than professional scientific discourse at work here.
People have been circulating unreviewed preprints since 1991 via arXiv and its predecessor, Paul Ginsparg’s xxx.lanl.gov preprint server, run on a 286 computer under a desk at Los Alamos (see below).
The problem comes with taking a preprint as seriously as a peer-reviewed paper. ArXiv does have moderators whose job seems to be to recategorize preprints by subject area (if needed) or to reject nutjob papers submitted.
How Myrhvold’s submission fared under that sort-of-scrutiny is unknown, but this case does put pressure on the system’s weak point — the lack of review.
In any event, if he submits it to a real journal, it’ll get (I hope!) real review and flaws will be dealt with. Whether the paper survives that review is, I guess, up to the reviewers and Myrhvold.
Myhrvold is a bright guy, but, even with all the pitfalls of peer review (particularly for physical science papers at Nature), if it’s not in the peer-reviewed literature, it should be ignored. Simply being wealthy and smart does not buy you a free pass to the scientific discourse table.
Well then, what DOES? Scientific discourse is so essentially egalitarian that anybody can participate. As reputations accrue, others learn soon enough who is reliable and who not.
But the door is necessarily open to all. And this is the great thing about arxiv. Yes, it has many warts. But the stranglehold on discourse by profit-making entities is appalling.
This entire episode will fall to the gods of scientific truth.
He has published it now, but his argument is that he shouldn’t have
had to publish a paper on it to be taken seriously when is is trying to expose scientific misconduct. For new scientific ideas, he would say peer review should be required, but not for criticisms of existing papers.
I need to go through it in detail, but at first glance, I don’t see anything terribly wrong with Mr. Myhrvold’s paper.
In terms of professional standards, it has been submitted to Icarus, a peer-reviewed journal which is quite relevant to the paper’s topic. I don’t really like the current practice of posting papers to arxiv.org before they have been reviewed and accepted. But this is becoming a common practice among scientists, and isn’t really a reason to criticize Mr. Myhrvold in particular.
In terms of technical details, the way to convert visible and IR brightness into an asteroid’s size, albedo, thermal properties, etc. is nontrivial. It involves many assumptions, and he is correct to point out the uncertainties introduced by making those assumptions. This differs from the usual, textbook methods for calculating uncertainty, which are based on the idea that the analysis process is mathematically correct (i.e. no assumptions involved) and simply propagates the measurement uncertainties to the uncertainties in derived quantities. The textbook approach does not cover “If you had assumed X instead of Y the result would be different by Z%.” So there is always some debate when issues like this come up. Sometimes harsh words are involved, since these considerations always increase the uncertainty, and people often prefer to present precise results.
The professional literature also isn’t a marvel of clarity. Issues like radius versus diameter are easy for people outside the field to mix up. The literature for everything from the size of asteroids to sub-micron particles frequently talks about “500 meter objects” or “2 micron particles” without every bothering to say whether that’s a radius, diameter or whatever (and, since most of these objects aren’t round, it’s usually whatever.) I am not saying that the professionals are being intentionally obscure. But they do have a tendency to skip over many details, because “everyone knows” what they mean.
It’s been a long time since I saw a paper that really followed the old standard of reproducibility. That is, describing the methodology in sufficient detail that a reader could duplicate the results. So the NeoWISE team can’t really complain if Mr. Myhrvold tried and failed to reproduce their results, and wants to publish a paper saying so.
Thank you. Everybody else’s job just looks SO easy, doesn’t it? Not true. Example: why diameter wouldn’t be 2xR in the case of space objects escapes me. People who need to know this stuff, do.
And I’d wager that reproducing somebody else’s study happens very infrequently, unless there’s glory in the disproof (arsenic life, anyone?)
Michael, in “why diameter wouldn’t be 2xR”, I think fcrary is referring to the fact that most asteroids are perfect spheres, they’re blobby lumps. You can approximate may of them as ellipsoids with major and minor radius or diameter, but that’s still a simplification. The main point is that Myhrvold’s error was pointed out to him, he changed the equation, but not the graphical results, and isn’t sharing his calculation methods with others, so people can’t examine why his results still have the error.
As pointed out by David Herald, one of the major players in the asteroid occultation timing community (a worldwide consortium of observers who participate in measuring the sizes, shapes, and orbital paths of asteroids), on page 72 of Dr. Myhrvold’s paper he shows how big of a discrepancy there is between NASA’s estimates of their sizes of three asteroids and his own:
Asteroid NASA Myhrvold
(208) Lacrimosa 45.0km 146.5km
(306) Unitas 51.6km 83.8km
(757) Portlandia 36.7km 6.6km
As Herald notes, Dr. Myhrvold somehow overlooks the fact that all three of these asteroids’ diameters have been independently measured via the stellar occultation method by members of the International Occultation Timing Association (IOTA), giving the following results:
Asteroid IOTA
(208) Lacrimosa 48 x 42km
(306) Unitas 61 x 44km
(757) Portlandia 39 x 34km
As is obvious, the NEOWISE (NASA) estimates are in good agreement with the occultation measurements, and Dr. Myhrvold’s numbers aren’t even close. I don’t think any “cosmetic” fixes are going to change that.
Looking at Mr Myhrvold’s paper, I’m not sure if your criticism is relevant. In cases where we have additional data, the NeoWISE numbers look good. Mr Myhrvold seems be concerned with the much larger number of asteroids where we have no additional data. How certain are the results in those cases? Are the techniques, which works well for asteroids with additional data, accurate for asteroids in general? That might seem right. But the additional data comes from a rare and potentiality atypical subset of asteroids. How do you properly include that in the published uncertainty in the measurements? That’s a difficult problem, and showing that the NeoWISE got it right on three occasions isn’t proof that their approach is perfect.
One would think that if Myhrvold had a good case, he would have chosen examples where independent evidence didn’t contradict his conclusions. The fact that he seems to be ignorant of such contradictory evidence doesn’t speak well for his diligence in considering ways that he may be in error.
I’m still working through the paper, but I get the feeling he considered the independent evidence either irrelevant or actually problematic. Specifically, that using an atypical subset of the data (those cases where independent evidence exists) may bias the analysis and results. By that logic, you shouldn’t chose examples which reflect biased sampling. I’m not sure I agree, but the idea isn’t absurd.
When someone strikes out three times in a row, maybe the batting skills should be questioned.
I was disappointed in the headline from Science- of all places!- using the word ‘accuses’. And further the characterization of the person responsible enough to actually look at the data attempting to understand or indeed replicate the results, describing him as a dilettante. The man is far from it.
That’s what science is about. Shame on ‘Science’ and kudos to Mr. Myhrvold. Scientific research at its very finest.