Persuade Me to Listen

Persuade Me to Listen: Math blogger Scott Aaronson, reacting to the question of whether he'd read a paper purported to be a major breakthrough nicely sums up a problem I ran into some time back:

If I read all such papers, then I wouldn't have time for anything else. It's an interesting question how you decide whether a given paper crosses the plausibility threshold or not. For me personally, the AKS "PRIMES in P" paper somehow crossed it whereas this one somehow doesn't.

Of course, I'd welcome an opinion from anyone who's actually read the paper.

Continuing, he noted that, "Three commenters wrote in to say the paper looked good. Then the author found a bug and retracted it." The mathematician then went on to discuss "Ten Signs a Claimed Mathematical Breakthrough is Wrong", but not without admitting inspiration from an amusing (but instructive) piece in Discover Magazine, "The Alternative-Science Respectability Checklist". There, author Sean Carroll makes a similar point to the mathematician I quoted above:

Scientists can't possibly pay equal attention to every conceivable hypothesis, they would literally never do anything else. Whether explicitly or not, they typically apply a Bayesian prior to the claims that are put before them. Purported breakthroughs are not all treated equally; if something runs up against their pre-existing notions of how the universe works, they are much less likely to pay it any attention. So what does it take for the truly important discoveries to get taken seriously? [link omitted]

Carroll then goes on to elaborate, in great detail, why a scientist dismissing a purported breakthrough does so, although he makes the point very sarcastically.

I'll share a passage I found particularly amusing.

Likewise, if your breakthrough is an experiment, it had better be a dramatically obvious one -- and the more you are violating cherished scientific beliefs, the more dramatic the effect had better be. If what you're claiming requires a re-arrangement of the energy levels in organic molecules, in flagrant disregard of the Schrodinger equation, you are going to need much more than a two- or three-sigma effect. And, equally importantly, you have to be up front about what the apparatus is, so that anyone can reproduce the experiment. No fair saying "Well, if you come into my lab, I'll turn it on and show you how it works." And "This experiment was done in the '70′s in a secret underground lab in Gdansk, and the KGB has suppressed the lab notebooks" isn't any better. If you're actually playing the role of a scientist, share your procedure with everyone, so that they can become true believers themselves. If, on the other hand, you just want to make money, then by all means don't tell anyone; just start producing the free energy (or amazing stretchy widgets, or whatever) and sell it on the open market. The millions of dollars that will doubtless flow your way will be very comforting as you rail against the establishment for failing to appreciate your genius.

What Aaronson and Carroll are describing (from the standpoint of someone being told of an alleged breakthrough) are heuristics, and each admits that such rules-of-thumb can cause them to miss a breakthrough from time to time. But both have a valuable points for anyone wanting to challenge an orthodoxy. (1) Actually know what the orthodoxy says and why. (2) Know why said orthodoxy is wrong and you are correct. (3) Be able to cause someone who upholds the orthodoxy, in terms they will understand, why they should listen to you.

The obviousness of these points explains (via weariness) the sarcasm in Carroll's piece: As he implies and I have often seen in my own personal experience, most people who champion something from left field have failed to do at least one of the above things, and usually, one of the first two. Only rarely is someone who has done all his homework such a poor communicator that he will only look like a crackpot. But it does happen.

-- CAV