Real peer review has never been tried - Works in Progress

Peer review was a huge, expensive intervention. By one estimate, scientists collectively spend 15,000 years reviewing papers every year. It can take months or years for a paper to wind its way through the review system, which is a big chunk of time when people are trying to do things like cure cancer and stop climate change. And

... See more

The system of peer-review and publishing has worked for a long time when we were actually dependent on them for things like physical distribution of magazines, manuscripts, etc. Today, journals use the industry's reliance on peer-reviewed publications as a proxy for quality in order to coordinate a lot of unpaid labor. Authors are somewhat dependen... See more

There seems to be an open debate on whether the current system of peer-review actually weeds out papers correctly. This experiment re-submitted 12 already approved papers and 89% of the peer reviewers said the paper shouldn’t be published. More recently hundreds of sham papers managed to get through the process as scammers posed as guest editors.

Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been runn

... See more

It didn’t. In all sorts of different fields, research productivity has been flat or declining for decades, and peer review doesn’t seem to have changed that trend. New ideas are failing to displace older ones. Many peer-reviewed findings don’t replicate, and most of them may be straight-up false. When you ask scientists to rate

... See more

Left to its own devices, research tends to become more specialized and abstracted from the real-world problems that motivated it and to which it remains relevant.

This suggests that a problem may be tackled effectively not my commissioning more research, but by assuming most or all of the solution can already be found in various scientific journals,... See more