In “Newton, Einstein, Watson and Crick, were not peer reviewed“, Jo Nova questions whether peer review is valid at all. I think the answer is somewhat more nuanced. This is an extended version of a comment made.

Before dismissing peer review, we should ask are the boundaries of peer review. That is what peer review can achieve and what it cannot.

Proper peer review should check that the thesis of paper is original and properly references other works in the field. It should also make sure that the claims made are coherent, not demonstrably false, have a reason (or reasons) for originality, and all assumptions are clearly stated. It might also check to ensure that certain ethical boundaries are not breached. There is more basic checking, like that of an editor.

Peer review cannot determine if the following criteria are valid:-

(1) The ultimate truth. Make sure that the claims made are the last word on the subject. That is the thesis will never be falsified, contradicted, or supplanted by more general theories.

(2) The best to date. Determine that the thesis is superior to what is already available. There is a place for literature reviews to compare and contrast the existing body of knowledge.⁽ⁱ⁾

(3) That every point is correct, or every assumption known and stated.

(4) That every conjecture that the paper is built upon is correct, or every assumption is valid. Certain stated hypotheses or conjectures might be themselves based upon other conjectures. Assumptions might be accepted, but be false or exclude other, contradictory but quite valid, lines of enquiry.

(5) That a paper is hugely significant, or of little consequence.

(6) That a paper is of outstanding quality, against mediocre.

(7) That the absence of, superior, contradictory views in the academic literature is not a demonstration of the truth or quality of a research program.

Academic study is a combination of building on the work of that has gone before, whilst noticing the empirical or logical gaps and anomalies. It can be quite valid to making conjectures upon conjectures, as long as you do not lose sight that the falsification of a root conjecture will partially or completely undermine every piece of work built upon it.⁽ⁱⁱ⁾ In climatology the vast majority of papers are built upon looking at the consequences of the catastrophic warming hypothesis. Falsifying CAGW will mean entire research programs will be null and void. That includes many studies in other areas such as economics and public-policy making.

Notes

1. For instance, the Journal of Economic Literature has long-performed this service in economics.
2. Until Andrew Wiles proved Fermat’s last theorem, large areas of mathematical proofs relied upon a conjecture. Watch the video here.