Books finished in October:
(Warning: reviews are unpolished and quickly written.)

Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science (2020) by Stuart Ritchie. The title summarizes the book’s topic well.

Chapter 2 recounts some egregious cases of fraud, like Stapel and Macchiarini, who made up data and lied about patients dying and suffering, respectively. But perhaps more worrying is the likely more widespread low-level p-hacking, selective reporting of findings and various small adjustments of the analysis and data to obtain publishable results that he turns to next. I know from personal experience that these are big issues in social science, but learnt here that they exist also in the harder sciences. The unfortunate result is that we cannot trust the scientific literature.

I believe Ritchie is spot on when he identifies the perverse incentives faced by researchers as a major factor in these problems. Good publications are the key to tenure, promotions, salary increases, funding, prestige and the competition to obtain these publications have tempted many to tweak their analysis, underplay uncertainty and oversell their findings. What can be done? There is room for more honest behavior from individuals, but it is too much to hope for that that will be all that is needed.

In the final chapter, Fixing science, Ritchie highlights a 2016 meta-analysis on antidepressant drug trials that illustrates many of the preceding problems and details how publication bias, p-hacking/outcome switching, spin, and citation bias may distort the scientific literature [The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression by de Vries and others.]. Instead of focusing on a specific solution, Ritchie lists a number of possible improvements. Some of these are ok, though not attacking the more structural problems or seem to labor intensive to do so, like naming and shaming, independent investigation of misconduct, algorithms to detect fake data/images/etc., hiring on merit rather than bean counting and a review service for pre-prints. Suggestions that seem to me to have more potential are journals being more acceptive of replications and null results (but needs to be accompanied by credit given by others), more comprehensive analysis like multiverse approaches and pre-registration, ideally combined with reviews in a registered report format. However, the problems with the trustworthiness of the scientific literature are so large that we should try multiple approaches.

Highly recommended.

Ratings and previous books are in the library.