The myth of null-hypothesis significance testing in scientific research
Null-hypothesis significance testing (NHST) causes researchers to draw erroneous conclusions from their data. NHST is still ubiquitously used …
by Gerben ter Riet
COMPARE is an initiative by Ben Goldacre. COMPARE (http://compare-trials.org/) takes the fight against outcome reporting bias one step further. COMPARE identifies new trials in NEJM, Lancet, JAMA, Annals of Internal Medicine and BMJ and compares the published results against the pre-specified outcomes in the corresponding study protocol and/or trial (pre-)registration text. They publish their results on the web and ask the pertinent journal to publish the discrepancies and, thereby, set the scientific records straight. At the time of writing (29/02/2016) the score was as follows: Trials checked (67); perfectly reported (9); pre-specified outcomes not reported (301); novel outcomes silently reported (357); letters sent (58); published (6); unpublished after 4 weeks (36); rejected (16). Unsurprisingly, the outcome “switching” is not a random process; p-values being an obnoxious driver.
The COMPARE website also features the team’s correspondence with the editorial boards. While BMJ have responded admirably, NEJM and Annals seem to have entangled themselves in debates that make one worry about their true scientific intentions. As always, the devil is in the details. What exactly is a “pre-specified outcome”? COMPARE uses a very strict definition, arguing that “protocols published after the start of a trial, by definition, cannot contain pre-specified outcomes.” This raises the question of what the “pre” refers to: Before the first patient entered, or before the statistical analysis started or something else. Make up your own mind and enter a reply on the COMPARE website. An excellent reply suggested that COMPARE should publish their full results on PubPeer and PubMed Commons, guaranteeing permanent links between the published trials and COMPARE’s findings, whether favourable or not! Why wait until stubborn editors change their ways?
NRIN devotes a great deal of attention to the website’s content and would greatly appreciate your suggestions of documents or links you believe to belong on this website.
This selection is an incomplete convenience sample, and does not reflect NRIN’s vision or opinion. Including an item does not necessarily mean that we agree with the authors nor does it imply we think unmentioned items are of poorer quality.
Please report any suggestions, inaccuracies or nonfunctioning hyperlinks etc. that you discover. Thanks in advance!Contact