The myth of null-hypothesis significance testing in scientific research
Null-hypothesis significance testing (NHST) causes researchers to draw erroneous conclusions from their data. NHST is still ubiquitously used …
By Ana Marušić and Joerg Meerpohl
To inform medical decision-making, respect the altruism of trial participants and maintain public trust in clinical research, such evidence should be made available in an easily accessible and unbiased way. However, it is estimated that about half of research findings are either not published at all or only selectively. Healthcare professionals and policymakers are therefore frequently unable to make decisions based on the entire relevant research evidence.
The EU-funded project OPEN (Overcome failure to Publish nEgative fiNdings) that ran from November 2011 to October 2013 brought together academics and stakeholders from across Europe to develop evidence-informed recommendations and strategies for overcoming the failure to publish negative research findings.
The project recently published its recommendations for key stakeholders in health research to reduce the incomplete dissemination of research findings from clinical trials*. These recommendations were based on the findings of 11 work packages and the discussions during a 2-day workshop in May 2013. This workshop was attended by the OPEN project partners, researchers from another EU-funded project on publication bias – UNCOVER, and selected key stakeholders from across the world. General recommendations (Table 1) are directed to all stakeholders, while 47 specific recommendations (Table 2) target funding agencies, pharmaceutical and device industry, research institutions, researchers (systematic reviewers and trialists), research ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit assessment institutions and legislators.
The results of the OPEN project will hopefully complement and support ongoing and future activities aiming for more transparency in clinical trial results dissemination.
The recommendations from the OPEN project coincide with a number of important decisions at the European Medicines Agency (EMA) and the European Parliament, to increase the transparency of clinical trials in Europe.
Meerpohl JJ, Schell LK, Bassler D, Gallus S, Kleijnen J, Kulig M, La Vecchia C, Marušić A, Ravaud P, Reis A, Schmucker C, Strech D, Urrútia G, Wager E, Antes G; OPEN project consortium. Evidence-informed recommendations to reduce dissemination bias in clinical research: conclusions from the OPEN (Overcome failure to Publish nEgative fiNdings) project based on an international consensus meeting. BMJ Open. 2015 May 5;5(5):e006666.
|1.1||All stakeholders should raise awareness about dissemination bias and measures to reduce it (strong recommendation)|
|1.2||All stakeholders should disseminate and facilitate the implementation of targeted OPEN recommendations as outlined in table 2 (strong recommendation)|
|1.3||All stakeholders should promote trial registration and posting of results, and support initiatives that facilitate searches across multiple trial registries (strong recommendation)|
|1.4||All stakeholders should support activities to systematically, with rigorous methodology, synthesise information from studies (strong recommendation)|
|1.1||Funding agencies should include a statement on dissemination bias and the requirement for the dissemination of research results in all calls for proposals (strong recommendation)|
|1.2||Funding agencies should include the requirement for grantees to provide a dissemination plan for funded projects in all calls for proposals (strong recommendation)|
|1.3||Funding agencies should include the requirement for grantees to explicitly declare that the results of funded research will be disseminated, regardless of the nature of findings, in all funding contracts (strong recommendation)|
|1.4||Funding agencies should implement measures to ensure that the evaluation process of funded projects does not end with the project’s final report, but instead is followed up until all agreed data have been disseminated (recommendation)|
|1.5||Funding agencies should consider providing incentives for researchers who disseminate their results, or, alternatively, withhold a part of the funding until a project’s results are adequately disseminated (recommendation)|
|1.6||Funding agencies should create a publicly accessible database of all grants awarded and on how their results were disseminated in order to keep an accurate record of funded projects and publication outcomes (recommendation)|
|2.1||Pharmaceutical and medical device companies should make their policies concerning the dissemination of methods and results of clinical trials publicly accessible (strong recommendation)|
|2.2||Pharmaceutical and devices companies should register all clinical trials in a public registry before the recruitment of the first participant (strong recommendation)|
|2.3||Pharmaceutical and devices companies should make their trial protocols+amendments (as submitted to RECs) available on the publication/dissemination of results (strong recommendation)|
|2.4||Pharmaceutical and devices companies should publish/disseminate complete summary results (aggregate data) of all trials conducted and provide access to their CSRs (for clinical trials) on request (recommendation)|
|3.1||Research institutions should provide guidance and training about the implications of and possible measures for avoiding dissemination bias (strong recommendation)|
|3.2||Research institutions should not accept any funding that includes clauses that prevent the dissemination of data (strong recommendation)|
|3.3||Research institutions should mandate the dissemination of complete summary results of all clinical trials (strong recommendation)|
|4.1||Researchers conducting SRs, MAs and NMAs should follow the best practices25 38 for performing SRs (especially those practices concerning the search for trials and the assessment of the impact of dissemination bias26) (strong recommendation)|
|4.2||Systematic reviewers should make SR protocols and the results of SRs informing clinical care publicly available (strong recommendation)|
|5.1||Trialists should register every trial they plan to conduct before the recruitment of the first participant (strong recommendation)|
|5.2||Trialists should disseminate complete summary results (as soon as possible, but no later than 12 months) from all clinical trials they conduct, that is, through journal publications and results/posting of results in registers (strong recommendation)|
|5.3||Trialists should make trial protocols publicly available both within the register where the trial is registered and as appendix/supporting material with the journal publication (strong recommendation)|
|6.1||RECs should require the registration of all clinical trials before the recruitment of the first participant (strong recommendation)|
|6.2||RECs should require that applicants commit to making complete summary results publicly available (recommendation)|
|6.3||RECs should encourage applicants to share anonymised individual patient-level data on request (strong recommendation)|
|6.4||RECs should require that applicants provide annual reports describing the dissemination of their study results (strong recommendation)|
|7.1||Trial registries should enable the reporting of aggregate summary results (strong recommendation)|
|7.2||Trial registries should enable and encourage the registration of observational human studies (recommendation)|
|7.3||Trial registries outside English-speaking countries should facilitate the registration process for non-English-speaking trialists (strong recommendation)|
|7.4||Initiatives should be developed that enable non-English speakers to use the information contained in trial registers (recommendation)|
|7.5||Trial registries should enable and encourage the inclusion of links to publications and other permanent data sources (eg, PubMed, other bibliographical databases, data repositories) in trial registry entries (strong recommendation)|
|7.6||All trial registries should comply with the WHO International Standards for Clinical Trials Registries (strong recommendation)|
|8.1||Journal editors and publishers should remove all barriers to publishing negative or inconclusive studies and consider studies for publication regardless of the direction of their findings or their sources of funding (strong recommendation)|
|8.2||All journals should make trial registration a requirement for publication (strong recommendation)|
|8.3||Journals should check all submitted manuscripts against study protocols and/or trial registry entries to detect selective reporting (strong recommendation)|
|8.4||Journal editors should publish editorials and commentaries about the problem of dissemination bias and the benefits of trial registration (recommendation)|
|8.5||Journals should check for redundant publication of results by using text-matching software and asking peer reviewers about papers reporting the same findings (strong recommendation)|
|9.1||Regulation of pharmaceutical products should be extended to cover other therapeutic and diagnostic agents, such as medical devices and biologicals (strong recommendation)|
|9.2||Responsible authorities (such as the EMA for drugs) should mandate that all clinical trials in humans falling under their remit are registered in an EU database that is publicly accessible (strong recommendation)|
|9.3||Responsible authorities (such as EMA for drugs) should mandate that, on a trial’s registration in an EU database, the full protocol approved by the REC, including the potential protocol amendments, is submitted and made publicly available as a searchable document (strong recommendation)|
|9.4||Responsible authorities (such as EMA for drugs) should mandate that the full report, including all results (eg, CSR) of a trial, is made available in the same registry that the trial was registered in, in a timely fashion (ie,1 year after trial completion or inactivity) for all trials registered in the EU database (strong recommendation)|
|9.5||Responsible authorities (such as EMA for drugs) should ensure that trial sponsors failing to comply with such result submission requirements are sanctioned (strong recommendation)|
|10.1||Benefit assessment institutions should make their methods and processes of benefit assessment publicly available, in order to achieve better transparency and understanding (strong recommendation)|
|10.2||Benefit assessment institutions should aim for a higher degree of collaboration between institutions to facilitate the detection of further (unpublished) data and to foster data sharing (strong recommendation)|
|10.3||Benefit assessment institutions should use the full evidence base available for an intervention for their assessments (strong recommendation)|
|10.4||Benefit assessment institutions should specify their course of action if they find that the evidence base for an assessment is deemed incomplete (eg, no adequate proof of benefit based on incomplete data set) (strong recommendation)|
|10.5||Benefit assessment institutions should request from legislators the following items which will allow the consideration of all study results (disclosure of full protocols and full CSRs): A. A legal obligation for manufacturers to submit all requested evidence, B. Public access to EMA databases, C. Public access to protocols and full study reports, (strong recommendation)|
|11.1||Legislators should make prospective registration of clinical trials in humans mandatory (strong recommendation)|
|11.2||Legislators should ensure that all data related to the health of patients and the public are NOT commercially confidential/proprietary information (strong recommendation)|
|11.3||Legislators should institute a legal obligation for manufacturers to submit all data and other required information for the formal decision-making process (strong recommendation)|
|11.4||Legislators should ensure that the raw data (anonymised individual patient data) are made publicly available for all clinical trials (registered in the EU database) (strong recommendation)|
CSR, clinical study report; EMA, European Medicines Agency; EU, European Union; MA, meta-analysis; NMA, network meta-analysis; REC, research ethics committee; SR, systematic review.
NRIN devotes a great deal of attention to the website’s content and would greatly appreciate your suggestions of documents or links you believe to belong on this website.
This selection is an incomplete convenience sample, and does not reflect NRIN’s vision or opinion. Including an item does not necessarily mean that we agree with the authors nor does it imply we think unmentioned items are of poorer quality.
Please report any suggestions, inaccuracies or nonfunctioning hyperlinks etc. that you discover. Thanks in advance!Contact