Once the sample of articles has been finalized, three reviewers will be randomly assigned to review each scientific article and the associated media article. Two will serve as primary reviewers and the third person (the arbitrator) will be responsible for consolidating the two independent review forms, as described in more detail below. All reviewers will be given an anonymous reviewer ID. The reviewers will not know the identities of any of the other reviewers at any time, including after the study. All communication between them will occur through the study administrators, and the file linking the reviewers’ identifying information with their numerical ID will be encrypted. The screening arbitrator from phase 2) will serve as the primary administrator for phase 3).
Reviewers
Reviewers were recruited voluntarily through a variety of means, including peer reference and public notices. All reviewers at this phase have completed at least a master’s degree or equivalent in a relevant health and/or quantitative sciences field, with documented coursework relating to quantitative causal inference. Master’s degree equivalency includes current enrolment in a relevant doctoral degree program if the reviewer has passed his/her qualifying examinations.
All reviewing is on a purely voluntary basis with no financial incentives. The qualifications of all reviewers are confirmed by the authors of this document, and the mix of academic backgrounds will be reported in the final document. While the reviewer list with the identifies of the reviewers will be published, no information revealing which reviewers reviewed which article will be published or made available to anyone except study administrators.
Review tool
The review tool [editor’s note: review took can be found here] is based on existing review protocols and guidance, where appropriate, from the literature on systematic literature reviews and meta-analysis. The primary guidance for the development of this tool is from the Cochrane Handbook for Systematic Reviews of Interventions, most notably the GRADE approach.6 Systematic reviews and meta-analyses, by construction, answer a focused research question/set of questions, making them of limited use for audiences who do not share the specific research interest. They also typically focus on literature from randomized controlled trials, though many tools include guidance for observational studies. For this study, we seek to systematically assess individual study-level strength of evidence and language across a wide variety of subjects, focusing on studies based on observational data. This review tool is, as of the writing of this protocol, the first of its kind.
The review tool contains three major sections: 1) scientific article strength of causal evidence, 2) scientific article language assessment, 3) media article language assessment.
The section of the review tool pertaining to the scientific study strength of evidence will require the reviewers to independently identify and assess several components of the study. Reviewers will identify the main study questions, the study design, the generalizability of the sample. They will also assess issues related to selection bias and the definition of the exposure, outcome, and covariates. Finally, reviewers will evaluate the strength of the statistical methods used and provide a summary assessment of the study as a whole. The summary assessment ranks the overall quality of evidence (very high – very low). This ranking is based on several summary assessment criteria, including a rating of the quality of evidence, determination of the direction of bias, whether the study (based on the concerns outlined above) over or underestimates the true effect, and a ranking of the generalizability of the study’s findings. The language assessment reviews the language used by the author(s) and the strength of causal statements in both the media and scientific articles.
Following the review of the scientific article, the review tool has questions related to the media article with two main objectives: 1) to ascertain the extent to which the media article deviates from the findings in the scientific article; and 2) to review the language used by the media article and the extent to which the causal language matches the language used in the scientific article. Once both reviewers have filled out the extraction form, the third independent reviewer will be notified and begin the process of consolidation and reconciliation (as necessary) of the two extraction forms.
The user interface for the review tool was developed using Google Forms and is implemented as an online survey.
Prior to the start of phase 3a, a group training meeting will occur with all reviewers in order to create consensus on the review process and standardize review methods. This will include live reviewing of an article generated from the above-described 2016 dataset using the review tool. Reviewers will also have an opportunity to discuss comments on the review tool at this phase. Study administrators may consider edits to the review tool and process based on these comments, and will note any deviation or changes to the original protocol.