After the initial sample is created, a more thorough screening procedure is applied to selected media and scientific articles which match our inclusion criteria. The final outcome of this sampling procedure is a list of media/scientific article sets containing 50 items, representing the most popular news articles in terms of sharing from each Twitter and Facebook, with a confirmation of the inclusion criteria, a link to the news article, and PDF of both the media and scientific articles.
To apply this inclusion criteria, the validation procedures are carried out by two independent screening reviewers and one screening arbitrator. The screening arbitrator receives the results of the screening process from the two independent reviewers, and makes a decision for inclusion/exclusion in the case of disputes. In order to train agreement between the two reviewers before screening, the two independent reviewers will be given a training set of 20 articles generated from articles published in 2016 (thus not included in the main analysis) using the same generation procedure as above, and discuss discrepancies between their respective lists.
This review process can reject a media/scientific article pair at any level (defined below). The reviewers will assess the pair for all inclusion criteria relevant at that level. If an article is determined to fail to meet any of the inclusion criteria at a given level, a note on the reason(s) for rejection will be made, and the reviewer will move to the next article. The reviewer will not continue to assess subsequent levels. If the reviewer determines that the study either meets our inclusion criteria or that there is not enough information to reject at the given level, the reviewer will proceed to review the media article at the next level.
The order of levels of screening are below:
- Media article title
- Review for inclusion / exclusion criteria
- If not rejected, continue to media article text level
- Media article text
- Review for inclusion / exclusion criteria
- If not rejected, attempt to identify the scientific article associated with the media article
- If scientific article identified and abstract located, continue to scientific article abstract level
- Scientific article abstract
- Review for inclusion / exclusion criteria
- If not rejected, accept the article into the final review sample and continue to data extraction and storage
Each independent reviewer will proceed down the combined list of media articles, starting from the first, until each independently accepts 75 articles.
After the two primary reviewers complete their assessment of the media and scientific articles for inclusion / exclusion criteria, the screening arbitrator will reconcile the lists for items to eliminate, making the final judgement when reviewers disagree. The screening arbitrator may also reject a media/study pair in the case that the screening arbitrator finds a violation of the inclusion criteria among the articles accepted by the two independent reviewers. In the case that the list produced does not contain at least 50 articles because of a large number of rejections, the sampling procedures can be performed again to produce the required number of articles.
After reconciliation, all three reviewers will attempt to locate the original scientific journal article identified in the screening review. These will be saved as PDFs, given a randomly generated numerical identifier, named “Scientific Article <identifier>.PDF,” and stored in a shared directory. The media article will similarly be printed as a PDF, preserving the original formatting and presentation of the media article, given a numerical identifier, saved as “Media Article <identifier>.PDF”, and stored in a shared directory. If this is not feasible and/or the PDF process is not deemed sufficiently readable, the reviewer will copy the article into a word processor, format, and save as PDF.
After the screening arbitrator has completed his/her review and reconciliation process, the arbitrator will manually identify any instances where multiple media articles are found covering the same scientific article. The review set will contain all media articles found from the phase 2 review associated with that scientific article. In the case that it is discovered that a media and/or scientific article does not meet our inclusion criteria after phase 2 is complete, the screening reviewer maintains the ability to post-hoc remove the article(s) and replace it (them) with the next article(s) on the consolidated list. For this reason, the screening reviewer will not be a reviewer in phase 3), and will instead act as a study administrator.
Note that the author information in the media and scientific articles is not removed before passing on to the main review process due to two practical implementation issues surrounding the feasibility of anonymity in popular literature. The first is feasibility of anonymity, as the selected articles are very likely to be known to the reviewers beforehand, limiting reviewer availability in the case the anonymity of authors is not possible. The second is that the phase 2 screening reviewers, who are exposed to the article authors during phase 2), are selected from the phase 3) reviewer set. This study relies instead on reviewers to self-declare conflicts of interest and recuse themselves of reviewing in phase 3).