Statistical analysis will be primarily based on the arbitrator-consolidated reviews. Agreement statistics, such as Cohen’s kappa,* from the primary review will also be assessed, primarily on the summary question statistics.
Statistical analysis will be primarily descriptive, reporting on the distribution of responses from the 50-100 arbitrator-consolidated assessments. Associations between the responses from different questions will be assessed by simple regression methodologies, such as ordinary least squares (OLS) and logistic regression as appropriate. All analysis will be performed in R software. Code used for analysis will be made publicly available at the time of publication.
* Editor’s note, 3/27/18: Cohen’s kappa was not reported in the final submitted publication, favoring simpler % matching categories and % within 1 category statistics for clarity.