Triple-masking v Double-masking: a Trial of Scientific Publication in Public Health

Overview

The trial is designed to determine whether knowledge of the identity of the authors, their institutions and of the reviewers of a given article submitted to American Journal of Public Health impacts the final editorial decision. The concept of triple masked editorial process (where editors are unaware of author identities and affiliations in addition to the authors' and reviewers' identities being masked from one another) compared to a double masked process (only authors' and reviewers' identities are hidden from one another; editors are aware of author and reviewer identities) has not been formally tested in a scientific journal, and particularly not in medicals social science, or public health journal. Triple-masking is expected to lead to greater acceptance of articles submitted overall because it will neutralize biases against some authors and reviewers because of who they are or the institutions they are related to. The triple masked editorial process, in which the editors, the authors and the reviewers ignore their respective identities will be compared to a double masked, in which the editor knows the identity of the authors and reviewers, because double masked is currently the editorial process used by the American Journal of Public Health. Even though only manuscripts are randomized, the trial will collect information about the identity of the authors (eg, genes, race/ethnicity, seniority) and the reviewers to be used for secondary analysis. In this sense it is dealing with human subjects and has obtained an exemption from the Institutional Review Board of Queens College.

Aim: To determine whether masking (i.e., triple-masked: editors, authors, reviewers) or not (i.e., double-masked: authors, reviewers) the identity, seniority, gender, race, ethnicity, and institutions of the authors and of the reviewers to the editors leads to a greater proportion of accepted manuscripts compared with double-masked review. Additionally, we will conduct exploratory analyses to determine whether the differences are related to the identity, seniority, gender, race, ethnicity, and institutions of the authors and of the reviewers. Trial design: two group, parallel, superiority trial with 1:1 allocation Study setting: The trial will solely take place at the American Journal of Public Health in a virtual capacity. All submissions will be electronic via the Editorial Manager (https://www.editorialmanager.com/ajph/default2.aspx). Eligible manuscripts in the blinded editorial process arm will be masked electronically before being entered in the trial by the editorial assistant, who will also place all manuscripts in Dropbox folders to be accessed by the editors. The questionnaire that will be sent to all authors, reviewers, and editors is housed in Survey Monkey and will be administered via email and/or text messaging by someone external to the trial. Inclusions: All eligible, new submissions (research, notes from the field, non-commissioned editorials, analytic essays, systematic reviews) for the regular issue and supplements from September 5, 2023 to March 1, 2024. Exclusions: Commissioned editorials, book reviews, Letters to the Editor, Public Health of Consequence editorials, Editor's Choice, and re-submissions Randomization procedure: 1:1 allocation of eligible articles to either triple-masked or usual process in the order of submission, after exclusion of non-eligible submissions by the editorial assistant. Planned generation: Randomization, stratified by three article types (ie, Research Articles, Analytic Essay, Editorial \& Notes from the Field), by blocks of random sizes to balance articles by type over time, and to conceal allocation order. Administration/implementation: Editorial assistant allocates the submissions following computer generated allocation orders for each of the three article types. Masking securities: Editors will be restricted from accessing the Editorial Manager for the duration of the project. Additionally, since articles won't be assigned to editors, they won't be able to see anything from their immediate accounts. This should block them from the masked information and preserve the integrity of the project. Editors will be able to search for reviewers with specific expertise from an external, cleaned nominative version of the Editorial Manager database. They won't however know who, from the reviewers selected, accepted the invitation and wrote the review. In this sense, reviews' authorship will be masked to the editor. Procedures: 1\. Questionnaire: All authors of new, eligible submissions will receive an email from a research assistant external to the trial that will explain the on-going trial. The email will invite them to fill out an optional, short, Survey Monkey questionnaire asking for age, gender, race, ethnicity, career position, and institution location. The questionnaire data will be collected in a stand-alone database, handled by the research assistant. Missing value are expected to be equally distributed in both arms of the study. However, for secondary analysis they may be correlated with the outcome. The same questionnaire will be sent but only once to the editors. Primary outcome: Difference in the proportion of manuscripts submitted within the six-month window that are accepted for publication (i.e., not rejected at any stage); risk difference, unadjusted Secondary outcome: Difference in the proportion of manuscripts submitted within the six-month window that are sent for peer review (i.e., not desk rejected); risk difference, unadjusted Proportion accepted in the double-masked group: \~10% Clinically meaningful difference: 5% Framework: Superiority N= 1500 (750 manuscripts in each group), 80% power to detect a difference of 4.8% (RR=1.48) in acceptance rate with 5% type I error for a two-sided test. This translates to an acceptance rate of 14.8% for the triple=masked. Additionally, the trial is projected to have 84% power to detect a difference of 5% (RR=1.5), and 90% power to detect a difference of 5.5% (RR=1.55) while remaining a 5% type I error for a two-sided test. The statistical power to detect differences if they exist in subgroups defined by article type or authors identities will be very small because of the small numbers of articles. It is expected that the proportion of articles with a diverse authorship will not exceed one third of the submissions in each groups. Ethics committee The protocol has been exempted from institutional review board by the IRB of Queens College CUNY, where the editor in chief and PI of the supporting grant has his primary appointment (Appendix B).