Impact of "Spin" on the Interpretation of Results of Randomized Trials in the Field of Cancer

Overview

Randomized controlled trials (RCT) are the gold standard for therapeutic evaluation. Rapid dissemination of trial results and their translation into clinical practice is particularly important. Abstracts published or presented in conferences are a large, rapid and free method to disseminate these results. However, this mode of dissemination may have serious consequences for patients if abstracts are not an accurate and unbiased reflection of the trial results. Investigators have great freedom when writing their abstracts and articles: they can choose the data and decide how to present it. Consequently, they have many opportunities to shape readers' impressions of their results, that is, to add "spin" (ie, spin is a specific way of reporting to convince readers that the beneficial effect of the experimental treatment is higher than shown by the results). Objective: To assess the impact of spin on the interpretation of results in abstracts of randomized controlled trials with non-statistically significant results in the field of cancer.

The investigators planned a RCT in which the unit of randomization is the abstract. The planning, implementation, analysis and writing of this study followed the CONSORT Statement12. Design The investigators performed a RCT comparing the interpretation of results in abstracts a) with and b) without spin. 1. Identification of abstracts of randomized trials with spin. The investigators selected a sample of negative published RCTs (i.e. non statistically significant results) with a spin in abstract conclusion identified in previous works. Inclusion criteria were 1) RCTs with non-statistically significant primary outcome, 2) spin in the abstract conclusion. 2. Abstracts' modification Selected abstracts were systematically rewritten to contain no "spin". All abstracts were presented in the same format without the names of authors, references, or the name of the journal in which it was published, and the names of treatments were masked by using generic terms. 3. Assessment Abstracts were assessed by corresponding authors of randomized trials, experts of specific grants and investigator of trials registered in http://clinicaltrials.gov. Experts were invited to participate in our study. They were not informed of the objectives of our study. Each clinician was randomized to evaluate 1 abstract with or 1 abstract without spin. 4. The primary endpoint was the interpretation of abstract results by participants. All readers participating in the study evaluated the abstract of randomized trial and answered the following question: Based on this abstract, do you think treatment A would be beneficial to patients? Randomization characteristics The randomization list was established in blocks of 30. So, each participant could evaluate 1 abstract with spin or 1 abstract with no spin. The list and the block size were not disclosed to investigators. Allocation concealment was obtained by use of a computerized randomization system. If a participant logged on the system but did not evaluate any abstract, the assessments were excluded and the abstracts were automatically allocated to the next participant. Eligibility criteria Evaluators participating in this study were as follows: 1. The "corresponding authors" of published RCTs in the field of cancer. The investigators systematically searched Medline via PubMed to identify all publications of RCTs in 2012 and if needed 2011, 2010. The investigators collected all e-mails of "corresponding authors" of these items. 2. Experts of a French national grant in the field of cancer All potential participants were invited by e-mail to participate in a study on the interpretation of results of abstracts of clinical trials. If they agreed to participate, they used an Internet link that gave them access to abstracts to assess. 3. Investigators of trials registered in http://clinicaltrials.gov. Intervention and comparators Selection of abstracts with spin We selected from previous work and personnel collection13-16, 30 abstracts of 2 parallel arms negative RCTs (ie, non-statistically significant primary outcome) evaluating treatment in the field of cancer and having spin in the abstract conclusion according to a classification developed previously13. We excluded abstract with very unusual type of spin such as a focus on another study objective, with spin related only to safety, or study comparing the same treatment but with different dosage or different mode of administration or different duration of treatment. We also excluded abstract of RCTs with small sample size (\<100). Abstract were numbered from 1 to 30 and were classified in two categories: 10 abstracts (numbers 11/12/14/16/20/22/24/27/28/30) with high level of spin (i.e., no acknowledgment of the non statistically significant primary outcome in the conclusion) and 20 with low level of spin. Development of abstracts without spin The abstracts with spin were systematically rewritten without spin. One researcher rewrote each abstract according to specific guidelines described in the box. All abstracts had the same number of words +/-25. All abstracts without spin were evaluated independently by another researcher any disagreement were discussed and the abstract was modified according to the consensus reached. Thus, for each study, the investigators had 2 versions of the abstract: 1 with spin and 1 without spin. Abstracts with and without spin were presented in the format of the original abstract with the same typography. The names of authors, references, journal name, registration number, trial name or acronym, and article title were deleted, and the treatment names and description were systematically masked with generic terms (e.g., treatment A and comparator B). If needed some information that could help identifying the treatment were deleted. Outcomes Primary outcome The primary endpoint was participants' interpretation of the abstracts. All study participants evaluated each abstract. They answered the following question on a numerical scale graded from 0 (not at all likely) to 10 (very likely) • Based on this abstract, do you think treatment A would be beneficial to patients? (answer: numerical scale from 0-10) Secondary outcomes The secondary endpoints were the assessment of the study quality, the study importance, the interest in reading the full text and the probability of being published. For each abstract, the participants answered the following: * Rate the overall rigor of the study methodology? (scale 0-10) * Rate the importance of the study (scale 0-10) * Are you interested in reading the full article for the study described in this abstract? (scale 0-10) * Do you think it would be interesting to run another trial evaluating this treatment? (scale 0-10) Blinding The primary endpoint is the interpretation of the abstract by participants. This outcome is very subjective. A recent study has shown that lack of blinding is responsible for an overestimation of treatment effect in randomized trials with subjective primary endpoints17. To minimize bias, the investigators proposed to have participants blinded to the study hypothesis. All participants were informed that they were participating in a survey on the interpretation of abstracts of clinical trials. They were not informed of the objectives and assumptions of the study before the end of the study when reporting the results. Other data collected The investigators also collected the changes to abstracts in terms of words deleted and words added to each abstract. Sample size Each participant will read 1 abstract with or 1 abstract without spin. A sample of 266 assessments of abstracts is needed to show an effect size of 0.4 with primary outcome assuming a mean difference of 1 point and a common standard deviation based on a pilot study of 2.5 with a power of 90% and an alpha risk equal to 5%. Thirty abstracts with spin and 30 abstracts without spin are available. Theoretically, each abstract must be read the same number of times according to randomization group. Considering all these elements, it will be necessary to include 300 participants (150 in each arm). Each abstract will be read 5 times in each group (abstract with low level of spin will be read 200 times (100 in each group) and abstract with high level of spin will be read 100 times (50 in each group)). Statistical analysis Statistical analysis will be undertaken independently and blindly with use of Statistical Analysis Software (SAS) v9.3 by a statistician of the Centre for Clinical Epidemiology. A statistical analytic plan will be developed and validated before searching the database. The statistical analysis plan will be revised during the study to take into account any amendments to the protocol or any other change having an impact on the statistical analysis originally planned. All versions will be kept on file for review. A T test will be used for analysis of primary and secondary outcomes. This statistical plan can be modified according to available data. For instance, if the number of readings for a particular vignette is unbalanced in groups, a multi-level model (a mixed model for clustered data) will be used to compare adjusted means for the primary outcome measures between the 2 arms. This means that statistical analysis will be adjusted for unbalanced repartition of abstracts in each group. The same model will be applied to secondary outcomes and to the subgroup abstracts with high level of spin. A secondary analysis will compare the Intraclass Correlation (ICC) between readers of abstracts with spin and abstracts without spin by computing the 95% confidence interval of the difference between ICCs (by a bootstrap method).