Randomized Trial of the Positive Action Program in Chicago Schools and Extension to Grade 8

Overview

This project focuses on social and character development of elementary and middle school-aged children and responds to an urgent national need that schools improve their capacity to address a range of student outcomes, including social skills, character, behavior, academic achievement and health outcomes. This study is a school-based randomized trial to evaluate the Positive Action program. The Positive Action program was designed to promote social and character development and improve behavior and school performance.

This project is one of seven in a multi-site trial of different programs that has been nationally implemented. Oregon State University (OSU) and University of Illinois at Chicago (UIC) are conducting a school-based randomized trial to evaluate the efficacy of the Positive Action program (PA) to find out how the program works, to determine the effectiveness of the Positive Action program on reducing negative behaviors (including health-related behaviors), increasing positive behaviors and improving academic achievement of elementary school students. The Positive Action program was designed to promote social and character development (respect, responsibility, altruism, civic virtue, prosocial behavior) in ways that reduce anti-social behavior (violence, substance use, delinquency) and improve school performance (attendance, test scores). Fourteen eligible schools selected during winter 2004 are comprised of 7 matched pairs (treatment and control); the schools were matched on a 'risk score' composed of multiple school characteristics. Students in grade 3 in the 2004-05 school year, their parents, and their teachers and principals were surveyed at baseline (Fall 2004), spring and fall of 2005, spring 2006 and spring 2007. Evaluation is based on multiple kinds of process, mediator variable and outcome data from school records (attendance, transience, grades, test performance, disciplinary actions and suspensions, and changes in school and student population characteristics), student records, student surveys, parent surveys, teacher ratings and surveys, and administrator surveys, collected from schools in both conditions (except information about delivery of the Positive Action program). The work being done at OSU is confined to Dr. Flay's overall supervision of all aspects of the project, and data analysis using de-identified data received from Dr. DuBois at UIC and research paper writing. The work being done at UIC, directed by Dr. David DuBois, includes all of the intervention work, data collection, data entering, and some data analysis and report writing. The U.S. Department of Education/IES hired a national contractor, Mathematica Policy Research, Inc. (MPR) to conduct core surveys at all sites of the multi-site trial through spring 2007. In addition, OSU/UIC is administering a site-specific student survey that is complementary to the multi-site surveys during all waves of data collection. As the project funding followed Dr. Flay's move from UIC to OSU in September 2005, OSU IRB provides a review for the overall project. As of April 2008, new funding allows continuation of the study through March 2012 and follows the target cohort of students through the end of 8th grade as they and their teachers and principals are surveyed fall 2008, spring 2009 and again along with their parents in spring 2010. Data collection for the continuation study also includes collection of height and weight of children and process evaluation data from students and teachers. There will be no involvement of MPR. Data collection was completed June 2010. Consent Rates and Mobility: Parental consent was obtained before students, parents or teachers completed surveys when students were in grade 3. Seventy-nine percent of parents provided consent at baseline. Students joining the study at later waves were consented at that time; consent rates for them ranged from 65% to 78% for Waves 2-5. All students were re-consented for the second phase of funding at Wave 6 (beginning of grade 7); consent rates were lower at Waves 6 through 8 ( ≈ 58 to 64%). This is consistent with previous studies that have found that consent rates drop as grade levels increase. The percentages of consenting parents who provided reports on their children were 72.3%, 58.9%, 52.2%, 50.5%, and 72.9% at Waves 1, 2, 4, 5 and 8, respectively. Two factors that likely increased parent response rate at Wave 8 were (1) an increase in the financial incentive for completing the parent report and (2) an intensive period of phone outreach to families to note the incentive increase and to encourage survey completion. Percentages of consented students for whom teachers completed ratings were 74.6%, 74.8%, 72.4%, 78.3%, 74.4%, and 92.7% for Waves 1, 2, 4, 5, 7, and 8, respectively. At Wave 8, we introduced an additional school-level incentive for 100% rates of teacher survey completion, which likely resulted in the increase in completed teacher ratings. Mobility patterns were identified using results from a latent class analysis in which a 5-class solution was found to be the most appropriate fit for the data: 1) stayers (average study duration of 5.72 years, N = 158), 2) temporary participants (1.30 years, only in grades 4 or 5; N=196), 3) late joiners (1.38 years; N=308); 4) early leavers (0.94 years; N=263), and 5) late leavers (3.23 years; N=287). Planned Statistical Analyses: Because the trial was cluster-focused, we assessed students who entered schools after the beginning of the trial (joiners), but did not follow individual students who stopped attending the study schools (leavers). From the standpoint of students, across time they could be considered a "dynamic" (i.e. changing) grade cohort. Multilevel models will be used to take into account variation at the school and student levels. Missing data will be addressed using the missing-at-random (MAR) assumption, as it is unlikely that a single unmeasured variable or set of variables would predict missingness for all students who left or joined the trial schools after randomization We propose a three-level (occasions of measurement nested within students nested within schools) growth-curve model for analyzing treatment effects on various student-level outcomes. These models will account for all observations and model school differences. This approach allows for a complete analysis of the multiple waves of available data and takes into account the patterns of change over time. Random-intercept growth-curve models will first be estimated. Following the random-intercept model, a random-coefficient model will be run to test whether there is significant variation in student change across time, rather than all students in each condition having the same change pattern. A Likelihood Ratio Chi-square (LR) test will be used to compare model fit with and without the random coefficient. If a model with a random time coefficient provides a significantly better fit for a given outcome, it will be reported as the final model. Intervention effects on scales collected only at later waves (Waves 5 or 6 onwards) will be tested with the intercept set at the endpoint (Wave 8) with the condition term indicating a possible difference in effects at the last (Wave 8). Because only 14 schools are in this trial, and the PA effect is tested at the school level in a cluster-randomized trial, we will conduct several sensitivity analyses. First, we will assess the statistical significance of the PA coefficient estimate and its standard error using the t-distribution with 12 degrees of freedom: 14 schools - 1 (the condition effect) - 1 = 12 df providing for a more conservative approach. A second approach will be a pair-level analysis, estimated as a four-level model: occasions of measurement nested within students, nested within schools, nested within matched pairs. In addition to the student-level survey data, several school-level archival measures will be analyzed. Because these data are at the school level, the growth-curve models will be two-level (observations within schools) rather than three-level. Because of the small amount of data (the number of schools times the number of waves) and the resulting power limitations, these analyses will use the random-intercept model only. We will test for moderation by gender and by student mobility. The moderation tests will reveal for whom the program has its effects; that is, these tests will allow us to assess whether program effects differ by gender or a child's mobility. We will not test for moderation by ethnicity because it is highly confounded with school, with 3 pairs of schools having a mostly African-American enrollment and 2 pairs of schools having a mostly Hispanic enrollment. While all 14 schools were retained throughout the CRCT, the student population in this trial was highly mobile. Thus, it is important to test for potential moderating effects of student mobility patterns. A recent approach to analyzing mobility patterns is latent class analysis (LCA). The mobility patterns described above can then be tested as a moderator of program effects; that is, examining whether students with different mobility patterns have different program effects. Mediation analyses will allow us to examine the PA program's mechanisms of action. We will first estimate the bivariate effect of X on Y without the mediator included in the model. Then, we will simultaneously estimate the direct effect of X on Y with the mediator included in the model, as well as the mediated effect, which consists of the effect of X on M × M on Y.