Design and Analysis of Group-Randomized Trials

We are interested in building the capacity of social scientists, foundation leaders, and public officials to carry out credible evaluation studies of group-based interventions. We have funded a team of researchers including Stephen Raudenbush (University of Chicago), Howard Bloom (MDRC), Jessaca Spybrook (Western Michigan University, and Andres Martinez (University of Michigan) to conduct this capacity-building work. Their suggested readings are below. The team also engineered and continues to update Optimal Design software to assist researchers.

Please note: In 2014, the William T. Grant Foundation retired our 10-year initiative on understanding youth social settings. In its place, we launched a new research initiative on understanding the programs, policies, and practices that reduce inequality in youth outcomes. The resources developed under the social settings initiative remain highly relevant, however, as settings are a critical lens for examining inequality.
  • A Conceptual Framework for Studying the Sources of Variation in Program Effects 2013, June.
    This working paper, by MDRC, was written by Michael Weiss, Howard Bloom, and Thomas Brock. It brings together new thinking about how to get more useful information from impact evaluations and is designed to help researchers capture data that will lead to stronger insights about why interventions succeed or fail.
  • When Is the Story in the Subgroups? Strategies for Interpreting and Reporting Intervention Effects on Subgroups 2010, November.
    This MDRC working paper by Howard Bloom and Charles Michalopoulos examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a sample study. The paper considers how and why subgroups are important for applied research; alternative ways to define subgroups; different research questions that motivate subgroup analyses, and the importance of pre-specifying subgroups prior to analysis. It also explicitly examines the conditions under which subgroup findings might be considered confirmatory evidence for decision making and conditions under which subgroups finds should be considered only exploratory and thus a basis for testing future hypotheses. The paper has been accepted for publication in Prevention Science.
  • Designing and Analyzing Studies That Randomize Schools to Estimate Intervention Effects on Student Academic Outcomes Without Classroom-Level Information 2011, April.
    This MDRC working paper written by Pei Zhu, Robin Jacob, Howard Bloom and Zeyu Xu provides practical guidance for researchers who are designing and analyzing studies that randomize schools to estimate the impacts of educational interventions on student academic outcomes without information about how students are clustered within classrooms. The paper demonstrates empirically that when schools are randomized and information about clustering at the classroom level does not exist, accounting for the clustering of students within schools is enough to produce valid estimate of intervention effects and their statistical significance. The paper has been accepted for publication in Educational Evaluation and Policy Analysis.
  • New Empirical Evidence for the Design of Group Randomized Trials in Education 2009, December
    This MDRC working paper, written by Robin Jacob, Pei Zhu, and Howard S. Bloom, offers guidance for designing group-randomized studies to measure the impacts of educational interventions. Specifically, the authors provide new empirical information about the values of parameters that influence the precision of impact estimates, include a discussion of the error in estimates of key design parameters, and discuss the implications of those errors for design decisions. (57 pages, 396kb PDF)
  • The March 2007 issue of Education Evaluation and Policy Analysis, Volume 29, No. 1, features three articles that are recommended reading on this topic.
    • Pages 5-29: "Strategies for Improving Precision in Group-Randomized Experiments" by Stephen W. Raudenbush, Andres Martinez, and Jessaca Spybrook (24 pages)
    • Pages 30-59: "Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions" by Howard S. Bloom, Lashawn Richburg-Hayes, and Alison Rebeck Black (29 pages, $ download fee)
    • Pages 60-87: "Intraclass Correlation Values for Planning Group-Randomized Trials in Education" by Larry V. Hedges and E. C. Hedberg (27 pages, $ download fee)
  • Using Covariates to Improve Precision 2005, November
    This working paper, by Howard Bloom and colleagues, examines how controlling statistically for baseline covariates (especially pretests) improves the precision of studies that randomize schools to measure the impacts of educational interventions on student achievement. (121 pages, 573.99kb PDF)
  • Strategies for Improving Precision in Group-Randomized Experiments 2005, November
    This paper, by Stephen Raudenbush and colleagues, aims to clarify conditions under which the use of blocking and covariance adjustment can reduce the number of groups required to achieve adequate power in group-randomized studies. (42 pages, 326kb PDF)
  • Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions 2008. Howard S. Bloom, Carolyn J. Hill, Alison Rebeck Black, and Mark W. Lipsey.
    This MDRC working paper presents a framework for interpreting the magnitudes of estimated effect sizes plus a series of empirical benchmarks for doing so in the context of studies that estimate impacts of educational interventions on student academic achievement.
  • Some Food for Thought About Effect Size 2004, March
    This document by Howard Bloom presents a series of vignettes that illustrate how the widely-used standardized measure of "effect size" can have markedly different implications in different settings. These vignettes are intended to stimulate thinking about how small the minimum detectable effect must be for experiments in different settings and thus what their sample requirements are. (6 pages, 24kb PDF)