Search
Resources

Design and Analysis of Group-Randomized Trials

We are interested in building the capacity of social scientists, foundation leaders, and public officials to carry out credible evaluation studies of group-based interventions. We have funded a team of researchers including Stephen Raudenbush (University of Chicago), Howard Bloom (MDRC), Jessaca Spybrook (Western Michigan University, and Andres Martinez (University of Michigan) to conduct this capacity-building work. Their suggested readings are below. The team also provides a consultation service and Optimal Design software to assist researchers.
RECOMMENDED GENERAL READING
  • Statistical Analysis and Optimal Design for Cluster Randomized Trials 1997; American Psychological Association
    This document by Stephen Raudenbush gives a more detailed account of design concerns for these studies with attention to the potential value of covariates in increasing precision and power and therefore reducing the number of clusters required. (15 pages, 961.79kb PDF)
  • Randomizing Groups to Evaluate Place-Based Trials 2005, June
    Howard Bloom presents a discussion of the rationale for, operating procedures for, and statistical implications of evaluation studies that randomize intact groups to measure the impact of interventions. The discussion focuses on a wide range of issues with respect to the design and analysis of such studies. "Randomizing Groups to Evaluate Place-Based Trials" is a chapter from the book, Learning More From Social Experiments: Evolving Analytic Approaches, edited by Howard Bloom. (90 pages, 746.55kb PDF)
  • Empirical Issues in the Design of Group-Randomized Studies to Measure the Effects of Interventions for Children 2008, July
    This MDRC working paper on research methodogy, by Howard Bloom and colleagues, “provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children.” (85 pages, 734kb PDF)
  • Modern Regression Discontinuity Analysis 2009, December
    This MDRC working paper by Howard Bloom provides a detailed discussion of the theory and practice of modern regression discontinuity (RD) analysis for estimating the effects of interventions or treatments focused on individuals or clusters of individuals. (62 pages, 642kb PDF)
  • When Is the Story in the Subgroups? Strategies for Interpreting and Reporting Intervention Effects on Subgroups 2010, November.
    This MDRC working paper by Howard Bloom and Charles Michalopoulos examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a sample study. The paper considers how and why subgroups are important for applied research; alternative ways to define subgroups; different research questions that motivate subgroup analyses, and the importance of pre-specifying subgroups prior to analysis. It also explicitly examines the conditions under which subgroup findings might be considered confirmatory evidence for decision making and conditions under which subgroups finds should be considered only exploratory and thus a basis for testing future hypotheses. The paper has been accepted for publication in Prevention Science.
IMPROVING PRECISION OF GROUP-RANDOMIZED TRIALS
  • Designing and Analyzing Studies That Randomize Schools to Estimate Intervention Effects on Student Academic Outcomes Without Classroom-Level Information 2011, April.
    This MDRC working paper written by Pei Zhu, Robin Jacob, Howard Bloom and Zeyu Xu provides practical guidance for researchers who are designing and analyzing studies that randomize schools to estimate the impacts of educational interventions on student academic outcomes without information about how students are clustered within classrooms. The paper demonstrates empirically that when schools are randomized and information about clustering at the classroom level does not exist, accounting for the clustering of students within schools is enough to produce valid estimate of intervention effects and their statistical significance. The paper has been accepted for publication in Educational Evaluation and Policy Analysis.
  • New Empirical Evidence for the Design of Group Randomized Trials in Education 2009, December
    This MDRC working paper, written by Robin Jacob, Pei Zhu, and Howard S. Bloom, offers guidance for designing group-randomized studies to measure the impacts of educational interventions. Specifically, the authors provide new empirical information about the values of parameters that influence the precision of impact estimates, include a discussion of the error in estimates of key design parameters, and discuss the implications of those errors for design decisions. (57 pages, 396kb PDF)
  • The March 2007 issue of Education Evaluation and Policy Analysis, Volume 29, No. 1, features three articles that are recommended reading on this topic.
    • Pages 5-29: "Strategies for Improving Precision in Group-Randomized Experiments" by Stephen W. Raudenbush, Andres Martinez, and Jessaca Spybrook (24 pages)
    • Pages 30-59: "Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions" by Howard S. Bloom, Lashawn Richburg-Hayes, and Alison Rebeck Black (29 pages, $ download fee)
    • Pages 60-87: "Intraclass Correlation Values for Planning Group-Randomized Trials in Education" by Larry V. Hedges and E. C. Hedberg (27 pages, $ download fee)
  • Using Covariates to Improve Precision 2005, November
    This working paper, by Howard Bloom and colleagues, examines how controlling statistically for baseline covariates (especially pretests) improves the precision of studies that randomize schools to measure the impacts of educational interventions on student achievement. (121 pages, 573.99kb PDF)
  • Strategies for Improving Precision in Group-Randomized Experiments 2005, November
    This paper, by Stephen Raudenbush and colleagues, aims to clarify conditions under which the use of blocking and covariance adjustment can reduce the number of groups required to achieve adequate power in group-randomized studies. (42 pages, 326kb PDF)
GUIDANCE ON EFFECT SIZES
  • Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions 2008. Howard S. Bloom, Carolyn J. Hill, Alison Rebeck Black, and Mark W. Lipsey.
    This MDRC working paper presents a framework for interpreting the magnitudes of estimated effect sizes plus a series of empirical benchmarks for doing so in the context of studies that estimate impacts of educational interventions on student academic achievement.
  • Some Food for Thought About Effect Size 2004, March
    This document by Howard Bloom presents a series of vignettes that illustrate how the widely-used standardized measure of "effect size" can have markedly different implications in different settings. These vignettes are intended to stimulate thinking about how small the minimum detectable effect must be for experiments in different settings and thus what their sample requirements are. (6 pages, 24kb PDF)