Consultation on Design of Group-Randomized Studies
The William T. Grant Foundation, in collaboration with Stephen Raudenbush and his colleagues, supports a free consultation service for researchers, funders, and policymakers. The service covers the design of group-randomized studies of interventions designed to promote the development of youth between ages 8-25, and/or the assessment of reliability and validity of measures of social settings. Designs not involving randomization of groups or measures of social settings are beyond the scope of the service.
The focus of the questions should be one or more of the following, related to group-randomized studies:
- Sample size determination (e.g., how many groups should be randomized to be able detect a given treatment effect at a certain level of statistical power? How many persons per group should be sampled?)
- Statistical power (e.g., how much statistical power will I have to detect a specified effect?)
- Minimum detectable effect size (e.g., how big an effect can I detect with my study at a specified level of power?)
- Assessment of reliability of setting-level measures (e.g., what is the reliability of a given setting-level measure? How can it be improved?)
- Statistical power accounting for the reliability of setting-level measures (e.g., how much statistical power will I have to detect a specified effect on a setting-level outcome considering the outcome reliability?)
To use the service, send an email to Andres Martinez at amzzz@umich.edu and include:
- A brief description of your study.
- Specific information that might help the consultants team, such as a brief description of your outcome variable(s), the effect size you expect the intervention to produce (or that you might want to detect), the importance of estimating effects for subgroups, the availability of baseline covariates, or the results of any pilot studies or past research that inform the design of your research.
Optimal Design Software
Optimal Design is a software package, developed by Stephen Raudenbush and colleagues, which helps researchers plan group-randomized trials, also called setting-level experiments.
In October 2011, the software was upgraded. Optimal Design with Empirical Information (OD+) is a major extension of Optimal Design. The new software includes a series of empirical estimates of plausible parameter values for determining the minimum effect size that can be detected by a given number, size, and treatment/group mix of randomized groups. This information (estimated values of intraclass correlations for groups and R-squares for baseline covariates) is currently available in OD+ for studies that randomize elementary, middle, or high schools to estimate average intervention effects on student reading and math achievement and for studies that randomize preschools or Head Start centers to estimate average intervention effects on social-emotional or cognitive student outcomes. In the future, we will add this type of information for other types of groups and outcomes. In addition, OD+ now provides tabular output (to complement its existing graphical output) that can be printed or saved to document the research design process.
The software (in both .zip and .exe downloads) and a manual containing software documentation are available at the links below.
Optimal Design with Empirical Information. Updated October 2011.
Please note, you may receive an error message when you install the software. To correct the issue, we recommend a package from Microsoft. If this doesn't solve the problem, please email richard@pikachu.harvard.edu.
Suggested citations
Tool: Raudenbush, S.W., Spybrook, J., Congdon, R., Liu, X., Martinez, A., Bloom, H., & Hill, C. (2011). Optimal Design Plus Empirical Evidence (Version 3.0)
Documentation: Spybrook, J., Bloom, H., Congdon, R., Hill, C. Martinez, A., & Raudenbush, S.W. (2011). Optimal Design Plus Empirical Evidence: Documentation for the “Optimal Design” Software Version 3.0.