Recasting the Secondary School Classroom as a Context for Positive Youth Development

Principal Investigators: Joseph Allen, Ph.D., University of Virginia; Robert Pianta, Ph.D., University of Virginia
December 2006–April 2013
$ 1,401,445

Can a program that provides regular feedback from a master teacher on the classroom performance of early-career, secondary school teachers change teachers’ interactions with students? If so, can the improvements be maintained in the next year? The researchers are implementing the My Teaching Partner (MTP) method, an intervention that encourages teachers to be better interpreters of youth behavior in order to create more positive interactions. Eighty teachers working in a Virginia school district were divided into experimental and control groups (40 teachers in each). Both groups are participating in a professional development workshop. The experimental group is receiving additional support in the form of the MTP training, which includes ongoing observation of teachers in the classroom and individualized feedback. Initial results, published in Science, indicate that the intervention produces substantial achievement gains at a very modest cost (equivalent to a nine percentile bump up in the average student’s academic performance at a cost of less than $40/student). This impact appears mediated by precisely the observed changes in teacher-student interactive behavior that were targeted by the program. Given the study’s initial success, the investigators received a supplemental award to develop a manual for My Teaching Partner for secondary schools (MTP-S) and context-sensitive replication strategies. Preliminary findings have indicated that students in MTP classes show greater observed student engagement at the end of the school year than those in control classes. Moreover, these observational findings seem to be mediated by improved instructional support for students. Student engagment and motivation also yielded significant effects that favor the MTP group.

Student Incorporation and the Sociocultural Contexts of Schools

Principal Investigators: Prudence Carter, Ph.D., Stanford University
December 2006–November 2009
$456,582

This study asks how school practices and student interactions either facilitate or impede students' abilities to be “culturally flexible”—or, able to participate in and cross different social and cultural settings in schools. What are the characteristics of both students’ group dynamics and the culture of high schools that promote or discourage black and Latino students’ inclusion in school life and academic advancement? Are these characteristics different in schools that are predominantly minority, more multi-racial, or predominantly white? This in-depth study generated theory about how cultural practices between students and teachers, the social organization of students (for example, in tracking and participation in extracurricular activities), and school policies about student self-presentation and discipline affect the incorporation of black and Latino students. Data was collected in two high schools in a northeastern city and two in a southern city. This project used a number of data collection methods, including ethnographic observations and interviews, semi-structured and group interviews, and student surveys. Findings from this study indicated several factors that could preclude full school engagement from students of different racial and socioeconomic backgrounds. These included the different approaches that educators take to the practice of equity in schools and the fact that similar groups of students are organized differently across school contexts. Educators in some schools often did not pay attention to the myriad ways in which students in minority groups were not fully incorporated into everyday school activities and how the coding of certain courses and extracurricular activities created boundaries for them.

Promoting Evidence-Based Decision-Making in Youth Mentoring Programs

Principal Investigators: David DuBois, Ph.D., University of Illinois at Chicago
July 2007–June 2010
$196,917

This Distinguished Fellow award to Dr. DuBois, a professor at the University of Illinois at Chicago, allowed him to work with Big Brothers Big Sisters of Metropolitan Chicago (BBBSMC) and Big Brothers Big Sisters of America (BBBSA). He worked in each of the major operational divisions at BBBSMC, shadowing staff and participating in activities that helped him better understand the work of front-line staff and how the organization is run. While at BBBSA, he worked with senior staff on a number of initiatives to translate research into practice and to promote evidence-based decision-making. Dr. DuBois also visited other BBBSA site agencies and participated in regional and national BBBSA conferences and meetings. Dr. DuBois reports that his Fellowship experience impressed on him the finely honed abilities and repositories of experience that were required in order for agency staff to effectively anticipate and manage the complex logistics of on-the-ground programming. Furthermore, he notes that his Fellowship enriched his appreciation of “the critical importance of providing ongoing support for mentoring relationships once they are established in programs, but also the formidable challenges that can make it difficult to do this job well.” For example, he notes that when managing mentoring cases, both time constraints and performance metrics led him (and his colleagues) to inevitably gravitate toward gauging success by volume (i.e., the percentage of calls completed on time) more than by quality. Dr. DuBois has also developed a deeper appreciation of the tensions that can exist between the press for timely release of research findings that are likely to be of significant interest to practice and policy stakeholders and the challenge of ensuring that such communications are effective and ultimately productive. In the event that initial findings and conclusions end up changing substantially, as is not uncommon, he is now wary that the benefits of early communication can be more than outweighed by the costs, which may include both greater confusion/uncertainty and reduced perceptions of credibility vis-à-vis research and its relevance among practice/policy stakeholders.

Toward an Understanding of Classroom Context: A Validation Study

Principal Investigators: Drew Gitomer, Ph.D., Educational Testing Services; Courtney Bell, Ph.D., Educational Testing Services
September 2008–June 2012
$813,910

The Classroom Assessment Scoring System (CLASS) is a promising observational instrument that measures student-teacher interactions in elementary school classrooms. Gitomer and Bell, along with colleagues from RAND (McCaffrey) and the University of Virginia (Pianta and Hamre), are testing the validity of the CLASS-S—an altered version of the CLASS meant for use in secondary school classrooms. They are also assessing different ways to implement the CLASS in an effort to optimize its efficiency. The investigators gathered data in 82 eighth and ninth grade algebra classrooms using three different strategies for completing the CLASS—the conventional observation by a classroom rater, video recordings of classrooms, and teacher self-assessment. For the validity analyses, the researchers collected measures of teacher and student characteristics, teachers’ math content knowledge, teachers’ and students’ views of intelligence, teachers’ beliefs about mathematics, and value-added measures based on student performance on the state standardized test and a pre-post test of algebra. In 2010, the investigators received a supplemental grant for additional data collection and scoring. Using factor analytic techniques, the empirical data support the theoretical model of CLASS-S. Sources of variance that have been identified include raters, mode (video versus live), lessons, experience in rating, and time of year. For instructional support and emotional support, scores decrease across the school year and also decrease along with raters’ experience and ongoing training. Overall, students are not learning much algebra. However, pre-test performance and gains are greater for eighth grade classes. The eighth grade classrooms also have higher CLASS-S scores, meaning that relatively stronger students do experience better teaching. However, whether this is the result of stronger teachers, rather than contextual factors that include the students, is unclear. In fact, while the relationship between CLASS-S scores and VAM is positive when other measures of prior achievement are not taken into account, positive correlations are not observed when adjustments for prior achievement are made. This raises important issues about the nature of the inference from CLASS scores to teacher quality. Overall, classroom organization dimension scores are high, most readily scored by observers, and most likely to be associated with teachers’ own evaluation of their practice. Emotional and instructional support dimension scores are much lower, more difficult to score, and less likely to be associated with teacher self-ratings.

Development of Self-Direction in Youth-Program-Family Interaction Systems: Latino and Non-Latino Adolescents

Principal Investigators: Reed Larson, Ph.D., University of Illinois at Urbana-Champaign; Marcela Raffaelli, Ph.D., University of Illinois at Urbana-Champaign
May 2010–May 2013
$640,034

How do adolescents’ engagement in youth programs and their interactions with program leaders and parents contribute to their development of self-direction? Many project-based youth programs (e.g., arts- or community service-based) cite helping youth set and achieve goals as a major program objective. Although many of these programs succeed in cultivating self-direction, little is known about how these skills develop and how adults support this development. This study examines how cultural factors, adolescent-parent interactions, and parent-leader interactions shape youth experiences in programs. Particular attention is being given to low- income and Latino youth. The study involves 12 project-based programs (focused on art, technology, science, or leadership and service) serving low-income youth ages 13 to 19 in Chicago, central Illinois, and Minneapolis-St. Paul. Youth, parents, and program leaders are being interviewed and asked to complete questionnaires three to four times over the course of the year, and investigators are observing program sessions, using the Youth Program Quality Assessment to examine youth-staff interactions associated with youth developing self-direction.

Policy Ideas, Entrepreneurs, and Education Research

Principal Investigators: Lorraine McDonnell, Ph.D., University of California, Santa Barbara; M. Steven Weatherford, Ph.D., University of California, Santa Barbara; Stephanie Dean, Ph.D., Hunt Institute
July 2010–June 2013
$453,620

The Common Core State Standards (CCSS) initiative has succeeded in developing K–12 content standards in mathematics and English-language arts, and in persuading 46 states to adopt them. This study examines the CCSS movement to understand how education research is used in developing and promoting policy ideas. It focuses on the role that policy entrepreneurs (politicians, advocates, foundations, and researchers) play in interpreting, framing, and applying research in policymaking, and how its use compares with other types of evidence such as practitioner knowledge, personal experience, public opinion, and philosophical beliefs. Study data include: interviews with CCSS leaders, public officials, researchers, and interest group representatives; participant observations; and document content analysis. In addition, adoption and implementation of the CCSS is being tracked in California, Indiana, Massachusetts, and Tennessee. A primary purpose of the study is to discern what forms of evidence are most useful to education reformers and the various audiences they seek to persuade about the merits of their policy ideas. As of June 2011, the investigators had archived more than 800 artifacts related to CCCS and conducted 48 structured interviews at the national level and in California, Massachusetts, and Tennessee. Due to political shifts and an unexpected acceleration in the adoption of the CCCS, the investigators have amended their framework to separate the decision-making about whether to use research-based evidence from how research-based evidence is used. Their research has revealed three main themes: (1) advocates have claimed that the CCCS are research- or evidence-based in order to depoliticize the debate; (2) research has been more influential in defining the problem that CCCS will presumably solve, but has been used less to shape the solution; (3) the CCCS movement demonstrates both the benefits of cooperation between research producers and users and the limitations of research as it competes with other political incentives such as constituent demands. The investigators presented key findings at the 2012 Annual Meeting of the American Education Research Association, which can be found here.