Keynote Presentations

From Framework Fragmentation to Synthesis in Research on Teaching Quality: Importance, First Steps, and Challenges

Professor Anna-Katharina Praetorius & Assistant Professor Charalambos Y. Charalambous 

Abstract

The last decades have accumulated significant empirical evidence attesting to the critical role that teaching has for student learning. Unsurprisingly then, interest in developing and using classroom observation frameworks and instruments to study teaching quality has escalated, leading to a plethora of such tools. Although this variation is not necessarily problematic, scholars have repeatedly warned about the fragmentation of the field of teaching quality and the local production of frameworks and instruments that impede attempts to produce cumulative knowledge in order to move the field forward. In this presentation, we provide an overview of attempts undertaken over the past years to work more synergistically by exploring how different frameworks/instruments can work complementarily to each other. Expanding on this work, we outline a line of research that brought together different scholars to reflect on teaching quality (a) as captured through different classroom observation frameworks and instruments; (b) by focusing more closely on the theoretical underpinnings of studying teaching. We discuss challenges encountered along the way and outline next steps that can help move the field forward.  

Video recording

Standardized Observation Manuals as Lenses into Teaching Quality: Findings from Nordic Classrooms Using the PLATO Manual

Professor Kirsti Klette

Video recording

Moving from Institutional Practices to Challenging the Robustness of Conclusions: The Case of Rater Error

Mark White, QUINT Postdoctoral Fellow, University of Oslo, Department of Teacher Education and School Research

Abstract

Many practices in educational research are institutionalized; we engage in them as a ritual because it is how we were taught or what colleagues do. One such institutionalized practice when using observation instruments is double scoring. In theory, double scoring is meant to provide confidence that codes have the intended meaning by showing that other, similarly trained raters would have produced very similar codes. In practice, double scoring is ritualized such that standard rater agreement statistics (e.g., Cohen's Kappa) are estimated and compared to standardized thresholds (Wilhelm, Rouse, & Jones, 2018).  However, high rater agreement statistics provide little assurance that other raters would produce results that would lead to similar conclusions.  In fact, in some sub-fields, such as video studies, rater agreement statistics well below established standards are routinely accepted with little testing of how rater error might affect study conclusions. In this talk, I argue for the need to move away from such institutionalized practices and towards approaches that examine the robustness of study conclusions to potential validity threats.  I highlight this principle with the example of rater error from the use of observation systems when trying to measure features of teaching quality within the educational sciences.  I demonstrate how to connect rater error estimates to specific study conclusions in order to examine whether the substantive conclusions of a study are robust to the observed levels of rater error found in the study.  I further argue for broader adoption of the ethos of this approach in order to align methods with modern validity theory, which holds that validity lies with the specific interpretation or use of data, not in compliance with institutionalized rituals.

Video recording