QUINT PhD Summer Institute & Conference 2022: Report

Conference report

The third QUINT Conference “Theorizing and Measuring Teaching Quality: Instruments, Evidence and Interpretations” contributed to further academic discussions in the field.

The event began with PhD Summer Institute, two days devoted to PhD papers, followed by parallel sessions with papers by experts in the field from all over the world. In this way, the format of the conference represented a unique platform for exchange of expertise between early career researchers and the more experienced ones. Alongside the main papers sessions, there were three keynote presentations, a video coding workshop, a panel debate on Icelandic policies on teaching quality, and a separate session dedicated to one of the central projects in QUINT, the LISA Nordic Study (website link with information on the project).

Papers in the parallel sessions represented a broad variety of topics, such as approaches to evaluating and measuring teaching quality in Social Science, Math, L1 and L2; teacher education and professional development; and subject-specific aspects of teaching quality.

One of the keynote presenters at the conference, Professor Daniël Muijs, said:

“The conference stood out for the clarity of focus and the high quality of research presented. This high quality was evident across the board, from PhD’s to experienced researchers. I felt that the conference moved the field forward in terms of understanding research into classroom processes. There was the feeling of an international community of scholars coming together.”

Abstracts from all paper sessions are available for download from the conference page.

QUINT PhD Summer Institute 2022

Over the course of two days, doctorial students from around the world, including the QUINT PhD fellows, presented and discussed their work in education research. 27 PhD papers were presented across various topics ranging from studies of cognitive activation in classrooms, to student and teachers’ perceptions of teaching quality. Presenters fielded questions from their peers, as well as from senior researchers, including the conference’s keynote presenters.

Hannah Bijlsma, PhD Fellow and Researcher at the University of Twente noted:

"Measuring teaching quality validly and reliably is a challenge. But it is not impossible. At the QUINT conference, we discussed what to consider when observing and/or scoring lessons and assessing teachers and schools. This is valuable for my own research, which is on lesson quality in Dutch schools.”

Patrick Schreyer, Doctoral candidate from Germany, DIPF, Leibniz Institute for Research and Information in Education said:

“The QUINT Conference 2022 was a welcome opportunity for personal exchange with researchers in the field of classroom research. I particularly liked the consensus regarding critical and reflective consideration of theorizing and prior measurements of instructional quality. This was evident not only in the excellent keynotes of the conference, but also in the methodologically rich and diverse contributions, both quantitative but also increasingly qualitative, of all participants.”

Keynotes dig into detail on methodological challenges

Three distinguished keynote presenters held talks that went into depth on the challenges of measuring teaching quality, drawing insights from their extensive research on teaching quality specifically.

Professor Daniël Muijs – Dean of the Faculty of Education and Society at Academica University of Applied Sciences in Amsterdam

In his keynote, Professor Daniël Muijs discussed classroom observations, focusing on the difficulties in achieving reliability and making valid interpretations when measuring teaching quality. In relation to reliability, Professor Muijs discussed issues such as observer bias, halo-effect, inter-rater reliability and the preferred number of observations. When discussing and problematizing validity issues, Professor Muijs raised critical points about different ways of conceptualizing teaching quality and what researchers can and cannot observe, as well as the consequences of not attending to the curriculum, classroom activities and the cultural context.

According to Professor Muijs’ suggestions for improving classroom observations, researchers need to a) focus on lesson sequences building towards a particular goal, instead of individual lessons. In addition, researchers should not expect the same behaviours to be observed, or to be effective, in different lessons. Furthermore, researchers need to b) understand the context of the lessons they are studying, and one way to do so is to use differentiated observation instruments for different lesson sections as well as understand the aim and purposes of lessons. Researchers also need to c) attend to curriculum and content, not just the process of teaching, for example by examining tasks in terms of the level of content teachers are aiming at. Finally, researchers need to d) engage more substantially in training to increase reliability using multiple observers/observations per teacher, as well as multiple perspectives.

Professor Courtney Bell – Director of the Wisconsin Center for Education Research (WCER) and Professor of Learning Sciences at the University of Wisconsin, Madison

Professor Bell’s keynote highlighted two different paths that educational research on teaching quality can take; information directly relevant and useful for practitioners one the one hand, and accumulating high quality research over time on the other hand. Professor Bell used these pathways to illustrate how the use of assessment information presents tradeoffs for key research issues when selecting common tools for measuring teaching quality in K-12 classrooms. The tradeoffs are definitions of teaching quality, measurement mode, data collection burden and validity.

Drawing on three well known studies, Professor Bell concluded that the definitions of teaching quality that researchers use affect both the cost of implementation, required training, the applicability to research/practice contexts, and the validity of scores. In terms of validity, Professor Bell argues that mode of measurement as well as the instruments we choose can shape foundational understanding of teaching quality. In addition, modes are related to the domains one can measure, burden of data collection and the validity of scores, and that details of how modes are related to these vary by at least which country and what teaching practice.

The overall take away message from Professor Bell is that the field of teaching quality needs both pathways in order to advance the state of knowledge, and in order for schools to be using tools that are reasonable and research-based. Bell stated that researchers have the responsibility to bring their social science to problems that the world faces, as they are uniquely positioned to think about these problems and thus play the role of accumulating and bridging knowledge between research and practice.

Professor Pam Grossman – Dean of the Graduate School of Education Teaching, Learning, and Leadership Division George and Diane Weiss Professor of Education

In her keynote, Professor Grossman talked about how to study core practices for Project-Based Learning (PBL). She presented a framework for PBL Core Practices pointing to four key practices: collaborative, authentic, iterative, and disciplinary, and she explained how this framework was used in a teacher professional learning intervention. In her talk, she presented 1) how teachers’ enactments of core PBL practices changed over the course of the Professional Development (PD), 2) how the quality of students’ work product changed over the course of the PD, and 3) whether there was a link between changes in teacher practice and changes in student work. The findings showed that the PD had effects on teachers’ practices in the dimensions of disciplinary practices, iteration practices, and – to some extent –student collaboration. They also found that at follow-up, student work was more likely to include indicators connected with disciplinary practices, and that in general, teacher practice grew in disciplinary practices and iteration practices.

Professor Grossman also highlighted issues that were puzzling in the study: Teachers reported feeling most familiar with disciplinary practices in the initial survey, even though this was the area in which the most change occurred. Teachers also reported learning most about practice in the authentic and collaborative domains, but the researchers saw little evidence of change in videos or in student work. The implication for practice that was highlighted based on this research, was that high quality instruction for PBL involves dramatic pedagogical transformation, and that PD focused on the core practices to support PBL can help shift teacher practice, but it may take longer to develop less familiar skills related to authenticity and collaboration. The implication for practice is that we need instruments that are well aligned to the goals of PBL as there are obvious challenges in using standardized tests of basic skills to assess high quality PBL, and there is a need for tools that measure collaboration and student agency. There is also value in using tools that are aligned around a common vision of high-quality instruction for PBL.

Workshop on coding classroom observations shows overlaps and discrepancies between frameworks

The first day of the conference featured a workshop where researchers applied a coding framework of their choice a video recording from a Nordic classroom, borrowed from the LISA Nordic study. Participants presented their framework and their findings to their workgroup. A total of 12 different frameworks were applied to footage from either a Language Arts or a Mathematics classroom, and were then discussed by the researchers in the group. This workshop was co-organised and financed by the research project Synthesizing Research on Teaching Quality (SYNTEQ), coordinated by QUINT Centre Director Professor Kirsti Klette. 

The results of these discussions gave insight into similarities and differences across frameworks and methods. Interestingly, despite differences in theoretical grounding and level of details, different frameworks revealed similar results in terms of patterns of teaching quality identified. The exercise also made visible some limitations of video-data, and illustrated the kinds of additional information different frameworks rely on to make statements about teaching quality.

The workshop represents a new approach to exchanging knowledge on teaching observation frameworks. The feedback from participants was positive, with many saying that the workshop was inspiring and useful for understanding how different research environments work with classroom video data.

Patrick Schreyer, Doctoral candidate at DIPF, Leibniz Institute for Research and Information in Education said

“A particular highlight for me was the coding workshop, in which participants had the opportunity to discuss their observation systems and approaches based on two collectively viewed instructional videos”

Hannah Bijlsma, PhD candidate and Researcher at the University of Twente, said

“The coding workshop made me realize how important a well-developed measuring instrument is and what influence the underlying concept of the instrument has on the rating of the lesson”

Further coding workshops will likely be arranged in the future, and the format will continue to be developed.

Panel Debate – Icelandic policies on teaching quality

Conference organisers used the opportunity of visiting Iceland to invite members of the national and local education community to a panel debate on Icelandic education policy. Panelists included Icelandic education ministers, union leaders, teachers, school leadership, and researchers. The discussion moderated by Professor Kirsti Klette (QUINT/UiO) and Professor Nikolaj Elf (QUINT/SDU) centred on the current state of teaching policy in Iceland and pathways to improving teaching quality in the country.

On the question of how the panelists would define quality in teaching, they responded that in Icelandic schools, quality in teaching is often linked to the well-being of the students in the classrooms and in schools. Therefore there has been focus on how students feel themselves in schools, and how the Icelandic teachers can contribute and facilitate this feeling this comfort. However, the panelists recognise that there is a rapidly growing group of students who feel disconnected at school. All panelists seemed to agree that defining teaching quality is in general a complex matter to define and measure. The LISA Nordic study in Iceland has revealed that Icelandic classrooms seem to score slightly lower on some categories of teaching quality than the other Nordic countries. These are preliminary findings, further analysis will reveal  possible clarifications for these scores. 

Towards a joint community of teachers and researchers

The panelists covered a variety of other topics, such as the current effort to improve the tools available to teachers, the current initiatives in education policy in Iceland, and the relationship between education researchers and practitioners. Professional development for teachers emerged as an important theme.

The Icelandic Teachers’ Trade Union leader, Ragnar Þór Pétursson, expressed concern over what he sees as a division between academics and teachers, saying too many education researchers lack teaching experience, and too many teachers lack good research literacy. He urged greater involvement of educational researchers’ in the education system.

“There’s no reason that academics shouldn’t be full-functioning members of the school communities. You don’t have to be distant, looking at the world through the lens of a camera. If educators and researchers manage to connect better, we can do magic.”

Teachers’ professional development and training is one forum where these connections can be made and strengthened, said Pétursson.  QUINT Icelandic researchers have been working closely with teachers and policy-makers in QUINT projects on Professional Development (PD) under research theme Using videos to support teachers' professional learning. 

Better tools for teachers

When asked what kind of help or resources are available to teachers in Iceland, Ingvi Hrannar Ómarsson – specialist in the Ministry of Education and Children – responded that they are currently working on revising the standardised tests.

“The changes that we are hoping to do after that are to create more support for teachers, to create more resources for them, and to stop with competition and start with collaboration. We should trust the teachers and say ‘you know your students best, we’re not going to tell you to measure all your students on this particular day.’ Instead, we should create toolboxes where teachers can access what they need and what they think their students need. We want tools where teachers can use the tool, see the results, and then adjust their teaching.”

The evolution of teaching policy

Anna Kristín Sigurðardóttir, a QUINT researcher, a theme 3 co-leader, and Professor at the University of Iceland, reflected on the changes occurring in how policy addresses teaching quality.

“Teaching is a complex task, it’s a whole system that hangs together. And when we look at educational policy in Iceland and other countries, we have had our neo-liberal approach with an emphasis on competition, measurements and things like that. But now I think we are moving towards professionalization, and acknowledging that what actually happens in the classroom is the most important if we want to develop the quality of the whole system.”

Kolbrún Þorbjörg Pálsdóttir, Dean of the School of Education at the University of Iceland added

“We mustn’t forget that schools have changed tremendously. Societies have changed the role of the school, and the role of teachers has changed. It’s not just about the importance of teaching and subjects and doing well on exams, it’s about the growth of the person, it’s about becoming the best version of yourself possible, and a good citizen. I think we take that to heart in policy, but maybe not in practice, it’s very challenging work.”

The QUINT projects Learning to notice in teacher education, Video to support excellence in teaching (VIST) and The Coherence and Assignments in Teacher Education (CATE) Nordic Study all examine ways of improving teaching quality through measures taken at the teacher education and professional development levels. Work is underway by QUINT researchers in Iceland that builds upon these projects.

Quotes have been edited for length and clarity.

LISA Nordic session

In a special session, titled “Standardized instruments and observation manuals as lenses into teaching quality: findings from Nordic classrooms using student perception surveys and the PLATO manual”, three key researchers from LISA Nordic presented papers on insights gained from the project. LISA Nordic Study is one of the central projects in the QUINT centre, coordinated by Centre Director Professor Kirsti Klette (University of Oslo) and Professor Michael Tengberg (Karlstad University). 

The session included three papers focusing on teaching quality in Language Arts classrooms in the Nordic countries, as well as comments from the discussant, Professor Pamela Grossman (University of Pennsylvania).

Paper 1 (Klette et al.) reported on a standardized classroom observation manual (PLATO) and discussed challenges linked to analyzing teaching quality. Paper 2 (Roe et al.) focused on how student perception surveys (the Tripod survey) might inform researchers about features of teaching quality. And Paper 3 (Gisladottir et al.) reported on the stated purpose and use of feedback (based on PLATO scorings) in Icelandic classrooms. 

Paper 1: “Standardized Observation Manuals as Lenses into Teaching Quality: Findings from Nordic Classrooms Using the PLATO Manual”

Presenters: Professor Kirsti Klette and QUINT Postdoctoral Fellow Camilla G. Magnusson (QUINT/University of Oslo).

Co-authors: Astrid Roe, Marte Blikstad-Balas, Jennifer M. Luoto (QUINT/University of Oslo), and Michael Tengberg (QUINT/University of Karlstad).

The paper reported on a predefined observation protocol in a Nordic context, to decompose teaching into key elements and identify key patterns and features for improvement, and to check for variation, using the Classroom Discourse element (CD) as an example.

The presentation focused on the research design for the LISA Nordic study as well as providing insight into the PLATO manual (Protocol for Language Arts Teaching Observations) and the findings across Nordic classrooms. The findings showed that in language arts, there was a general pattern of low scores on scaffolding elements and high scores on classroom environment elements. However, scores on the classroom discourse element (CD) varied across countries, and in-depth analyses of high scores in CD revealed common features across classrooms: use of questions, teachers revoicing and prompting, and often related to literature discussions. The paper’s authors concluded that PLATO works well within a Nordic context, that there are strong similarities in PLATO scores across Nordic LA classrooms, however, there is variation in scores on some elements (CD as an example). They highlighted that PLATO is able to capture features of teaching quality in a Nordic context, and that teachers’ instructional patterns across contexts are similar rather than different, although they find variation on the element level.

Paper 2: “Student perceptions as an indicator of teaching quality: A report from Nordic lower secondary language arts classrooms”

Presenter: Astrid Roe (QUINT/University of Oslo).

Co-authors: Michael Tengberg (QUINT/Karlstad University), Berglind Gísladóttir (QUINT/University of Iceland), and Anders Stig Christensen (QUINT/UCL University College in Denmark).

This paper reported on patterns of prevalent instructional practices between Nordic countries, and to explore validity aspects of the student responses as an indicator of teaching quality. The study reported on results from the Tripod student survey in five Nordic contexts (Norway, Sweden, Denmark, Iceland, and Finland), measuring student perceptions of teaching quality based on seven dimensions: care, control, clarity, challenge, captivate, confer, and consolidate, on a five-point Likert scale. The results focused mainly on score distributions. Across all countries, scores ranged from a total mean of 3.56 on the dimension “confer” to 3.99 on the dimension “care”. The total mean by country ranged from 3.61 in Iceland to 4.27 in Finland. Variations between countries on the different dimensions revealed that Sweden scored a little higher on “care”, Iceland a little lower on “control” and “clarify”, and Finland a bit lower on “challenge”. There were large variations between classrooms.

The paper’s authors concluded that there is some evidence suggesting that the Tripod survey may provide reliable data on Nordic students’ perception of teaching quality, but critical aspects of validity are yet to be explored. There are some recurrent differences between countries, but variation within countries is also considerable. However, in general, Nordic students appear to be satisfied with the quality of language arts teaching, especially with the quality of instructional explanations and the provision of care and support.

Paper 3: “Stated purpose and use of feedback in Icelandic lower secondary classrooms. Results from video-recordings”

Presenter: Berglind Gísladóttir, QUINT/University of Iceland

Co-authors: Birna María Svanbjörnsdóttir and Sólveig Zophoníasdóttir (QUINT/University of Akureyri)

The aim of this paper was to provide insight into teaching quality in Icelandic lower secondary schools and to assess the quality indicators “purpose” and “feedback” in language arts lessons and mathematics. The researchers reported on 72 video-recorded lessons that were analysed using the PLATO protocol. Their findings show some evidence regarding the quality of teaching in lower secondary classrooms in Iceland, related to the feedback students receive from their teachers and the coherence of the lessons observed. In contrast to previous findings from the Nordic countries, the researchers found limited evidence for a clear connection between the elements of purpose and feedback. However, similar to previous Nordic studies, they found that in a majority of the observed lessons, purpose and feedback were at the lower end of the PLATO observation protocol, indicating limited evidence of quality feedback to students and coherent lessons.

Conference pictures