today and
we have Anna and have a Loomis to going to present today for us about content specificity and
observation systems and issues around that so we're very happy for that and the recording is already
ongoing and we will stop the recording after the presentation of open up for discussion as we always
do so yes the floor is yours thank you
thanks Jennifer
so hello everyone and thank you for joining us today for this presentation we will also like to
thank the Queen's group for organizing this series of seminars on observation systems and especially
Mark and Jennifer for all the work that you have been doing in organizing these so our presentation
today is the third in a series of presentations during this for semester in which we have been
in thinking about the role of the content when starting teaching quality and as you can see from the
title of our presentation we're going to make a claim about the importance of adding content to the
equation when we study teaching quality and also discuss what it might look like so that
presentation is going to be going to be split into two parts of Tralee first we're going to be
discussing about
out the importance of taking content more seriously into account and then I'm going to turn it to
Anna and she's going to be discussing about the importance of synthesizing across content genetic
and content specific dimensions should present a model that will develop on synthesizing then share
some first findings from such attempts and then reflect on what it means to synthesize share some
affordances of this approach has approached and some open issues
so why should we take content more seriously into account if we look back in the studies that have
been conducted for decades in educational Effectiveness research we will realize that we have been
focusing on genetic Dimensions when we studied teaching quality and its relation to student learning
and in essence we were making this very strong
underlying assumption often times these was implicit other times it was more explicit that
Effectiveness Travis across content areas and subjects and if we want to become somewhat critical
about what we have been doing in most of the times we have been focusing on mathematics and we have
been more explicitly or more possibly made making arguments about teaching quality in general Asif
one size fit
hole
but research recent findings actually challenge the strong assumption and help us realize that this
but not have been the case and I'm going to share with you very selected findings there are several
but in the interest of time I'm going to go through three studies very quickly so the first study
study that was conducted by Jana and colleagues in 2016 and studying which they
floor teaching quality in German and in English to subject matters that are both languages and they
were using the three basic Dimensions with classroom management student support constructive support
and cognitive activation in this particular study they were analyzing they were partitioning the
teacher effects into that subject specific part and their clothes subjects perception effect that is
not related to
the subject matter per se so if we compare the green parts to them yellow Parts when it come comes
to classroom management the ratio is actually 10th one and one would say okay if this if this is the
case actually we shouldn't be worried about the content because you know the nonspecific part is
much bigger than the specific part but then moving to constructive support
they're the ratio is also much smaller was 221 which actually makes us strong Claim about the
importance of considering subject specificity and in that particular study the authors did not
examine cognitive activation and they were making the assumption that had they been studying these
the ratio could have been even smaller
another study that we conducted with colleagues here in Cyprus we were focusing on to subject
matters that are particularly distinct mathematics and physical education and in that particular
study we included 18 teachers who are studying both mathematics and physical education to the same
students and we explore Effectiveness using two different approaches first patient
additional scores and notice that we have been using the dynamic models to do so in both subject
matters which is a more generic instruments and then based on student learning based on value-added
scores then what we did for each of those approaches we cluster teachers into three groups based on
their findings low medium and high effective for each subject matter so you get tables like this
where you cross tab
relate the two subject matters and if subject would not play any wrong then most or all of the
teachers will be on the diagonal like here
but as you can see that's not the case about 40% of the teachers or on the diagonal where is about
60% percent of the teachers were off the Diablo and interestingly enough three teachers one 16 and
13 were two levels apart when it comes to the two different subject matters the situation was
slightly better in when it comes
student learning but even here half of the teachers were off the diagonal
and because these are two Studies by ourselves their Scholars have also explore these this issue
show and a recent study by Julie Cohen using data from the Met study actually showed similar finding
so in that particular study she explore mathematics and English language arts and out of the 120
teachers she found out that only 11 percent
were consistently using high-quality explicit instruction in both subjects murders 90% about 90% of
the teachers were not so we're increasingly realising that content matters and that one size does
not actually it all which is not surprising because back in the 1980s Schulman this making it was
making this very strong claim that would need to be attended
to the content in that by ignoring the content were missing important pieces of information
interestingly enough even in educational Effectiveness research Scholars of the beginning of the
21st century were making claims about attending to the content there's a very interesting book on
differential Effectiveness and I think it more or less has gone unnoticed and in this book Scholars
are making
a strong Claim about the importance of attending two different variables that actually can influence
teaching quality and one of them being the content self and currently enough during the last year in
scholars in educational Effectiveness research are indeed attending to the role of the content more
systematically not just as a variable that is added to the mixture but
is a condition that actually influence teaching quality and I think that's really important because
by doing so were in essence starting to actually challenging assumptions that will have been making
for years which are actually turn out or appear to be very strong so colleagues in 2020 have been
noted very legitimately that there is a paradox
it's in how we study teaching although we argue that we want to measure teaching quality in
different subject matters and actually take the subject into consideration we were already using
conceptualizations and framework that appear to be more subject independent and actually this is a
paradox that we need to be taking to consideration when we study teaching apart from
becoming more critical about the assumptions that we have been using for years in educational
Effectiveness in debt in general and in teaching quality in particular were increasingly realizing
that different sub to Madras impose different demands on the teacher so it's not enough to study
mathematics and then pretend that things transfer across different subject matters we have several
different subject matters and it just suffice
Isis to start thinking how things transfer from example from music to Arts to geography to history
to physical education into English one by raising this question one can start realizing white why it
is important to consider in the continent and Anna during the second part of the presentation will
be discussing how bringing scorer from different subject matters together and actually
us start answering such questions so what going to conclude the first part by actually making a
claim about the importance of combining content genetic and content specific approaches if content
is very important then we need to start thinking about the combination of these different approaches
and there are both conceptual and empirical reasons for doing so an argument that
Anna and myself have been making for the past five or six years is that teaching is disappointed
particularly complex phenomenon and teaching quality is also really hard to understand if we just
focus on just one part of this phenomenon and just used one lens which actually get us to the very
well-known Indian Fable of the Kleinman starting different
so the elephant and would do not want to become like those you know blind men than women but at the
same time just acknowledging the importance of using different lenses is not enough plurality is
important but it can become fruitful when we bring together different approaches and traditions and
when these different approaches and traditions come to interaction and stand speak to each other
we're also increasingly realizing that for years we have been using a multitude of Frameworks and
models you know starting from for example in mathematics in queue I IQ a.m. scan and then moving to
more Patrick Frameworks like true or YouTube or using more generic Frameworks like east of or the
dynamic model of educational Effectiveness so we have a ton of those frame
looks and models right now but we're increasingly realizing and this is an argument that several
Scholars including ourselves have been making that different these different approaches and
Frameworks need to be brought together because doing so actually promote communication and mutual
understanding it can help us reduce their over generation of Frameworks models and theories they are
important but when we start developing more and more than communication becomes harder this can also
bringing together different approaches can also help us bring more perseverance into our fears and
occasionally Effectiveness research and teaching quality in particular and then we can have more
arguments about comparing approach for studies and
remote cumulative knowledge so as an example if we go back to the meta-analysis that have been run
then one can raise a question and concern as to whether what what is included in this meta-analysis
actually comparable because we have been using very different approaches to conceptualize and to
capture things and by not producing humility of knowledge it's really hard to create knowledge
that's going to help
practitioners but they are also empirical reasons for doing so and I'm going to use just one that
more variance can be explained when combining the genetic and the content specific approaches with
colleagues here in Cyprus will have been running a couple of studies in physical education
mathematics showing that when considering the combination of these two different approaches we can
explain
barbarians I'm going to share with you just a snippet from this study in mathematics here we use it
was a secondary analysis from the team's data sets the 2007 and the trostle 2011 and actually what
we showed in this study was the importance of the combination of course the study had its own
limitations because teams has not
been developed by taking to consideration content generic and content-specific Frameworks but there
were some statements there in the teacher and the student questionnaires that reflected this genetic
and discontent specific aspect so if we focus only on a Content genetic aspects this is the variance
that can be explained you know the variance that you can see in the first panel then left on site
of the Seeker if we focus only on the content specific this is what we get in the middle but we can
get more of the variance when focusing on the combination which is what we get on the right hand
side and side and you know just as an example really and component2 characteristic examples for
example in Congo you can explain about double as much of the variance as compared to just focusing
on one
practice that contains specific practice and of course this varies across countries and it's really
an open question why this is the case but this is another interesting question so the conclusion is
that it is really important to combine generic and content specific aspects but but the question is
okay how can we do that and I'm going to leave are not to answer the very hard question
thanks a lot you should now be able to see my screen to that correct right so yeah if one agrees
that we should be thinking more harder about how to combine generic and content-specific asked quite
aspects the question is not how can we do that and we can think of the generic and the content
specific dimensions also the Frameworks as a continuum
and now the question then starts exactly how this Continuum can be brought together in working on
this or thinking about this we have been using a systematic developed by ISTA Bruna she suggested
that if we're interested in combining these two types we could either use an additive approach so we
take the generic and the subject specific separately and we just add them in trying to measure
as a teaching quality another approach would be to integrate the two so we would use for example
let's say cognitive activation and we combined both the generic aspects of cognitive activation and
the subject specific aspects into one dimension so we don't have separate Dimensions one generic and
the other one subject specific but they are integrated in the dimensions
and the third approach would be to start more not from the generic Dimensions kind of integrating
both perspectives but starting from the subject specific perspective in the example from mathematics
and thinking again in mathematics how could we comprehensively address certain Dimensions which
leads again to kind of integrating both the content specific and the generic but thought from
um the more content specific perspective so the latter two are kind of related to each other but the
starting point if it is a different one without knowing the systematic in the beginning how long was
a now when we were starting our work quite some years ago we were starting with the second approach
the integrative approach to starting more from water dimensions of teaching quality that kind of
assumed to travel across subjects and then think about how can we
kind of further develop them to really capture both and so yeah this is the the focus also in in the
following in thinking about now how we could kind of develop such an integrative approach we were
referring to the Work by Big Nash Josh and politica by the way a very interesting paper if you don't
know it already please read it it's really a very nice piece on theorizing
teaching and combining different models and theories and based on that work we developed this yeah
tiny little model and how we think it might be useful thinking about combining and that the the
content specific and generic approaches and the main argument for the beginning is that we cannot
just combine things the first and very important step is that we need to
make ourselves understandable so all the assumptions that we're putting into our work and often
times think implicit we need to make them explicit we need to Define everything we're using very
explicitly so that others can understand us and they should be doing the same so that we can I I can
understand the other approaches well so this is the first step because only if we understand the
others and they understand us we can be thinking about how can we
actually bring different approaches together
The Next Step would then start being to start to compare these different approaches and then
synthesize whatever is possible to synthesize also again based obviously on the fact that we cannot
synthesize everything so if there are things that are conflicting we cannot synthesize them but
oftentimes the different approaches is we've been finding in the literature using a more generic and
the more content specific approach they can be synthesized because they are kind of based on similar
as
functions and if we done so we obviously not done but we need to go back all the time making sure in
the kind of cyclical approach that we're taking the things that do not work so far into account and
start the entire process again and again also thinking about does the empirical evidence support our
assumptions the theories we have in the field today support the assumptions or do we need to go back
and revise certain assumptions or certain
ideas that we have about how to synthesize things in the following I would like to present very
shortly to like main results of two special issues with Bendix conducting over the last year's in
the in German journals so probably don't know these so far and therefore we selected them because it
might be new information for you that this interesting
they have been published in German general called indirect listen shop and in this in these two
special issues our idea was to bring several subjects together and discuss exactly these questions
of how what or first starting bye what do we know in different subjects about teaching quality and
how can we in the next step then maybe also start synthesizing the first special issue we try to
have not too many
subjects but have them as had originally as possible so we had Natural Sciences which is quite close
to mathematics still and then history and physical education as kind of other animals of subjects
and then added to the special issue another special issue where we included more subjects biology
history physical education the mathematics German English as a second language or foreign language
religion Arts and Music and what we gave them as a task is
provide an overview of research on teaching quality in their specific subjects and we also asked
them to use a common reference point which I will not go into much detail today which we ask them to
use the main teach model which he'll Amazon I developed and the idea of the common reference point
is that comparing different approaches who use very different language different structure is nearly
impossible and so I idea was to ask the
experts themselves to kind of have the overview of research in their own area and kind of integrated
into one categorisation and then the categorization is the same across the subject so that we can
start comparing actually what are the results of these two special issues so some findings kind of
indicate that one size actually fits also a large part of the sub dimensions in the dimensions that
we distinguish in the
each model can be found across all the subjects that we had in these two special issues so this was
the first very nice finding but as you can imagine this is not the end of the story so there are
also quite some parts where one size does not fit all fits rather so one of the dimensions where we
did see that we need considerable adjustments as cognitive activations
white early get got clear that cognitive activation because it was developed with a focus on
mathematics does not really work as it is now in all the other subjects so many of the other
subjects suggested that it needs additions for example the subjects as Arts music they were
considering that we need something more aesthetic combined with a cognitive because they were saying
well we have also these aspects of bad
if activation but the aesthetic experience of something is so important it's closely intertwined
with the cognitive activation we cannot think them independently so it needs to be kind of a broader
Dimension and there were similar arguments with other additions to cognitive activation in other
subjects
on the sub Dimension level we also had some suggestions for additions and I give you one example
here for example in classroom management we had like time and behavior management originally and the
physical add people and the Natural Sciences People based on the literature review said that they
need something like a sub-dimension on space and materials management additionally because this is
so Central to their subjects and without that nothing
works like the the children are endangered because then they can hurt themselves and physical
education but also Natural Sciences if they're conducting experiments and the material on the space
is not properly prepared than we do have a problem and this is only one of the examples that we
usually do not emphasize that much but that were like strongly emphasized by these colleagues one of
the questions
that erase them obviously is
how many things can we kind of ADD and change and still talk about the same construct so is
construct equivalents across subjects still given and this is an open question how far we can get
actually and still label it for example classroom management and kind of talk about the same thing
across subjects
if we go even one level down from the dimensions to the sub dimensions and then observable
indicators we also found that subject specific adjustments are necessary for example if we take the
positive relationship between teachers and students as part of the socio emotional support in
physical education again the same subject they were emphasizing that something like the adequate
handling of physical exposure in order to reduce embarrassment is very important
Orton to physical education and this is an observable indicator that I have never seen in all the
other Frameworks and instruments that are taking more from mathematics but interestingly as soon as
we started discussing about these additions it's was often the case that the other subjects who have
not like included this said well make sense for our subject to but we have not attended to that so
far
so taking this little example together with what Harlem was has pretty presented before what are the
affordances of that we've like reach so far in working on these issues well one obvious thing is
that mathematics should not constantly be the point of reference we're using in research on teaching
quality in the future and that we need to take more seriously
flee the subject demands of teaching into account
connected to that we need to
avoid that we over simplification things and that we kind of take certain things for granted for
example the idea that cognitive activation kind of travels across subjects probably not correct
what we can get with such an like enhancing view is that we get a more comprehensive picture of what
teaching quality actually is and what could also be the case and we've been realizing this in our
own work also over the last year's that it helps to kind of strengthening the bonds of a community
that is of researchers interested in teaching quality often times we have these kind of separate
groups that work more or less independently
Lee and if we take more into the focus are yeah we are all interested in research on teaching
quality then we can work better together and yeah creates from the bones
but obviously the story is not solved already in that quite some challenges and also open issues so
far the first one obviously the more we take these different subjects into account the more complex
it gets and it's easily super overwhelming with this complexity and additionally it also requires
that we work interdisciplinary and this is also very challenging because we have coming from
different traditions
as we use different language we have different perspectives on things and even the this first step
of like making one self understandable and understanding others sometimes takes a lot of time and
energy as well
thinking more about the open issues