Trial lecture - time and place
Adjudication committee
- Professor Marie Wiberg, Umea University, Sweden
- Senior Researcher Martin Hooper, American Institutes for Research, USA
- Professor Arne Ola Lervag, Department of Education, University of Oslo
Chair of defence
Professor Rolf Vegar Olsen, Center for Educational Measurement, University of Oslo
Supervisors
- Professor Johan Braeken, Center for Educational Measurement, University of Oslo
Summary
This thesis explores two specific applications of null baseline model comparisons in the context of quantitative research in international large-scale educational assessments. Both applications make use of a null baseline model in which all observed variables are assumed to be uncorrelated. Articles 1 and 2 focus on model fit evaluation with the Comparative Fit Index (CFI). Here, the fit of a model of interest is compared against the fit of the null baseline model. Two simulation studies clarified the meaning and behavior of the CFI, as well as the consequences for the commonly used rule-of-thumb for model fit evaluation, as a function of the null baseline model.
Both articles end with the general reminder that incremental fit indices are relative measures with the null baseline model as a standardized metric, and thus their values should not be compared in an absolute sense nor should universal rules-of-thumb be adopted. Articles 3-6 focus on random response behavior in the TIMSS 2015 student questionaire. To assess the prevalence of and to identify those students likely engaging in random response behavior, we adopted a mixture item response theory (IRT) approach. In this approach, a relative comparison definition for random responders is used based on a contrast between a measurement model (reflecting regular response behavior) and a uniform null baseline model (reflecting random response behavior).
The articles investigated the prevalence, impact, and characteristics of random responders, as well as where and how often random response behavior occurs. Results showed that: (i) The average prevalence of random responders was estimated at 7.5% [0-38%]; (ii) The overall impact of random responders on aggregated-level was fairly limited; (iii) Scale position and scale character were important determinants for the prevalence of random responders; (iv) Certain groups of students (e.g., students in higher grades or male students) were more likely to be identified as random responders; and (v) Random responding is not necessarily a consistent behavior across the questionnaire. This thesis was supported by a research grant [FRIPRO-HUMSAM261769] of the Norwegian Research Council and has been carried out at CEMO: Centre for Educational Measurement at the University of Oslo.