In educational large-scale assessments (LSA), the method of plausible values (PVs) is used to correct measurement error in the achievement test and to represent students’ (latent) proficiency scores while taking covariates from the background questionnaire, such as learning attitudes or interests, into account (Mislevy, 1991). This method follows the multiple imputation (MI) approach of Rubin (1987) by considering the latent proficiency scores as missing data, thus generating predictions for students’ proficiency from a scaling model that is based on both the achievement test data and the covariates in the background questionnaire. However, the scaling procedures employed in the generation of PVs require that the covariates are completely observed. This raises the question of how PVs should be generated from the scaling model when the covariates in the background model contain missing data (Rutkowski, 2011; von Davier, 2013).
In the present talk, we consider different strategies for dealing with missing data in the covariates of the scaling model. This includes the procedures currently employed in educational LSAs such as PISA, which rely on recoding the covariates with missing data before they are entered into the scaling model. In addition, we consider different strategies for treating the missing data that rely on nested and non-nested MI. In this context, non-nested MI refers to procedures that attempt to treat measurement error and missing data simultaneously (i.e., in a single stage), whereas nested MI refers to strategies that generate imputations for missing data and PVs in two consecutive stages (Harel, 2007; Rubin, 2003).
Finally, we present the results from a simulation study that compared these methods in a number of different settings. We show that the procedures currently employed in PISA can lead to biased parameter estimates when the data are not missing completely at random. By contrast, nested and non-nested MI are shown to provide unbiased estimates even with systematically missing data. In addition, we show that simplified procedures on the basis of nested MI which use only a single imputation in the second stage can provide similar results without the need for specialized software implementing the pooling methods required for nested MI. In this context, we emphasize the important differences in perspective of those involved in the scaling of the achievement data on the one hand and those performing secondary analyses on the basis of PVs on the other hand. We close with a discussion of our findings and consider possible consequences for current and future practices of handling missing covariate data in the scaling model in educational LSAs.