Advances in educational technology provide teachers and schools with a wealth of information about student performance. A critical direction for educational research is to harvest the available longitudinal data to provide teachers with real-time diagnoses about students' skill mastery. Cognitive diagnosis models (CDMs) offer educational researchers, policy-makers, and practitioners with a psychometric framework for designing instructionally relevant assessments and diagnoses about students' skill profiles. Still, methodological challenges prevent the widespread application of CDMs in educational measurement and classroom assessments. This paper considers problems of fundamental problems of identifiability, model selection, and the validation of expert knowledge.
Accurate inferences for CDM model parameters and student classifications require knowledge about the latent processes and attributes students need to succeed on educational tasks. The CDM Q matrix indicates which attributes are needed for each item and is central to implementing CDMs. In most applications of CDMs, content experts specify Q. The general unavailability of Q for most content areas and datasets poses a barrier to widespread applications of CDMs and recent research accordingly developed fully exploratory methods to estimate Q. However, current methods do not always offer clear interpretations of the uncovered skills and existing exploratory methods do not use expert knowledge to estimate Q. In fact, estimating Q without the use of available expert knowledge may be sub-optimal. Instead, incorporating expert knowledge during Q estimation may enhance interpretation of uncovered attributes and could assist with cognitive theory development. That is, using an exploratory method with expert knowledge may help to identify residual, or unexplained, attributes that are not predicted by cognitive theory. In such cases, exploratory CDM results can be shared with experts and subsequent conversations may serve to refine cognitive theories.
We consider an exploratory CDM framework that directly uses expert knowledge about item features by developing a new model to relate expert knowledge to the Q matrix using a latent, multivariate regression model. We report new sufficient conditions for identifying model parameters that impose fewer restrictions and are more likely to be satisfied in empirical applications. We show how the developed method can be used to validate which of the underlying attributes are predicted by experts and to identify residual attributes that remain unexplained by expert knowledge. We report Monte Carlo evidence about the accuracy of selecting active expert-predictors and present an application using Tatsuoka's fraction-subtraction dataset. Our analyses partially support expert knowledge and we uncovered two additional attributes that were not previously specified by experts. In general, the results of such analyses could be used to validate expert knowledge and shared with experts to determine if the residual attributes describe previously unidentified cognitive skills. We conclude the paper with a discussion of how the exploratory CDM approach can aid educational measurement in practice with particular focus on the settings where the goal is to provide fine-grained assessment of educational interventions a longitudinal setting.