Continuous item calibration in computerized adaptive testing

In computerized adaptive testing (CAT), knowledge about the item parameters of the test items in the pool is required to select the next item. These item parameters are estimated based on item responses collected in a calibration study using an item response theory (IRT) model. In several potential application areas of computerized adaptive testing (CAT), constructing large numbers of items prior to the test’s initial use, and/or carrying out a calibration study with a large sample is not feasible. Correspondingly, for applications such as written standardized exams, psychological tests used in personnel selection, for clinical diagnoses or in research, CAT is typically not used, even though it would be advantageous here too. To extend the application range of CAT, a new continuous calibration strategy is presented and illustrated. This calibration strategy is applicable when setting up a CAT anew or when converting a linear test into a computerized adaptive test. The basic ideas of the strategy are (a) item calibration oriented on the time and capacity available for test development, (b) utilizing item responses across periodical assessments for item calibration purposes, (c) maintaining the measured scale over time, and (d) continuously increasing the adaptivity of the test during its operational use. In the presentation, I will describe the key elements of the new continuous calibration strategy and present results from a comprehensive simulation study. The simulation is based on a factorial design with the between factors IRT model (1PL, 2PL), sample size (50, 100, 300), item parameter estimation method (MML, Bayesian), and the within factor test cycle (1, ..., 11). The results showed a promising performance of the proposed strategy even for very small sample sizes. Based on a detailed presentation of the results, I will conclude with aspects of the continuous calibration strategy that should be covered by future research prior to its operational use.