In computer adaptive testing, the multistage design with branching rules at decision points has seen some important applications in recent years. A feasibility study of a multistage computer-based adaptive test was carried out in 2016 as part of a system-wide program to be used in schools. The aim of this feasibility study was to evaluate the design and the branching rules. This paper will present the design, the implementation and the evaluation results.
The adaptive test was designed for three grade levels covering four different domains. The adaptive design is a three-stage tailored test design with a specified branching model with two decision points. Each test module consisted of a total of six mutually exclusive testlets with varying average testlet difficulties. Each participating student was administered three testlets, one at each stage, assigned according to the branching rule based on the student performance up to the decision point. All items used in the multistage adaptive test were calibrated based on data collected previously from traditional paper tests using the Rasch model using ACER ConQuest. Test items were assigned to the testlets and branching rules were developed based on these existing item difficulties.
The test data in the feasibility study were analysed, test items were recalibrated. Item difficulties and student abilities were estimated on a common scale. The cut scores predefined in the branching rules were evaluated. The result shows that most of the cut scores worked well, a few of them needed a small adjustment (within ±2 raw score points).
The recalibration results show that the difficulty of testlet in stage 1 which was administrated to all students is approximately the average of all testlets together. The test characteristic curves show each of five paths are well separated from each other for the most part of the ability range.
The student proficiencies were estimated using weighted maximum likelihood (WLEs). Student ability distributions were compared by test path. The multistage adaptive test had functioned well in terms of assigning students of different ability levels to the appropriate test paths. The data had clearly showed the branching rules had successfully directed students of higher abilities to the more difficult test paths and students of lower abilities to the easier test paths. There were marked difference in the range of student abilities between students who were assigned different paths at the end of stage 1, and the tailored test design further separated the students into different test paths based on ability at the end of stage 2.