This paper presents an example of how educational measurement can contribute to the next generation of assessment systems. It outlines a joint initiative by Australian Council for Educational Research Centre for Global Education Monitoring (ACER-GEM) and UNESCO Institute of Statistics (UIS) to develop empirically supported vertical scales in mathematics and reading. The scales will play a role in improving the quality of measuring and monitoring learning outcomes within countries (including those in developing and conflict-affected contexts), and address the challenges associated with between-country comparisons. Such advancement is essential to ensuring the Sustainable Development Goal (SDG) agenda is achieved and that all countries, including those not participating in large scale assessments, have the opportunity to participate and benefit.
The vertical scales describe learning progressions for reading and mathematics, across the range of proficiencies that typically develop throughout compulsory schooling. The aim is to enable countries to examine and report the outcomes of their assessment activities using a common methodology. Despite the high level of participation in learning assessments, clearly defined vertical scales and intra- as well as inter-assessment comparability remain limited. This presents particular challenges for measuring progress against the SDGs for learning outcomes. The learning goals and targets will only have meaning and utility if they are underpinned by empirically derived common scales that accommodate results from a range of different assessment programs. Vertical scales provide a means to assess the emerging competencies of learners, and to explore cognitive growth and trends in growth over time. The development of the vertical scales allows policy makers, education practitioners and education investors to not only quantify and compare learner proficiency, but also describe it in a meaningful way.
The vertical scales are based on an empirically analysis of the relative difficulties of items across assessment programs international, grounded in a conceptual framework taking in the current state-of-the-art of reading and mathematics theory. To permit comparison of the difficulty of the different item sets mapped to the vertical scales, a pairwise comparison methodology (BTL model) was employed. More than 500 items from 14 assessment programs were included in the analysis and more the 30 000 comparisons were made for each of reading and mathematics . The purpose of this comparison was to generate a set of difficulty estimates across the entire item set used in the initial steps of development of the vertical scales for reading and mathematics respectively. A pairwise comparison of items enables the different assessment programs from which those items were sourced to be aligned, allowing inferences to be made as to the underlying learning progression represented by the items. By modelling the cumulative information provided by multiple comparisons from many content specialists, estimates of the difficulties of items on a latent scale were obtained. Excellent evidence was generated that the pairwise estimates recovered the within-assessment program item parameters (where published). This presentation will present the methodology undertaken to create the vertical scale, outline the broad findings, and discuss implications and next steps for validation.