Exploring log files in international large-scale assessments: Methods, practices and tools

The new era of large-scale assessments involves the administration of the tests in a computer-based format. Examples in the international surveys which have conducted assessments in this format are: the Programme for International Student Assessment (PISA) and Programme for the International Assessment of Adult Competencies (PIAAC). Beyond collecting the correct/incorrect answers for each item, both assessments also collect the interactions between respondents and the computer testing application during the course of the test administration. Respondents' actions (e.g. starting a unit, clicking in a bottom, time spending until inputting an answer) within the tool are recorded in log files. The Organization for Economic Co-operation and Development (OECD), responsible for both programmes, has released such files publicly since the 2012 cycle of PISA and PIAAC. This work aims to present the potential and limits of such files, the up-to-date literature, as well as examining tools to extract and/or conduct correct analysis from this data. For this objective, this study will: (a) give researchers an overview of what they can expect for a first analysis as well the existed tools to work with these log files; (b) present a range of methods and practices that have been using such files to address issues relating to test-taking behaviour and strategies followed by respondents when answering to test items.