Title: Towards a more nuanced conceptualization of examiner/rater stringency in performance assessments
Abstract: It is well known that examiners vary in their judgments of performance in clinical examinations (often called Objective Structured Clinical Examinations=OSCEs). In this talk, I briefly review the problem and use mixed models (random intercepts and slopes) on a large examination dataset (n=442 exams) to estimate candidate, item and examiner effects on scores. This work demonstrates that examiner stringency behaviour is more complex than often thought. This is particularly important when standard setting is examinee-centred – i.e. where the patterns of scoring/grading in the exam are used to produce cut-scores and pass/fail decisions. The implications for OSCE practice, including examiner training, ‘fair’ score adjustment and standard setting, are discussed.