Title: Standard errors and null-hypothesis significance tests for reliability coefficients
Abstract: Reliability analysis is one of the most conducted analyses in applied psychometrics. It entails the assessment of reliability of both item scores and scale scores using coefficients that estimate the reliability (e.g. Cronbach's alpha), estimate measurement precision (e.g., estimated standard error of measurement), or estimate the contribution of individual items to the reliability (e.g., corrected item-total correlations). Most statistical software packages used in the social and behavioral sciences offer these reliability coefficients. Standard errors and null-hypothesis significance tests (NHSTs) are generally unavailable for reliability coefficients, which is a bit ironic for coefficients that are about measurement precision. One of the tasks of my six-month sabbatical leave was to derive these standard errors and NHSTs. In this presentation, I will discuss the dilemmas and challenges of this task. In particular, I will discuss (1) categorical marginal models (CMMs), which can serve as a framework for finding the correct sampling distributions of complex statistics, (2) the challenges of estimating CMMs when a large number of items is involved, and (3) the challenges of developing user-friendly software. Finally, I will show what I think reliability-analysis computer output should look like in the future.