CREATE Seminar: Investigating the effect of experience sampling study design on careless and insufficient effort responding identified with a screen-time-based mixture model

When designing experience sampling method (ESM) studies for clinical assessment, researchers must navigate a delicate balance between obtaining fine-grained snapshots of phenomena of interest and avoiding undue participant burden, which can lead to participants' disengagement and compromise data quality. To guide that process, we investigated how questionnaire length and sampling frequency impact careless and insufficient effort responding (C/IER) as an important yet understudied aspect of ESM data quality. To this end, we made use of existing experimental ESM data (Eisele et al., 2022) from 163 students randomly assigned to one of two questionnaire lengths (30/60 items) and one of three sampling frequencies (3/6/9 assessments per day). In our pre-registered analyses, we employed a novel mixture modeling approach (Ulitzsch et al., 2024) that leverages screen time data to disentangle attentive responding from C/IER and allows investigating how the occurrence of C/IER evolved within and across ESM study days. We further investigated the relationship between model-implied C/IER and other data quality indicators, such as self-reported C/IER, attention checks, and non-compliance. We found sampling frequency, but not questionnaire length to impact C/IER, with higher frequencies resulting in higher overall C/IER proportions and sharper increases of C/IER across, but not within days. These effects proved robust across various model specifications. Our findings contrast previous studies on non-compliance, suggesting that participants may employ different strategies to lower the different types of burden imposed by questionnaire length and sampling frequency. Implications for designing ESM studies in the context of clinical assessment are discussed.