Title: Does the Procedure Matter? Applying a Multiverse Analysis Approach to Negative Emotion Differentiation
Abstract: Negative emotion differentiation (NED) – the extent to which individuals are able to experience and distinguish between emotional states of negative valence in a fine-grained manner – is commonly assessed using momentary emotion ratings in ambulatory assessment studies. However, there is variation in how researchers process NED data, for example, at which level of compliance participants are excluded. The objective of the present research was to use a multiverse analysis approach (i.e., conducting the analysis of interest across all reasonable combinations of decisions) to scrutinize whether varying decisions in NED assessment and NED data processing affect the robustness of empirical findings on NED’s relationships with other constructs.
The multiverse analysis included decisions regarding the compliance threshold for exclusion, the item set, the calculation of the NED index, the handling of negative indices, and the transformation of indices. The analyses of interest were twofold: First, we examined the robustness of the bivariate correlation between NED and depressiveness on the person level. Second, we investigated the robustness of the buffering effect of NED on stress reactivity, which corresponds to a cross-level interaction in a multilevel model. For both analyses, we performed a separate multiverse analysis. Preliminary results from three different data sets suggest that both analyses of interest are not robust across the included decisions and that there is variation in effect size.
The results of the multiverse analyses highlight the need for careful data preparation decisions, as they may affect empirical findings on NED. The results support recent calls for more research transparency.