Counterintuitive discrepancy in cross-correlation signs between ctsem and lme4

[MaurizioSicorello]

Dear all,

I fitted a cross-lagged models with two latent variables in ctsem to assess the cross-correlations over time. One variable measured affect (higher values mean better mood) and one variable measured difficulties in emotion regulation (higher values mean more difficulties). Weirdly the cross-correlations have a counterintuitive sign, where better mood leads to more emotion regulation problems; and more emotion regulation problems lead to better mood. Both variables are correctly coded. I also fitted a lagged model with nlme4, where the cross-correlation has the expected sign.

I would really appreciate some ideas concerning this phenomenon! I could imagine I am not understanding something fundamental about ctsem.

Below is my ctsem and nlme code with some outputs.

Best wishes,
Maurizio

persModel_ct_td <- ctModel(type = "ct",
                           n.latent = 2,  # only affect and SDERS now
                           n.manifest = 2,
                           n.TDpred = 1,
                           manifestNames = c("affect_valence_scaled", "SDERS_Total_scaled"),
                           latentNames = c("affect", "SDERS"),
                           TDpredNames = "Stressors_scaled",
                           
                           LAMBDA = diag(2),  # identity matrix for manifest-latent mapping
                           
                           DRIFT = matrix(c(
                             "drift_Affect_Affect",    "drift_Regulation_Affect",
                             "drift_Affect_Regulation","drift_Regulation_Regulation"
                           ), 2, 2, byrow = TRUE)
)


persModel_ct_fit2 <- ctStanFit(datalong=df,
                              ctstanmodel=persModel_ct_td)

ctStanDiscretePars(persModel_ct_fit2, plot=TRUE, indices = 'CR', times = seq(from=0, to=48, by = 1))

Here is my nlme code, where I focused on the correlation of SDERS (emotion regulation problems) on affect at the next time point (lagged backwards). The predictors are person-mean centered. As I understood ctsem, it takes care of this by modelling a person-wise intercept per default. The distance between measurements was approximately 1-3 hours.

persModel <- lme(data=df, AFFECT_ESM_X_lag_minus1 ~ affect_valence_PMcen_scaled*S.DERS_Total_PMcen_scaled, 
                   random = list(PARTICIPANT_ID =~ 1, DayInd =~1), na.action = "na.exclude", control=lmeControl(msMaxIter = 100, opt = "optim"))

Linear mixed-effects model fit by REML
  Data: df 
       AIC      BIC    logLik
  24394.86 24442.45 -12190.43

Random effects:
 Formula: ~1 | PARTICIPANT_ID
        (Intercept)
StdDev:   0.9316594

 Formula: ~1 | DayInd %in% PARTICIPANT_ID
        (Intercept) Residual
StdDev:   0.5211351 1.389205

Fixed effects:  AFFECT_ESM_X_lag_minus1 ~ affect_valence_PMcen_scaled * S.DERS_Total_PMcen_scaled 
                                                          Value  Std.Error   DF  t-value p-value
(Intercept)                                            5.988748 0.06777041 4740 88.36818  0.0000
affect_valence_PMcen_scaled                            0.174139 0.01962063 4740  8.87530  0.0000
S.DERS_Total_PMcen_scaled                             -0.065456 0.02017161 4740 -3.24495  0.0012
affect_valence_PMcen_scaled:S.DERS_Total_PMcen_scaled -0.029538 0.01564515 4740 -1.88797  0.0591
 Correlation: 
                                                      (Intr) af__PM_ S.DERS
affect_valence_PMcen_scaled                           -0.005               
S.DERS_Total_PMcen_scaled                              0.009  0.296        
affect_valence_PMcen_scaled:S.DERS_Total_PMcen_scaled  0.069 -0.091   0.127

Standardized Within-Group Residuals:
        Min          Q1         Med          Q3         Max 
-4.79478864 -0.54438170  0.04843245  0.58704126  3.58370219 

Number of Observations: 6621
Number of Groups: 
            PARTICIPANT_ID DayInd %in% PARTICIPANT_ID 
                       222                       1878

[cdriveraus]

This is exceptionally common. Cross-effects are extremely sensitive to the rest of the model specification. ctsem includes random intercepts by default, as well as measurement error and correlated random fluctuations. Ignoring any of these can dramatically bias estimates, and as far as I'm aware no other software estimates all of these by default (which is not to say that ctsem is 'correct', but hopefully less wrong by default). with nlme I assume there is no measurement error or random fluctuations considered. You probably also want to consider the random correlation to show it's influence in your temporal effect plots:

ctStanDiscretePars(persModel_ct_fit2, plot=TRUE, indices = 'CR', times = seq(from=0, to=48, by = 1),type='observational')

For more on flipping signs, difficult inference etc under observational conditions, see these two:

https://osf.io/preprints/osf/xdf72_v1

https://osf.io/preprints/psyarxiv/y3e9d_v1/

[MaurizioSicorello]

Thank you for the input :)

[MaurizioSicorello]

In case other people have similar issues I want to give an update on the results.

I'd say the second preprint Charles suggested contained the solution to the specific problem (note that the argument for the given function currently appears to be "observational = TRUE"). This leads to the auto- and cross-regressions (AR/CR) shown below.

The simplest summary of the problem, as I understand it, would be:

Issues arise especially through unobserved confounders. The classic display of AR/CR effects shows what happens when there is an intervention on a single variable based on the fitted dynamic model (figure in the original post). In our study, this would be for example what happens to affect over time when emotion regulation difficulties (ERD) are manipulated (e.g., through a hypothetical mindfulness intervention). Still, we did not experimentally manipulate ERD in our data. In our observational data, the same unobserved influences are likely to both decrease affect ("worse emotion") and increase ERD (e.g., when a person experiences something bad). These kinds of influences are largely unobserved*. Therefore, instead of looking what happens after hypothetical interventions on a single isolated variable (like ERD), it might more informative to look at what happens to the system when it receives a typical "shock", in which both affect and ERD can be affected. This is achieved by setting the "observational"-argument to "TRUE".

In the resulting plot the effects become plausible. There, a typical "shock" (e.g., unobserved event) leads to a negative momentary cross-regression between affect and ERD. This means that more problems co-occur with worse affect, which is exactly what we would expect. As I understand the second preprint, a cross-regression in this plot needs to be interpreted in light of the auto-regression; for example, because ERD predicting affect could be simply due to the auto-regressive decay of affect. Therefore, the important take-away in my case would be, as I understand it, that the CR effect of ERD on affect decays more slowly than affect decays on its own according to the AR effect.

My interpretation would be that after shocks (e.g., unobserved negative experiences), (1) people have co-occuring negative affect and ERD and that (2) ERD slows down the return of negative affect to its baseline, likely due to ERD's slower-changing nature.

*Self-reported stressors were included in the model as a time-dependent predictor and I wondered whether there is a way to inspect the CR/AR functions with this stressor variable as an "observed shock"

[cdriveraus]

Thanks for the update! Yes in general in the presence of common causes, of which high correlation in the system noise (yours looks close to 1!) is one good (but not comprehensive) indicator, it is not very plausible to try to make claims about what would happen in an experimental intervention condition, which is what your initial plot above and the typical interpretation assumes.