Linear probing other datasets & pretraining collapse

[SamGijsen]

Hello,

Thank you for sharing your work!

I've ran into some issues running both your linear probing and pretraining setup and would really appreciate any input on what I'm doing wrong.

• Linear probing:
  On UKBB data this works well, but I get chance-level performance for any other dataset I've tried. I thought this might be because of the large offsets each ROI has for UKBB data specifically (mean might be 10000 for one ROI and 8000 for another). Multiple other datasets I know do not have these large specific offsets and thus, when one applies either robust scaling (or z-score standardization as included in your code) the resulting data distribution is sufficiently different that linear probing fails. I don't have the ADNI or HCP-Aging data you use in the paper - do they have similar ROI-specific offsets, or am I wrong to think this is relevant?

• Pretraining:
  I've ran your pretraining code on the UKBB but see collapse early on, without loss recovering from zero thereafter. Is this instability something you saw during your experiments? Would you have any recommendations on how to resolve this? Initial attempts to adjust the EMA parameter do not seem to prevent this.

Thanks