Question about Contrastive Learning Collapse

[EdgarAllanPoo]

Thank you very much for your work. May I ask have you ever encounter a condition during experiments where your model collapses (only able to reconstruct certain classes only)? I have tried to replicate your work as a basis on other datasets . However, my experiments only able to reconstruct 2 types of images regardless of the pair in the test set. Looking forward to your reply!

[PaulScotti]

Without further information all I can say is that our model clearly doesnt collapse into just 2 types of images (you can see the variety of possible outputs here: https://github.com/MedARC-AI/MindEyeV2/tree/main/figs)

So I imagine perhaps your model had a problem during training...

[PaulScotti]

It is possible there are some kinds of representational collapse that is more fine-grained than 2 classes though, the generalizability of the possible reconstructed outputs is an interesting research direction. I'd also direct your attention to the zebra example in MindEye1 which touches on this nuance, stating that there is at least enough representational fidelity to, for example, figure out the exact correct corresponding zebra image out of a pool of similar-looking zebras