Some questions about 257 x 1024 CLIP L

[dongyangli-del]

Hi, @PaulScotti , thanks for your great work! I'd like to know whether the model used for the CLIP L is the pre-trained model of ViT/L-14, or the pre-trained encoder of the GIT model.

As far as I know, although the output shape of both is 257 x 1024, the GIT model is fine-tuned for image caption tasks and has better results. On the contrary, the image features obtained by the ViT/L-14 image encoder are difficult to generate image descriptions directly through the GIT model.

Looking forward to your reply, thank you.

[PaulScotti]

The specific notebook you are looking for is here: https://huggingface.co/datasets/pscotti/mindeyev2/blob/main/Train_bigG_to_large.ipynb

If you have more questions please let me know, sorry for my delay in responding to you

I'll also note that image captioning aspect of MindEye2 was developed pretty rushed right before submitting to conference, so I'm sure there are better and simpler ways to get image captioning working