Changing image encoder

[oscarloch]

Hi!

I was wondering:
What do you think is the easiest way to change the image encoder from CvT to UniFormer?

I want to make this change, but I’m a little confused about how to do it.
My initial idea is to edit the file:
modules/transformers/single_model/modelling_single.py
However, I’m not entirely sure if this is the only .py file I need to modify or which functions in that file I can reuse when implementing UniFormer, and which ones I cannot recycle.

Thanks again for all the help! This is a really useful model =).

[anicolson]

Hi @oscarloch,

In the lightning_module, e.g.,

cxrmate/modules/lightning_modules/single.py

Line 218 in b106927

if self.warm_start_modules:

,

Change this:

    if self.warm_start_modules:
        encoder = CvtWithProjectionHead.from_pretrained(encoder_ckpt_name, config=config_encoder)
        decoder = transformers.BertLMHeadModel(config=config_decoder)
        self.encoder_decoder = SingleCXREncoderDecoderModel(encoder=encoder, decoder=decoder)
    else:
        config = transformers.VisionEncoderDecoderConfig.from_pretrained(encoder_decoder_ckpt_name)
        self.encoder_decoder = SingleCXREncoderDecoderModel(config=config)

To this (ignoring self.warm_start_modules to make things easier):

        encoder = transformers.AutoModel.from_pretrained('aehrc/uniformer_base_tl_384', trust_remote_code=True)
        decoder = transformers.BertLMHeadModel(config=config_decoder)
        self.encoder_decoder = SingleCXREncoderDecoderModel(encoder=encoder, decoder=decoder)

This should also be the change for the multi-image and longitudinal, multi-image cases.

I haven't tested this, so let me know what errors occur (will probably be to do with the shape of the UniFormer output versus CvT, but we should be able to fix that easily).

A.