Question about retraining/fine-tuning EncoderModel with new words in `t5.get_tokenizer()`

[Kevin7720]

That I have added some new words to t5.get_tokenizer() as shown below:

def get_tokenizer(name):
    tokenizer = T5Tokenizer.from_pretrained(name, model_max_length=MAX_LENGTH)
    new_words  =['XXX', 'OOO', ......]
    tokenizer.add_tokens(new_words)
    return tokenizer

I would like to understand if I need to retrain or fine-tune the EncoderModel after adding these new words to the tokenizer. How will this modification affect the model's performance or behavior?

This question is related to the Imagen project, and I want to ensure that I am following the correct approach when incorporating new words into the tokenizer.

[jacobwjs]

I'm not exactly sure what you mean, but what you proposed won't get you there.

See here:
https://github.com/huggingface/transformers/blob/70b49f023c9f6579c516671604468a491227b4da/src/transformers/tokenization_utils_base.py#L863

When you add new tokens to the vocabulary (and add the entry in the embedding layer), you'll end up with randomly initialized values corresponding to the new token(s).

[Kevin7720]

Thank you for your reply! Your answer has been incredibly helpful to me.