Support Speculative Decoding

[JOHW85]

This could be used for LLMs and hopefully for encoder-decoder models like using the smaller NLLB coupled with the bigger NLLB models

[wsxiaoys]

This looks be a duplicate of #1234

[guillaumekln]

It's the same idea but I'm not sure it refers to the same implementation? There is also "Speculative sampling" which seem to refer to yet another implementation/algorithm of this concept.

[epinnock]

How hard would it be to implement a really naive version of this with ctranslate2? I would like to pick this up if possible

[guillaumekln]

Implementing this feature in the most basic form may be already possible with the existing Generator API. You could use generate_batch with a small model, and then use forward_batch with a big model to validate the output. The limitation of this approach is that when the big model does not agree, you have to start the generation from scratch and not at the first mismatched position.

[GaetanBaert]

Hello,

I'm looking to use speculative decoding for whisper : https://huggingface.co/blog/whisper-speculative-decoding

Is it possible with CTranslate2 ?