Different generation parameters in the same batch

[juliensalinas]

Hello team,

Today, batch generation works like the HF generate() function: it accepts several input texts but generation parameters (like temperature, top k, etc.) apply to the whole batch, so it is not possible to use different parameters within the same batch.

Is it because using different parameters in the same batch would degrade performance so much that it would defeat the purpose of batch generation?

Ideally it would be awesome if one could do something like this:

generator.generate_batch([
    {"input":input_1, "max_length":30, "sampling_topk":10},
    {"input":input_2, "max_length":150, "sampling_topk":50},
    {"input":input_3, "max_length":10, "sampling_topk":50},
    ...
])

For example this is something that can be achieved with NVIDIA Faster Transformers: https://github.com/NVIDIA/FasterTransformer/blob/main/examples/pytorch/gpt/gpt_example.py

Thank you!