[WIP] interleave prefill steps with a decode step

[sducouedic]

Interleave consecutive prefill operations are interleaved with a decode step to minimize interruptions of current running requests. This mitigates the peaks in inter-token latency (ITL). A prefill is skipped if the previous step was also a prefill.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[yannicks1]

nit: I guess you mean not allowed, not now allowed in the comment

[maxdebayser]

I think it makes sense to rate limit the prefills, but shouldn't this be regulated by how many requests are in the batch? Let's say that the current batch is completely empty, and I send N requests, then it will take 2*N to ramp up to full compute utilization.