[WIP] Scheduler experimentation: don't shift tkv for new sequence

[yannicks1]

Description

Do not merge, ongoing experiment.

Related Issues

Issue: #746
Opened as closed this PR: #610.

Test Plan

Checklist

• I have read the contributing guidelines
• My code follows the project's code style (run bash format.sh)
• I have added tests for my changes (if applicable)
• I have updated the documentation (if applicable)
• My commits include a Signed-off-by: line (DCO compliance)

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

[yannicks1]

bot:bench
AGENT=AIOPS
NUM_PROMPTS=10

[yannicks1]

bot:bench
MAX_CONCURRENT=8

[yannicks1]

bot:bench
MAX_CONCURRENT=4
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
NUM_PROMPTS=4545

[yannicks1]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
NUM_PROMPTS=600

[waleedqk]

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
NUM_PROMPTS=600

[yannicks1]

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

[yannicks1]

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

[yannicks1]

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8

[yannicks1]

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8

[yannicks1]

bot:bench
NUM_PROMPTS=1000
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

[waleedqk]

bot:bench
NUM_PROMPTS=10
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4