Custom bench metrics - #1009

Merged

yannicks1 merged 126 commits into

torch-spyre:mainfrom

sducouedic:custom_bench_metrics

Aug 17, 2026

Commits on May 21, 2026

Add stub of function to calculate required blocks
maxdebayser
committed
f393964
fix typo
maxdebayser
committed
526fac3
input_batch adaptation for preemption
sducouedic
committed
0c69697
run pre-commit hook
sducouedic
committed
9cc4310
make constraint greedy when scheduling new requests
sducouedic
committed
7e2c516
Fill get_blocks_required_for_decode_batch
maxdebayser
committed
baa4de5
Merge branch 'decode_holdback' of github.com:maxdebayser/vllm-spyre into decode_holdback
maxdebayser
committed
0c37074

Commits on May 22, 2026

temporary tests
sducouedic
committed
cf457b9

Commits on May 26, 2026

correct volumetric violated prefill holdback test
sducouedic
committed
0baa5ec

Commits on May 27, 2026

remove max-context-len constraint entirely
sducouedic
committed
9777fc6

Commits on May 28, 2026

scheduler predict next tkv
sducouedic
committed
86a4aa9

Commits on May 29, 2026

rename holdback to pausing
sducouedic
committed
88a7c4d
rename handle pausing and resuming
sducouedic
committed
ed9b1ab
bugfix: wrong tkv when expanding to new block
sducouedic
committed
e39dc32
continued: rename holdback to pausing
sducouedic
committed
117f7f2
complete decode pausing tests
sducouedic
committed
cd28178
handle paused decoding requests abort
sducouedic
committed
2053a3b
fix tests
sducouedic
committed
6a4c605
only add requests that were paused
sducouedic
committed
8555656
fix tests
sducouedic
committed
01e43b2

Commits on May 30, 2026

fix tests
sducouedic
committed
867e38c

Commits on Jun 1, 2026

pause/resume concept in intput batch
sducouedic
committed
15cabb9

Commits on Jun 2, 2026

small bugs catched
sducouedic
committed
2195185
handle finish of paused requests
sducouedic
committed
41ee95e
clean the tests
sducouedic
committed
944ed67
logits_processor advanced test: also check the output
sducouedic
committed
75eb053
remove superseded design idea for block reservation
maxdebayser
committed
0530cab

Commits on Jun 3, 2026

Fix input batch removal of requests
maxdebayser
committed
84cf5db
cleanup test code
sducouedic
committed
4bd1c83
factor out common part
sducouedic
committed
9d5f9d0

Commits on Jun 4, 2026

Commits on Jun 5, 2026

log preemption
sducouedic
committed
b39f262

Commits on Jun 9, 2026

custom bench serve: first version
sducouedic
committed
82167b5
allow to use sendnn-bench serve
sducouedic
committed
b523729
add some debug prints
sducouedic
committed
e850da5
bugfix processes isolations
sducouedic
committed
acfc886
bug fixed, remove prints
sducouedic
committed
a6a4590
cleaner output print
sducouedic
committed
2250d9e

Commits on Jun 10, 2026

pretty print final output
sducouedic
committed
baf5388
inject per request results to vllm bench serve .json file
sducouedic
committed
f846041
standalone metric: num prefill chunks
sducouedic
committed
cae6cc7
add benchmark tests
sducouedic
committed
92b78a5
scheduler: move bench attributes to dedicated dataclass
sducouedic
committed
3fc7054
custom detailed timeline
sducouedic
committed
8a17565
surface decode metrics
sducouedic
committed
71dbc31
timeline legend show itl range
sducouedic
committed
6e4394e

Commits on Jun 11, 2026

Commits on Jun 12, 2026

modelrunner: use dedicated dataclass for bench data
sducouedic
committed
b54fd0c
remove decode_tkv
sducouedic
committed
ffeac1f
update tests
sducouedic
committed
dcca355
track all metrics in scheduler
sducouedic
committed
d23408e
Always allow last prefill chunks onto the decode batch if it doesn't cause pausing
maxdebayser
committed
6014899
debug tests
sducouedic
committed
5f28438
update timeline
sducouedic
committed
636d4c1
add claude skill
sducouedic
committed
885f037
cleanup block pool before tests
maxdebayser
committed
b2556a9
display prefix cache hit
sducouedic
committed
3a51f7d
Merge branch 'main' into decode_holdback
maxdebayser
committed
3009cca
display left-padding
sducouedic
committed
c4233b7
refactor pause/resume
maxdebayser
committed
182306f
remove cond2
maxdebayser
committed
e2f7ecf
merge with decode_holdback
sducouedic
committed
e10d1bb
display prefix cache hit
sducouedic
committed
11f4051

Commits on Jun 15, 2026

address PR review comments
maxdebayser
committed
49db7f8
Merge branch 'main' into decode_holdback
maxdebayser
committed
fd5321a
address review comment
maxdebayser
committed
6243d35
fix test
maxdebayser
committed
a19dea2
display left-padding
sducouedic
committed
dcf2459

Commits on Jun 16, 2026

don't prioritize any endpoint so that we support --skip-chat-template
sducouedic
committed
224c9aa
display pause time
sducouedic
committed
0cac781
Merge remote-tracking branch 'sop-fork/custom_bench_metrics' into save_custom_bench_metrics
sducouedic
committed
8359657
rename 'previous_step_was_prefill' to 'step_is_prefill'
sducouedic
committed
28d4456
address pause/resume logic comments
sducouedic
committed
d1dbe75
Merge remote-tracking branch 'max/decode_holdback' into save_custom_bench_metrics
sducouedic
committed
a99e662
Merge remote-tracking branch 'origin/main' into decode_holdback
sducouedic
committed
42b1528
Merge remote-tracking branch 'max/decode_holdback' into save_custom_bench_metrics
sducouedic
committed
c78a340
support multi endpoints
sducouedic
committed
f83d4f4
extract bench part of update_from_output
sducouedic
committed
bb6ce98
Merge remote-tracking branch 'origin/main' into custom_bench_metrics
sducouedic
committed
72d16e8
address md error
sducouedic
committed
0b8b170
address precommit hook error
sducouedic
committed
d1af985
fix tests
sducouedic
committed
7722d20
timestamp right before return
sducouedic
committed
6e455fa
Merge remote-tracking branch 'sop-fork/custom_bench_metrics' into save_custom_bench_metrics
sducouedic
committed
1aa8ca4
rename prev_was_prefill to is_prefill
sducouedic
committed
2a262b1
don't put dedicated test workflow
sducouedic
committed
7a3ec53

Commits on Jun 17, 2026

count the number of requests blocked by missing kv blocks
sducouedic
committed
192af4b

Commits on Jun 18, 2026

restore cond2
sducouedic
committed
5260e9c
per request metrics json file save: one line per metric
sducouedic
committed
db632e1
Merge remote-tracking branch 'origin/main' into custom_bench_metrics
sducouedic
committed
e21b6fd

Commits on Jun 24, 2026

remove dead code
sducouedic
committed
dc53819

Commits on Jun 30, 2026

Commits on Aug 10, 2026

Merge remote-tracking branch 'upstream/main' into custom_bench_metrics
sducouedic
committed
2f750fd
Merge remote-tracking branch 'origin/custom_bench_metrics' into decode_pausing_bench_metrics
sducouedic
committed
040502f
restore diverging to main
sducouedic
committed
3f6ef78
bench pausing starting
sducouedic
committed
76d73e6
restore round_up function
sducouedic
committed
92da4e9
Merge pull request #1 from sducouedic/decode_pausing_bench_metrics
sducouedic
authored
6ae0167
format
sducouedic
committed
72507da
test fix multimodal
sducouedic
committed
988c482

Commits on Aug 11, 2026

add metrics descriptions
sducouedic
committed
0eaf86d
add prefilling breakdown metrics
sducouedic
committed
f88f697
bench metrics description: move it to file instead of print
sducouedic
committed
1082287

Commits on Aug 12, 2026

bugfix: time_ts was using end of first chunk instead of start

sducouedic
and
yannicks1
committed
dc1a72d
more thorough prefill phase breakdown test
sducouedic
committed
65af154
fix ordering
yannicks1
committed
f105f25
Merge pull request #3 from yannicks1/fix-ordering
sducouedic
authored
db1f1ab
test patch_serving compatibility
sducouedic
committed
2be9330
test main_async marker compatibility
sducouedic
committed
a0844f0

Commits on Aug 13, 2026

force set explicit name when --save-result or --plot-timeline
sducouedic
committed
a178f5c
bugfix: number of left-padding blocks can momentally go negative
sducouedic
committed
3e3fd99
handle detailed_timeline_plot in warning message
sducouedic
committed
68e5288
bugfix failing test
sducouedic
committed
8f579b4
add sendnn-bench docs
sducouedic
committed
e828990

Commits on Aug 14, 2026

Commits on Aug 17, 2026

Update tests/benchmarks/test_bench_metrics.py

yannicks1
and
sducouedic
authored
81b8c47
Merge pull request #4 from yannicks1/custom-bench-robustness
sducouedic
authored
9a289de