Custom bench metrics

[sducouedic]

Description

This PR allows to run benchmarks with additional custom metrics in addition to the regular TTFT, ITL values. It patches vllm bench to reuse the base implementation, but on top of it collects additional metrics such as the waiting time in the queue, or the waiting time in the queue or the number of chunked prefill count.

Note: we can use the client patching to collect and print information from the SimModel in PR #990. Additionally the patching should also allow to inject real number of output tokens to SimModel (because as a sim model, it cannot by itself generate a realistic number of output tokens), enabling more realistic perf simulation without --ignore-eos and --custom-output-len -1 parameters.

Contributions

• sendnn-bench serve instead of vllm bench. We register a custom spyre-chat backend that collects per-request custom metrics from SSE responses, in addition to the existing ttft, itl values

• Injection of per-request sendnn results into the existing output JSON file created by --save-detailed

• using --describe-metrics flag saves the sendnn metrics descriptions to a sendnn_bench_metrics_description.txt file

• Detailed timeline.html which displays for each request the waiting time and individual chunked prefill times (instead of only the ttft), as well as waiting time of decodes

• test for the patching and metrics collection

• A claude skill to easily add new metrics. The user has to give a detailed description of the new metric, how and where to compute the value, then the skill allows Claude to know all the places that require modifications for the new metric to appear in the .json output file and printed result. It also tells how to adapt the tests to integrate the new metrics. Examples:

  /add-bench-metric add a metric for the prefix cache hit percent, which is based on the number of chunks saved from cache hit over expected number of prefill chunks, for each request. The expected number of chunked prefill for a given request is math.ceil(request.num_prompt_tokens / self.chunk_size)

  /add-bench-metric a list of number of left-padding blocks for the request, one value for each decode step. The list for each request should be the same length of as the decode_latencies or decode_start_times. Ignore the exisitng left_padding variables et recompute it completely: left_padding = max_num_blocks - req_num_blocks, where max_num_blocks = math.ceil(tkv / block_size), and req_num_blocks = math.ceil(req.num_computed_tokens / block_size)

• Usage docs: https://vllm--1009.org.readthedocs.build/projects/spyre/en/1009/user_guide/detailed_performance_measurement.html

Usage

1. Export SENDNN_INFERENCE_BENCH_METRICS_ENABLED=1 to indicate to the server to collect sendnn metrics, and also set the patchings
2. Launch the server as usual: vllm serve {model} --max-model-len {model-len} --max-num-seqs {num_seqs}
3. Launch the benchmarking client spyre-bench serve --model ibm-granite/granite-3.3-8b-instruct --save-result --describe-metrics ...

Output:

============ Serving Benchmark Result ============
Successful requests:                     XXXX
Failed requests:                         XXXX
Maximum request concurrency:             XXXX
Benchmark duration (s):                  XXXX
Total input tokens:                      XXXX
Total generated tokens:                  XXXX
Request throughput (req/s):              XXXX
Output token throughput (tok/s):         XXXX
Peak output token throughput (tok/s):    XXXX
Peak concurrent requests:                XXXX
Total token throughput (tok/s):          XXXX
---------------Time to First Token----------------
Mean TTFT (ms):                          XXXX
Median TTFT (ms):                        XXXX
P99 TTFT (ms):                           XXXX
P100 TTFT (ms):                          XXXX
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          XXXX
Median TPOT (ms):                        XXXX
P99 TPOT (ms):                           XXXX
P100 TPOT (ms):                          XXXX
---------------Inter-token Latency----------------
Mean ITL (ms):                           XXXX
Median ITL (ms):                         XXXX
P99 ITL (ms):                            XXXX
P100 ITL (ms):                           XXXX
----------------End-to-end Latency----------------
Mean E2EL (ms):                          XXXX
Median E2EL (ms):                        XXXX
P99 E2EL (ms):                           XXXX
P100 E2EL (ms):                          XXXX
================= SenDNN Metrics =================
Total prefill chunks processed:          XXXX
---------------- Queue Wait Time -----------------
Mean Queue Wait Time (ms):               XXXX
Median Queue Wait Time (ms):             XXXX
P99 Queue Wait Time (ms):                XXXX
P100 Queue Wait Time (ms):               XXXX
------------- Chunked Prefill Count --------------
Mean Num Chunked Prefills:               XXXX
Median Num Chunked Prefills:             XXXX
P99 Num Chunked Prefills:                XXXX
P100 Num Chunked Prefills:               XXXX
------------ Chunked Prefill Latency -------------
Mean Chunk Prefill Latency (ms):         XXXX
Median Chunk Prefill Latency (ms):       XXXX
P99 Chunk Prefill Latency (ms):          XXXX
P100 Chunk Prefill Latency (ms):         XXXX
-------------- Decode Step Latency ---------------
Mean Decode Step Latency (ms):           XXXX
Median Decode Step Latency (ms):         XXXX
P99 Decode Step Latency (ms):            XXXX
P100 Decode Step Latency (ms):           XXXX
---------------- Prefix Cache Hit ----------------
Mean Prefix Cache Hit (%):               XXXX
Median Prefix Cache Hit (%):             XXXX
P99 Prefix Cache Hit (%):                XXXX
P100 Prefix Cache Hit (%):               XXXX
-------------- Left Padding Blocks ---------------
Mean Left Padding Blocks:                XXXX
Median Left Padding Blocks:              XXXX
P99 Left Padding Blocks:                 XXXX
P100 Left Padding Blocks:                XXXX
==================================================
INFO SenDNN metric descriptions written to results/sendnn_bench_metrics_description.txt
INFO Spyre metrics injected into results/spyre-chat-infqps-concurrency4-granite-3.3-8b-instruct-20260811-133931.json

Related Issues

None

Test Plan

Test the patching works correctly, that the metrics are saved to .json file correctly and also printed correctly

pytest tests/benchmarks/test_bench_metrics.py -m cpu

Checklist

• I have read the contributing guidelines
• My code follows the project's code style (run bash format.sh)
• I have added tests for my changes (if applicable)
• I have updated the documentation (if applicable)
• My commits include a Signed-off-by: line (DCO compliance)

[yannicks1]

these positional args are a bit fragile if upstream signature changes. Can we add a compat test to detect this?

[sducouedic]

added some tests to catch upstream changes in CI

[yannicks1]

this seems risky. can we just always set an explicit_name?

[sducouedic]

Now I force the user to set --result-dir and --result-filename whenever --save-result is set

[yannicks1]

also risky if that formatting changes upstream. We could add an compat test to detect this too.

[sducouedic]

Added a compat test so that if this changes in upstream, the CI will detect it

[sducouedic]

Thanks for the review @yannicks1 I addressed your comments

[yannicks1]

lgtm, thanks for this addition. will be very useful to get a better understanding of HW perf!