Introducing Sim mode by yannicks1 · Pull Request #990 · torch-spyre/sendnn-inference

yannicks1 · 2026-06-03T15:22:31Z

Description

Sim-mode lets us run the full plugin stack but mocks the forward pass (no op), while still:

Exercising all the real plugin code paths so we can catch correctness/constraint bugs without execution of model forward passes on either CPU or Spyre.
Producing a rough perf estimate for arbitrary workloads, by charging a user-supplied virtual cost per prefill chunk and per decode step. Useful for scheduler exploration and for quickly previewing the gain from features like request pausing without waiting for a real hardware run.

How it works

When SENDNN_INFERENCE_SIM_MODE=1, the runner swaps SpyreCausalLM for a no-op MockSpyreCausalLM and a SimState singleton accumulates a virtual clock alongside the real one.
Each forward step charges SENDNN_INFERENCE_SIM_PREFILL_MS or SENDNN_INFERENCE_SIM_DECODE_MS regardless of shape.
When a request finishes, its virtual lifecycle (per-token timestamps, prefill/decode durations, ITLs) is written to sim_metrics.jsonl, in roughly the same shape as the existing request_metrics.jsonl so analysis tooling carries over. Wall-clock request_metrics.jsonl writes are skipped in sim mode since they would all be 0.

Code reuse: The MockSpyreCausalLM from tests/ was reused for this implementation. It has been moved into the package sendnn_inference and is imported for the InstrumentedModelRunner in tests/.

Knobs

SENDNN_INFERENCE_SIM_MODE=1: activate the feature (default 0)
SENDNN_INFERENCE_SIM_PREFILL_MS: virtual cost in milliseconds per prefill chunk (default 0)
SENDNN_INFERENCE_SIM_DECODE_MS: virtual cost in milliseconds per prefill chunk (default 0)

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

github-actions · 2026-06-03T15:22:48Z

👋 Hi! Thank you for contributing.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

maxdebayser · 2026-06-03T17:13:01Z

Given that we can run in eager mode and that it also exercise the full model code, what is the use case for this?

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 · 2026-06-04T08:16:35Z

Given that we can run in eager mode and that it also exercise the full model code, what is the use case for this?

for constraint checking: This mocks the fwd call (takes ~0 ms), eager CPU execution fwd calls are taking a lot of time.

for perform approx: By providing a rough estimate of values for prefill and decode times, I was already able to get an approx of the vllm bench perf within 1-5%. This simulated bench takes magnitudes less time than running it for real (obviously not possible to get this data with eager on CPU)

maxdebayser · 2026-06-04T21:18:09Z

for perform approx: By providing a rough estimate of values for prefill and decode times, I was already able to get an approx of the vllm bench perf within 1-5%. This simulated bench takes magnitudes less time than running it for real (obviously not possible to get this data with eager on CPU)

Makes sense to me. Thanks!

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

sducouedic

This is a very cool idea, we should use it with long runs on scheduler changes

I left a few comments. But I have a general concert with tp > 1, if it could happen that decode and prefills gets recorded too many times, one for each model runner. Have you tried running with tp > 1?

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 · 2026-06-14T09:00:18Z

bot:test

yannicks1 · 2026-06-14T09:00:57Z

bot:bench
NUM_PROMPTS=1000
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

maxdebayser

Very nice, especially test mock model code reuse. I've left a few suggestions around the code structure.

maxdebayser · 2026-06-15T18:21:43Z

+            self._ensure_file()
+            assert self._fp is not None
+            self._fp.write(json.dumps(record) + "\n")


Suggestion: make self_.ensure_file() a context manager

maxdebayser · 2026-06-15T18:23:26Z

+def get_sim_state() -> SimState:
+    global _sim_state
+    if _sim_state is None:
+        _sim_state = SimState()
+    return _sim_state


The only place where _sim_state is referenced is the model runner, so it can me an attribute instead of a global singleton.

sducouedic · 2026-06-17T21:28:43Z

+
+        first_token_t = token_emit_times[0]
+        last_token_t = token_emit_times[-1]
+        ttft = first_token_t - rec.virtual_arrival


does the ttft still capture the waiting time in the queue this way? I see that rec.virtual_arrival is set with the first apparition of the request in record_step, which is called by the model_runner.execute_model() method, meaning it corresponds to the time of first chunked prefill

good catch, I did add this and now rec.virtual_arrival is set upon request arrival

sducouedic · 2026-06-17T22:02:31Z

+
+class SimState:
+    def __init__(self) -> None:
+        self.virtual_clock_seconds: float = 0.0


I think we should set the virtual_clock_seconds with the real time.time() and maintain only the difference to the real time. This would allow to capture all the overheads in the engine in addition to just the prefill time and decode time, if anything becomes suboptimal outside of self.model(), it will be captured by the sim-model:

initialization:

self.virtual_clock_seconds: float = time.time()

self.diff_to_real_time: float = 0.0

update in record_step():

self.diff_to_real_time += step_seconds

self.virtual_clock_seconds = time.time() + self.diff_to_real_time <-- we add the diff to time.time() instead of incrementing the virtual clock directly

this would also fix the issue of missing the waiting time in the ttft: ~~we add self.diff_to_real_time to the existing req.arrival_time value to get the virtual arrival time~~ edit: actually this wouldn't work correctly because self.diff_to_real_time is the current difference now, the req.arrival_time is in the past. So probably waiting time can be computed with time.time() - req.arrival_time (keep real time reference), then added to the ttft: ttft = time_last_token - time_first_chunked_prefill + waiting_time. Hope that makes sense.

… sim mode Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 · 2026-06-22T12:44:57Z

I am currently rethinking this. please wait with further review

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

## Description This PR allows to run benchmarks with additional custom metrics in addition to the regular TTFT, ITL values. It patches `vllm bench` to reuse the base implementation, but on top of it collects additional metrics such as the waiting time in the queue, or the waiting time in the queue or the number of chunked prefill count. Note: we can use the client patching to collect and print information from the SimModel in PR #990. Additionally the patching should also allow to inject real number of output tokens to SimModel (because as a sim model, it cannot by itself generate a realistic number of output tokens), enabling more realistic perf simulation without `--ignore-eos` and `--custom-output-len -1` parameters. ### Contributions * `sendnn-bench serve` instead of `vllm bench`. We register a custom `spyre-chat` backend that collects per-request custom metrics from SSE responses, in addition to the existing ttft, itl values * Injection of per-request sendnn results into the existing output JSON file created by `--save-detailed` * using `--describe-metrics` flag saves the sendnn metrics descriptions to a `sendnn_bench_metrics_description.txt` file * Detailed `timeline.html` which displays for each request the waiting time and individual chunked prefill times (instead of only the ttft), as well as waiting time of decodes * test for the patching and metrics collection * A claude skill to easily add new metrics. The user has to give a detailed description of the new metric, how and where to compute the value, then the skill allows Claude to know all the places that require modifications for the new metric to appear in the .json output file and printed result. It also tells how to adapt the tests to integrate the new metrics. Examples: > **/add-bench-metric** add a metric for the prefix cache hit percent, which is based on the number of chunks saved from cache hit over expected number of prefill chunks, for each request. The expected number of chunked prefill for a given request is math.ceil(request.num_prompt_tokens / self.chunk_size) > **/add-bench-metric** a list of number of left-padding blocks for the request, one value for each decode step. The list for each request should be the same length of as the decode_latencies or decode_start_times. Ignore the exisitng left_padding variables et recompute it completely: left_padding = max_num_blocks - req_num_blocks, where max_num_blocks = math.ceil(tkv / block_size), and req_num_blocks = math.ceil(req.num_computed_tokens / block_size) * Usage docs: https://vllm--1009.org.readthedocs.build/projects/spyre/en/1009/user_guide/detailed_performance_measurement.html ### Usage 1. Export `SENDNN_INFERENCE_BENCH_METRICS_ENABLED=1` to indicate to the server to collect sendnn metrics, and also set the patchings 2. Launch the server as usual: `vllm serve {model} --max-model-len {model-len} --max-num-seqs {num_seqs}` 3. Launch the benchmarking client `spyre-bench serve --model ibm-granite/granite-3.3-8b-instruct --save-result --describe-metrics ...` Output: ``` ============ Serving Benchmark Result ============ Successful requests: XXXX Failed requests: XXXX Maximum request concurrency: XXXX Benchmark duration (s): XXXX Total input tokens: XXXX Total generated tokens: XXXX Request throughput (req/s): XXXX Output token throughput (tok/s): XXXX Peak output token throughput (tok/s): XXXX Peak concurrent requests: XXXX Total token throughput (tok/s): XXXX ---------------Time to First Token---------------- Mean TTFT (ms): XXXX Median TTFT (ms): XXXX P99 TTFT (ms): XXXX P100 TTFT (ms): XXXX -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): XXXX Median TPOT (ms): XXXX P99 TPOT (ms): XXXX P100 TPOT (ms): XXXX ---------------Inter-token Latency---------------- Mean ITL (ms): XXXX Median ITL (ms): XXXX P99 ITL (ms): XXXX P100 ITL (ms): XXXX ----------------End-to-end Latency---------------- Mean E2EL (ms): XXXX Median E2EL (ms): XXXX P99 E2EL (ms): XXXX P100 E2EL (ms): XXXX ================= SenDNN Metrics ================= Total prefill chunks processed: XXXX ---------------- Queue Wait Time ----------------- Mean Queue Wait Time (ms): XXXX Median Queue Wait Time (ms): XXXX P99 Queue Wait Time (ms): XXXX P100 Queue Wait Time (ms): XXXX ------------- Chunked Prefill Count -------------- Mean Num Chunked Prefills: XXXX Median Num Chunked Prefills: XXXX P99 Num Chunked Prefills: XXXX P100 Num Chunked Prefills: XXXX ------------ Chunked Prefill Latency ------------- Mean Chunk Prefill Latency (ms): XXXX Median Chunk Prefill Latency (ms): XXXX P99 Chunk Prefill Latency (ms): XXXX P100 Chunk Prefill Latency (ms): XXXX -------------- Decode Step Latency --------------- Mean Decode Step Latency (ms): XXXX Median Decode Step Latency (ms): XXXX P99 Decode Step Latency (ms): XXXX P100 Decode Step Latency (ms): XXXX ---------------- Prefix Cache Hit ---------------- Mean Prefix Cache Hit (%): XXXX Median Prefix Cache Hit (%): XXXX P99 Prefix Cache Hit (%): XXXX P100 Prefix Cache Hit (%): XXXX -------------- Left Padding Blocks --------------- Mean Left Padding Blocks: XXXX Median Left Padding Blocks: XXXX P99 Left Padding Blocks: XXXX P100 Left Padding Blocks: XXXX ================================================== INFO SenDNN metric descriptions written to results/sendnn_bench_metrics_description.txt INFO Spyre metrics injected into results/spyre-chat-infqps-concurrency4-granite-3.3-8b-instruct-20260811-133931.json ``` ## Related Issues None ## Test Plan Test the patching works correctly, that the metrics are saved to .json file correctly and also printed correctly ```bash pytest tests/benchmarks/test_bench_metrics.py -m cpu ``` ## Checklist - [x] I have read the [contributing guidelines](https://docs.vllm.ai/projects/spyre/en/latest/contributing) - [x] My code follows the project's code style (run `bash format.sh`) - [x] I have added tests for my changes (if applicable) - [x] I have updated the documentation (if applicable) - [x] My commits include a `Signed-off-by:` line (DCO compliance) --------- Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Signed-off-by: Sophie du Couédic <sop@zurich.ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com> Co-authored-by: Max de Bayser <mbayser@br.ibm.com> Co-authored-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 added 3 commits June 2, 2026 10:31

initial commit: sim model

0bdb256

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

rmv sim summary script

78fb2fe

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

consolidate files for sim model and state

da51175

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 added 3 commits June 4, 2026 10:01

fix format

b0618fb

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Merge branch 'main' into sim-model

4c1f588

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

fix merge conflict

e4ddf8c

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

merge main into branch

a281ef4

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

sducouedic reviewed Jun 11, 2026

View reviewed changes

yannicks1 added 4 commits June 14, 2026 09:54

lazy import sim dependencies

78769f8

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

assert tp 1 for sim mode

3519906

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

assert backend eager for sim mode

5195497

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

add lock to file write

cc25f21

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 requested a review from sducouedic June 14, 2026 08:36

yannicks1 changed the title ~~[WIP] Introducing Sim mode~~ Introducing Sim mode Jun 14, 2026

yannicks1 marked this pull request as ready for review June 14, 2026 08:36

yannicks1 requested review from prashantgupta24, rafvasq and tdoublep as code owners June 14, 2026 08:36

joerunde reviewed Jun 15, 2026

View reviewed changes

Comment thread sendnn_inference/v1/worker/spyre_model_runner.py Outdated

maxdebayser requested changes Jun 15, 2026

View reviewed changes

sducouedic mentioned this pull request Jun 16, 2026

Custom bench metrics #1009

Merged

5 tasks

sducouedic reviewed Jun 17, 2026

View reviewed changes

Introduce SimulatedChunkedPrefillModelRunner for better separation of…

79b3e66

… sim mode Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 added 3 commits June 18, 2026 13:37

open metrics file in context manager

3a88249

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

account for waiting time in ttft

ba527aa

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Merge branch 'main' into sim-model

f17ce6c

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 marked this pull request as draft June 22, 2026 12:45

yannicks1 added 5 commits June 22, 2026 15:32

small cleanup

186011f

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

rewrite record step

8639d4c

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

skip warmup requests

3c45d36

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

fix inconsistency with vllm bench

23f0f0d

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

write decode and prefill times

fc54bf5

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Uh oh!

Conversation

yannicks1 commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How it works

Knobs

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

maxdebayser commented Jun 3, 2026

Uh oh!

yannicks1 commented Jun 4, 2026

Uh oh!

maxdebayser commented Jun 4, 2026

Uh oh!

sducouedic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yannicks1 commented Jun 14, 2026

Uh oh!

yannicks1 commented Jun 14, 2026

Uh oh!

Uh oh!

maxdebayser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maxdebayser Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

maxdebayser Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

sducouedic Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

yannicks1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

sducouedic Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yannicks1 commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yannicks1 commented Jun 3, 2026 •

edited

Loading

sducouedic left a comment •

edited

Loading

sducouedic Jun 17, 2026 •

edited

Loading