fix(qwen3): preserve explicit stop-token causes

docs/models/qwen3/model-crate.md

@@ -1,7 +1,7 @@
 # Qwen3-4B Model Crate
 
 **Created**: 2026-05-03
-**Last touched**: 2026-07
+**Last touched**: 2026-08
 **TL;DR**: `crates/pegainfer-qwen3` now owns Qwen3 config, weights, execution, scheduler, tests, benches, and kernel plan. Root `pegainfer` loads Qwen3 through a generic `EngineHandle` and no longer contains `Qwen3Model`, `Qwen3Executor`, `ModelRuntimeConfig`, root Qwen3 tests, or `src/model/qwen3/*`. The old `ModelForward` path has been removed; decode length-limit now emits the final token before `Finished`. Long-context `bs=1` TPOT was traced to non-partition FlashInfer paged decode under-filling the GPU; Qwen3 runtime gates FlashInfer split-K decode on `padded_bs<=32` (the `seq_len>=1024` gate was dropped in #437) with a 64-token chunk floor (`Tuned` capped at 64 chunks; opt-in `--batch-invariant` pins a fixed 160-token split), cutting 4k/64 serving steady TPOT from about `11.7ms` to `6.46ms` on RTX 5090. Qwen3 now keeps a single model-crate bench entry: `qwen3_kernel_snapshot`, a JSON snapshot runner with warm/cold-L2 latency, default-on CUPTI counters, and compare. Correctness/truth is intentionally out of this snapshot for now.
 
 ## Determinism scope
@@ -10,6 +10,18 @@
 
 It also **requires `--no-prefix-cache` and rejects `--kv-offload`**. A prefix hit advances a prompt's `chunk_start` off the request-local chunk grid, re-cutting its prefill chunks according to prior traffic. `--no-prefix-cache` closes that — except under offload, where it only disables HBM retention and deliberately leaves prefix *matching* on (host-tier reuse is the point of that mode), so that pair is rejected outright.
 
+## Stop contract
+
+The stepped Qwen3 scheduler keeps the request's EOS policy and explicit
+`stop_token_ids` independent in `StopPolicy`. When a generated token triggers
+either policy, that token is retained exactly once in the internal
+`RequestUpdate`; decode results carry its real logprob when one is available,
+and the terminal carries a typed `StopCause`. `StopCause::Eos` maps to
+`finish_reason=stop` without a wire
+`stop_reason`; an explicit request stop maps to the actual token ID.
+`ignore_eos=true` disables only model EOS and does not disable explicit request
+stop IDs. A length finish has no token-level stop cause.
+
 ## Preparation
 
 - **Read**:
@@ -157,7 +169,7 @@ pub fn start_engine(
 ### Step 7: Retire ModelForward and Fix Length Limit
 - Deleted `pegainfer_core::model::{ModelForward, GenerationState}` and removed the root `src/model.rs` re-export.
 - Deleted the Qwen3 `forward.rs` compatibility path. Qwen3 tests that used it now build their baselines from `batch_prefill(bs=1)` plus `batch_decode(bs=1)`, so they exercise the same phase APIs as production.
-- Fixed Qwen3 decode length-limit handling by adding `DecodeEffect::EmitAndFinish`. EOS behavior is unchanged: EOS finishes without emitting the stop token. Length limit now emits the sampled final token, then sends `Finished { finish_reason: Length }`.
+- Fixed Qwen3 decode length-limit handling by adding `DecodeEffect::EmitAndFinish`. The stepped stop contract now retains an EOS trigger in the internal token update and records `StopCause::Eos`; the bridge omits EOS from wire `stop_reason`. Length limit emits the sampled final token, then sends `Finished { finish_reason: Length }`.
 - Regenerated `test_data/Qwen3-4B.json` because every length-limited golden output now includes the final requested token.
 - Re-ran `bench_serving snapshot` on the CUDA validation host and pulled back `bench_snapshots/rtx-5090/qwen3-4b.json`; `decode_heavy (1024,256)` now records `generated_tokens min=max=avg=256`.
 - Performance stayed within noise on RTX 5090:

docs/subsystems/frontend/frontend-architecture.md

@@ -13,8 +13,9 @@ An engine is a set of schedulers, each a `Scheduler` implementation driven by th
 ```
 pegainfer-frontend/src/engine/
 ├── step.rs              # the wire: RequestId, Request, StepOutputs { Vec<RequestUpdate> },
-│                        #   RequestUpdate { scheduled, tokens, logprobs, cached_tokens,
-│                        #   prompt_echo, kv_transfer, terminal }, Terminal
+│                        #   Request { ..., stop_policy }, RequestUpdate { scheduled, tokens,
+│                        #   logprobs, cached_tokens, prompt_echo, kv_transfer, terminal },
+│                        #   Terminal { ..., stop_cause }
 ├── request_lifecycle.rs # typestate handles: QueuedRequest ─admit→
 │                        #   ActiveRequest ─finish/fail/defer→ consumed; every
 │                        #   transition is by-move, a dropped handle emits Failed
@@ -35,6 +36,7 @@ Design decisions worth knowing before touching it:
 
 - **Step-batched wire.** One message per scheduler step, not one channel per request: the scheduler's natural output unit is the step batch, and per-request channels were tried and rejected (the scheduler for-loop over N channels was the bottleneck). The protocol stack demuxes.
 - **Flat `RequestUpdate`.** All facts a step produced for one request travel in one struct, so intra-request ordering is structure, not convention. This is what makes `defer_finish` safe: a P/D prefill executor can withhold a request's `Finished` until its KV saves are peer-visible and send it later from any thread — the deferred message carries the request's entire buffered update, so late delivery cannot reorder.
+- **Independent stop policy and cause.** `Request.stop_policy` keeps model EOS handling separate from explicit request stop IDs. A token-driven finish retains the triggering token in `RequestUpdate` and carries `Terminal::Finished.stop_cause`; decode paths include its real logprob when available, so the stepped bridge can report the actual explicit stop ID without reconstructing it. `ignore_eos` affects only model EOS. Legacy producers may leave the cause empty while they are migrated individually.
 - **Typestate lifecycle.** `QueuedRequest` (queued) and `ActiveRequest` (streaming) are owned tokens; admit/reject/retire/finish/fail consume them, so "terminal exactly once, nothing after it" cannot be miscoded — it does not compile. A handle dropped without a transition emits `Failed` from its `Drop`, which is also how a crashed scheduler answers every in-flight request: the driver drops the scheduler, the handles fall, the terminals ship.
 - **Emitter as single writer.** Schedulers never touch the channel; they call `StepEmitter` methods against their handles. The emitter stamps `ScheduledInfo` at admission, tallies token counts (terminal counts derive from the tally, never from model-side arithmetic), and `commit_step` publishes the whole step in one send.
 - **Pure polling driver.** `spawn_scheduler` owns the serve loop: drain submissions, `Scheduler::step`, publish load, commit. No idle/park distinction — the scheduler owns the GPU and spinning on it costs nothing anyone else could use; async KV I/O (prefetch, decode-overlap prefill) is naturally absorbed by polling. An idle iteration ends in a `spin_loop` hint (relaxes the core's issue slots, no latency cost — busy iterations never pause). The loop exits when the frontend drops the handle and the queue drains.
@@ -93,7 +95,7 @@ All six lines are onboarded. Adding a model line = write `model_line.rs` in the
 
 ## Protocol stacks
 
-**`vllm` (current default, fleet-proven).** Impersonates a vLLM EngineCore process over in-process ZMQ/msgpack because upstream `vllm-server` assumes the engine is a separate process. HTTP routes, OpenAI types, tokenizer, chat templates, Prometheus live in the external `vllm-server`/`vllm-metrics`/`vllm-text` crates. `SteppedEngineBridge` translates each `RequestUpdate` 1:1 into an EngineCore output (wall-clock timestamps are reconstructed from the contract's `Instant`s via a per-bridge unix anchor; a `Finished{Stop}` appends the stop sentinel token, which is how usage keeps counting the suppressed EOS).
+**`vllm` (current default, fleet-proven).** Impersonates a vLLM EngineCore process over in-process ZMQ/msgpack because upstream `vllm-server` assumes the engine is a separate process. HTTP routes, OpenAI types, tokenizer, chat templates, Prometheus live in the external `vllm-server`/`vllm-metrics`/`vllm-text` crates. `SteppedEngineBridge` translates each `RequestUpdate` 1:1 into an EngineCore output (wall-clock timestamps are reconstructed from the contract's `Instant`s via a per-bridge unix anchor; a typed `StopCause` reports the actual request stop token, while the synthetic sentinel remains only as a compatibility fallback for producers that provide no cause).
 
 **`dynamo` (planned second stack).** dynamo's `lib/llm` in-process path removes the wire protocol entirely (`EngineConfig::InProcessTokens` + `run_input`). The step contract was shaped so this stack can consume `StepOutputs` directly without impersonation overhead. Decision gate: prototype, A/B against the vllm stack, let TTFT/step-overhead numbers pick the default.
 

pegainfer-frontend/src/engine/driver.rs

@@ -108,6 +108,7 @@ mod tests {
     use super::super::step::Request;
     use super::super::step::RequestId;
     use super::super::step::Terminal;
+    use super::super::stop::StopPolicy;
     use super::*;
     use crate::engine::FinishReason;
 
@@ -163,6 +164,7 @@ mod tests {
         Request {
             prompt_tokens: vec![1, 2],
             params: crate::sampler::SamplingParams::default(),
+            stop_policy: StopPolicy::default(),
             max_tokens,
             lora_adapter: None,
             kv_transfer_params: None,
@@ -197,6 +199,7 @@ mod tests {
             terminal,
             Some(Terminal::Finished {
                 reason: FinishReason::Length,
+                stop_cause: None,
                 prompt_tokens: 2,
                 completion_tokens: 3,
             })

pegainfer-frontend/src/engine/ledger.rs

@@ -38,6 +38,7 @@ use super::step::RequestUpdate;
 use super::step::ScheduledInfo;
 use super::step::StepOutputs;
 use super::step::Terminal;
+use super::stop::StopCause;
 
 /// One open account: the request's admission facts and running tally. The
 /// payload is not here — it went to the scheduler at `submit`; the account is
@@ -239,12 +240,23 @@ impl RequestLedger {
 
     /// Finish the request. Token counts come from the ledger's tally.
     pub fn finish(&mut self, id: RequestId, reason: FinishReason) {
+        self.finish_with_cause(id, reason, None);
+    }
+
+    /// Finish a request while preserving a typed token-level stop cause.
+    pub fn finish_with_cause(
+        &mut self,
+        id: RequestId,
+        reason: FinishReason,
+        stop_cause: Option<StopCause>,
+    ) {
         let account = self.close(id);
         let AccountState::Active { completion_tokens } = account.state else {
             panic!("finish on {id} before admission");
         };
         self.statement.entry(id).terminal = Some(Terminal::Finished {
             reason,
+            stop_cause,
             prompt_tokens: account.prompt_len,
             completion_tokens,
         });
@@ -279,6 +291,16 @@ impl RequestLedger {
     /// this step — tokens included — folds into the returned message, so late
     /// delivery cannot reorder against the step stream.
     pub fn defer_finish(&mut self, id: RequestId, reason: FinishReason) -> DeferredFinish {
+        self.defer_finish_with_cause(id, reason, None)
+    }
+
+    /// Defer a finish while preserving a typed token-level stop cause.
+    pub fn defer_finish_with_cause(
+        &mut self,
+        id: RequestId,
+        reason: FinishReason,
+        stop_cause: Option<StopCause>,
+    ) -> DeferredFinish {
         let account = self.close(id);
         let AccountState::Active { completion_tokens } = account.state else {
             panic!("defer_finish on {id} before admission");
@@ -289,6 +311,7 @@ impl RequestLedger {
             .unwrap_or_else(|| RequestUpdate::empty(id));
         update.terminal = Some(Terminal::Finished {
             reason,
+            stop_cause,
             prompt_tokens: account.prompt_len,
             completion_tokens,
         });
@@ -374,6 +397,7 @@ mod tests {
     use super::super::request_lifecycle::StepReceiver;
     use super::super::step::Request;
     use super::super::step::Terminal;
+    use super::super::stop::StopPolicy;
     use super::super::wiring::SchedulerHandle;
     use super::super::wiring::scheduler_pair;
     use super::*;
@@ -382,6 +406,7 @@ mod tests {
         Request {
             prompt_tokens: prompt,
             params: crate::sampler::SamplingParams::default(),
+            stop_policy: StopPolicy::default(),
             max_tokens: 8,
             lora_adapter: None,
             kv_transfer_params: None,
@@ -402,7 +427,9 @@ mod tests {
         backend.ledger.admit(id);
         backend.ledger.push_tokens(id, &[10, 11], &[]);
         backend.ledger.set_cached_tokens(id, 2);
-        backend.ledger.finish(id, FinishReason::Stop);
+        backend
+            .ledger
+            .finish_with_cause(id, FinishReason::Stop, Some(StopCause::Token(11)));
         backend.ledger.commit_step();
 
         let mut steps = handle_steps(handle);
@@ -419,6 +446,7 @@ mod tests {
             update.terminal,
             Some(Terminal::Finished {
                 reason: FinishReason::Stop,
+                stop_cause: Some(StopCause::Token(11)),
                 prompt_tokens: 3,
                 completion_tokens: 2,
             })
@@ -479,7 +507,9 @@ mod tests {
         let id = backend.ledger.register(envelope).id;
         backend.ledger.admit(id);
         backend.ledger.push_tokens(id, &[7], &[]);
-        let deferred = backend.ledger.defer_finish(id, FinishReason::Length);
+        let deferred = backend
+            .ledger
+            .defer_finish_with_cause(id, FinishReason::Length, None);
         backend.ledger.commit_step();
 
         let mut steps = handle_steps(handle);
@@ -498,6 +528,7 @@ mod tests {
             update.terminal,
             Some(Terminal::Finished {
                 reason: FinishReason::Length,
+                stop_cause: None,
                 prompt_tokens: 2,
                 completion_tokens: 1,
             })

pegainfer-frontend/src/engine/mod.rs

@@ -39,6 +39,7 @@ mod request;
 mod request_lifecycle;
 mod sink;
 mod step;
+mod stop;
 mod wiring;
 
 pub use control::*;
@@ -52,4 +53,5 @@ pub use request::*;
 pub use request_lifecycle::*;
 pub use sink::*;
 pub use step::*;
+pub use stop::*;
 pub use wiring::*;

pegainfer-frontend/src/engine/step.rs

@@ -14,6 +14,8 @@ use std::time::Instant;
 
 use super::event::FinishReason;
 use super::event::TokenLogprob;
+use super::stop::StopCause;
+use super::stop::StopPolicy;
 
 /// In-process routing id for one generate request, minted by
 /// [`super::SchedulerHandle::submit`] from a per-scheduler counter. `Copy` and
@@ -49,6 +51,7 @@ impl std::fmt::Display for RequestId {
 pub struct Request {
     pub prompt_tokens: Vec<u32>,
     pub params: crate::sampler::SamplingParams,
+    pub stop_policy: StopPolicy,
     pub max_tokens: usize,
     pub lora_adapter: Option<String>,
     /// Opaque router/P-D metadata from the request's
@@ -228,6 +231,10 @@ impl fmt::Display for RejectReason {
 pub enum Terminal {
     Finished {
         reason: FinishReason,
+        /// Present for token-driven stop finishes. The triggering token remains
+        /// in `RequestUpdate.tokens`, with its real logprob in the matching
+        /// `RequestUpdate.logprobs` entry.
+        stop_cause: Option<StopCause>,
         prompt_tokens: usize,
         completion_tokens: usize,
     },

pegainfer-frontend/src/engine/stop.rs

@@ -0,0 +1,94 @@
+/// How a request treats end-of-sequence tokens.
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+pub enum EosPolicy {
+    /// Do not stop on model EOS tokens.
+    Ignore,
+    /// Use the model executor's configured EOS set.
+    #[default]
+    ModelDefault,
+    /// Stop only on this protocol-provided primary EOS token.
+    Token(u32),
+}
+
+/// Request-scoped token stopping policy.
+///
+/// EOS is kept separate from caller stop tokens because the vLLM protocol
+/// reports them differently: EOS has no 'stop_reason', while a request stop
+/// reports the actual matching token ID.
+#[derive(Clone, Debug, Default, Eq, PartialEq)]
+pub struct StopPolicy {
+    pub eos: EosPolicy,
+    pub token_ids: Vec<u32>,
+}
+
+impl StopPolicy {
+    /// Classify a token using vLLM's priority: EOS first, then the request's
+    /// explicit stop-token set.
+    #[must_use]
+    pub fn classify(
+        &self,
+        token_id: u32,
+        is_model_eos: impl FnOnce(u32) -> bool,
+    ) -> Option<StopCause> {
+        let is_eos = match self.eos {
+            EosPolicy::Ignore => false,
+            EosPolicy::ModelDefault => is_model_eos(token_id),
+            EosPolicy::Token(eos_token_id) => token_id == eos_token_id,
+        };
+
+        if is_eos {
+            Some(StopCause::Eos(token_id))
+        } else if self.token_ids.contains(&token_id) {
+            Some(StopCause::Token(token_id))
+        } else {
+            None
+        }
+    }
+}
+
+/// The token-level cause of a normal stop finish.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum StopCause {
+    /// A primary or model-default EOS token.
+    Eos(u32),
+    /// A token from the request's explicit stop-token set.
+    Token(u32),
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn model_default_classifies_model_eos() {
+        let policy = StopPolicy::default();
+
+        assert_eq!(
+            policy.classify(99, |token_id| token_id == 99),
+            Some(StopCause::Eos(99))
+        );
+    }
+
+    #[test]
+    fn ignored_eos_does_not_disable_an_explicit_stop() {
+        let policy = StopPolicy {
+            eos: EosPolicy::Ignore,
+            token_ids: vec![99],
+        };
+
+        assert_eq!(
+            policy.classify(99, |token_id| token_id == 99),
+            Some(StopCause::Token(99))
+        );
+    }
+
+    #[test]
+    fn eos_wins_when_the_same_id_is_also_an_explicit_stop() {
+        let policy = StopPolicy {
+            eos: EosPolicy::Token(99),
+            token_ids: vec![99],
+        };
+
+        assert_eq!(policy.classify(99, |_| false), Some(StopCause::Eos(99)));
+    }
+}