[RFC] Support score centering: additive rollout-training mismatch correction in bypass mode #8009

Implementation in PR #8010

Motivation

Every rollout correction in verl today is multiplicative: TIS / MIS / IcePop / KPop / TOPR / CPPO all reweight or mask the sampled token by a ratio of trainer and sampler probabilities. Score Centering Stabilizes Off-policy Reinforcement Learning (Marek & Ryabinin, 2026-09-17) shows the instability under training-inference mismatch comes from a "drift" term E_q[R] * E_q[∇log p] that is nonzero whenever sampler q differs from trainer p, and that it acts as distillation toward the (quantized / stale) sampler in a feedback loop. Score centering removes it additively: subtract the expected score under the sampler from each token's score. It has no hyperparameters, is a no-op on-policy, composes with token-level TIS / MIS, and in the paper is the only method that trains stably under severe sampler quantization (0.6B to 30B). The same paper reports SC + TIS / MIS as best under staleness.

verl has no additive correction of this kind, and no issue or PR proposes one.

Algorithm and references

• Paper: arXiv 2609.20807, Eq. 8 / 14 (top-k form, k = 128; k = 32 matches full-vocab centering in all their settings).
• Official code: https://github.com/martin-marek/score-centering (JAX; README carries a PyTorch reference loss).
• Prior integration: slime merged it on 2026-09-23 as [slime] Support score centering THUDM/slime#2405, Megatron + SGLang, REINFORCE only.

Per token, with sampler top-k head H, q sampler probs, p trainer probs, tail-mass ratio ρ = (1 − Σ_H q) / (1 − Σ_H p), IS rule f (1 for plain SC, min(r, c) for TIS, band mask for MIS), α = ρ f(1/ρ):

L = -A * ( sg[f(p_y/q_y)] * log p_y  -  Σ_{v∈H} sg[q_v f(p_v/q_v) - α p_v] * log p_v )

Only the k head log-probs of the trainer are needed; the tail is modeled as proportional to the trainer's distribution.

Proposed design

Score centering is a 2-policy objective (current trainer vs. sampler), which is exactly verl's bypass mode with loss_type="reinforce", so it goes there rather than as a new loss_mode.

• Rollout: actor_rollout_ref.rollout.topk_log_probs: int = 0. When set, the vLLM server requests logprobs=k instead of the hard-coded 0, raises max_logprobs (same as the distillation teacher does), and returns the top-k (ids, logprobs) per generated token. Batch keys rollout_topk_ids (int32) and rollout_topk_log_probs (fp32), laid out over the full sequence [B, P+R, k] exactly like teacher_ids / teacher_logprobs, including the padding template and the no-padding conversion. Requires calculate_log_probs=True, top_p=1, top_k=-1, logprobs_mode=processed_logprobs.
• Trainer: a logits_processor_func hook (the slot the distillation top-k loss already uses) gathers the trainer's log-probs at the head ids from the full logits and emits a per-token scalar sc_correction plus detached head-mass diagnostics. The gather uses a chunked, vocab-parallel-aware autograd function so memory stays O(chunk * V); incompatible with use_fused_kernels, like distillation.
• Loss / config: algorithm.rollout_correction.score_centering: bool = False, valid only with bypass_mode=True, loss_type="reinforce", rollout_is in {None, "token"}. compute_policy_loss_reinforce adds A * sc_correction after the existing IS weighting; the head coefficients reuse the same rollout_is_threshold rule (float = TIS clamp, "lo_hi" = IcePop / MIS). Presets bypass_pg_sc, bypass_pg_token_tis_sc, bypass_pg_token_icepop_sc. Metrics actor/sc_correction, actor/sc_sampler_head_mass, actor/sc_train_head_mass.

Plan

1. One PR, end to end on vLLM + FSDP: config, rollout plumbing, hook, loss, presets, docs, CPU tests (on-policy no-op, equality with exact full-vocab centering when k = V, drift cancellation under constant reward, TIS / MIS composition, output parsing and alignment), plus a 2-GPU smoke.
2. Follow-ups: SGLang rollout (top_logprobs_num), Megatron TP path, v1 trainer fields, sequence-level IS composition, top_p < 1 replay as in slime#2406.
3. Evidence for the PR: reward curves and rollout_probs_diff with an fp8 sampler, comparing bypass_pg vs +SC vs +TIS vs +TIS+SC.

Related but not overlapping: #7712 (generic auxiliary objectives; SC is a rollout correction, not a reward-side term) and #7856 (tail bucket for OPD; same tail-mass idea on the distillation side).