Repository navigation
[MLX] Avoid standalone target forwards between DFlash rounds - #25337
Merged
Merged
Conversation
…ash window correction, and DFlash2 support with CI coverage. [ghstack-poisoned]
…ash window correction, and DFlash2 support with CI coverage. [ghstack-poisoned]
…ash window correction, and DFlash2 support with CI coverage. [ghstack-poisoned]
…ash window correction, and DFlash2 support with CI coverage. [ghstack-poisoned]
[ghstack-poisoned]
Contributor
Author
|
Stack from ghstack (oldest at bottom): |
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25337
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
…overage; rebase the performance follow-up and refresh the complete stack ordering. [ghstack-poisoned]
…overage; rebase the performance follow-up and refresh the complete stack ordering. [ghstack-poisoned]
mergennachin
added a commit
that referenced
this pull request
Nov 20, 2026
MLX DFlash decoding currently runs a standalone target forward before each draft and verification round. For eligible singleton token batches, draft directly from the uncommitted input token and let verification commit the anchor and accepted proposals. Preserve the existing path for mixed inputs, empty pending context, and context tails. This applies to both original DFlash and DFlash2. Verify greedy proposals against the target sample already drawn for each absolute position. This keeps RNG selection stable across prompt chunking and batch composition when the shortcut is enabled, and removes retained vocabulary probability arrays. The original baseline passes the schedule test; the inconsistency occurs in the shortcut without this sampling change. Native sampling, batching, and serialized DFlash suites pass, including temperature 0.8 distributions, top-k/top-p, cache lifecycle, shared-payload input offsets, and fallback paths. An input-sensitive original-DFlash fixture also exercises target-only/speculative parity, reset, clone, and rewind. A wrong-anchor mutation passes the old continuation test and fails the new slice checks for both draft formats. The same 96 schedule cases pass on the unchanged baseline, sampling-only control, and combined implementation; the shortcut before the sampling change fails for both draft formats. Lintrunner and git diff --check pass. Measured actual serving through the unmodified loopback `/v1/chat/completions` endpoint on M5 Max (64 GB), with full Q4_K_M target/draft models and FP16 activations. Baseline is the parent stack (83884a8004); production workers differ only in the MLX executor. One active request, four short prompts, 64 generated tokens, three measured repetitions per prompt after excluded warmups (288 measured requests), seed 42, top-k 0/top-p 1, `ignore_eos=true`, greedy drafts and target temperature 0 or 0.8. Fresh sessions, no prefix reuse; variant order AB/BA/AB. Model startup and cooling pauses between groups are excluded. These rates describe warmed single requests under controlled starting temperatures; sustained server capacity was not measured. Server effective decode uses `(output_tokens - 1) / native_decode_seconds`, from the native runner's first output batch to completion. Speculative bursts give multiple tokens the same timestamp, including any burst delivered with the first token. HTTP end-to-end throughput includes preparation, tokenization, scheduling, inference and streaming through `[DONE]`. Both are pooled token counts divided by pooled latency; SSE fragments are not counted as tokens. | Drafter | Temperature | Proposals | Server effective decode tok/s, before → after | HTTP end-to-end tok/s, before → after | |---|---:|---:|---:|---:| | DFlash | 0 | 3 | 48.51 → 59.85 (+23.4%) | 42.65 → 50.88 (+19.3%) | | DFlash | 0 | 7 | 60.49 → 80.53 (+33.1%) | 51.23 → 64.69 (+26.3%) | | DFlash | 0 | 15 | 84.50 → 108.88 (+28.8%) | 66.90 → 81.14 (+21.3%) | | DFlash | 0.8 | 3 | 47.33 → 59.24 (+25.2%) | 41.70 → 50.40 (+20.8%) | | DFlash | 0.8 | 7 | 60.05 → 75.39 (+25.5%) | 50.78 → 61.31 (+20.7%) | | DFlash | 0.8 | 15 | 64.43 → 91.26 (+41.6%) | 53.72 → 70.81 (+31.8%) | | DFlash2 | 0 | 3 | 48.00 → 62.00 (+29.2%) | 41.82 → 51.99 (+24.3%) | | DFlash2 | 0 | 7 | 67.08 → 94.85 (+41.4%) | 55.44 → 72.59 (+30.9%) | | DFlash2 | 0 | 15 | 87.19 → 118.54 (+35.9%) | 67.97 → 85.07 (+25.2%) | | DFlash2 | 0.8 | 3 | 45.88 → 59.07 (+28.8%) | 40.20 → 49.84 (+24.0%) | | DFlash2 | 0.8 | 7 | 62.68 → 85.47 (+36.4%) | 52.27 → 66.85 (+27.9%) | | DFlash2 | 0.8 | 15 | 86.35 → 94.84 (+9.8%) | 67.21 → 72.06 (+7.2%) | GPU frequency/activity/temperature and macOS thermal state were sampled every 200 ms. All measured request-window samples reported nominal thermal pressure, AC power and low power mode disabled, with 100.0% telemetry coverage. Maximum measured GPU temperature was 81.6 C. Per-case GPU active-time-weighted clocks ranged 1.490–1.510 GHz for baseline and 1.601–1.610 GHz for candidate (+6.3% to +8.0% paired differences). Within a fixed case and variant, repetition-average clocks varied by at most 0.8%. Clocks were monitored, not locked, so these are whole-system serving gains. An exploratory warmer run triggered fair thermal pressure and clocks around 1.0 GHz; it is excluded in full. Each group started after two seconds of nominal thermal state with GPU <=60 C / CPU <=70 C. These are experimental starting conditions, not hardware throttling thresholds. All greedy visible HTTP responses match baseline, and seeded visible-text replay is stable in each variant across all conditions. Sampled old/new responses can differ at the same seed because target RNG coupling changes, so sampled gains include different token streams and acceptance patterns. Per-prompt HTTP means below baseline in this corpus: DFlash2, T=0.8, N=15, explanation: -6.7%; DFlash2, T=0.8, N=15, planning: -4.0%. A separate earlier, unmonitored native DFlash2/N=15/T=0.8 matched-sampler control produces identical tokens and improves prefill-plus-generation throughput 24.3% (warm decode 31.6%); that control is not an HTTP measurement. These means cover a small corpus with three repetitions, and longer FP16 runs have existing near-tie differences across verification shapes. Concurrency was not benchmarked. CUDA code and interfaces are unchanged. Follow-up to #24811, tracked in #23011. Stacked on #25336. Authored with AI assistance from OpenAI Codex. ghstack-source-id: 920071f ghstack-comment-id: 7033163531 Pull-Request: #25337
This was referenced Nov 20, 2026
…nd update the wheel ownership probe, and guard the MLX-only export test in builds without generated MLX files. Rebase the full stack onto fetched origin/main. [ghstack-poisoned]
….py for Buck, restore GGUF draft capacity with regression coverage, and clarify native test oracles. Rebase the remaining stack after #25335 merged. [ghstack-poisoned]
….py for Buck, restore GGUF draft capacity with regression coverage, and clarify native test oracles. Rebase the remaining stack after #25335 merged. [ghstack-poisoned]
mergennachin
added a commit
that referenced
this pull request
Nov 21, 2026
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included. Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged. The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON. M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency. | Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain | |---|---:|---:|---:|---:|---:|---:| | Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% | | Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% | | dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% | | dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% | | dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% | | dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% | | dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% | | dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% | 192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison. Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass. Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review. Authored with AI assistance from OpenAI Codex. ghstack-source-id: fe2526c ghstack-comment-id: 7061815924 Pull-Request: #25410
[ghstack-poisoned]
mergennachin
added a commit
that referenced
this pull request
Nov 21, 2026
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included. Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged. The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON. M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency. | Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain | |---|---:|---:|---:|---:|---:|---:| | Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% | | Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% | | dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% | | dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% | | dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% | | dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% | | dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% | | dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% | 192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison. Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass. Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review. Authored with AI assistance from OpenAI Codex. ghstack-source-id: 59cdb6e ghstack-comment-id: 7061815924 Pull-Request: #25410
This PR needs a
|
mergennachin
added a commit
that referenced
this pull request
Nov 21, 2026
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included. Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged. The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON. M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency. | Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain | |---|---:|---:|---:|---:|---:|---:| | Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% | | Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% | | dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% | | dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% | | dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% | | dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% | | dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% | | dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% | 192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison. Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass. Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review. Authored with AI assistance from OpenAI Codex. ghstack-source-id: 59cdb6e ghstack-comment-id: 7061815924 Pull-Request: #25410
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MLX DFlash decoding currently runs a standalone target forward before each draft and verification round. For eligible singleton token batches, draft directly from the uncommitted input token and let verification commit the anchor and accepted proposals. Preserve the existing path for mixed inputs, empty pending context, and context tails. This applies to both original DFlash and DFlash2.
Verify greedy proposals against the target sample already drawn for each absolute position. This keeps RNG selection stable across prompt chunking and batch composition when the shortcut is enabled, and removes retained vocabulary probability arrays. The original baseline passes the schedule test; the inconsistency occurs in the shortcut without this sampling change.
Native sampling, batching, and serialized DFlash suites pass, including temperature 0.8 distributions, top-k/top-p, cache lifecycle, shared-payload input offsets, and fallback paths. An input-sensitive original-DFlash fixture also exercises target-only/speculative parity, reset, clone, and rewind. A wrong-anchor mutation passes the old continuation test and fails the new slice checks for both draft formats. The same 96 schedule cases pass on the unchanged baseline, sampling-only control, and combined implementation; the shortcut before the sampling change fails for both draft formats. Lintrunner and git diff --check pass.
Measured actual serving through the unmodified loopback
/v1/chat/completionsendpoint on M5 Max (64 GB), with full Q4_K_M target/draft models and FP16 activations. Baseline is the parent stack (83884a8004); production workers differ only in the MLX executor. One active request, four short prompts, 64 generated tokens, three measured repetitions per prompt after excluded warmups (288 measured requests), seed 42, top-k 0/top-p 1,ignore_eos=true, greedy drafts and target temperature 0 or 0.8. Fresh sessions, no prefix reuse; variant order AB/BA/AB. Model startup and cooling pauses between groups are excluded. These rates describe warmed single requests under controlled starting temperatures; sustained server capacity was not measured.Server effective decode uses
(output_tokens - 1) / native_decode_seconds, from the native runner's first output batch to completion. Speculative bursts give multiple tokens the same timestamp, including any burst delivered with the first token. HTTP end-to-end throughput includes preparation, tokenization, scheduling, inference and streaming through[DONE]. Both are pooled token counts divided by pooled latency; SSE fragments are not counted as tokens.GPU frequency/activity/temperature and macOS thermal state were sampled every 200 ms. All measured request-window samples reported nominal thermal pressure, AC power and low power mode disabled, with 100.0% telemetry coverage. Maximum measured GPU temperature was 81.6 C. Per-case GPU active-time-weighted clocks ranged 1.490–1.510 GHz for baseline and 1.601–1.610 GHz for candidate (+6.3% to +8.0% paired differences). Within a fixed case and variant, repetition-average clocks varied by at most 0.8%. Clocks were monitored, not locked, so these are whole-system serving gains. An exploratory warmer run triggered fair thermal pressure and clocks around 1.0 GHz; it is excluded in full. Each group started after two seconds of nominal thermal state with GPU <=60 C / CPU <=70 C. These are experimental starting conditions, not hardware throttling thresholds.
All greedy visible HTTP responses match baseline, and seeded visible-text replay is stable in each variant across all conditions. Sampled old/new responses can differ at the same seed because target RNG coupling changes, so sampled gains include different token streams and acceptance patterns. Per-prompt HTTP means below baseline in this corpus: DFlash2, T=0.8, N=15, explanation: -6.7%; DFlash2, T=0.8, N=15, planning: -4.0%. A separate earlier, unmonitored native DFlash2/N=15/T=0.8 matched-sampler control produces identical tokens and improves prefill-plus-generation throughput 24.3% (warm decode 31.6%); that control is not an HTTP measurement. These means cover a small corpus with three repetitions, and longer FP16 runs have existing near-tie differences across verification shapes. Concurrency was not benchmarked. CUDA code and interfaces are unchanged.
Follow-up to #24811, tracked in #23011. Stacked on #25336.
Authored with AI assistance from OpenAI Codex.