Skip to content

[MLX] Fuse native decode operators and save M5 measurements - #25410

Draft
mergennachin wants to merge 21 commits into
mainfrom
gh/mergennachin/51/head
Draft

mergennachin wants to merge 21 commits into
mainfrom
gh/mergennachin/51/head

Conversation

@mergennachin

Copy link
Copy Markdown
Contributor

Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included.

Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged.

The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in backends/mlx/benchmarks/native_decode_fusions.md; machine-readable aggregates and checkpoint hashes are in the adjacent JSON.

M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled (completion_tokens - 1) / server decode seconds; HTTP throughput includes prefill and full client latency.

Configuration Temp Retained pairs Decode tok/s before -> after Decode gain Prefill gain HTTP gain
Solo 0 7/12 26.36 -> 26.91 +2.10% -1.27% +1.60%
Solo 0.8 7/12 26.01 -> 26.69 +2.59% +0.51% +2.11%
dflash p7 0 12/12 89.96 -> 91.71 +1.95% +1.92% +1.91%
dflash p7 0.8 11/12 78.41 -> 79.26 +1.09% +1.92% +1.45%
dflash2 p7 0 11/12 99.64 -> 101.70 +2.06% +1.32% +1.67%
dflash2 p7 0.8 11/12 88.47 -> 90.79 +2.62% +1.53% +2.12%
dflash2 p15 0 10/12 123.51 -> 124.85 +1.08% +1.43% +1.28%
dflash2 p15 0.8 11/12 103.82 -> 106.64 +2.72% +1.41% +2.06%

192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison.

Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass.

Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review.

Authored with AI assistance from OpenAI Codex.

…ash window correction, and DFlash2 support with CI coverage.

[ghstack-poisoned]
…ash window correction, and DFlash2 support with CI coverage.

[ghstack-poisoned]
…ash window correction, and DFlash2 support with CI coverage.

[ghstack-poisoned]
…ash window correction, and DFlash2 support with CI coverage.

[ghstack-poisoned]
[ghstack-poisoned]
…overage; rebase the performance follow-up and refresh the complete stack ordering.

[ghstack-poisoned]
…overage; rebase the performance follow-up and refresh the complete stack ordering.

[ghstack-poisoned]
…nd update the wheel ownership probe, and guard the MLX-only export test in builds without generated MLX files. Rebase the full stack onto fetched origin/main.

[ghstack-poisoned]
…nd update the wheel ownership probe, and guard the MLX-only export test in builds without generated MLX files. Rebase the full stack onto fetched origin/main.

[ghstack-poisoned]
…nd update the wheel ownership probe, and guard the MLX-only export test in builds without generated MLX files. Rebase the full stack onto fetched origin/main.

[ghstack-poisoned]
…nd update the wheel ownership probe, and guard the MLX-only export test in builds without generated MLX files. Rebase the full stack onto fetched origin/main.

[ghstack-poisoned]
…nd update the wheel ownership probe, and guard the MLX-only export test in builds without generated MLX files. Rebase the full stack onto fetched origin/main.

[ghstack-poisoned]
…he native FP16 fusion experiments as a draft.

[ghstack-poisoned]
…he native FP16 fusion experiments as a draft.

[ghstack-poisoned]
…he native FP16 fusion experiments as a draft.

[ghstack-poisoned]
…the native FP16 fusion experiments as a draft.

[ghstack-poisoned]
@mergennachin

mergennachin commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

@pytorch-bot

pytorch-bot Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25410

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 28 Cancelled Jobs, 5 Pending, 12 Unclassified Failures

As of commit 704bbb4 with merge base f63b372 (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

….py for Buck, restore GGUF draft capacity with regression coverage, and clarify native test oracles. Rebase the remaining stack after #25335 merged.

[ghstack-poisoned]
….py for Buck, restore GGUF draft capacity with regression coverage, and clarify native test oracles. Rebase the remaining stack after #25335 merged.

[ghstack-poisoned]
….py for Buck, restore GGUF draft capacity with regression coverage, and clarify native test oracles. Rebase the remaining stack after #25335 merged.

[ghstack-poisoned]
mergennachin added a commit that referenced this pull request Nov 21, 2026
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included.

Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged.

The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON.

M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency.

| Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain |
|---|---:|---:|---:|---:|---:|---:|
| Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% |
| Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% |
| dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% |
| dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% |
| dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% |
| dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% |
| dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% |
| dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% |

192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison.

Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass.

Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review.

Authored with AI assistance from OpenAI Codex.
ghstack-source-id: fe2526c
ghstack-comment-id: 7061815924
Pull-Request: #25410
[ghstack-poisoned]
[ghstack-poisoned]
mergennachin added a commit that referenced this pull request Nov 21, 2026
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included.

Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged.

The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON.

M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency.

| Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain |
|---|---:|---:|---:|---:|---:|---:|
| Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% |
| Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% |
| dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% |
| dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% |
| dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% |
| dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% |
| dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% |
| dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% |

192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison.

Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass.

Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review.

Authored with AI assistance from OpenAI Codex.
ghstack-source-id: 59cdb6e
ghstack-comment-id: 7061815924
Pull-Request: #25410
mergennachin added a commit that referenced this pull request Nov 21, 2026
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included.

Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged.

The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON.

M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency.

| Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain |
|---|---:|---:|---:|---:|---:|---:|
| Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% |
| Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% |
| dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% |
| dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% |
| dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% |
| dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% |
| dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% |
| dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% |

192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison.

Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass.

Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review.

Authored with AI assistance from OpenAI Codex.
ghstack-source-id: 59cdb6e
ghstack-comment-id: 7061815924
Pull-Request: #25410
Base automatically changed from gh/mergennachin/50/head to main November 21, 2026 16:28

This branch was previously deployed

1 inactive deployment
cadence — 704bbb40 Deployed Nov 21, 2026 by mergennachin via hifi-op-test / hifi4 #39967
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant