Skip to content

Commit 0973dd3

Browse files
committed
[MLX] Fuse native decode operators and save M5 measurements
Save the native MLX fusion experiments as an opt-in draft above #25337. Fuse FP16 SwiGLU/sigmoid gating, RoPE arithmetic, RMSNorm/residual addition and target logit soft cap, and share identical RoPE phase buffers during Muse Glimmer MLX export. Preserve native quantized matmuls, packed weights and existing rounding boundaries. All five export flags default off. No INT8 activation kernels, architecture dispatch, quantized-linear rewrites or MLX submodule changes are included. Review the shared runtime changes first: cache custom-kernel callables for the lifetime of a loaded program and allow empty outputs. Then review the guarded pattern matchers and exact-output tests, followed by the MLX-only phase-sharing hook. CUDA paths are unchanged. The saved serving results below compare gate-only against gate + RoPE + norm/add + soft cap + shared phases. They are incremental gains, not a single full-draft-versus-parent measurement. Both arms include the shared runtime fixes. Earlier gate-only versus unfused results and isolated kernel timings are preserved in `backends/mlx/benchmarks/native_decode_fusions.md`; machine-readable aggregates and checkpoint hashes are in the adjacent JSON. M5 Max, 64 GiB; serial HTTP serving, six prompts including 512/2048-token prefill, 64 generated tokens, seed 42, top-k 0/top-p 1, no prefix reuse. Two counterbalanced run-order blocks at temperatures 0 and 0.8. Effective decode is pooled `(completion_tokens - 1) / server decode seconds`; HTTP throughput includes prefill and full client latency. | Configuration | Temp | Retained pairs | Decode tok/s before -> after | Decode gain | Prefill gain | HTTP gain | |---|---:|---:|---:|---:|---:|---:| | Solo | 0 | 7/12 | 26.36 -> 26.91 | +2.10% | -1.27% | +1.60% | | Solo | 0.8 | 7/12 | 26.01 -> 26.69 | +2.59% | +0.51% | +2.11% | | dflash p7 | 0 | 12/12 | 89.96 -> 91.71 | +1.95% | +1.92% | +1.91% | | dflash p7 | 0.8 | 11/12 | 78.41 -> 79.26 | +1.09% | +1.92% | +1.45% | | dflash2 p7 | 0 | 11/12 | 99.64 -> 101.70 | +2.06% | +1.32% | +1.67% | | dflash2 p7 | 0.8 | 11/12 | 88.47 -> 90.79 | +2.62% | +1.53% | +2.12% | | dflash2 p15 | 0 | 10/12 | 123.51 -> 124.85 | +1.08% | +1.43% | +1.28% | | dflash2 p15 | 0.8 | 11/12 | 103.82 -> 106.64 | +2.72% | +1.41% | +2.06% | 192 measured requests plus 192 warmups completed; all 96 paired visible outputs match. 80 pairs meet the predeclared GPU-active-weighted frequency difference <=1%, nominal thermals, AC/no-low-power, normal memory pressure and >=95% telemetry coverage. Frequencies are monitored, not hardware-locked. Both retained order blocks have positive aggregate decode/HTTP gains, but individual workloads and solo prefill are mixed. Two repetitions per prompt and different retained cohorts limit conclusions; this is not a proposal-count comparison. Validation: 25 gate plus 44 rotary/norm/add/soft-cap exported exact-byte cases, 18 full-vocabulary logit comparisons and 36 speculative token/acceptance comparisons per experiment, plus full constant/graph audits. Coverage includes empty/dynamic/strided tensors, extreme finite FP16 values, large/reset RoPE positions, exposed intermediates and fallback dtypes. The native-only snapshot reruns all 69 operator cases, the custom-kernel cache runtime tests and the existing offgraph suite. Formatting, flake8 and diff checks pass. Follow-up experiments for #24811, tracked in #23011. Kept as a draft to preserve code and results pending review. Authored with AI assistance from OpenAI Codex. ghstack-source-id: fe2526c ghstack-comment-id: 7061815924 Pull-Request: #25410
1 parent ab3198f commit 0973dd3

20 files changed

Lines changed: 2007 additions & 10 deletions

‎backends/mlx/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,20 @@ lists, graph dumps, per-node support decisions, and serialization details:
9191
ET_MLX_DEBUG=1 python -m executorch.backends.mlx.examples.llm.export_llm_hf ...
9292
```
9393

94+
### Experimental decode fusions
95+
96+
`ET_MLX_FUSE_GATES=1` experimentally fuses FP16 SwiGLU and sigmoid gating while
97+
retaining native MLX matmuls and FP16 rounding boundaries. It supports strided
98+
inputs; shared activation outputs and broadcasting retain their existing paths.
99+
100+
Additional opt-in fusions are `ET_MLX_FUSE_ROPE=1` (FP32 rotation around FP16
101+
activations), `ET_MLX_FUSE_NORM_ADD=1` (FP16 RMSNorm and residual addition), and
102+
`ET_MLX_FUSE_SOFT_CAP=1` (FP32 logit soft cap). Muse Glimmer's MLX DFlash export
103+
also accepts `ET_MLX_SHARE_ROPE_PHASE=1` to share identical RoPE phase calculations.
104+
These preserve the native quantized matmuls and remain disabled by default.
105+
See the [saved M5 measurements](benchmarks/native_decode_fusions.md) for results
106+
and validation limits.
107+
94108
---
95109

96110
## Directory Layout

0 commit comments

Comments
 (0)