Skip to content

[executorch][cuda] NVFP4 linear and embedding dispatch - #25342

Open
Gasoonjia wants to merge 5 commits into
gh/gasoonjia/217/basefrom
gh/gasoonjia/217/head
Open

Gasoonjia wants to merge 5 commits into
gh/gasoonjia/217/basefrom
gh/gasoonjia/217/head

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Adds eager and export-time linear/embedding dispatch for CudaNVFP4Tensor, compile-time precision and split-K selection, unit-scale integration, and a configurable prefill output dtype. Gemma CUDA weight packers accept NVFP4 tensors.

PTE tests exercise serialized runtime programs, dynamic input sizes, fixed-prefill mode selection, split-K, per-output-row scales, and unit-scale behavior.

Authored with AI assistance (agentcloud).

Differential Revision: D132997584

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25342

Note: Links to docs will display an error until the docs builds have been completed.

❌ 30 Pending, 2 Unclassified Failures

As of commit 0906c7c with merge base 9f49fe2 (image):

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 32m 0s —— View job


Request Changes: the new NVFP4 dispatch has no SM89 guard, makes importing quantize_op_dispatch require Triton, picks W4A4 vs W4A16 numerics by autotune timing behind undocumented env vars, and none of its new GPU tests run on any CI runner.

PR Review: #25342

Summary

This PR adds eager and export F.linear/F.embedding dispatch for CudaNVFP4Tensor. That covers autotuned W4A4 vs W4A16 and split-K, unit-scale GEMV, and an optional fp32 prefill output. It also adds NVFP4 routing in the Gemma 4 CUDA packers, plus PTE tests. The issues below need fixing or discussion before this lands.

Backward Compatibility

  • import executorch.backends.cuda.quantize_op_dispatch now requires Triton.
    • nvfp4_dispatch.py:17-28 imports nvfp4_output, nvfp4_quantized_gemm, nvfp4_tuning and quantized_gemm_family at module level.
    • Any one of those runs backends/cuda/triton/kernels/__init__.py, which eagerly imports conv/fused_moe/int4_matmul/sdpa/topk, and with them triton.
    • On the base branch, no dispatch or tensor module in this package imports Triton. int4_dispatch.py–int8_dispatch.py defer their kernel imports into the handlers. The function-level imports at :51, :74, :132 and :196 follow the same pattern, but the module-level imports defeat them.
    • CPU-only and macOS torch wheels don't ship Triton. CPU-side packing code that imports the package will fail at import there, e.g. examples/models/gemma4_31b/tests/test_cuda_packers.py:18 and quant/tests/test_pack_cuda.py:18.
    • CI won't catch this, because the only job that runs those files is on A10G, which has Triton.
    • Fix:
      • Move _MODE_*, _prefill, _decode and the triton::nvfp4_auto_* ops into nvfp4_quantized_gemm.py, next to the other triton::nvfp4_* ops.
      • Inline nvfp4_output.py into the dispatch. It contains no kernel and has a single consumer.
    • Fix this →
  • gemma4_31b:quant doesn't declare //executorch/backends/cuda:nvfp4_tensor.
    • cuda_packers.py:62 and :137 now import CudaNVFP4Tensor unconditionally at function entry.
    • Any Buck consumer of :quant that doesn't get nvfp4_tensor some other way now fails for every weight type (INT4/INT5/INT6/INT8/GGUF), not just NVFP4.
    • The new test_nvfp4_packers target declares the dep itself, which hides the gap.
    • Add the dep next to coalesced_int4_tensor in examples/models/gemma4_31b/BUCK:17-19. Fix this →

API Design

  • The default auto policy picks numerics by timing.
    • On SM120, nvfp4_auto_prefill/nvfp4_auto_decode (nvfp4_dispatch.py:87-111) let AOTI autotuning choose between W4A16 and W4A4 for each layer. W4A4 quantizes activations to FP4.
    • Accuracy therefore depends on which candidate was faster on the export machine. The choice can differ between layers, between prefill and decode, and between two exports of the same checkpoint, and the user can't see it.
    • Precision should be an explicit, deterministic choice. Default to W4A16, make W4A4 opt-in through a tensor attribute (like act_per_tensor_scale/unit_scale) or a recipe, and limit autotuning to split-K within the chosen precision.
  • Undocumented process-global env vars change numerics and output dtype at trace time.
    • ET_CUDA_NVFP4_ACT is read from os.environ inside the handler (nvfp4_dispatch.py:119).
    • ET_CUDA_NVFP4_PREFILL_OUTPUT is read behind assume_constant_result (nvfp4_output.py:17).
    • Neither is recorded on the tensor or in a compile spec, and no doc or README mentions either one.
    • The fp32-output contract is also applied unevenly:
      • The fallback at nvfp4_dispatch.py:130 ignores output_dtype, so CPU eager returns bf16 where CUDA/export returns fp32.
      • Static-small decode keeps the input dtype while prefill returns fp32, so one layer's output dtype differs between the prefill and decode methods.
      • fp32 activations reach the kernels only under this policy, and _launch_w4a16_impl (nvfp4_quantized_gemm.py:970-971) then rounds them to bf16.
    • Suggest turning both settings into tensor attributes or recipe/compile-spec options, and documenting them.
    • The one-line module docstring at nvfp4_dispatch.py:5 and the package docstring in quantize_op_dispatch/__init__.py (which still lists only INT4–INT8) should also cover NVFP4.

Infrastructure

  • None of the new GPU/PTE tests run in any CI.
    • Every new GPU test skips below SM89 (e.g. test_nvfp4_export.py:29-32, test_nvfp4_fusion.py:24-27). The W4A4, auto, unit-scale and fixed-prefill cases also skip without SM120.
    • The new fbcode test_nvfp4_dispatch target runs on subplatform = "A100" (SM80), and its PTE tests return under is_fbcode() before executing.
    • OSS runs backends/cuda/tests on A10G (SM86, .github/workflows/cuda.yml:529).
    • Only the CPU max_norm test and the packer tests actually execute. Kernel numerics, dispatch routing, split-K export and runtime execution are not checked by any automation.
    • Either add an SM89+/SM120 runner or subplatform, or state in the test plan that these tests are manual-only.
  • test_nvfp4_packers makes a backend test target depend on an example target (//executorch/examples/models/gemma4_31b:quant). It only exercises cuda_packers.py, so it belongs in examples/models/gemma4_31b/tests/test_cuda_packers.py. Fix this →

Testing

  • New paths with no coverage:
    • The _embedding success path is untested: CUDA F.embedding/nn.Embedding, the swapped-argument aten.embedding branch (nvfp4_dispatch.py:193), export/PTE, and the CPU fallback. test_embedding_bias_dispatch calls the raw nvfp4_embedding kernel, and the only test that reaches _embedding is the CPU max_norm rejection (test_nvfp4_fusion.py:33-46).
    • The default path on SM89/SM90 (auto → nvfp4_w4a16, :167-170) is never exported through F.linear with dynamic M. The dispatch PTE cases that run on those GPUs are all static M=1 (GEMV), and the auto cases skip without native FP4.
    • No test sets ET_CUDA_NVFP4_PREFILL_OUTPUT=fp32.
    • The packers' ExportableNVFP4Tensor → from_exportable branch is untested.
    • Every fixture uses register_buffer, but the packers produce an nn.Parameter, which goes through detach. Should one PTE case use a Parameter?
    • Fix this →
  • Checks that can't fail, and fixture hygiene:
    • The post-load env changes (test_nvfp4_export.py:137-146, test_nvfp4_fixed_prefill.py:88) can never fail, because nothing in the compiled .so reads ET_CUDA_NVFP4_ACT.
    • Uniform block scales (test_nvfp4_export.py:40, test_nvfp4_split_k_export.py:40) would hide block-scale indexing bugs across split-K. Use random E4M3 bytes, as test_nvfp4_unit_scale.py:36 does.
    • test_nvfp4_fixed_prefill.py:23 imports PackedLinear from another test module.
    • test_nvfp4_fixed_prefill.py:37-38/107-111 saves and restores the env by hand even though the test already takes monkeypatch.
    • test_nvfp4_unit_scale.py:145 hardcodes // 64 instead of using _MODE_STRIDE.
    • Fix this →

Performance

  • The fallbacks dequantize the whole weight.
    • _linear (nvfp4_dispatch.py:130) calls F.linear(flat, w.dequantize(x.dtype)) instead of chunked_dequant_linear (_gemm_family_dispatch.py:67). Every INT4–INT8 dispatch uses that helper, and it exists to avoid ~10 GiB of temporaries on the 262144-row lm_head.
    • _embedding (:195) dequantizes the full table just to gather a few rows. dequantize builds int64 codes plus fp32 per-element scales, which comes to ~20 GB of temporaries for a 262144×5376 table.
    • Triggers: CPU eager, and fp32 activations on CUDA outside the fp32-prefill policy. Under export, the full dequant is traced into the graph.
    • Fix this →

Code Quality

  • No SM89 gate before the Triton kernels.
    • On SM80/SM86 (A100, A10G), bf16/fp16 inputs still take the Triton branch. can_launch_triton only checks for a CUDA or fake tensor, and _has_native_fp4() == False just switches the mode to W4A16.
    • That path reaches nvfp4_gemv/nvfp4_w4a16 and nvfp4_embedding, which use tl.float8e4nv. Triton rejects that type below SM89, which is why this PR's own tests skip there.
    • The result is a Triton error during eager calls and AOTI compilation, where the code should fall back instead. That contradicts _gemm_family_dispatch.py:19 ("dispatch never raises for them").
    • Add an assume_constant_result capability ≥ (8, 9) check that takes the fallback or raises a clear error. Fix this →
  • detach/alias/t/_to_copy are registered in the dispatch module (nvfp4_dispatch.py:205-219), even though they define CudaNVFP4Tensor semantics.
    • aoti_packed_int4_tensor.py:139-151 registers the same handlers next to its class.
    • As written, ops like w.t() behave differently depending on whether quantize_op_dispatch has been imported, and the packers create nn.Parameters without importing it.
    • Fix this →
  • _has_native_fp4 (nvfp4_dispatch.py:33-35) duplicates _has_native_fp4_conversion (nvfp4_quantized_gemm.py:114-116). Is major >= 12 meant to exclude SM100/SM103 (B200/GB300) from W4A4? Fix this →
  • The NVFP4 branch is duplicated in pack_linear_for_cuda and pack_embedding_for_cuda (cuda_packers.py:70-84 and :145).
    • Neither the module docstring nor the function docstrings mention NVFP4.
    • The comment at :71-72 says the loader "pairs it by name and calls to_exportable_nvfp4_tensor itself". However, no non-test code calls to_exportable_nvfp4_tensor (defined at extension/llm/export/gguf.py:424). Does [executorch][cuda] NVFP4 fused gated GEMM and dispatch #25343 add that loader path? If not, GGUF NVFP4 always raises here.
    • Fix this →

Recommendation

Request Changes

Concrete fixes needed:

  • the missing SM89 guard, which is a hard failure on A100/A10G-class GPUs
  • the package-wide Triton import
  • the missing Buck dep
  • the full-weight dequant fallbacks

Needs discussion:

  • precision selected by timing
  • configuration through env vars

None of the new GPU tests run in CI, so none of the kernel or dispatch behavior in this PR is checked by automation.

Review steps
  • Gather PR context and diff
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

…code down so A100 CI runs the NVFP4 kernels

[ghstack-poisoned]
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Thanks, addressed:

  • Removed the unused nvfp4_prefill_fp32_enabled import here; it moves to [executorch][cuda] NVFP4 fused gated GEMM and dispatch #25343, where it is used.
  • examples/models/gemma4_31b:quant now depends on //executorch/backends/cuda:nvfp4_tensor.
  • The non-Triton _linear fallback honors the prefill output dtype, and the embedding fallback dequantizes only the gathered rows. A CPU test covers both.
  • Docs: the nvfp4_dispatch module docstring documents ET_CUDA_NVFP4_ACT and ET_CUDA_NVFP4_PREFILL_OUTPUT, and the quantize_op_dispatch package docstring lists NVFP4. Dispatch now imports _has_native_fp4_conversion instead of duplicating it, so it is also False on ROCm.
  • Tests: added a CPU test for the packers' ExportableNVFP4Tensor branch; switched to monkeypatch instead of restoring the env by hand; use _MODE_STRIDE instead of // 64. The PTE tests import is_fbcode/Runtime inside the test (main's pattern), so ROCm collection no longer needs pandas, and they skip on HIP.
  • SM89 gate: the software E4M3 decode now lands in [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411 (right after [executorch][cuda] NVFP4 fused gated GEMM and dispatch #25343), so these kernels compile on SM80/SM86.
    Not changed:
  • auto picks precision by timing within NVFP4's accuracy contract (W4A4 is what ModelOpt calibrates for); ET_CUDA_NVFP4_ACT=w4a16 forces weight-only.
  • quantize_op_dispatch is only imported on CUDA paths that need Triton.
  • The env policy is fixed at export time by design.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 30m 24s —— View job


Request Changes: three dispatch paths have no test, including the w4a16 prefill that pre-Blackwell GPUs use, CI can't run any of the GPU tests, and the linear fallback still dequantizes the full weight at once (about 11 GB for Gemma 4's lm_head).

PR Review: #25342

Summary

This PR adds CudaNVFP4Tensor linear and embedding dispatch to quantize_op_dispatch. Linear chooses between GEMV, W4A16 and W4A4 using ET_CUDA_NVFP4_ACT, with auto deciding by AOTI timing; embedding uses nvfp4_embedding. The Gemma 4 packers now accept NVFP4.

The remaining issues are mostly test coverage and one memory hotspot. The author already answered two points, the auto policy and the SM89 ordering. Both stay below as non-blocking notes and only cover what is still open.

Code Quality

  • nvfp4_dispatch.py:133-175: this PR on its own breaks SM80/SM86.
    • can_launch_triton (_gemm_family_dispatch.py:38-45) accepts any CUDA or fake tensor, so A100/A10G route to nvfp4_gemv, nvfp4_w4a16 and nvfp4_embedding.
    • Those kernels decode E4M3 with tl.float8e4nv (nvfp4_quantized_gemm.py:680-683, 871, 1088-1092), which Triton only supports on SM89+.
    • The result is a compile error, not the fallback that _gemm_family_dispatch.py:19 promises ("dispatch never raises").
    • Non-blocking as long as [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411 lands with this PR. If it might not, add an SM89 capability check to this PR's own commit. Fix this →
  • nvfp4_dispatch.py:55, 78, 137, 209: the function-level imports defer nothing. Lines 31-34 already import nvfp4_quantized_gemm at module load. int4_dispatch.py:98 imports lazily only because nothing at its top level pulls in Triton. Move _check_split, nvfp4_gemv, nvfp4_w4a16, nvfp4_w4a4 and nvfp4_embedding into the top-level import. Fix this →
  • examples/models/gemma4_31b/cuda_packers.py: the NVFP4 branches. Fix this →
    • The 15-line NVFP4 block appears twice, at 70-84 and 145-159.
    • The comment at 71-72 and 146-147 says "the loader pairs it by name and calls to_exportable_nvfp4_tensor itself", but no loader does that. The only users of to_exportable_nvfp4_tensor are gguf.py and its test, and extension/llm/export/load.py has no NVFP4 handling.
    • As a result, a GGUF NVFP4 checkpoint run through convert_quantized_tensors_for_cuda hits this ValueError, and the user can't act on it from that API. If [executorch][cuda] NVFP4 fused gated GEMM and dispatch #25343 adds the loader, say so in the comment.
    • The module docstring (7-36) and the function docstrings (48-54, 128-136) don't mention NVFP4. The embedding docstring still says the weight is converted to "gatherable int8".
    • Fix: one _pack_nvfp4(module, w) -> bool helper, an accurate comment, and updated docstrings.

API Design

  • Needs a maintainer decision (non-blocking): auto precision. I read the reply that W4A4 is within NVFP4's accuracy contract. That reply doesn't cover this consequence:
    • With the default auto, eager CUDA always runs the first candidate, _MODE_SPLITS[0], which is W4A16 with split 1 (launch_params.py:43-46). The CPU fallback (nvfp4_dispatch.py:133-135) always dequantizes.
    • So evaluating a model in eager measures W4A16 numerics, while the exported program can run W4A4 in any layer, picked by timing on the export host.
    • Nothing in the tensor or the .pte records which choice was made. The same is true of ET_CUDA_NVFP4_PREFILL_OUTPUT, which changes the output dtype of every prefill linear.
    • Options:
      • Make W4A4 opt-in, for example only when act_per_tensor_scale holds a calibrated ModelOpt input_scale.
      • Or move the policy out of env vars into an explicit export option or a tensor attribute (like unit_scale).

Infrastructure

  • No GPU path in this PR runs in CI. Fix this →
    • OSS unittest-cuda runs on A10G (SM86, .github/workflows/cuda.yml), and the Buck test_nvfp4_dispatch and test_nvfp4_kernels targets run on A100 (SM80, backends/cuda/tests/targets.bzl).
    • Every CUDA test is gated on SM89 or on native FP4 (SM120), so only the CPU tests run: test_nvfp4_packers.py and the first two tests in test_nvfp4_fusion.py. In fbcode, the PTE tests also return before execution.
    • Even after [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411, these SM120-only paths never run in any CI: auto, w4a4, mode_split tuning, triton::nvfp4_auto_{prefill,decode}, unit-scale, and fixed-prefill.
    • An export-only routing test would cover the dispatch logic on the existing runners:
      • monkeypatch nvfp4_dispatch._has_native_fp4 to True;
      • run torch.export (no AOTI) for each mode with M=1 and with dynamic M;
      • assert which triton::nvfp4_* target lands in ep.graph.

Testing

  • Dispatch paths with no test: Fix this →
    • The w4a16 prefill branch (nvfp4_dispatch.py:172-175) never goes through export, AOTI or a PTE.
      • The w4a16 cases of test_packed_linear_pte (test_nvfp4_export.py:50) and test_fused_row_scale_pte (test_nvfp4_fusion.py:138) use static M=1, which routes to nvfp4_gemv instead.
      • Lines 143-148 force w4a16 on every non-SM120 GPU (H100, L40S, RTX 40xx), so this is the only prefill path on those GPUs.
      • Add ("w4a16", 16, True).
    • The CUDA embedding dispatch never runs. test_embedding_bias_dispatch (test_nvfp4_kernels.py:115-125) calls the raw nvfp4_embedding op. So the CUDA branch of _embedding (nvfp4_dispatch.py:209-215) and the swapped aten.embedding argument order (197-198) are never executed, and no export test contains an embedding.
    • ET_CUDA_NVFP4_PREFILL_OUTPUT=fp32 is only exercised on the CPU fallback (test_nvfp4_fusion.py:42-45).
      • No test passes output_dtype to the kernels.
      • Nothing checks that dynamic-M prefill returns fp32 while M≤4 graphs keep the input dtype.
      • Nothing checks that output_dtype survives nvfp4_auto_prefill through AOTI.
  • Test hygiene: Fix this →
    • test_nvfp4_unit_scale.py:91-92 skips the w4a16 case without native FP4, even though _gemv_original handles UNIT_SCALE in software (nvfp4_quantized_gemm.py:649-650). Skip only auto, as test_nvfp4_export.py:53 does. Until [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411 lands, also add the (8, 9) bound to that file's CUDA marker.
    • test_nvfp4_fixed_prefill.py:22 imports PackedLinear from another test module. Move it into nvfp4_test_utils, next to shards and portable_weight.
    • test_nvfp4_packers.py:49 and :63 pass an nn.Linear to pack_embedding_for_cuda. Use an nn.Embedding for that case.

Performance

  • nvfp4_dispatch.py:135: the linear fallback dequantizes the whole weight at once. Fix this →
    • CudaNVFP4Tensor.dequantize builds an int64 N×K code tensor (nvfp4_tensor.py:129). For Gemma 4 31B's 262144×5376 lm_head (examples/models/gemma4_31b/model.md:16), the codes alone take about 11 GB, before the float copy.
    • _gemm_family_dispatch.py:29-35 exists to prevent exactly this, and the int4/5/6/8 dispatches all route their fallback through chunked_dequant_linear.
    • Triggers:
      • real CPU eager;
      • fp32 activations under export without the fp32 policy, where the whole-weight dequant is traced into the AOTI graph.
    • Fix: use chunked_dequant_linear, building a CudaNVFP4Tensor per row range the same way the embedding CPU fallback (199-208) does.

Recommendation

Request Changes

Blocking:

  • Test the w4a16 dynamic-M export path and the CUDA embedding dispatch. Pre-Blackwell GPUs use these paths, and CI runs none of them.
  • Route the linear fallback through chunked_dequant_linear.

Non-blocking:

Review steps
  • Gather PR context and diff
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Round 2:

  • The linear fallback dequantizes in N chunks through chunked_dequant_linear. A shared _weight_rows helper (used by the embedding fallback too) builds each chunk. A CPU test forces 32-row chunks.
  • Untested paths now covered:
    • CPU routing tests trace with fake tensors and assert which op each policy emits: W4A16 prefill, GEMV decode, auto prefill/decode, W4A4, and the FP32-prefill output_dtype (prefill gets FP32, decode keeps BF16).
    • The CUDA embedding dispatch is tested for both F.embedding and the swapped aten.embedding argument order.
    • test_packed_linear_pte[w4a16-37-True] covers the dynamic-M W4A16 prefill through export, AOTI and PTE; it passes on the RTX 5090.
  • Hygiene:
  • Not changed:

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 42m 0s —— View job


Request Changes: no CI job runs the default NVFP4 auto ops (_prefill/_decode) or the new FP32 prefill output policy, and several new tests cannot fail.

PR Review: #25342

Summary

This PR adds CudaNVFP4Tensor dispatch for F.linear/F.embedding and NVFP4 support in the Gemma 4 CUDA packers. The dispatch routes to W4A16, W4A4, or an AOTI-timed auto mode, and adds an FP32 prefill output policy. At this commit every GPU test in the PR skips on CI hardware: Buck runs on A100 and OSS on A10G, and both are below the SM89 gate. That leaves only the CPU fake-tensor tests. The first two Testing gaps below stay open after #25411, because they need native FP4 or have no test at all. The rest is test robustness and cleanup.

Infrastructure

  • Some direct Buck deps are missing and only resolve transitively (minor):

    • backends/cuda/BUCK:93-101: quantize_op_dispatch doesn't list :autotune, but nvfp4_dispatch.py:22 imports autotune.launch_params. It's only reached through :triton_kernels.
    • examples/models/gemma4_31b/BUCK:15-26: :quant doesn't list //executorch/extension/llm/export:nvfp4, which cuda_packers.py:55 imports. It's only reached through :gguf.
    • backends/cuda/tests/targets.bzl:635-642: test_nvfp4_packers imports extension.llm.export.gguf (test_nvfp4_packers.py:61) without declaring it.

    Fix this →

Testing

  • No CI job executes triton::nvfp4_auto_prefill/nvfp4_auto_decode or the _prefill/_decode bodies (major).

    • These are the default auto path, but _linear only emits them with native FP4 (nvfp4_dispatch.py:160-165). So they never run on A100/A10G, even after [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411.
    • Every GPU case that reaches them skips there: test_nvfp4_export.py:43-44, test_nvfp4_fixed_prefill.py:33, test_nvfp4_unit_scale.py:91-94.
    • The CPU test_linear_routing swaps both ops for recorders (test_nvfp4_fusion.py:90-95).
    • As a result, none of these is ever checked:
      • the divmod(mode_split, _MODE_STRIDE) decode and _check_split (:62-63, :85-87)
      • output_dtype forwarding (:102-103)
      • unit_scale forwarding to GEMV (:89, :116)
    • Routing also only uses a static M of 1 or 37. Every dynamic export reaches statically_known_true (:166, nvfp4_output.py:24-26) with a symbolic M, and that case is never tested.
    • A CPU fix:
      • Patch recorders only into nvfp4_quantized_gemm. The bodies import from it at call time, so the real ops run their bodies under fake tensors.
      • Record split_k, unit_scale and output_dtype.
      • Call _prefill/_decode with an explicit mode_split= for both modes. launch_params.py:352-354 then calls the body directly.
      • Add a FakeTensorMode(shape_env=ShapeEnv()) case, as test_int8_quantized_gemm.py:397 does.

    Fix this →

  • The FP32 prefill output policy never runs a real kernel or a PTE (major).

    • Only the CPU fallback (test_nvfp4_fusion.py:47-50) and the recorders (:98) set ET_CUDA_NVFP4_PREFILL_OUTPUT.
    • No test calls nvfp4_w4a16(..., output_dtype=torch.float32).
    • No test feeds FP32 activations to Triton. nvfp4_dispatch.py:142-144 newly allows this under the policy, and nvfp4_quantized_gemm.py:1027-1028 rounds them to bf16.
    • No test checks that assume_constant_result (nvfp4_output.py:15) bakes the FP32 dtype into a dynamic-M PTE.
    • The only FP32-output kernel test is the SM120-only W4A4 one (test_nvfp4_kernels.py:127-145).
    • Fix: parametrize test_packed_linear_pte over the policy, at least the dynamic w4a16 case, which runs on A100 once [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411 lands. Also add a W4A16 FP32-output kernel case.

    Fix this →

  • test_embedding_bias_dispatch never dispatches the embedding (minor).

    • test_nvfp4_kernels.py:167 calls nvfp4_embedding directly.
    • So the CUDA side of _embedding (nvfp4_dispatch.py:209-225: the aten.embedding argument order, indices.contiguous(), orig_dtype) only runs against a mocked kernel (test_nvfp4_fusion.py:111-130).
    • No export lowers an nn.Embedding holding the CudaNVFP4Tensor that pack_embedding_for_cuda now produces (cuda_packers.py:153).
    • Fix: call F.embedding(ids, w) and aten.embedding.default(w, ids) in that test.

    Fix this →

  • The value check in test_portable_nvfp4_is_packed can't fail (minor). portable_weight() has all-zero qdata (nvfp4_test_utils.py:30), so test_nvfp4_packers.py:56 compares zeros with zeros whatever the repack does. Give it random qdata first, as test_nvfp4_unit_scale.py:80-82 does. Fix this →

  • The ET_CUDA_NVFP4_ACT flips after the PTE loads can't fail; still open from round 1 (minor). See test_nvfp4_export.py:136-138,145 and test_nvfp4_fixed_prefill.py:87-88. Only the Python tracer reads the variable, so changing it after load affects nothing. The check the comment describes needs the flip between torch.export.export and to_edge_transform_and_lower. Otherwise, drop the flips and the comment. Fix this →

  • Tuning asserts depend on test order (minor).

    • len(picks) == 1 (test_nvfp4_export.py:96-98, test_nvfp4_fixed_prefill.py:65, test_nvfp4_unit_scale.py:153) needs the process-global _CACHE to miss. A cache hit records no pick (launch_params.py:365-368, 394-398).
    • assert config_graph_timings (test_nvfp4_export.py:90) likewise needs the Triton autotuner cache to miss. That is why :40 picks "a hint no other case tunes".
    • All five files share one process (targets.bzl:647), so a rerun, or a new case with the same shapes, fails.
    • Fix: clear both caches in an autouse fixture, as the int4/5/6/8 GEMM tests do with clear_launch_param_cache() (test_int8_quantized_gemm.py:393).

    Fix this →

Code Quality

  • Basic tensor ops live in the dispatch module instead of next to the class; still open from round 1.

    • nvfp4_dispatch.py:228-242 registers identity detach/alias, t and _to_copy. aoti_packed_int4_tensor.py:139-151 keeps the same handlers with its class.
    • So CudaNVFP4Tensor's own behavior depends on whether quantize_op_dispatch was imported. Take nn.Parameter(w) in _pack_nvfp4, which calls w.detach():
      • With the import, it returns w itself and sets _is_param on the caller's tensor.
      • Without it, it falls back to TorchAOBaseTensor's handling. I didn't verify what that does here.
    • test_nvfp4_packers.py:6 imports the dispatch first, so only the first path is tested.
    • _transpose also quietly turns aten.t into a full dense dequantize.

    Fix this →

  • The positional max_norm branch is dead. F.embedding passes everything after weight to handle_torch_function by keyword, so args[3] at nvfp4_dispatch.py:211 is never reached. The positional=True case at test_nvfp4_fusion.py:133-144 runs the same path as the keyword case. Use kwargs.get("max_norm") and drop the parametrization. Fix this →

  • The policy logic is split across two modules (optional).

    • nvfp4_output.py:23-28 repeats the statically_known_true(M <= NVFP4_DECODE_MAX_ROWS) check from nvfp4_dispatch.py:166.
    • nvfp4_output.py reads its variable through assume_constant_result, while _linear reads ET_CUDA_NVFP4_ACT with a bare os.environ.get (:135).
    • Compute small once in _linear, and read both variables the same way.

    Fix this →

Recommendation

Request Changes

Two gaps block approval:

  • The default auto ops have no test that executes them in CI. CPU tests with the existing fake-tensor recorders would cover them cheaply.
  • The FP32 prefill policy has no such test either. A W4A16 PTE case would cover it.

The other items are test robustness and cleanup.

Review steps
  • Gather PR context and diff
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Round 3:

  • The default auto ops now run in CPU tests:
    • The real _prefill/_decode bodies run with only the kernels recorded. Tests check the mode/split decode, output_dtype and unit_scale forwarding, and that an unbounded M rejects split-K.
    • nvfp4_auto_prefill/nvfp4_auto_decode run their first candidate outside tuning.
    • Routing covers a symbolic M (ShapeEnv), which never counts as statically small.
  • The FP32 prefill policy now runs on hardware we have:
  • Tests that could not fail:
    • test_embedding_bias_dispatch now goes through F.embedding and aten.embedding.
    • The packer value check uses random qdata.
    • The no-op post-load ET_CUDA_NVFP4_ACT flips are removed.
    • The PTE files clear the pick cache in an autouse fixture.
  • Also done:
    • the auto prefill gets the traced M's bound for split-K, so its timing agrees with the traced call;
    • direct Buck deps added (:autotune, :nvfp4, gguf/nvfp4 for the packer test);
    • max_norm is read from kwargs only.
  • Not changed: detach/t/_to_copy registration lives with the dispatch, which every CUDA export imports; the policy-module split is cosmetic.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 21m 59s —— View job


Request Changes: test_linear_routing imports a helper from a test module that isn't in the Buck test_nvfp4_dispatch target, so all 9 of its cases fail under Buck, and no test checks that a calibrated activation scale reaches the W4A4 kernels.

PR Review: #25342

Summary

This PR adds CudaNVFP4Tensor linear and embedding dispatch (GEMV, W4A16, W4A4 and an AOTI-timed auto mode), an FP32 prefill output policy, and NVFP4 support in the Gemma 4 CUDA packers. One new Buck failure blocks it. The rest are test gaps and a tie-break subtlety. I don't repeat points you've already declined: the auto/env-var policy, where detach/t/_to_copy are registered, and SM80/SM86 support through #25411.

Infrastructure

  • test_linear_routing fails to import under Buck (major). Fix this →
    • test_nvfp4_fusion.py:103 imports dynamic_rows from executorch.backends.cuda.tests.test_nvfp4_launch_validation.
    • That module is only in the srcs of the two launch-validation test targets (targets.bzl:657-658, 669-670). Test targets can't be deps, and test_nvfp4_dispatch (targets.bzl:647-654) doesn't list the file.
    • The import always runs, so all 9 parametrizations raise ModuleNotFoundError under Buck. OSS pytest passes only because the file is on disk.
    • Of the tests that run on this target's A100, this is the only one that checks _linear's routing. At this commit, the target's GPU tests skip below SM89.
    • Fix: move dynamic_rows into nvfp4_test_utils.py, as you did for PackedLinear, and import it from there in both files.
    • Also in this target: four of its files import autotune.launch_params directly, but the target gets :autotune only through :quantize_op_dispatch.

Testing

  • No test checks that the static activation scale reaches the kernels (major). Fix this →
    • _linear passes w.act_per_tensor_scale or 0.0 to nvfp4_w4a4, nvfp4_auto_decode and nvfp4_auto_prefill (nvfp4_dispatch.py:184, 193, 210). _prefill and _decode pass act_scale on positionally (:68-69, :91).
    • The recorders drop positional arguments:
      • record_calls keeps only output_dtype (test_nvfp4_fusion.py:56-57).
      • The recorder in test_auto_bodies_forward_mode_split keeps only kwargs (:130-131), so the 0.5 that test passes is never checked.
    • Every weight in the routing and PTE tests has act_per_tensor_scale=None (shards, PackedLinear, portable_weight).
    • So every test still passes if any step on this path hardcodes 0.0. A ModelOpt checkpoint with a calibrated input_scale would then silently switch W4A4 to dynamic per-row activation scaling.
    • Forced w4a4 at decode M (:178, which _linear checks before the decode branches) also has no routing or PTE case.
  • No test exports a w4a16 prefill with a 4096-row bound (minor). Fix this →
    • test_prefill_4096_keeps_compiled_mode_for_dynamic_m (test_nvfp4_fixed_prefill.py:39) runs only auto and w4a4, and both need native FP4.
    • Without native FP4 (H100, L40S, and A100 after [executorch][cuda] NVFP4: exact software E4M3 scale decode below SM89 #25411), _linear forces w4a16 (nvfp4_dispatch.py:171-176).
    • test_packed_linear_pte exports that mode only with max=64 (test_nvfp4_export.py:89). So _launch_w4a16 never chooses split-K against a 4096-row workspace bound through AOTI or in a PTE.
    • Adding "w4a16" takes three edits:
  • The FP16 cases of test_unit_scale_gemv_bitwise_same_tile can't fail (minor). Fix this →
    • UNIT_SCALE changes only the BF16 branch of both GEMV variants (nvfp4_quantized_gemm.py:638, :722). For FP16 it only replaces the loaded pts with 1.0, and the test's pts is all ones.
    • So 24 of the 48 cases (test_nvfp4_unit_scale.py:37) compare two launches that compute the same values.

Code Quality

  • The autotune tie-break now also picks the precision (minor). Fix this →
    • _MODE_SPLITS encodes mode * 64 + split (nvfp4_dispatch.py:44-47), and the autotuner picks the smallest candidate within 2% of the fastest (launch_params.py:389-392).
    • So every W4A16 candidate (1–32) beats every W4A4 candidate (65–96) when their times are within 2%. For example, W4A16 split-32 beats a faster W4A4 split-1.
    • That contradicts the documented reason for the rule, "fewer partials, less memory" (launch_params.py:34-35).
    • If W4A16 should win close calls, say so at _MODE_SPLITS. Otherwise encode split ahead of mode.

Recommendation

Request Changes

  • Blocking: the Buck import failure. The fix is to move one function.
  • Should land with it: the activation-scale assertions. The existing recorders make them cheap to add.
  • The other items are minor.
Review steps
  • Gather PR context and diff
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

This branch was successfully deployed

1 active (outdated) and 1 inactive deployments
cadence — 0906c7c1 Deployed Nov 21, 2026 by Gasoonjia via hifi-op-test / hifi4 #39977
upload-benchmark-results — ac0065cc Deployed Nov 21, 2026 by Gasoonjia via upload-benchmark-results #23004
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant