Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25338
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 1 PendingAs of commit ca5ea65 with merge base 9f49fe2 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
[ghstack-poisoned]
This PR needs a
|
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 21m 26s —— View job Requesting changes: nothing owns the NVFP4 per-tensor Detailed reviewPR Review: #25338SummaryThis PR teaches
The test gating and the 0xff reference coverage are also weaker than the test comments claim. API Design
Testing
Code Quality
RecommendationRequest Changes Items 1–3 block approval:
The other items are small. I couldn't run Python or the tests in this environment, so every finding comes from reading the code. All five changed files were reviewed.
|
…code down so A100 CI runs the NVFP4 kernels [ghstack-poisoned]
|
Thanks, addressed:
|
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 20m 21s —— View job Requesting changes: an NVFP4 Detailed reviewPR Review: #25338 (re-review)SummaryThis PR loads GGML NVFP4 blocks into
API Design
Testing
Code Quality
RecommendationRequest Changes Item 1 blocks approval. Today an NVFP4 Python couldn't be run in this environment, so all findings come from reading the code. The GGML bit layout was checked by hand, not against gguf-py or llama.cpp. All seven changed files were reviewed.
|
[ghstack-poisoned]
|
Round 2:
|
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 19m 30s —— View job Requesting changes for one line: the new Detailed reviewPR Review: #25338 (round 3)SummaryThis PR loads GGML NVFP4 blocks as Infrastructure
API Design
Code Quality
RecommendationRequest Changes Only item 1 blocks, and it's a one-line BUCK fix. Items 2 and 3 are minor. Python couldn't be run in this environment, so every finding comes from reading the code. All eight changed files were reviewed.
|
[ghstack-poisoned]
|
Round 3:
|
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 24m 46s —— View job Approve with three minor follow-ups: the generic GGUF loader now loads unpaired NVFP4 weights that only fail later at forward or export, Detailed reviewPR Review: #25338 (round 4)SummaryThis PR loads GGML NVFP4 blocks as API Design
Testing
RecommendationApprove Nothing blocks. Item 1 matters most before landing: it decides whether an NVFP4 GGUF fails at load or only at export, for every caller of the generic loader. Items 2 and 3 are small follow-ups. I couldn't run Python here, so every finding comes from reading the code. All eight changed files were reviewed. I didn't check the GGML NVFP4 bit layout against llama.cpp or gguf-py source.
|
Stack from ghstack (oldest at bottom):
ExportableGGUFTensornow accepts GGML NVFP4 (type 40) blocks: 64 elements per block as four K16 sub-blocks, four UE4M3 scale bytes followed by 32 bytes of E2M1 codes.dequantize_ggufdecodes them directly, andto_exportable_nvfp4_tensor()repacks them losslessly into the portable torchao NVFP4 layout (nibble reorder from GGML's eight-lows-then-eight-highs to even-element-low, GGML's sign-ignoring UE4M3 canonicalized tofloat8_e4m3fn, code 8 canonicalized to +0, optional per-tensor scale from the GGUF.scalecompanion). The 0xff scale byte decodes to 480 in GGML, which E4M3 cannot represent, so the conversion rejects it instead of silently changing values.ExportableNVFP4Tensorcarries the optional activation per-tensor scale the CUDA kernels use.Backends only need to understand
ExportableNVFP4Tensor; none of them has to parse GGML blocks.This is the first of three diffs adding NVFP4 to the CUDA backend (GGUF loading, CUDA kernels and dispatch, Muse Glimmer integration), followed by kernel follow-ups.
Authored with AI assistance.
Differential Revision: D131418678