Skip to content

[ET-VK][bf16] Lower bfloat16 dtype casts - #25230

Open
SS-JIA wants to merge 2 commits into
gh/SS-JIA/733/basefrom
gh/SS-JIA/733/head
Open

SS-JIA wants to merge 2 commits into
gh/SS-JIA/733/basefrom
gh/SS-JIA/733/head

Conversation

@SS-JIA

@SS-JIA SS-JIA commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Ulterior Motive

Running bfloat16 models, such as Muse Glimmer, natively through ET-VK. This stack adds the bfloat16 dtype and the elementwise operators that exercise it. This commit lets exported programs carry bfloat16 tensors and delegate casts to and from bfloat16.

Rationale

What: The partitioner and serializer accept bfloat16 tensors, and _to_dim_order_copy and _to_copy convert between bfloat16 and fp32, and from integers to bfloat16. Fusable patterns that carry bfloat16 tensors stay on the CPU.

Why: Casts bound every bfloat16 region of a model. Python scalar constants, as in x + 1, also export as int64 tensors that are cast to bfloat16. Programs exported with _skip_dim_order=True, as the LLM export flow does, cast with _to_copy. No fused operator has bfloat16 shaders.

Details

  • torch.bfloat16 serializes as VkDataType.BFLOAT16. VULKAN_T holds every serializable dtype. ALL_T, the default operator dtype set, excludes bfloat16, so each operator opts in; its contents are unchanged. The 64-bit dtype check uses VULKAN_T.
  • Tensor representation selection forces bfloat16 tensors to buffer storage.
  • _to_dim_order_copy and bfloat16:
    • Before: not partitioned.
    • After: partitioned for bfloat16 to fp32, fp32 or int32 / int64 to bfloat16, and bfloat16 layout copies. Other pairs involving bfloat16 stay on the CPU.
  • _to_copy accepts the same bfloat16 conversions. It copies buffers with view shaders instead of blitting, since blits need textures. On devices that store bfloat16 as fp32, a bfloat16 cast is then a same-dtype buffer copy.
  • New shader variants: view_convert_buffer for int32 to bfloat16, fp32 to bfloat16 and bfloat16 to fp32, and view_buffer for bfloat16.
  • Fusable patterns with bfloat16 tensors:
    • Before: their nodes skipped the per-operator dtype checks. A bfloat16 int4 weight-only linear, for example, fused into et_vk.linear_q4gsw, which has no bfloat16 shader, so the model failed to load on GPUs with bfloat16 support.
    • After: the whole pattern stays on the CPU, as patterns that need 64-bit downcasting already do.
  • Devices without bfloat16 support store and compute bfloat16 tensors in fp32 without rounding to bfloat16, so results can differ from eager bfloat16. The comment introducing VULKAN_T states this.

Differential Revision: D132361768

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25230

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit c76fbea with merge base ba0d826 (image):

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]
SS-JIA added a commit that referenced this pull request Nov 20, 2026
Pull Request resolved: #25230

## Ulterior Motive

Running bfloat16 models, such as Muse Glimmer, natively through ET-VK. This stack adds the bfloat16 dtype and the elementwise operators that exercise it. This commit lets exported programs carry bfloat16 tensors, and delegates casts to and from bfloat16 and the elementwise operators.

## Rationale

**What**: The partitioner and serializer accept bfloat16 tensors. `_to_dim_order_copy` and `_to_copy` convert between bfloat16 and fp32, and from integers to bfloat16, and the elementwise operators accept bfloat16. Fusable patterns that carry bfloat16 tensors stay on the CPU.

**Why**: Casts bound every bfloat16 region of a model. Python scalar constants, as in `x + 1`, also export as int64 tensors that are cast to bfloat16, so the elementwise operators, which the runtime already implements for bfloat16, are delegated together with the casts. Programs exported with `_skip_dim_order=True`, as the LLM export flow does, cast with `_to_copy`. No fused operator has bfloat16 shaders.

## Details

- `torch.bfloat16` serializes as `VkDataType.BFLOAT16`. `VULKAN_T` holds every serializable dtype. `ALL_T`, the default operator dtype set, excludes bfloat16, so each operator opts in; its contents are unchanged. The 64-bit dtype check uses `VULKAN_T`.
- Operators that accept bfloat16:
  - **Before**: none.
  - **After**: the casts below, plus binary ops, floor division, unary ops, clamp, and `pow` and `mul` by a scalar, through the new `FP_BF16_T` and `FP_BF16_INT_T` dtype sets.
- Tensor representation selection forces bfloat16 tensors to buffer storage.
- `_to_dim_order_copy` and bfloat16:
  - **Before**: not partitioned.
  - **After**: partitioned for bfloat16 to fp32, fp32 or int32 / int64 to bfloat16, and bfloat16 layout copies. Other pairs involving bfloat16 stay on the CPU.
- `_to_copy` accepts the same bfloat16 conversions. It copies buffers with view shaders instead of blitting, since blits need textures. On devices that store bfloat16 as fp32, a bfloat16 cast is then a same-dtype buffer copy.
- New shader variants: `view_convert_buffer` for int32 to bfloat16, fp32 to bfloat16 and bfloat16 to fp32, and `view_buffer` for bfloat16.
- Fusable patterns with bfloat16 tensors:
  - **Before**: their nodes skipped the per-operator dtype checks. A bfloat16 int4 weight-only linear, for example, fused into `et_vk.linear_q4gsw`, which has no bfloat16 shader, so the model failed to load on GPUs with bfloat16 support.
  - **After**: the whole pattern stays on the CPU, as patterns that need 64-bit downcasting already do.
- Devices without bfloat16 support store and compute bfloat16 tensors in fp32 without rounding to bfloat16, so results can differ from eager bfloat16. The comment introducing `VULKAN_T` states this.
ghstack-source-id: 479777277
@exported-using-ghexport

@bypass-github-export-checks

Differential Revision: [D132361768](https://our.internmc.facebook.com/intern/diff/D132361768/)

This branch was previously deployed

1 inactive deployment
cadence — c76fbeaa Deployed Nov 20, 2026 by SS-JIA via hifi-op-test / hifi4 #39685
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant