Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25230
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit c76fbea with merge base ba0d826 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
Pull Request resolved: #25230 ## Ulterior Motive Running bfloat16 models, such as Muse Glimmer, natively through ET-VK. This stack adds the bfloat16 dtype and the elementwise operators that exercise it. This commit lets exported programs carry bfloat16 tensors, and delegates casts to and from bfloat16 and the elementwise operators. ## Rationale **What**: The partitioner and serializer accept bfloat16 tensors. `_to_dim_order_copy` and `_to_copy` convert between bfloat16 and fp32, and from integers to bfloat16, and the elementwise operators accept bfloat16. Fusable patterns that carry bfloat16 tensors stay on the CPU. **Why**: Casts bound every bfloat16 region of a model. Python scalar constants, as in `x + 1`, also export as int64 tensors that are cast to bfloat16, so the elementwise operators, which the runtime already implements for bfloat16, are delegated together with the casts. Programs exported with `_skip_dim_order=True`, as the LLM export flow does, cast with `_to_copy`. No fused operator has bfloat16 shaders. ## Details - `torch.bfloat16` serializes as `VkDataType.BFLOAT16`. `VULKAN_T` holds every serializable dtype. `ALL_T`, the default operator dtype set, excludes bfloat16, so each operator opts in; its contents are unchanged. The 64-bit dtype check uses `VULKAN_T`. - Operators that accept bfloat16: - **Before**: none. - **After**: the casts below, plus binary ops, floor division, unary ops, clamp, and `pow` and `mul` by a scalar, through the new `FP_BF16_T` and `FP_BF16_INT_T` dtype sets. - Tensor representation selection forces bfloat16 tensors to buffer storage. - `_to_dim_order_copy` and bfloat16: - **Before**: not partitioned. - **After**: partitioned for bfloat16 to fp32, fp32 or int32 / int64 to bfloat16, and bfloat16 layout copies. Other pairs involving bfloat16 stay on the CPU. - `_to_copy` accepts the same bfloat16 conversions. It copies buffers with view shaders instead of blitting, since blits need textures. On devices that store bfloat16 as fp32, a bfloat16 cast is then a same-dtype buffer copy. - New shader variants: `view_convert_buffer` for int32 to bfloat16, fp32 to bfloat16 and bfloat16 to fp32, and `view_buffer` for bfloat16. - Fusable patterns with bfloat16 tensors: - **Before**: their nodes skipped the per-operator dtype checks. A bfloat16 int4 weight-only linear, for example, fused into `et_vk.linear_q4gsw`, which has no bfloat16 shader, so the model failed to load on GPUs with bfloat16 support. - **After**: the whole pattern stays on the CPU, as patterns that need 64-bit downcasting already do. - Devices without bfloat16 support store and compute bfloat16 tensors in fp32 without rounding to bfloat16, so results can differ from eager bfloat16. The comment introducing `VULKAN_T` states this. ghstack-source-id: 479777277 @exported-using-ghexport @bypass-github-export-checks Differential Revision: [D132361768](https://our.internmc.facebook.com/intern/diff/D132361768/)
Stack from ghstack (oldest at bottom):
Ulterior Motive
Running bfloat16 models, such as Muse Glimmer, natively through ET-VK. This stack adds the bfloat16 dtype and the elementwise operators that exercise it. This commit lets exported programs carry bfloat16 tensors and delegate casts to and from bfloat16.
Rationale
What: The partitioner and serializer accept bfloat16 tensors, and
_to_dim_order_copyand_to_copyconvert between bfloat16 and fp32, and from integers to bfloat16. Fusable patterns that carry bfloat16 tensors stay on the CPU.Why: Casts bound every bfloat16 region of a model. Python scalar constants, as in
x + 1, also export as int64 tensors that are cast to bfloat16. Programs exported with_skip_dim_order=True, as the LLM export flow does, cast with_to_copy. No fused operator has bfloat16 shaders.Details
torch.bfloat16serializes asVkDataType.BFLOAT16.VULKAN_Tholds every serializable dtype.ALL_T, the default operator dtype set, excludes bfloat16, so each operator opts in; its contents are unchanged. The 64-bit dtype check usesVULKAN_T._to_dim_order_copyand bfloat16:_to_copyaccepts the same bfloat16 conversions. It copies buffers with view shaders instead of blitting, since blits need textures. On devices that store bfloat16 as fp32, a bfloat16 cast is then a same-dtype buffer copy.view_convert_bufferfor int32 to bfloat16, fp32 to bfloat16 and bfloat16 to fp32, andview_bufferfor bfloat16.et_vk.linear_q4gsw, which has no bfloat16 shader, so the model failed to load on GPUs with bfloat16 support.VULKAN_Tstates this.Differential Revision: D132361768