TestDistributedContextParallelSelfAttn::test_context_parallel_ring_attn[window_size(-1, -1)-NO_SCAN-BALANCED-BSHD_KVPACKED-CAUSAL-BF16-1-2-128xCPx2-8-128-n8_dp1_cp4_tp2]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

Process completed with exit code 1.

JAX mGPU tests FAILED.

test_layers_with_overlap_fp8[Linear-row_tensor_parallel-fp8_current_scaling]::AssertionError: Warning: Latency not found for MI_M=16, MI_N=16, MI_K=128, mi_input_type=BFloat8Float8_fnuz. Returning latency value of 32 (really slow).

test_layers_with_overlap_fp8[Linear-row_tensor_parallel-fp8_delayed_scaling]::AssertionError: Warning: Latency not found for MI_M=16, MI_N=16, MI_K=128, mi_input_type=BFloat8Float8_fnuz. Returning latency value of 32 (really slow).

test_layers_with_overlap_bf16[ TransformerLayer - DGRAD+RS - BF16 ]::AssertionError: [aiter WARNING] unsupported condition in fwd_v3!!! data type: bf16

test_layers_with_overlap_bf16[ MultiheadAttention - DGRAD+RS - BF16 ]::AssertionError: [aiter WARNING] unsupported condition in fwd_v3!!! data type: bf16

test_layers_with_overlap_bf16[ LayerNormMLP - DGRAD+RS - BF16 ]::AssertionError: timeout: sending signal TERM to command ‘torchrun’

test_layers_with_overlap_bf16[ LayerNormLinear - COL-PARALLEL - DGRAD+RS - BF16 ]::AssertionError: timeout: sending signal TERM to command ‘torchrun’

test_layers_with_overlap_bf16[ Linear - COL-PARALLEL - DGRAD+RS - BF16 ]::AssertionError: timeout: sending signal TERM to command ‘torchrun’

test_layers_with_overlap_bf16[ Linear - ROW-PARALLEL - BF16 ]::AssertionError: timeout: sending signal TERM to command ‘torchrun’

test_split_reduce_scatter_overlaps[True-fp8]::AssertionError: Warning: Latency not found for MI_M=16, MI_N=16, MI_K=128, mi_input_type=BFloat8Float8_fnuz. Returning latency value of 32 (really slow).

test_split_reduce_scatter_overlaps[True-none]::AssertionError: timeout: sending signal TERM to command ‘torchrun’

Process completed with exit code 1.

PyTorch mGPU tests FAILED.

Process completed with exit code 1.

JAX tests failed.

PyTorch tests failed.

TestGroupedDense::test_grouped_dense_grad_fp8[scaling_mode_ScalingMode.CURRENT_TENSOR_SCALING-fwd_bwd_dtype0-input_shape_8_64_128_256]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe1-126m-1-6-torch.float16]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe1-126m-1-6-torch.bfloat16]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe1-126m-1-6-torch.float16]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe0-126m-1-3-torch.float32]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe1-126m-1-6-torch.float16]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe2-126m-2-6-torch.bfloat16]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

test_grouped_linear_accuracy[True-True-True-True-True-recipe1-126m-1-6-torch.float16]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

TestTELayers::test_sanity[None-grouped_linear]::did not finish: process hard-exited while running this test (per-test timeout, segfault, or OOM); see the suite .log

Process completed with exit code 1.

JAX tests FAILED.

PyTorch tests FAILED.

Process completed with exit code 1.

Core tests failed.

PyTorch tests failed.

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-128-2048-dtype0-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.45 GiB is allocated by PyTorch, and 181.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-64-4096-dtype2-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 384.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.45 GiB is allocated by PyTorch, and 181.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-64-4096-dtype1-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 384.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.45 GiB is allocated by PyTorch, and 181.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-64-4096-dtype0-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.45 GiB is allocated by PyTorch, and 181.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-64-2048-dtype2-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.45 GiB is allocated by PyTorch, and 181.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-64-2048-dtype1-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.45 GiB is allocated by PyTorch, and 181.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-1.0-64-2048-dtype0-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 251.98 GiB of which 0 bytes is free. Of the allocated memory 27.26 GiB is allocated by PyTorch, and 372.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-0.5-256-4096-dtype2-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 1.50 GiB. GPU 0 has a total capacity of 251.98 GiB of which 128.00 MiB is free. Of the allocated memory 26.26 GiB is allocated by PyTorch, and 1014.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-0.5-256-4096-dtype1-False]::torch.OutOfMemoryError: HIP out of memory. Tried to allocate 1.50 GiB. GPU 0 has a total capacity of 251.98 GiB of which 128.00 MiB is free. Of the allocated memory 26.01 GiB is allocated by PyTorch, and 1.24 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

test_fused_qkv_rope[True-1-_overlapping_grad-sbhd-0-0.5-256-4096-dtype0-False]::RuntimeError: Comparing

Process completed with exit code 1.

Core tests FAILED.

PyTorch tests FAILED.

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

14 more failures omitted from annotations; see the job summary for the full list.

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

1319 more failures omitted from annotations; see the job summary for the full list.