test_cp_with_fused_attention[False-None-False-False-False-p2p-thd-cp_1_1-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_1_1', 'qkv_format=thd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-thd-cp_1_0-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_1_0', 'qkv_format=thd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_3_4-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_3_4', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_3_2-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_3_2', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_2_3-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_2_3', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_2_2-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_2_2', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_2_0-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_2_0', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_1_4-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_1_4', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_1_1-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_1_1', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

test_cp_with_fused_attention[False-None-False-False-False-p2p-sbhd-cp_1_0-bf16]::subprocess.CalledProcessError: Command '['python3', '-m', 'torch.distributed.launch', '--nproc-per-node=2', '/workspace/tests/pytorch/attention/run_attention_with_cp.py', 'dtype=bf16', 'model=cp_1_0', 'qkv_format=sbhd', 'kernel_backend=FusedAttention', 'cp_comm_type=p2p', 'fp8_bwd=False', 'fp8_dpa=False', 'fp8_mha=False', 'scaling_mode=None', 'f16_O=False', 'is_training=True', 'log_level=WARNING']' returned non-zero exit status 1.

Process completed with exit code 1.

PyTorch mGPU tests FAILED.

Canceling since a higher priority waiting request for PR Automatic CI-refs/pull/608/merge exists

The operation was canceled.

Canceling since a higher priority waiting request for PR Automatic CI-refs/pull/608/merge exists

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/download-artifact@v4, actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/download-artifact@v4, actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/download-artifact@v4, actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/download-artifact@v4, actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

31 more failures omitted from annotations; see the job summary for the full list.

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/download-artifact@v4, actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/download-artifact@v4, actions/upload-artifact@v4. Actions will be forced to run with Node.js 24 by default starting June 16th, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/