Skip to content

Commit f63b372

Browse files
authored
Upload the wheels to R2 from a workflow_run workflow (#25356)
Executorch wheels are not uploaded to R2 today. test-infra's reusable build workflows (`_binary_upload.yml`) upload to R2 only when the caller passes the R2 secrets, and none of executorch's build workflows does. `pytorch/executorch` is already on the R2 allowlist there, so only the secrets are missing. Passing the repo-level `secrets.R2_*` from the build workflows would let a workflow in a pull request that gets a ciflow tag read them (pytorch/FBGEMM#6852 showed this in FBGEMM). A reusable workflow does not see the caller's environment secrets, and a calling job cannot set `environment:`, so the R2 upload gets its own workflow, as in pytorch/vision#9765, pytorch/audio#4244 and pytorch/FBGEMM#6855. `upload_wheels_r2.yml` runs on `workflow_run` for the seven wheel builds: Linux, Aarch64 Linux, Linux CUDA, Aarch64 Linux CUDA, Windows, Windows CUDA and macOS. A `workflow_run` workflow always comes from main. It only acts when the build was a push to `nightly` in this repo. `get-matrix` reads the build matrix, which each build workflow now saves as the `build-matrix` artifact (the filtered matrix for the CUDA ones), and `upload-r2` (`environment: pytorchbot-env`) downloads each wheel from the build run and copies it to R2 with the same steps as test-infra's `_binary_upload.yml`. The run is named after the build run it handled, and its summary links to it. The architecture in the artifact name follows what each build passes to the upload: `x86_64` for Linux, `aarch64` for Aarch64 Linux, `x64` for Windows, none for macOS. I checked these, and the CUDA names, against the artifacts of the 2026-11-20 nightly runs (for example `pytorch_executorch__3.11_cpu_x86_64`, `..._cu132_aarch64`, `..._cpu_x64`, `pytorch_executorch__3.11_cpu_`). Not included: the two Native wheel workflows. They pass `trigger-event: development`, so they never upload, and the comment there says executorch-native is not published yet. The build workflows' own S3 upload (an AWS role through OIDC) is unchanged. The only change to them is the `save-build-matrix` job. **DO NOT MERGE** before the `R2_ACCOUNT_ID`, `R2_ACCESS_KEY_ID` and `R2_SECRET_ACCESS_KEY` secrets exist in the `pytorchbot-env` environment of pytorch/executorch. Rollout: 1. A repo admin adds the three secrets to `pytorchbot-env`. 2. Merge this. 3. Check that the next nightly run triggers "Upload wheels to R2" and that the wheels reach R2. Not verified: a `workflow_run` workflow only runs once it is on main, so none of this has run in executorch. pytorch/audio#4244 ran this design end to end, with the environment secrets, on 2026-11-19. I checked that the workflows parse as YAML and that `shellcheck` is clean on the new scripts, and that the matrix fields the upload reads (`build_name`, `python_version`, `desired_cuda`, `upload_to_base_bucket`) are kept by `filter_cuda_matrix.py`. Created with Claude Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>
1 parent 549a384 commit f63b372

8 files changed

Lines changed: 231 additions & 0 deletions

‎.github/workflows/build-wheels-aarch64-linux.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ jobs:
4646
with-rocm: disabled
4747
python-versions: '["3.11", "3.12", "3.13", "3.14", "3.15"]'
4848

49+
save-build-matrix:
50+
needs: generate-matrix
51+
runs-on: ubuntu-latest
52+
steps:
53+
- name: Save the build matrix
54+
env:
55+
MATRIX: ${{ needs.generate-matrix.outputs.matrix }}
56+
run: printf '%s' "${MATRIX}" > build-matrix.json
57+
- name: Upload the build matrix
58+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
59+
with:
60+
name: build-matrix
61+
path: build-matrix.json
62+
4963
build:
5064
needs: generate-matrix
5165
permissions:

‎.github/workflows/build-wheels-cuda-aarch64-linux.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,20 @@ jobs:
8181
echo "${MATRIX_BLOB}"
8282
echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}"
8383
84+
save-build-matrix:
85+
needs: filter-matrix
86+
runs-on: ubuntu-latest
87+
steps:
88+
- name: Save the build matrix
89+
env:
90+
MATRIX: ${{ needs.filter-matrix.outputs.matrix }}
91+
run: printf '%s' "${MATRIX}" > build-matrix.json
92+
- name: Upload the build matrix
93+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
94+
with:
95+
name: build-matrix
96+
path: build-matrix.json
97+
8498
build:
8599
needs: filter-matrix
86100
permissions:

‎.github/workflows/build-wheels-cuda-linux.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,20 @@ jobs:
8080
echo "${MATRIX_BLOB}"
8181
echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}"
8282
83+
save-build-matrix:
84+
needs: filter-matrix
85+
runs-on: ubuntu-latest
86+
steps:
87+
- name: Save the build matrix
88+
env:
89+
MATRIX: ${{ needs.filter-matrix.outputs.matrix }}
90+
run: printf '%s' "${MATRIX}" > build-matrix.json
91+
- name: Upload the build matrix
92+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
93+
with:
94+
name: build-matrix
95+
path: build-matrix.json
96+
8397
build:
8498
needs: filter-matrix
8599
permissions:

‎.github/workflows/build-wheels-cuda-windows.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,20 @@ jobs:
7878
echo "${MATRIX_BLOB}"
7979
echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}"
8080
81+
save-build-matrix:
82+
needs: filter-matrix
83+
runs-on: ubuntu-latest
84+
steps:
85+
- name: Save the build matrix
86+
env:
87+
MATRIX: ${{ needs.filter-matrix.outputs.matrix }}
88+
run: printf '%s' "${MATRIX}" > build-matrix.json
89+
- name: Upload the build matrix
90+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
91+
with:
92+
name: build-matrix
93+
path: build-matrix.json
94+
8195
build:
8296
needs: filter-matrix
8397
strategy:

‎.github/workflows/build-wheels-linux.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ jobs:
4646
with-rocm: disabled
4747
python-versions: '["3.11", "3.12", "3.13", "3.14", "3.15"]'
4848

49+
save-build-matrix:
50+
needs: generate-matrix
51+
runs-on: ubuntu-latest
52+
steps:
53+
- name: Save the build matrix
54+
env:
55+
MATRIX: ${{ needs.generate-matrix.outputs.matrix }}
56+
run: printf '%s' "${MATRIX}" > build-matrix.json
57+
- name: Upload the build matrix
58+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
59+
with:
60+
name: build-matrix
61+
path: build-matrix.json
62+
4963
build:
5064
needs: generate-matrix
5165
permissions:

‎.github/workflows/build-wheels-macos.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,20 @@ jobs:
5050
with-rocm: disabled
5151
python-versions: '["3.11", "3.12", "3.13", "3.14", "3.15"]'
5252

53+
save-build-matrix:
54+
needs: generate-matrix
55+
runs-on: ubuntu-latest
56+
steps:
57+
- name: Save the build matrix
58+
env:
59+
MATRIX: ${{ needs.generate-matrix.outputs.matrix }}
60+
run: printf '%s' "${MATRIX}" > build-matrix.json
61+
- name: Upload the build matrix
62+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
63+
with:
64+
name: build-matrix
65+
path: build-matrix.json
66+
5367
build:
5468
needs: generate-matrix
5569
permissions:

‎.github/workflows/build-wheels-windows.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,20 @@ jobs:
4848
with-rocm: disabled
4949
python-versions: '["3.11", "3.12", "3.13", "3.14", "3.15"]'
5050

51+
save-build-matrix:
52+
needs: generate-matrix
53+
runs-on: ubuntu-latest
54+
steps:
55+
- name: Save the build matrix
56+
env:
57+
MATRIX: ${{ needs.generate-matrix.outputs.matrix }}
58+
run: printf '%s' "${MATRIX}" > build-matrix.json
59+
- name: Upload the build matrix
60+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
61+
with:
62+
name: build-matrix
63+
path: build-matrix.json
64+
5165
build:
5266
needs: generate-matrix
5367
strategy:
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
name: Upload wheels to R2
2+
run-name: Upload wheels from ${{ github.event.workflow_run.name }} (run ${{ github.event.workflow_run.id }})
3+
4+
# Runs from main whenever a nightly wheel build finishes, picks up the wheels that
5+
# build saved as GitHub artifacts and uploads them to R2. The upload step lives
6+
# here, not in the build workflows, so a pull request cannot change it, and the R2
7+
# credentials come from the pytorchbot-env environment.
8+
9+
on:
10+
workflow_run:
11+
workflows:
12+
- Build Linux Wheels
13+
- Build Aarch64 Linux Wheels
14+
- Build Linux CUDA Wheels
15+
- Build Aarch64 Linux CUDA Wheels
16+
- Build Windows Wheels
17+
- Build Windows CUDA Wheels
18+
- Build macOS Wheels
19+
types:
20+
- completed
21+
22+
permissions: {}
23+
24+
concurrency:
25+
group: ${{ github.workflow }}-${{ github.event.workflow_run.id }}
26+
cancel-in-progress: false
27+
28+
jobs:
29+
get-matrix:
30+
if: ${{ github.repository == 'pytorch/executorch' && github.event.workflow_run.event == 'push' && github.event.workflow_run.head_branch == 'nightly' && github.event.workflow_run.head_repository.full_name == github.repository }}
31+
runs-on: ubuntu-22.04
32+
permissions:
33+
actions: read
34+
outputs:
35+
matrix: ${{ steps.matrix.outputs.matrix }}
36+
architecture: ${{ steps.matrix.outputs.architecture }}
37+
steps:
38+
- name: Link the build run
39+
shell: bash
40+
env:
41+
WORKFLOW_NAME: ${{ github.event.workflow_run.name }}
42+
RUN_URL: ${{ github.event.workflow_run.html_url }}
43+
RUN_BRANCH: ${{ github.event.workflow_run.head_branch }}
44+
run: echo "Uploading the wheels of [${WORKFLOW_NAME} (${RUN_BRANCH})](${RUN_URL})" >> "${GITHUB_STEP_SUMMARY}"
45+
46+
- name: Download the build matrix
47+
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
48+
with:
49+
name: build-matrix
50+
run-id: ${{ github.event.workflow_run.id }}
51+
github-token: ${{ secrets.GITHUB_TOKEN }}
52+
53+
- name: Read the build matrix
54+
id: matrix
55+
shell: bash
56+
env:
57+
WORKFLOW_NAME: ${{ github.event.workflow_run.name }}
58+
run: |
59+
set -euo pipefail
60+
echo "matrix=$(jq -c . build-matrix.json)" >> "${GITHUB_OUTPUT}"
61+
# The architecture in the artifact name, as the build workflows pass it
62+
# to the upload (the macOS one passes none)
63+
case "${WORKFLOW_NAME}" in
64+
"Build Aarch64 Linux Wheels" | "Build Aarch64 Linux CUDA Wheels") echo "architecture=aarch64" >> "${GITHUB_OUTPUT}" ;;
65+
"Build Windows Wheels" | "Build Windows CUDA Wheels") echo "architecture=x64" >> "${GITHUB_OUTPUT}" ;;
66+
"Build macOS Wheels") echo "architecture=" >> "${GITHUB_OUTPUT}" ;;
67+
*) echo "architecture=x86_64" >> "${GITHUB_OUTPUT}" ;;
68+
esac
69+
70+
upload-r2:
71+
needs: get-matrix
72+
runs-on: ubuntu-22.04
73+
environment: pytorchbot-env
74+
timeout-minutes: 30
75+
permissions:
76+
actions: read
77+
contents: read
78+
strategy:
79+
fail-fast: false
80+
matrix: ${{ fromJSON(needs.get-matrix.outputs.matrix) }}
81+
name: upload-r2-${{ matrix.build_name }}
82+
steps:
83+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
84+
with:
85+
repository: pytorch/test-infra
86+
ref: main
87+
path: test-infra
88+
89+
- uses: ./test-infra/.github/actions/set-channel
90+
91+
- uses: ./test-infra/.github/actions/setup-binary-upload
92+
with:
93+
repository: pytorch/executorch
94+
ref: ""
95+
python-version: ${{ matrix.python_version }}
96+
cuda-version: ${{ matrix.desired_cuda }}
97+
arch: ${{ needs.get-matrix.outputs.architecture }}
98+
upload-to-base-bucket: ${{ matrix.upload_to_base_bucket }}
99+
100+
- name: Download the wheel
101+
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
102+
with:
103+
name: ${{ env.ARTIFACT_NAME }}
104+
path: pytorch/executorch/dist/
105+
run-id: ${{ github.event.workflow_run.id }}
106+
github-token: ${{ secrets.GITHUB_TOKEN }}
107+
108+
- name: Upload package to R2
109+
shell: bash
110+
working-directory: pytorch/executorch
111+
env:
112+
R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID }}
113+
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
114+
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
115+
AWS_DEFAULT_REGION: auto
116+
run: |
117+
set -ex
118+
if [[ -z "${R2_ACCOUNT_ID}" || -z "${AWS_ACCESS_KEY_ID}" || -z "${AWS_SECRET_ACCESS_KEY}" ]]; then
119+
echo "::error::R2 secrets are missing from the pytorchbot-env environment"
120+
exit 1
121+
fi
122+
123+
# shellcheck disable=SC1090
124+
source "${BUILD_ENV_FILE}"
125+
126+
pip install awscli==1.32.18
127+
128+
PYTORCH_S3_BUCKET_PATH=${PYTORCH_S3_BUCKET_PATH/"s3://pytorch"/"s3://pytorch-downloads"}
129+
for pkg in dist/*; do
130+
shm_id=$(sha256sum "${pkg}" | awk '{print $1}')
131+
aws s3 cp "$pkg" "${PYTORCH_S3_BUCKET_PATH}" \
132+
--metadata "checksum-sha256=${shm_id}" --endpoint-url "https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com"
133+
done

0 commit comments

Comments
 (0)