Skip to content

[BE] Split up download.pytorch.org index update job #7185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[BE] Split up download.pytorch.org index update job #7185

wants to merge 9 commits into from

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Sep 18, 2025 •
edited
Loading

The jobs is getting slower and close to 1 hours now https://github.com/pytorch/test-infra/actions/workflows/update-s3-html.yml and I hate waiting an hour for the index to update. It's also awkward trying to explain to folks that they need to wait an hour for the index to update after a package is uploaded.

My proposal is to split up the job and run it on more runners for each accelerators cuda, rocm, xpu, cpu.

I also accidentally add a local file to the folder in my previous PR #7182 here.

Testing

https://github.com/pytorch/test-infra/actions/runs/17815392432

@vercel Vercel
Copy link

vercel bot commented Sep 18, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Preview Updated (UTC)
torchci Ignored Ignored Preview Sep 18, 2025 3:23am

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 18, 2025
@huydhn huydhn marked this pull request as draft September 18, 2025 01:22
@huydhn
Buggy code
b39df73 
Signed-off-by: Huy Do <[email protected]>
@huydhn huydhn requested review from atalman and malfet September 18, 2025 01:41
@huydhn
Fix lint
4eb09da 
Signed-off-by: Huy Do <[email protected]>
@huydhn huydhn marked this pull request as ready for review September 18, 2025 01:42
@huydhn huydhn changed the title Split up s3 update job [BE] Split up s3 update job Sep 18, 2025
@huydhn huydhn changed the title [BE] Split up s3 update job [BE] Split up download.pytorch.org index update job Sep 18, 2025
@huydhn
Finish testing
098a3ce 
Signed-off-by: Huy Do <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@atalman atalman Awaiting requested review from atalman

@malfet malfet Awaiting requested review from malfet

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@huydhn