[Feature Request] Add official support for onnxruntime-gpu on ARM64/aarch64 platforms

[abhishek-iitmadras]

Describe the feature request

Issue Description
Currently, onnxruntime-gpu package lacks official support for ARM64/aarch64 architecture, limiting GPU acceleration capabilities on increasingly popular ARM-based platforms.

Current Situation

No official pre-built wheels for onnxruntime-gpu on ARM64/aarch64
Limited documentation for ARM64 GPU deployment

Technical Details

• Environment:
  Architecture: ARM64/aarch64
  Platform: Various (like AWS Graviton, etc.)
  Python Versions: 3.8+ compatibility needed

Proposed Solution

Official pre-built wheels for ARM64/aarch64
CI/CD pipeline additions for ARM64 builds

Would appreciate any feedback or guidance on how to make this happen?

Describe scenario use case

Use Case :
Growing adoption of ARM64 in edge/hpc computing.
Cloud deployments on ARM64-based servers (AWS Graviton, etc.)
Machine learning workloads on newer ARM-based development machines
IoT and embedded systems requiring GPU acceleration

[abhishek-iitmadras]

cc @snnn @skottmckay @edgchen1 @fs-eire @tiantun @hariharans29 @yuslepukhin @mszhanyi @baijumeswani @yufenglee @pengwa @pranavsharma @guoyu-wang @centwang @HectorSVC @natke @adrianlizarraga @wangyems @jchen351 @faxu @RandySheriff @fdwr @jywu-msft @askhade @RyanUnderhill @PeixuanZuo @thiagocrepaldi @tracysh @smk2007 @chilo-ms @xadupre @Honry @RandyShuai @zhanghuanrong @shahasad @liqunfu @jeffbloo

[skottmckay]

Do you have a specific use case? Different GPUs require different things. e.g. CUDA vs AMD vs integrated. Mobile vs server is also very different.

You mention AWS Graviton but I don't see a GPU option with that.

[adamreeve]

I'm also interested in this feature. We'd like to use onnxruntime with CUDA on Linux and arm64 Nvidia Grace CPUs. It would be great if we could install wheel's from PyPI for this rather than need to build onnxruntime ourselves.

I imagine that space pressure on PyPI is a concern, as you already need to remove old versions to make space for new releases (eg. #22747), and adding another architecture would make this worse. If that's a blocker to adding a new architecture then maybe providing builds from an Azure devops feed like you do for CUDA 11 builds would be an option?

You mention AWS Graviton but I don't see a GPU option with that.

G5g instances are Graviton machines with an Nvidia GPU.

[adamreeve]

If this is something that would be accepted, I or someone else in my team could help update the CI pipelines to add arm64 CUDA builds. We could probably use the existing onnxruntime-linux-ARM64-CPU-2019 machine pool for builds, but would ideally be able to run tests on arm64 GPU machines too. It doesn't look like there any arm64 machines with NVidia GPUs available in Azure from what I could see though, so that might be a problem. Is there a way around that, or would it be acceptable to create the builds but leave them untested for now?

[Retia-development]

Just wanted to add something to this.
I have the issue that it's very hard to get a working build for the NVidia jetson Orin (aarch64/CUDA 12.6/multiple branches including main/v19.2). I couldn't manage to get a working build after spending multiple days on it. Just putting it here since it seemed relevant to this issue.
So having a prebuild onnxruntime-gpu for that plaform would help a lot for development purposes.

[adamreeve]

Hi, is there any update on this? We'd be happy to contribute CI changes to set this up if those changes would be accepted. Given many of Nvidia's latest systems run arm64 processors this is starting to look like quite a big omission.