This directory pins the user-space environment Aboleth is built and tested with, and the script that recreates it.
bootstrap.sh: recreates the pinned environments and builds everythingenvironment.json: machine-readable manifest of the reference host and package environmentrequirements.freeze.txt: exact uv/Python package resolutionrequirements.base.txt: direct dependencies used to resolve the environmentgin-requirements.freeze.txt: exact NCCL 2.30.7 GIN comparison environmentgin-requirements.base.txt: resolver-consistent Torch nightly environment before the NCCL overrideefa-requirements.txt: packages for the separate EFA environment (bootstrap.sh --efa)configure-efa-gda-driver.sh: installs NVIDIA kernel-module options (PeerMappingOverride=1,NVreg_EnableStreamMemOPs=1) that let the GPU map NIC doorbell pages without root; takes effect after a rebootbootstrap-gdrcopy.sh: builds a pinned GDRCopy for the NCCL GIN baseline on EFASHA256SUMS: checksums for the reproducibility files
The repository pins an unmodified upstream Triton submodule. Aboleth's dialect, operations, passes, and Python package are built entirely out of tree.
| GPUs | Network | Doorbell path | Notes |
|---|---|---|---|
| 8× H100 80GB (NVLink, NV18) | mlx5 RoCE v2 (ConnectX VF) | direct (root), staged | Reference host in environment.json: driver 570.211.01, CUDA 12.8.93; the driver does not grant unprivileged UAR mapping |
| 8× H100 SXM5 (NVSwitch) | 8× mlx5 native InfiniBand NDR, one HCA per GPU | direct without root (PeerMappingOverride=1), staged |
Same software stack |
| 4× H100 per node, multi-node | mlx5 InfiniBand | staged | No nvidia_peermem: GPU memory registers through dma-buf |
8× B200 (sm_100) |
mlx5 RoCE v2, rail-optimized | direct, staged | Needs the Torch substitution and settings below |
| 8× H100 (AWS p5.48xlarge) | EFA gen-2 | staged EFA, EFA host proxy | GPU-rung EFA doorbells are platform-gated (EFA_ABI.md) |
The AMD MI355X HIP runtime and Pensando Ionic backend are experimental and not covered by this table.
Install an NVIDIA driver and CUDA toolkit (the pinned environment targets the 570-series driver and CUDA 12.8), rdma-core/OFED userspace and development headers, zlib, libxml2 and libzstd development headers, Git, curl, GCC/G++, and Make. Then:
git clone --recurse-submodules https://github.com/ParCoreLab/aboleth.git
cd aboleth
sha256sum -c repro/SHA256SUMS
./repro/bootstrap.shThe default command installs uv 0.11.28 and CPython 3.12.13 under the project,
recreates .venv and .tools/venvs/gin, restores both exact package
environments, builds stock Triton with extension support, builds the
out-of-tree Aboleth compiler plugin, and builds the Aboleth, NVSHMEM, and NCCL
GIN binaries. It requires network access for pinned Python wheels and
Triton's pinned LLVM package. Dependencies outside default locations are
passed through to CMake:
./repro/bootstrap.sh -- -DRdmaCore_ROOT=/opt/rdma-core -DLibfabric_ROOT=/opt/amazon/efaThe GIN environment installs the pinned CUDA 12.6 Torch nightly and then
deliberately replaces Torch's NCCL 2.29.3 dependency with NCCL 2.30.7. The
bootstrap verifies the loaded libnccl.so.2 before compiling the benchmark.
Run the compiler, Python and unprivileged GPU suites:
./repro/bootstrap.sh --keep-env --verifyAlso run the root-required RDMA and workload tests (needs passwordless sudo unless the driver grants unprivileged UAR mapping) on compatible hardware:
./repro/bootstrap.sh --keep-env --hardware-testsThe runtime detects native InfiniBand versus RoCE, and rank-local runtimes
select the GPU-nearest mlx5 HCA from PCI topology. ABOLETH_HCA,
ABOLETH_PORT, and ABOLETH_GID_INDEX provide explicit overrides
(transport-control.md). Where the driver
refuses to map the NIC doorbell into the GPU, the runtime falls back to staged
submission, which needs no privileges (RDMA_ABI.md).
The pinned torch cannot run on this GPU. requirements.freeze.txt pins
torch==2.13.0+cu126, whose arch_list covers sm_50…sm_90, so every GPU
test dies with no kernel image is available for execution on the device. The
same torch version built against CUDA 12.9 carries sm_100 and stays on cu12
wheels, leaving the pinned nvidia-nvshmem-cu12 untouched:
./.tools/bin/uv pip install --python .venv/bin/python \
--index-strategy unsafe-best-match \
--extra-index-url https://download.pytorch.org/whl/cu129 "torch==2.13.0+cu129"This deviates from the pinned environment; revert with
uv pip sync repro/requirements.freeze.txt. Build the device bitcode for the
deployment target as well, e.g.
-DABOLETH_RDMA_BITCODE_ARCHS=sm_100a -DABOLETH_EFA_BITCODE_ARCHS=sm_100a.
On SM100 the RDMA lowering's scalar descriptor loads can be faster than the
default; select them with ABOLETH_PASSES=-invariant-descriptors
(COMPILER_SEMANTICS.md).
A host whose inbox ib_core cannot load nvidia_peermem registers GPU memory
through dma-buf; NVSHMEM then needs NVSHMEM_IB_DISABLE_DMABUF=0. On a
rail-optimized RoCE fabric, set ABOLETH_GID_INDEX to the routable RoCE v2
entry (transport-control.md).
The prebuilt LLVM needs development headers. Triton's vendored LLVM
exports ZLIB::ZLIB and LibXml2::LibXml2; without zlib1g-dev,
libxml2-dev and libzstd-dev, its CMake configure fails with "the target was
not found". The runtime .so.1 files are usually already present — it is the
headers and the dev symlinks that are missing.
Always install Triton through scripts/build-triton.sh. Running
pip install --editable third_party/triton directly omits TRITON_EXT_ENABLED=1
and TRITON_BUILD_DIR, which silently builds a second Triton at hidden
visibility, overwrites libtriton.so, and disables plugin loading entirely.
Every aboleth.* op then fails with 'triton._C.libtriton.ir.builder' object has no attribute 'create_aboleth_put'. Check with
nm -D --defined-only third_party/triton/python/triton/_C/libtriton.so | wc -l:
a correct build exports on the order of a hundred thousand symbols, a broken
one exports 1.