ci: point uv at the baked venv so tests pass on the /next runner image by HarikrishnanBalagopal · Pull Request #509 · torch-spyre/hf-adapters

HarikrishnanBalagopal · 2026-09-10T15:22:16Z

What this does

The ARC self-hosted runner sets have been switched to boot from the
icr.io/ai_sw_accel/2.0/next/<product>:amd64-dev images (spyre-frameworks
merge, live). Those images ship the project venv baked and activated at
/home/senuser/.venv. With no hint to uv, uv run re-creates an empty project
.venv and re-syncs on every card job, so each Spyre test breaks at setup.

This sets uv to reuse the baked venv:

_test_matrix.yaml top-level env: gains UV_NO_SYNC: "1" and
UV_PROJECT_ENVIRONMENT: /home/senuser/.venv, so uv run points at the baked
venv and skips its implicit sync.
The check uv.lock job (ubuntu-latest, no baked venv) blanks both vars at
job level, so the top-level values do not leak into setup-uv@v5 and fail it.
The two make model-module-tests steps drop the now-redundant
source .venv/bin/activate.
gather-runner-info echoes VIRTUAL_ENV / UV_NO_SYNC /
UV_PROJECT_ENVIRONMENT for diagnostics.

Why at the workflow level

This mirrors spyre-frameworks PR #1390, which bakes the same two vars into the
image layer. That PR is not yet merged, so the workflow-level fix keeps CI green
on the /next runners in the meantime. Once #1390 lands and the image carries
the vars, this top-level env: can be removed.

Issue

https://github.com/torch-spyre/hf-adapters/actions/runs/34507171144/job/102972671665?pr=183#step:5:22

ImportError while loading conftest '/home/senuser/hf-adapters/hf-adapters/tests/conftest.py'.
tests/conftest.py:55: in <module>
    from transformers import AutoModelForCausalLM, PretrainedConfig
E   ModuleNotFoundError: No module named 'transformers'
make: *** [Makefile:107: embed-compare-tests] Error 4

Validation

The identical change was validated on the shadow branch
(shadow-resource-validation-v2, PR #329) against the fresh /next image: the
card suites went from all-red-at-setup to green (~124 card jobs), the 3 VLM e2e
legs passed, and the only residual reds were image-independent (a Wikimedia HTTP
429 on the CLIP fixture download, which also flakes on the legacy control, and
the check uv.lock env leak this PR fixes). Shadow run:
https://github.com/torch-spyre/hf-adapters/actions/runs/34478885987

This is the non-shadow production version: it excludes the shadow-only
image_torch_spyre_shadow label swap in test_pull_request.yaml, so it targets
the real prod runner labels.

The ARC runner sets now boot from the icr.io/ai_sw_accel/2.0/next/<product> images, which ship the project venv baked and activated at /home/senuser/.venv. Without telling uv about it, `uv run` re-creates an empty project .venv and re-syncs on every card job, so each Spyre test breaks at setup. Set UV_NO_SYNC=1 and UV_PROJECT_ENVIRONMENT=/home/senuser/.venv at the _test_matrix.yaml top level so uv reuses the baked venv and skips the implicit sync. Neutralize both on the ubuntu-hosted check-uv-lock job, which has no baked venv and would otherwise fail inside setup-uv. Drop the now-redundant `source .venv/bin/activate` from the two model-module-tests steps. Echo VIRTUAL_ENV / UV_NO_SYNC / UV_PROJECT_ENVIRONMENT from gather-runner-info for diagnostics. This mirrors spyre-frameworks PR #1390, which bakes the same vars into the image layer; the workflow-level fix keeps CI green on the /next runners until that image rebuild lands. Signed-off-by: Harikrishnan Balagopal <Harikrishnan.Balagopal@ibm.com>

BenjSz

lgtm

spyre-ci · 2026-09-10T18:47:18Z

⚙️ Auto-merge enabled — dispatched /spyre-test for 2dfac342 because the required Spyre Test gate had not run on this commit.

_{Automated: enabling auto-merge requests the gate, so the PR is not left waiting on a check nobody asked for. Re-enabling auto-merge on this same commit will not dispatch again.}

spyre-ci · 2026-09-10T18:47:44Z

❌ spyre-test: failure

Triggered by: /spyre-test comment

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴<br/>image · 16ec4926d4c8"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 0aa666e8acc0"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · b8b89d99752a"]
  end
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_torch_spyre_torch_spyre_dev sOk;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;

✅ orch trigger-pr-validation — green · arches amd64 · fp amd64=5d4c7455

level	component	arch	build	smoke	unit	integration	trunk	regression	perf
L0	torch-spyre/torch-spyre-dev	amd64	✅ ok	⏳	·	🟡 gha	·	·	·
L1	hf-adapters/hf-adapters-dev	amd64	✅ ok	🟢	·	🟢 gha	·	·	·
L2	spyre-inference/spyre-inference-dev	amd64	✅ ok	🟢	·	🟡 gha	·	·	·

GHA test runs:

[hf-adapters/amd64] integration: SUCCESS https://github.com/torch-spyre/hf-adapters/actions/runs/34518120438
[torch-spyre/amd64] integration: FAILURE https://github.com/torch-spyre/torch-spyre/actions/runs/34517627295
[spyre-inference/amd64] integration: FAILURE https://github.com/torch-spyre/spyre-inference/actions/runs/34519199084

⚠️ advisory failures only — mergeable, see below

Build: built 3

Tests: passed 3 · blocking 0 · advisory 2 · infra/inconclusive 0 · no signal 0

Failures by kind

⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

spyre-ci · 2026-09-10T18:58:55Z

GHA test runs:

[torch-spyre/x86_64] integration: DISPATCHED https://github.com/torch-spyre/torch-spyre/actions/runs/34517627295

spyre-ci · 2026-09-10T19:03:42Z

GHA test runs:

[hf-adapters/x86_64] integration: DISPATCHED https://github.com/torch-spyre/hf-adapters/actions/runs/34518120438

spyre-ci · 2026-09-10T19:14:22Z

GHA test runs:

[spyre-inference/x86_64] integration: DISPATCHED https://github.com/torch-spyre/spyre-inference/actions/runs/34519199084

ashokponkumar · 2026-09-10T19:52:29Z

/spyre-test

spyre-ci · 2026-09-10T19:53:03Z

❌ spyre-test: failure

Triggered by: /spyre-test comment: #509 (comment)

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_spyre_backend_spyre_backend_dev["spyre-backend/spyre-backend-dev ↺<br/>image · eddfefe0e289"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴 ↺<br/>image · 09ec21ef38de"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🔴<br/>image · 00f2afeac7d0"]
  end
  subgraph Lamd64_3["amd64 L3 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · 94e8d8cd9da3"]
  end
  n_amd64_spyre_backend_spyre_backend_dev --> n_amd64_torch_spyre_torch_spyre_dev
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_spyre_backend_spyre_backend_dev sReused;
  class n_amd64_torch_spyre_torch_spyre_dev sReused;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;

✅ orch trigger-pr-validation — green · arches amd64 · fp amd64=059f8ea6

level	component	arch	build	smoke	unit	integration	trunk	regression	perf
L0	spyre-backend/spyre-backend-dev	amd64	✅ ok (reused)	·	·	·	·	·	·
L1	torch-spyre/torch-spyre-dev	amd64	✅ ok (reused)	🟢	·	🟡 gha	·	·	·
L2	hf-adapters/hf-adapters-dev	amd64	✅ ok	🟢	·	🟡 gha	·	·	·
L3	spyre-inference/spyre-inference-dev	amd64	✅ ok	🟢	·	🟡 gha	·	·	·

GHA test runs:

[torch-spyre/amd64] integration: FAILURE https://github.com/torch-spyre/torch-spyre/actions/runs/34526369252
[hf-adapters/amd64] integration: FAILURE https://github.com/torch-spyre/hf-adapters/actions/runs/34527108575
[spyre-inference/amd64] integration: FAILURE https://github.com/torch-spyre/spyre-inference/actions/runs/34528078805

⚠️ advisory failures only — mergeable, see below

Build: built 2 · reused 2

Tests: passed 1 · blocking 0 · advisory 4 · infra/inconclusive 0 · no signal 0

Failures by kind

⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
⚠️ advisory (does not block) · hf-adapters/amd64 integration: UNSTABLE
⚠️ advisory (does not block) · hf-adapters/amd64 gha:integration: FAILURE
ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

hf-adapters/amd64 gha:integration, hf-adapters/amd64 integration, torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

HarikrishnanBalagopal · 2026-09-10T19:57:09Z

This change is complete and does what it says. On the live /next :amd64-dev runner image, the baked, pre-activated venv at /home/senuser/.venv made uv run --active pick a venv that uv sync never populated, so every card test failed collection with ModuleNotFoundError: No module named 'transformers'. Pointing UV_PROJECT_ENVIRONMENT at the baked venv (and UV_NO_SYNC=1) makes the two converge on one complete venv.

Result on the latest run (34496557541): 129 pass, 8 fail (down from a full-matrix collection failure without this change). The 8 remaining failures are all external to this PR:

4 VLM e2e jobs (granite-vision-4.1-4b, Ministral-3-3B, each plus its pod-level retry) fail in DDC compile, not in Python:
dxp_standalone ... returned non-zero exit status 1, Program verification failed for core N node sdsc_11, Immediate value out of boundary ... LX_MVLOOPCNT on a sdsc_fused__scaled_dot_product_... kernel. This is a known DDC compiler boundary defect whose fix has already merged in the toolchain and is being rebuilt into the runner image. It clears once the /next image rebakes with the new toolchain build. Not related to the venv change.
1 seq-classification job (distilbert sst-2) fails at setup with a transient git clone error: RPC failed; curl 92 HTTP/2 stream ... not closed cleanly, fatal: early EOF. Network flake; a re-run recovers it.
3 fan-in gates (run spyre unit tests x2, Spyre test matrix result) only aggregate the above.

No remaining failure is caused by this change. The venv fix itself is proven by the 129 green card jobs that previously could not even collect. Once the /next image carries the deeptools #4691 RPM, a re-run should be fully green.

spyre-ci · 2026-09-10T20:26:19Z

GHA test runs:

[torch-spyre/x86_64] integration: DISPATCHED https://github.com/torch-spyre/torch-spyre/actions/runs/34526369252

spyre-ci · 2026-09-10T20:33:39Z

GHA test runs:

[hf-adapters/x86_64] integration: DISPATCHED https://github.com/torch-spyre/hf-adapters/actions/runs/34527108575

spyre-ci · 2026-09-10T20:43:35Z

GHA test runs:

[spyre-inference/x86_64] integration: DISPATCHED https://github.com/torch-spyre/spyre-inference/actions/runs/34528078805

ashokponkumar · 2026-09-10T20:50:18Z

/spyre-test

spyre-ci · 2026-09-10T20:50:51Z

❌ spyre-test: failure

Triggered by: /spyre-test comment: #509 (comment)

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_spyre_backend_spyre_backend_dev["spyre-backend/spyre-backend-dev ↺<br/>image · b6af73edcd2c"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴 ↺<br/>image · fc1752087216"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 7dc77fe364ea"]
  end
  subgraph Lamd64_3["amd64 L3 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · c28982d013ad"]
  end
  n_amd64_spyre_backend_spyre_backend_dev --> n_amd64_torch_spyre_torch_spyre_dev
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_spyre_backend_spyre_backend_dev sReused;
  class n_amd64_torch_spyre_torch_spyre_dev sReused;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;

✅ orch trigger-pr-validation — green · arches amd64 · fp amd64=913da65a

level	component	arch	build	smoke	unit	integration	trunk	regression	perf
L0	spyre-backend/spyre-backend-dev	amd64	✅ ok (reused)	·	·	·	·	·	·
L1	torch-spyre/torch-spyre-dev	amd64	✅ ok (reused)	🟢	·	🟡 gha	·	·	·
L2	hf-adapters/hf-adapters-dev	amd64	✅ ok	🟢	·	🟢 gha	·	·	·
L3	spyre-inference/spyre-inference-dev	amd64	✅ ok	🟢	·	🟡 gha	·	·	·

GHA test runs:

[hf-adapters/amd64] integration: SUCCESS https://github.com/torch-spyre/hf-adapters/actions/runs/34531486567
[torch-spyre/amd64] integration: FAILURE https://github.com/torch-spyre/torch-spyre/actions/runs/34531194141
[spyre-inference/amd64] integration: FAILURE https://github.com/torch-spyre/spyre-inference/actions/runs/34531795945

⚠️ advisory failures only — mergeable, see below

Build: built 2 · reused 2

Tests: passed 3 · blocking 0 · advisory 2 · infra/inconclusive 0 · no signal 0

Failures by kind

⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

spyre-ci · 2026-09-10T21:15:49Z

GHA test runs:

[torch-spyre/x86_64] integration: DISPATCHED https://github.com/torch-spyre/torch-spyre/actions/runs/34531194141

spyre-ci · 2026-09-10T21:18:53Z

GHA test runs:

[hf-adapters/x86_64] integration: DISPATCHED https://github.com/torch-spyre/hf-adapters/actions/runs/34531486567

spyre-ci · 2026-09-10T21:22:13Z

GHA test runs:

[spyre-inference/x86_64] integration: DISPATCHED https://github.com/torch-spyre/spyre-inference/actions/runs/34531795945

HarikrishnanBalagopal requested review from BenjSz and kmehant as code owners September 10, 2026 15:22

HarikrishnanBalagopal force-pushed the hfa-uv-venv-main branch from 6e8d63c to 2dfac34 Compare September 10, 2026 15:34

BenjSz approved these changes Sep 10, 2026

View reviewed changes

HarikrishnanBalagopal mentioned this pull request Sep 10, 2026

[DO NOT MERGE] control: hf-adapters suite on legacy runners (baseline for the /next shadow PR) #183

Open

ashokponkumar enabled auto-merge September 10, 2026 18:47

ashokponkumar disabled auto-merge September 10, 2026 20:47

ashokponkumar enabled auto-merge September 10, 2026 20:47

Uh oh!

Conversation

HarikrishnanBalagopal commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Why at the workflow level

Issue

Validation

Uh oh!

BenjSz left a comment

Choose a reason for hiding this comment

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ spyre-test: failure

⚠️ advisory failures only — mergeable, see below

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

ashokponkumar commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ spyre-test: failure

⚠️ advisory failures only — mergeable, see below

Uh oh!

HarikrishnanBalagopal commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

ashokponkumar commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ spyre-test: failure

⚠️ advisory failures only — mergeable, see below

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

spyre-ci Bot commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HarikrishnanBalagopal commented Sep 10, 2026 •

edited

Loading

spyre-ci Bot commented Sep 10, 2026 •

edited

Loading

spyre-ci Bot commented Sep 10, 2026 •

edited

Loading

spyre-ci Bot commented Sep 10, 2026 •

edited

Loading