feat(nodes): run heavy-model tests only on hardware that can run them (`requiresHardware`)

[asclearuc]

Related: #2258 adds Qwen3-VL 4B, Mage-VL 4B, SAM 3 and LLMDet-large. These models need more GPU or unified memory than a typical laptop has, so their full tests can't run everywhere. This ticket makes such tests skip, or fail in strict mode, on machines that can't run them, so #2258 can add real end-to-end tests for them.

Problem Statement

builder nodes:test-full runs every fulltest profile on whatever machine it starts on. No CI runner has a GPU, so these tests run on developer machines and GPU boxes, and their hardware varies widely. The 4B models from #2258 (Qwen3-VL, Mage-VL) have 8–10 GB of weights and load at float32 on Apple Silicon. On a laptop their tests run out of memory or fall back to something unusably slow.

Related gaps in the node test harness:

• Timeouts. The timeout of a test group is parsed but never applied; the only limit is the 600 s pytest-timeout default.
• Env vars. When a requires environment variable is missing, the test silently disappears. The docs say it is skipped.
• "Heavy" tests. A test counts as heavy when its node has the gpu capability or appears in a hardcoded list (_HEAVY_TEST_NODES). The tag is unreliable: prompt and question carry it but load no model.
• Parallel runs. Nothing enforces --dist loadgroup, so a custom --dist runs heavy tests side by side.
• Model downloads. nodes:test-full has no pre-download step, so models download inside the timed tests. A pre-download step also has to:
  • fetch only the models of tests that will run;
  • respect pinned revisions;
  • fetch only the files the loaders read, not whole repos (e.g. 6.4 GB for SmolVLM-500M's 0.95 GB of weights).
• Order-dependent skips. Tests that import a node's package directly (psycopg2, img2table, python-docx, …) skip until some earlier pipeline happened to load that node on the same dist, so results depend on the checkout's history.

Proposed Solution

1. Contract. A test group can declare requiresHardware:

   • Machine types cuda / mps / cpu; a type that isn't listed isn't allowed.
   • Per-type minimums: vramGb (free VRAM on cuda:0) and ramGb (total memory), plus an optional timeout.
   • false means the group needs no special hardware.

   Example:

   "requiresHardware": { "cuda": { "vramGb": 11 }, "mps": { "ramGb": 32, "timeout": 1800 } }

2. Probe. A new module, ai.common.utils.hardware, predicts the device pick_torch_device() would pick, without importing torch: NVML for NVIDIA GPUs (honouring CUDA_VISIBLE_DEVICES), the platform for Apple Silicon, psutil for memory.

3. Gate (nodes/test/framework/gate.py, used by conftest.py):

   • The probe runs once, in the main pytest process, and xdist workers receive its result.
   • A test that can't run here is skipped with a [hardware] reason. The reason names the processes holding VRAM when free VRAM is the problem.
   • Before each gated CUDA test, the harness waits up to 60 s for the declared VRAM to be free, and otherwise fails.

4. Remote servers and strict mode.

   • ROCKETRIDE_TEST_DEVICE, ROCKETRIDE_TEST_VRAM_GB and ROCKETRIDE_TEST_RAM_GB describe the test server's hardware when it isn't this machine.
   • A non-local ROCKETRIDE_URI with no override skips gated tests with a [remote] reason.
   • ROCKETRIDE_TEST_HARDWARE_STRICT=1 turns hardware skips into failures, so a GPU box can't report green by skipping everything.

5. Timeouts. The group timeout and the per-type timeout are applied, but only above the suite default.

6. Parallel runs.

   • A test is heavy when its group declares requiresHardware.
   • Heavy tests run in hwN xdist groups. ROCKETRIDE_TEST_HW_LANES=1|N|auto sets how many groups exist; auto sizes them from free VRAM/RAM and the declared needs.
   • Any other --dist mode with several workers stops the run before it starts when heavy tests are selected.

7. Warmup. A new pytest option, --warmup-models[=plan], downloads the models of the selected tests that will run, at their pinned revisions and only the files the loaders read.

   • nodes:test-full runs it before the test server starts.
   • --warmup=plan|off lists the downloads or skips the pass.
   • Other tooling can call the same pass.

8. Report. A new builder flag, --list-skipped[=hardware|remote|env|libs|marker], lists the selected tests that will be skipped, grouped by reason, without starting a server.

9. Test dependencies.

   • nodes:test / nodes:test-full install nodes/test/requirements.txt through depends().
   • That file now lists the node packages the tests import directly, so those tests stop depending on which pipelines ran before.

10. Visible env skips. A missing requires env var becomes a visible [env] skip.

11. Annotations.

    • Every test group of the model-loading nodes declares requiresHardware; prompt / question declare false.
    • BiRefNet HR moves into its own fulltest group.
    • A contract test requires a declaration on every group of a gpu node and every group with a memory_gb profile.

12. Docs. docs/development/nodes/testing.md covers the contract, the environment variables, warmup, the skip report, parallel runs and test dependencies. It also fixes the timeout / requires descriptions.

Acceptance Criteria

• On an 8 GB GPU, groups that need more are skipped with a reason, and --warmup=plan lists none of their models.
• Strict mode turns those skips into failures at setup.
• -n 2 --dist load with heavy tests selected stops before any test runs.
• With ROCKETRIDE_TEST_HW_LANES=auto and 4 workers:
  • an 80 GB GPU spreads heavy tests across 4 groups;
  • an 8 GB GPU keeps them in 1.
• --list-skipped prints its report, exits 0 and starts no server.
• Test dependencies:
  1. --list-skipped=marker lists the missing-package skips;
  2. nodes:test runs;
  3. the same report no longer lists them.
• builder test downloads no models and doesn't run the fulltests.
• The contract test fails on a missing declaration, or when cuda.vramGb is below a profile's memory_gb.
• No declared value is below what a test actually uses, measured on a real GPU and on CPU.

Alternatives Considered

• Use memory_gb as the CUDA minimum. Rejected: it is a bf16 allocation hint for the model server and says nothing about MPS or CPU.
• Keep the gpu capability as the "heavy" signal. Rejected: it is a product flag, and some text-only nodes carry it.
• Probe with torch. Rejected: it imports torch into every pytest worker and initialises CUDA there.
• Keep warmup as a standalone script with its own filter. Rejected: it can't follow -k / -m and duplicates discovery.
• Install every node's requirements before the tests. Rejected: a fresh machine would pull torch-scale packages the tests don't need.
• Install versions from constraints.lock. Rejected: the engine doesn't read the lockfile yet, and its plain PyPI torch pin would replace the CUDA build.

Affected Modules

• nodes (pipeline)
• ai

Related

• feat(vision): SmolVLM caption default, Qwen3-VL/Mage-VL profiles, SAM 3 segmentation, transformers 5.x #2258: adds the large vision models that motivate this gating. After rebasing on this change it declares requiresHardware for its caption groups and adds full-test groups for LLMDet-large and SAM 3.
• feat(nodes): gate heavy-model tests on hardware (requiresHardware) #2314: the pull request for this issue.
• rocketride-ai/rocketride-saas#715, rocketride-ai/rocketride-saas#716: the rocketride-saas side.

Out of Scope / Follow-ups

• Confirm the BiRefNet HR and feat(vision): SmolVLM caption default, Qwen3-VL/Mage-VL profiles, SAM 3 segmentation, transformers 5.x #2258 4B values on a GPU with at least 16 GB.
• Make ROCKETRIDE_TEST_HW_LANES=auto the default once the values are confirmed.
• Let the test server report its hardware, so remote targets need no environment override.
• Exact (lowering) group timeouts.

[github-actions]

🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.