Skip to content

fix: fail fast when a rule exceeds the --max-memory/--max-threads cap#49

Merged
ShixiangWang merged 1 commit into
Traitome:mainfrom
andrewbudge:fix/max-memory-preflight
Jun 30, 2026
Merged

fix: fail fast when a rule exceeds the --max-memory/--max-threads cap#49
ShixiangWang merged 1 commit into
Traitome:mainfrom
andrewbudge:fix/max-memory-preflight

Conversation

@andrewbudge

Copy link
Copy Markdown
Collaborator

Problem

A rule that declares more memory (or threads) than an explicit --max-memory / --max-threads budget can never be scheduled — but today that breach is only discovered lazily, inside the per-rule resource check, once execution actually reaches the offending rule. Earlier rules run and do real work first, so a long pipeline can grind through hours of jobs only to die mid-run on a rule that was impossible from the start.

Repro (before this PR):

$ oxo-flow run wf.oxoflow --max-memory 100   # rule 2 declares memory = "8G"
Running: [1/2] cheap_first
  ✓ cheap_first (0.1s)                         # real work done...
Running: [2/2] hungry_second
Error: resource exhausted: rule 'hungry_second' requires ... 8192MB (available: 100MB)

Fix

Add a pre-flight feasibility check (scheduler::check_budget_feasibility) that compares every scheduled rule's declared request against the configured cap before any rule executes, and aborts with a clear, aggregated error naming each breaching rule:

$ oxo-flow run wf.oxoflow --max-memory 100
Error: resource budget too small for 1 rule(s); no rules were run:
  - rule 'hungry_second' requires 8192MB memory but --max-memory caps the run at 100MB

cheap_first no longer runs — no wasted work.

Auto-detect (no explicit cap) is intentionally not treated as a hard limit: that case stays a warning (validate_resources_against_system), since an overcommit against detected physical RAM may still succeed.

Tests

  • 4 unit tests in scheduler.rs: explicit memory breach, explicit threads breach, within-budget, and None-cap-is-not-a-hard-limit.
  • 1 CLI integration test (cli_run_fails_fast_when_rule_exceeds_max_memory) asserting the run fails, the error names the rule + cap, and the earlier rule's output is never produced.

cargo fmt --check, cargo clippy --workspace -- -D warnings, and the full --workspace test suite all pass.

A rule that declares more memory (or threads) than an explicit
--max-memory / --max-threads budget can never be scheduled, but the
breach was only discovered lazily, inside the per-rule resource check,
once execution reached that rule. Earlier rules ran and did real work
first, so a long pipeline could grind through hours of jobs only to die
mid-run on a rule that was impossible from the start.

Add a pre-flight feasibility check (scheduler::check_budget_feasibility)
that compares every scheduled rule's declared request against the
configured cap before any rule executes, and abort with a clear,
aggregated error naming each breaching rule. Auto-detect (no explicit
cap) is intentionally not treated as a hard limit — that case stays a
warning, since an overcommit may still succeed.
@ShixiangWang

Copy link
Copy Markdown
Member

great!

@ShixiangWang
ShixiangWang merged commit e3bfb0b into Traitome:main Jun 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants