fix: fail fast when a rule exceeds the --max-memory/--max-threads cap#49
Merged
Merged
Conversation
A rule that declares more memory (or threads) than an explicit --max-memory / --max-threads budget can never be scheduled, but the breach was only discovered lazily, inside the per-rule resource check, once execution reached that rule. Earlier rules ran and did real work first, so a long pipeline could grind through hours of jobs only to die mid-run on a rule that was impossible from the start. Add a pre-flight feasibility check (scheduler::check_budget_feasibility) that compares every scheduled rule's declared request against the configured cap before any rule executes, and abort with a clear, aggregated error naming each breaching rule. Auto-detect (no explicit cap) is intentionally not treated as a hard limit — that case stays a warning, since an overcommit may still succeed.
Member
|
great! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A rule that declares more memory (or threads) than an explicit
--max-memory/--max-threadsbudget can never be scheduled — but today that breach is only discovered lazily, inside the per-rule resource check, once execution actually reaches the offending rule. Earlier rules run and do real work first, so a long pipeline can grind through hours of jobs only to die mid-run on a rule that was impossible from the start.Repro (before this PR):
Fix
Add a pre-flight feasibility check (
scheduler::check_budget_feasibility) that compares every scheduled rule's declared request against the configured cap before any rule executes, and aborts with a clear, aggregated error naming each breaching rule:cheap_firstno longer runs — no wasted work.Auto-detect (no explicit cap) is intentionally not treated as a hard limit: that case stays a warning (
validate_resources_against_system), since an overcommit against detected physical RAM may still succeed.Tests
scheduler.rs: explicit memory breach, explicit threads breach, within-budget, andNone-cap-is-not-a-hard-limit.cli_run_fails_fast_when_rule_exceeds_max_memory) asserting the run fails, the error names the rule + cap, and the earlier rule's output is never produced.cargo fmt --check,cargo clippy --workspace -- -D warnings, and the full--workspacetest suite all pass.