When the test command has no mode flags, all four modes become the same command

[sdubagun-amd]

What happens

GEAK turns the user's test command into a COMMANDMENT with four modes: --correctness, --benchmark, --full-benchmark, and --profile.

But if the user's command has none of GEAK's mode flags - for example a plain:

timeout 600 python op_tests/test_rmsnorm2dFusedAddQuant.py

then GEAK puts the same unchanged command into all four modes. Every mode just re-runs the exact same thing. In a real run, the generated COMMANDMENT.md had four identical timeout 600 python ... bodies.

Why it happens

• src/minisweagent/run/preprocess_v3/tools.py:111 - _substitute_mode_flag: when the command has no known flag, it returns the command unchanged (line 127). So nothing distinguishes the four modes.
• _make_tool_commandment_from_user_command then emits that same command for every mode.
• The harness-synthesis path (tools.py:184, _try_synthesize_shell_contract_harness) only triggers for compound cmd_a && cmd_b commands (if "&&" not in cmd: return None), so a single plain command never gets a proper per-mode harness.

Real-world impact

Two problems showed up in a kernel run because of this:

1. Wrong operation benchmarked. The harness defaults to --mode 1 (plain rmsnorm), but one kernel was add_rmsnorm, which needs --mode 2. Since no mode was ever chosen, that kernel was measured on the wrong op.
2. Timeouts. With no shape/size arguments chosen per mode, every mode ran the harness's full default sweep and hit the 600s limit before optimization could begin.

Expected vs actual

• Expected: the four modes are genuinely different - correctness runs a small quick check, benchmark/full-benchmark use the right op and sizes, profile wraps the benchmark. If a mode truly cannot be built, say so clearly.
• Actual: all four modes are identical copies of the user's command, and a missing flag silently produces duplicates instead of a warning.

Suggested direction

Let the preprocessor's orchestrator LLM author the actual command for each of the four modes (choosing the right op / --mode, small shapes that finish under the timeout, and iteration counts that honor GEAK_BENCHMARK_ITERATIONS), instead of mechanically copying one command into all four. When a mode cannot be authored, emit a visible warning rather than a silent duplicate.

[sdubagun-amd]

Update — PR #264 addresses this, and the scope grew during testing

While fixing the reported flag-less case, end-to-end testing surfaced several adjacent test-command shapes that had the same class of failure. PR #264 now covers the full matrix below.

The test instruction is…	Prompt lists shapes?	Result
has a flag + a "supports all 4 modes" note	yes / no	works
exactly one flag, no note	no	works — A1 happy path
exactly one flag, no note	yes	fixed — a shapes pre-check routes it to the harness generator (A2) before A1, so the prompt shapes are honored instead of triggering the full default sweep / timeout
plain command, no flag, no &&	yes / no	works
build + correctness + benchmark joined by &&	no	works
build + correctness + benchmark joined by &&	yes	fixed — shapes pre-check routes to the harness generator so shapes are honored
two different tests joined by && (e.g. mode 1 && mode 2)	yes / no	fixed — detected as an amalgamation (a && with no leading build/compile prefix) at both the CLI and preprocessor layers; refused with PATH_A_FLAG_MISSING and routed to the harness generator, which builds one harness emitting one aggregate metric
no command, prompt has shapes	yes	works
no command, no shapes	no	works
a ready-made test file	—	works

How the fixes work

1. Amalgamation && (the most dangerous — silently dropped one metric): the original CLI split && blindly into correctness/performance before the preprocessor saw it. Now is_amalgamation_command (a && with no leading compile prefix) is checked at both the CLI (mini.py) and the preprocessor (commandment_from_user_command, flag-independent), refusing and routing to the harness generator. Routing by build-token subsumes "same script twice" — the generator handles both same-script and different-script cases.
2. Shapes ignored (single-flag or build-&& + prompt shapes → default sweep timeout): a shapes pre-check in the orchestrator runs before the A1/A2 split and diverts shape-bearing commands to the harness generator.
3. A separate baseline bug found during testing — the synthesized harness ran but emitted no latency because GEAK_WORK_DIR wasn't exported to the baseline subprocess — is also fixed.

Note: the shape-routing (cases #2/#3) lives in orchestrator prose because the deterministic tool is shape-blind, so those paths have no deterministic unit-test net and are covered by manual end-to-end checks.