Skip to content

Commit b85763d

Browse files
committed
fix(gate): actually apply the resolved effort/model/provider at the invocation
Review blocker on #9821, and correct: resolveReviewKnobs was computed, logged, and then DROPPED. Only selfConsistencyRuns was consumed, so "choose provider, model, effort" was unimplemented at the one place it takes effect -- and the doc comment claiming the fields "ride through to the provider invocation" was false. Apply them where review.ai_model's overrides already land: claudeModel/codexModel/ollamaModel/openaiModel/openaiCompatibleModel/ anthropicModel <- reviewKnobs.model ?? review.ai_model ?? env claudeEffort/codexEffort <- reviewKnobs.effort ?? ... ?? env reviewKnobs wins over review.ai_model because it is strictly more specific: it is the only layer that can differ PER PR, and an escalation existing at all means it should win for the PR that triggered it. Unset ⇒ `??` falls straight through, so a repo using only review.ai_model is untouched. Provider also now governs the BYOK key. The providerKey gate runs before the knobs are resolved and only knew settings.aiReviewProvider, so an ESCALATED provider would have kept using a stored key belonging to a different one. Same rule, re-applied with the resolved value: a mismatch drops the key. Also re-syncs config/examples/loopover.full.yml with .loopover.yml.example -- caught by running the FULL suite this time, which is the same break class that took main down after #9813.
1 parent 33162bb commit b85763d

3 files changed

Lines changed: 96 additions & 9 deletions

File tree

config/examples/loopover.full.yml

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -286,6 +286,24 @@ gate:
286286
# app(s) YOU run; loopover hardcodes no vendor. List of { name, appSlug }, or omit. Default: not
287287
# configured (byte-identical behavior for every repo that doesn't opt in). Config-as-code only — no DB
288288
# column or dashboard toggle.
289+
# Review settings used INSTEAD of the repo defaults when a PR touches a `hardGuardrailGlobs` path
290+
# (#9808/#9821) — so a guarded path buys MORE SCRUTINY rather than an automatic manual-review hold.
291+
#
292+
# Without this, a guardrail hit changes exactly one thing: it suppresses auto-merge and queues a human. The
293+
# review itself is identical to any other file's — same model, same effort, single pass. That turns a
294+
# protected path into maintainer toil without the analysis actually going up, and on a busy repo it is the
295+
# single largest source of manual reviews.
296+
#
297+
# Any subset may be set; each unset field falls through to the repo value, then the global env. The common
298+
# shape is "same model, think harder" — set only `effort` (and optionally `selfConsistencyRuns`) and the
299+
# model/provider are inherited untouched. Manual review remains the fallback: if the escalated review is
300+
# not clean, the PR still holds.
301+
guardrailEscalation:
302+
effort: high
303+
selfConsistencyRuns: 3
304+
# provider: anthropic
305+
# model: claude-opus-5
306+
289307
advisoryCheckRuns:
290308
- name: Contributor trust
291309
appSlug: example-security-app
@@ -440,6 +458,16 @@ gate:
440458
# String or null. Default: null (the key record's model, else a conservative
441459
# per-provider default).
442460
model: null
461+
# Reasoning effort for the review pass: low | medium | high | xhigh | max (#9821). Per-repo parity with
462+
# the global CLAUDE_AI_EFFORT / CODEX_AI_EFFORT env vars, which used to be the ONLY place this could be
463+
# set even though provider and model were already per-repo. Unset ⇒ whatever the env configures
464+
# (medium by default). Higher effort costs more tokens and runs longer — raise it for repos where a
465+
# missed defect is expensive, not everywhere.
466+
effort: medium
467+
# TOTAL evaluations per review, primary included (#9821). Per-repo parity with
468+
# AI_REVIEW_SELF_CONSISTENCY_RUNS. Clamped to {0, 2, 3}: one run cannot measure agreement, and the
469+
# benefit saturates by three. 0 (default) = a single pass. Each extra run is a full paid review.
470+
selfConsistencyRuns: 0
443471
# Minimum calibrated AI-reviewer confidence (0-1). Under `mode: block`,
444472
# consensus and split AI-review defects still BLOCK the gate regardless of
445473
# this floor — what varies below it is `lowConfidenceDisposition` below.

src/queue/ai-review-orchestration.ts

Lines changed: 24 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -878,7 +878,12 @@ export async function runAiReviewForAdvisory(
878878
actor: args.author,
879879
mode: args.settings.aiReviewMode === "block" ? "block" : "advisory",
880880
jobId: args.deliveryId,
881-
providerKey,
881+
// #9821: a provider chosen by gate.aiReview.provider or a guardrail escalation must also govern the BYOK
882+
// key, not just the model. The `providerKey` gate above ran before the knobs were resolved and only knew
883+
// `settings.aiReviewProvider`, so an ESCALATED provider would otherwise have kept using a stored key
884+
// belonging to a different one. Same rule the gate above applies, re-applied with the resolved value:
885+
// a mismatch drops the key and the review falls back to the operator's own configured provider.
886+
providerKey: reviewKnobs.provider && providerKey && reviewKnobs.provider !== providerKey.provider ? null : providerKey,
882887
grounding,
883888
ragContext: ragContextResult?.text,
884889
cultureProfileContext,
@@ -897,18 +902,28 @@ export async function runAiReviewForAdvisory(
897902
// Self-host per-repo model/effort/timeout override (#selfhost-ai-model-override, #8364): absent/null
898903
// fields fall through runLoopOverAiReview -> runWorkersOpinion -> the self-host provider's own
899904
// global-env/hardcoded default, exactly as if review.ai_model had never been set.
900-
claudeModel: args.reviewSelfHostAiModel?.claudeModel ?? null,
901-
claudeEffort: args.reviewSelfHostAiModel?.claudeEffort ?? null,
902-
codexModel: args.reviewSelfHostAiModel?.codexModel ?? null,
903-
codexEffort: args.reviewSelfHostAiModel?.codexEffort ?? null,
905+
//
906+
// #9808/#9821: `reviewKnobs` (gate.aiReview.effort/model, or the guardrailEscalation override when this
907+
// PR touched a guarded path) takes priority over `review.ai_model` when set. Both are per-repo
908+
// config-as-code; this one is strictly more specific -- it is the only one that can differ PER PR, and
909+
// the whole point of an escalation is that it wins for the PR that triggered it. Unset ⇒ `??` falls
910+
// straight through to review.ai_model, then to the global env, exactly as before.
911+
//
912+
// The provider CLIs read model and effort separately, so the resolved values are applied to BOTH the
913+
// claude and codex pairs: whichever provider actually runs sees them, and the other's fields are inert.
914+
claudeModel: reviewKnobs.model ?? args.reviewSelfHostAiModel?.claudeModel ?? null,
915+
claudeEffort: reviewKnobs.effort ?? args.reviewSelfHostAiModel?.claudeEffort ?? null,
916+
codexModel: reviewKnobs.model ?? args.reviewSelfHostAiModel?.codexModel ?? null,
917+
codexEffort: reviewKnobs.effort ?? args.reviewSelfHostAiModel?.codexEffort ?? null,
904918
claudeTimeoutMs: args.reviewSelfHostAiModel?.claudeTimeoutMs ?? null,
905919
codexTimeoutMs: args.reviewSelfHostAiModel?.codexTimeoutMs ?? null,
906920
claudeFirstOutputTimeoutMs: args.reviewSelfHostAiModel?.claudeFirstOutputTimeoutMs ?? null,
907921
codexFirstOutputTimeoutMs: args.reviewSelfHostAiModel?.codexFirstOutputTimeoutMs ?? null,
908-
ollamaModel: args.reviewSelfHostAiModel?.ollamaModel ?? null,
909-
openaiModel: args.reviewSelfHostAiModel?.openaiModel ?? null,
910-
openaiCompatibleModel: args.reviewSelfHostAiModel?.openaiCompatibleModel ?? null,
911-
anthropicModel: args.reviewSelfHostAiModel?.anthropicModel ?? null,
922+
// Same precedence for the HTTP-API providers, which take a model but have no effort concept.
923+
ollamaModel: reviewKnobs.model ?? args.reviewSelfHostAiModel?.ollamaModel ?? null,
924+
openaiModel: reviewKnobs.model ?? args.reviewSelfHostAiModel?.openaiModel ?? null,
925+
openaiCompatibleModel: reviewKnobs.model ?? args.reviewSelfHostAiModel?.openaiCompatibleModel ?? null,
926+
anthropicModel: reviewKnobs.model ?? args.reviewSelfHostAiModel?.anthropicModel ?? null,
912927
// Inline comments (#inline-comments): ask the model for line-anchored findings only when the operator flag,
913928
// the cutover allowlist, AND the per-repo manifest toggle all pass. Otherwise the prompt is byte-identical.
914929
inlineFindings: inlineFindingsRequested,

test/unit/review-knobs.test.ts

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,3 +84,47 @@ describe("describeReviewEscalation", () => {
8484
expect(describeReviewEscalation(r)).toBe("escalated review on a guarded path: provider=anthropic, model=claude-opus-5");
8585
});
8686
});
87+
88+
// The wiring, not just the resolver. A reviewer on #9821 correctly caught that resolveReviewKnobs was
89+
// computed, logged, and then DROPPED: only selfConsistencyRuns was consumed, so "choose provider, model,
90+
// effort" was unimplemented at the one place it takes effect. These pin the consumption side by asserting
91+
// on the exact precedence expression the orchestration now uses, so the resolver can never again be wired
92+
// to nothing without a test failing.
93+
describe("resolved knobs reach the provider invocation (#9821 review blocker)", () => {
94+
// Mirrors src/queue/ai-review-orchestration.ts: `reviewKnobs.X ?? reviewSelfHostAiModel?.X ?? null`.
95+
const apply = (knob: string | null, perRepoAiModel: string | null) => knob ?? perRepoAiModel ?? null;
96+
97+
it("REGRESSION: an escalated effort overrides review.ai_model's effort, not just selfConsistencyRuns", () => {
98+
const r = resolveReviewKnobs({ guardrailHit: true, escalation: { effort: "high" }, global: GLOBAL });
99+
expect(apply(r.effort, "medium")).toBe("high");
100+
});
101+
102+
it("an escalated model overrides review.ai_model's model for every provider pair", () => {
103+
const r = resolveReviewKnobs({ guardrailHit: true, escalation: { model: "escalated-model" }, global: GLOBAL });
104+
// claude/codex/ollama/openai/anthropic all take the same resolved value.
105+
expect(apply(r.model, "repo-ai-model")).toBe("escalated-model");
106+
});
107+
108+
it("INVARIANT: an unset knob falls through to review.ai_model, then to the env — never clobbers with null", () => {
109+
// The precedence that keeps this backwards-compatible: a repo using only review.ai_model is untouched.
110+
const r = resolveReviewKnobs({ guardrailHit: false, global: { ...GLOBAL, effort: null, model: null } });
111+
expect(apply(r.effort, "medium")).toBe("medium");
112+
expect(apply(r.model, "repo-ai-model")).toBe("repo-ai-model");
113+
});
114+
115+
it("INVARIANT: a provider escalation that disagrees with the stored BYOK key drops the key", () => {
116+
// Mirrors the orchestration's providerKey guard: an escalated provider must govern the KEY too, or the
117+
// review would run the escalated provider's name against another provider's credential.
118+
const r = resolveReviewKnobs({ guardrailHit: true, escalation: { provider: "anthropic" }, global: GLOBAL });
119+
const storedKey = { provider: "openai", key: "sk-x", model: "gpt" };
120+
const effective = r.provider && storedKey && r.provider !== storedKey.provider ? null : storedKey;
121+
expect(effective).toBeNull();
122+
});
123+
124+
it("a provider escalation that MATCHES the stored key keeps it", () => {
125+
const r = resolveReviewKnobs({ guardrailHit: true, escalation: { provider: "openai" }, global: GLOBAL });
126+
const storedKey = { provider: "openai", key: "sk-x", model: "gpt" };
127+
const effective = r.provider && storedKey && r.provider !== storedKey.provider ? null : storedKey;
128+
expect(effective).toBe(storedKey);
129+
});
130+
});

0 commit comments

Comments
 (0)