Skip to content

Commit 0c371b7

Browse files
BunsDevclaudeCopilot
authored
Auto routing: Jev judges each turn, code picks model, effort, and fast mode (#10)
* Auto routing: Jev judges each turn, code picks model, effort, and fast mode Adds an ⚡ Auto toggle per thread. Before a turn, a fixed set of typed questions (task kind, complexity, blast radius, wants-speed, needs-deep- reasoning, depends-on-prior-turns) goes to TypeSafe's Jev over a compact digest of the request — never file contents. A deterministic policy turns the judgments into backend/model/effort/fast within user bounds (posture, effort ceiling, confidence floor, daily premium budget, backend switching) and leaves an expandable receipt in the transcript. Without a key, or after an auth/billing rejection, a built-in heuristic judges and says so. Hand-picked models on Auto threads teach a per-task tier offset stored in ~/.modex/app/routing-fit.json; failures nudge it too. Claude gets --effort and --settings fastMode; Codex gets per-turn effort and serviceTierForTurn and now reports serviceTiers. Settings gains an Auto routing section with judge status and a reset for the learned fit. Coding turns still run only through the CLIs; AGENTS.md states the judge as the one optional network call. docs/auto-routing.md covers the design, the CLI/local-model survey, and how to verify. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Settings dialog scrolls when the window is shorter than its content The Auto routing section made the modal taller than the macOS CI runner's window, so Playwright could not reach the Cancel button ("element is outside of the viewport"). Cap the dialog at the viewport and let it scroll. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Change assistant text output to a placeholder Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding 'Exclude the current user item from state history' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Implement abort controller for request handling Add abort controller handling for request timeout Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 2c4db82 commit 0c371b7

29 files changed

Lines changed: 1687 additions & 44 deletions

‎AGENTS.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,10 @@ itself driving one of them.
3737

3838
- `apps/desktop` — the Electron app (`src/main` process + backends, `src/renderer` React UI,
3939
`e2e/` Playwright). `packages/core` — the offline scripted engine used by demos and tests.
40-
- Real models run only through the `claude` and `codex` CLIs. There is no API mode; do not
41-
add one.
40+
- Real models run only through the `claude` and `codex` CLIs. There is no API mode for
41+
coding turns; do not add one. The one network call outside the CLIs is the optional Auto
42+
routing judge (Jev, `apps/desktop/src/main/engine/routing/`), which answers typed questions
43+
about a request and never runs a turn or sees file contents; without `TYPESAFE_API_KEY`
44+
it is replaced by a built-in heuristic. See `docs/auto-routing.md`.
4245
- Scratch scripts go in `apps/desktop/.probes/` (git-ignored). Playwright wipes
4346
`apps/desktop/test-results/` on every run.

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,13 @@ modex/
2424
the CLI reports (`model/list` for Codex — every current GPT model with its reasoning-effort
2525
row; the `fable`/`opus`/`sonnet`/`haiku` latest aliases for Claude), default preselected, no
2626
free-text entry.
27+
- **⚡ Auto routing.** Turn on Auto and a fast judge — TypeSafe's Jev, or a built-in
28+
heuristic when no `TYPESAFE_API_KEY` is set — reads each request before the turn and picks
29+
the model, reasoning effort, and fast mode within the bounds you set (posture, effort
30+
ceiling, daily premium-turn budget, confidence floor). Every turn gets a one-line receipt
31+
explaining the pick; picking a model by hand teaches Auto your preference for that kind of
32+
task. The coding turn itself still runs only through the CLIs. See
33+
[docs/auto-routing.md](docs/auto-routing.md).
2734
- **Projects & threads.** Open any local folder as a project. Each project holds threads;
2835
threads run **in parallel** and independently, each with its own backend, model, and mode.
2936
- **Worktree threads.** `⑂` starts a thread in a fresh `git worktree` so agents never step on

‎apps/desktop/e2e/app.spec.ts‎

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ let repo: string;
4343

4444
test.beforeAll(async () => {
4545
({ home, repo } = seedHome());
46-
app = await electron.launch({ args: [appDir], cwd: appDir, env: { ...process.env, MODEX_HOME: home, MODEX_E2E: "1" } });
46+
app = await electron.launch({ args: [appDir], cwd: appDir, env: { ...process.env, MODEX_HOME: home, MODEX_E2E: "1", MODEX_NO_LOGIN_PATH: "1", TYPESAFE_API_KEY: "" } });
4747
page = await app.firstWindow();
4848
await page.waitForLoadState("domcontentloaded");
4949
});
@@ -173,11 +173,51 @@ test("⇧⌘N creates a worktree thread and the header shows its branch and path
173173

174174
test("state survives a relaunch: the thread and its transcript are restored", async () => {
175175
await app.close();
176-
app = await electron.launch({ args: [appDir], cwd: appDir, env: { ...process.env, MODEX_HOME: home } });
176+
app = await electron.launch({ args: [appDir], cwd: appDir, env: { ...process.env, MODEX_HOME: home, MODEX_NO_LOGIN_PATH: "1", TYPESAFE_API_KEY: "" } });
177177
page = await app.firstWindow();
178178
await expect(page.locator(".threads .thread")).toHaveCount(2);
179179
await page.locator(".threads .thread").nth(1).click();
180180
await expect(page.locator(".approval .approval-answer")).toHaveText("Approved");
181181
await expect(page.locator(".msg.assistant").last()).toContainText("added CONTRIBUTING.md");
182182
await expect(page.locator(".pill.plan")).toHaveText(/Plan/);
183183
});
184+
185+
test("⚡ Auto: the judge picks a model before the turn and leaves an expandable receipt", async () => {
186+
// A fresh thread (default mode: chat). No TypeSafe key in this environment, so the built-in heuristic judges.
187+
await page.keyboard.press("Meta+n");
188+
await expect(page.locator(".threads .thread")).toHaveCount(3);
189+
await expect(page.locator(".btn.toggle.auto")).not.toHaveClass(/on/);
190+
await expect(page.locator(".pill.auto")).toHaveCount(0);
191+
await page.locator(".btn.toggle.auto").click();
192+
await expect(page.locator(".btn.toggle.auto")).toHaveClass(/on/);
193+
await expect(page.locator(".pill.auto")).toHaveText(/Auto/);
194+
await page.locator(".composer textarea").focus();
195+
await page.keyboard.type("What does this repo do?");
196+
await page.keyboard.press("Meta+Enter");
197+
const route = page.locator(".route").first();
198+
await expect(route).toBeVisible();
199+
await expect(route.locator(".route-label")).toContainText("Auto picked");
200+
await expect(route.locator(".route-label")).toContainText("Mock · mock");
201+
await expect(route.locator(".route-meta")).toContainText("quick answer · heuristic");
202+
await expect(route.locator(".route-body")).toHaveCount(0);
203+
await route.locator(".route-head").click();
204+
await expect(route.locator(".route-body")).toContainText("No TypeSafe API key found; used the built-in heuristic.");
205+
await expect(route.locator(".route-body")).toContainText("quick answer · complexity");
206+
await page.screenshot({ path: path.join(appDir, "test-results", "e2e-auto-route.png") });
207+
// The receipt sits between the user message and the agent's first reply.
208+
await expect(page.locator(".transcript > *").nth(0)).toHaveClass(/msg user/);
209+
await expect(page.locator(".transcript > *").nth(1)).toHaveClass(/route/);
210+
// Stop the scripted run; the Auto pill and the receipt survive.
211+
await page.keyboard.press("Meta+.");
212+
await expect(page.locator(".pill.status")).toHaveText("Idle");
213+
await expect(page.locator(".route")).toHaveCount(1);
214+
await expect(page.locator(".pill.auto")).toHaveText(/Auto/);
215+
// Settings explains why the heuristic judged, counts the auto turn, and exposes the policy knobs.
216+
await page.locator(".sidebar button", { hasText: "Settings" }).click();
217+
const status = page.locator("[data-testid=routing-status]");
218+
await expect(status).toContainText("No TYPESAFE_API_KEY found");
219+
await expect(status).toContainText("1 auto turn so far");
220+
await expect(page.locator(".modal select").filter({ has: page.locator("option[value=economy]") })).toHaveValue("balanced");
221+
await page.locator(".modal button", { hasText: "Cancel" }).click();
222+
await expect(page.locator(".modal")).toHaveCount(0);
223+
});

‎apps/desktop/src/main/engine/backends/claude.ts‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,16 @@ export class ClaudeBackend implements Backend {
1212
readonly id = "claude" as const;
1313
constructor(private readonly bin = process.env.MODEX_CLAUDE_BIN ?? "claude", private readonly spawnImpl = spawn) {}
1414

15+
/** `claude --effort` levels (from `claude --help`); no default so the CLI's own setting applies. */
16+
static readonly EFFORTS = ["low", "medium", "high", "xhigh", "max"];
17+
1518
static readonly MODELS: ModelInfo[] = [
1619
// Aliases always point at the newest model of each family (resolved by the CLI on every run);
1720
// a full model id such as "claude-fable-5-1" can be typed as well.
18-
{ id: "fable", label: "Fable (latest)", description: "Newest Fable — currently claude-fable-5-1", isDefault: true },
19-
{ id: "opus", label: "Opus (latest)", description: "Newest Opus — currently claude-opus-5-5" },
20-
{ id: "sonnet", label: "Sonnet (latest)", description: "Newest Sonnet — currently claude-sonnet-5" },
21-
{ id: "haiku", label: "Haiku (latest)", description: "Newest Haiku — currently claude-haiku-4-5" },
21+
{ id: "fable", label: "Fable (latest)", description: "Newest Fable — currently claude-fable-5-1", isDefault: true, efforts: ClaudeBackend.EFFORTS },
22+
{ id: "opus", label: "Opus (latest)", description: "Newest Opus — currently claude-opus-5-5", efforts: ClaudeBackend.EFFORTS },
23+
{ id: "sonnet", label: "Sonnet (latest)", description: "Newest Sonnet — currently claude-sonnet-5", efforts: ClaudeBackend.EFFORTS },
24+
{ id: "haiku", label: "Haiku (latest)", description: "Newest Haiku — currently claude-haiku-4-5", efforts: ClaudeBackend.EFFORTS },
2225
];
2326

2427
async listModels(): Promise<ModelInfo[]> {
@@ -35,6 +38,9 @@ export class ClaudeBackend implements Backend {
3538
else if (opts.mode === "agent") args.push("--permission-mode", "acceptEdits");
3639
else args.push("--permission-mode", "bypassPermissions");
3740
if (opts.model) args.push("--model", opts.model);
41+
if (opts.effort) args.push("--effort", opts.effort);
42+
// Fast mode is a settings key in Claude Code; `--settings` accepts inline JSON for this session only.
43+
if (opts.fast) args.push("--settings", JSON.stringify({ fastMode: true }));
3844
if (opts.resume) args.push("--resume", opts.resume);
3945
for (const d of opts.addDirs ?? []) args.push("--add-dir", d);
4046
return args;

‎apps/desktop/src/main/engine/backends/codex.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,11 @@ export class CodexBackend implements Backend {
9595
const res = await this.request<{ data: CodexModel[] }>("model/list", {});
9696
return res.data
9797
.filter((m) => !m.hidden)
98-
.map((m) => ({ id: m.id, label: m.displayName || m.id, description: m.description, isDefault: m.isDefault, efforts: m.supportedReasoningEfforts?.map((e) => e.reasoningEffort), defaultEffort: m.defaultReasoningEffort }));
98+
.map((m) => ({
99+
id: m.id, label: m.displayName || m.id, description: m.description, isDefault: m.isDefault,
100+
efforts: m.supportedReasoningEfforts?.map((e) => e.reasoningEffort), defaultEffort: m.defaultReasoningEffort,
101+
serviceTiers: m.serviceTiers?.map((t) => t.id), defaultServiceTier: m.defaultServiceTier ?? undefined,
102+
}));
99103
}
100104

101105
async dispose(): Promise<void> {
@@ -258,6 +262,8 @@ export class CodexBackend implements Backend {
258262
sandboxPolicy: pol.sandboxPolicy,
259263
model: opts.model || null,
260264
effort: opts.effort ?? null,
265+
// "fast" is the Codex fast-mode service tier; only this turn, the thread's tier is untouched.
266+
serviceTierForTurn: opts.fast ? "fast" : null,
261267
})
262268
.then((r) => {
263269
turnId = r.turn.id;
@@ -310,6 +316,8 @@ interface CodexModel {
310316
isDefault: boolean;
311317
supportedReasoningEfforts?: { reasoningEffort: string }[];
312318
defaultReasoningEffort?: string;
319+
serviceTiers?: { id: string; name?: string; description?: string }[];
320+
defaultServiceTier?: string | null;
313321
}
314322
interface CodexItem {
315323
type: string;

‎apps/desktop/src/main/engine/backends/types.ts‎

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
1-
import type { ApprovalAnswer, BackendId, Mode } from "../../../shared/types.js";
1+
import type { ApprovalAnswer, BackendId, Mode, ModelInfo } from "../../../shared/types.js";
2+
3+
export type { ModelInfo };
24

35
export interface ToolStart {
46
id: string;
@@ -45,8 +47,10 @@ export interface TurnOptions {
4547
/** Plan mode: think and propose, never edit. */
4648
plan: boolean;
4749
model: string;
48-
/** Reasoning effort (Codex). */
50+
/** Reasoning effort (Codex `effort`, Claude `--effort`). */
4951
effort?: string;
52+
/** Fast mode for this turn (Codex "fast" service tier, Claude fastMode setting). */
53+
fast?: boolean;
5054
/** Resumable handle from a previous turn on this thread. */
5155
resume?: string;
5256
/** Extra directories the agent may write to (worktree threads pass the project root). */
@@ -67,15 +71,6 @@ export interface Backend {
6771
dispose(): Promise<void>;
6872
}
6973

70-
export interface ModelInfo {
71-
id: string;
72-
label: string;
73-
description?: string;
74-
isDefault?: boolean;
75-
efforts?: string[];
76-
defaultEffort?: string;
77-
}
78-
7974
/** Splits a byte stream into complete lines; keeps the trailing partial line. */
8075
export class LineBuffer {
8176
private buf = "";

‎apps/desktop/src/main/engine/demo.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ export async function runDemo(o: DemoOptions): Promise<void> {
3131
// A second, finished thread so the sidebar shows history.
3232
const earlier = await o.runner.createThread(project.id, { mode: "agent", backend: "mock" });
3333
o.runner.updateThread(earlier.id, { title: "Explain the approval policy module" });
34-
const thread = await o.runner.createThread(project.id, { mode: "chat", backend: "mock" });
34+
// Auto is on for the demo thread so the transcript shows a routing decision (heuristic when no key).
35+
const thread = await o.runner.createThread(project.id, { mode: "chat", backend: "mock", auto: true });
3536
const turn = o.runner.send(thread.id, "Add a CONTRIBUTING.md with the three-step workflow (install, test, PR).");
3637
await waitFor(() => o.runner.status(thread.id) === "waiting" || o.runner.status(thread.id) === "idle" || o.runner.status(thread.id) === "error");
3738
await settle();
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
import type { BackendId, ModelInfo } from "../../../shared/types.js";
2+
3+
/**
4+
* The routing catalogue: every model a backend's CLI lists, placed on a capability ladder.
5+
*
6+
* Tier 0 = fastest/cheapest, 3 = most capable. The CLIs report names and descriptions but not
7+
* cost or capability, so tiers come from a small table of known families plus description
8+
* hints; unknown models land in the middle. The ladder is rebuilt from the live `model/list`
9+
* each time the catalogue refreshes, so new models appear without a Modex release.
10+
*/
11+
export type Tier = 0 | 1 | 2 | 3;
12+
13+
export interface Candidate {
14+
backend: BackendId;
15+
model: string;
16+
label: string;
17+
tier: Tier;
18+
efforts: string[];
19+
defaultEffort?: string;
20+
/** Service tier id that means "fast mode" for this model (Codex), if any. */
21+
fastTier?: string;
22+
isDefault?: boolean;
23+
}
24+
25+
const KNOWN: { re: RegExp; tier: Tier }[] = [
26+
// Claude Code aliases and full ids. Aliases resolve to the newest model of each family.
27+
{ re: /^(claude-)?haiku/, tier: 0 },
28+
{ re: /^(claude-)?sonnet/, tier: 1 },
29+
{ re: /^(claude-)?opus/, tier: 2 },
30+
{ re: /^(claude-)?fable/, tier: 3 },
31+
// Codex catalogue (Sept 2026). Ordering within GPT-6 follows the CLI's own descriptions when present.
32+
{ re: /^gpt-5\.5/, tier: 0 },
33+
{ re: /^gpt-5\.6/, tier: 1 },
34+
{ re: /^gpt-6-luna/, tier: 2 },
35+
{ re: /^gpt-6-sol/, tier: 2 },
36+
{ re: /^gpt-6-astra/, tier: 3 },
37+
{ re: /mini|nano|small|lite|flash/, tier: 0 },
38+
{ re: /^mock$/, tier: 1 },
39+
];
40+
41+
/** Description hints only move a model when the name is unknown. */
42+
function hintTier(description: string | undefined): Tier | null {
43+
const d = (description ?? "").toLowerCase();
44+
if (!d) return null;
45+
if (/most (capable|intelligent|powerful)|frontier|hardest|complex/.test(d)) return 3;
46+
if (/fast(est)?|cheap|quick|lightweight|latency|everyday|simple tasks/.test(d)) return 0;
47+
return null;
48+
}
49+
50+
export function tierOf(model: ModelInfo): Tier {
51+
const id = model.id.toLowerCase();
52+
for (const k of KNOWN) if (k.re.test(id)) return k.tier;
53+
return hintTier(model.description) ?? 2;
54+
}
55+
56+
export function toCandidate(backend: BackendId, m: ModelInfo): Candidate {
57+
const fastTier = m.serviceTiers?.find((t) => /^fast$|fast/i.test(t));
58+
return { backend, model: m.id, label: m.label, tier: tierOf(m), efforts: m.efforts ?? [], defaultEffort: m.defaultEffort, fastTier, isDefault: m.isDefault };
59+
}
60+
61+
/** Candidates for a backend, lowest tier first; within a tier the CLI default wins. */
62+
export function ladder(backend: BackendId, models: ModelInfo[]): Candidate[] {
63+
return models.map((m) => toCandidate(backend, m)).sort((a, b) => a.tier - b.tier || Number(Boolean(b.isDefault)) - Number(Boolean(a.isDefault)));
64+
}
65+
66+
/** The best candidate at or below `tier`; if the ladder has nothing that low, its lowest rung. */
67+
export function pick(rungs: Candidate[], tier: Tier): Candidate | undefined {
68+
if (!rungs.length) return undefined;
69+
const atOrBelow = rungs.filter((c) => c.tier <= tier);
70+
if (!atOrBelow.length) return rungs[0];
71+
const top = atOrBelow[atOrBelow.length - 1]!.tier;
72+
return atOrBelow.find((c) => c.tier === top && c.isDefault) ?? atOrBelow.find((c) => c.tier === top);
73+
}

0 commit comments

Comments
 (0)