Skip to content

perf(framework): run an analyzer's units concurrently - #54

Merged
elecnix merged 1 commit into
mainfrom
perf/unit-parallelism
Aug 9, 2026
Merged

perf(framework): run an analyzer's units concurrently#54
elecnix merged 1 commit into
mainfrom
perf/unit-parallelism

Conversation

@elecnix

@elecnix elecnix commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Problem

Sessions were the only axis of fan-out. runAnalyzer awaited each unit in turn, so one session could never have more than a single call in flight, however the limits were set.

Measured on a real corpus at 40-way session concurrency:

per-call latency ~1.5 s (flat at both 10- and 40-way)
theoretical throughput at 40 lanes ~26 calls/s
actual ~7.8 calls/s (~30% utilisation)
going 10 → 40 concurrency only 1.7×

The lanes were not waiting on the provider. Many were doing deterministic work — turn construction, tokenising, matching — and issuing no calls at all, while a session with 100 terms to judge serialised them through one lane for over two minutes.

Raising --llm-concurrency could not fix this: the ceiling was how many sessions happened to be issuing calls at the same instant, not the semaphore. And the degenerate case was worse — --session X alone ran at concurrency 1 regardless of any flag.

Change

Units now run through the existing mapWithConcurrency, bounded by a new unitConcurrency. The analyze command sets it to the LLM concurrency, so a single session can reach the gate by itself.

Layered under the existing limits, not replacing them:

  • the global semaphore around the LLM caller is still the only thing deciding provider load
  • better-sqlite3 is synchronous, so persistence cannot interleave mid-write
  • unitConcurrency defaults to 1, so nothing changes for callers that do not opt in

One honest behaviour change

Fail-fast on configuration faults is preserved, but units already in flight now finish. A bad model spec costs at most one concurrency width of error nodes instead of exactly one — still bounded, against the 113,992 it produced before fail-fast existed. Covered by test.

Test plan

  • npm test429 passing. New tests/component/unit-parallelism.test.ts pins what could plausibly break under concurrency: one node per unit with no duplicates, idempotent re-runs, the configured limit respected, default-sequential behaviour unchanged, and early stop on a configuration fault.
  • node --import tsx test/integration/test-commands.ts21 passing.
  • npx tsc --noEmit clean.

🤖 Generated with Claude Code

Sessions were the only axis of fan-out. `runAnalyzer` awaited each unit in turn,
so one session could never have more than a single call in flight however the
limits were set.

Measured on a real corpus at 40-way session concurrency: per-call latency held
steady around 1.5s, which should sustain ~26 calls/s across 40 lanes, but actual
throughput was ~7.8/s — roughly 30% utilisation. The lanes were not waiting on
the provider; many were busy with deterministic work (turn construction,
tokenising, matching) and issuing no calls at all, while a session with 100 terms
to judge serialised them through one lane for well over two minutes.

Raising --llm-concurrency did not fix it, and could not: the ceiling was how many
sessions happened to be doing LLM work at the same instant, not the semaphore.
Going 10 → 40 bought only 1.7x. The degenerate case was worse still — `--session
X` on its own ran at concurrency 1 regardless of any flag.

Units now run through mapWithConcurrency, bounded by a new `unitConcurrency`,
which the analyze command sets to the LLM concurrency so a single session can
reach the gate by itself.

This is layered under the existing limits rather than replacing them. The global
semaphore around the LLM caller is still the only thing deciding provider load,
and better-sqlite3 is synchronous, so persistence cannot interleave mid-write.
`unitConcurrency` defaults to 1, so nothing changes for callers that do not opt in.

Fail-fast on configuration faults is preserved with one honest difference: units
already in flight finish, so a bad model spec now costs at most one concurrency
width of error nodes instead of exactly one. That is still bounded, against the
113,992 it produced before fail-fast existed.

Tests cover the properties that could plausibly break under concurrency: one node
per unit with no duplicates, idempotent re-runs, the configured limit respected,
default-sequential behaviour unchanged, and early stop on a configuration fault.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@elecnix
elecnix merged commit dd470b0 into main Aug 9, 2026
3 checks passed
@elecnix
elecnix deleted the perf/unit-parallelism branch August 9, 2026 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant