Multitask AI engine calls: concurrent LLM turns via a consumer worker pool

[XargonWan]

Summary

Make Synth capable of running multiple AI/LLM engine calls concurrently instead of one at a time. Today every user-facing turn is fully serialized, so a single slow generation (notably the non-cancellable Selenium engine) blocks all other conversations until it finishes.

Current behaviour (the bottleneck)

There are two independent serialization points:

1. Single message-queue consumer — core/message_queue.py::_consumer_loop() is one asyncio task that pulls one item from the PriorityQueue and awaits the whole turn (await asyncio.wait_for(processing_task, timeout=...)) before pulling the next. Only the background band (priority <= PRIORITY_LOW) runs detached.
2. Global LLM chain lock — core/plugin_instance.py::_llm_chain_lock (_llm_chain_lease()) serializes every LLM chain across tasks, so even work launched from separate tasks cannot overlap.

The HTTP adapters (httpx/aiohttp) and ExternalCortexEngine.generate_response are already fully async with no locks, so the transport layer is already parallelizable — the two points above are what force sequential execution.

Proposed approach

Introduce a pool of N consumer workers plus a redesign of the global lock so it no longer serializes generation.

Phase A — consumers config

• New exposed config var (e.g. LLM_CONSUMERS), default 1 (identical to current behaviour), user-raisable, clamped to a sane max. This doubles as the feature flag and the global concurrency cap.

Phase B — Consumer worker pool (core/message_queue.py)

• Replace the single _consumer_task with a list of N identical _consumer_loop() workers all draining the same PriorityQueue.
• _start_consumer_task() → _start_consumer_tasks(n); _supervisor_loop() keeps the whole pool alive.
• Priority stays an ingress gate; completion order is not guaranteed (accepted).
• Keep compact_similar_messages under the existing _get_lock() to stay race-free across workers.
• Background branch and task_cancellable/ghost-reply logic unchanged.

Phase C — Redesign _llm_chain_lock (core/plugin_instance.py)

• The lock today also guards mutation of the module-global plugin during the trainer cortex switch (load_plugin swaps it, then it is restored). This is real shared mutable state and cannot be removed naively.
• Fix: pass the engine instance explicitly to the turn (or hold a short lock only around the switch/restore), leaving adapter.chat_completion outside any global lock.
• With consumers == 1 behaviour is identical to today; reentrancy via _llm_chain_depth preserved.

Phase D — Selenium engine

• No Synth-side lock needed: the selenium-llm-engine has its own internal FIFO queue manager and serializes concurrent requests itself. The _clamp_messages_to_char_budget mitigation stays unchanged. The Selenium engine itself is not modified (per AGENTS.md).

Acceptance criteria

• consumers = 1 → sequential behaviour unchanged (existing queue tests pass).
• consumers = 3 → three turns on different interface_paths run concurrently (mock engine with sleep; wall-time ≪ sum).
• Trainer cortex switch with >1 worker does not corrupt the global plugin under concurrent turns.
• ruff format / ruff check --fix / scoped ty check clean on touched files.

Scope / out of scope

• Out of scope: modifying the Selenium engine (external container), touching the already-async HTTP adapters, runtime resizing of the pool (may be a follow-up).

Key references

• core/message_queue.py — _consumer_loop (~L1243), turn await (~L1901), background branch (~L1836), _start_consumer_task (~L2196), _supervisor_loop (~L2207).
• core/plugin_instance.py — _llm_chain_lock (~L91), _llm_chain_lease (~L119), trainer switch + restore (~L402-475).
• core/external_endpoints/bridges/cortex_bridge.py — generate_response (~L767), adapter.chat_completion (~L813).