Skip to content

[FEATURE]: Chat completions run on the runtime #782

Description

@justintime4tea

Summary

On nightly, POST /v1/chat/completions prepares an agent, begins a run, takes the run's event receiver with AgentRun::take_agent_events(), reads content from the StreamItem stream, projects the rest through run_event_sse, and on disconnect, timeout, or shutdown cancels through the run's token — all inside the request's task. It is the reference implementation of a run, and it is why a run cannot outlive a request.

Move it onto the runtime. The handler attaches to the session as a claiming observer with presence, starts a run, projects the subscription to the wire it serves today, and detaches when the client goes. Cancel-on-unclaimed (#781) is today's behavior exactly.

Goals

  • The handler attaches to the session first — claiming, present, cursor Live ([FEATURE]: Observers declare whether they collect, claim, or attend #781) — then describes the run — agent, the request's forwarded headers and client tools, the messages, the chat liveness default — and the runtime prepares and starts it ([FEATURE]: A runtime that owns sessions and their runs #780). Because the claim is on the session, the handler is the claimant from the run's first event and Started is the first thing it streams. It never holds a PreparedAgent or an AgentRun.
  • The handler is three things in a row: the Subscription, project_sse, and Body::from_stream. No chunk channel between the run and the response body, no result oneshot for the collected path, no channel end anywhere in the handler but the stream inside the subscription.
  • One projection over SessionEvent. SessionEventPayload::Agent goes through run_event_sse as it stands on nightly; content comes from the journal's TextDelta / Reasoning / Completed rather than from StreamItem, which makes this handler the first consumer of the copy tee_content makes; a terminal lifecycle event ends the stream the way the item stream ending does now. If [FEATURE]: SSE Producer from Agent Events #576 has landed, its producer is that projection.
  • aura.session_info and aura.mcp_status, which the handler emits before the agent produces anything, come from the run's Started lifecycle event and the Agent(McpStatus) the run's writer task appends right after it from the prepared agent's server status ([FEATURE]: A runtime that owns sessions and their runs #780), so a late observer sees them too.
  • Disconnect, timeout, and shutdown become a detach with a reason. The runtime's cancel-on-unclaimed does what the handler's disconnect arms do now, from one place.
  • A Gap on the handler's subscription is an error, never a skip. Today a slow client back-pressures the LLM stream; under the journal the handler is one subscriber, and lagging would hand the user a hole in the answer. The handler ends the stream with an error frame — the same outcome a2a-server gives a subscriber that "fell behind active execution" — and the broadcast capacity is sized so a live client does not lag short of pathology.
  • The non-streaming path starts a run the same way and drains it.
  • Wire output unchanged, demonstrably: the existing integration suites pass untouched, and a parity test drives one scripted run through the pre-runtime handler and the runtime-backed one and compares frames byte for byte. The differential test [FEATURE]: Agent event schema and broker adapter #618 added did not survive PR Deliver a run's events through its handle and retire the request registries #738, so this is new, and it avoids the shared state that made the old one flaky ([BUG]: agent_event_differential::a_full_tool_turn_matches is flaky under parallel test threads #733).
  • ActiveRequestTracker retired; the runtime's drain ([FEATURE]: A runtime that owns sessions and their runs #780) is what shutdown waits on. Whatever still needs the tracker is a gap in the runtime, not a reason to keep it.
  • The orchestrator's StreamItem channel (orchestration::factory) is left without a consumer, since content now comes from the journal. Removing it, and ToolCallObserver with it, is [FEATURE]: Remove the request-scoped broker path #627's; this issue does not touch orchestration.

Data structures

Proposed. The projection is a function; the only new type is what it returns.

/// Projects one session event onto the OpenAI-plus-`aura.*` wire. `None` for
/// events the wire has no frame for (an observer attaching, a liveness decision,
/// anything belonging to a different run of the session).
pub fn project_sse(
    event: &SessionEvent,
    run: &RunId,                        // the run this response serves; other runs' events are None
    correlation: &CorrelationContext,   // session id, trace id — applied here, as today
    custom_events: bool,                // AURA_CUSTOM_EVENTS; approvals ignore it
) -> Option<SseFrame>;

pub enum SseFrame {
    Chunk(ChatCompletionChunk),         // from Agent(TextDelta | Reasoning | Completed)
    Custom(AuraStreamEvent),            // from Agent(ToolStart | ToolProgress | ...) via run_event_sse
    Done,                               // from Lifecycle(Finished | Cancelled | Parked)
    Error(StreamError),                 // from Lifecycle(Failed), and from JournalItem::Gap
}

Additional Context

Session handling is unchanged: chat_session_id from the request or generated, returned in X-Chat-Session-Id. It is the session the handler attaches to and the key of the stream the run is written to (#779), which is what makes a later attach by session or by run (#785) possible.

Searched Issues

  • No similar issues found

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions