This document describes the server engine structure implemented in the current code. It is a reference, not a roadmap.
server_main
├─ builds BackendPlan
├─ creates ModelBackend
├─ transfers ModelBackend ownership to LuceEngine
└─ constructs HttpServer, which borrows LuceEngine
LuceEngine
├─ owns ModelBackend
├─ owns the serving thread
└─ selects one serving loop
├─ serial: HttpServer::worker_loop()
└─ concurrent: HttpServer::scheduler_loop(SeqEngine &)
HttpServer
├─ owns sockets and HTTP protocol state
├─ owns the current ServerJob queue
├─ prepares model-ready GenerateRequest values
└─ formats tokens and terminal results for clients
The backend outlives HttpServer because server_main constructs
LuceEngine before the server. HttpServer stops and joins the serving loop
before its transport and cache state is destroyed.
BackendPlan is the validated, normalized input to backend construction. Its
groups describe the effective model, placement, cache, speculation, runtime,
and architecture-specific values selected during startup.
The backend factory accepts a plan, projects its values into the selected
architecture config, and returns one owned ModelBackend. Architecture
configs own persistent strings; they do not retain pointers into the plan.
Files:
server/src/common/backend_plan.cppserver/src/common/backend_plan_internal.hserver/src/common/backend_factory.{h,cpp}
ModelBackend is the common model-resource interface implemented by each
architecture adapter. A concrete backend owns weights, caches, snapshots,
model-specific execution state, and any architecture-specific helpers.
Its complete-request generation operations remain the execution mechanism for
serial serving. A backend that supports continuous batching also exposes a
borrowed SeqEngine; that object is owned by the backend and remains valid for
the backend lifetime.
File: server/src/common/model_backend.h
LuceEngine is the runtime owner. It owns exactly one ModelBackend and at
most one serving thread.
Its current public lifecycle is:
explicit LuceEngine(std::unique_ptr<ModelBackend> backend);
ModelBackend & backend() noexcept;
bool start_serving(ServingLoops loops, bool allow_concurrent);
void stop_serving();start_serving() selects the concurrent loop only when both conditions hold:
- the caller permits concurrent local serving; and
- the backend exposes a
SeqEngine.
Otherwise it starts the serial loop. The selected callback is owned by the
worker thread. stop_serving() requests shutdown and joins that thread before
returning. Destruction also calls stop_serving() and then destroys the owned
backend, whose concrete destructor performs backend shutdown.
Files: server/src/engine/luce_engine.{h,cpp}
HttpServer owns the HTTP-facing state:
- listen and client sockets;
- request parsing and response formatting;
- SSE state and client-disconnect detection;
- tokenizer-dependent request preparation;
- prefix-cache policy and server status;
- the current intrusive
ServerJobqueue.
It borrows LuceEngine and, through it, a ModelBackend. The references are
valid for the complete HttpServer lifetime.
At startup the server supplies its established serial and concurrent loops to
LuceEngine. Upstream PFlash forwarding disables concurrent local serving,
so the serial worker remains selected for that configuration.
Files:
server/src/server/http_server.{h,cpp}server/src/server/scheduler.cpp
GenerateRequest is the model-ready input shared by backend execution paths.
It owns every token sequence that may be retained during generation:
- prompt tokens;
- speculative hint tokens;
- stall-detection sequences;
- thinking-budget close tokens.
GenerateResult owns the completed token vector, timings, typed failure, and
generation metadata. Neither type contains HTTP or JSON state.
File: server/src/common/generation_types.h
SeqEngine is the model-side capability used by concurrent serving. It owns
slot allocation, per-sequence model state, KV capacity, batched prefill and
decode execution, sampling, and retirement.
HttpServer::scheduler_loop() currently owns admission order, fair prefill
selection, cancellation, response construction, and non-blocking delivery.
All SeqEngine calls come from the single serving thread owned by
LuceEngine.
File: server/src/common/concurrency/seq_engine.h
CLI arguments
-> BackendPlan::build()
-> create_backend(plan)
-> LuceEngine(std::move(backend))
-> HttpServer(engine, tokenizer, config)
-> LuceEngine::start_serving(...)
Backend validation and normalization happen before construction. Reporting, tokenizer setup, and backend construction read the same effective plan values.
client thread
-> enqueue ServerJob
serving thread
-> worker_loop()
-> prepare prompt, cache state, GenerateRequest, and callbacks
-> ModelBackend::generate() or restore_and_generate()
-> stream or format GenerateResult
-> complete ServerJob
client thread
-> close request connection
Only the serving thread mutates model execution state.
client thread
-> enqueue ServerJob
serving thread
-> scheduler_loop(SeqEngine &)
-> admit requests into model slots
-> select bounded prefill work and all live decode rows
-> SeqEngine::step()
-> buffer and flush client output without blocking other slots
-> retire completed, failed, or cancelled slots
Admission order and slot identity remain stable until retirement. A slow or disconnected client cannot block model progress for other live slots.
HttpServer::shutdown()
-> set stopping flag and wake the request queue
-> LuceEngine::stop_serving()
-> invoke request_stop callback
-> join serving thread
-> close SSE clients
-> drain remaining ServerJob values
-> release server-owned cache and transport state
stop_serving() is idempotent and serializes stop/restart through the full
worker join. HttpServer also calls it when run() exits. LuceEngine
destroys the backend only after the serving thread has joined; the concrete
backend destructor owns its single shutdown call.
The ownership boundary is active: LuceEngine owns the backend and execution
thread, while HttpServer owns transport. Request coordination has not fully
crossed that boundary yet: ServingLoops and the ServerJob queue connect the
runtime owner to the existing HTTP worker and scheduler.
A transport-independent generation channel (bounded request queue plus consumer/producer endpoints for token batches, progress, and the terminal result) is the intended next step for request submission; it is not part of this change and lands with its HTTP caller.