Symptom
When the inference backend faults, mlxk serve stays up, keeps accepting requests, and fails every
one of them. Observed during a long unattended local run: a Metal command-buffer timeout ended the
request in flight —
mlx-vlm generate() failed: [METAL] Command buffer execution failed:
Caused GPU Timeout Error (00000002:kIOGPUCommandBuffer...)
— and roughly 70 consecutive requests afterwards failed identically until the run was stopped. The
process never recovered and never exited. It held its port and, being a constant, /health would
have kept answering healthy throughout (see #64).
Impact
This is the least useful failure mode for a supervised deployment. The documented countermeasure —
run the server under systemd or another supervisor — only takes effect if the process actually
exits. A process that is alive but permanently unable to serve is invisible to that mechanism, and
each request it accepts is a request that fails.
On the wire the failures surface as 500 internal_error with retryable: false, which is the
same envelope a deterministic, permanent backend error produces. A client cannot tell the two apart
and cannot decide whether retrying, failing over, or waiting is correct.
Cause
Not established.
Suspected but unverified: MLX state corruption when a primitive throws during eval
(ml-explore/mlx#3675, merged upstream, not in any release at the time of writing) — an exception out
of eval_gpu skips the stream epilogue, leaving half-finished streams and uncommitted command
buffers. That would explain the absence of recovery. It does not explain slowdowns observed earlier
in the same run, which produced bit-identical output and resolved on their own.
Also unverified: whether a model reload inside the same process recovers, or whether the queue is
poisoned process-wide. This decides whether the right reaction is recovery or an honest exit.
Repro (induced, not yet run)
Drive the prefill command buffer past the GPU watchdog — a very large image, or --max-edge raised
far beyond normal — then issue an ordinary request against the same process and observe whether it
succeeds.
Note: this deliberately triggers a GPU fault; the display may stutter while it happens.
Symptom
When the inference backend faults,
mlxk servestays up, keeps accepting requests, and fails everyone of them. Observed during a long unattended local run: a Metal command-buffer timeout ended the
request in flight —
— and roughly 70 consecutive requests afterwards failed identically until the run was stopped. The
process never recovered and never exited. It held its port and, being a constant,
/healthwouldhave kept answering
healthythroughout (see #64).Impact
This is the least useful failure mode for a supervised deployment. The documented countermeasure —
run the server under systemd or another supervisor — only takes effect if the process actually
exits. A process that is alive but permanently unable to serve is invisible to that mechanism, and
each request it accepts is a request that fails.
On the wire the failures surface as 500
internal_errorwithretryable: false, which is thesame envelope a deterministic, permanent backend error produces. A client cannot tell the two apart
and cannot decide whether retrying, failing over, or waiting is correct.
Cause
Not established.
Suspected but unverified: MLX state corruption when a primitive throws during
eval(ml-explore/mlx#3675, merged upstream, not in any release at the time of writing) — an exception out
of
eval_gpuskips the stream epilogue, leaving half-finished streams and uncommitted commandbuffers. That would explain the absence of recovery. It does not explain slowdowns observed earlier
in the same run, which produced bit-identical output and resolved on their own.
Also unverified: whether a model reload inside the same process recovers, or whether the queue is
poisoned process-wide. This decides whether the right reaction is recovery or an honest exit.
Repro (induced, not yet run)
Drive the prefill command buffer past the GPU watchdog — a very large image, or
--max-edgeraisedfar beyond normal — then issue an ordinary request against the same process and observe whether it
succeeds.
Note: this deliberately triggers a GPU fault; the display may stutter while it happens.