docs(ai): record that MLX context overflow is undetectable

[albertoelopez]

Summary

mlx_lm.server cannot have a context-overflow pattern, and this documents why so nobody tries to add one.

Given a prompt beyond the model's context it does not reject, error, or truncate. It prefills the entire prompt past the model's own max_position_embeddings until the GPU runs out of memory, then the server process aborts mid-request:

Prompt processing progress: 38912/50013
Prompt processing progress: 40960/50013
"POST /v1/chat/completions HTTP/1.1" 200
libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed: Insufficient Memory
(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

The client observes a dropped connection, so none of the three cases in isContextOverflow apply: there is no errorMessage to match, no usage to compare against the context window, and no length stop.

Changes

• Document the behaviour in overflow.ts alongside the other backends, in both the provider list and the "Unreliable detection" section.
• Add regression tests asserting that transport failures (fetch failed, terminated, socket hang up, ECONNRESET, ECONNREFUSED) stay unclassified.
• Changelog entry under [Unreleased].

The tests are the point. A dropped socket is not evidence of overflow — it is equally consistent with a server crash or a network fault — so matching one to "support MLX" would misclassify every such failure. The tests make that regression fail loudly.

No behaviour changes; isContextOverflow is untouched.

Reproduction

Measured against mlx_lm.server 0.31.3 serving mlx-community/Qwen3-14B-4bit (max_position_embeddings 40960) on an M4 / 24 GB, sending a ~50k-token prompt. The crash is memory-bound, so the exact threshold varies with host memory, but the failure mode does not: MLX never reports overflow.

The practical mitigation is configuration rather than detection — declare a contextWindow the host can actually hold so compaction runs before the limit is reached. Note that a KeepAlive launchd agent restarts the server after such a crash, which makes it easy to miss.

Testing

• packages/ai: npx tsx ../../node_modules/vitest/dist/cli.js --run test/overflow.test.ts — 14 passed
• npm run check from the repo root — clean

Note

Document that MLX context overflow is undetectable in isContextOverflow

• Documents in overflow.ts that mlx_lm.server aborts the process on context overflow rather than returning an error, making overflow undetectable via response inspection.
• Adds tests in overflow.test.ts asserting that transport-level errors (fetch failed, socket hang up, ECONNRESET, etc.) are not classified as context overflow.
• Warns explicitly in JSDoc not to treat dropped connections as overflow and suggests configuration-based mitigation for MLX users.

^{Macroscope summarized cea8555.}