Inference Fabric is a model-agnostic inference reuse and continuity layer that sits in front of an existing language-model server. It avoids whole redundant model executions when reuse is demonstrably safe, collapses identical work already in flight, preserves scoped project continuity, and leaves backend-native KV/prefix caching to the inference engine.
Research preview by Emerson Francis Dowell, founder of Superior MI.
In the Turn 17 isolated mixed-workload benchmark against Qwen3.6-35B-A3B-TQ3_4S, Fabric captured 24 of 24 deliberately safe reuse opportunities, reduced backend model executions from 100 to 76, reduced backend-observed work by 16.7%, and passed the correctness gate.
This was not a duplicate-only benchmark. The 100-request workload contained:
- 35 novel requests
- 20 exact-repeat requests
- 15 paraphrases
- 10 multi-turn conversation requests
- 5 continuity/privacy probes
- 15 duplicate-burst requests
See BENCHMARKS.md for methodology, caveats, raw results, and the duplicate-heavy stress tests.
Within the mixed workload, exact-repeat traffic measured 8,713.2 ms median direct vs 7.69 ms through Fabric, approximately 1,133x lower median latency for the category. The median exact cache hit itself was 6.96 ms. Cross-arm parity remained 100%.
A 15-request burst representing three unique computations required 15 direct backend executions but only three through Fabric. Twelve duplicate requests coalesced behind in-flight work and backend-observed work fell by 79.9%.
Fabric does not treat a related sentence as permission to return an old answer.
- deterministic exact reuse: ON
- concurrent single-flight: ON
- near/typo candidate discovery: ON
- direct semantic/paraphrase bypass: OFF
- answer-basis prompt injection: OFF by default, experimental
- backend-native prompt/KV cache: backend-owned
- private user/project scopes: isolated
In the mixed benchmark, paraphrase retrieval found related candidates but bypassed the model zero times. Novel and conversational traffic also produced zero direct-response reuse.
Prefix/KV caching and Inference Fabric solve different layers of repeated work.
A backend prefix cache can avoid recomputing a shared prompt prefix, but the model still performs the remaining prefill and generation. Fabric can avoid the entire model invocation when a completed computation is safely reusable, or collapse multiple identical simultaneous misses into one actual inference.
The intended stack is complementary:
Application
|
Inference Fabric
| whole-response reuse, single-flight, continuity
v
Model server
| native KV / prefix caching, batching, decoding
v
Model
The integration boundary is the server capability, not a hardcoded model name.
The current lab backend is a TQ3/llama.cpp-compatible server, but the core is designed so that:
- unknown OpenAI-shaped servers can use a conservative generic adapter;
- recognized backends can expose richer capabilities such as slots, native cache observations, or timing counters;
- exact reuse identity remains model/backend aware so a response from one model is not silently replayed as if another model produced it.
Requirements:
- Python 3.11+
- SQLite through Python
- an already-running compatible inference server
Install:
chmod +x install.sh
./install.shAuto-discover a local server:
~/Projects/InferenceFabric/fabric connect --autoOr point Fabric at one explicitly:
~/Projects/InferenceFabric/fabric connect http://127.0.0.1:8080Start the gateway:
~/Projects/InferenceFabric/fabric serveThen point OpenAI-shaped clients at:
http://127.0.0.1:9000/v1
instead of the model server directly.
Fabric exposes reuse state in HTTP response headers:
X-Inference-Fabric-Cache: MISS | HIT | COALESCED
X-Inference-Fabric-Request-Id: <current request>
X-Inference-Fabric-Source-Computation: <source computation on reuse>
X-Inference-Fabric-Source-Created: <original computation time on reuse>
Cached responses receive fresh transport IDs/timestamps, and stale backend timing fields are not replayed as if a new model execution occurred.
~/Projects/InferenceFabric/fabric bench mixed \
--mode quick \
--backend-cache isolated \
--workers 4 \
--burst-workers 8Then measure the combined stack with backend-native prompt caching:
~/Projects/InferenceFabric/fabric bench mixed \
--mode quick \
--backend-cache native \
--workers 4 \
--burst-workers 8See REPRODUCIBILITY.md.
The full August 2026 technical report includes architecture, methodology, raw findings, limitations, related systems, portability design, and the author/project context.
This is a research preview, not a claim that every workload becomes 24%, 80%, or 98% cheaper. Savings depend on reuse density and request semantics. The repository intentionally includes neutral and negative findings so results can be evaluated rather than marketed in isolation.
Current priorities:
- native-cache mixed benchmark
- randomized/interleaved cold-request A/B methodology
- multi-slot serving tests
- additional model families and sizes
- continuity quality/cost evaluation
- optional semantic retrieval with explicit false-reuse evaluation
Superior MI is a planned Michigan-rooted effort organized around three long-term branches: Superior MI Labs for research and software, Superior MI Compute for local inference and infrastructure, and Superior MI Foundation for public-interest technology, education, access, community work, and environmental stewardship.
Inference Fabric is a Superior MI Labs project.
This project is currently being developed independently without institutional funding. See SUPPORT.md for the founder's note and ways collaboration, equipment, compute, workspace, funding, business help, or paid technical work could materially help.
Contact: EmersonDowell@gmail.com
Release-candidate default: Apache License 2.0. See LICENSE.
See CITATION.cff.