Run an LLM service for your teams or customers.

Connect model providers, create plans with model access, pricing, and rate limits, and assign them to organizations. Apps use an OpenAI-compatible API. You manage access and usage through the Admin UI.

Self-hosted. MIT licensed. SSO and audit logs included.

Customer Plans

Define a Growth plan with two models, set their prices and limits, and assign it to five customer organizations. When the plan changes, review and activate a new version without configuring each customer separately.

Service tiers manage these plans. Assign organizations to follow the active version so they receive plan updates together.

Admin UI

Manage organizations, access, and usage in the browser.

One Endpoint For Your Apps

from openai import OpenAI

# Before: call OpenAI directly.
client = OpenAI(api_key="sk-...")

# After: send the same OpenAI-compatible requests through DeltaLLM.
client = OpenAI(
    base_url="http://localhost:4002/v1",
    api_key="sk-deltallm-key",
)

Your application keeps its OpenAI request format. DeltaLLM handles provider credentials, routing, policy checks, caching, failover, and spend tracking.

What It Handles

• Unified API - Use one OpenAI-compatible endpoint across OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Groq, and other providers.
• Scoped API keys - Issue virtual keys with model allowlists, rate limits, budgets, owners, and expiry.
• Customer plans - Assign versioned model packages to organizations through service tiers, with model access, rate limits, and pricing managed together.
• Routing and failover - Route by strategy, retry failed deployments, and separate provider credentials from application code.
• Batch API - Run embeddings and non-streaming chat completions asynchronously through OpenAI-compatible files and batches, even when upstream providers are synchronous.
• MCP gateway - Register external MCP servers and expose approved tools through controlled gateway flows.
• Guardrails - Detect PII and prompt injection before provider calls.
• Spend and usage - Attribute cost by key, team, organization, model, and provider.
• Admin UI - Manage models, credentials, route groups, teams, users, usage, audit logs, and settings in the browser.
• Operations - Export Prometheus metrics, request logs, audit events, cache behavior, and health checks.

Performance

Latest complete run: 11 October 2026. All four rates passed.

Rate	Duration	Successful / scheduled	p95 / p99	Result
50 RPS	600 s	30,000 / 30,000	22.86 / 26.15 ms	Passed
100 RPS	600 s	60,000 / 60,000	23.53 / 28.09 ms	Passed
200 RPS	600 s	120,000 / 120,000	28.16 / 41.67 ms	Passed
500 RPS	600 s	300,000 / 300,000	95.15 / 179.39 ms	Passed

Synthetic one-token provider on a twelve-CPU, 12 GiB VM. See the RPS results for setup and limits.

Quick Start With Docker

Use Docker Compose for the shortest local evaluation path. It starts DeltaLLM, PostgreSQL, and Redis.

git clone https://github.com/deltawi/deltallm.git
cd deltallm
cp config.example.yaml config.yaml

For a first local request, enable one-time model bootstrap in config.yaml:

general_settings:
  model_deployment_source: db_only
  model_deployment_bootstrap_from_config: true

Generate the required secrets:

python3 -c 'import secrets; print("DELTALLM_MASTER_KEY=sk-" + secrets.token_hex(20) + "A1")'
python3 -c 'import secrets; print("DELTALLM_SALT_KEY=" + secrets.token_hex(32))'

Create .env with the generated values and a provider key:

DELTALLM_MASTER_KEY=sk-your-generated-master-key
DELTALLM_SALT_KEY=your-generated-salt-key
OPENAI_API_KEY=sk-your-openai-key
PLATFORM_BOOTSTRAP_ADMIN_EMAIL=admin@example.com
PLATFORM_BOOTSTRAP_ADMIN_PASSWORD=ChangeMe123!

Start the stack:

docker compose --profile single up -d --build

Wait for the health endpoint to return {"status":"ok"}, then send a request. In this terminal, export the same master key you put in .env: Compose reads that file for its containers, but it does not export variables into your shell.

curl http://localhost:4002/health/liveliness

export DELTALLM_MASTER_KEY='paste-your-generated-master-key'

curl http://localhost:4002/v1/chat/completions \
  -H "Authorization: Bearer $DELTALLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Hello from DeltaLLM"}
    ]
  }'

Open the Admin UI at http://localhost:4002.

Full Docker setup, environment variables, the multi-instance evaluation profile, and Dockerfile usage are covered in the Docker guide. Compose is not the production high-availability reference.

Deployment Paths

Path	Use it for	Start here
Railway	Managed evaluation with hosted PostgreSQL and Redis	Railway deployment
Docker Compose	Local evaluation, demos, small teams, simple self-hosting	Docker guide
Kubernetes	Multi-instance production, ingress, autoscaling, and managed infrastructure	Kubernetes guide
Local development	Backend, UI, and contributor workflow from the repository	Installation guide

Documentation

• Getting started
• Gateway usage examples
• Configuration reference
• Model deployments
• Features
• Admin UI guide
• API reference
• Deployment guide
• Security model

Contributing

• Report issues
• Request features
• Report a vulnerability privately
• PRs are welcome. Start with the local installation guide.
• Documentation changes follow the documentation contribution guide.

If DeltaLLM solves a problem for you, consider starring the repository. Share your use case in Discussions so we can improve the workflows that matter to you.

License

See LICENSE.