sovereign-llm-quickstart

An air-gap-capable, on-premise LLM stack you can stand up in one docker-compose up and ship into a network with zero outbound connectivity. Four services. One exposed port. Hash-only audit log.

 ___________________________________________________________________
|                                                                   |
|   S O V E R E I G N   L L M   Q U I C K S T A R T                 |
|       built for the offline kingdom                               |
|                                                                   |
|              ~ * ~  air-gap ready  ~ * ~                          |
|                                                                   |
|       ~~~~~~  ~~~~~~  ~~~~~~                                      |
|        \ | /   \ | /   \ | /                                      |
|         \|/     \|/     \|/         <-- the audit jellyfish       |
|         (o)     (o)     (o)             watching every prompt     |
|         /|\     /|\     /|\                                       |
|        // \\   // \\   // \\                                      |
|___________________________________________________________________|

Described in paper/paper.pdf: Sovereign LLM Quickstart: An Air-Gap-Capable Reference Stack for On-Premise LLM Deployment with Hash-Only Audit Logging.

---

What's in the box

 ┌─ port 443 (HTTPS, basic-auth) ──┐
 |                                 |
 |     nginx ────────────────┬─────┼─► open-webui  (chat UI)
 |       │                   │     |
 |       ├─► /api/generate ──┼─► audit  ─►  ollama  (inference)
 |       └─► /audit ─────────┘     |
 └─────────────────────────────────┘

Four services on an internal Docker network — only Nginx is reachable from the host. Ollama, Open-WebUI, and the audit service have no host ports. The Docker network is the perimeter.

Service	Image	Job
ollama	ollama/ollama:0.16.2	Local inference, GGUF model loader
open-webui	ghcr.io/open-webui/open-webui:0.5.4	Chat front-end
audit	locally built	FastAPI proxy + SQLite hash-only audit log
nginx	nginx:1.27-alpine	TLS terminator + basic-auth gate

---

Quickstart

git clone https://github.com/thunderstornX/sovereign-llm-quickstart.git
cd sovereign-llm-quickstart

# Generate a self-signed TLS cert (4096-bit RSA, 365-day validity)
./nginx/gen_certs.sh

# Generate the Nginx basic-auth file
BASIC_AUTH_USER=admin PASS='please-rotate-this-immediately' ./scripts/gen_htpasswd.sh

# Bring the stack up
docker-compose up -d

# Pull a model into Ollama (default: llama3.2:1b, ~1.3GB)
./models/pull_models.sh
# Or override:    MODELS="llama3.2:1b mistral:7b" ./models/pull_models.sh

# Sanity check
curl -k https://localhost/health

The chat UI is at https://localhost/. Your browser will scream about the self-signed cert — that is the deal with self-signed.

---

The audit log

Every POST /api/generate call (whether from the WebUI or from your own client) is proxied through the audit service, which records:

Column	Type	What it is
timestamp_utc	TEXT	ISO-8601 UTC when the request arrived
user	TEXT	From X-Forwarded-User (Nginx basic-auth)
model	TEXT	Model the prompt was sent to
prompt_hash	TEXT	SHA-256 hex of the prompt (no plaintext stored!)
response_hash	TEXT	SHA-256 hex of the response
prompt_tokens	INT	From Ollama's prompt_eval_count
response_tokens	INT	From Ollama's eval_count
latency_ms	INT	Round-trip time we observed at the audit layer
upstream_status	INT	HTTP status from Ollama (or 0 if unreachable)

An attacker who later compromises the audit DB still cannot reconstruct prompt text — they only get hashes. If your compliance regime requires plaintext retention, run a separate encrypted-content store keyed by the same prompt_hash.

Query it:

curl -ku admin:secret \
    "https://localhost/audit?user=alice&since=2026-05-01&limit=50" | jq .

---

Air-gap deployment

The stack is designed to install on a network with no public-internet egress. The full procedure is in docs/air_gap_guide.md. The shape of it:

1. On a connected box: docker pull the four pinned images + docker save to a tar + git archive the repo + sha256sum everything.
2. Transfer the tar bundle (USB, one-way diode, whatever your policy says).
3. On the air-gapped box: verify SHAs, docker load, restore the model volume, run docker-compose up.
4. Run a tcpdump filter to verify zero outbound traffic. Empty pcap == success.

---

Tests

14 pytest cases covering the audit DB, the FastAPI endpoints, X-Forwarded-User attribution, upstream-error path, query filtering, and Pydantic input validation. Run them:

python -m pytest tests/ -v

---

Real benchmarks (this machine only — and we tell you which one)

Measured on an Intel Core i5-8250U @ 1.60GHz with 16GB RAM running qwen2.5:0.5b (the largest model that fits in available RAM on this profile, given a busy desktop session):

Metric	Value
Cold-start latency	3446 ms (model load + first inference)
Warm p50 latency	561 ms
Warm mean latency	558 ms
Warm max latency	754 ms
Warm mean tokens/sec	16.53
Warm max tokens/sec	26.53

The original spec mentioned three hardware profiles (workstation/server/Pi). We only measure machines we have. The script records the host profile in the CSV header so re-runs on different hardware produce comparable rows — fabricating numbers for hardware we cannot run is precisely the thing this portfolio refuses to do.

Re-run on your host:

ollama pull qwen2.5:0.5b
python -m benchmarks.benchmark --model qwen2.5:0.5b --repeats 2

---

ISO 27001 mapping

docs/iso27001_mapping.md maps stack components to 22 controls under A.5 (organisational) and A.8 (technological) of ISO/IEC 27001:2022 Annex A — and is honest about which controls the stack does not cover (OIDC SSO, rate limiting, content moderation, DLP) so reviewers can plan compensating controls.

---

Repository layout

.
├── audit/
│   ├── audit_service.py    # FastAPI proxy + hash-only logging
│   ├── banner.py           # ASCII jellyfish startup banner
│   ├── db.py               # SQLite WAL connection helper
│   ├── models.py           # Pydantic request / response models
│   ├── schema.sql          # audit_log table + indices
│   └── Dockerfile          # python:3.12-slim + 4 deps
├── nginx/
│   ├── nginx.conf          # TLS + basic-auth + 3 routes
│   └── gen_certs.sh        # self-signed cert generator
├── docker-compose.yml      # 4 services, internal network
├── docs/
│   ├── air_gap_guide.md    # docker save / docker load procedure
│   └── iso27001_mapping.md # Annex A control mapping
├── benchmarks/
│   ├── benchmark.py        # cold/warm latency + tok/s measurement
│   └── results.csv         # current measurement
├── tests/                  # 14 pytest cases
├── models/
│   └── pull_models.sh      # default: llama3.2:1b
├── scripts/
│   ├── gen_htpasswd.sh     # openssl apr1 (no apache2-utils needed)
│   └── render_terminal.py  # ANSI-aware terminal-to-PNG
└── paper/
    ├── paper.tex           # IEEE 4 pages
    ├── paper.pdf
    └── figures/

---

Citing this work

@software{bhutto2026sovereignllm,
  author    = {Bhutto, Ali Murtaza},
  title     = {sovereign-llm-quickstart},
  year      = {2026},
  doi       = {10.5281/zenodo.20480448},
  url       = {https://github.com/thunderstornX/sovereign-llm-quickstart},
  orcid     = {0009-0007-2787-943X}
}

The DOI above is the concept DOI — it always resolves to the latest release. Version 1.0.0 is archived at 10.5281/zenodo.20480449. Related research:

• Legal and Ethical Framework for OSINT Investigations

---

License

MIT © 2026 Ali Murtaza Bhutto

~ AMB · ORCID 0009-0007-2787-943X · v1.0 · 2026 ~