Wisp v0.5.1 SIGSEGV ~2.5h into runtime behind StartOS proxy (inbound path)

[wksantiago]

Summary

Wisp v0.5.1 segfaults (signal 11 / SIGSEGV) roughly 2.5 hours into normal operation behind the StartOS reverse proxy. StartOS catches the dead child, fails the primary health check, and auto-restarts the daemon (recovery works), but the relay drops all live connections each time.

Observed

00:03:03  info: Wisp v0.5.1 starting / Listening on 0.0.0.0:7777 / Storage: /data/wisp (sync=meta)
04:32:18  stop -> Shutdown complete
04:32:19  /backup/create  (completes 04:34:26)
04:34:26  start -> Wisp v0.5.1 starting (clean)
07:02:12  Unknown Error: signal: 11 (SIGSEGV)
          startos::bins::container_cli ... src/service/effects/subcontainer/sync.rs:794
          Error: wisp exited with code 1
          Health Check primary failed: primary daemon crashed
07:02:12  (auto-restart) info: Wisp v0.5.1 starting ...

The sync.rs:794 location is the StartOS supervisor reporting the child's exit signal, not the fault site. The fault is inside the wisp binary.

Triage

• Not the spider. configure-spider/getInput confirms enabled: false, so the known upstream spider read-loop crashes ([BUG] wisp core dumped on initial sync wisp#104, #100) are ruled out — that code path does not run.
• Not the XFF/trust_proxy code. Reviewed extractClientIp / normalizeIp / bucketKey in wisp v0.5.1: bounds-checked slice ops and bufPrint (the IPv6 /64 key fits its [19]u8 exactly), and the WS path @memcpys the IP out before the request buffer is freed. (Also moot — current master seeds max_connections_per_ip: 100000 and does not set trust_proxy.)
• Not OOM. An OOM kill is signal 9; this is signal 11 — a genuine memory-safety fault.
• Leading suspect: the inbound httpz websocket read path — the one area v0.5.1 churned (httpz repin, Re-pin httpz to the simplified websocket lost-read fix wisp#98 "websocket lost-read/reorder fix"). Fits the signature: clean start, segfault hours into live client traffic.

Why there is no stack trace

The package builds wisp with -Doptimize=ReleaseFast (Dockerfile:41), which strips safety checks and the panic trace, so StartOS can only surface a bare signal 11. We cannot pin the exact fault line from these logs.

Proposed next step

Build wisp ReleaseSafe so the next fault emits a Zig panic with file:line (as in privkeyio/wisp#104) instead of an opaque SIGSEGV; then file an upstream wisp bug with the trace. Optionally enable core dumps for a backtrace.

Notes

• Package pins v0.5.0 (Dockerfile:29) but the crashing instance runs v0.5.1 — reconcile.
• Reproducible ~2.5h into steady inbound traffic, spider disabled.

[kwsantiago]

Resolved in wisp v0.5.4 (packaged here as 0.5.4:0).

(Note: this issue was auto-closed earlier by the ReleaseSafe build PR #9, which only added diagnosable traces. The actual fix came after that build gave us a usable stack trace.)

Root cause: a use-after-free in http.zig's epoll worker, not wisp code. In one event batch the worker could process a .signal (which frees a connection via disown -> http_conn_pool.release + conn_mem_pool.destroy) and then a .recv for that same connection, dereferencing the freed HTTPConn in getState() -> SIGSEGV (worker.zig:1679, self == null, 0x2f8 = _state's offset). It surfaced ~2.5-3.3h into sustained inbound traffic, regardless of spider state.

Fix: defer processSignal until the event batch is fully drained, so every .recv runs while its connection is still alive (it sees the .handover state and skips it via the existing guard); the free happens safely afterward. Carried in the pinned http.zig fork; upstream PR karlseguin/http.zig#214.

Validation: the ReleaseSafe build ran 13h+ under real load with no crash, vs crashing every ~3h before.