fix(sell): add obol sell resume so offers recover after a host reboot

[bussyjd]

Problem

After a host reboot, the public seller catalog stays empty even though the stack looks healthy: storefront serves 200, controller/verifier/catalog pods are all Running, but /api/services.json returns [].

Root cause (diagnosed live on the rc14 prod seller after a reboot):

1. Docker's restart policy resurrects the k3d node containers at boot, so the cluster comes back without anyone running obol stack up.
2. The sell-resume path (resumeSellOffers → re-apply ServiceOffer → startDetachedInferenceGateway) is only invoked from the stack up action (cmd/obol/main.go), so it never fires.
3. The host-side x402 gateway started by obol sell inference died with the host. The ServiceOffer survives in etcd but sits at:

   UpstreamHealthy=False: Get "http://<name>.llm.svc.cluster.local:8402/health": connect: connection refused
   Ready=False
   

   → excluded from the catalog → [] → buyers see nothing.

Fix

• obol sell resume — exposes the existing resume path as a first-class subcommand so a reboot no longer requires a full stack up to recover. Idempotent: gateways with a live PID on disk are skipped; the kubectl applies re-assert existing objects.
• obol sell resume --install-boot-unit (Linux/systemd, opt-in) — writes + enables a systemd user unit (obol-sell-resume.service, WantedBy=default.target, pre-start sleep for the API server) so resume runs automatically at boot. Prints the loginctl enable-linger hint required for boot-without-login. Non-Linux returns actionable guidance instead.
• Refreshes the stale resumeSellOffers doc comment (it still claimed gateways are not restarted; startDetachedInferenceGateway has done so since the resume feature shipped).

No behavior change to obol stack up — it keeps calling the same resume path.

Tests

• TestSellResumeCommand_Registered — registration + --install-boot-unit guard (mirrors the existing TestStackUpAction_CallsResumeSellOffers source-guard pattern).
• TestRenderResumeBootUnit — pins ExecStart/Environment/WantedBy/pre-start sleep of the emitted unit.
• TestInstallResumeBootUnit_PlatformBehavior — non-Linux refuses with guidance; Linux writes the unit under $HOME even when systemctl is unavailable (enable steps are best-effort warnings).

go build ./... and go test ./cmd/obol -count=1 green. Branched off latest main (tree-identical to v0.10.0-rc14).

Follow-up (out of scope)

The long-term shape noted in the existing code comment stands: package the inference gateway as an in-cluster Deployment so the cluster supervises it like Traefik/LiteLLM and the host-side PID plumbing disappears.

Coverage: all ServiceOffer types (review follow-up)

Per review feedback, c7ca58f generalizes resume beyond inference+http: the sell-http/ store becomes the persisted-ServiceOffer ledger (dir name kept for already-shipped files), and every offer surface persists into it — sell agent (both creation sites), the agent-backed demo (offer only; replaying an Agent CR would mint a fresh wallet), and the legacy demo as a v1 List bundle (namespace + backend + offer) so resume restores a working demo. sell mcp has no ServiceOffer and is documented as not resumed. Lifecycle holes closed: agent delete drops the agent's ledger entries; sell update refreshes the ledger from the live CR so resume can't revert a payTo/price change. resumeSellOffers simplifies to one guard + two phases with a pure, tested loader.

[bussyjd]

Live reboot validation on the seller box found two bugs in the resume path as originally pushed — both fixed in 29426c0:

1. Binary skew — startDetachedInferenceGateway preferred the installed BinDir/obol (rc11) over the binary running sell resume. The arg-builder emits the current flag surface (--description, renamed in rc12), so the relaunched gateway died with flag provided but not defined: -description. Fix: spawn the running executable; also emit --register-description (parsed by every released CLI) for the BinDir fallback.

2. Validation drift — with a current binary doing the relaunch, the slash-in-model rule (newer than the persisted descriptor) rejected --model AEON-7/Qwen3.6-…, so pre-rule offers could never resume at all. Fix: spawned gateway carries OBOL_SELL_RESUME_REPLAY=1; the rule downgrades to a warning for replays, still hard-fails new offers.

Repro sequence: reboot host → cluster auto-recovers via Docker restart policy, catalog goes [], route 404s → sell resume re-applied manifests but the gateway died instantly both ways above. Re-running the full unattended reboot test with the fixed binary + boot unit next.

[OisinKyne]

will we need to make this more general than only for selling? some part of me wonders should this be like obol stack resume and/or we just have obol stack up or a version of it set to trigger for a reboot? like if you're away from your node your agent may not be able to use this to fix itself

[bussyjd]

Live unattended reboot test: PASSED at cc7ef06.

Timeline (zero intervention after shutdown -r): SSH back in 38s → boot unit fires (sleep 30 → /readyz wait passes immediately) → resume re-applies manifests + relaunches gateway → offer Ready=True at t+68s. Verified from the public internet: catalog lists the offer, unpaid probe returns 402 with x402 requirements, ERC-8004 discovery 200. Gateway is running inside the unit's cgroup with the unit active (exited).

Three additional fixes were required beyond the original two commits, all found by this live test:

Commit	Bug	Fix
29426c0	Relaunch spawned the installed (older) obol → flag provided but not defined: -description; new binary rejected the persisted slashed model name	Spawn the running executable; emit --register-description (parsed by every released CLI); OBOL_SELL_RESUME_REPLAY=1 downgrades the slash rule to a warning for replays
080a346	Boot unit pinned BinDir/obol (rc11 has no sell resume); boot race — kubectl applies run before the cluster API is up, warn-and-continue, nothing retries	Pin the installing binary into ExecStart; sell resume waits up to 3min for /readyz
cc7ef06	systemd killed the "detached" gateway when the oneshot unit finished (setsid doesn't escape the cgroup) — unit reported success, gateway dead, log empty	RemainAfterExit=yes keeps the cgroup (and gateway) alive; systemctl --user stop becomes a deliberate gateway kill

Each has a pinned regression test (TestResumeGatewayBinaryPrefersRunningExecutable, TestResumeGatewayEnviron, TestValidateSellInferenceModelName, TestWaitForClusterAPI, TestRenderResumeBootUnit). go build ./... + go test ./cmd/obol -count=1 green.

[bussyjd]

will we need to make this more general than only for selling? … should this be like obol stack resume?

Generalized in c7ca58f — sell resume now covers every ServiceOffer type, not just inference:

Surface	Persisted	Resume behavior
sell inference	own store (inference/)	re-apply cluster artifacts + relaunch detached host gateway
sell http	manifest ledger	re-apply
sell agent	manifest ledger (both creation sites)	re-apply; offer waits on its Agent CR with a clear controller condition if the agent is gone
sell demo (legacy)	ledger, as a v1 List bundle (ns + backend Deployment + Service + offer)	re-apply → working demo, not an offer with a dead upstream
sell demo quant (agent-backed)	ledger, offer only	deliberate: replaying an Agent CR mints a fresh wallet and orphans funds on the old one
sell mcp	—	no ServiceOffer exists (foreground server); documented as not resumed

Two lifecycle holes the coverage matrix surfaced, both fixed: agent delete now drops the agent's ledger entries (else resume replays ghost offers forever), and sell update refreshes the ledger from the live post-patch CR (else the next resume kubectl-applies the old payTo/price back — silently reverting an intentional payment change).

On stack resume vs sell resume: obol stack up already runs this exact resume path, so the "recover everything" verb exists; what a reboot uniquely kills is host-side sell processes, which is what this command (plus the --install-boot-unit systemd unit, for the away-from-node case you raised) targets. If we later grow more host-side state worth resuming, promoting this to stack resume is a rename, not a redesign.

[bussyjd]

Adversarial review of c7ca58f (12 agents, findings reproduced against a live cluster) confirmed three majors in the generalization — all fixed in 9b1c146:

1. agent delete swept ledger entries for offers still alive in etcd. Nothing actually deletes the agent's ServiceOffers (the finalizer leaves the namespace + offers) — and a survivor reconciles back to Ready, paying the deleted agent's wallet, if the name is ever reused. deleteCRDAgent now deletes the namespace's offers in-cluster, and the ledger sweep only runs when the cluster was reachable.
2. sell stop didn't refresh the ledger — an etcd-wiping stack down/up replayed the pre-drain manifest and resurrected a deliberately stopped offer fully live. The drain patch now refreshes like sell update.
3. Agent-offer replay failed after stack recreation (namespaces "agent-<name>" not found) — the documented "offer waits on the missing agent" never happened. Agent offers now persist as a v1 List bundling the agent namespace (canonical labels) with the offer; the Agent CR stays excluded so no fresh wallet is ever minted by a replay.

Also closed the inference half of delete⇒no-resume: sell delete tombstones the inference descriptor (DeletedAt, kept for list/status history, cleared by re-creating the offer) so resume can't relaunch a deleted offer's gateway. Full go test ./... green.