Epic: Move staging to a dedicated Contabo VM (gateway + testnet Bee)

[crtahlin]

Epic: Move the staging environment to a dedicated VM on provider C (gateway + testnet Bee)

Today staging and production run on the same host and share the same Bee node. That coupling is the root of the propagation-debugging pain in #236: we could not tell whether staging upload failures were gateway bugs or node/connectivity problems, because staging experiments write to the same (mainnet) Bee that prod depends on. Testing Flow B (#225) there also means spending real xBZZ on every createBatch.

This epic stands up an isolated staging environment on its own VM on provider C — a Sepolia testnet Bee + the staging gateway on one box — so staging is fully decoupled from prod, free to test against (testnet funds), and inspectable (we control the host + logs).

A companion epic (#249) moves production to C after staging is proven.

Why this shape

• A fresh, self-contained VM cleanly decouples staging from the shared prod host/Bee with no dependency on the current setup.
• Light Bee, not full node: the gateway buys batches via the chain client and stamps are signed off-chain, so staging needs only upload/pushsync — a light node (no reserve disk, no BZZ stake) is sufficient and cheap.
• Testnet = Sepolia (chain id 11155111) — the gateway's GNOSIS_CHAIN_ID=11155111 path already exists; an existing funded Sepolia signer wallet means no new real spend to test Flow A + Flow B end-to-end.

Networking is the thing to get right (the #236 lesson)

The old node showed networkAvailability=Unavailable (outbound-dial failures, likely IPv6 egress). On the new VM we control egress, so every Bee bring-up must verify: p2p port (1634) reachable, working IPv4 and IPv6 egress, connectedPeers climbing, and networkAvailability=Available — observable now via the debug proxy + /health node section — before any upload test is trusted.

Sub-issues

Dependency-ordered. The VM + Bee gate everything; the retests gate sign-off.

• Provision + harden the staging VM (Docker, firewall, egress check) #241 — Provision + harden the staging VM (Docker, firewall, egress check)
• Deploy the Sepolia testnet Bee as a light node + verify network availability #242 — Deploy the Sepolia testnet Bee as a light node + verify network availability
• Deploy the staging gateway pointed at the local testnet Bee #243 — Deploy the staging gateway pointed at the local testnet Bee
• DNS + TLS for the staging subdomain #244 — DNS + TLS for the staging subdomain
• Repoint CI/CD: dev branch deploys to the staging VM #245 — Repoint CI/CD: dev branch deploys to the staging VM
• Document the staging install as a reproducible runbook #246 — Document the install as a reproducible runbook (for future installs)
• Re-test Flow A (pre-signed chunks) on the new staging #247 — Re-test Flow A (pre-signed chunks, Epic: Agent-controlled uploads — stamped-chunk forwarding with prepaid bandwidth credit (Flow A) #218) on the new staging
• Re-test Flow B (buy stamps for an external owner) on the new staging #248 — Re-test Flow B (buy stamps for an external owner, Epic: Flow B — gateway purchases postage batches owned by an external wallet (createBatch on Gnosis) #225) on the new staging

Acceptance criteria (epic)

• Staging runs on its own VM; no shared Bee with production.
• Staging Bee is a Sepolia light node with networkAvailability=Available and a healthy peer count.
• dev pushes auto-deploy to the staging gateway via CI/CD (old shared-host dev deploy retired).
• Staging gateway reachable over HTTPS on its own subdomain.
• The full install is documented as a runbook reproducible for future installs (incl. prod).
• Flow A (pre-signed chunks) verified end-to-end on staging: sign → POST /chunks → Bee accepts → retrievable.
• Flow B verified end-to-end on staging: gateway createBatch(owner=…) → owner signs a stamp → upload via Flow A → retrievable; spend is testnet only.

Constraints / notes

• Reuse the existing funded Sepolia signer wallet (keys held locally; never commit or paste keys/addresses into issues/PRs).
• No secrets, IPs, hostnames, or provider-account details in this public repo — keep them in the private store referenced by the runbook.
• Branch from dev; never push to dev/main directly. No customer/private data on the staging node.

[crtahlin]

Status: largely executed, with two deliberate divergences

A new VM on provider C is live and serving both environments. What was built differs from this epic in ways worth recording, because one of them has an ongoing cost.

Done

• Provision + harden the staging VM (Docker, firewall, egress check) #241 provision + harden — Ubuntu 24.04, Docker, ufw (22/80/443 + p2p only), 4 GB swap. Public IP directly on eth0, no NAT.
• Deploy the staging gateway pointed at the local testnet Bee #243 gateway on the same box as its Bee, reaching it over the compose network. The Bee API is never published to the host — it has no authentication.
• DNS + TLS for the staging subdomain #244 DNS + TLS — both hostnames moved, per-host Let's Encrypt certificates via Caddy.
• Repoint CI/CD: dev branch deploys to the staging VM #245 CI/CD repointed — DEPLOY_RUNNERS now targets the new host only; the previous host's provenance containers are stopped.
• Document the staging install as a reproducible runbook #246 runbook — partially: deploy/README.md and monitoring/alerting/README.md cover the proxy and alerting. The Bee/VM install itself is not yet written up.
• Epic: Move production to provider C (after staging is proven) #249 production moved too — ahead of this plan rather than after it.

Divergence 1: mainnet Bee, not Sepolia testnet

This epic specifies a Sepolia testnet Bee so staging costs nothing to exercise. What was deployed is a second mainnet Bee — separate node, separate wallet, but real xBZZ.

The cost is not hypothetical. Staging burned roughly 8.9 BZZ in 13 hours through a stamp-pool defect (#274), which on a testnet node would have been free. Staging is currently at 0.80 BZZ and alerting.

The propagation isolation this epic wanted is achieved — staging and production now have separate Bee nodes, so staging experiments cannot affect production's node. The funding isolation is not.

Divergence 2: one VM, both environments

Staging and production share the new host rather than staging having its own. Cheaper, and they are properly separated at the Bee level, but a host-level failure still takes both.

Consequences for the remaining issues

• Deploy the Sepolia testnet Bee as a light node + verify network availability #242 is not done as written. Either close it as superseded, or keep it as the path to a genuinely free staging environment.
• Re-test Flow A (pre-signed chunks) on the new staging #247 / Re-test Flow B (buy stamps for an external owner) on the new staging #248 (re-test Flow A and Flow B on the new staging) are still open, and Flow B in particular means real createBatch spend on mainnet — which is precisely what the testnet plan was meant to avoid.

[crtahlin]

Reconciliation — the plan and what was built have diverged

Recording this rather than leaving six issues describing a system nobody is going to build.

The plan: a dedicated Contabo VM for staging, running a Sepolia testnet Bee.

What exists: one box running both environments — production and staging gateways, plus two mainnet Bee nodes with separate keystores, data volumes, p2p ports and Gnosis RPC endpoints. That was a deliberate cost decision, and mainnet was chosen so staging behaves like production.

issue	status
#241 provision + harden the VM	done — one box, not a staging-only one
#242 Sepolia testnet Bee	not done, deliberately — mainnet instead
#243 staging gateway → local Bee	partly — local, but mainnet
#244 DNS + TLS	done
#245 CI/CD repoint	done
#246 reproducible runbook	not done
#247 re-test Flow A	not done
#248 re-test Flow B	not done

What the change bought: staging that exercises real chain economics. The issues it surfaced this month — shared-RPC rate limiting (#296), the postage minimum-validity floor (#258), batch status contradicting its own TTL (#252) — all depend on that and would not have appeared against a testnet.

What it costs: staging spends real BZZ. It is at 0.0139 BZZ with an empty pool right now and cannot exercise any stamp flow, which weakens verification for everything tested there.

Open decision, on #242: accept mainnet staging and fund it as an operating cost, or add a testnet node alongside so cheap flows run free.

#246 and #247/#248 remain genuinely outstanding, and #246 is worth more now than when it was written — the install is no longer the one it was going to describe.