Conversation
int(d.get(key, 0)) raises on a present-but-null or empty value: the default
only applies when the key is ABSENT. Bee returns amount as null on every
/batches entry — verified on a live node, all 421 of them — and the merged
view only fills it in from /stamps, which is incomplete while the node is
starting. Production hit this on a deploy:
ERROR Error syncing stamps from Bee node:
invalid literal for int() with base 10: ''
Because the try wrapped the entire loop, one unreadable record discarded
every other known stamp. Before the replenish guard added earlier, that read
as an empty pool and bought a full reserve on every check interval, which
fits the observed purchase volume far better than one reserve per restart.
Adds coerce_int() in swarm_api and uses it at the three sites that parsed Bee
numeric fields directly, including get_all_stamps_processed() — where the
exception originated and propagated out to fail the caller's whole sync.
Parsing in the pool sync moves inside the loop, so an unreadable record is
skipped, logged, and deliberately KEPT in state for the next cycle rather
than dropped, since the field is often populated moments later.
Bee gates /topology and /status behind its startup, answering them with 503 "Node is syncing" until it has replayed the postage snapshot — minutes on a cold start — while /health and /addresses answer immediately. Both cases leave topo and status empty, and the summary reported the same warning for each: "could not query the Bee node". So a node that had just reported its version and overlay in the same response was described as unreachable, sending an operator to debug connectivity that was fine. This cost real diagnostic time during a node migration. The summary now checks whether anything answered — bee_status or overlay — and says the node is starting up when it did. The healthy flag is unchanged in both branches: a node that cannot report topology is not asserted healthy whichever reason applies, and network_availability stays None during startup so the gateway does not flip to degraded.
…utages Two defects on the stamp purchase path, both of which made a caller error look like an infrastructure failure. Bee enforces a floor of currentPrice * minimumValidityBlocks and requires the amount to be strictly greater. minimumValidityBlocks is 17280, exactly 24 * 720, so calculating an amount for the documented 24h minimum landed precisely on the floor and was always rejected — the minimum the API declares (ge=24) could never succeed. Bee also re-evaluates currentPrice when it executes the purchase, and the price moves (observed 41516 -> 77610 within an hour), so an amount that was sufficient when calculated could be short by the time it was validated. calculate_stamp_amount now carries a 5% margin for price drift and, when the caller supplies Bee's reported minimumValidityBlocks, lifts the amount above the floor Bee actually enforces rather than one inferred from local constants. Both the purchase and extend paths pass it through. Separately, any httpx error was reported as 502 "The Bee node may be unavailable". A 4xx from Bee means the request was wrong, not that the node is down, and the message it returns names the problem exactly. Bee's 4xx now surfaces with its own status and message; 5xx and connection failures still report 502, since those genuinely are the node. Existing tests that encoded the exact old formula are updated to the new contract rather than loosened, so a future change to the margin has to be deliberate.
docker-compose.yml defaulted SWARM_BEE_API_URL to a specific node. That was correct when exactly one host ran this stack and stopped being correct the moment a second one existed. It becomes actively harmful once that node is retired: a host without an override would start, report itself up, and fail every Swarm operation, because nothing checks that the endpoint is real. Both endpoints are now required. Compose refuses to start and names the variable it needs, which is a better failure than a gateway wired to a node that no longer answers. This also removes one of the private network addresses committed to a public repository — a value that exposes internal topology while being useless to anyone outside that network. Verified: with the variables set the rendering is unchanged; unset, compose fails with "required variable SWARM_BEE_API_URL_DEV is missing a value" and the suggested value. The deployment host already sets both in its host env file, so no deploy is affected.
POST /api/v1/chunks/credit priced the x402 payment from a different parse of ?mb than the handler credited from. The pricing dependency used int(), which raises on a float-formatted string and fell back to the 100 MB minimum. The endpoint used Pydantic's lax int coercion, which accepts "1000000.0" as 1000000. A caller passing ?mb=1000000.0 was therefore quoted for 100 MB and credited 1,000,000 MB — four orders of magnitude of bandwidth for the minimum price. Both paths now derive the value from parse_topup_mb(), which accepts only a plain integer. The endpoint takes the parameter as a string and parses it itself rather than letting Pydantic coerce, so no value can be acceptable to one path and not the other. Anything unparseable is refused with 422, which is what makes the pricing fallback safe: a request priced at the floor never completes. Booleans are rejected explicitly, since bool is an int subclass and ?mb=true would otherwise credit 1 MB. The existing below-minimum behaviour is unchanged: such a request is still refused with TOPUP_TOO_SMALL rather than floored.
POST /api/v1/stamps/for-owner priced the x402 payment by inspecting the raw
JSON body with isinstance checks, while the batch was built from the
Pydantic-validated model. The two disagree: isinstance("22", int) and
isinstance(20.0, int) are both False, so a string- or float-formatted depth
fell through to the depth-17 / 24h defaults while the endpoint created exactly
what was asked for.
depth=22 , dur=168 -> priced depth=22, 168h | built depth=22, 168h
depth='22', dur='168' -> priced depth=17, 24h | built depth=22, 168h
Batch cost scales as amount(duration) * 2^depth, so the string form was quoted
at a small fraction of the batch created, and the shortfall came out of the
gateway's own Gnosis wallet. The endpoint is off by default behind an owner
allow-list and capped by STAMP_FOR_OTHERS_MAX_BZZ, but there is no idempotency
key, so the call can be repeated.
The dependency now validates the body with StampForOwnerRequest and prices
from get_effective_depth() and duration_hours — the same values the endpoint
uses. A body the endpoint would reject prices at the smallest defaults rather
than guessing, and is refused there anyway.
This also removes the isinstance(depth, int) branch that would have treated a
boolean depth as depth 1, since bool is an int subclass. Pydantic rejects it,
so it was not exploitable, but it showed the branch was the wrong way to read
the value.
calculate_usable_status() collapsed six distinct causes into one boolean: missing, expired, expiring within the safety threshold, full, invalid depth, and unreadable data. Consumers had nothing else to render, and rendered it as "Expired". That produced a listing where a batch expiring at 12-49 was labelled Expired beside batches expiring at 12-48 labelled Usable — the displayed signal and the honest one pointing in opposite directions. An agent sorting by expiry and taking the longest-lived stamp selects the unusable one. The cause was almost certainly a full batch: capacity exhaustion makes a stamp unusable regardless of how long it lives, so a longer-lived batch legitimately fails while shorter-lived ones pass. Under a label saying "Expired" that reads as a contradiction. Adds get_unusable_reason(), returning a code and message, and surfaces both on the stamp response as unusableReason and unusableMessage. The caller's correct response differs per cause — a full batch will never become usable, while an expiring one can be topped up — so the cause has to reach the caller. calculate_usable_status() is kept as a thin wrapper so existing callers are unaffected, and "expiring soon" is now distinct from "expired" rather than both being reported as expiry.
gateway_wallet_bzz_balance and gateway_wallet_xdai_balance reported a number with no indication of which wallet it belonged to. An alert firing on them could not name the wallet to fund, so the address was written into the Grafana alert text by hand. That literal named one specific Bee node. It was therefore already wrong for the other environment, and became wrong for both when that node was replaced. The alert correctly reported a low balance and then pointed at a decommissioned wallet — worse than reporting nothing, because an operator following it sends real funds to a dead address. Observed today: both the development and main alerts quoted the same retired wallet while reporting correct, different balances. The address is already returned by the preflight balance checks; it simply never reached the metric. Both gauges now carry a `wallet` label, so alerts can template the correct address per environment. Each environment has exactly one wallet, so this adds no meaningful cardinality. Note this changes series identity for these two metrics, as adding any label does. Existing queries that do not filter on `wallet` continue to work.
Review of this branch found the retry behaviour it claimed did not work. _save_state() writes self._pool.keys(), so adding an unparseable batch ID to valid_ids only affected whether a save happened, never what was saved — the record was dropped from state on the next rewrite, exactly the outcome the change set out to prevent. _save_state() now takes extra_ids for batch IDs that are ours but not in the pool, and the sync passes the ones it could not parse. The test that was supposed to cover this passed against the broken code: it never triggered a rewrite, because the state file is only written when something was removed, so it asserted against the file it had written itself. It now includes a stamp that is genuinely gone from the node, which forces the rewrite and makes the assertion real.
…ence Survive unreadable fields when syncing the stamp pool
…-node Distinguish a starting-up Bee node from an unreachable one on /health
Review found the purchase path propagating Bee's 4xx while the extend path still reported every failure as 502 'The Bee node may be unavailable'. Both call calculate_stamp_amount and both talk to Bee, so a caller extending a stamp can hit the same minimum-validity refusal and would have been told the node was down. Adds the two matching tests. They also mock the stamp lookup the extend handler performs first, which the initial versions missed and which surfaced as a 500 rather than the assertion under test.
Clear Bee's minimum-validity floor and stop masking its refusals as outages
Review noted that the troubleshooting guide tells an operator to run docker-compose up --build locally to reproduce a deploy problem, and that this now fails without the two Bee endpoint variables. The failure names the missing variable, but the documented command should work as written.
Require an explicit Bee endpoint instead of defaulting to one node
Review caught this branch introducing a second BYTES_PER_MB while fixing a bug caused by two sources of truth for the same value. If one copy were ever changed to 1024*1024 and the other left at 10^6, pricing and crediting diverge again — the same defect under a different name. The endpoint now imports the constant from the credit module, and a test asserts both refer to the same object and that the pricing path does not hardcode the conversion. Also expands the OpenAPI description for the mb parameter, which is now typed as a string: it states that whole numbers are required, that values like '100.0' are rejected, and why it is parsed the way it is.
Parse the bandwidth top-up amount in one place, not two
Pricing quotes the batch the model describes, and the endpoint's depth and duration caps currently refuse an over-cap request. If either were changed to clamp instead, the endpoint would build something smaller than was quoted and the caller would overpay — the same divergence this branch removes, reintroduced one layer further down. Adds tests asserting the caps raise, that neither clamps, and that pricing deliberately does not apply the caps itself: quoting the requested batch is what keeps both paths describing the same thing.
Price the for-owner batch from the model the endpoint builds it from
Review of this branch found the reason never reached a caller. Two causes: get_all_stamps_processed() assembles its result dict field by field, and the new keys were set on the intermediate merged_stamp rather than listed there — so the model declared unusableReason and nothing ever populated it. A field that is always null is worse than no field, because it reads as "no reason available" rather than "not implemented". The reason was also only derived on one branch. When the node itself reported a stamp unusable, get_unusable_reason() was never called, so precisely the stamps a caller most needs explained carried no explanation. Resolving the merge with dev also required a deliberate choice: dev now has coerce_int(), which returns its default on unreadable input. Using it for batchTTL here would turn a TTL we cannot parse into 0 and report the batch as "expired" — collapsing the exact distinction this change exists to make. The raising form is kept, with a comment, so unreadable data is reported as unreadable. Adds tests asserting the reason survives serialisation and reaches the caller through get_all_stamps_processed, not merely that the model declares it.
Report why a stamp is unusable instead of only that it is
Review checked every alert for remaining hardcoded addresses. One is left: Low Base ETH names 0xc87688A4... which is currently correct, unlike the Bee wallet that this branch was written for. It is the same latent trap though — a literal in alert text that nothing keeps in step with the running configuration, correct only until the address changes. check_base_eth_balance() already returns the address, so labelling the gauge lets that alert stop hardcoding it as well and removes the last place this failure can recur.
Label wallet balance metrics with the address they describe
The minimum top-up was enforced; the maximum was not, so one request could credit an unbounded amount. The payment is priced before it is credited, so economics discourage an absurd request — but that guarantee rests on the pricing and crediting paths agreeing about the number, which is exactly what could not be assumed before they were made to share one parser. Adds BANDWIDTH_CREDIT_MAX_TOPUP_MB (default 1 TB, far above any legitimate top-up), refused above with TOPUP_TOO_LARGE mirroring TOPUP_TOO_SMALL, and wires it through config, .env.example and deploy.yml as the contributing guide requires. Two existing test settings helpers build their settings from MagicMock, whose attributes compare truthy — an unset ceiling made every top-up look too large. Both now set it explicitly, with a note, since the same trap catches any future numeric setting added to those helpers.
Bound a single bandwidth credit top-up
The seven alert rules existed only inside Grafana: no review, no history, and no way to distinguish a deliberate change from an accidental one. Two things that cost real time today came from that. A wallet address hardcoded in three alert descriptions went stale when the Bee node was replaced. The alerts kept reporting correct balances while pointing at a decommissioned wallet, so an operator following one would have sent funds to a dead address. Nothing flagged it because nothing was watching the text. A false negative sat in Stamp Pool Exhausted for an unknown period: it summed pool availability across hosts, so an exhausted pool on one host could be masked by another. It was found only by exporting the rules and reading them by hand. Exports the rule definitions to monitoring/alerting/alert-rules.json with the commands to re-export and re-apply. A non-empty git diff after an export now means someone changed a rule without recording it, which is the point. Contact points are deliberately excluded: they carry the Telegram bot token and chat ID. UIDs are excluded too, being per-instance.
Keep the Grafana alert rules in version control
The Caddy configuration existed only at /etc/caddy/Caddyfile on one host. It encodes decisions that are not recorded anywhere else: which hostname maps to which gateway port, that the containers bind to loopback so the proxy is the sole entry point, upstream timeouts sized for slow Swarm uploads, and that access logging goes to stdout because writing under /var/log/caddy fails under the packaged unit's sandboxing. Rebuilding the host meant reconstructing that from memory, and it was already clobbered once: a change made directly on the host was reverted by an upload from a stale local copy, and the site served self-signed certificates until it was noticed. Hostnames are read from the environment so the file carries no deployment-specific values, supplied by a systemd drop-in. Verified before and after applying it to the live host: `caddy adapt` renders byte-identical HTTP app configuration to the hand-written file it replaces, and both hostnames return 200 with valid Let's Encrypt certificates. The previous file is kept on the host as a timestamped backup.
…ntrol Keep the reverse proxy configuration in version control
docker compose up -d --force-recreate with no service argument recreates every service in the file, so a push to dev restarted production and a push to main restarted staging. That is not free: the stamp pool purchases a reserve on startup, and one such restart cost roughly 0.7 BZZ for nothing. The deploy now force-recreates only the gateway belonging to the branch being deployed, with --no-deps so depends_on does not drag the other environment in. A second plain `up -d` starts host-level services — monitoring, and the bundled Bee nodes when their profile is active — without --force-recreate, so anything already running is left alone. That second call also covers alloy, which declares depends_on the staging gateway and would otherwise never be started by a main-branch deploy. This addresses the restart half of the problem. The other half — both branches sharing one checkout directory, so whichever deployed last determines the compose file on disk — is unchanged and still tracked, because fixing it means moving the Bee and monitoring services out of the per-branch project and onto a shared network, which should not be done without a maintenance window.
Recreate only the service the deploying branch owns
By default docker compose derives the project name from the directory name, and the project name prefixes both container and volume names. The Bee volumes are swarm_connect_bee-data and swarm_connect_bee-dev-data purely because the checkout lives at /opt/swarm_connect. That makes wallet keys hostage to a filesystem path. Deploying from any other directory would look for volumes that do not exist, create empty ones, and start Bee with no keystore — a fresh wallet, with the funded ones orphaned and unreachable. swarm.key is 490 bytes and exists in exactly one place. Pinning COMPOSE_PROJECT_NAME makes the path irrelevant to identity. It changes nothing today, since the pinned value matches what the directory already produces, and it is what makes per-branch checkout directories safe to introduce later. Written after the host env file so nothing there can override it.
Pin the compose project name so volume identity survives a path change
The Bee nodes and the metrics agent were defined alongside the gateways, so both branches described them and whichever deployed last decided their configuration. A change merged to one branch was silently reverted by the next deploy of the other — that is how the Bee p2p port fix was undone twenty minutes after it merged, with nothing failing to indicate it. They are not branch-specific, so they now live in docker-compose.host.yml and are applied with an explicit -f. The gateways keep docker-compose.yml and stay per-branch. Both use the same COMPOSE_PROJECT_NAME, pinned in the previous change, so this is a file reorganisation rather than a migration: containers and volumes keep their names and the gateways still reach the nodes at http://bee:1633. Confirmed by rendering the new file — the volume resolves to name: swarm_connect_bee-data, the existing volume. Also drops alloy's depends_on the staging gateway, which now lives in another file. It was never load-bearing, since Alloy retries a target that is not yet up, and it had a real cost: a main-branch deploy would not start the metrics agent at all, because it hung off the other environment's container.
Define the host-level services once, not in both branches
Two defects that made an underfunded pool expensive to diagnose and costly to run. Bee's message was discarded. A failed purchase logged only httpx's status line, "Client error '400 Bad Request' for url ...", so the actual cause never appeared anywhere. Finding out that Bee was saying "out of funds" required calling it by hand. Bee's message is now extracted and appended to both the log line and the errors array that /api/v1/pool/check returns — the same treatment the API purchase path got earlier. Failures were retried every cycle. A purchase that fails is almost always persistent — out of funds, or an amount below Bee's minimum validity — but the pool tried again on every check, and attempted each stamp the reserve was short rather than stopping at the first failure. Each attempt holds a request handler for as long as Bee takes to refuse. A failed depth now records an exponential backoff, 60s doubling to an hour, and the loop stops at the first failure rather than working through the shortfall. Backoff is per depth, so a depth-20 failure does not block depth-17, and it is cleared on the next success. A skipped depth reports why in the errors array rather than being silently absent.
…koff Back off after a failed pool purchase, and say what Bee refused
Both nodes pointed at one free Gnosis RPC from a single BEE_RPC_ENDPOINT, so every chain operation from both — block-number polling, balance reads, chainstate, batch creation — shared one rate limit. That is not theoretical. A test run drove a few hundred stamp purchases on staging and the endpoint began returning 429; production's Bee log carries 30 occurrences of 'Too Many Requests'. When it bites, chainstate and wallet reads fail, x402 cannot price a request, and stamp purchases fail — while /health still reports a healthy node, because peer connectivity is unaffected and nothing reports on the chain backend. BEE_DEV_RPC_ENDPOINT lets the staging node use its own, falling back to the shared value when unset so a host that has not split them is unchanged. Verified both renderings: with only the shared variable both nodes get it; with both set each node gets its own. This reduces the blast radius. It does not make a free shared endpoint a suitable production dependency — see #296.
* Watch the Gnosis RPC connection Nothing did. When an endpoint started refusing calls, the first sign was a user getting "Failed to fetch wallet information from Swarm API" — the chain-reading endpoints degrade while everything else keeps working, so overall status stays green and nobody looks. Finding #296 meant reading Bee's logs by hand, per container, and only because a test suite happened to be running at the time. The 429s had been accumulating for a day. Bee already exports what is needed; it was simply never scraped. Alloy now collects bee_eth_backend_* from both nodes: total calls and errors, the block-number cache load errors that specifically break /chainstate, per-method call counters, and observed block time. Only those 22 series and `up` are kept — Bee exposes 874 per node and forwarding all of them would multiply Grafana Cloud ingest for no benefit. The API port stays unpublished, so this is reachable only on the compose network and adds no exposure. Four panels on gateway-overview: error rate, calls by method, cache load errors, and observed block time. Block time moves before the error rate does when an endpoint starts serving stale results. Two alert rules, with deliberately different noData handling: Gnosis RPC Errors >5% for 15m noData=OK unpaused Bee Node Not Scraped up<1 for 10m noData=Alerting PAUSED The first is OK on no data so it does not double-report: absent metrics are a scrape fault, not an error rate, and the second rule covers that. The second ships paused because it treats absence as failure, and until this config is deployed `up{instance=~"bee.*:1633"}` does not exist — provisioning it live would fire it before anything was wrong. Unpause after confirming the series arrives; the README spells out the order. 5% comes from measurement, not a guess. Production sat near 1.1% cumulative over 19h, and both nodes measured zero errors across several hundred calls in steady state, so a threshold at zero would be noise. What 5% deliberately tolerates is a restart: a node catching up produced 5,462 errors in its first forty minutes and none afterwards, which the 15m window absorbs. Config only — no application code changes. Alloy config validated with `alloy validate`; all six PromQL expressions parsed with promtool. Full suite: 1073 passed, 16 skipped. Implements #298. * Do not publish Bee's up under labels that mean "gateway" Review of the previous commit found two problems in it. The Bee scrapes forwarded `up` unchanged. The Gateway Down alert matches up{job=~"prometheus.scrape.*"}, and the new components satisfy that regex, so a stopped Bee node — or simply a host not running the `bee` compose profile — would have paged as a downed gateway, at critical severity, with triage steps for the wrong service. The Gateway Status panel would have shown four series instead of two, mapping Bee nodes to UP/DOWN as though they were gateways. `up` is now renamed to bee_up in the relabel chain before forwarding. The absence rule queries bee_up, so nothing keyed on `up` sees Bee at all. The dashboard JSON was also rewritten wholesale by a formatter: 1,300 lines of reformatting around 4 new panels, which buries the change and makes every future diff of this file noisy. Restored to the original compact style with the panels appended in matching form — 36 added lines, nothing else touched. alloy validate exit 0; bee_up parsed with promtool; both JSON files valid.
They have not been since 2026-08-25. The config is bind-mounted as a single FILE, which pins the inode, and `git pull` replaces the file rather than editing it in place — so the container kept reading the old inode. Compose could not help: the service definition never changed, so `up -d` saw nothing to do. Verified on the host. The file on disk was from today, the file the container had open was from 2026-08-25 12:38, and their inodes differed: host: 2026-08-26 09:25:43 inode=1600172 container: 2026-08-25 12:38:57 inode=1575124 The failure was silent, which is the worst part. Every deploy since then reported success and changed nothing about Alloy. The host label and the env() -> sys.env() migration were both dead on arrival, and the Bee scrape targets from #298 would have been too — the deploy went green while the monitoring it was supposed to enable did not exist. Alloy is now recreated on every deploy. It holds no state worth keeping and starts in about a second, so doing it unconditionally is cheap and removes the class of problem rather than the instance. Mounting the directory instead would fix the inode issue but not the reload: Alloy reads its config at startup. The Bee nodes are deliberately left alone — they are stateful, and restarting them costs peer connections and sync progress. Full suite: 1073 passed, 16 skipped. Workflow YAML parses.
* Apply the dashboard and alert rules from this repo Nothing did. The JSON under monitoring/ described what should be in Grafana; getting it there was a curl in a README, run by hand. So the two drifted with no way to tell them apart: a merged change to either file looked exactly like one that had been applied. That is why the panels and rules from #298 are not in Grafana. They were committed, validated and deployed, and no step existed that would have put them there. scripts/apply_grafana.py applies both. Matching is by rule title, so it is idempotent — running it twice updates rather than duplicating. Per-instance UIDs are read from the target and never committed, which is why the exported rules in this repo carry folderUID: null; the folder is resolved by title instead. Three things it deliberately does not do: - Delete rules present in Grafana but absent here. One may be a deliberate hand-made addition, and removing it silently during an unrelated run is worse than reporting it. - Create the alert folder. Guessing where alerts should live is how they end up somewhere nobody is watching. - Send X-Disable-Provenance. The rules already in the instance were created through the API, and a write that disagrees is refused with 409 alerting.provenanceMismatch. It waits for the stack to wake first. A sleeping Grafana Cloud stack answers 404 or 503 with a body that reads exactly like a wrong hostname, which has cost time before. The token must be a service-account token for the instance (glsa_ prefix). The GRAFANA_CLOUD_API_TOKEN on the gateway hosts is a different credential — a glc_ Cloud Access Policy token for Prometheus remote-write — and returns 401 here. Verified again today against the live instance. The script checks the prefix and refuses early rather than failing against the API. --dry-run needs no credentials: 36 panels, 9 rules, one of them paused. * Unpause the scrape-gap rule; record how the token is stored Both are now applied to the instance, which is the first time anything in monitoring/ has been applied by a committed, repeatable step rather than a hand-run curl. Bee Node Not Scraped shipped paused because bee_up did not exist yet and a rule that treats absence as failure would have fired on provisioning. bee_up is now present for both environments, so the documented precondition is met and it is unpaused. Confirmed live in Grafana Cloud after applying: bee_up development=1 main=1 bee_eth_backend_total_rpc_calls development=1240 main=1383 bee_eth_backend_total_rpc_errors development=0 main=0 up{job=~"prometheus.scrape.*"} only the two gateways That last line matters: the Bee nodes do not appear under `up`, so the rename to bee_up holds in real data and the Gateway Down alert cannot be triggered by a stopped Bee node. That was the regression caught in review of #298. Dashboard is at version 11 with panels 80-83. Nine rules live, all active.
The panel queried:
sum(rate({__name__=~"bee_eth_backend_calls_.*"}[5m])) by (__name__)
rate() strips __name__. With the name gone all 28 series carry an identical
label set, and Prometheus refuses the whole query:
vector cannot contain metrics with the same labelset
So the panel was not merely empty, it was broken — and it only showed itself
once there was real data to query. promtool validated the syntax and could not
have caught this: a semantically wrong query parses perfectly.
The metric name is now copied into a real `method` label before the rate is
taken, via label_replace over a subquery. Verified against the live datasource:
28 series in 139ms.
development eth_call 2.7407 req/s
development balance 0.5889 req/s
main eth_call 0.2870 req/s
main balance 0.1000 req/s
development filter_logs 0.0389 req/s
The other three panels from #298 were checked at the same time and do return
data: RPC error rate 0 on both environments, observed block time 5.0s on main
and 5.6s on development against Gnosis's ~5s target.
…money (#305) * Stop anyone who can resolve the hostname from spending the gateway's money POST /api/v1/pool/check calls check_and_replenish(), which buys postage batches with the gateway's own Gnosis funds. It had no authentication and no payment gate, and sits outside the x402-protected prefix, so an anonymous caller reached it directly and got a 200. Production now holds 18.56 BZZ. This is the same mechanism that took staging from 0.80 to 0.0139 BZZ. That was a test suite doing it by accident; the endpoint made it available to anyone. It is now gated by a signature from POOL_ADMIN_ADDRESSES — the mechanism already used for the Bee diagnostics proxy, extracted into app/services/signed_auth.py so both share one implementation. Two deliberate choices in that gate: A SEPARATE allow-list from DEBUG_ALLOWED_ADDRESSES. Reading a node's topology and spending its wallet are different privileges, and an operator trusted with the first is not automatically trusted with the second. A DIFFERENT message prefix. Diagnostics signs "swarm-connect-debug:<ts>", this signs "swarm-connect-pool-check:<ts>". Sharing the prefix would have made every debug signature a spending signature for as long as its timestamp stayed fresh. Two tests pin this in both directions. Empty allow-list answers 404 rather than 401, so an unconfigured gateway does not advertise that a privileged endpoint exists. That is also the default, which closes the exposure everywhere on deploy — including production, where DEBUG_ALLOWED_ADDRESSES is currently unset and this one will be too until it is configured deliberately. The second half of #292 is the blocking. The handler awaited the whole check; a purchase takes ~16s and the reserve is five, so a call could hold a connection for over a minute. It now returns 202 and schedules the work, pointing at GET /api/v1/pool/status for the result. check_and_replenish() also gains a single-flight guard. The timer alone could not overlap, but an operator can now trigger a run while a scheduled one is in flight, and two overlapping checks each see the same shortfall and each buy to cover it — twice the reserve, paid for twice. One test needed correcting after it failed: it asserted the handler returns quickly, measured by wall-clock through TestClient. TestClient drives the whole ASGI cycle and Starlette runs background tasks inside it, so the stopwatch was timing the background work too and would have failed even though the real client had been released. It now asserts the ORDER at the ASGI level — the response body is emitted before maintenance completes — which is the property that actually matters. Full suite: 1084 passed, 16 skipped. Closes #292. * Tighten the single-flight guard; drop the model the 202 made unused Review of the previous commit found two things in it. The guard used an asyncio.Lock with `if lock.locked(): skip` followed by `async with lock`. Those are two steps: a second caller can pass the locked() test while the first has not acquired yet, then WAIT on the acquire instead of skipping — running a redundant replenish afterwards and holding a background task for its duration. The money outcome was still safe, because the second run finds the reserve already met, but the skip it was supposed to perform did not happen. A plain boolean is both simpler and actually race-free: there is no await between testing it and setting it, and asyncio is single-threaded, so the two cannot interleave. Reset in a finally, so a raising check does not wedge the guard shut. ManualCheckResponse was left behind when the endpoint moved to 202. Nothing referenced it. A response model that no longer describes any response is the kind of thing someone later wires back up by mistake. Full suite: 1084 passed, 16 skipped.
tests/test_integration_gateway.py defaulted GATEWAY_URL to the production gateway and skipped only when that host was unreachable. Production is normally up, so `pytest tests/` — the gate CLAUDE.md requires before any PR — fired real traffic at it on every run, including every run made while fixing other things today. The known consequence (#233) is that it made the gate non-deterministic. The free tier allows three writes a minute, the built-in pacer does not fully absorb that when local timing shifts, and tests failed with 429 instead of the status they asserted — so the one check everything else depends on failed for reasons unrelated to the change under test. The consequence not in the issue is worse. The usable_stamp fixture PURCHASES A POSTAGE BATCH when it cannot find a usable local one: print("\nNo usable local stamp found, purchasing new stamp...") That is real BZZ, on production, from a routine local test run. It has been harmless only because production happens to hold usable stamps — five, as it turns out. An empty pool would have made the pre-PR gate spend money. Three switches, because these are three separate decisions: RUN_LIVE_TESTS=1 run the live modules at all GATEWAY_URL=... which gateway (now localhost, not production) ALLOW_LIVE_STAMP_PURCHASE=1 permit the fixture to spend Reachability is not a gate. Whether production happens to be up says nothing about whether this run intended to touch it — that was the original mistake, and matching test_x402_live.py's RUN_LIVE_TESTS pattern fixes it. Defaulting the URL to localhost means production cannot be reached by omission, only by naming it. And opting in to live tests is not opting in to buying postage, so the purchase carries its own switch and skips with a message that names it. Verified on each axis: default 9 skipped, 0.04s, no network RUN_LIVE_TESTS=1 (default target) 9 skipped, nothing listening on localhost RUN_LIVE_TESTS=1 against staging 3 passed, 6 skipped naming the spend switch Side effect worth having: the full suite drops from 84s to 21s, because it is no longer waiting on network round-trips to a remote host. 1075 passed, 25 skipped. Closes #233.
#233 made these opt-in and moved the default off production, which was the point. But localhost was the wrong alternative: someone opting in to live tests rarely has a gateway and a Bee node running locally, so the useful case needed extra configuration before it worked at all, and the obvious next step for anyone hitting that is to set GATEWAY_URL to whatever they can reach. Staging keeps the property that matters — production is never reached by omission, only by being named — while making `RUN_LIVE_TESTS=1 pytest ...` do something useful on its own. Verified: 3 passed, 6 skipped against staging with no other configuration. Deriving the target from the current git branch was considered and rejected. Being on `main` locally would point these at production, which is the hazard this module already had once. And the branch carries no useful signal: these tests exercise a DEPLOYED gateway, and the branch in a working tree is by definition not deployed — so matching it would test the environment you are about to deploy into rather than the change you are making. There is also no CI job to branch-match against; deploy.yml only deploys, and provenance-smasher already handles branch-aware targeting for the suite that runs in CI. Default run is unchanged and still hermetic: 1075 passed, 25 skipped.
…red (#309) Every rule was set to Grafana's middle NoData option, and the notification policy has a single route with no filters — everything reaches Telegram. So when metrics stopped arriving, SEVEN alerts fired at once: Critical BZZ Balance, Gateway Down, Low xDAI and four more, repeating every four hours until someone intervened. Not one of them was true. The balance was not critical, it was unknown. Seven alarms for one fault, all saying the wrong thing, is how a channel gets muted. Each of these rules answers a question about a measured value. None of them can answer anything when no measurement arrives, so none of them tries any more: all nine now stay silent when the query returns nothing, and speak only about data that actually met their threshold. One rule watches for measurements stopping. It is titled for the Telegram message rather than for the query — "No Data — System May Be Down" — because what the reader needs at that moment is what it might mean, not which series was empty. Its text says the system may be down OR monitoring may have broken, and does not pretend to know which, since the query genuinely cannot distinguish them. It also says that nothing else will alert while it is firing, so silence elsewhere is not read as good news. count(up{job=~"prometheus.scrape.*"} or bee_up) Returns a number whenever anything is reporting, and nothing at all when everything stops. The threshold (< 1) is deliberately unreachable while data exists — a count is at least 1 if anything reported — so the alert is driven entirely by the no-data path and cannot be suppressed by an unusual but healthy reading. Verified against live data: 4 targets reporting, two gateways and two Bee nodes. The trade-off is recorded in the README rather than left implicit: if the watchdog itself fails, the result is silence instead of noise. The old arrangement was noisy but hard to miss. This is a deliberate judgement that one accurate alarm beats seven misleading ones. Applied to the instance: 10 rules live, 1 firing on no data, 9 silent, none left on the middle option. Closes #263.
GET /metrics was reachable by anyone on both public hostnames, serving 146 series: wallet addresses, BZZ and xDAI balances, gateway version, which features are enabled, and request volumes. That is a funding position, readable by whoever is deciding whether draining it is worth attempting — and the endpoint was never meant to be public. Blocked at the reverse proxy rather than in the application, because the proxy is the only public route in. Alloy scrapes the container directly over the compose network (provenance_gateway:8000) and never traverses Caddy, so Grafana is completely unaffected — same metrics, same dashboards, same alerts. The gateway ports are bound to 127.0.0.1, so an operator on the host still reads /metrics as before. 404 rather than 403, matching the pool-check gate: a refusal confirms the endpoint is there. /health gains `version` and `uptime_seconds`, which is what makes the block safe to ship. Both were previously only readable from /metrics, and provenance-smasher uses them to tell "this test failed" from "the gateway was redeployed underneath the run" — a distinction that already saved one false investigation today. Blocking /metrics without this would have degraded that check to comparing "unknown" with "unknown" forever, which is exactly the silent-no-op failure mode that check itself was written to catch. Uptime falling is the reliable half: redeploying the same commit leaves the version string unchanged. Neither field is a disclosure /health was not already making — it reports the Bee node's version, api_version and overlay in the same response. Two provenance-smasher tests read /metrics through the public URL. Both already skip when it does not return 200, so they degrade rather than fail; they lose coverage that only ever worked because the endpoint was exposed. Caddyfile validated with `caddy validate`, and the adapted JSON confirms both hosts route /metrics to a 404. Full suite: 1075 passed, 25 skipped. Closes #188.
PR #311 reported conflicts in deploy/Caddyfile, monitoring/alerting/README.md and monitoring/alerting/alert-rules.json. None of them were real disagreements about content. The cause is squash-merged promotions. #304 and #306 were squashed, so main's history contains none of dev's commits and the merge base is #267 — from before any of this work. Files that arrived on main via a squashed promotion look, to git, like files created independently on both branches, which is the "add/add" conflict it reported. Resolved by taking dev's version of all three, after checking that dev is strictly ahead rather than assuming it: deploy/Caddyfile +32 -0 alerting/README.md +30 -0 alert-rules.json +85 -8 The only removals are the eight noDataState lines, seven NoData and one Alerting, all becoming OK — which IS the #263 change, not a loss. Verified afterwards that each file matches origin/dev byte for byte, that main holds no file dev lacks, that the rules still parse with the single watchdog present, and that the Caddyfile still blocks /metrics on both sites. Useful side effect: this merge makes main an ancestor of dev, so the next promotion has a correct merge base and will show only what actually changed rather than forty files. Merging #311 with a merge commit rather than a squash keeps it that way. Full suite: 1075 passed, 25 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four changes, and one of them closes an inconsistency I created on production yesterday.
Read the file count carefully
GitHub will show a large diff. That is the merge-base artefact again: #304 and #306 were squash-merged, so
mainanddevshare no recent history and the whole previous promotion reappears even thoughmainalready has it.The real content difference is 7 files, 308 insertions:
Merging with a merge commit rather than a squash would fix this permanently — dev's commits become ancestors of main, and future promotion PRs would show only what actually changed.
Why this should not wait
Applying the Caddy block to the host hit both hostnames, but the
/healthfields that replace it are app-level and only ondev:So production has
/metricsblocked with nothing offering the replacement. provenance-smasher's redeploy check reports itself inoperative against production until this lands. It warns rather than lying — that was deliberate — but it is a gap I created by applying a proxy change ahead of the app change it depends on.What is in it
/healthreportsversionanduptime_seconds(#188). Both were only readable from/metrics, which is no longer public. They are what lets an external checker tell "this test failed" from "the gateway was redeployed underneath the run" — a distinction that saved a false investigation on 2026-08-26. Uptime falling is the reliable half; redeploying the same commit leaves the version unchanged. Neither is a disclosure/healthwas not already making: it reports Bee's version, api_version and overlay in the same response.Alerting: one rule for missing data, nine that speak only about measurements (#263). Every rule used Grafana's middle
NoDataoption and the notification policy has a single unfiltered route, so when metrics stopped, seven alerts fired at once — "Critical BZZ Balance", "Gateway Down", "Low xDAI" and four more, every four hours. None were true; the balance was not critical, it was unknown. Now one alarm fires saying the system may be down or monitoring may have broken, and admits it cannot distinguish them.The test gate no longer touches production (#233). It defaulted to the production gateway and skipped only when production was unreachable, so every local run sent real traffic there — and the
usable_stampfixture purchases a postage batch when it cannot find a usable one. Real BZZ, from a routine test run. Now opt-in on three switches, defaulting to staging. Verified with a shim raising on any non-localhost connection: zero outbound calls. Side effect: the suite dropped from 84s to 21s./metricsreturns 404 publicly (#188). Already live on the host; this syncs the repo copy, which is version control only and not applied by the deploy.Closes on merge
#188, #233, #263 — all fixed and deployed, still open because PRs merging into
devnever auto-close.Pre-flight
ok, bee healthy, 134 peers, pool at full reserve (5)ssl_verify_result=0Deploying recreates
provenance_gatewayonly. The pool is at full reserve, so the restart triggers no purchase.