Conversation
The top-up loop took every batch in the pool regardless of depth, never
consulting the reserve config:
stamps_to_topup = [s for s in self._pool.values() if s.status == AVAILABLE]
So with a depth's target set to zero, the pool went on paying to keep those
batches alive. The incoherence shows on acquisition: a depth with a target of
zero is NOT replaced when it is handed out. The pool was maintaining inventory
it had already decided not to keep.
The cost is not marginal. A depth-20 batch costs eight times a depth-17 one, so
the two unwanted mediums on production cost more per day than the entire
intended pool — 0.295 against 0.018.
Batches at a dropped depth are now left to expire. They stay usable until they
do and sync removes them once gone, so nothing is discarded; the pool simply
stops paying. Each is logged once per cycle with the reason, so an operator
seeing a batch decay knows it is deliberate.
Found by the user asking why a depth set to zero was still being topped up
rather than accepting a manual state-file edit as the remedy. It was the right
question: editing the state would have fixed one instance of a general bug, and
the next configuration change would have reintroduced it.
Two tests, one of which matters more than it looks: that every held depth IS
still topped up when all are wanted, so the filter cannot quietly become an
excuse to stop maintaining the pool.
Both initially failed because the pool is disabled by default and a sync-health
guard blocks replenishment — the tests were wrong, not the fix.
Full suite: 1103 passed, 25 skipped.
…324) The deploy ran `rm -rf swarm_connect && git clone`, deleting the working checkout before knowing whether a replacement could be obtained. Today the clone failed and the host was left with no /opt/swarm_connect at all. The running containers were unaffected — they do not need the directory — but nothing could be deployed or recreated until it was restored by hand, which is a bad state to be in during an incident. It now clones to swarm_connect.new and swaps only on success. A failed deploy leaves the previous checkout exactly as it was. The clone is also authenticated now, with the job's own repository-scoped token. Anonymous git operations are rate limited per IP and every deploy clones the full repository, so the limit is reachable. The failure mode is worth recording because it is thoroughly misleading: ref discovery still returns 200, so the repository looks reachable, and only the follow-up upload-pack POST gets 401 — which git reports as fatal: could not read Username for 'https://github.com' That reads like a permissions or credentials problem on a repository which is public and was returning 200 to curl at the same moment. Diagnosis took a while for a second reason: the first failure was something else entirely. The runner had wedged — its log showed the previous job completing and then no record of the next one, while GitHub attributed that job to it and failed it after exactly ten minutes with no steps recorded. Restarting the runner service brought it back, and only then did the real error appear. Full suite: 1103 passed, 25 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge with a merge commit, not a squash.
Four files. Two fixes: one that stops production wasting money daily, one that stops a failed deploy destroying the host's checkout.
The pool stops paying for depths it has dropped
Top-up took every batch in the pool regardless of depth, never consulting the reserve config. With a depth's target at zero the pool no longer replaces that depth when it is handed out — so it was maintaining inventory it had already decided not to keep.
Production is doing exactly this right now:
Two surplus mediums, extended daily, that will never be replaced. The balance moved 16.8281 → 16.4908 overnight, and roughly 0.295 of that 0.337 was those two mediums — against 0.018 for the one batch the pool actually wants. The surplus costs sixteen times the intended pool.
After this they are simply left to expire. They stay usable until they do, and sync removes them once gone, so nothing is discarded. No manual state-file edit is needed, which was the alternative I had been about to reach for.
Each skipped batch is logged once per cycle with the reason, so an operator watching one decay knows it is deliberate.
A failed deploy no longer destroys the checkout
The deploy ran
rm -rf swarm_connect && git clone, deleting the working checkout before knowing a replacement could be obtained. Yesterday the clone failed and the host was left with no/opt/swarm_connectat all — running containers were fine, but nothing could be deployed or recreated until it was restored by hand.It now clones to a temporary directory and swaps only on success, and authenticates with the job's own repository-scoped token, because anonymous git operations are rate limited per IP and every deploy clones the full repository.
That failure was worth recording for how it presents: ref discovery still returns 200 —
curlconfirmed the repository reachable at the same moment — and only the follow-up fetch gets 401, which git reports ascould not read Usernameon a repository that is public.Tests
Two on the top-up filter. The second matters more than it looks: that every held depth is still topped up when all are wanted, so the filter cannot quietly become an excuse to stop maintaining the pool.
Full suite: 1103 passed, 25 skipped.
Verified on staging
The fix is confirmed running in the staging container, not merely merged. The deploy path itself was exercised twice yesterday during the recovery, so it is freshly proven.