Repository navigation
DRAFT — polis-jobs: parked jobs are rediscovered from the database after a restart (needs 000026, P-086) - #2983
Draft
colinmegill wants to merge 15 commits into
Draft
colinmegill wants to merge 15 commits into
colinmegill wants to merge 15 commits into
Conversation
server/postgres/bin/apply-migration.sh refused 000024 ("has no apply
policy here"), so an existing database had no checked path from
polis-queue/2 to /3. The wrapper now accepts 000024:
- seal: down/000024-files.sha256
- rights: superuser or SET membership in polis_queue_owner (as 000023)
- chain: 000023 installed and polis_queue_install reads polis-queue/2
(a second run is refused here, since the install then reads /3)
- rows: every queue and job table is empty (000024 alters the queue
tables' CHECK constraints, ACCESS EXCLUSIVE until COMMIT)
- lock line: names the four queue tables; no lock on conversations
- post-check: polis_queue_install reads polis-queue/3
The usage text now prints the whole header (it was cut at line 60).
test_000024_down.sh gains check (w): refused without 000023 and with a
queued job (catalog unchanged both times); --preflight-only passes and
changes nothing; the apply exits 0 with post-check /3 and the same
schema dump as a plain psql apply; a second run is refused at preflight.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… schema change) pq_sweep deletes one bounded page of expired queue history per call (P-083's bounds), recorded in a new polis_queue_sweeps ledger; pq_class_parked lists a class's parked jobs with their unproven attempts (P-086 restart-safe rediscovery); pq_queue_usage reports the queue's bytes and the last finished sweep; pq_class_depth gains oldest_eligible_at and answers polis-queue/4. Three indexes serve the sweep's scans. The contract version stays polis-queue/3. Sealed down script and its test (forward, backward byte-identical, refusals, the chain unwinding through 000024 and 000023). The poller's depth decoder accepts the /4 reply; the server and queue-rs tests learn the new functions and the jobs_v4 template. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…o queue/retention-000026
…rows are reported, not refused Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…needs migration 000026) The daemon runs retention itself, page by page through pq_sweep, only when its claim pass has nothing to start (cost-reduction plan P-083: the sweep never goes ahead of a job). The SQL decides cadence (once per 24 h per env), the single sweeper and every bound; each finished sweep is one polis_jobs.sweep/1 line with its counts. POLIS_JOBS_SWEEP_MAX_PAGES (100) and POLIS_JOBS_SWEEP_CHECK_SECONDS (900) bound it. On a database without 000026 the sweep says so once and turns itself off. Stacked on the 000026 draft: a schema-dependent change, draft too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…eaper tick (needs migration 000026) Cost-reduction plan P-086, restart-safe recovery: the reaper re-reads the class's parked jobs (pq_class_parked) instead of watching only the rows it saw park, so a restarted daemon still sees them. An attempt this daemon's journal knows is watched as before; one it does not know (its box is gone) prints polis_jobs.alarm/1 reason=exit_unproven on every tick until an operator records the proof. No proof is inferred from elapsed time. Without 000026 the rediscovery says so once and stays off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
# Conflicts: # server/postgres/bin/apply-migration.sh
…eft 'latest' to the job ids' order) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… (a grep that exits early made pipefail flaky) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Delphi Coverage Report
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observable changes: on a database with migration 000026, a
polis-jobsworker re-reads its class's parked jobs on every reaper tick. A parked attempt whose exit is unproven and that this worker's journal does not know (the box that ran it is gone) now printspolis_jobs.alarm/1with"reason":"exit_unproven","rediscovered":trueon every tick until an operator records the proof; before, a restarted worker never saw such a row and printed nothing. On a database without 000026 it printspolis_jobs parked rediscovery off: …once. Nothing infers proof from elapsed time; no job state changes. Draft: it stands on the 000026 schema ruling (#2978). Not a schema change itself; nothing runs a daemon.What it is
Cost-reduction plan P-086, "restart-safe recovery of parked and exit-proof state" (part of its PR 3). The reaper's watch set (
reaper.rs) was built only from transitions the process saw, so it was empty after a restart (finding in the operating-surface map, reproduced there). Now, after each reap pass, the reaper pages throughpq_class_parked(env, class, after_job, 100)(000026): for each parked job with unproven attempts, an attempt the journal knows goes into the existing watch (the journal recovery proves its exit when it can; the existing alarm after lease + one tick still applies); an attempt it does not know raises the alarm line above, each tick. The way out is the operator's recorded proof (pq_end_attempt … 'confirm_exit'with the evidence, after reading the instance's termination), after which the job is claimable as before.Not in this PR (rest of P-086 PR 3/4): the durable inventory keys on the readiness line, disk free, the
confirm-exitsubcommand, drain on SIGTERM.Gates (mm2)
cargo fmt --check, clippy-D warningsboth feature sets, unit 41 + 7 + 4,jobs_integration26 passed against postgres:17, including the newa_parked_job_is_rediscovered_after_a_restart_and_alarms_until_proven(worker A killed with its child and journal; B parks the row and stops; C, which never saw the transition, alarmsexit_unprovenevery tick, invents no proof, and runs the job once the proof is recorded, after which the alarm stops) and the /3 case (one "rediscovery off" line, the worker keeps working).test_gateandsign_lint: pass.Stacked on #2981 (the built-in sweep), itself on #2978; their commits show here until they land.
🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s