Skip to content

capacity line: a revision field; the poller and ops parsers read older and newer revisions - #2987

Open
colinmegill wants to merge 2 commits into
edgefrom
ops/capacity-line-revisions
Open

colinmegill wants to merge 2 commits into
edgefrom
ops/capacity-line-revisions

Conversation

@colinmegill

Copy link
Copy Markdown
Member

Observable changes: the small math poller's math_poller.capacity/1 line, and the readiness line's embedded capacity object, gain one key, "rev":2. The server's ops "Math engine" page now parses capacity lines of every revision (production's nine-count line, edge's twelve-count line, and the revisions #2984 and #2985 add) instead of counting the other revisions as malformed; each parsed row carries rev, and a key the parser does not know is shown as nothing (null for an older line's missing keys; a newer line's extra keys are dropped). The Python depth decoder also reads 000026's polis-queue/4 depth reply (it read only /3). No CloudWatch filter changes (they select schema/class/role, which are unchanged). No schema change. No Lambda.

Why

The connections map (§3) found the TS parser and the Python decoder are closed per revision: edge has 12 count keys, #2984 14, #2985 18, under one schema string, and the actual transpiled parser accepted 3/3 matching revisions and refused 6/6 mixed ones. A rolling deploy (old and new pollers logging at once) and retained log history therefore lose ops observations. Separately, edge's poller refuses 000026's /4 depth reply, so applying 000026 under it would blank its depth. Production (stable) logs a third shape, nine keys.

Rule

  • The line carries rev, a minor version under the same schema string. Each revision only adds keys (CAPACITY_REV_KEYS in delphi/polismath/poller/capacity.py, mirrored in server/src/ops/readinessLine.ts).
  • Revision 1 = production's nine counts; revision 2 = edge's twelve. Both were logged without rev; an unrevisioned line is revision 1 or 2, told apart by its exact key set.
  • A decoder reads revisions 1..own+8 (REV_FORWARD). An older line must carry exactly its own revision's keys (newer known keys decode as null). A newer line must carry every key the decoder knows; the keys only its revision declares must be plain names with count-or-null values and are dropped, never relayed. Within the decoder's own revisions the shape stays closed (an undeclared key is refused).

Tests

  • delphi/tests/poller/fixtures/capacity-revisions.txt: both unrevisioned shapes, revisions 2, 3, 4 (primary and standby) and a forward revision 5. test_capacity_revisions.py pins this branch's revision line to the real build_line output and decodes every line; server/__tests__/unit/opsCapacityRevisions.test.ts parses the same file with the TS parser, including the readiness line's embedded object. Negative cases: undeclared key, missing declared key (each revision and both unrevisioned shapes), forward key not a count or not a plain name, revision out of range or not an integer.
  • Cross-branch matrix on the work box, actual emitters (build_line from a real CapacityRouter) × actual parsers (Python parse_line/validate_counts and the TS parser through ts-node), primary line + standby line + embedded object, both languages:
parser \ emitter stable edge (before) this PR (rev 2) #2984 (rev 3) #2985 (rev 4)
stable (deployed) 6/6 0/6 0/6 0/6 0/6
edge (before) 0/6 6/6 0/6 0/6 0/6
this PR 6/6 6/6 6/6 6/6 6/6
#2984 6/6 6/6 6/6 6/6 6/6
#2985 6/6 6/6 6/6 6/6 6/6

The six mixed combinations among the three new revisions all pass. The first two rows are the parsers already written; nothing here can change what they accept. The deployed stable parser already refuses edge's twelve-key line today, so the next promotion's roll has that window whatever this PR does; from this PR on, a roll does not.

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

colinmegill and others added 2 commits October 6, 2026 19:44
The small poller's capacity line (and the readiness line's capacity object)
now carries "rev" (1 here). A line without it predates revisioning and is
revision 1. Python decode_counts and the server's TypeScript decodeCounts
read revisions 1..own+8: an older line carries exactly its own keys (newer
ones decode as null); a newer line must still carry every key the decoder
knows, and the keys its revision declared are checked as counts and dropped,
never relayed. Within a decoder's own revisions the shape stays closed.

The Python depth decoder reads both 000024's polis-queue/3 reply and
000026's polis-queue/4 (with oldest_eligible_at), so a poller keeps its depth
when 000026 is applied under it.

delphi/tests/poller/fixtures/capacity-revisions.txt holds a line per revision
(pre-revision, 1, 2, 3 and a forward 4); the Python test pins this
revision's line to the real emitter and both decoders read every line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…lve revision 2

Production (stable) logs nine counts; edge added large_leased,
large_poisoned and large_parked. Both shapes were logged without a rev, so
both are kept readable: an unrevisioned line is revision 1 or 2, told apart
by its keys. This emitter writes rev 2 explicitly. The fixture now holds
both unrevisioned shapes, revisions 2..4 and a forward 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 0 100%
components/init.py 1 0 100%
components/config.py 165 70 58%
conversation/init.py 2 0 100%
conversation/conversation.py 1291 264 80%
conversation/group_vote_counts.py 38 2 95%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 395 178 55%
database/postgres.py 414 95 77%
delphi_storage/init.py 1 0 100%
delphi_storage/codec.py 295 23 92%
delphi_storage/golden_corpus.py 60 9 85%
empty_output.py 29 2 93%
engine_adapter.py 317 265 16%
job_child/init.py 325 49 85%
job_child/census.py 101 13 87%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 38 85%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 197 7 96%
pca_kmeans_rep/pca.py 162 19 88%
pca_kmeans_rep/repness.py 226 5 98%
poller/init.py 4 0 100%
poller/admission.py 318 15 95%
poller/backfill.py 936 32 97%
poller/capacity.py 524 29 94%
poller/capacity_queue.py 255 14 95%
poller/coordinator_bridge.py 317 317 0%
poller/math_writer.py 113 3 97%
poller/promotion.py 112 4 96%
poller/readiness.py 368 13 96%
poller/rebuild_child.py 133 6 95%
poller/service.py 665 52 92%
poller/worker_pool.py 140 0 100%
queue/executor.py 223 111 50%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 891 640 28%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/attribution_capture.py 49 0 100%
replay/axis_continuity.py 347 24 93%
replay/certify.py 999 59 94%
replay/crosslang.py 128 2 98%
replay/decision_ties.py 55 8 85%
replay/diagnostics.py 50 7 86%
replay/driver.py 73 1 99%
replay/dsn_admission.py 36 1 97%
replay/equiv_query.py 10 0 100%
replay/event_ingress.py 95 1 99%
replay/fixture_bundle.py 815 62 92%
replay/fixture_config.py 266 11 96%
replay/fixture_extract.py 285 12 96%
replay/fixture_generate.py 103 1 99%
replay/fixture_samples.py 157 14 91%
replay/fixture_selection.py 119 0 100%
replay/fixture_survey.py 147 11 93%
replay/legacy_pca.py 29 17 41%
replay/polarity.py 319 25 92%
replay/poller_equiv.py 776 99 87%
replay/prodclone.py 193 2 99%
replay/real_data.py 146 10 93%
replay/schedule.py 205 10 95%
replay/served_math.py 159 4 97%
replay/shard_bench.py 189 65 66%
replay/stagecompare.py 890 161 82%
replay/stages.py 256 40 84%
replay/stepcompare.py 86 1 99%
replay/store.py 120 11 91%
replay/tie_capture.py 107 19 82%
replay/types.py 71 6 92%
run_math_pipeline.py 284 93 67%
tools/engine_rebuild.py 230 11 95%
types.py 22 0 100%
umap_narrative/501_calculate_comment_extremity.py 113 38 66%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 434 275 37%
umap_narrative/801_narrative_report_batch.py 837 620 26%
umap_narrative/803_check_batch_status.py 267 148 45%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 234 234 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 331 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 580 466 20%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 379 229 40%
umap_narrative/topic_naming.py 148 148 0%
utils/cli_flags.py 15 0 100%
utils/clj_hash.py 44 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 63 39 38%
utils/output_profile.py 66 2 97%
utils/serialization.py 10 5 50%
utils/vote_convention.py 159 5 97%
Total 24319 8161 66%

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant