Skip to content

Delphi on Postgres: queue runs, run-bound results, readers behind a switch - #3007

Open
colinmegill wants to merge 2 commits into
edgefrom
dynamo-on-postgres-edge
Open

colinmegill wants to merge 2 commits into
edgefrom
dynamo-on-postgres-edge

Conversation

@colinmegill

Copy link
Copy Markdown
Member

Replaces #3004, rebuilt on edge after the job tables shipped (#3003) and the routing prerequisite landed (#3006).

Observable changes: none by default. The reader switch (DELPHI_RESULT_BACKEND) defaults to DynamoDB; Delphi's queue path and the importer are present but inactive until turned on.

What it adds: Delphi through the queue as dependent per-step jobs; results stored in Postgres tied to their run; server routes and the report able to read them behind the switch; an importer command for existing DynamoDB results.

Proof (work box): end-to-end scenario with DynamoDB stopped; default replay byte-exact. Hosted CI is the witness for the full server unit suite (flaky on the box, tracked separately).

🤖 Generated with Claude Code

https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

…ind a switch

Delphi runs through the job queue as dependent per-step jobs; each step's
result is stored in Postgres tied to its run. Readers can read from
Postgres behind a switch that defaults to DynamoDB, so merging changes
nothing until the switch is turned on. An importer command copies existing
DynamoDB results; it is not run automatically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 128 107 16%
benchmarks/bench_repness.py 81 65 20%
benchmarks/bench_update_votes.py 38 28 26%
benchmarks/benchmark_utils.py 34 0 100%
components/init.py 1 0 100%
components/config.py 165 70 58%
conversation/init.py 2 0 100%
conversation/conversation.py 1291 264 80%
conversation/group_vote_counts.py 38 2 95%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 396 178 55%
database/postgres.py 414 95 77%
delphi_storage/init.py 1 0 100%
delphi_storage/codec.py 295 23 92%
delphi_storage/golden_corpus.py 60 9 85%
delphi_storage/legacy_import.py 168 14 92%
delphi_storage/postgres.py 49 22 55%
delphi_storage/resource.py 173 34 80%
empty_output.py 29 2 93%
engine_adapter.py 317 265 16%
job_child/init.py 325 49 85%
job_child/census.py 101 13 87%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 38 85%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/group_k_smoother.py 28 0 100%
pca_kmeans_rep/legacy_kmeans.py 197 8 96%
pca_kmeans_rep/pca.py 162 19 88%
pca_kmeans_rep/repness.py 226 5 98%
poller/init.py 4 0 100%
poller/admission.py 318 15 95%
poller/backfill.py 936 32 97%
poller/capacity.py 485 27 94%
poller/capacity_queue.py 248 13 95%
poller/coordinator_bridge.py 317 317 0%
poller/math_writer.py 113 3 97%
poller/promotion.py 112 4 96%
poller/readiness.py 368 13 96%
poller/rebuild_child.py 133 5 96%
poller/service.py 665 52 92%
poller/worker_pool.py 140 0 100%
queue/executor.py 223 111 50%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 20 89%
regression/comparer.py 891 640 28%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 130 85 35%
replay/init.py 2 0 100%
replay/attribution_capture.py 49 0 100%
replay/axis_continuity.py 347 24 93%
replay/certify.py 999 59 94%
replay/crosslang.py 128 2 98%
replay/decision_ties.py 55 8 85%
replay/diagnostics.py 50 7 86%
replay/driver.py 73 1 99%
replay/dsn_admission.py 36 1 97%
replay/equiv_query.py 10 0 100%
replay/event_ingress.py 95 1 99%
replay/fixture_bundle.py 815 62 92%
replay/fixture_config.py 266 11 96%
replay/fixture_extract.py 285 12 96%
replay/fixture_generate.py 103 1 99%
replay/fixture_samples.py 157 14 91%
replay/fixture_selection.py 119 0 100%
replay/fixture_survey.py 147 11 93%
replay/legacy_pca.py 29 17 41%
replay/polarity.py 319 25 92%
replay/poller_equiv.py 776 99 87%
replay/prodclone.py 193 2 99%
replay/real_data.py 146 10 93%
replay/schedule.py 205 10 95%
replay/served_math.py 159 4 97%
replay/shard_bench.py 189 65 66%
replay/stagecompare.py 890 161 82%
replay/stages.py 256 40 84%
replay/stepcompare.py 86 1 99%
replay/store.py 120 11 91%
replay/tie_capture.py 107 19 82%
replay/types.py 71 6 92%
run_math_pipeline.py 284 93 67%
tools/engine_rebuild.py 230 11 95%
types.py 22 0 100%
umap_narrative/501_calculate_comment_extremity.py 113 38 66%
umap_narrative/502_calculate_priorities.py 136 136 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 311 311 0%
umap_narrative/702_consensus_divisive_datamapplot.py 435 275 37%
umap_narrative/801_narrative_report_batch.py 666 455 32%
umap_narrative/803_check_batch_status.py 268 148 45%
umap_narrative/llm_factory_constructor/init.py 2 0 100%
umap_narrative/llm_factory_constructor/model_provider.py 234 107 54%
umap_narrative/narrative_data.py 178 65 63%
umap_narrative/numerical_stages.py 54 25 54%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 9 7 22%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 332 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 581 466 20%
umap_narrative/reset_conversation.py 161 52 68%
umap_narrative/run_pipeline.py 337 207 39%
umap_narrative/topic_naming.py 148 11 93%
utils/cli_flags.py 15 0 100%
utils/clj_hash.py 44 0 100%
utils/env_flags.py 13 0 100%
utils/general.py 63 39 38%
utils/output_profile.py 66 2 97%
utils/serialization.py 10 5 50%
utils/vote_convention.py 159 5 97%
Total 24697 7876 68%

With DELPHI_RESULT_BACKEND=postgres, job submission is admitted to the
Postgres queue and narrative/statement writes go to Postgres, so nothing
writes DynamoDB once the switch is on. The default stays DynamoDB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant