Skip to content

Import: give the job daemon a queue DSN its parser can read - #3057

Merged
colinmegill merged 1 commit into
edgefrom
import-worker-daemon-dsn
Oct 14, 2026
Merged

colinmegill merged 1 commit into
edgefrom
import-worker-daemon-dsn

Conversation

@colinmegill

Copy link
Copy Markdown
Member

Observable change: the detached production import can start its job daemon instead of failing immediately with "transport refused: unparseable DSN".

The import requires its queue login to use libpq verify-full TLS, so QUEUE_DATABASE_URL carries sslmode=verify-full and a pinned sslrootcert; the daemon parses DSNs with tokio-postgres, which rejects those parameters. Under POLIS_JOBS_TRANSPORT=tls the daemon now gets the same DSN with only the libpq TLS parameters removed; everything else is kept byte for byte and the derived DSN is never logged. The Python-side verify-full requirement is unchanged, and the daemon still verifies the server certificate (built-in roots off, only the pinned CA trusted, hostname checked, exact host allowlist).

Tests: Python unit tests on the exact production DSN shape; a queue-rs test that the derived DSN parses under TLS and the libpq form is refused. The participant-facing request path is not touched.

🤖 Generated with Claude Code

The detached production import requires its queue login to reach the database
with libpq verify-full TLS, so QUEUE_DATABASE_URL carries sslmode=verify-full and
a pinned sslrootcert. The importer started the Rust job daemon with that same
URL, which the daemon's tokio-postgres parser rejects ("transport refused:
unparseable DSN"). The existing tests use a loopback DSN, so they never hit it.

Under POLIS_JOBS_TRANSPORT=tls the daemon now gets the same postgresql:// URL with
only sslmode (verify-full or require) and sslrootcert removed, which the daemon
reproduces: built-in roots off, only the pinned CA trusted, hostname checked,
host restricted to an exact allowlist. Every other span is kept byte for byte,
and the derived DSN is never logged. Keyword/value DSNs and TLS policy
parameters the daemon does not reproduce (client certificates, CRLs, protocol
bounds and so on) are refused rather than dropped. The Python-side verify-full
requirement is unchanged.

Tests: Python unit tests on the exact production DSN shape, encoded keys, odd
passwords and the refusals, and a queue-rs test
that the derived DSN parses under TLS while the libpq form is refused. The
participant-facing request path is not touched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
@github-actions

Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
polismath/init.py 2 0 100%
polismath/benchmarks/bench_pca.py 128 107 16%
polismath/benchmarks/bench_repness.py 81 65 20%
polismath/benchmarks/bench_update_votes.py 38 28 26%
polismath/benchmarks/benchmark_utils.py 34 0 100%
polismath/components/init.py 1 0 100%
polismath/components/config.py 165 70 58%
polismath/conversation/init.py 2 0 100%
polismath/conversation/conversation.py 1291 264 80%
polismath/conversation/group_vote_counts.py 38 2 95%
polismath/conversation/manager.py 131 42 68%
polismath/database/init.py 1 0 100%
polismath/database/dynamodb.py 392 176 55%
polismath/database/postgres.py 414 95 77%
polismath/delphi_storage/init.py 1 0 100%
polismath/delphi_storage/codec.py 295 22 93%
polismath/delphi_storage/embedding_spool.py 58 6 90%
polismath/delphi_storage/golden_corpus.py 60 9 85%
polismath/delphi_storage/import_report.py 51 3 94%
polismath/delphi_storage/legacy_import.py 171 16 91%
polismath/delphi_storage/postgres.py 49 22 55%
polismath/delphi_storage/production_import.py 573 124 78%
polismath/delphi_storage/resource.py 179 32 82%
polismath/delphi_storage/scan_pacing.py 77 0 100%
polismath/delphi_storage/writer.py 193 18 91%
polismath/empty_output.py 29 2 93%
polismath/engine_adapter.py 317 265 16%
polismath/job_child/init.py 356 52 85%
polismath/job_child/census.py 101 8 92%
polismath/pca_kmeans_rep/init.py 5 0 100%
polismath/pca_kmeans_rep/clusters.py 257 38 85%
polismath/pca_kmeans_rep/corr.py 98 17 83%
polismath/pca_kmeans_rep/group_k_smoother.py 28 0 100%
polismath/pca_kmeans_rep/legacy_kmeans.py 197 8 96%
polismath/pca_kmeans_rep/pca.py 162 19 88%
polismath/pca_kmeans_rep/repness.py 226 5 98%
polismath/poller/init.py 4 0 100%
polismath/poller/admission.py 318 15 95%
polismath/poller/backfill.py 936 32 97%
polismath/poller/capacity.py 485 27 94%
polismath/poller/capacity_queue.py 248 13 95%
polismath/poller/coordinator_bridge.py 317 317 0%
polismath/poller/math_writer.py 113 3 97%
polismath/poller/promotion.py 112 4 96%
polismath/poller/readiness.py 368 13 96%
polismath/poller/rebuild_child.py 133 5 96%
polismath/poller/service.py 665 52 92%
polismath/poller/worker_pool.py 140 0 100%
polismath/queue/executor.py 223 111 50%
polismath/regression/init.py 4 0 100%
polismath/regression/clojure_comparer.py 188 20 89%
polismath/regression/comparer.py 891 640 28%
polismath/regression/datasets.py 135 27 80%
polismath/regression/recorder.py 36 27 25%
polismath/regression/utils.py 130 85 35%
polismath/replay/init.py 2 0 100%
polismath/replay/attribution_capture.py 49 0 100%
polismath/replay/axis_continuity.py 347 24 93%
polismath/replay/certify.py 999 59 94%
polismath/replay/crosslang.py 128 2 98%
polismath/replay/decision_ties.py 55 8 85%
polismath/replay/diagnostics.py 50 7 86%
polismath/replay/driver.py 73 1 99%
polismath/replay/dsn_admission.py 36 1 97%
polismath/replay/equiv_query.py 10 0 100%
polismath/replay/event_ingress.py 95 1 99%
polismath/replay/fixture_bundle.py 815 62 92%
polismath/replay/fixture_config.py 266 11 96%
polismath/replay/fixture_extract.py 285 12 96%
polismath/replay/fixture_generate.py 103 1 99%
polismath/replay/fixture_samples.py 157 14 91%
polismath/replay/fixture_selection.py 119 0 100%
polismath/replay/fixture_survey.py 147 11 93%
polismath/replay/legacy_pca.py 29 17 41%
polismath/replay/polarity.py 319 25 92%
polismath/replay/poller_equiv.py 776 99 87%
polismath/replay/prodclone.py 193 2 99%
polismath/replay/real_data.py 146 10 93%
polismath/replay/schedule.py 205 10 95%
polismath/replay/served_math.py 159 4 97%
polismath/replay/shard_bench.py 189 65 66%
polismath/replay/stagecompare.py 890 161 82%
polismath/replay/stages.py 256 40 84%
polismath/replay/stepcompare.py 86 1 99%
polismath/replay/store.py 120 11 91%
polismath/replay/tie_capture.py 107 19 82%
polismath/replay/types.py 71 6 92%
polismath/run_math_pipeline.py 284 93 67%
polismath/tools/engine_rebuild.py 230 11 95%
polismath/types.py 22 0 100%
polismath/utils/cli_flags.py 15 0 100%
polismath/utils/clj_hash.py 44 0 100%
polismath/utils/env_flags.py 13 0 100%
polismath/utils/general.py 63 39 38%
polismath/utils/output_profile.py 66 2 97%
polismath/utils/serialization.py 10 5 50%
polismath/utils/vote_convention.py 159 5 97%
umap_narrative/501_calculate_comment_extremity.py 113 38 66%
umap_narrative/502_calculate_priorities.py 136 103 24%
umap_narrative/700_datamapplot_for_layer.py 507 507 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 435 275 37%
umap_narrative/801_narrative_report_batch.py 666 455 32%
umap_narrative/803_check_batch_status.py 268 148 45%
umap_narrative/llm_factory_constructor/init.py 2 0 100%
umap_narrative/llm_factory_constructor/model_provider.py 234 107 54%
umap_narrative/narrative_data.py 178 65 63%
umap_narrative/numerical_stages.py 54 25 54%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 268 268 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 9 7 22%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 332 289 13%
umap_narrative/polismath_commentgraph/utils/storage.py 609 489 20%
umap_narrative/reset_conversation.py 162 53 67%
umap_narrative/run_pipeline.py 342 212 38%
umap_narrative/topic_naming.py 148 11 93%
Total 25721 8020 69%

@colinmegill
colinmegill merged commit ded3e5b into edge Oct 14, 2026
32 of 33 checks passed
@colinmegill
colinmegill deleted the import-worker-daemon-dsn branch October 14, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant