[DO NOT MERGE] feat(etl): седмичен дайджест „Седмицата в пари“ — производител + понеделнишки cron (#167A) - #324
Draft
DiyanaDimitrova wants to merge 443 commits into
Draft
Conversation
…text
Two PROSE_NUMBER_PATTERNS in report-schema.ts false-matched ordinary prose and
made emit_report fail (ok:false) on legitimate reports — observed live: Q18
"Колко похарчи Столична община през 2023 г." failed terminally, Q2 needed a
retry (see docs/ai-assistant-chat-testing-2026-07-02.md).
- percent idiom `на\s+сто`: pin "сто" as a standalone word with `(?!\p{L})`, so
the "сто" word-family no longer matches — "на стойност" (ubiquitous in
procurement), the entity "Столична община", "на стотици". "12 на сто" / "на
сто%" still match.
- grouped thousands `\d{1,3}(?:[.,\s…]\d{3})+`: add a trailing `(?!\d)` so a
four-digit run is not read as a 3-digit group — `MM.YYYY` / `DD.MM.YYYY` dates
(01.2026, 01.02.2026) are no longer flagged as "01.202". A real grouped amount
always ends on a 3-digit group, so nothing valid is lost.
Adds regression tests for both classes (date notation + the сто-family) and
asserts the genuine percent idiom and grouped amounts are still caught.
The assistant answered "no match" for entities that plainly exist (e.g.
Столична община / authority 000696327). Root cause: the model resolved names
with `LIKE`/`=` on `authorities.name`, but SQLite folds case for ASCII only —
Cyrillic is compared case-sensitively and `upper()`/`lower()` leave it
unchanged. Names are stored mixed/UPPERCASE ("СТОЛИЧНА ОБЩИНА"), so a
title-case `LIKE '%Столична община%'` silently returns 0 rows. semantic_search
(vector) also missed it, leaving no working path.
Fix: add a `find_entity` tool that resolves a name to its exact join id via the
site's FTS5 `search_index` (unicode61 tokenizer folds case + diacritics for
Cyrillic and Latin), reusing the same ranked prefix-AND query the website uses
(`searchMatchQuery` from @sigma/db). It returns `search_index.ref` — the
authority_id / bidder_id the model joins on. run_sql can't do this (its parser
rejects FTS MATCH); find_entity runs a server-authored, parameterized query.
Also updates the data dictionary to steer the model to `find_entity` and warn
that `LIKE`/`=` on name is case-sensitive for Cyrillic.
Tests cover the id-per-kind formatting (the Столична община case), the
too-short-term and no-match paths, and graceful degradation on a query error.
…condense feat(assistant): condense old chat turns into a recap message for the POST
When the weak model gathered real data (run_sql results) but never produced a VALID emit_report within the step budget — a number in prose (correctly gated), a malformed block shape, or simply running out of steps — the turn dead-ended on "Справката не можа да бъде съставена" even though the answer sat in ctx.results. Observed live for Q18 (Столична община 2023): find_entity resolved the id and run_sql returned 250,264,972.88 € / 293 contracts, but the model put the figure in a text block and hit the budget, so no report rendered. Add a server-side fallback: after the model loop finishes, if no valid report was emitted but results exist, synthesize a minimal report from the actual data and inject it as a tool-emit_report part (the shape the dock already renders). - report-fallback.ts: pure buildFallbackReport(results, question) — picks the last non-empty result and renders it as a `totals` block (single numeric row) or a `table` (rankings/breakdowns). Values bind through the SAME bindReport path, so they stay server-owned (never model-written); a fixed number-free title can't trip the E2 gate. - agent.ts: wrap the model stream in createUIMessageStream; after the loop, inject the fallback report unless the model finalized, errored, or gathered no data. onError also resolves the finish latch so the stream can never hang. Tests cover the totals/table/empty/last-non-empty/huge-number cases and guessFormat. Full assistant suite 319 passed / 1 skipped; typecheck + prettier clean.
…fields rules Review follow-ups on PR #24: - redactEmitReportOutput now also masks a `tool-input-error` (raised when the model's report args fail input-schema validation) — its errorText can echo schema/column shape, the same leak the output-side redaction already closes. - Fold INTERNALS_NON_DISCLOSURE_RULE into NO_INTERNAL_FIELDS_RULE: one rule now forbids SQL/column/tool internals in BOTH the chat prose and report `text`/ `callout` blocks (they overlapped after #20 merged). - Rename the test's `c` chunk-builder to `asChunk` for readability.
…column names) The server-synthesized fallback showed raw SQL column names to the reader (total_spent_eur, contracts_count). Map them to Bulgarian display labels via a curated dictionary keyed on the columns the model actually produces, with a de-snaked/capitalised degrade for anything unmapped (never the raw identifier). Only the DISPLAY label changes — bound values still reference the real column.
…lint) Same one-line formatting as #30; the fallback branch inherits the base's unformatted describe-schema.ts, so `prettier --check .` reds this PR's Lint job. Identical change → merges cleanly regardless of order.
Add a 'Resolution — follow-ups landed' section: the emit-shape false-positives (prose gate) and the newly-found Cyrillic entity-lookup bug are fixed (#27), the server-side finalizer backstops the never-finalize cases (#31), and finishReason 'other' is documented as a benign provider quirk. Includes the live re-verification table (Q18/Q2/Q19).
Update PR #24 to the advanced base (find_entity, chat-history condense, prose-number gate fix). One conflict — useAssistantChat.ts imports: keep BOTH the phase (isPhasePart/AssistantPhase) and the base's condenseForPost; their bodies auto-merged (condenseForPost in the POST prep, phase state/reset intact). Prettier-format describe-schema.ts (the base's find_entity commit left it unformatted; the PR merge commit includes the base tip, so CI flags it).
…, stuck spinner) Follow-ups from the PR-31 UI question sweep: 1. Empty completion (Q3) — a model turn that returns no report, no run_sql data, and no prose (finishReason 'other', 0 tokens) dead-ended on a BLANK transcript: the server fallback needs rows to synthesize, so it can't fire. agent.ts now captures finishReason + whether prose was produced, and writes an explicit Bulgarian affordance when the turn would otherwise be empty (a legit prose-only reply is left untouched). Empty completions are now logged. 2. Broken percentage (Q8) — the single-offer "Дял по стойност" totals metric rendered a raw euro sum as "1342360573264,6%". Root causes: the by-value canonical query returned only euro sums (no share column), and nothing validated a percent magnitude. Fixes: describe-schema now exposes single_offer_share (0..1); bindReport rejects a percent slot bound to a non-ratio (→ model retries); formatCell renders an em-dash for an impossible ratio; the fallback downgrades a name-guessed percent to a number when the value isn't a ratio; system prompt now states percent must reference a 0..1 share. Shared isImplausibleRatio threshold keeps binder and renderer from drifting. 3. Stuck spinner (Q13) — a fallback-finalized report leaves the model's original emit_report part orphaned at input-available, so the "Подготвям справка…" spinner never cleared. AssistantTranscript now gates the pending indicator on busy + last turn, so it clears once the stream settles. Also bounds the per-year rollup query to the current period (matching the monthly query) so stray future-dated rows (2027–2029) no longer appear. Tests added across report-schema, render-format, report-fallback, and AssistantTranscript. Full web unit (729) + golden (150) suites pass.
…hat then succeeds A first emit_report that returns ok:false is normally retried (the loop re-forces emit_report) and then lands ok:true — but the transcript flashed „Справката не можа да бъде съставена." during the retry window, contradicting the report that appeared a moment later. Gate the failure line on the last turn being settled (!busy): while the turn is in flight the pending indicator carries the state; an earlier turn that genuinely ended ok:false still shows its failure line. Observed on Q13 (Разход по години) re-verification after the previous fix commit.
…o stray years)
Airtight follow-up to the year-rollup fix. The describe-schema template bounded the
year query, but that was guidance the weak model could drop on an exploratory query —
which then returned out-of-coverage rows (2016/2029, source date-quality errors) and
led the model to both chart them and narrate a wrong range ("от 2016 до 2026").
Add a conditional gate in assertDefaultFilters: a base-contracts query that BUCKETS by
signed_at (substr(signed_at,1,4|7) in SELECT/GROUP BY) must bracket the date range —
either a raw signed_at range (upper AND lower) or a pinning =/IN/BETWEEN on signed_at
(raw or derived). Fully-unbounded buckets, or ones constrained only by IS NOT NULL /
the GLOB well-formedness check, are rejected so the model adds the window. This mirrors
the site's own trend query (packages/db/src/queries/trend.ts) and is conditional, so
non-temporal totals/top-N are untouched — the totals⇄rollup reconciliation basis is
unchanged. With the model never seeing stray-dated rows, it no longer narrates years
outside coverage either.
- assertDefaultFilters: bucketsBySignedAt + seriesIsDateBounded (range or pin).
- describe-schema DATA_TRAPS: state the mandatory series window + "don't cite years
outside 2020–2026".
- golden fixtures 03/11/20: add the coverage window to their bucketed SQL so they pass
the tightened gate (22 already pins via substr(signed_at)= '2023').
Tests: 6 new gate cases (unbounded/GLOB-only rejected; range/pin/period accepted;
non-temporal unaffected). Full unit (737) + golden (150) suites pass.
…rmat + log bind miss Address three review findings on the report fallback finalizer: - Single-row [name, number] results dropped the label: a totals block is now used only when the single row is ENTIRELY numeric; a row that also carries a text/label column (entity name, period) routes to a 1-row table so that context is preserved instead of silently dropped. - guessFormat stole counts into money via the generic "total" token: the count shape is now checked before the broad money pattern, gated on the absence of a hard currency token (total_count -> number, while total_spent_eur / won_eur stay money). - The fallback failing to bind was a silent no-op: log a warn with the bind errors so the "had data, still no report" case is observable.
…t version - add a 60s settle backstop around the model-loop wait so the stream wrapper can never hang if the SDK fails to fire onFinish/onError; bail without synthesizing when it trips (loop state is indeterminate) - derive PROMPT_VERSION from a FNV-1a hash of the canonical system prompt so semantic edits to system-prompt.ts / describe-schema.ts re-fingerprint automatically instead of relying on a manual date bump - hoist the duplicated guessFormat call in buildFallbackReport - assert the exact default-filters callout count (3) instead of > 0
A totals item is a single headline aggregate, but the weak model bound one to row 0 of a multi-row result — the live 'Разход по години' report showed 'Общ разход 2020-2026: 762,1 млн. €', which was merely the 2020 row (~61x below the real ~46,6 млрд. € sum). The value is a genuine cell so no existing gate caught it. - bindReport now rejects a totals item whose ref points to a result with >1 row, so the model retries with a proper SELECT SUM/COUNT (or renders the series as a table/timeseries). facts stays exempt (row highlight). - strengthen the emit-report block guide to state totals must reference a single-row aggregate. - test the reject path; all golden fixtures (single-row totals) still pass.
The server-side finalizer inferred money from the bare aggregate words total/sum without a currency token, and the count-shape guard omitted bids/offers — so a bid/offer count aliased total_bids/sum_offers rendered as euros (e.g. total_bids=5 -> "Обща стойност (€): 5,00 €"). - guessFormat: add оферт|bids|offers to the count shape; return money only on a hard currency token or a sum/total/won that isn't count-shaped. - COLUMN_LABELS: gate the "(€)" label on a currency token (or a non-count total/sum) and add a "Брой оферти" label. - tests: cover the total_bids repro end-to-end plus guessFormat/humanizeColumn.
… a number in the dock The dock chip's leadStat reads the first totals/facts item, so a table-only report shows just a title. Add HEADLINE_TOTALS_RULE: for a list/breakdown report with a meaningful summary aggregate, lead with a totals block (full-population SUM(amount_eur) / COUNT of contracts) computed by a DEDICATED aggregate query and referenced by handle — never pointed at a list row, never written in prose. The chip then surfaces it automatically; no dock change. Conditional (skips flows / period-only timeseries / no-aggregate cases) to avoid manufacturing a misleading headline on a public transparency report. Bumps PROMPT_VERSION. Prompt-only: the golden replay never builds the prompt, so the fixtures are untouched.
…um-first Review found the rule listed flows/timeseries in BOTH the trigger set and the skip clause — a weak model could manufacture a misleading SUM on a Sankey or a period timeseries. Trigger only on table/bar; flows/timeseries stay in the skip clause. Also pin the first totals item to the money total (leadStat shows only the first item, so the chip leads with the sum, not the count).
Resolve PROMPT_VERSION toward the computed sp_${fnv1a(buildSystemPrompt)} form — it
auto-captures #32's HEADLINE_TOTALS_RULE, superseding the manual literal.
# Conflicts:
# apps/web/app/lib/assistant/agent.ts
…allback feat(assistant): server-side last-resort report finalizer (never dead-end with real data)
…feat/assistant-stream-phases # Conflicts: # apps/web/app/lib/assistant-dock/AssistantTranscript.test.tsx # apps/web/app/lib/assistant-dock/AssistantTranscript.tsx # apps/web/app/lib/assistant/agent.ts # apps/web/app/lib/assistant/system-prompt.test.ts
…hases feat(assistant): stream only coarse phases to the dock, not the raw agent loop
# Conflicts: # apps/web/app/app.css
…ide) Verify a Cloudflare Turnstile token before buffering the body or any paid model/D1 work. No-op until TURNSTILE_SECRET is provisioned (dev/preview/staging), so it activates only at the launch gate. New turnstile.ts (verify + reject helpers, fail-closed) + tests; hooked after firstPartyRejection; TURNSTILE_SITE_KEY var + Env typing. Client widget/token header is the paired follow-up.
…ing (#36) Softens missing table display columns to null cells with a recorded warning (link id cols + chart value cols still hard-error); mobile-dock onOpenReport threading; bindReport tests + warning surfacing.
…nistic gates (#38) Adds a risk-scaled LLM verifier as role ④ behind the deterministic gates: every report (model path + deterministic fallback) passes through verify() before persist/stream, so nothing reaches the user unverified. Strip-only output channel.
…-artifact 404 nikimilenkov Finding 1 (should-fix): seed-weekly-digest.mjs defaulted BUCKET to the PRODUCTION `sigma-reports` and printed `--remote` put/delete commands — a copy-paste could publish fabricated digests (naming real institutions) on the public site and delete real cron artifacts (recentWeeks() overlaps live weeks). Default to sigma-reports-dev; withhold prod `--remote` commands unless SIGMA_REPORTS_NAME=sigma- reports AND ALLOW_PROD_SEED=1, with a loud warning either way. lyubomir MINOR (defense-in-depth): the meta noindex on /weeks/:iso doesn't cover the React Router `.data` twin (JSON, bypasses <head>). Set `X-Robots-Tag: noindex` at the worker for the detail path (matches `/weeks/:iso` + `/weeks/:iso.data`), leaving the archive `/weeks` indexable. Integration tests for both twins + the archive. nikimilenkov Finding 2 (minor): readStoredReport does no shape validation, so a valid- JSON-but-wrong-shape artifact 500'd every request. The loader now treats a malformed artifact (missing report.blocks / provenance.freshness) as absent → 404. Test added. 1203 web + 71 etl tests pass; typecheck + prettier clean.
…umer feat(weekly-digest): consumer render layer + /weeks routes (#167B)
…af-index+JSON-LD midt-bg#212, semgrep midt-bg#255, osv midt-bg#271); keep sharp pin + all security overrides
…o feat/weekly-digest # Conflicts: # apps/etl/src/index.ts # apps/web/app/lib/assistant/describe-schema.ts # osv-scanner.toml # pnpm-lock.yaml
… workers-types dep Resolves two review findings on PR #80: - MINOR (review 5078506825): the base seed only paired value_flag='value_suspect' with a NULL amount_eur, so „summed in total, excluded from largest/top" held by construction. Add a suspect row carrying the week's HIGHEST non-NULL amount_eur and assert it lands in getWeeklyTotal (flag-independent basis) yet is filtered from getWeeklyLargestContract / getWeeklyTopContracts (value_flag='ok' guard). - MEDIUM (strict review 5094463521): @sigma/report used R2Bucket/ExecutionContext ambient types without declaring @cloudflare/workers-types; add it as a devDependency so the package type-resolves in isolation.
Review of #80 flagged that the two shim modules used `export * from '@sigma/report'`, which re-exported the ENTIRE barrel — making ./emit-report-schema and ./report-schema identical aliases of the whole package and silently hiding any name collision in the barrel. Replace the wildcards with explicit named re-exports that mirror exactly the surface each real module (packages/report/src/{emit-report-schema,report-schema}.ts) always exposed, keeping the module boundaries narrow. Verified: apps/web typecheck clean under verbatimModuleSyntax (proves every re-exported symbol resolves in the barrel — export parity — and all import sites still resolve); @sigma/report 230 tests and @sigma/web 1213 tests pass; @sigma/report already declared in apps/web dependencies; no dangling references to the removed r2-report-object fixture.
…udit The CI "check" job (osv-scanner) failed: undici 7.28.0 is affected by 5 advisories (GHSA-4cwx-7wf7-3272 HIGH + four Medium), all fixed in 7.29.0. Raise the undici floor. This branch carries the security overrides in BOTH pnpm-workspace.yaml and a legacy package.json `pnpm.overrides` block; the package.json block is the one pnpm actually applies here (it appears in the lockfile's overrides), so bump it too — as `^7.29.0` (bounded to the 7.x line, matching #87), not the open `>=7.29.0` selector which pulled undici 8.x. pnpm-workspace.yaml is set to the same `^7.29.0` + comment, mirroring #87 so the sibling branches stay in sync for the umbrella merge. Verified: osv-scanner v2.4.0 `scan source -L pnpm-lock.yaml` → "No issues found" (exit 0); lockfile diff is undici-only (7.28.0 → 7.29.0); frozen install clean; @sigma/etl 81, @sigma/web 1213 tests pass.
…t passes The sync pulled in upstream's coverage ratchet (check-coverage.mjs + coverage-baseline.json), which measures apps/web against a 91% lines floor via v8 include: ['app/**']. The assistant's data corpus under app/** (golden replay fixtures, R2 report fixtures — 34 .json files, ~169 lines) was counted at 0%, sinking apps/web to 89.17%. JSON has no executable lines; excluding it restores the true source coverage (92.58%). Real untested source stays counted.
Resolve conflicts: - packages/db/src/queries/index.ts: keep both weekly + related-persons exports - apps/web/workers/app.integration.test.ts: keep both noindex describe blocks (weeks + conflicts) - apps/web/app/lib/assistant/describe-schema.ts: keep the @sigma/report shim (#167A); carry base's annex_total_suspect value_flag additions into packages/report/src/describe-schema.ts - osv-scanner.toml / pnpm-workspace.yaml: keep the fuller HEAD suppression docs - package.json: drop the stale pnpm.overrides block so pnpm 10 reads overrides from pnpm-workspace.yaml (base's migration); regenerate pnpm-lock.yaml with pnpm 10.33.0
…o 0007 Resolves both blockers from the review of #80. BLOCKER 1 (accuracy) — synthetic orphan contracts (parent tender procedure_type= 'неизвестна', title='(без предмет)', is_synthetic=1 per 0006) can carry a non-NULL amount_eur with value_flag='ok', so the eight weekly.ts indicators were over-counting sums, could surface a '(без предмет)' row as the week's largest, and could make the volume count non-zero — bypassing the zero-row publish gate. Add `is_synthetic != 1` to the shared WEEK_FILTER so all eight queries (and any future one) inherit the guard, matching precompute.sql's sector/authority/company rollups. Correct the top-of-file and getWeeklyCounts comments; rewrite the reconciliation comment — home_totals.value_eur is SUM(amount_eur) over ALL contracts and does NOT itself filter is_synthetic, so the week is a strict subset of it and 'week ≤ home' is a valid loose bound, not an equal basis (the review's 'home_totals additionally filters is_synthetic' was inaccurate for that rollup). Add a discriminating test: a synthetic value_flag='ok' row with the week's highest amount is excluded from total/largest/top, and a synthetic-only week counts 0. BLOCKER 2 (migration collision) — the base advanced to 0006, so 0004_weekly_digests.sql collided with 0004_cpv_division_stats.sql. Renumber to 0007_weekly_digests.sql (git mv; table name unchanged) and fix the ticket's stale reference. Also updates the ticket's pipeline note to drop persistReport({immutable:true}), matching the code (the object is overwritten in place on re-issue, so immutable would stale-serve). Verified: @sigma/db 350 tests, @sigma/etl 81 tests pass; db+etl typecheck clean; prettier clean.
…ase merge Merging the advanced base (feat/ai-assistant-contracts) into feat/weekly-digest renumbered the migration chain — is_synthetic moved to 0012 and the base now reaches 0012 (0007 is taken by 0007_amendment_value_suspect). The weekly_digests migration (previously renumbered 0004→0007 for the review of #80) collided again, so move it to 0013_weekly_digests.sql. Refresh the two now-stale references: weekly.ts's is_synthetic provenance comment (0006 → 0012) and the ticket's migration path (0007 → 0013). No duplicate migration prefixes remain. Verified post-merge: @sigma/db 521 tests, @sigma/etl 89 tests pass; db typecheck clean; osv-scanner exit 0; check:docs ok.
CI 'Coverage ratchet' failed: @sigma/report (new in #167A T1) has a "test" script but no entry in coverage-baseline.json, and no vitest.config, so it emitted no coverage-summary.json and check-coverage.mjs fails closed on the missing baseline key. Add packages/report/vitest.config.ts using the shared sharedCoverage(['src/**']) preset (mirrors packages/shared) so it emits coverage/coverage-summary.json like every other workspace, and add its baseline entry (lines 96.3, branches 86.9 — floored from the measured 96.33/86.93). Verified: full 'pnpm test -- --coverage' + check-coverage.mjs exits 0, report row 96.33%/86.93%, no workspace below baseline.
DiyanaDimitrova
changed the title
DiyanaDimitrova
marked this pull request as draft
August 19, 2026 12:10
DiyanaDimitrova
changed the title
Drop docs/tickets/167a-weekly-digest-producer.md and 167b-weekly-digest-consumer.md (the folder held only these two, so docs/tickets/ goes with them) and their two index entries in docs/README.md. check:docs stays green — no dangling references remain.
Follow-up to 82ad5c3, which deleted the 167a/167b ticket files but (due to a staging slip) landed without the docs/README.md edit that removes their two index links. Remove them now so check:docs has no dangling references.
Drop the midt-bg#167 numeric prefix from the implementation-plan filename; update the single in-repo reference (docs/README.md). git mv preserves history; check:docs stays green.
Remove .github/workflows/preview.yml and preview-reap.yml — the per-PR `sigma-pr-<n>` preview-env deploy + reaper. They are fork-only infrastructure (gated to same-repo branches, keyed to a `preview` GitHub Environment), absent from midt-bg:main and from the #79 branch that targets it. Dropping them so the upstream PR (midt-bg#324) carries the same shape as #79 instead of proposing the fork's preview setup to the main repo. check:docs stays green (the dev-environment docs reference them as inline code, not validated links).
lyubomir-bozhinov
left a comment
lyubomir-bozhinov
left a comment
Collaborator
There was a problem hiding this comment.
Дълбок преглед, с фокус върху CI/деплой границата. Кодът на дайджеста е грижлив; една конкретна находка — тестова/preview настройка, изтекла в комитнатия конфиг:
apps/etl/wrangler.toml:5 — workers_dev = true (на main е false). Конфигът го описва обратно на четири места:
wrangler.toml:7-9: „workers_dev = false above … unreachable without a route/workers_dev";src/index.ts:34:DIGEST_TRIGGER_ENABLED„Committed 'false'" — аwrangler.toml:41комитва"true";src/index.ts:297: „in production this handler is unreachable";src/digest-trigger.ts:3: целият security модел стъпва на „workers_dev=false, no route, so this surface is unreachable in [prod]".
scripts/wrangler-render.mjs не пипа workers_dev (0 срещания), deploy.yml също не — тъй че комитнатите workers_dev=true + DIGEST_TRIGGER_ENABLED="true" тръгват дословно към prod. Ефект: sigma-etl получава публичен *.workers.dev URL и on-demand trigger-ът е ENABLED; остава само DIGEST_TRIGGER_TOKEN (unset → 404). От трите слоя защита (недостижим + флаг off + токен) остава един, а четирите коментара заблуждават бъдещ оператор, че повърхността е недостижима.
Съгласно index.ts:297 („where a preview env opts in") opt-in-ът е замислен per-env, не като комитнат default. Предложение: върни workers_dev = false и DIGEST_TRIGGER_ENABLED = "false" в комита; preview средата да ги вдига през var/render.
Останалото по CI е чисто: ephemeral preview workflow-ите (preview.yml/reap/teardown) НЕ са внесени в upstream; wrangler-render коректно сменя ETL gateway account-а (ред 131-136, SIGMA_AI_GATEWAY_ACCOUNT); двата понеделнишки crons (0 6/0 7) се раутват отделно по event.cron. Дифът е надмножество на #79 — очаквано до сливането му.
Collaborator
|
@DiyanaDimitrova — подсещане за находката от прегледа по-горе: |
Review of midt-bg#324 caught a dev-testing config leaked into the committed wrangler.toml (introduced by 5e760b0 'enable workers.dev route for dev digest-trigger testing'): - workers_dev = true → false (main is false) - DIGEST_TRIGGER_ENABLED = "true" → "false" Neither scripts/wrangler-render.mjs nor deploy.yml rewrites these, so the committed literals went verbatim to prod: sigma-etl would get a public *.workers.dev URL AND the on-demand HTTP trigger would be ENABLED, collapsing the three-layer defence (unreachable + flag off + bearer token) to just the DIGEST_TRIGGER_TOKEN secret. Both values also contradicted four in-code comments that state the surface is committed-false / unreachable (wrangler.toml:7-9, src/index.ts:34 & 297, src/digest-trigger.ts:3). The trigger opt-in is per-env by design (index.ts:297) — a preview env raises these via var/render, they are not a committed default. @sigma/etl 89 tests pass; toml valid.
Contributor
Author
|
Точна находка — оправено в
И двете бяха изтекли от Проверено: Благодаря и за потвърждението по останалото (preview workflow-ите не са внесени upstream, ETL gateway account рендерът, раздалечените понеделнишки crons). |
Contributor
Author
|
@lyubomir-bozhinov — оправено, същата находка от прегледа. В
Потвърдено, че |
Collaborator
|
Рейловете за данни/libel и Major (заварено infra, но deploy.yml — който този PR пипа — е мястото) — Minor — |
…e refresh boundary Two findings from the review of midt-bg#324. MAJOR — deploy.yml never forwarded SIGMA_TURNSTILE_SITE_KEY to the render step. wrangler- render.mjs reads it (:98) and swaps the per-account key over the committed one (:246) only when non-empty, but the job env set just SIGMA_BUILD_ID/ASSISTANT_ENABLED/ENVIRONMENT/ AI_GATEWAY_ACCOUNT — so the value was always '', the swap a no-op, and the committed prod (domain-bound) Turnstile SITE key shipped to every environment. The moment a non-prod env turns on the assistant (SIGMA_ASSISTANT_ENABLED=true), the prod key can't validate on that domain and the bot-gate (useTurnstileGate) silently breaks. Add SIGMA_TURNSTILE_SITE_KEY: ${{ vars.SIGMA_TURNSTILE_SITE_KEY }} alongside the other four (same missing-var class as SIGMA_AI_GATEWAY_ACCOUNT). Verified via render dry-run: set → dev key renders, unset → committed prod key stays. MINOR — PROMPTS_CRON '0 6 * * 1' fired at exactly the Monday 06:00 REFRESH_CRON slot, so scheduled() ran twice and generateSuggestedPrompts raced itself on the same D1 (idempotent per-slot upserts, so wasted compute + last-write-wins refreshed_at, not corruption). Since the 6-hourly refresh already regenerates prompts and PROMPTS_CRON is only the coarse weekly fallback, move it to '5 6 * * 1' — runs just AFTER the Monday refresh (what a fallback should do) and clears every refresh slot (:00). wrangler.toml [triggers].crons updated in lockstep (cron-guard test enforces crons.ts == wrangler.toml). @sigma/etl 89 tests pass (incl. cron-guard); deploy.yml valid; render swap verified.
Contributor
Author
|
И двете оправени в Major — Проверено с render dry-run:
Minor — Проверено: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Реализира #167A — Седмичен дайджест: Производител (конвейер · данни · генериране) по плана в
docs/implementation-plans/167-weekly-digest.md.Какво прави функцията
Това е генериращата част на „Седмицата в пари" — автоматичен седмичен обзор на обществените поръчки. Всеки понеделник ETL работникът:
Дев Б консумира готовия
StoredReport; това PR само го произвежда (SSR маршрутът/weeks/{ISO}и рендерерът са извън обхвата — #167B).Ключово за точността: два гейта се изпълняват преди каквато и да е заявка или разход за модел — уредена седмица и ненулев брой редове. Всяко неудостоверено число се маха, вместо да се пренапише, и никога не се записва.
main, и до сливането на #79 това е надмножествоОгледало е на PR-а във fork-а lyubomir-bozhinov/sigma#80, където дайджестът стъпва върху
feat/ai-assistant-contracts. Този клон не съществува в това хранилище, затова PR-ът е насочен къмmain.Следствие: докато #79 (
feat(ai-assistant): conversational analytic layer (BgGPT)) не се слее вmain, разликата тук е надмножество — включва несляната работа по ai-assistant/contracts (конвейерът emit→bind→validate→verify на@sigma/report, скелето на ETL cron-а, ADR-0007), от която дайджестът зависи, плюс самата седмична добавка отгоре.Препоръчана последователност: първо ревю/сливане на #79, след което това се свежда само до седмичния дайджест. Фокусираната, вече ревюирана разлика живее в #80 във fork-а — ползвайте нея за прегледа; това PR пренася същата работа в
midt-bg/sigma.Какво съдържа (седмичната добавка върху #79)
T1 — извличане на
@sigma/reportcloudflare:*), за да върви един и същ конвейер emit→bind→validate→verify и в уеб worker-а, и в ETL cron-а. Преместен сgit mv(запазена история);@sigma/webреекспортира от него (стеснено, изрични реекспорти, неexport *), така че чат-тестовете минават непроменени — доказателството срещу дрейф.persist.ts:StoredReport/Provenance,persistReport(bucket, key, stored),readStoredReport. R2 идва като инжектиранR2Bucket— без зашити имена на binding.T2 — слой за данни
0013_weekly_digests.sql:weekly_digests(iso_week PK, as_of, refreshed_at, status, total_eur)— само архивен индекс; самият доклад стои в R2 като единствен източник на истина. (Номериран0013след синхронизацията с напредналата база — веригата вече стига до0012.)packages/db/src/queries/weekly.ts: по една функция на показател a–h. Всеки показател изключва синтетичните „сирашки" договори (is_synthetic != 1, през споделенияWEEK_FILTER, като rollup-ите в precompute.sql); парите са сamount_eur IS NOT NULL, сектор презsubstr(cpv_code,1,2), единична оферта наbids_received=1зад праг ≥20 извадка, топ-10 с id-та за връзки.SUM(amount_eur)сhome_totals.value_eurи логва при отклонение (валидна свободна горна граница — седмицата е строго подмножество; никога не приравнява броячи).T3 — ETL cron
DIGEST_CRON = '0 7 * * 1'(понеделник 07:00 UTC, след опреснението в 06:00), добавен в реда, който cron-guard очаква.weekly-digest.ts: котва за свежест → ГЕЙТ 1 уредена седмица (ADR-0007) → ГЕЙТ 2 къс път при нула редове → заявки a–h → сверка → emit блокове → разказ от модела (BgGPT през AI Gateway) →bindReport→ гейтfindProseNumbers→ ограничено регенериране → verifier премахва неудостовереното → все още невалидно ⇒ шаблон без AI →persistReport(…)(безimmutable— обектът се презаписва на място при преиздаване, така чеimmutableби бил капан за остаряло сервиране) → UPSERT → структуриран лог.wrangler.tomlполучава binding-итеAIиREPORTSплюс kill-switch променливата;wrangler-render.mjsе обновен съответно.Тестове
Писани заедно с кода (изискване за TDD по плана):
persist.tsсрещу златната фикстура; ISO-седмичният помощник на W52/W53/W01.value_flag='ok'и най-високата сума в седмицата се изключва от total/largest/top, а седмица само от синтетични редове дава брой 0 (гейтът за нула редове не се заобикаля).put; невалидно след N регенерации ⇒ записва fallback; kill-switch изключен ⇒ не публикува. cron-guard разширен заDIGEST_CRON.Бележки за ревюто
/companies/{ЕИК}; това е грешно за bidder-и по име (name:<name>), затова връзките минават презentityHref/hrefForEntity, а не форматиране на ЕИК на ръка./weeks/{ISO},ReportBlockRenderer,ReportAiWatermark.