diff --git a/docs/adr/0030-legacy-fx-backfill.md b/docs/adr/0030-legacy-fx-backfill.md
new file mode 100644
index 000000000..1db153be1
--- /dev/null
+++ b/docs/adr/0030-legacy-fx-backfill.md
@@ -0,0 +1,77 @@
+# ADR-0030 — Насочен backfill на заварените NULL amount_eur (следствие от #158)
+
+- **Статус:** Прието
+- **Дата:** 2026-07-22
+- **Обхват:** `scripts/backfill-fx.mjs`, `scripts/import.mjs` (slice FX gate), served D1
+
+## Контекст
+
+Преди ADR-0029 cron derive-ът вървеше с празна `fx_rates`: договорите в чужда валута от този
+прозорец седят в served D1 с `amount_eur = NULL` (извън всички суми) и — по-коварно — с грешно
+решени fx-зависими клонове на `value_flag` (`eff_eur` беше NULL, затова `value_suspect`/`review`
+пропадаха до `ok`). Cron поправката спира нови случаи, но не лекува заварените: те са извън
+21-дневния й прозорец.
+
+Алтернативи: (а) пълен CLI re-derive — пълна коректност, но изисква целия raw корпус, повторен
+ingest и часове работа за шепа редове; (б) насочена поправка на място — минути, без staging, но
+дублира derive формули. Избираме (б) с **емпирична гаранция срещу drift**: equivalence тест, който
+derive-ва един и същ корпус С курсове (еталон) и БЕЗ курсове + backfill, и изисква байтова
+идентичност на всяка EUR колона, флаг и rollup таблица.
+
+## Решение
+
+`scripts/backfill-fx.mjs` (report по подразбиране, `--apply` за поправка; същите цели като
+`load-fx.mjs`: `--work-db` / локална / `--remote` D1):
+
+1. **Щета — предикат по коренната причина.** `currency NOT IN ('BGN','EUR') AND fx_rate IS NULL AND
+ value_flag <> 'value_suspect'`: чужд договор без `fx_rate` е точно този, който прозорецът остави
+ безкурсов. `fx_rate IS NULL` е коренът; `amount_eur IS NULL` беше само най-видимият **симптом** и
+ изпуска един клас — чужд договор, чиято водеща стойност е анекс в **EUR** (`current_value_currency
+ = 'EUR'`, #245): `amount_eur` му е коректно попълнено по номинал, но `fx_rate` / `signing_value_eur`
+ са NULL. Предикатът по `fx_rate` лови и него. Плюс **само-флагов** клас: `value_flag` зависи и от
+ валутата на **тендерната оценка** — договор в BGN/EUR с чужда валута на оценката е с коректно
+ `amount_eur`, но с грешно пропаднали `value_suspect`/`review` клонове; открива се чрез
+ преизчисляване на флага с налични курсове и сравнение. Офлайн report-ът брои непроверимите
+ кандидати (`flagUnverified`) и излиза с код 1.
+2. **Курсове:** липсващото покритие (валути на договорите + чужди валути на тендерните оценки,
+ по дата на подписване) се тегли през споделената FX логика от ADR-0029 (`fx.ts`: lookback, URL,
+ валидация, host-pinning) и се upsert-ва в `fx_rates`. Fetch грешка при оставаща дупка → изключение
+ (fail loudly); валута извън ECB → остава NULL, отчита се.
+3. **Поправка на редовете:** `fx_rate` от курса към датата на подписване; преизчисляване на
+ fx-зависимите флагове (`value_suspect` печели над всеки текущ флаг, `review` само от `ok` —
+ същият приоритет като derive-а); `value_suspect` се ремонтира от тендерната оценка (както в
+ derive-а). EUR колоните се преизчисляват през **същата стълбица като derive-а** (`toEur`), а не
+ с наивно `amount × fx_rate`: всяка родна стойност се конвертира по **своята** валута — `amount`
+ по `trusted_currency`, `current_value` по `current_value_currency` (миграция 0002 — EUR анексът
+ по номинал, а не по курса на подписване, #245), `signing_value` по валутата на договора; чуждият
+ клон ползва курса на договора, EUR/BGN клоновете — номинал/фиксинг. `annex_suspect` остава с
+ потиснато `current_value_eur`. BGN/EUR редовете конвергират към номинал/фиксинг (идемпотентно).
+4. **Rollups:** поправените редове пълнят `refresh_touched_*` и се пускат собствените батчове на
+ `refresh-slice.sql` (company/authority totals, flow-pairs, search index, globals, cleanup) —
+ нула дублиран rollup SQL. Поправка + rollups излизат като **един** exec (една
+ wrangler/sqlite3 инвокация), а прекъснат пуск се лекува: останалите `refresh_touched_*`
+ таблици (cleanup ги трие последен) са трайното доказателство кои редове са с изостанали
+ rollups — следващият пуск ги засича в самото начало (преди early return и преди какъвто и да е
+ fetch) и препуска rollup батчовете върху тях. Report режимът също ги отчита и излиза с код 1.
+5. **Gate parity:** `runSliceDerive` в `import.mjs` вече също минава `assertFxPopulated` — и то
+ на същото място като full derive-а: след derive групите и **преди** rollup групите (сплит по
+ споделения `REFRESH_SLICE_ROLLUP_GROUPS`, с drift-тест). Гейт след rollup-ите би гърмял
+ шумно, но върху вече записани повредени суми.
+
+## Последствия
+
+- Заварените редове се лекуват за минути, без raw корпус; идемпотентно (второ пускане: нула
+ fetch-ове, нула промени).
+- **Остатъци (документирани):** (1) клонът на `value_low` „< 1000 € подписана И < 5% от оценката"
+ чете **собствената** оценка на договора (`estimated_value` / `procurement_currency` от raw), която
+ не оцелява в served `contracts` — затова ремонтиран ред, който би бил `value_low`, остава `ok`.
+ За **сумарните** rollup-и това е безобидно (`value_low` се сумира като `ok` — една и съща
+ `trusted_native` формула), **но НЕ и за `cpv_division_stats`**, който филтрира `value_flag = 'ok'`:
+ такъв ред замърсява процентилите на своята CPV дивизия. Backfill-ът брои горната граница на класа
+ (`summary.valueLowUnverifiable` — ремонтиран `ok` ред с чужда валута и подписана стойност < 1000 €,
+ чийто 5%-клон е неразрешим офлайн) и я отчита в summary-то; **пълният CLI re-derive е пътят към
+ абсолютна коректност на процентилите**. (2) ред, който би станал `value_suspect` без тендерна
+ оценка, при derive би отпаднал изцяло (`display_native IS NULL` филтър) — backfill-ът не трие
+ редове; (3) валута извън ECB остава непоправима и се отчита (`remaining` / `flagUnverified`).
+- Equivalence тестът е и drift-аларма: промяна във формулите на `refresh-slice.sql` (напр. #245/
+ #261 по валутата на анексите) ще го счупи шумно и backfill-ът трябва да се пренастрои съзнателно.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 697a92014..b77af6808 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -15,3 +15,4 @@
| [0005](0005-blue-green-d1-rollback.md) | Blue/green D1 слотове за rollback на refresh | Прието |
| [0006](0006-eop-wins-dedup.md) | Dedup на два източника: EOP печели по `contract_number` | Прието |
| [0029](0029-worker-native-fx-load.md) | Worker-native FX зареждане в cron refresh-а и режим на отказ | Прието |
+| [0030](0030-legacy-fx-backfill.md) | Насочен backfill на заварените NULL `amount_eur` | Прието |
diff --git a/packages/db/src/backfill-fx.test.ts b/packages/db/src/backfill-fx.test.ts
new file mode 100644
index 000000000..428a940b6
--- /dev/null
+++ b/packages/db/src/backfill-fx.test.ts
@@ -0,0 +1,552 @@
+///
+// Legacy-NULL FX backfill (#158 follow-up): contracts derived during the bug window (cron ran the
+// derive with an empty fx_rates) sit in the SERVED D1 with amount_eur = NULL — and, subtler, with
+// value_flag mis-assigned (eff_eur was NULL, so value_suspect/review classification fell through
+// to 'ok'). scripts/backfill-fx.mjs repairs the served rows in place and refreshes the rollups
+// through refresh-slice.sql's own touched-scoped batches.
+//
+// The crown test is EQUIVALENCE: ground truth = the real derive WITH rates present; damaged twin =
+// the same corpus derived WITHOUT rates, then backfilled with the same rates. Every EUR column,
+// every flag, and every rollup table must come out identical — that is what makes the backfill's
+// repair SQL drift-proof against scripts/refresh-slice.sql.
+import { readFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { DatabaseSync } from 'node:sqlite';
+import { describe, expect, it, vi } from 'vitest';
+import { backfillFx, repairSql, reportFxDamage } from '../../../scripts/backfill-fx.mjs';
+
+const root = resolve(dirname(fileURLToPath(import.meta.url)), '../../..');
+const initSql = readFileSync(resolve(root, 'packages/db/migrations/0000_init.sql'), 'utf8');
+const flowIdxSql = readFileSync(
+ resolve(root, 'packages/db/migrations/0001_flow_pairs_bidder_index.sql'),
+ 'utf8',
+);
+// refresh-slice.sql reads contracts.current_value_currency (migration 0002) — the served DB the
+// backfill runs against has it applied, so the fixture must too.
+const cvCurrencySql = readFileSync(
+ resolve(root, 'packages/db/migrations/0002_current_value_currency.sql'),
+ 'utf8',
+);
+const stagingSql = readFileSync(resolve(root, 'scripts/work-staging-schema.sql'), 'utf8');
+const refreshSliceSql = readFileSync(resolve(root, 'scripts/refresh-slice.sql'), 'utf8');
+
+const FETCHED_AT = '2026-07-22T00:00:00Z';
+const SIGNED = '2026-05-20';
+// ECB business days only — no rate ON the signing date, the derive carries 05-19 forward.
+const USD_RATES: Record = {
+ '2026-05-16': { EUR: 0.88 },
+ '2026-05-19': { EUR: 0.9 },
+};
+const USD_RATE = 0.9;
+const PEG = 1.95583;
+
+interface Contract {
+ unp: string;
+ num: string;
+ value: number;
+ ccy: string;
+ estimate?: number;
+ estCcy?: string;
+ current?: number;
+ annexCcy?: string;
+}
+
+// One tender + one contract per row; SUSP/REV exercise the flag re-classification the bug window
+// got wrong, ANNEX exercises the amendments/current_value path, CHF stays unpricable end to end.
+// The BGN-REV/BGN-SUSP/ANNEXSUSP trio is the FLAG-ONLY damage class: the contract's own EUR
+// columns derive fine (peg / covered rate), but the TENDER estimate's currency was uncovered, so
+// the value_suspect/review branches mis-resolved — invisible to the amount_eur damage predicate.
+const CORPUS: Contract[] = [
+ { unp: 'UNP-OK', num: 'C-USD-OK', value: 100000, ccy: 'USD' },
+ { unp: 'UNP-BGN', num: 'C-BGN', value: 1000, ccy: 'BGN' },
+ { unp: 'UNP-EUR', num: 'C-EUR', value: 2500, ccy: 'EUR' },
+ {
+ unp: 'UNP-SUSP',
+ num: 'C-USD-SUSP',
+ value: 3000000,
+ ccy: 'USD',
+ estimate: 10000,
+ estCcy: 'BGN',
+ },
+ { unp: 'UNP-REV', num: 'C-USD-REV', value: 60000, ccy: 'USD', estimate: 5000, estCcy: 'EUR' },
+ { unp: 'UNP-CHF', num: 'C-CHF-UNPRICED', value: 5000, ccy: 'CHF' },
+ { unp: 'UNP-ANNEX', num: 'C-USD-ANNEX', value: 10000, ccy: 'USD', current: 12000 },
+ // BGN contract, USD estimate: eff 51,129 € ≥ 10 × 1,800 € → truth 'review', window 'ok'.
+ {
+ unp: 'UNP-BGN-REV',
+ num: 'C-BGN-REV',
+ value: 100000,
+ ccy: 'BGN',
+ estimate: 2000,
+ estCcy: 'USD',
+ },
+ // BGN contract, USD estimate: eff 511,292 € > 200 × 1,800 € → truth value_suspect (amount
+ // repaired to the estimate), window 'ok' — with amount_eur populated all along.
+ {
+ unp: 'UNP-BGN-SUSP',
+ num: 'C-BGN-SUSP',
+ value: 1000000,
+ ccy: 'BGN',
+ estimate: 2000,
+ estCcy: 'USD',
+ },
+ // USD contract ballooned by annex (1,000 → 500,000): truth value_suspect (current-based eff
+ // 450,000 € > 200 × est), window annex_suspect — eff must be current-based, not amount-based.
+ {
+ unp: 'UNP-ANXS',
+ num: 'C-USD-ANNEXSUSP',
+ value: 1000,
+ ccy: 'USD',
+ estimate: 2000,
+ estCcy: 'USD',
+ current: 500000,
+ },
+ // Euro-annex on a FOREIGN contract: USD signing, EUR-denominated annex → current_value_currency
+ // = 'EUR' ≠ contract currency. Probes the euro-annex trap (#245): the derive converts
+ // current_value at par (EUR), NOT via the USD signing rate.
+ {
+ unp: 'UNP-EURANX',
+ num: 'C-USD-EURANX',
+ value: 40000,
+ ccy: 'USD',
+ current: 45000,
+ annexCcy: 'EUR',
+ },
+];
+
+function seedCorpus(db: DatabaseSync): void {
+ db.exec(initSql);
+ db.exec(flowIdxSql);
+ db.exec(cvCurrencySql);
+ db.exec(stagingSql);
+ for (const c of CORPUS) {
+ db.prepare(
+ `INSERT INTO raw_tenders
+ (source, dataset_year, fetched_at, unp, tender_id, procedure_type, procurement_subject,
+ cpv_code, contract_kind, estimated_value, currency, authority_name, authority_eik,
+ authority_type, num_lots, published_at)
+ VALUES ('eop:tenders:2026-06-01', 2026, ?, ?, ?, 'open', 'Backfill tender', '45000000',
+ 'works', ?, ?, 'Authority BF', '123456789', 'public', 1, '2026-05-15')`,
+ ).run(FETCHED_AT, c.unp, `T-${c.unp}`, c.estimate ?? null, c.estCcy ?? 'BGN');
+ db.prepare(
+ `INSERT INTO raw_contracts
+ (source, dataset_year, dataset_variant, fetched_at, needs_enrichment, document_number,
+ published_at, unp, tender_ext_id, procedure_type, procurement_subject, cpv_code,
+ contract_kind, estimated_value, procurement_currency, authority_name, authority_eik,
+ authority_type, contract_number, contract_date, signing_value, currency,
+ contract_subject, contractor_eik, contractor_name, contractor_country, bids_received)
+ VALUES ('eop:contracts:2026-06-01', 2026, 'eop', ?, 0, ?, '2026-06-01', ?, ?, 'open',
+ 'Backfill tender', '45000000', 'works', NULL, NULL, 'Authority BF', '123456789',
+ 'public', ?, ?, ?, ?, 'Backfill contract', '987654321', 'Bidder BF', 'BG', 3)`,
+ ).run(FETCHED_AT, `DOC-${c.num}`, c.unp, `T-${c.unp}`, c.num, SIGNED, c.value, c.ccy);
+ }
+ // Annexes: current_value moves value → current for every row that declares one.
+ for (const c of CORPUS) {
+ if (c.current === undefined) continue;
+ db.prepare(
+ `INSERT INTO raw_amendments
+ (source, dataset_year, dataset_variant, fetched_at, seq_no, document_number,
+ contract_number, contract_date, published_at, unp, authority_eik, authority_name,
+ procurement_subject, contract_kind, value_before, value_after, value_delta, currency,
+ description)
+ VALUES ('eop:annexes:2026-06-01', 2026, 'eop', ?, '1', ?, ?, ?,
+ '2026-06-01', ?, '123456789', 'Authority BF', 'Backfill tender', 'works',
+ ?, ?, ?, ?, 'Increase')`,
+ ).run(
+ FETCHED_AT,
+ `AMD-${c.num}-1`,
+ c.num,
+ SIGNED,
+ c.unp,
+ c.value,
+ c.current,
+ c.current - c.value,
+ c.annexCcy ?? c.ccy,
+ );
+ }
+}
+
+function insertUsdRates(db: DatabaseSync): void {
+ for (const [date, { EUR }] of Object.entries(USD_RATES)) {
+ db.prepare(
+ 'INSERT INTO fx_rates (base_currency, rate_date, eur_per_unit, source, fetched_at) VALUES (?, ?, ?, ?, ?)',
+ ).run('USD', date, EUR, 'ecb:frankfurter', FETCHED_AT);
+ }
+}
+
+function derive(db: DatabaseSync): void {
+ db.exec(refreshSliceSql);
+}
+
+function dropStaging(db: DatabaseSync): void {
+ for (const t of [
+ 'raw_contracts',
+ 'raw_tenders',
+ 'raw_amendments',
+ 'raw_ocds_parties',
+ 'raw_ocds_lots',
+ ]) {
+ db.exec(`DROP TABLE IF EXISTS ${t}`);
+ }
+}
+
+function groundTruthDb(): DatabaseSync {
+ const db = new DatabaseSync(':memory:');
+ seedCorpus(db);
+ insertUsdRates(db);
+ derive(db);
+ return db;
+}
+
+function damagedDb(): DatabaseSync {
+ const db = new DatabaseSync(':memory:');
+ seedCorpus(db);
+ derive(db); // no rates: the bug-window derive
+ dropStaging(db); // the served D1 carries no raw staging
+ return db;
+}
+
+function runnerFor(db: DatabaseSync) {
+ return {
+ query: (sql: string) => db.prepare(sql).all() as Record[],
+ exec: (sql: string) => {
+ db.exec(sql);
+ },
+ };
+}
+
+function fxFetchMock(): typeof fetch {
+ return vi.fn(async (input: string | URL | Request) => {
+ const url = new URL(String(input));
+ if (url.searchParams.get('base') === 'USD') {
+ return new Response(JSON.stringify({ rates: USD_RATES }), { status: 200 });
+ }
+ // CHF (and anything else): frankfurter does not serve it.
+ return new Response(JSON.stringify({ message: 'not found' }), { status: 404 });
+ }) as unknown as typeof fetch;
+}
+
+async function backfill(db: DatabaseSync) {
+ return backfillFx(runnerFor(db), {
+ fetchFn: fxFetchMock(),
+ fetchedAt: FETCHED_AT,
+ refreshSliceSql,
+ });
+}
+
+const CONTRACT_COLS =
+ 'contract_number, amount, currency, value_flag, amount_eur, fx_converted, fx_rate, signing_value_eur, current_value_eur';
+const ROLLUP_QUERIES = {
+ contracts: `SELECT ${CONTRACT_COLS} FROM contracts ORDER BY contract_number`,
+ company_totals:
+ 'SELECT bidder_id, name, won_eur, contracts, authorities, first_date, last_date FROM company_totals ORDER BY bidder_id',
+ authority_totals:
+ 'SELECT authority_id, name, spent_eur, contracts, suppliers, avg_eur, first_date, last_date FROM authority_totals ORDER BY authority_id',
+ flow_pairs:
+ 'SELECT authority_id, bidder_id, won_eur, contracts FROM flow_pairs ORDER BY authority_id, bidder_id',
+ home_totals:
+ 'SELECT contracts, value_eur, authorities, bidders, suspect, first_date, last_date, as_of FROM home_totals',
+ sector_totals: 'SELECT division, contracts, value_eur FROM sector_totals ORDER BY division',
+ facet_counts: 'SELECT facet, key, contracts, value_eur FROM facet_counts ORDER BY facet, key',
+ search_index:
+ "SELECT kind, ref, title, ident, amount FROM search_index WHERE kind IN ('contract','company','authority') ORDER BY kind, ref",
+ data_freshness: 'SELECT source, as_of, rows FROM data_freshness ORDER BY source',
+ // cohort-stats (full-rebuild percentiles over priced 'ok' rows) — the backfill must refresh it
+ // after repair, so the equivalence must cover it.
+ cpv_division_stats:
+ 'SELECT division, priced_contracts, p25_eur, median_eur, p75_eur, p90_eur, p95_eur, p99_eur FROM cpv_division_stats ORDER BY division',
+};
+
+describe('legacy-NULL FX damage (bug-window derive)', () => {
+ it('reproduces: foreign rows derive NULL amount_eur and mis-classify as ok', () => {
+ const db = damagedDb();
+ const rows = db
+ .prepare(
+ "SELECT contract_number, value_flag, amount_eur FROM contracts WHERE currency NOT IN ('BGN','EUR') AND amount_eur IS NULL ORDER BY contract_number",
+ )
+ .all() as { contract_number: string; value_flag: string; amount_eur: number | null }[];
+ expect(rows).toHaveLength(6);
+ for (const r of rows) {
+ expect(r.amount_eur).toBeNull();
+ // The dangerous part: even the 300x-estimate contract classified 'ok' with no rate to
+ // price it (the ballooned-annex row fell one branch further, to annex_suspect).
+ expect(r.value_flag).toBe(r.contract_number === 'C-USD-ANNEXSUSP' ? 'annex_suspect' : 'ok');
+ }
+ const bgn = db
+ .prepare("SELECT amount_eur FROM contracts WHERE contract_number = 'C-BGN'")
+ .get() as { amount_eur: number };
+ expect(bgn.amount_eur).toBeCloseTo(1000 / PEG, 6);
+ // The flag-only damage class: BGN contracts with a USD estimate price fine via the peg but
+ // mis-classify as 'ok' — invisible to any amount_eur-based predicate.
+ for (const num of ['C-BGN-REV', 'C-BGN-SUSP']) {
+ const r = db
+ .prepare('SELECT value_flag, amount_eur FROM contracts WHERE contract_number = ?')
+ .get(num) as { value_flag: string; amount_eur: number | null };
+ expect(r.value_flag).toBe('ok');
+ expect(r.amount_eur).not.toBeNull();
+ }
+ // The euro-annex damage class: a foreign contract whose headline value is a EUR-denominated
+ // annex has amount_eur populated AT PAR (so amount_eur IS NULL misses it), yet still carries
+ // NULL fx_rate / signing_value_eur — caught only by the fx_rate-IS-NULL predicate.
+ const eur = db
+ .prepare(
+ "SELECT amount_eur, fx_rate, signing_value_eur FROM contracts WHERE contract_number = 'C-USD-EURANX'",
+ )
+ .get() as { amount_eur: number; fx_rate: number | null; signing_value_eur: number | null };
+ expect(eur.amount_eur).toBe(45000);
+ expect(eur.fx_rate).toBeNull();
+ expect(eur.signing_value_eur).toBeNull();
+ });
+
+ it('ground truth: the same corpus derived with rates prices and flags correctly', () => {
+ const db = groundTruthDb();
+ const flag = (num: string) =>
+ db
+ .prepare('SELECT value_flag, amount_eur FROM contracts WHERE contract_number = ?')
+ .get(num) as {
+ value_flag: string;
+ amount_eur: number | null;
+ };
+ expect(flag('C-USD-OK')).toEqual({ value_flag: 'ok', amount_eur: 100000 * USD_RATE });
+ expect(flag('C-USD-SUSP').value_flag).toBe('value_suspect');
+ expect(flag('C-USD-SUSP').amount_eur).toBeCloseTo(10000 / PEG, 4);
+ expect(flag('C-USD-REV')).toEqual({ value_flag: 'review', amount_eur: 60000 * USD_RATE });
+ expect(flag('C-USD-ANNEX').amount_eur).toBeCloseTo(12000 * USD_RATE, 6);
+ expect(flag('C-CHF-UNPRICED').amount_eur).toBeNull();
+ // Flag-only class, priced correctly all along but classified via the USD estimate:
+ expect(flag('C-BGN-REV').value_flag).toBe('review');
+ expect(flag('C-BGN-REV').amount_eur).toBeCloseTo(100000 / PEG, 4);
+ expect(flag('C-BGN-SUSP').value_flag).toBe('value_suspect');
+ expect(flag('C-BGN-SUSP').amount_eur).toBeCloseTo(2000 * USD_RATE, 6);
+ expect(flag('C-USD-ANNEXSUSP').value_flag).toBe('value_suspect');
+ expect(flag('C-USD-ANNEXSUSP').amount_eur).toBeCloseTo(2000 * USD_RATE, 6);
+ // Euro-annex: current_value (45000 EUR) converts at par, NOT via the USD signing rate.
+ expect(flag('C-USD-EURANX')).toEqual({ value_flag: 'ok', amount_eur: 45000 });
+ });
+});
+
+describe('backfillFx', () => {
+ it('reports the damage before repairing', () => {
+ const db = damagedDb();
+ const report = reportFxDamage(runnerFor(db));
+ expect(report.total).toBe(7);
+ expect(report.byCurrency).toEqual({ CHF: 1, USD: 6 });
+ // The three flag-only candidates (USD tender estimates, no rate loaded) are unverifiable
+ // offline — surfaced instead of silently ignored.
+ expect(report.flagUnverified).toBe(3);
+ expect(report.interrupted).toBe(false);
+ });
+
+ it('EQUIVALENCE: backfilled damaged DB matches the with-rates derive on every surface', async () => {
+ const truth = groundTruthDb();
+ const db = damagedDb();
+
+ const summary = await backfill(db);
+ expect(summary.repaired).toBe(6);
+ expect(summary.reflagged).toBe(3);
+ expect(summary.remaining.map((r) => r.currency)).toEqual(['CHF']);
+
+ for (const [table, sql] of Object.entries(ROLLUP_QUERIES)) {
+ expect(db.prepare(sql).all(), `table ${table}`).toEqual(truth.prepare(sql).all());
+ }
+ const rates = (d: DatabaseSync) =>
+ d.prepare('SELECT base_currency, rate_date, eur_per_unit FROM fx_rates ORDER BY 1, 2').all();
+ expect(rates(db)).toEqual(rates(truth));
+ });
+
+ it('is idempotent: a second run fetches nothing and changes nothing', async () => {
+ const db = damagedDb();
+ await backfill(db);
+ const before = ROLLUP_QUERIES.contracts ? db.prepare(ROLLUP_QUERIES.contracts).all() : [];
+
+ const fetchFn = fxFetchMock();
+ const summary = await backfillFx(runnerFor(db), {
+ fetchFn,
+ fetchedAt: FETCHED_AT,
+ refreshSliceSql,
+ });
+ // CHF is still damaged (unpricable) → one retry for CHF only; USD is covered, never re-fetched.
+ expect(summary.repaired).toBe(0);
+ const calls = (fetchFn as unknown as ReturnType).mock.calls.map((c) =>
+ new URL(String(c[0])).searchParams.get('base'),
+ );
+ expect(calls).toEqual(['CHF']);
+ expect(db.prepare(ROLLUP_QUERIES.contracts).all()).toEqual(before);
+ });
+
+ it('report-only mode mutates nothing', () => {
+ const db = damagedDb();
+ const before = db.prepare(ROLLUP_QUERIES.contracts).all();
+ reportFxDamage(runnerFor(db));
+ expect(db.prepare(ROLLUP_QUERIES.contracts).all()).toEqual(before);
+ });
+
+ it('fails loudly when the rate fetch fails while damage remains', async () => {
+ const db = damagedDb();
+ const fetchFn = vi.fn(async () => {
+ throw new Error('network down');
+ }) as unknown as typeof fetch;
+ await expect(
+ backfillFx(runnerFor(db), { fetchFn, fetchedAt: FETCHED_AT, refreshSliceSql }),
+ ).rejects.toThrow(/network down|fx/i);
+ });
+
+ it('resumes an interrupted run: heals stale rollups from the leftover touched tables', async () => {
+ const truth = groundTruthDb();
+ // The interrupted state: rates loaded and the row repair committed, but the process died
+ // before ANY rollup group ran — refresh_touched_* linger, rollups still hold bug-window sums.
+ const db = damagedDb();
+ insertUsdRates(db);
+ db.exec(repairSql());
+
+ const report = reportFxDamage(runnerFor(db));
+ expect(report.interrupted).toBe(true);
+ // The repaired USD rows no longer match the damage predicate — without the leftover-table
+ // heal, their rollup contribution would be unrecoverable (the HIGH from the security review).
+ expect(report.byCurrency).toEqual({ CHF: 1 });
+
+ const summary = await backfill(db);
+ expect(summary.healed).toBe(true);
+ for (const [table, sql] of Object.entries(ROLLUP_QUERIES)) {
+ expect(db.prepare(sql).all(), `table ${table}`).toEqual(truth.prepare(sql).all());
+ }
+ // Fully healed: touched tables are gone again.
+ expect(reportFxDamage(runnerFor(db)).interrupted).toBe(false);
+ });
+
+ it('heals an interrupted run even when no damaged rows remain (early-return path)', async () => {
+ // With a CHF rate present the interrupted run repaired EVERY row — the next run sees zero
+ // damage, and before the fix would have returned early leaving the rollups stale forever.
+ const chfRate = (d: DatabaseSync) =>
+ d
+ .prepare(
+ 'INSERT INTO fx_rates (base_currency, rate_date, eur_per_unit, source, fetched_at) VALUES (?, ?, ?, ?, ?)',
+ )
+ .run('CHF', '2026-05-19', 1.05, 'ecb:frankfurter', FETCHED_AT);
+
+ const truth = new DatabaseSync(':memory:');
+ seedCorpus(truth);
+ insertUsdRates(truth);
+ chfRate(truth);
+ derive(truth);
+
+ const db = damagedDb();
+ insertUsdRates(db);
+ chfRate(db);
+ db.exec(repairSql()); // interrupted before any rollup group
+
+ const report = reportFxDamage(runnerFor(db));
+ expect(report.total).toBe(0);
+ expect(report.interrupted).toBe(true);
+
+ const fetchFn = fxFetchMock();
+ const summary = await backfillFx(runnerFor(db), {
+ fetchFn,
+ fetchedAt: FETCHED_AT,
+ refreshSliceSql,
+ });
+ expect(summary.healed).toBe(true);
+ expect(summary.repaired).toBe(0);
+ expect((fetchFn as unknown as ReturnType).mock.calls.length).toBe(0);
+ for (const [table, sql] of Object.entries(ROLLUP_QUERIES)) {
+ expect(db.prepare(sql).all(), `table ${table}`).toEqual(truth.prepare(sql).all());
+ }
+ expect(reportFxDamage(runnerFor(db)).interrupted).toBe(false);
+ });
+
+ // The documented value_low blind spot (ADR-0030 residual): the derive's value_low "< 1000 EUR
+ // signed AND < 5% of the estimate" branch reads the contract's OWN estimated_value /
+ // procurement_currency from raw staging, which the served `contracts` table does not carry. A
+ // repaired tiny foreign row that should be value_low stays 'ok' — neutral for SUM rollups but NOT
+ // for cpv_division_stats (filters value_flag='ok'). This test PINS that boundary: the backfill
+ // must surface the class (summary.valueLowUnverifiable) rather than hide it, and the known
+ // percentile divergence from a full re-derive is asserted so any future change is deliberate.
+ it('surfaces the value_low blind spot and pins its cpv_division_stats divergence', async () => {
+ // Two USD works contracts in CPV division 45: one tiny (500 USD, own estimate 100000 EUR →
+ // truth value_low, excluded from percentiles) and one normal (50000 USD → ok, counted).
+ const seedTwo = (db: DatabaseSync, withRates: boolean) => {
+ db.exec(initSql);
+ db.exec(flowIdxSql);
+ db.exec(cvCurrencySql);
+ db.exec(stagingSql);
+ const insert = (num: string, signing: number, estOwn: number | null) => {
+ db.prepare(
+ `INSERT INTO raw_tenders
+ (source, dataset_year, fetched_at, unp, tender_id, procedure_type, procurement_subject,
+ cpv_code, contract_kind, estimated_value, currency, authority_name, authority_eik,
+ authority_type, num_lots, published_at)
+ VALUES ('eop:tenders:2026-06-01', 2026, ?, ?, ?, 'open', 'T', '45000000', 'works', NULL,
+ 'BGN', 'A', '123456789', 'public', 1, '2026-05-15')`,
+ ).run(FETCHED_AT, `U-${num}`, `T-${num}`);
+ db.prepare(
+ `INSERT INTO raw_contracts
+ (source, dataset_year, dataset_variant, fetched_at, needs_enrichment, document_number,
+ published_at, unp, tender_ext_id, procedure_type, procurement_subject, cpv_code,
+ contract_kind, estimated_value, procurement_currency, authority_name, authority_eik,
+ authority_type, contract_number, contract_date, signing_value, currency,
+ contract_subject, contractor_eik, contractor_name, contractor_country, bids_received)
+ VALUES ('eop:contracts:2026-06-01', 2026, 'eop', ?, 0, ?, '2026-06-01', ?, ?, 'open',
+ 'T', '45000000', 'works', ?, ?, 'A', '123456789', 'public', ?, ?, ?, 'USD',
+ 'C', '987654321', 'B', 'BG', 3)`,
+ ).run(
+ FETCHED_AT,
+ `DOC-${num}`,
+ `U-${num}`,
+ `T-${num}`,
+ estOwn,
+ estOwn ? 'EUR' : null,
+ num,
+ SIGNED,
+ signing,
+ );
+ };
+ insert('C-USD-LOW', 500, 100000);
+ insert('C-USD-BIG', 50000, null);
+ if (withRates) insertUsdRates(db);
+ derive(db);
+ };
+
+ const truth = new DatabaseSync(':memory:');
+ seedTwo(truth, true);
+ const db = new DatabaseSync(':memory:');
+ seedTwo(db, false);
+ dropStaging(db);
+
+ const summary = await backfill(db);
+
+ // Priced correctly: the tiny row's amount_eur is right (500 × 0.9 = 450).
+ const low = db
+ .prepare("SELECT amount_eur, value_flag FROM contracts WHERE contract_number = 'C-USD-LOW'")
+ .get() as { amount_eur: number; value_flag: string };
+ expect(low.amount_eur).toBeCloseTo(450, 6);
+ // But the flag cannot be recomputed to value_low offline — it stays 'ok' (the residual)…
+ expect(low.value_flag).toBe('ok');
+ expect(
+ truth.prepare("SELECT value_flag FROM contracts WHERE contract_number = 'C-USD-LOW'").get(),
+ ).toEqual({ value_flag: 'value_low' });
+ // …and the backfill SURFACES it rather than hiding it.
+ expect(summary.valueLowUnverifiable).toBe(1);
+
+ // The pinned consequence: cpv_division_stats diverges from a full re-derive — the backfill
+ // counts the mislabeled row (2 priced, median dragged down), truth excludes it (1 priced).
+ const cs = (d: DatabaseSync) =>
+ d.prepare('SELECT division, priced_contracts, median_eur FROM cpv_division_stats').get();
+ expect(cs(truth)).toEqual({
+ division: '45',
+ priced_contracts: 1,
+ median_eur: 50000 * USD_RATE,
+ });
+ expect(cs(db)).toEqual({ division: '45', priced_contracts: 2, median_eur: 450 });
+ // Full re-derive is the path to absolute percentile correctness (ADR-0030).
+ });
+
+ it('leaves flag-clean BGN/EUR rows and their rollup contributions untouched', async () => {
+ const db = damagedDb();
+ const sql =
+ "SELECT contract_number, amount_eur, value_flag FROM contracts WHERE contract_number IN ('C-BGN','C-EUR') ORDER BY 1";
+ const before = db.prepare(sql).all();
+ await backfill(db);
+ expect(db.prepare(sql).all()).toEqual(before);
+ });
+});
diff --git a/packages/ingest/src/refresh.test.ts b/packages/ingest/src/refresh.test.ts
new file mode 100644
index 000000000..253a90bdb
--- /dev/null
+++ b/packages/ingest/src/refresh.test.ts
@@ -0,0 +1,27 @@
+// Drift alarm for REFRESH_SLICE_ROLLUP_GROUPS (#158): scripts/import.mjs splits the slice derive
+// on this list to gate FX between "rows derived" and "rollups written", and backfill-fx.mjs
+// re-runs exactly these groups over a touched set. A renamed or re-ordered @refresh-batch in
+// refresh-slice.sql must fail HERE, not silently skip a rollup.
+import { readFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+import { REFRESH_SLICE_ROLLUP_GROUPS, refreshSliceStatementGroups } from './refresh';
+
+const refreshSliceSql = readFileSync(
+ resolve(dirname(fileURLToPath(import.meta.url)), '../../../scripts/refresh-slice.sql'),
+ 'utf8',
+);
+
+describe('REFRESH_SLICE_ROLLUP_GROUPS', () => {
+ it('is the contiguous tail of refresh-slice.sql, in file order', () => {
+ const names = refreshSliceStatementGroups(refreshSliceSql).map((g) => g.name);
+ expect(names.slice(-REFRESH_SLICE_ROLLUP_GROUPS.length)).toEqual([
+ ...REFRESH_SLICE_ROLLUP_GROUPS,
+ ]);
+ // …and none of them leaks into the derive half of the split.
+ for (const name of names.slice(0, -REFRESH_SLICE_ROLLUP_GROUPS.length)) {
+ expect(REFRESH_SLICE_ROLLUP_GROUPS).not.toContain(name);
+ }
+ });
+});
diff --git a/packages/ingest/src/refresh.ts b/packages/ingest/src/refresh.ts
index 4159da90e..d0a2144d1 100644
--- a/packages/ingest/src/refresh.ts
+++ b/packages/ingest/src/refresh.ts
@@ -49,6 +49,21 @@ export interface RefreshSliceStatementGroup {
statements: string[];
}
+/** refresh-slice.sql's rollup groups (file order, contiguous tail of the file): everything that
+ * aggregates the served rows. Callers that must gate between "rows derived" and "rollups written"
+ * (the FX gate in scripts/import.mjs, #158) or re-run only the rollups over a touched set
+ * (scripts/backfill-fx.mjs) split on this list. */
+export const REFRESH_SLICE_ROLLUP_GROUPS = [
+ 'company-totals',
+ 'authority-totals',
+ 'flow-pairs',
+ 'entity-search-index',
+ 'contract-search-index',
+ 'globals',
+ 'cohort-stats',
+ 'cleanup',
+] as const;
+
const REFRESH_BATCH_MARKER = /^--\s*@refresh-batch\s+([a-z0-9][a-z0-9-]*)\s*$/i;
/**
diff --git a/scripts/backfill-fx.d.mts b/scripts/backfill-fx.d.mts
new file mode 100644
index 000000000..b3cd15f1c
--- /dev/null
+++ b/scripts/backfill-fx.d.mts
@@ -0,0 +1,68 @@
+// Type surface for the legacy-NULL FX backfill (#158 follow-up). The implementation is plain ESM
+// (.mjs) because it runs directly under `node` with no build step; this declaration gives the
+// TypeScript equivalence suite (packages/db/src/backfill-fx.test.ts) real types.
+
+/** A read/write runner over the served DB: `query` returns rows, `exec` applies a script. Backed
+ * by the sqlite3 CLI or wrangler d1 in the CLI, and by node:sqlite in tests. */
+export interface FxRunner {
+ query: (sql: string) => Array>;
+ exec: (sql: string) => void;
+}
+
+export interface FxDamageReport {
+ /** damaged rows: foreign currency, NULL amount_eur, not value_suspect */
+ total: number;
+ byCurrency: Record;
+ rows: Array>;
+ /** leftover refresh_touched_* tables: a prior repair/refresh died before its rollup refresh */
+ interrupted: boolean;
+ /** rows whose value_flag depends on a foreign tender-estimate rate that is not loaded — the
+ * flag cannot be verified offline; --apply loads the rates and recomputes */
+ flagUnverified: number;
+}
+
+export interface FxFetchOutcome {
+ currency: string;
+ start?: string;
+ end?: string;
+ /** rates inserted for this currency */
+ loaded?: number;
+ status: 'ok' | 'unsupported' | 'invalid' | 'error';
+ detail?: string;
+}
+
+export interface FxBackfillSummary {
+ repaired: number;
+ /** flag-only repairs: rows whose recomputed value_flag differed (any contract currency) */
+ reflagged: number;
+ /** repaired rows priced to 'ok' with a tiny (< 1000 EUR) signing value whose value_low
+ * 5%-of-estimate classification is undecidable offline (the raw estimate is not served) — an
+ * upper bound on rows that may skew cpv_division_stats percentiles (ADR-0030 residual) */
+ valueLowUnverifiable: number;
+ /** rows still unpriced after the repair (no usable ECB rate) */
+ remaining: Array<{
+ id: unknown;
+ contract_number: unknown;
+ currency: unknown;
+ signed_at: unknown;
+ }>;
+ fetched: FxFetchOutcome[];
+ /** true when a prior interrupted run's stale rollups were healed first */
+ healed: boolean;
+ before: FxDamageReport;
+}
+
+export function reportFxDamage(runner: FxRunner): FxDamageReport;
+
+/** The row-repair script (rates → flags → EUR columns); exported for the interruption tests. */
+export function repairSql(): string;
+
+export function backfillFx(
+ runner: FxRunner,
+ options: {
+ fetchFn?: typeof fetch;
+ fetchedAt: string;
+ refreshSliceSql: string;
+ api?: string;
+ },
+): Promise;
diff --git a/scripts/backfill-fx.mjs b/scripts/backfill-fx.mjs
new file mode 100644
index 000000000..32068f124
--- /dev/null
+++ b/scripts/backfill-fx.mjs
@@ -0,0 +1,513 @@
+#!/usr/bin/env node
+// Legacy-NULL FX backfill (#158 follow-up). Contracts derived while fx_rates was empty (the cron
+// bug window) sit in the SERVED D1 with amount_eur = NULL — silently missing from every rollup —
+// and with the fx-dependent value_flag branches (value_suspect / review) mis-resolved to 'ok'.
+// This repairs the served rows in place: load the missing ECB rates (shared fx.ts logic), recompute
+// fx_rate / flags / EUR columns for exactly the damaged rows, then refresh the rollups through
+// refresh-slice.sql's own touched-scoped batches — no staging, no re-ingest, no duplicated rollup
+// SQL. See docs/adr/0030-legacy-fx-backfill.md.
+//
+// node scripts/backfill-fx.mjs # report damage on local D1, exit 1 if any
+// node scripts/backfill-fx.mjs --apply # repair local D1
+// node scripts/backfill-fx.mjs --apply --remote # repair the served D1
+// node scripts/backfill-fx.mjs --work-db=data/work/sigma.db [--apply]
+import { execFileSync } from 'node:child_process';
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { resolve, dirname } from 'node:path';
+import { fileURLToPath, pathToFileURL } from 'node:url';
+import {
+ FX_LOOKBACK_DAYS,
+ FX_SOURCE,
+ addDays,
+ assertSameFinalHost,
+ fxSeriesUrl,
+ isCurrencyCode,
+ isIsoDate,
+ parseFxSeries,
+} from '../packages/ingest/src/fx.ts';
+import {
+ refreshSliceStatementGroups,
+ REFRESH_SLICE_ROLLUP_GROUPS,
+} from '../packages/ingest/src/refresh.ts';
+
+const root = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+const PEG = 1.95583;
+
+const stripControls = (s) => String(s).replace(/[\x00-\x1F]/g, '');
+const sqlStr = (s) => (s == null ? 'NULL' : `'${stripControls(s).replace(/'/g, "''")}'`);
+
+// The damage predicate: a foreign-currency contract with no fx_rate is one the cron bug window left
+// un-priced. `fx_rate IS NULL` is the ROOT cause; `amount_eur IS NULL` was only its most visible
+// symptom — a foreign contract whose headline value is a EUR-denominated annex has amount_eur
+// populated at par yet still carries NULL fx_rate / signing_value_eur (the euro-annex case, #245),
+// and must be repaired too. value_suspect rows are repaired from the procedure estimate, not FX.
+const damage = (p = '') =>
+ `${p}currency NOT IN ('BGN','EUR') AND ${p}fx_rate IS NULL AND ${p}value_flag <> 'value_suspect'`;
+const DAMAGE = damage();
+
+// A damaged row the loaded rates can actually price: a fx_rate exists for its currency within the
+// lookback of its signing date. Distinguishes repairable (rate loaded) from ECB-unpriceable (CHF).
+const PRICEABLE = `EXISTS (
+ SELECT 1 FROM fx_rates f
+ WHERE f.base_currency = contracts.currency
+ AND f.rate_date <= contracts.signed_at
+ AND f.rate_date >= date(contracts.signed_at, '-${FX_LOOKBACK_DAYS} days')
+ )`;
+
+// Latest usable rate for `ccyExpr` at `dateExpr` — keep in sync with the fx_rate subqueries in
+// scripts/refresh-slice.sql (same bounds, FX_LOOKBACK_DAYS carry-forward).
+const rateAt = (dateExpr, ccyExpr) =>
+ `(SELECT f.eur_per_unit FROM fx_rates f
+ WHERE f.base_currency = ${ccyExpr}
+ AND f.rate_date <= ${dateExpr}
+ AND f.rate_date >= date(${dateExpr}, '-${FX_LOOKBACK_DAYS} days')
+ ORDER BY f.rate_date DESC LIMIT 1)`;
+
+// Tender estimate in EUR at the contract's signing date — keep in sync with the proc_est_eur CASE
+// in scripts/refresh-slice.sql.
+const procEstEur = `(SELECT CASE
+ WHEN t.estimated_value IS NULL THEN NULL
+ WHEN COALESCE(NULLIF(t.currency, ''), 'BGN') = 'EUR' THEN t.estimated_value
+ WHEN COALESCE(NULLIF(t.currency, ''), 'BGN') = 'BGN' THEN t.estimated_value / ${PEG}
+ ELSE t.estimated_value * ${rateAt('contracts.signed_at', `NULLIF(t.currency, '')`)}
+ END FROM tenders t WHERE t.id = contracts.tender_id)`;
+
+const procEstNative = `(SELECT t.estimated_value FROM tenders t WHERE t.id = contracts.tender_id)`;
+
+// The contract's own signing-date rate. current_value may be denominated in a DIFFERENT currency
+// than the contract (current_value_currency, migration 0002 — e.g. a 2026 EUR annex on a foreign
+// contract), but the derive still converts the foreign branch at the CONTRACT's rate, switching
+// only the EUR/BGN par/peg branches by the value's own currency. Keep this ladder byte-identical to
+// refresh-slice.sql's amount_eur / current_value_eur / signing_value_eur / eff_eur CASEs (#245).
+const contractRate = rateAt('contracts.signed_at', `NULLIF(contracts.currency, '')`);
+const contractCurrency = `COALESCE(NULLIF(contracts.currency, ''), 'BGN')`;
+const cvCurrency = `COALESCE(NULLIF(contracts.current_value_currency, ''), NULLIF(contracts.currency, ''), 'BGN')`;
+const toEur = (nativeExpr, ccyExpr) => `CASE
+ WHEN ${ccyExpr} = 'EUR' THEN ${nativeExpr}
+ WHEN ${ccyExpr} = 'BGN' THEN ${nativeExpr} / ${PEG}
+ WHEN ${contractRate} IS NOT NULL THEN ${nativeExpr} * ${contractRate}
+ ELSE NULL
+ END`;
+
+// The currency of the trusted (headline) native value — current_value's currency when present,
+// else the contract's; annex_suspect trusts the signing side instead. Mirrors refresh-slice.sql's
+// trusted_currency. Evaluated AFTER step 3 reclassifies value_flag.
+const trustedCurrency = `CASE
+ WHEN contracts.value_flag = 'annex_suspect' THEN
+ CASE WHEN contracts.signing_value IS NOT NULL THEN ${contractCurrency} ELSE ${cvCurrency} END
+ ELSE
+ CASE WHEN contracts.current_value IS NOT NULL THEN ${cvCurrency} ELSE ${contractCurrency} END
+ END`;
+
+// The derive's effective value in EUR (refresh-slice.sql's eff_eur): current_value at its own
+// currency when present, else signing_value at the contract currency.
+const EFF = `(CASE
+ WHEN contracts.current_value IS NOT NULL THEN ${toEur('contracts.current_value', cvCurrency)}
+ ELSE ${toEur('contracts.signing_value', contractCurrency)}
+ END)`;
+
+// The fx-dependent part of the derive's value_flag CASE (same precedence: value_suspect wins over
+// any current flag, 'review' only upgrades 'ok' — the branches between them don't depend on FX
+// and are already encoded in the current flag). Used both to DETECT flag-only damage and to
+// repair it, so the two can never disagree.
+const NEW_FLAG = `CASE
+ WHEN ${EFF} > 2000000000 OR (${procEstEur} >= 1000 AND ${EFF} > 200 * ${procEstEur}) THEN 'value_suspect'
+ WHEN contracts.value_flag = 'ok' AND ${procEstEur} > 0 AND ${EFF} >= 10 * ${procEstEur} THEN 'review'
+ ELSE contracts.value_flag END`;
+
+// Flag-only damage: the value_flag CASE also depends on the TENDER estimate's currency — a BGN/EUR
+// (or priced foreign) contract whose tender estimate is foreign got its value_suspect/review
+// branches mis-resolved when that rate was missing, with amount_eur perfectly populated. Detected
+// by recomputing the flag with rates present and comparing.
+const FLAG_CANDIDATE = `contracts.value_flag <> 'value_suspect'
+ AND contracts.signed_at IS NOT NULL
+ AND (SELECT t.estimated_value IS NOT NULL
+ AND COALESCE(NULLIF(t.currency, ''), 'BGN') NOT IN ('BGN','EUR')
+ FROM tenders t WHERE t.id = contracts.tender_id)`;
+
+// Rollup batches to re-run after the repair — refresh-slice.sql's own statements, scoped by the
+// refresh_touched_* tables we fill from the repaired set. Order matters (file order); the list
+// itself lives next to the parser (packages/ingest/src/refresh.ts) and is shared with import.mjs.
+const ROLLUP_GROUPS = REFRESH_SLICE_ROLLUP_GROUPS;
+
+/** Damaged rows in the served DB: foreign currency, NULL amount_eur, not value_suspect.
+ * `interrupted` — a prior repair/refresh committed its row updates but died before finishing the
+ * rollup refresh: the refresh_touched_* tables it left behind (cleanup drops them last) still
+ * scope exactly the rows whose rollups are stale. */
+export function reportFxDamage(runner) {
+ const rows = runner.query(
+ `SELECT c.id, c.contract_number, c.currency, c.signed_at, c.amount, c.value_flag
+ FROM contracts c WHERE ${damage('c.')}
+ ORDER BY c.currency, c.signed_at, c.contract_number`,
+ );
+ const byCurrency = {};
+ for (const r of rows) byCurrency[r.currency] = (byCurrency[r.currency] ?? 0) + 1;
+ const interrupted =
+ runner.query(
+ `SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'refresh_touched_contracts'`,
+ ).length > 0;
+ // Flag candidates whose tender-estimate rate is missing: their value_flag cannot be verified
+ // offline — only counted here; --apply loads the rates and recomputes.
+ const flagUnverified = Number(
+ runner.query(
+ `SELECT COUNT(*) AS n FROM contracts
+ WHERE ${FLAG_CANDIDATE}
+ AND NOT EXISTS (
+ SELECT 1 FROM fx_rates f
+ WHERE f.base_currency = (SELECT NULLIF(t.currency, '') FROM tenders t WHERE t.id = contracts.tender_id)
+ AND f.rate_date <= contracts.signed_at
+ AND f.rate_date >= date(contracts.signed_at, '-${FX_LOOKBACK_DAYS} days')
+ )`,
+ )[0]?.n ?? 0,
+ );
+ return { total: rows.length, byCurrency, rows, interrupted, flagUnverified };
+}
+
+// Rate coverage gaps: contract currencies for the damage set, plus foreign tender-estimate
+// currencies for EVERY flag candidate (the value_suspect / review re-classification needs them
+// whatever the contract's own currency is), per signing date.
+function coverageGaps(runner) {
+ return runner.query(
+ `WITH needs (currency, d) AS (
+ SELECT c.currency, c.signed_at FROM contracts c
+ WHERE ${damage('c.')} AND c.signed_at IS NOT NULL
+ UNION
+ SELECT (SELECT NULLIF(t.currency, '') FROM tenders t WHERE t.id = contracts.tender_id),
+ contracts.signed_at
+ FROM contracts WHERE ${FLAG_CANDIDATE}
+ )
+ SELECT currency, MIN(d) AS min_date, MAX(d) AS max_date, COUNT(DISTINCT d) AS missing_dates
+ FROM needs n
+ WHERE n.currency IS NOT NULL AND NOT EXISTS (
+ SELECT 1 FROM fx_rates f
+ WHERE f.base_currency = n.currency AND f.rate_date <= n.d
+ AND f.rate_date >= date(n.d, '-${FX_LOOKBACK_DAYS} days')
+ )
+ GROUP BY currency ORDER BY currency`,
+ );
+}
+
+async function loadMissingRates(runner, { fetchFn, fetchedAt, api }) {
+ const gaps = coverageGaps(runner);
+ const fetched = [];
+ const failures = [];
+ for (const gap of gaps) {
+ const c = String(gap.currency);
+ if (!isCurrencyCode(c) || !isIsoDate(gap.min_date) || !isIsoDate(gap.max_date)) {
+ fetched.push({ currency: c, status: 'invalid' });
+ continue;
+ }
+ const start = addDays(String(gap.min_date), -FX_LOOKBACK_DAYS);
+ const end = String(gap.max_date);
+ try {
+ const url = fxSeriesUrl(c, start, end, api);
+ const res = await fetchFn(url);
+ assertSameFinalHost(url, res.url);
+ if (res.status === 404) {
+ fetched.push({ currency: c, start, end, status: 'unsupported' });
+ continue;
+ }
+ if (!res.ok) throw new Error(`HTTP ${res.status}`);
+ const { rows, warnings } = parseFxSeries(await res.json(), c, `${start}..${end}`);
+ for (const w of warnings) console.warn(` ! ${w}`);
+ if (rows.length > 0) {
+ const values = rows
+ .map(
+ (r) =>
+ `(${sqlStr(r.currency)}, ${sqlStr(r.rateDate)}, ${r.eurPerUnit}, ${sqlStr(FX_SOURCE)}, ${sqlStr(fetchedAt)})`,
+ )
+ .join(',\n ');
+ runner.exec(
+ `INSERT OR REPLACE INTO fx_rates (base_currency, rate_date, eur_per_unit, source, fetched_at) VALUES\n ${values};`,
+ );
+ }
+ fetched.push({ currency: c, start, end, loaded: rows.length, status: 'ok' });
+ } catch (error) {
+ const detail = error instanceof Error ? error.message : String(error);
+ fetched.push({ currency: c, start, end, status: 'error', detail });
+ failures.push(`${c}: ${detail}`);
+ }
+ }
+ if (failures.length > 0) {
+ throw new Error(`fx backfill: rate fetch failed — ${failures.join('; ')}`);
+ }
+ return fetched;
+}
+
+// The row repair over an existing refresh_touched_contracts set. Idempotent over correctly
+// derived rows (recomputing with the same inputs converges), so a resumed run can safely re-apply
+// it to a leftover touched set.
+function repairRowsSql() {
+ return [
+ // Re-resolve the fx-dependent value_flag branches the bug window could not evaluate —
+ // NEW_FLAG mirrors the value_flag CASE in scripts/refresh-slice.sql. The value_low
+ // 5%-of-estimate branch needs raw-staging columns (the contract's own estimated_value /
+ // procurement_currency) that do not survive to the served rows, so a repaired row that should be
+ // value_low stays 'ok'. Neutral for SUM rollups (value_low is summed like 'ok'), but NOT for
+ // cpv_division_stats, which filters value_flag='ok' — such a row pollutes its CPV percentiles.
+ // Surfaced via summary.valueLowUnverifiable; full re-derive is the fix (ADR-0030).
+ `UPDATE contracts SET value_flag = (${NEW_FLAG})
+ WHERE id IN (SELECT id FROM refresh_touched_contracts);`,
+
+ // Rows classified value_suspect: repaired from the procedure estimate, exactly like the
+ // derive (amount := proc_est_native, amount_eur := proc_est_eur, timeline columns NULL).
+ `UPDATE contracts SET
+ amount = COALESCE(${procEstNative}, amount),
+ amount_eur = ${procEstEur},
+ signing_value_eur = NULL,
+ current_value_eur = NULL
+ WHERE id IN (SELECT id FROM refresh_touched_contracts) AND value_flag = 'value_suspect';`,
+
+ // Remaining rows: recompute the EUR columns from the served native values via the derive's own
+ // per-value currency ladder (toEur). amount == display_native == trusted_native for these rows,
+ // so amount_eur converts at trusted_currency; current_value at its own currency (the euro-annex
+ // case, #245); signing at the contract currency. BGN/EUR rows converge to par/peg and any
+ // flag-only BGN/EUR repair recomputes to its existing (correct) value — idempotent.
+ `UPDATE contracts SET
+ amount_eur = ${toEur('contracts.amount', trustedCurrency)},
+ signing_value_eur = CASE WHEN contracts.signing_value IS NULL THEN NULL
+ ELSE ${toEur('contracts.signing_value', contractCurrency)} END,
+ current_value_eur = CASE WHEN contracts.value_flag = 'annex_suspect' OR contracts.current_value IS NULL THEN NULL
+ ELSE ${toEur('contracts.current_value', cvCurrency)} END
+ WHERE id IN (SELECT id FROM refresh_touched_contracts) AND value_flag <> 'value_suspect';`,
+ ].join('\n');
+}
+
+export function repairSql() {
+ return [
+ // 1. Capture every row we will repair — the PRICEABLE damage set (a rate now exists for its
+ // currency+date) AND the flag-only candidates whose recomputed flag differs — plus their
+ // entities, for the rollup refresh (refresh-slice.sql's own touched-table mechanism, see its
+ // setup batch). Captured BEFORE step 2 sets fx_rate, since the damage predicate keys on
+ // fx_rate IS NULL. ECB-unpriceable rows (no rate) are excluded and stay in `remaining`.
+ `DROP TABLE IF EXISTS refresh_touched_contracts;`,
+ `DROP TABLE IF EXISTS refresh_touched_bidders;`,
+ `DROP TABLE IF EXISTS refresh_touched_authorities;`,
+ `CREATE TABLE refresh_touched_contracts (id TEXT PRIMARY KEY);`,
+ `CREATE TABLE refresh_touched_bidders (bidder_id TEXT PRIMARY KEY);`,
+ `CREATE TABLE refresh_touched_authorities (authority_id TEXT PRIMARY KEY);`,
+ `INSERT INTO refresh_touched_contracts SELECT id FROM contracts WHERE ${DAMAGE} AND signed_at IS NOT NULL AND ${PRICEABLE};`,
+ `INSERT OR IGNORE INTO refresh_touched_contracts
+ SELECT id FROM contracts WHERE ${FLAG_CANDIDATE} AND (${NEW_FLAG}) <> contracts.value_flag;`,
+ `INSERT INTO refresh_touched_bidders SELECT DISTINCT bidder_id FROM contracts WHERE id IN (SELECT id FROM refresh_touched_contracts);`,
+ `INSERT INTO refresh_touched_authorities SELECT DISTINCT t.authority_id FROM contracts c JOIN tenders t ON t.id = c.tender_id WHERE c.id IN (SELECT id FROM refresh_touched_contracts);`,
+
+ // 2. Price the touched foreign rows at their signing-date rate.
+ `UPDATE contracts SET fx_rate = ${rateAt('contracts.signed_at', 'contracts.currency')}
+ WHERE id IN (SELECT id FROM refresh_touched_contracts)
+ AND COALESCE(NULLIF(currency, ''), 'BGN') NOT IN ('BGN','EUR');`,
+
+ // 3–5. The row repair itself (also runs standalone when resuming an interrupted run).
+ repairRowsSql(),
+ ].join('\n');
+}
+
+/**
+ * Repair the served DB in place. Throws when a rate fetch fails while damage remains (fail loudly
+ * rather than leave silent NULLs); currencies ECB cannot price are returned in `remaining`.
+ *
+ * Interruption-safe: the repair + rollup refresh go out as ONE exec (one wrangler/sqlite3
+ * invocation), and if a prior run died between its row updates and its rollup refresh, the
+ * refresh_touched_* tables it left behind are detected up front and its rollups are healed
+ * first — before the early return and before any network fetch, so even a failing rate fetch
+ * cannot strand them a second time.
+ */
+export async function backfillFx(runner, { fetchFn = fetch, fetchedAt, refreshSliceSql, api }) {
+ const rollupSql = refreshSliceStatementGroups(refreshSliceSql)
+ .filter((g) => ROLLUP_GROUPS.includes(g.name))
+ .map((g) => g.statements.map((s) => `${s};`).join('\n'))
+ .join('\n');
+
+ const before = reportFxDamage(runner);
+ let healed = false;
+ if (before.interrupted) {
+ // Scoped by the leftover touched tables. Re-applying the row repair too makes a partial
+ // failure INSIDE the interrupted run's repair (not just between repair and rollups)
+ // converge on retry; its cleanup drops the tables.
+ runner.exec(`${repairRowsSql()}\n${rollupSql}`);
+ healed = true;
+ }
+ if (before.total === 0 && before.flagUnverified === 0) {
+ return {
+ repaired: 0,
+ reflagged: 0,
+ valueLowUnverifiable: 0,
+ remaining: [],
+ fetched: [],
+ healed,
+ before,
+ };
+ }
+
+ const fetched = await loadMissingRates(runner, { fetchFn, fetchedAt, api });
+ const reflagged = Number(
+ runner.query(
+ `SELECT COUNT(*) AS n FROM contracts WHERE ${FLAG_CANDIDATE} AND (${NEW_FLAG}) <> contracts.value_flag`,
+ )[0]?.n ?? 0,
+ );
+ // value_low blind spot: the derive's value_low "< 1000 EUR signed AND < 5% of the estimate" branch
+ // needs the contract's OWN estimated_value / procurement_currency, which are NOT served (dropped
+ // from the contracts table). A row we price to 'ok' with a tiny (< 1000 EUR) signing value is
+ // exactly where that branch MIGHT fire — undecidable offline. Counted pre-repair (rates loaded,
+ // rows still match the damage predicate): they may enter cpv_division_stats percentiles where a
+ // full re-derive would exclude them (ADR-0030 residual). Upper bound — a genuinely small estimate
+ // leaves the row correctly 'ok'.
+ const valueLowUnverifiable = Number(
+ runner.query(
+ `SELECT COUNT(*) AS n FROM contracts
+ WHERE ${DAMAGE} AND signed_at IS NOT NULL AND ${PRICEABLE}
+ AND (${NEW_FLAG}) = 'ok'
+ AND contracts.signing_value IS NOT NULL
+ AND ${toEur('contracts.signing_value', contractCurrency)} < 1000`,
+ )[0]?.n ?? 0,
+ );
+ runner.exec(`${repairSql()}\n${rollupSql}`);
+
+ const after = reportFxDamage(runner);
+ return {
+ repaired: before.total - after.total,
+ reflagged,
+ valueLowUnverifiable,
+ remaining: after.rows.map((r) => ({
+ id: r.id,
+ contract_number: r.contract_number,
+ currency: r.currency,
+ signed_at: r.signed_at,
+ })),
+ fetched,
+ healed,
+ before,
+ };
+}
+
+// ---------------------------------------------------------------------------------------------
+// CLI plumbing (same target model as scripts/load-fx.mjs: sqlite work DB / local D1 / remote D1).
+function arg(name) {
+ const hit = process.argv.find((a) => a === `--${name}` || a.startsWith(`--${name}=`));
+ if (!hit) return undefined;
+ const eq = hit.indexOf('=');
+ return eq === -1 ? true : hit.slice(eq + 1);
+}
+
+function cliRunner() {
+ const apiDir = resolve(root, 'apps/web');
+ const workDb = arg('work-db');
+ if (workDb === true) throw new Error('--work-db needs a value: --work-db=');
+ const remoteFlag = process.argv.includes('--remote') ? '--remote' : '--local';
+ const persistTo = arg('persist-to');
+ if (workDb && process.argv.includes('--remote'))
+ throw new Error('--work-db and --remote are mutually exclusive');
+ const d1Name = process.env.SIGMA_D1_NAME || 'sigma';
+ const persistArgs =
+ remoteFlag === '--local' && persistTo ? ['--persist-to', String(persistTo)] : [];
+ // wrangler is a devDependency: `node scripts/backfill-fx.mjs` runs without node_modules/.bin on
+ // PATH, so resolve the workspace binary first and fall back to a global install.
+ const localWrangler = resolve(root, 'node_modules/.bin/wrangler');
+ const wrangler = existsSync(localWrangler) ? localWrangler : 'wrangler';
+
+ const query = (sql) => {
+ if (workDb) {
+ const out = execFileSync('sqlite3', ['-json', String(workDb), sql], {
+ encoding: 'utf8',
+ maxBuffer: 64 * 1024 * 1024,
+ }).trim();
+ return out ? JSON.parse(out) : [];
+ }
+ const out = execFileSync(
+ wrangler,
+ ['d1', 'execute', d1Name, remoteFlag, ...persistArgs, '--json', '--command', sql],
+ { cwd: apiDir, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 },
+ );
+ return JSON.parse(out.slice(out.indexOf('[')))[0].results;
+ };
+
+ const exec = (sql) => {
+ const outDir = resolve(root, 'data');
+ mkdirSync(outDir, { recursive: true });
+ const file = resolve(outDir, 'fx-backfill.sql');
+ writeFileSync(file, `${sql}\n`);
+ if (workDb) {
+ // One transaction per exec: -bail aborts on the first error and the process exit rolls the
+ // open transaction back, so a failed repair leaves the file exactly as it was.
+ execFileSync('sqlite3', ['-bail', String(workDb)], {
+ input: `BEGIN;\n${readFileSync(file, 'utf8')}\nCOMMIT;\n`,
+ stdio: ['pipe', 'inherit', 'inherit'],
+ });
+ } else {
+ execFileSync(
+ wrangler,
+ ['d1', 'execute', d1Name, remoteFlag, ...persistArgs, '--file', file],
+ {
+ cwd: apiDir,
+ stdio: 'inherit',
+ },
+ );
+ }
+ };
+
+ return { query, exec };
+}
+
+async function main() {
+ const runner = cliRunner();
+ const report = reportFxDamage(runner);
+ console.log(`legacy-NULL foreign-currency contracts: ${report.total}`);
+ for (const [ccy, n] of Object.entries(report.byCurrency)) console.log(` ${ccy}: ${n}`);
+ for (const r of report.rows.slice(0, 10)) {
+ console.log(
+ ` ${r.contract_number} ${r.currency} ${r.signed_at} amount=${r.amount} [${r.value_flag}]`,
+ );
+ }
+ if (report.rows.length > 10) console.log(` … ${report.rows.length - 10} more`);
+ if (report.flagUnverified > 0) {
+ console.log(
+ `value_flag unverifiable offline (foreign tender-estimate currency without a loaded rate): ${report.flagUnverified}`,
+ );
+ }
+ if (report.interrupted) {
+ console.warn(
+ 'leftover refresh_touched_* tables: an earlier refresh/repair died before its rollup refresh — rollups may be stale.',
+ );
+ }
+
+ const dirty = report.total > 0 || report.flagUnverified > 0 || report.interrupted;
+ if (!process.argv.includes('--apply')) {
+ if (dirty) {
+ console.log('\nrun with --apply to repair (rates + EUR columns + flags + rollups).');
+ process.exit(1);
+ }
+ console.log('clean — nothing to do.');
+ return;
+ }
+ if (!dirty) {
+ console.log('clean — nothing to do.');
+ return;
+ }
+
+ const refreshSliceSql = readFileSync(resolve(root, 'scripts/refresh-slice.sql'), 'utf8');
+ const summary = await backfillFx(runner, {
+ fetchedAt: new Date().toISOString(),
+ refreshSliceSql,
+ });
+ console.log(`\nrepaired: ${summary.repaired}, reflagged: ${summary.reflagged}`);
+ if (summary.valueLowUnverifiable > 0) {
+ console.warn(
+ `value_low unverifiable offline (tiny foreign value, raw estimate not served): ${summary.valueLowUnverifiable} — may skew cpv_division_stats percentiles; run a full re-derive for absolute correctness (ADR-0030).`,
+ );
+ }
+ if (summary.healed) console.log('healed the interrupted run’s stale rollups.');
+ for (const f of summary.fetched) {
+ console.log(
+ ` fx ${f.currency} ${f.start ?? ''}..${f.end ?? ''} → ${f.loaded ?? 0} rates [${f.status}]`,
+ );
+ }
+ if (summary.remaining.length > 0) {
+ console.warn(`\nstill unpriced (no ECB rate available — see ADR-0030):`);
+ for (const r of summary.remaining) {
+ console.warn(` ${r.contract_number} ${r.currency} ${r.signed_at}`);
+ }
+ }
+}
+
+const isMain = process.argv[1] && pathToFileURL(resolve(process.argv[1])).href === import.meta.url;
+if (isMain) await main();
diff --git a/scripts/import.mjs b/scripts/import.mjs
index 578a1132b..90d530d53 100644
--- a/scripts/import.mjs
+++ b/scripts/import.mjs
@@ -18,6 +18,7 @@ import { computeCatchupWindow, daysInWindow } from '../packages/ingest/src/ocds.
import {
dropTransientStagingStatements,
refreshSliceStatementGroups,
+ REFRESH_SLICE_ROLLUP_GROUPS,
} from '../packages/ingest/src/refresh.ts';
import { assertIntegrity } from './integrity-checks.mjs';
import { buildAnomalyReport, formatAnomalyReport } from './anomaly-report.mjs';
@@ -141,7 +142,8 @@ function assertFxPopulated() {
const missing = Number(rows[0]?.missing_fx ?? 0);
if (missing > 0) {
console.error(
- `!! FX assertion failed: ${missing} foreign-currency contracts have NULL amount_eur after normalize.`,
+ `!! FX assertion failed: ${missing} foreign-currency contracts have NULL amount_eur after normalize.\n` +
+ ' Pre-existing legacy rows (cron bug window, #158)? Repair first: node scripts/backfill-fx.mjs --apply',
);
process.exit(1);
}
@@ -168,7 +170,8 @@ function assertFxPopulatedSqlite(dbPath) {
const missing = Number(rows[0]?.missing_fx ?? 0);
if (missing > 0) {
console.error(
- `!! FX assertion failed: ${missing} foreign-currency contracts have NULL amount_eur after normalize.`,
+ `!! FX assertion failed: ${missing} foreign-currency contracts have NULL amount_eur after normalize.\n` +
+ ' Pre-existing legacy rows (cron bug window, #158)? Repair first: node scripts/backfill-fx.mjs --apply --work-db ',
);
process.exit(1);
}
@@ -247,14 +250,21 @@ async function runSliceDerive() {
run('node', ['scripts/load-fx.mjs', '--apply', ...passthru]);
execSql(resolve(root, 'scripts/load-nuts.sql'));
execSql(resolve(root, 'scripts/seed-state-owned.sql'));
- runRefreshSliceBatches();
+ // Same FX gate as the full derive (#158), at the same point: after the rows are derived and
+ // BEFORE any rollup is written — a gate after the rollups would fire loudly but leave the
+ // corrupted totals already served (mirrors assertFxPopulated's slot before precompute.sql).
+ runRefreshSliceBatches((g) => !REFRESH_SLICE_ROLLUP_GROUPS.includes(g.name));
+ assertFxPopulated();
+ runRefreshSliceBatches((g) => REFRESH_SLICE_ROLLUP_GROUPS.includes(g.name));
await assertIntegrity(d1, { label: 'slice derive (D1)' });
reportAnomalies(d1, 'slice derive (D1)');
}
-function runRefreshSliceBatches() {
+function runRefreshSliceBatches(include = () => true) {
const refreshSlicePath = resolve(root, 'scripts/refresh-slice.sql');
- const groups = refreshSliceStatementGroups(readFileSync(refreshSlicePath, 'utf8'));
+ const groups = refreshSliceStatementGroups(readFileSync(refreshSlicePath, 'utf8')).filter(
+ include,
+ );
const batchDirParent = resolve(root, 'data/work');
mkdirSync(batchDirParent, { recursive: true });
const batchDir = mkdtempSync(resolve(batchDirParent, 'refresh-slice-'));