fix(legislation): keep the article index when importing editions

[overthelex]

LEXAI-1957.

Симптом

get_legislation_section{rada_id:"1618-15", article_number:"350-1"} → not found, хотя ст. 350-1 ЦПК действует.

Замер

Считались только статьи, присутствующие в действующей редакции по npa.article (max ed_date), для актов из public.legislation. Это важно: подсчёт «есть ли строка» смешивает исключённые статьи с недоступными, а ЦПК 105-1 и 11-1 действительно вычеркнуты при переписывании 2017 года, и «не найдено» на них правильный ответ.

	в действующей редакции	доступно	недоступно
с индексом	1 707	541	1 166 (68.3%)
обычных	15 982	15 472	510 (3.2%)

Из 1 166 только 148 имели строку без флага. У 1 018 строки не было вовсе, поэтому это не бэкфил is_current.

Причина

Рада пишет индекс с пробелами вокруг дефиса, в базе заголовок хранится как Стаття 350 - 1 .. Регулярка импортёра (\d+(?:-\d+)?) не допускает ни пробелов, ни en/em dash, поэтому захватывала 350, сталкивалась с настоящей статьёй 350 и теряла строку на ON CONFLICT DO NOTHING. Отсюда неизменные 500 статей в каждой редакции ЦПК с 2004 по 2028 при 525 в npa.article.

Проверенные и отброшенные версии: флаги is_current (148 из 1 166), en dash (в npa.edition_text для 1618-15 нашлось 33 текста с ASCII-дефисом и 0 с en dash), жёсткий лимит 500 (в импортёре его нет, по другим актам число статей гуляет).

Что сделано

ARTICLE_NUMBER_PATTERN встал рядом с normalizeArticleNumber, все три регулярки импортёра собираются из него. Было четыре написания одного и того же, стало одно.

Миграция 192 чинит уже сохранённое: недостающие действующие статьи прикрепляются к собственной дате текущего снимка акта, чтобы у акта осталась ровно одна текущая редакция. Идемпотентна, и пропускается там, где схемы npa нет.

Применено на проде

1 344 строки (1 149 с индексом), помечены metadata.task = LEXAI-1957.

	было	стало
с индексом	68.3% недоступно	1.0%
обычных	3.2% недоступно	2.0%

Живой MCP на 1618-15 350-1 отдаёт «Підсудність». Повторный запуск миграции вставляет 0.

Тест

Строит регулярку из отгруженной константы, а не повторяет её. Проверен на способность падать: на старом шаблоне даёт 350 - 1 → 350.

🤖 Generated with Claude Code

---

Summary by cubic

Keeps indexed article numbers (e.g., “350-1”) when importing Rada editions and backfills missing in‑force indexed articles (LEXAI-1957). Previously, the importer captured only the base number (“350”), collided with existing rows, and skipped inserts, so lookups like get_legislation_section(..., "350-1") returned “not found”.

• Importer now builds its regexes from ARTICLE_NUMBER_PATTERN and normalizes to the stored form, handling spaces and en/em dashes.
• Adds 192_backfill_indexed_articles.sql to insert missing in‑force indexed articles using each act’s current snapshot version_date; idempotent and skips when the npa schema is absent.
• Tests assert the shipped pattern and normalization to prevent drift.
• Required action: run migration 192.

^{Written for commit 29c8cb4. Summary will update on new commits.}

[cubic-dev-ai]

5 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="mcp_backend/src/migrations/192_backfill_indexed_articles.sql">

<violation number="1" location="mcp_backend/src/migrations/192_backfill_indexed_articles.sql:40">
P1: When an act already has current rows at multiple `version_date` values, this chooses one insertion date but leaves every older current snapshot flagged. Demote other current versions or select and normalize one snapshot before inserting, otherwise current-filtered lookups can return duplicate editions.</violation>

<violation number="2" location="mcp_backend/src/migrations/192_backfill_indexed_articles.sql:45">
P1: When `npa.article` contains a future edition, this selects its text while marking the backfilled rows current. Select the latest `ed_date <= CURRENT_DATE`, with an explicit fallback only when every edition is future-dated.</violation>
</file>

<file name="scripts/rada/import-historical-editions.ts">

<violation number="1" location="scripts/rada/import-historical-editions.ts:230">
P2: When Rada serializes the dash or surrounding spaces as HTML entities, this regex misses the indexed heading because it accepts only literal Unicode/ASCII characters. Decode the relevant HTML entities before all heading regexes, then pass the decoded capture to `normalizeArticleNumber`.</violation>
</file>

<file name="mcp_backend/src/services/act-number.ts">

<violation number="1" location="mcp_backend/src/services/act-number.ts:217">
P3: Inserting the new JSDoc block and constant between `normalizeArticleNumber`'s doc comment and the function leaves that doc (about the LONGEST-FIRST alternation in normalizeArticleNumber) orphaned directly above the unrelated article-number constant. `normalizeArticleNumber` now has no attached doc comment. Move the normalizeArticleNumber doc so it sits directly on the function, after the constant.</violation>

<violation number="2" location="mcp_backend/src/services/act-number.ts:217">
P2: The documented fix does not hold when the pattern is embedded in the importer's regex. The comment claims the stored ЦПК heading is «Стаття 350 - 1 .» (space before the period), but the new unit test only composes `Стаття\s+(pattern)` — it omits the `\.?</b>` trailer that import-historical-editions.ts actually appends. With that trailer, `Стаття 350 - 1 .</b>` yields NO MATCH, because the capture group consumes the space after «1» and `\.?</b>` cannot bridge the remaining space before the period (verified with the real preBoldRegex and plainText regex `\.\s*`, both of which require the dot right after the capture). As a result the headline 350-1 / 1618-15 case may still be lost by the primary preBold path; the added test passes only because it ignores the trailer that exposes the mismatch. Add a test that composes the full `<b>Стаття (...) \.?</b>` regex against «Стаття 350 - 1 .», and make the importer's trailer tolerate the space (e.g. `\.?\s*`) before the tag.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When an act already has current rows at multiple version_date values, this chooses one insertion date but leaves every older current snapshot flagged. Demote other current versions or select and normalize one snapshot before inserting, otherwise current-filtered lookups can return duplicate editions.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At mcp_backend/src/migrations/192_backfill_indexed_articles.sql, line 40:

<comment>When an act already has current rows at multiple `version_date` values, this chooses one insertion date but leaves every older current snapshot flagged. Demote other current versions or select and normalize one snapshot before inserting, otherwise current-filtered lookups can return duplicate editions.</comment>

<file context>
@@ -0,0 +1,90 @@
+  END IF;
+
+  WITH snap AS (        -- each act's own current snapshot date
+    SELECT l.id AS legislation_id, lower(l.rada_id) AS nreg, max(la.version_date) AS vd
+    FROM legislation l
+    JOIN legislation_articles la ON la.legislation_id = l.id AND la.is_current
</file context>

[cubic-dev-ai]

P1: When npa.article contains a future edition, this selects its text while marking the backfilled rows current. Select the latest ed_date <= CURRENT_DATE, with an explicit fallback only when every edition is future-dated.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At mcp_backend/src/migrations/192_backfill_indexed_articles.sql, line 45:

<comment>When `npa.article` contains a future edition, this selects its text while marking the backfilled rows current. Select the latest `ed_date <= CURRENT_DATE`, with an explicit fallback only when every edition is future-dated.</comment>

<file context>
@@ -0,0 +1,90 @@
+    JOIN legislation_articles la ON la.legislation_id = l.id AND la.is_current
+    GROUP BY 1, 2
+  ), ed AS (            -- the in-force edition in the clean corpus
+    SELECT nreg, max(ed_date) AS ed FROM npa.article GROUP BY 1
+  ), plan AS (
+    SELECT s.legislation_id, s.vd, a.art_no, a.title, a.body
</file context>

Suggested change

	SELECT nreg, max(ed_date) AS ed FROM npa.article GROUP BY 1
	SELECT nreg, coalesce(max(ed_date) FILTER (WHERE ed_date <= CURRENT_DATE), max(ed_date)) AS ed FROM npa.article GROUP BY 1

[cubic-dev-ai]

P2: When Rada serializes the dash or surrounding spaces as HTML entities, this regex misses the indexed heading because it accepts only literal Unicode/ASCII characters. Decode the relevant HTML entities before all heading regexes, then pass the decoded capture to normalizeArticleNumber.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/rada/import-historical-editions.ts, line 230:

<comment>When Rada serializes the dash or surrounding spaces as HTML entities, this regex misses the indexed heading because it accepts only literal Unicode/ASCII characters. Decode the relevant HTML entities before all heading regexes, then pass the decoded capture to `normalizeArticleNumber`.</comment>

<file context>
@@ -217,10 +218,19 @@ function extractArticlesFromEditionHtml(html: string): Array<{ article_number: s
+// (LEXAI-1957). normalizeArticleNumber folds the dashes and strips the spaces so
+// the stored value matches npa.article.art_no character for character.
+  const preBoldRegex = new RegExp(
+    `<b>Стаття\\s+(${ARTICLE_NUMBER_PATTERN})\\.?</b>\\s*([\\s\\S]*?)(?=<b>Стаття\\s+\\d|</pre>\\s*$|$)`, 'g');
   let match;
   while ((match = preBoldRegex.exec(html)) !== null) {
</file context>

[cubic-dev-ai]

P2: The documented fix does not hold when the pattern is embedded in the importer's regex. The comment claims the stored ЦПК heading is «Стаття 350 - 1 .» (space before the period), but the new unit test only composes Стаття\s+(pattern) — it omits the \.?</b> trailer that import-historical-editions.ts actually appends. With that trailer, Стаття 350 - 1 .</b> yields NO MATCH, because the capture group consumes the space after «1» and \.?</b> cannot bridge the remaining space before the period (verified with the real preBoldRegex and plainText regex \.\s*, both of which require the dot right after the capture). As a result the headline 350-1 / 1618-15 case may still be lost by the primary preBold path; the added test passes only because it ignores the trailer that exposes the mismatch. Add a test that composes the full <b>Стаття (...) \.?</b> regex against «Стаття 350 - 1 .», and make the importer's trailer tolerate the space (e.g. \.?\s*) before the tag.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At mcp_backend/src/services/act-number.ts, line 217:

<comment>The documented fix does not hold when the pattern is embedded in the importer's regex. The comment claims the stored ЦПК heading is «Стаття 350 - 1 .» (space before the period), but the new unit test only composes `Стаття\s+(pattern)` — it omits the `\.?</b>` trailer that import-historical-editions.ts actually appends. With that trailer, `Стаття 350 - 1 .</b>` yields NO MATCH, because the capture group consumes the space after «1» and `\.?</b>` cannot bridge the remaining space before the period (verified with the real preBoldRegex and plainText regex `\.\s*`, both of which require the dot right after the capture). As a result the headline 350-1 / 1618-15 case may still be lost by the primary preBold path; the added test passes only because it ignores the trailer that exposes the mismatch. Add a test that composes the full `<b>Стаття (...) \.?</b>` regex against «Стаття 350 - 1 .», and make the importer's trailer tolerate the space (e.g. `\.?\s*`) before the tag.</comment>

<file context>
@@ -202,6 +202,20 @@ export function looksLikeOfficialNumber(raw: string): boolean {
+ * ON CONFLICT DO NOTHING — 1 018 in-force indexed articles had no row at all
+ * (LEXAI-1957). Pair it with normalizeArticleNumber to get the stored form.
+ */
+export const ARTICLE_NUMBER_PATTERN = String.raw`\d+(?:\s*[-–—]\s*\d+)?`;
+
 export function normalizeArticleNumber(raw: string): string {
</file context>

[cubic-dev-ai]

P3: Inserting the new JSDoc block and constant between normalizeArticleNumber's doc comment and the function leaves that doc (about the LONGEST-FIRST alternation in normalizeArticleNumber) orphaned directly above the unrelated article-number constant. normalizeArticleNumber now has no attached doc comment. Move the normalizeArticleNumber doc so it sits directly on the function, after the constant.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At mcp_backend/src/services/act-number.ts, line 217:

<comment>Inserting the new JSDoc block and constant between `normalizeArticleNumber`'s doc comment and the function leaves that doc (about the LONGEST-FIRST alternation in normalizeArticleNumber) orphaned directly above the unrelated article-number constant. `normalizeArticleNumber` now has no attached doc comment. Move the normalizeArticleNumber doc so it sits directly on the function, after the constant.</comment>

<file context>
@@ -202,6 +202,20 @@ export function looksLikeOfficialNumber(raw: string): boolean {
+ * ON CONFLICT DO NOTHING — 1 018 in-force indexed articles had no row at all
+ * (LEXAI-1957). Pair it with normalizeArticleNumber to get the stored form.
+ */
+export const ARTICLE_NUMBER_PATTERN = String.raw`\d+(?:\s*[-–—]\s*\d+)?`;
+
 export function normalizeArticleNumber(raw: string): string {
</file context>