feat(nipo): persist UIPV dossier (docs+payments) + refresh prod egress IPs

[overthelex]

Що і навіщо (LEXAI-1835)

Демо для МСП (свідоцтво №67482) показує блок «Досьє» — документообіг і платежі/держмито. Ці дані (data_docs, data_payments) приходять на верхньому рівні запису SIS open-data (сусіди внутрішнього data), але обидва імпортери зберігали в raw_data лише data, тому досьє втрачалось. Артефакт довелось збирати ручним викликом sis-API.

Тепер досьє зберігається штатно і віддається через search_registry.

Зміни

• 166_ip_dossier_columns.sql — додає data_payments/data_docs (JSONB) до opendata_trademarks і opendata_patents (ідемпотентно, ADD COLUMN IF NOT EXISTS).
• import-uipv-multi-ip.py / import-uipv-async.py — extract + upsert обох полів досьє для ТМ і патентів (той самий патерн, що вже є для raw_data).
• registry-catalog.ts — data_payments/data_docs додано до selectColumns для trademarks і patents, тож search_registry повертає досьє (напр., причина припинення ТМ 67482: позов 2023 → ухвала → припинення 2025; історія зборів 85 грн 2006 → 4950 грн 2016).

Побічно — оновлення egress-IP (task 1)

SOURCE_IPS у import-uipv-multi-ip.py дрейфнув: 4 з 10 зашитих EIP звільнено/переприв'язано. Замінено на 15 EIP-backed secondary-IP, які зараз на інстансі i-04e39a795576e33f0 (ENI eni-043d37237c0523292) — звірено через AWS describe-addresses і перевірено, що всі підняті на хості. ~15 незалежних rate-limit-бакетів SIS замість 10.

Валідація

• python3 -m py_compile обох імпортерів — OK.
• registry-catalog.ts — typecheck чистий (зміна лише в рядку selectColumns).
• Прод-версію import-uipv-multi-ip.py взято за базу — у PR тільки цільові правки (SOURCE_IPS + досьє); локальний незакоммічений експеримент THREADS_PER_IP=5 навмисно не включено (він порушив би per-IP rate-limit).

Розгортання / нотатки

• Міграція 166 накотиться через CI/CD.
• Імпортери раніше не трекались у main (жили лише в прод-checkout b83e89d1 + локально untracked). Цей PR вперше версіонує їх. Після мержа прод-checkout треба синхронізувати, щоб наступний sync писав досьє.
• Наявні рядки (~389K ТМ / ~347K патентів) отримають досьє при повторному sync/бекфілі; поточний фоновий re-pull винаходів+промзразків стартував зі старим кодом і досьє не пише — окремий бекфіл після мержа.

Parent: LEXAI-1833 · Closes LEXAI-1835

🤖 Generated with Claude Code

---

Summary by cubic

Persist and expose the UIPV/NIPO dossier fields (documents and payments) for trademarks and patents, so search_registry returns full case and fee history. Also refresh prod egress IPs to 15 EIP-backed addresses to expand throughput. Implements LEXAI-1835.

• New Features

  • DB: add data_payments and data_docs (JSONB) to opendata_trademarks and opendata_patents (idempotent).
  • Importers: add scripts/opendata/nipo/import-uipv-multi-ip.py and scripts/opendata/nipo/import-uipv-async.py; extract and upsert dossier fields for trademarks and patents.
  • API/search: include data_payments and data_docs in registry-catalog.ts selectColumns so search_registry returns them.
  • Infra: update SOURCE_IPS in import-uipv-multi-ip.py to 15 current EIP-backed secondary IPs to improve rate-limit headroom.

• Migration

  • Migration 166 applies via CI/CD.
  • After merge, sync the prod importer checkout and re-run imports to backfill dossier for existing records.

^{Written for commit 3de691f. Summary will update on new commits.}

[cubic-dev-ai]

7 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/opendata/nipo/import-uipv-async.py">

<violation number="1" location="scripts/opendata/nipo/import-uipv-async.py:168">
P2: Patent re-imports can leave registry details stale for changed registration dates, English titles, abstracts, IPC codes, owner country, and inventor names because the conflict update omits those inserted columns. Consider updating all extracted patent fields on conflict so `search_registry` reflects the latest SIS record.</violation>

<violation number="2" location="scripts/opendata/nipo/import-uipv-async.py:235">
P3: The `session` parameter in `fetch_page` is always passed but never read — the function creates its own per-IP `ClientSession` internally. Removing the unused parameter clarifies the contract (each call binds to a specific IP via a fresh session) and eliminates the false hint that an outer session could be reused.</violation>

<violation number="3" location="scripts/opendata/nipo/import-uipv-async.py:240">
P1: HTTPS imports can accept spoofed SIS responses because this connector disables certificate verification for every request made through it. Keeping the `local_addr` binding while using aiohttp's default SSL verification preserves the multi-IP behavior without weakening transport security.</violation>

<violation number="4" location="scripts/opendata/nipo/import-uipv-async.py:304">
P3: The `ssl_ctx = ssl.create_default_context()` variable is created but never used anywhere. It can be removed. If the intent was to eventually pass a proper SSL context into the per-IP sessions, that work was left incomplete — the sessions currently use `ssl=False` (no verification), which is a separate concern. Removing the unused assignment keeps the codebase clean.</violation>
</file>

<file name="scripts/opendata/nipo/import-uipv-multi-ip.py">

<violation number="1" location="scripts/opendata/nipo/import-uipv-multi-ip.py:59">
P2: Host resolution and socket family are inconsistent: DNS may return IPv6 first but requests always use an IPv4 socket. This can cause intermittent connection failures depending on resolver answer order.</violation>

<violation number="2" location="scripts/opendata/nipo/import-uipv-multi-ip.py:340">
P1: Per-IP throttling is not actually enforced: global queue scheduling allows two workers to run `process_page` with the same `ip` simultaneously. That can exceed SIS per-IP limits and create avoidable 429/retry churn; a serial worker/queue per IP would keep the intended 1 req/sec behavior.</violation>

<violation number="3" location="scripts/opendata/nipo/import-uipv-multi-ip.py:364">
P1: Checkpoint-save logic risks skipping pages on resume, potentially silently dropping thousands of records. The checkpoint saves the page number of whichever page happened to complete most recently via `as_completed`, not the maximum completed page or the minimum uncompleted page. With 15 IPs driving parallel requests, out-of-order completion is highly likely. If the process restarts after a checkpoint, it resumes from that saved page, skipping every uncompleted page with a lower number.

A robust approach tracks the minimum page that hasn't yet been confirmed complete (e.g., maintain a `min_uncompleted` counter via an atomic set or sorted completion tracking), and saves that as the resume point.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: HTTPS imports can accept spoofed SIS responses because this connector disables certificate verification for every request made through it. Keeping the local_addr binding while using aiohttp's default SSL verification preserves the multi-IP behavior without weakening transport security.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-async.py, line 240:

<comment>HTTPS imports can accept spoofed SIS responses because this connector disables certificate verification for every request made through it. Keeping the `local_addr` binding while using aiohttp's default SSL verification preserves the multi-IP behavior without weakening transport security.</comment>

<file context>
@@ -0,0 +1,378 @@
+    for attempt in range(MAX_RETRIES):
+        await limiter.acquire()
+        try:
+            connector = aiohttp.TCPConnector(local_addr=(limiter.ip, 0), ssl=False)
+            async with aiohttp.ClientSession(connector=connector, timeout=TIMEOUT) as ip_session:
+                async with ip_session.get(url, headers={
</file context>

[cubic-dev-ai]

P1: Per-IP throttling is not actually enforced: global queue scheduling allows two workers to run process_page with the same ip simultaneously. That can exceed SIS per-IP limits and create avoidable 429/retry churn; a serial worker/queue per IP would keep the intended 1 req/sec behavior.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-multi-ip.py, line 340:

<comment>Per-IP throttling is not actually enforced: global queue scheduling allows two workers to run `process_page` with the same `ip` simultaneously. That can exceed SIS per-IP limits and create avoidable 429/retry churn; a serial worker/queue per IP would keep the intended 1 req/sec behavior.</comment>

<file context>
@@ -0,0 +1,397 @@
+        futures = {}
+        for i, page_num in enumerate(pages):
+            ip = ips[i % len(ips)]
+            futures[executor.submit(process_page, page_num, obj_type_num, table, ip)] = page_num
+
+        for future in as_completed(futures):
</file context>

[cubic-dev-ai]

P1: Checkpoint-save logic risks skipping pages on resume, potentially silently dropping thousands of records. The checkpoint saves the page number of whichever page happened to complete most recently via as_completed, not the maximum completed page or the minimum uncompleted page. With 15 IPs driving parallel requests, out-of-order completion is highly likely. If the process restarts after a checkpoint, it resumes from that saved page, skipping every uncompleted page with a lower number.

A robust approach tracks the minimum page that hasn't yet been confirmed complete (e.g., maintain a min_uncompleted counter via an atomic set or sorted completion tracking), and saves that as the resume point.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-multi-ip.py, line 364:

<comment>Checkpoint-save logic risks skipping pages on resume, potentially silently dropping thousands of records. The checkpoint saves the page number of whichever page happened to complete most recently via `as_completed`, not the maximum completed page or the minimum uncompleted page. With 15 IPs driving parallel requests, out-of-order completion is highly likely. If the process restarts after a checkpoint, it resumes from that saved page, skipping every uncompleted page with a lower number.

A robust approach tracks the minimum page that hasn't yet been confirmed complete (e.g., maintain a `min_uncompleted` counter via an atomic set or sorted completion tracking), and saves that as the resume point.</comment>

<file context>
@@ -0,0 +1,397 @@
+
+                # Checkpoint
+                os.makedirs(CHECKPOINT_DIR, exist_ok=True)
+                json.dump({"page": page_num, "imported": stats["imported"], "ts": time.strftime("%Y-%m-%dT%H:%M:%S")},
+                          open(cp_file, "w"))
+
</file context>

[cubic-dev-ai]

P2: Patent re-imports can leave registry details stale for changed registration dates, English titles, abstracts, IPC codes, owner country, and inventor names because the conflict update omits those inserted columns. Consider updating all extracted patent fields on conflict so search_registry reflects the latest SIS record.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-async.py, line 168:

<comment>Patent re-imports can leave registry details stale for changed registration dates, English titles, abstracts, IPC codes, owner country, and inventor names because the conflict update omits those inserted columns. Consider updating all extracted patent fields on conflict so `search_registry` reflects the latest SIS record.</comment>

<file context>
@@ -0,0 +1,378 @@
+                             status, last_update, raw_data, data_payments, data_docs)
+                        VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
+                        ON CONFLICT (app_number) DO UPDATE SET
+                            registration_number=EXCLUDED.registration_number,
+                            registration_date=EXCLUDED.registration_date,
+                            holder_name=EXCLUDED.holder_name,
</file context>

[cubic-dev-ai]

P2: Host resolution and socket family are inconsistent: DNS may return IPv6 first but requests always use an IPv4 socket. This can cause intermittent connection failures depending on resolver answer order.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-multi-ip.py, line 59:

<comment>Host resolution and socket family are inconsistent: DNS may return IPv6 first but requests always use an IPv4 socket. This can cause intermittent connection failures depending on resolver answer order.</comment>

<file context>
@@ -0,0 +1,397 @@
+def resolve_host(host):
+    with _dns_lock:
+        if host not in _dns_cache:
+            _dns_cache[host] = socket.getaddrinfo(host, 443)[0][4][0]
+        return _dns_cache[host]
+
</file context>

Suggested change

	_dns_cache[host] = socket.getaddrinfo(host, 443)[0][4][0]
	_dns_cache[host] = socket.getaddrinfo(host, 443, socket.AF_INET)[0][4][0]

[cubic-dev-ai]

P3: The session parameter in fetch_page is always passed but never read — the function creates its own per-IP ClientSession internally. Removing the unused parameter clarifies the contract (each call binds to a specific IP via a fresh session) and eliminates the false hint that an outer session could be reused.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-async.py, line 235:

<comment>The `session` parameter in `fetch_page` is always passed but never read — the function creates its own per-IP `ClientSession` internally. Removing the unused parameter clarifies the contract (each call binds to a specific IP via a fresh session) and eliminates the false hint that an outer session could be reused.</comment>

<file context>
@@ -0,0 +1,378 @@
+        self.semaphore.release()
+
+
+async def fetch_page(session: aiohttp.ClientSession, url: str, limiter: IPRateLimiter) -> dict | None:
+    """Fetch one page with rate limiting and retries."""
+    for attempt in range(MAX_RETRIES):
</file context>

[cubic-dev-ai]

P3: The ssl_ctx = ssl.create_default_context() variable is created but never used anywhere. It can be removed. If the intent was to eventually pass a proper SSL context into the per-IP sessions, that work was left incomplete — the sessions currently use ssl=False (no verification), which is a separate concern. Removing the unused assignment keeps the codebase clean.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/opendata/nipo/import-uipv-async.py, line 304:

<comment>The `ssl_ctx = ssl.create_default_context()` variable is created but never used anywhere. It can be removed. If the intent was to eventually pass a proper SSL context into the per-IP sessions, that work was left incomplete — the sessions currently use `ssl=False` (no verification), which is a separate concern. Removing the unused assignment keeps the codebase clean.</comment>

<file context>
@@ -0,0 +1,378 @@
+    limiters = [IPRateLimiter(ip, THREADS_PER_IP, RATE_LIMIT) for ip in ips]
+
+    # Get total count
+    ssl_ctx = ssl.create_default_context()
+    async with aiohttp.ClientSession(timeout=TIMEOUT) as session:
+        first_url = f"{API_BASE}?obj_type={obj_type_num}&obj_state=2&page=1"
</file context>