fix(opendata): rate-limit nl_rechtspraak importer, opt-in court scope filter

[teosoph]

Summary

• Adds a real sliding-window rate limiter (shared/http_client.py) shared across all workers/IPs in a MultiIPSessionPool, wired in via an opt-in RATE_LIMIT_PER_SEC on BaseImporter.
• Enables it for nl_rechtspraak at 8 req/sec — data.rechtspraak.nl documents a 10 req/sec fair-use cap, and a plain 4-worker concurrency setting was measured hitting 11.7 req/s during pilot testing (worker count alone doesn't guarantee staying under a documented limit when per-request latency is low).
• Adds an optional ECLI-court-code scope filter (PILOT_COURTS env var, default ALL — no behavior change for existing deployments) for narrowing a run to specific courts (e.g. HR,GHAMS,GHARL,GHDHA,GHSHE for Hoge Raad + Gerechtshoven only).
• Adds a START_DATE override for import_dataset() — useful for a cold-start historical backfill where the existing "resume from MAX(decision_date) on target" logic would otherwise start from the year-2000 fallback sentinel.

All three additions are opt-in via env var / class attribute with defaults that preserve current behavior — no change to docker-compose.opendata.yml or any deployed target configuration.

Test plan

• python3 -m py_compile on all three changed files
• Ran the importer directly (venv, not via compose) against a scratch Postgres instance with PILOT_COURTS set to a 5-court subset and to ALL, confirmed the filter includes/excludes correctly
• Confirmed the rate limiter holds steady at the configured cap (measured 7.5-7.7 req/s against an 8/sec limit) across a multi-hour real run against data.rechtspraak.nl
• Not yet verified inside the actual opendata-importers Docker image / compose stack (only tested via direct python3 -m importers.nl_rechtspraak execution)

🤖 Generated with Claude Code

---

Summary by cubic

Adds a shared sliding-window rate limiter and enables it at 8 req/sec for the nl_rechtspraak importer to stay under the 10 req/sec fair-use cap. Also adds opt-in court scoping and a start-date override for targeted pilots and backfills; defaults keep current behavior.

• New Features
  • Global RateLimiter in shared/http_client.py that caps requests per rolling second across all workers/IPs in MultiIPSessionPool.
  • BaseImporter now accepts RATE_LIMIT_PER_SEC; nl_rechtspraak sets it to 8.0.
  • Optional court scope filter via PILOT_COURTS (comma-separated ECLI court codes; ALL or unset disables filtering).
  • Optional START_DATE to start historical runs from a specific day instead of MAX(decision_date) fallback.
  • No changes needed to existing configs; all additions are opt-in.

^{Written for commit 0ce455b. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="services/opendata-importers/importers/nl_rechtspraak.py">

<violation number="1" location="services/opendata-importers/importers/nl_rechtspraak.py:35">
P2: Court lists containing conventional spaces after commas silently exclude those courts. Normalize each `PILOT_COURTS` item before comparing it to uppercase ECLI court codes.</violation>
</file>

<file name="services/opendata-importers/shared/http_client.py">

<violation number="1" location="services/opendata-importers/shared/http_client.py:38">
P3: `RateLimiter.acquire()` crashes with `IndexError` if `max_per_sec` is 0 or negative. When the rate limiter is initialized with a non-positive limit, the first call enters `acquire()`, finds the deque empty, skips the eviction while-loop, evaluates `len(self._times) < self.max_per_sec` as False (0 < 0 or 0 < -N), and then reaches `self._times[0]` on an empty deque.

This won't trigger through the current `base.py` code path since `if self.RATE_LIMIT_PER_SEC:` uses truthiness (0.0 is falsy). But the `RateLimiter` class is public and could be used directly or extended with a computed value. Adding a guard in `__init__` would make it robust.

**Suggestion**: In `__init__`, validate and clamp: `if max_per_sec <= 0: raise ValueError("max_per_sec must be positive")`.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Court lists containing conventional spaces after commas silently exclude those courts. Normalize each PILOT_COURTS item before comparing it to uppercase ECLI court codes.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At services/opendata-importers/importers/nl_rechtspraak.py, line 35:

<comment>Court lists containing conventional spaces after commas silently exclude those courts. Normalize each `PILOT_COURTS` item before comparing it to uppercase ECLI court codes.</comment>

<file context>
@@ -26,6 +26,21 @@
+# unchanged; this is strictly opt-in for a narrower pilot/test run.
+# ECLI shape is ECLI:NL:<court-code>:<year>:<ordinal>.
+_PILOT_COURTS_ENV = os.environ.get("PILOT_COURTS", "ALL")
+_PILOT_COURTS = None if _PILOT_COURTS_ENV.strip().upper() == "ALL" else set(
+    _PILOT_COURTS_ENV.split(","))
+_ECLI_COURT_RE = re.compile(r"^ECLI:NL:([A-Z]+):")
</file context>

Suggested change

	_PILOT_COURTS = None if _PILOT_COURTS_ENV.strip().upper() == "ALL" else set(
	_PILOT_COURTS_ENV.split(","))
	_PILOT_COURTS = None if _PILOT_COURTS_ENV.strip().upper() == "ALL" else {
	court.strip().upper() for court in _PILOT_COURTS_ENV.split(",") if court.strip()
	}

[cubic-dev-ai]

P3: RateLimiter.acquire() crashes with IndexError if max_per_sec is 0 or negative. When the rate limiter is initialized with a non-positive limit, the first call enters acquire(), finds the deque empty, skips the eviction while-loop, evaluates len(self._times) < self.max_per_sec as False (0 < 0 or 0 < -N), and then reaches self._times[0] on an empty deque.

This won't trigger through the current base.py code path since if self.RATE_LIMIT_PER_SEC: uses truthiness (0.0 is falsy). But the RateLimiter class is public and could be used directly or extended with a computed value. Adding a guard in __init__ would make it robust.

Suggestion: In __init__, validate and clamp: if max_per_sec <= 0: raise ValueError("max_per_sec must be positive").

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At services/opendata-importers/shared/http_client.py, line 38:

<comment>`RateLimiter.acquire()` crashes with `IndexError` if `max_per_sec` is 0 or negative. When the rate limiter is initialized with a non-positive limit, the first call enters `acquire()`, finds the deque empty, skips the eviction while-loop, evaluates `len(self._times) < self.max_per_sec` as False (0 < 0 or 0 < -N), and then reaches `self._times[0]` on an empty deque.

This won't trigger through the current `base.py` code path since `if self.RATE_LIMIT_PER_SEC:` uses truthiness (0.0 is falsy). But the `RateLimiter` class is public and could be used directly or extended with a computed value. Adding a guard in `__init__` would make it robust.

**Suggestion**: In `__init__`, validate and clamp: `if max_per_sec <= 0: raise ValueError("max_per_sec must be positive")`.</comment>

<file context>
@@ -10,11 +12,38 @@
+                if len(self._times) < self.max_per_sec:
+                    self._times.append(now)
+                    return
+                sleep_for = 1.0 - (now - self._times[0])
+                await asyncio.sleep(max(sleep_for, 0.01))
+
</file context>