-
Notifications
You must be signed in to change notification settings - Fork 1
Background Jobs and Scripts
overthelex edited this page May 17, 2026
·
2 revisions
Background jobs and scripts in the SecondLayer project encompass automated data synchronization, batch document processing, database migrations, vectorization, and testing utilities. These components maintain the platform's legal data accuracy -- including court decisions, parliament bills, state registry information, EU and international data, and research datasets. They operate outside the primary request-response cycle, often running as scheduled cron tasks, CI/CD-triggered pipelines, or manual maintenance operations.
The system uses TypeScript for backend logic, Python for data-intensive scraping and ML pipelines, R for statistical analysis, and Shell for orchestration and deployment.
A GitHub Actions workflow (cron-edrsr-sync.yml) runs daily at 04:00 Kyiv time. It downloads the current year's court decision metadata ZIP from data.gov.ua and imports new records incrementally into prod PostgreSQL.
| Property | Value |
|---|---|
| Schedule |
0 1 * * * UTC (04:00 EET) |
| Runner | self-hosted, local |
| Script | scripts/edrsr/sync-edrsr-incremental.sh |
| Strategy |
ON CONFLICT DO NOTHING (partitioned table) |
| Manual trigger |
workflow_dispatch with optional year parameter |
| Coverage | 2008-2026 (2012 missing from data.gov.ua) |
A dedicated Docker container (opendata-sync) runs as a lightweight cron scheduler. It does not download data itself -- it delegates to existing import infrastructure via HTTP triggers.
| Target | Trigger endpoint | Sources |
|---|---|---|
| Backend (MVS, NAZK, UIPV) | POST /api/tools/start_import |
Government portals |
| NAIS registries (11 sources) | POST /api/admin/sync-registry |
OpenReyestr server |
cd opendata-sync
npm run dev # Local dev
npm run run-once # Single pass (all sources once)The mcp_rada server fetches data from the official Verkhovna Rada API. Available as npm scripts and standalone shell wrappers.
| npm script | Description | Cache TTL |
|---|---|---|
sync:deputies |
Bulk sync all deputies from RADA API | 7 days |
sync:laws |
Fetch and cache common laws (Constitution, Codes) from zakon.rada.gov.ua | 30 days |
sync:reference |
Sync reference data (factions, committees) | 7 days |
cleanup:cache |
Remove expired Redis cache entries | -- |
Shell scripts for batch operations:
| Script | Purpose |
|---|---|
scripts/rada/run-week-sync.sh |
Sync a week of RADA data (bills, votes, speeches) with configurable concurrency |
scripts/rada/sync-rada-data.sh |
Sync RADA data for a specific date via the MCP API |
scripts/rada/bulk-download-legislation.py |
Mass download legislation texts from zakon.rada.gov.ua |
scripts/rada/crawl-index-sequential.py |
Sequential crawl of RADA bill index pages |
scripts/rada/check-rada-changes.sh |
Detect new or updated bills since last sync |
Large-scale scripts for downloading, importing, and managing the State Court Decision Registry (~33M+ documents).
| Script | Purpose |
|---|---|
scripts/edrsr/sync-edrsr-incremental.sh |
Daily incremental sync (used by cron workflow) |
scripts/edrsr/download-and-import-fulltext.py |
Download full-text court decisions with multi-IP support |
scripts/edrsr/download-fulltext-multi-ip.py |
Multi-IP parallel downloader for fulltext (avoids rate limiting) |
scripts/edrsr/import-fulltext-parallel.py |
Parallel import of fulltext into PostgreSQL |
scripts/edrsr/backfill-fulltext-gaps.py |
Find and fill missing fulltext records |
scripts/edrsr/copy-new-to-prod-by-diff.py |
Transfer only new records from local to prod by comparing IDs |
scripts/edrsr/db-status.py |
Report partition sizes, row counts, gaps |
scripts/edrsr/edrsr-stats.py |
Aggregate statistics across all years |
scripts/edrsr/vectorize-cpk-batch.py |
Batch vectorization of civil cases (VoyageAI voyage-3.5 to Qdrant) |
scripts/edrsr/vectorize-kupap-batch.py |
Batch vectorization of administrative offense cases |
scripts/edrsr/backfill-tsv-prod.py |
Backfill TSV (full-text search) columns on prod |
The mcp_openreyestr server imports entities from data.gov.ua and multiple NAIS registries.
| npm script | Description |
|---|---|
import:entities |
Import legal entities from XML (data.gov.ua) |
import:debtors |
Import state debtors registry |
import:nais |
Download all NAIS registry datasets |
sync:registries |
Full sync of all 11 NAIS registries |
sync:weekly |
Weekly incremental registry sync |
sync:edrpou |
Sync EDRPOU (company registry) data |
import:street-renamings |
Import street renaming decommunization data |
Orchestration wrapper:
# Full parallel backfill -- 7 threads (3 EDRPOU + 4 NAIS)
./scripts/backfill-openreyestr.sh [local|stage] [--only=nais|edrpou] [--threads=10]Scripts for court session schedule and ECHR (European Court of Human Rights) data.
| Script | Purpose |
|---|---|
scripts/hudoc/download-hudoc.ts |
Download ECHR metadata and full texts from hudoc.echr.coe.int |
scripts/hudoc/import-echr-to-pg.ts |
Import HUDOC data into PostgreSQL (50 parallel workers) |
scripts/hudoc/prod-fast-import.py |
Fast production import of court session schedules |
scripts/hudoc/local-sharded-import.py |
Sharded local import with parallel processing |
# Download from OpenSanctions and import into PostgreSQL (50 workers)
./scripts/nsdc-sanctions/import-sanctions.sh [WORKERS=50]# Download and import ~737K records from data.gov.ua (JSON, Windows-1251)
./scripts/arma/import-arma-edraa.sh [--skip-download]| Script | Sources |
|---|---|
scripts/opendata/eu/import-eu-opendata.py |
EUR-Lex (regulations, directives), CURIA (CJEU), ECHR/HUDOC, EU Sanctions -- 100 async workers |
| Script | Purpose |
|---|---|
download-cendoj-autorotate.py |
Download CENDOJ court decisions with AWS EIP rotation |
download-cendoj-phase2.py / download-cendoj-phase2-pw.py
|
Phase 2 crawling with Playwright |
download-boe-legislation.py |
Download BOE legislation texts |
download-borme-section-c.py |
Download BORME corporate filings |
download-aeat-consultas.py |
Tax authority consultations |
download-consejo-estado.py |
Council of State opinions |
download-tribunal-constitucional.py |
Constitutional Court decisions |
download-eurlex-es.py |
EUR-Lex Spanish-language docs |
download-fiscalia.py |
Public prosecutor memoranda |
# Netherlands, Switzerland, Ireland, ICIJ Offshore Leaks -- 4 pipelines in parallel
python3 scripts/opendata/offshore/import-offshore-opendata.py [country] [--workers 50]# Download NIPO open data: trademarks and invention patents (2024-2026)
python3 scripts/opendata/nipo/download_nipo.py# Download acts/addendums/peny from spending.gov.ua -- 25 concurrent workers, 5 IPs
python3 scripts/opendata/spending/download_spending.py [--type acts] [--from 2024-01-01] [--resume]| npm script | Purpose |
|---|---|
process:docs |
Process existing documents (parse, summarize, extract) |
seed:admin |
Create initial admin user |
seed:test-account |
Create test user account for QA |
seed:matters |
Seed matter/case type reference data |
sync:dictionaries |
Sync judicial dictionaries (courts, judges, categories) |
backfill:decisions |
Backfill court decision metadata |
backfill:reyestr |
Backfill fulltext search index from EDRSR |
backfill:tsv |
Backfill TSV (tsvector) columns for full-text search |
discover:court |
Discover new court decisions from EDRSR API |
scrape:court |
Scrape court registry with configurable filters |
load:debt-cases |
Load debt-related court cases |
load:civil-cases |
Load civil property dispute cases |
compute:judge-analytics |
Compute judge performance analytics (statistics) |
test:email |
Test email delivery configuration |
The batch-process-documents.ts script (root-level) orchestrates mass document processing.
npm run batch-process -- --dir /path/to/docs --concurrency 5 --ops parse,summarize,extract_key_clausesSupports operations: parse, summarize, extract_key_clauses. Concurrency is configurable via --concurrency flag.
The unified backfill script runs all data source groups in parallel:
./scripts/backfill-all.sh [local|stage] [--years=2] [--step=all|dictionaries|decisions|legislation|rada-reference|rada-bills|registries]Steps: dictionaries, decisions, legislation, rada-reference, rada-bills, registries. Default runs all steps covering 2 years of data.
| Script | Purpose |
|---|---|
scripts/edrsr/vectorize-cpk-batch.py |
Civil cases: fulltext to VoyageAI voyage-3.5 embeddings, upsert to Qdrant |
scripts/edrsr/vectorize-kupap-batch.py |
Administrative offense cases vectorization |
scripts/rebuild-embeddings-voyage35.js |
Rebuild all embeddings with voyage-3.5 model |
scripts/reembed-qdrant.js |
Re-embed existing Qdrant vectors |
scripts/index-legislation-vectors.js |
Index legislation text as vectors |
| Script | Purpose |
|---|---|
scripts/citation-graph/extract-citations-fast.py |
Extract inter-case citation links from fulltext |
scripts/citation-graph/build-cocitation.py |
Build co-citation network |
scripts/citation-graph/analyze-communities.py |
Community detection in citation graph |
scripts/citation-graph/analyze-degree-centrality.py |
Degree centrality analysis |
scripts/citation-graph/analyze-temporal-dynamics.py |
Temporal evolution of citation network |
scripts/citation-graph/evaluate-precision.py |
Evaluate retrieval precision |
scripts/citation-graph/exp1-citation-vs-dense.py |
Experiment: citation-based vs dense retrieval |
scripts/citation-graph/run_temporal_eval.R |
R: temporal evaluation for arXiv paper |
scripts/citation-graph/06_statute_retrieval_eval.R |
R: statute retrieval evaluation |
| Script | Purpose |
|---|---|
scripts/dataset/build_hf_dataset.py |
Convert JSONL to HF parquet (train/val/test splits) |
scripts/dataset/extract_case_outcome.py |
Extract case outcomes for classification dataset |
scripts/dataset/build_bridge_paper_dataset.py |
Build dataset for bridge paper experiments |
scripts/dataset/upload-v5-to-hf.py |
Upload dataset v5 to HuggingFace Hub |
| Script | Purpose |
|---|---|
scripts/lextreme/extract-dataset.py |
Extract Ukrainian court decisions for LexTreme contribution |
scripts/lextreme/extract-full-local.py |
Full local extraction pipeline |
scripts/lextreme/upload-to-hf.py |
Upload to HuggingFace |
Each service has its own migration runner. Migrations are idempotent (IF NOT EXISTS / ON CONFLICT DO NOTHING).
| Service | Command | Location |
|---|---|---|
| mcp_backend | npm run migrate |
mcp_backend/src/migrations/ |
| mcp_rada | npm run migrate |
mcp_rada/src/migrations/ |
| mcp_openreyestr | npm run migrate |
mcp_openreyestr/src/migrations/ |
npm run db:setup # Create DB + run migrations (any service)The manage-gateway.sh script is the primary entry point for managing containerized environments.
cd deployment
./manage-gateway.sh start local # Start all services
./manage-gateway.sh deploy local # Full rebuild (--no-cache)
./manage-gateway.sh logs local # View service logs
./manage-gateway.sh stop local # Stop all services
./manage-gateway.sh status # Container status
./manage-gateway.sh health # Health check endpoints# Move old EDRSR partitions (2005-2017, ~540 GB) to HDD tablespace
./scripts/migrate-cold-storage.sh [--dry-run]Spins up a temporary EC2 Spot Instance from an EBS snapshot of prod, runs compute-judge-analytics, copies results back, and tears down.
./scripts/ephemeral-analytics-replica.sh # full run
./scripts/ephemeral-analytics-replica.sh --dry-run # show plan only
./scripts/ephemeral-analytics-replica.sh --cleanup # force cleanup orphaned resources| Script | Purpose |
|---|---|
scripts/sync-prod-to-local.sh |
Sync EDRSR fulltext from prod to local via SSH tunnel + pg_dump |
scripts/sync-edrsr-fulltext.sh |
Sync fulltext partitions between environments |
Headless Claude Code invocations for automation:
| Script | Purpose |
|---|---|
scripts/claude/git-housekeeping.sh |
Auto-commit, branch, push, and open PR for pending changes |
scripts/claude/health-check.sh |
Check container health, disk space, memory on current machine |
scripts/pre-deploy-check.sh |
Validate builds, docker-compose config, env vars, port conflicts |
| Script | Purpose |
|---|---|
scripts/testing/load-test-quality.ts |
Load & quality test for chat pipeline (45 tools x 2 queries, Bedrock) |
scripts/testing/run-all-tests.sh |
Orchestrate full test suite across all services |
scripts/testing/run-e2e-tests.sh |
Run Playwright E2E tests |
scripts/testing/test-batch-processing.sh |
Validate batch document processing tool |
scripts/testing/test-chat-tool-coverage.sh |
Verify all chat tools respond correctly |
scripts/testing/test-gateway.sh |
Test unified gateway routing |
scripts/testing/test-all-mcp-logging.sh |
Verify MCP request/response logging |
| Script | Purpose |
|---|---|
scripts/utilities/convert-to-txt.sh |
Convert DOCX/PDF test files to plain text |
scripts/utilities/convert-test-files-to-txt.ts |
TypeScript file conversion utility |
scripts/utilities/clear-minio-buckets.sh |
Clear all user buckets from MinIO storage |
scripts/prod-chat-logs.sh |
Retrieve chat logs from prod (DB + docker logs) by request ID |
scripts/patch-fast-xml-parser.js |
Postinstall patch for fast-xml-parser compatibility |
scripts/generate-*.py |
Blog banner image generators (fractal-based) |
| npm script (lexwebapp) | Purpose |
|---|---|
generate:sitemap |
Generate sitemap.xml for SEO |
test |
Run Vitest unit tests |
build |
Production build with TypeScript check |
| Workflow | Trigger | Purpose |
|---|---|---|
ci-local-deploy.yml |
Push to main | Build, test, deploy to local Docker, health check |
deploy-prod.yml |
After local CI or manual | Blue-green deploy to prod via SSH |
cron-edrsr-sync.yml |
Daily 04:00 Kyiv / manual | EDRSR incremental sync from data.gov.ua |
ci-mobile-release.yml |
Tag push | Build and release mobile app (Flutter) |
All CI runs on a self-hosted runner (local.legal.org.ua). Build-and-test is isolated on a KVM VM; deploy steps run on the host.