Repository navigation
Add PostGIS citation database, contract API, and explorer UI - #757
Merged
Merged
Conversation
- Enhanced `parking_clean.py` by adding `drop_incomplete` function to filter out rows with missing `issue_datetime` or `loc_lat`. - Updated `rebuild_clean` to invoke `drop_incomplete` post datetime creation. - Modified `parking_pipeline.py` to integrate the dropping of incomplete rows into the overall pipeline. - Adjusted `parking_postgis.py` to ensure `drop_incomplete` is executed after the cleaned table is rebuilt.
…lace points - Updated `datacontract.yaml` to include new valid values: "Neighborhood" and "City Council District". - Modified `Dockerfile` to load additional GeoJSON files for council districts, neighborhoods, and places. - Refactored `02_load_boundaries.sh` to streamline loading of boundary polygons and place points, adding functions for better code organization and including new data sources.
- Updated `docker-compose.yml` to include `SKIP_CITATIONS_LOAD` environment variable for controlling CSV loading on first boot. - Modified `Dockerfile` to install additional Python dependencies and include new scripts for loading contract citations. - Added `03_load_citations.sh` script to handle loading of parking citations from CSV files. - Introduced FastAPI application in `api/main.py` for querying parking citation data. - Created `query_contract.py` CLI for executing data-contract queries against the PostGIS database. - Implemented error handling and validation in the new API and CLI. - Added tests for FastAPI routes to ensure functionality and reliability.
…API features - Added a portable boundary loader script (`load_boundaries.sh`) to streamline the loading of GeoJSON files into the database. - Updated `02_load_boundaries.sh` to utilize the new loader, ensuring all boundary tables are populated on first container start. - Enhanced `README.md` with detailed instructions on the new loading process and added scripts for boundary checks. - Updated `requirements.txt` to include additional dependencies for improved functionality. - Introduced new methods in the API for region suggestions and fetching citations based on geographic regions.
- Introduced Leaflet CSS and JS files for enhanced map functionality. - Added marker and layer images to support map features. - Integrated Leaflet assets into the project to facilitate interactive map rendering.
…etup - Added a new "Quick start" section detailing the steps to build and load the PostGIS database using Docker. - Included commands for preflight checks, starting the PostGIS service, and setting up Python tooling for local exploration. - Provided default database connection details for user convenience.
…ionality - Updated the citation explorer UI to support a new "Compare mode" allowing users to select and compare two regions side by side. - Modified the backend API to handle requests for both single-region and compare mode, including validation for input regions. - Enhanced the JavaScript to toggle between single and compare modes, updating the UI accordingly. - Updated the CSS to style the new mode switch buttons and layout for comparison results. - Revised the README.md to reflect the new features and usage instructions for the citation explorer.
- Introduced `docker-compose.prod.yml` for production deployment of PostGIS, API, and explorer UI. - Created `.env.example` to provide a template for environment variables required in production. - Updated `Dockerfile` and `Dockerfile.api` to support production builds with necessary dependencies. - Enhanced `README.md` with detailed instructions for building and deploying the application on a VPS. - Added deployment documentation for IONOS and general VPS setups, including steps for database initialization and restoration. - Implemented health checks for services to ensure proper startup and readiness. - Removed obsolete `postgisdb.zip` file to streamline the project structure.
…ity with host bind-mounts
- Enhanced the "Quick start" section with a direct link to the Parking Citations dataset on the LA Open Data Portal. - Revised instructions for downloading the citations CSV to specify the required format and source for clarity.
Neither app could be shared with others: no authentication, no TLS, and
no abuse limits. Add credentials, put Caddy in front, and stop leaking
database internals in error responses.
Access control:
- API requires X-API-Key against the comma-separated API_KEYS, compared
in constant time. One key per consumer, so revoking one does not
disturb the rest.
- Explorer requires HTTP basic auth (WEB_USER / WEB_PASSWORD).
- Both fail closed: no credentials configured means 503, not open
access. ALLOW_UNAUTHENTICATED=1 opts out for localhost only.
- /docs and /openapi.json are off unless API_DOCS_PUBLIC=1.
TLS:
- Add a caddy service that terminates TLS with automatic Let's Encrypt
certificates; it is the only service publishing to the internet.
- Bind api/web to 127.0.0.1 in prod so they are reachable for local
curl checks but not from outside.
Abuse and query limits:
- Per-key sliding-window rate limiter (60/min API, 30/min explorer),
falling back to client IP when unauthenticated.
- Connections carry a 5s connect timeout and a 15s statement_timeout.
Without the latter one wide spatial query can hold a connection until
max_connections is exhausted.
Error handling:
- Classify exceptions in one place and return fixed strings. psycopg
errors are not QueryError subclasses, so an unreachable database
previously escaped the handler and returned Starlette's plain-text
500; matched explicitly now, with tests using real driver exceptions.
- Normalize FastAPI's own 422 body to the documented {"error": ...}
shape so clients handle one error format instead of two.
- Full detail goes to the container log, never to the client.
Credentials:
- Remove the "changeme" default from the Dockerfile, the service DSN,
and both compose files; POSTGRES_PASSWORD is now required.
- scripts/ensure_env.sh writes a .env with a generated password so
local setup stays one command, and gen_secrets.sh prints production
credentials.
- Redact the password from the citation loader's progress output.
Also: run the API image as non-root, pin dependencies via
constraints.txt for reproducible builds, gitignore dumps/*.dump, and
set pythonpath in pytest.ini.
Co-authored-by: Cursor <cursoragent@cursor.com>
Repeated runs on one machine hid first-run and cross-platform bugs: a populated volume, a warm image cache, and an existing .env all mask steps that would fail from scratch. Line endings (would have broken Windows outright): - Add a .gitattributes scoped to postgis_db pinning *.sh, *.sql, and *.yml to LF. Git for Windows defaults to core.autocrlf=true, and the image COPYs init/*.sh plus load_boundaries.sh into a Linux container, where a trailing \r fails as "$'\r': command not found". - Normalize 16 files that already had CRLF committed, including scripts/prod_restore.sh, which would have failed on the VPS on any platform. - Strip CR in the Dockerfile before chmod as a second line of defence. PowerShell 5.1 (the version that ships with Windows): - ensure_env.ps1 and gen_secrets.ps1 used RandomNumberGenerator::Fill, which is .NET Core 2.1+ only, so they would have crashed under the Windows PowerShell 5.1 that the .cmd shims invoke. Use RandomNumberGenerator::Create().GetBytes() instead, which works on both 5.1 and 7.x. - Write .env without a BOM; 5.1's Set-Content -Encoding utf8 adds one and Compose does not strip it. - Keep .ps1 files ASCII-only, since 5.1 reads BOM-less UTF-8 as ANSI. Verifying an install: - Add scripts/smoke_test.sh/.ps1/.cmd: one command checks containers, every boundary table, a real spatial query, and both web apps, printing PASS/FAIL per check and exiting non-zero on failure. The chart window is derived from the data because the contract caps a range at 3660 days and CSV years differ between dumps. - Add CITATIONS_LOAD_LIMIT so first boot can load a sample and finish in minutes. The multi-hour full load is the reason a clean end-to-end run was never practical. Also: read POSTGRES_USER/DB from .env in the db-status scripts instead of hardcoding them, and exclude .venv (~260 MB), .git, tests, and dumps from the Docker build context. README documents both this work and the preceding security changes: a clean-room checklist, Windows-specific hazards, the empty-volume reset trap, and the POSTGRES_PASSWORD upgrade note. Co-authored-by: Cursor <cursoragent@cursor.com>
A fresh build of the postgis target fails:
E: Failed to fetch .../libpython3.9-stdlib_3.9.2-1+deb11u7_amd64.deb
404 Not Found
E: Failed to fetch .../python3-pip_20.3.4-4+deb11u2_all.deb 404 Not Found
E: Unable to fetch some archives
exit code: 100
postgis/postgis:16-3.5 is built on Debian 11 (bullseye), which reached
end-of-LTS on 2026-08-31. Its .deb files have since been withdrawn from
deb.debian.org, but the index served by apt-get update still advertises
them, so update succeeds and the install then 404s on every Python
package. gdal-bin comes from bullseye/main and still resolves, which is
why the failure looks Python-specific.
This was not visible locally because a warm layer cache skips the RUN
entirely. Only a --no-cache build, or a machine that never built the
image, hits it - so it broke for a new contributor while continuing to
work here.
Take packages from archive.debian.org, which still serves bullseye:
- Drop bullseye-security. archive.debian.org has no such suite, and
leaving it in makes apt-get update exit non-zero, which now fails the
build since the step runs under `set -eux`.
- Archived Release files are past Valid-Until, so disable that check.
- apt.postgresql.org is a separate live repo and is left untouched.
Staying on bullseye is deliberate. postgis/postgis publishes no
bookworm/trixie build for PostGIS 3.5 on PG16; 17-3.5 is bullseye too,
and the only current option, 18-3.6, is Debian 13 but would force a
PostgreSQL major upgrade that existing data directories cannot be read
across. Moving there needs a pg_upgrade or dump/restore and should be
its own change.
Verified with `docker compose build --no-cache postgis`, then a fresh
container on a throwaway volume: healthy in 30s, PostGIS 3.5, and all
five boundary layers loaded via ogr2ogr (99 councils, 313 zipcodes, 15
districts, 114 neighborhoods, 257 places). polars 1.36.1 and psycopg
3.2.13 install and import.
Also document the failure in the README, including that a warm cache
hides it and that --no-cache is how to check what a new contributor gets.
Co-authored-by: Cursor <cursoragent@cursor.com>
glenflorendo
self-requested a review
September 15, 2026 18:28
glenflorendo
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
data-science/postgis_db/, a self-contained PostGIS deployment that turns the raw LA Parking Citations flat fileinto a queryable spatial database, plus two applications on top of it:
datacontract.yaml, so request/response schemas aregenerated from the contract rather than hand-maintained alongside it.
spreadsheet, in either single-region or compare mode.
Regions can be resolved five ways: ZIP code, neighborhood, neighborhood council, city council district, and an arbitrary
radius around a point. Boundary layers ship as GeoJSON and shapefiles and load automatically on first container start.
Operationally:
docker compose upbuilds the image, loads boundaries, ingests citations, and starts both apps.Citations load into a staging table and are promoted with an atomic swap, so a failed or partial load never leaves the
served table in a mixed state. Credentials are required rather than defaulted, the API authenticates with per-consumer
keys, the explorer uses HTTP basic auth, both are rate limited, and
docker-compose.prod.ymlputs Caddy in front forautomatic TLS. Database errors are logged server-side and returned as fixed generic strings so connection details and
SQL fragments never reach a client.
data-science/.gitignoregains a rule forpostgis_db/dumps/*, which holds multi-GBpg_dumpfiles staged for restore.Related Issues
Refs #695
This does not fully close #695 — it builds the destination and serving layer, not the scheduled ingestion framework
the issue describes. Against that issue's acceptance criteria:
init, but nothing schedules it. There is no cron, Airflow, or Prefect component.
plus atomic swap. There is no watermark, no upsert, and no retry logic.
pipeline health or load runs.
Happy to split those remaining criteria into their own issues, or to retarget this PR if you would rather track the
warehouse/serving layer separately from ingestion.
Testing
Unit/integration: 39 tests via
pytest, covering the contract models, region resolution, API handlers, and theauth/rate-limit/error-sanitization paths. Database-error tests use real
psycopgexceptions rather than mocks, afterfinding that an unreachable database bypassed the error handler and returned a plain-text 500 instead of a sanitized
503.
End-to-end:
scripts/smoke_test.sh(also.ps1/.cmd) checks container health, the PostGIS extension, row countsfor every boundary table, a real spatial chart query, and both web apps, printing PASS/FAIL per check and exiting
non-zero on failure.
Config: both
docker compose configand the prod overlay validate.Clean-room: verified from an empty volume on macOS. Set
CITATIONS_LOAD_LIMITto load a sample so a full runcompletes in minutes instead of hours. Windows was specifically targeted, since Git for Windows defaults to
core.autocrlf=trueand the image copies shell scripts into a Linux container where a trailing\rfails; a scoped.gitattributespins those files to LF and the Dockerfile strips CR as a second line of defence.Checklist
changes or issues.
Two boxes left unchecked deliberately:
apps/*andpackages/*, sopnpm verifydoes not reach these files. Tell me the preferred Python formatter and I will apply it; the code iscurrently Black-compatible.
and I did not want to claim otherwise.
Two notes for reviewers:
docs/CONTRIBUTING.mdsays to branch from and targetdev, but that branch does not exist, sothis targets
main. And the 119 files include boundary shapefiles and vendored Leaflet assets, so the reviewablesurface is much smaller than the diff stat suggests.