glenflorendo · Sep 15, 2026 · Sep 14, 2026
diff --git a/docs/01-overview.md b/docs/01-overview.md
@@ -0,0 +1,160 @@
+# Overview
+
+## Mission
+
+**Lucky Parking** is a [Hack for LA](https://www.hackforla.org/) project that helps city planners and community members explore Los Angeles parking citation data and make informed decisions about parking policy.
+
+The repository combines:
+
+- A **web map** for interactive exploration of citations by date and geography
+- A **data pipeline** for ingesting the city's multi-gigabyte citation export and keeping it fresh via API sync
+- A **legacy backend and data-science stack** from earlier project phases
+
+The project is **not complete**. The web app, API, and new data pipeline were built at different times and are not fully integrated. See [Roadmap & open questions](./08-roadmap-and-open-questions.md) for the current gap list.
+
+## Who is this for?
+
+| Audience | Primary entry point |
+|----------|---------------------|
+| Frontend / full-stack developers | [Web application](./03-web-application.md), [Monorepo structure](./02-monorepo-structure.md) |
+| Data / analytics contributors | [Data pipeline](./05-data-pipeline.md), [Data sources & schemas](./07-data-sources-and-schemas.md) |
+| API / infra contributors | [Backend API](./04-backend-api.md), [Monorepo structure](./02-monorepo-structure.md) |
+| New contributors | This page → [Monorepo structure](./02-monorepo-structure.md) → area of interest |
+
+## System at a glance
+
+```mermaid
+flowchart LR
+    subgraph user [User]
+        Browser[Browser]
+    end
+
+    subgraph web [Web tier — active]
+        Next["Next.js app<br/>/parking-insights"]
+        Config["/api/v1/public/config"]
+    end
+
+    subgraph external [External services]
+        Socrata["LA Open Data<br/>Socrata 4f5p-udkv"]
+        Mapbox["Mapbox<br/>maps + geocoding"]
+    end
+
+    subgraph api [API tier — legacy / unused by web]
+        Express["Express API"]
+        MongoDB[(MongoDB)]
+    end
+
+    subgraph data [Data tier — beta pipeline]
+        Pipeline["beta_pipeline<br/>Python + Polars"]
+        SQLite[(SQLite)]
+        PostGIS[(PostGIS)]
+    end
+
+    Browser --> Next
+    Next --> Config
+    Next --> Socrata
+    Next --> Mapbox
+
+    Express --> MongoDB
+    Socrata --> Pipeline
+    Pipeline --> SQLite
+    Pipeline --> PostGIS
+
+    Next -.->|"planned integration"| Express
+```
+
+## Three parallel data paths
+
+The same underlying citation records can flow through three different paths today:
+
+### 1. Web app → Socrata (live, limited)
+
+The Next.js app queries the [Socrata Query API v3](https://dev.socrata.com/docs/queries/) directly. Users pick a date range and up to two geographic areas; the app fetches matching citations and renders them on a Mapbox map.
+
+**Limitation:** Results are capped at **50 rows** per query (pagination not implemented). See [Web application](./03-web-application.md).
+
+### 2. Beta pipeline → SQLite / PostGIS (local analytics)
+
+The `data-science/beta_pipeline/` Python tools stream the ~6 GB CSV into SQLite or PostGIS using Polars batches. SQLite supports **incremental API sync**; PostGIS supports **spatial queries** and a cleaned analytics table.
+
+**Limitation:** PostGIS does not yet have API sync. The web app does not read from these databases. See [Data pipeline](./05-data-pipeline.md).
+
+### 3. Legacy pipeline → normalized PostGIS (older dataset)
+
+The `data-science/src/data/` scripts and Makefile target an **older dataset ID** (`wjz9-h9np`) and a normalized relational schema (`citation`, `vehicle`, `make`, etc.). This path predates the beta pipeline.
+
+**Limitation:** Dataset ID mismatch with current web app and beta pipeline. See [Legacy data science](./06-legacy-data-science.md).
+
+## Technology summary
+
+| Layer | Stack |
+|-------|-------|
+| Monorepo | Turborepo, pnpm workspaces |
+| Web | Next.js 16, React 19, TypeScript, Tailwind 4, Mapbox GL |
+| Shared UI | `@lucky-parking/design` (Radix-based components) |
+| API | Express 4, MongoDB driver, Zod validation |
+| Beta pipeline | Python 3.12, Polars, psycopg, uv (recommended) |
+| Legacy data science | pandas, geopandas, SQLAlchemy, Conda/Makefile |
+| Spatial DB | PostGIS 16 (Docker), SQLite (file) |
+| CI | GitHub Actions (lint, format, build, test) |
+
+## Primary dataset
+
+All current-facing work uses the LA City Open Data dataset **[Parking Citations (`4f5p-udkv`)](https://data.lacity.org/Transportation/Parking-Citations/4f5p-udkv)** — millions of rows, ~6 GB as CSV, updated on a rolling basis by the city.
+
+Details: [Data sources & schemas](./07-data-sources-and-schemas.md).
+
+## Getting started (short)
+
+**Web app only:**
+
+```bash
+pnpm install
+cp apps/web/.env.schema apps/web/.env   # add Mapbox + Socrata tokens
+cd apps/web && pnpm dev
+```
+
+Open `/parking-insights`.
+
+**Data pipeline:**
+
+```bash
+cd data-science/beta_pipeline
+uv venv --python 3.12 .venv
+uv pip install -r requirements.txt jupyter ipykernel --python .venv/bin/python
+```
+
+See [Data pipeline](./05-data-pipeline.md) for full workflows.
+
+## Document map
+
+```mermaid
+mindmap
+  root((Lucky Parking docs))
+    Overview
+      Mission
+      Three data paths
+    Monorepo
+      apps/web
+      apps/api
+      packages
+    Web app
+      Map + filters
+      Socrata queries
+      Zustand state
+    API
+      MongoDB GeoJSON
+      OpenAPI spec
+    Beta pipeline
+      SQLite sync
+      PostGIS clean table
+    Legacy DS
+      Makefile ETL
+      Old dataset ID
+    Data dictionary
+      23 columns
+      Clean schema
+    Roadmap
+      TODOs
+      Integration gaps
+```
diff --git a/docs/02-monorepo-structure.md b/docs/02-monorepo-structure.md
@@ -0,0 +1,186 @@
+# Monorepo structure
+
+Lucky Parking is a [Turborepo](https://turbo.build/) monorepo managed with **pnpm workspaces**. One install at the root pulls dependencies for all apps and packages.
+
+## Directory layout
+
+```
+lucky-parking/
+├── apps/
+│   ├── web/                 # Next.js frontend (primary user-facing app)
+│   └── api/                 # Express REST API (MongoDB)
+├── packages/
+│   ├── design/              # Shared React UI component library
+│   └── configs/             # Shared ESLint, Prettier, TypeScript, Tailwind configs
+├── data-science/
+│   ├── beta_pipeline/       # Modern Polars + SQLite/PostGIS pipeline
+│   ├── src/data/              # Legacy ETL scripts
+│   ├── notebooks/             # Jupyter notebooks (exploratory + archived)
+│   ├── references/            # Lookup tables (makes, violation codes, regex)
+│   ├── db/                    # Legacy PostGIS schema dump
+│   └── docker/                # Conda + Jupyter Lab container
+├── docs/                      # This documentation set
+├── scripts/                   # Shared tooling (e.g. typecheck.mjs)
+├── .github/workflows/         # CI (integration, compliance)
+├── package.json               # Root scripts via Turbo
+├── pnpm-workspace.yaml
+└── turbo.json
+```
+
+## Workspace packages
+
+```mermaid
+graph TB
+    subgraph apps [apps/]
+        WEB["@lucky-parking/web"]
+        API["@lucky-parking/api"]
+    end
+
+    subgraph packages [packages/]
+        DESIGN["@lucky-parking/design"]
+        CONFIGS["@lucky-parking/configs"]
+    end
+
+    WEB --> DESIGN
+    WEB --> CONFIGS
+    API --> CONFIGS
+    DESIGN --> CONFIGS
+```
+
+| Package | Path | Role |
+|---------|------|------|
+| `@lucky-parking/web` | `apps/web` | Next.js map application |
+| `@lucky-parking/api` | `apps/api` | Express citations API |
+| `@lucky-parking/design` | `packages/design` | Buttons, sidebar, calendar, dialog, etc. |
+| `@lucky-parking/configs` | `packages/configs` | Shared lint/format/TS/tailwind presets |
+
+## Turbo task pipeline
+
+Root `package.json` delegates to Turbo:
+
+| Command | Effect |
+|---------|--------|
+| `pnpm install` | Install all workspace dependencies |
+| `pnpm dev` | Start dev servers (persistent, uncached) |
+| `pnpm build` | Build apps/packages; depends on upstream `^build` |
+| `pnpm lint` | ESLint across workspaces |
+| `pnpm format` | Prettier |
+| `pnpm test` | Test task (limited coverage today) |
+| `pnpm check-types` | TypeScript checking |
+
+`turbo.json` defines task dependencies — for example, `build` waits for dependencies' builds and outputs `.next/` or `dist/`.
+
+## Prerequisites
+
+| Tool | Version (as documented) | Used for |
+|------|---------------------------|----------|
+| Node.js | 22+ (root); API Docker uses 22 | Web, API, tooling |
+| pnpm | 9 | Package management |
+| Python | 3.10+ (3.12 recommended) | `beta_pipeline` |
+| uv | latest | Recommended Python env manager |
+| Docker | optional | PostGIS for beta pipeline; legacy Jupyter image |
+
+## Environment files
+
+Each app documents its own env vars via `.env.schema`:
+
+| App / area | Schema file | Key variables |
+|------------|-------------|---------------|
+| Web | `apps/web/.env.schema` | `MAPBOX_ACCESS_TOKEN`, `SOCRATA_APP_TOKEN` |
+| API | `apps/api/.env.schema` | `DB_*`, `API_VERSION`, `COL_NAME_CITATIONS` |
+| Beta pipeline | documented in README | `SOCRATA_APP_TOKEN`, `DATABASE_URL` |
+
+**Note:** `.env` files are gitignored. Copy from `.env.schema` and fill in tokens.
+
+**Known issue:** The API code reads `COL_CITATIONS` but the schema documents `COL_NAME_CITATIONS`. See [Backend API](./04-backend-api.md).
+
+## Git hooks and CI
+
+```mermaid
+flowchart LR
+    Commit[git commit] --> Husky[Husky pre-commit]
+    Husky --> LS[lint-staged]
+    LS --> Prettier
+    LS --> ESLint
+    LS --> Typecheck
+
+    Push[git push / PR] --> GHA[GitHub Actions]
+    GHA --> Format
+    GHA --> Lint
+    GHA --> Build
+    GHA --> Test
+```
+
+- **Pre-commit:** Husky runs lint-staged (Prettier, ESLint, typecheck on staged files)
+- **CI:** `.github/workflows/integration.yaml` — format, lint, build, test on PRs
+- **Compliance:** `.github/workflows/compliance.yaml` — PR/issue linking rules
+
+## Running locally
+
+### Web (most common)
+
+```bash
+pnpm install
+cp apps/web/.env.schema apps/web/.env
+# Edit .env with Mapbox and Socrata tokens
+cd apps/web && pnpm dev
+```
+
+The root route redirects to `/parking-insights` (`apps/web/next.config.ts`).
+
+### API (standalone)
+
+```bash
+cd apps/api
+# Configure MongoDB connection via .env
+pnpm dev   # or see apps/api package.json
+```
+
+The web app does **not** call this API today.
+
+### Data pipeline
+
+See [Data pipeline (beta)](./05-data-pipeline.md). Lives outside the Node workspace but shares the repo.
+
+## What is not in the monorepo
+
+| Item | Location / note |
+|------|-----------------|
+| Citation CSV files | Downloaded locally; gitignored (`raw_data/`, etc.) |
+| SQLite DB files | Generated by pipeline; not committed |
+| Python `.venv` | Created per contributor; gitignored |
+| Production deploy config | Referenced in OpenAPI (`luckyparking.org`) but not fully documented in repo |
+
+## Package boundaries (design intent)
+
+```mermaid
+flowchart TB
+    subgraph presentation [Presentation]
+        WEB[apps/web]
+    end
+
+    subgraph shared [Shared libraries]
+        DESIGN[packages/design]
+    end
+
+    subgraph services [Services — partially used]
+        API[apps/api]
+    end
+
+    subgraph analytics [Analytics — offline]
+        BETA[beta_pipeline]
+        LEG[legacy data-science]
+    end
+
+    WEB --> DESIGN
+    WEB --> Socrata[Socrata API]
+    API --> Mongo[(MongoDB)]
+    BETA --> SQLite[(SQLite)]
+    BETA --> PostGIS[(PostGIS)]
+    LEG --> PostGIS
+
+    WEB -.-> API
+    WEB -.-> BETA
+```
+
+The monorepo currently optimizes for **shared frontend tooling** and **co-located data work**. Full-stack integration (web ↔ API ↔ local DB) remains future work.