Skip to content

Hardening and observability - #1

Merged
nathan-v merged 1 commit into
mainfrom
hardening
Jul 13, 2026
Merged

Hardening and observability#1
nathan-v merged 1 commit into
mainfrom
hardening

Conversation

@nathan-v

Copy link
Copy Markdown
Owner

This is a hardening and observability pass: it closes several input-validation/security gaps (log injection, decompression-bomb-style oversized images, unbounded frame decoding, unbounded camera creation), adds a documented path for injecting secrets via environment variables instead of YAML, and significantly expands the Grafana dashboard with summary stats and better-styled panels.

  • Security hardening

    • src/clockd/utils/log.py (new): sanitize_for_log() strips control characters/newlines from untrusted strings before logging, preventing log injection; applied to uploaded filenames in src/clockd/routers/process.py.
    • src/clockd/routers/calibrate.py: rejects decoded calibration images wider/taller than 16384px (MAX_IMAGE_DIM), guarding against decompression-bomb-style images that are small on disk but huge once decoded.
    • src/clockd/services/pipeline.py: the frame-decode loop now also enforces MAX_FRAMES directly (not just via container metadata validation), since metadata can under-report actual frame count; adds a warning and stops cleanly at the cap.
    • src/clockd/routers/cameras.py + src/clockd/config.py: new max_cameras server setting (default 50) caps how many cameras can be created via POST /cameras, returning 409 once the limit is reached.
  • Secrets via environment variables

    • src/clockd/config.py: ServerConfig now supports nested env vars (env_nested_delimiter="__") and overrides settings_customise_sources so env vars take precedence over server.yaml, letting secrets (NVR credentials, InfluxDB tokens) be injected at deploy time (e.g. Kubernetes Secrets) instead of living in the config file.
    • configs/server.yaml and README.md document the new env-var override pattern and example variable names.
    • deploy/k8s-example.yaml (new): a full hardened Kubernetes Deployment example (secrets via env vars, non-root, read-only root filesystem, dropped capabilities).
  • Grafana dashboard (grafana/clockd-dashboard.json): adds four new stat panels (Total Detections, Vehicles >25mph, Average Speed, Top Speed), restyles the speed timeseries (smoothed lines, raw-point overlay, red threshold shading above 25mph), reworks several panels (renamed "Vehicles Detected" → "Detections", "HTTP Requests" → "Requests/sec" with proper reqps unit), and hides legends across timeseries panels for a cleaner layout.

  • Tests: new/updated coverage for the oversized-image rejection, frame cap enforcement, env-var/YAML precedence and nested secret merging, and camera-limit enforcement (tests/test_calibrate.py, tests/test_pipeline_integration.py, tests/test_config.py, tests/test_api_cameras.py (new), tests/test_log_utils.py (new)).

  • Files changed: 12 tracked (479 insertions, 69 deletions) + 4 new untracked files (deploy/k8s-example.yaml, src/clockd/utils/log.py, tests/test_api_cameras.py, tests/test_log_utils.py)

…alidation/security gaps (log injection, decompression-bomb-style oversized images, unbounded frame decoding, unbounded camera creation), adds a documented path for injecting secrets via environment variables instead of YAML, and significantly expands the Grafana dashboard with summary stats and better-styled panels.

- **Security hardening**
  - `src/clockd/utils/log.py` (new): `sanitize_for_log()` strips control characters/newlines from untrusted strings before logging, preventing log injection; applied to uploaded filenames in `src/clockd/routers/process.py`.
  - `src/clockd/routers/calibrate.py`: rejects decoded calibration images wider/taller than 16384px (`MAX_IMAGE_DIM`), guarding against decompression-bomb-style images that are small on disk but huge once decoded.
  - `src/clockd/services/pipeline.py`: the frame-decode loop now also enforces `MAX_FRAMES` directly (not just via container metadata validation), since metadata can under-report actual frame count; adds a warning and stops cleanly at the cap.
  - `src/clockd/routers/cameras.py` + `src/clockd/config.py`: new `max_cameras` server setting (default 50) caps how many cameras can be created via `POST /cameras`, returning 409 once the limit is reached.
- **Secrets via environment variables**
  - `src/clockd/config.py`: `ServerConfig` now supports nested env vars (`env_nested_delimiter="__"`) and overrides `settings_customise_sources` so env vars take precedence over `server.yaml`, letting secrets (NVR credentials, InfluxDB tokens) be injected at deploy time (e.g. Kubernetes Secrets) instead of living in the config file.
  - `configs/server.yaml` and `README.md` document the new env-var override pattern and example variable names.
  - `deploy/k8s-example.yaml` (new): a full hardened Kubernetes Deployment example (secrets via env vars, non-root, read-only root filesystem, dropped capabilities).
- **Grafana dashboard** (`grafana/clockd-dashboard.json`): adds four new stat panels (Total Detections, Vehicles >25mph, Average Speed, Top Speed), restyles the speed timeseries (smoothed lines, raw-point overlay, red threshold shading above 25mph), reworks several panels (renamed "Vehicles Detected" → "Detections", "HTTP Requests" → "Requests/sec" with proper `reqps` unit), and hides legends across timeseries panels for a cleaner layout.
- **Tests**: new/updated coverage for the oversized-image rejection, frame cap enforcement, env-var/YAML precedence and nested secret merging, and camera-limit enforcement (`tests/test_calibrate.py`, `tests/test_pipeline_integration.py`, `tests/test_config.py`, `tests/test_api_cameras.py` (new), `tests/test_log_utils.py` (new)).

- Files changed: 12 tracked (479 insertions, 69 deletions) + 4 new untracked files (`deploy/k8s-example.yaml`, `src/clockd/utils/log.py`, `tests/test_api_cameras.py`, `tests/test_log_utils.py`)
@nathan-v nathan-v self-assigned this Jul 13, 2026
@nathan-v
nathan-v merged commit ac501bd into main Jul 13, 2026
3 checks passed
@nathan-v
nathan-v deleted the hardening branch July 13, 2026 19:03
nathan-v added a commit that referenced this pull request Jul 14, 2026
## Summary

Fixes the production CrashLoopBackOff that took Clockd offline for ~24h.

Kubernetes injects legacy Docker-link env vars for every Service in a
namespace: a Service named `clockd` produces
`CLOCKD_PORT=tcp://<cluster-ip>:8000` in every pod. Since #1 made env
vars override YAML config (for Secret-based credential injection), that
injected value beat the ConfigMap's `port: 8000` and failed integer
validation, crashing the app at startup on every restart.

## Changes

- **`src/clockd/config.py`**: wrap the env settings source to drop
`tcp://…` values for the `port` field. Real integer `CLOCKD_PORT` values
still override YAML as intended.
- **`deploy/k8s-example.yaml`**: set `enableServiceLinks: false` (with
an explanatory comment) so the example manifest doesn't reproduce the
collision. This is also the mitigation applied to the live deployment.
- **`tests/test_config.py`**: three regression tests — service-link
value ignored with YAML port, without YAML port, and a legitimate env
port still winning.

## Testing

- 151 passed locally (`pytest -q`), ruff check + format clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant