Is this the right issue type?
Summary
app/airflow currently has no tests at all. No pytest dependency, no test/ directory, no CI job running anything. Every DAG (scan report processing, auto mapping, rules export) has no automated verification.
Most of the actual logic isn't Python we can unit test, it's raw SQL run directly against tables that are owned and migrated by django app.
Shadow of the API's schema with nothing to catch migration changes.
There is a pure Python layer untested: the Excel-parsing helpers in SR_processing/helpers.py, the parameter validation functions in libs/utils.py, and (after separating fetch from shaping) the CSV/JSON-building logic in rules_export/file_services.py. None of this needs a database and all of it is easy to unit test as-is.
Proposed direction:
- Start with the standard Airflow pattern this project doesn't have yet: load the
DagBag and assert every DAG parses with no import errors. Catches broken DAGs before deploy, no infra needed.
- Add pytest and unit test the already-pure logic (Excel helpers, param validation) with plain functions and parametrize.
- Split "fetch data from Postgres" from "shape it into CSV/JSON" in the rules-export code, so the shaping logic can be tested without a DB.
- Separately test the SQL-heavy core: likely a Postgres-backed integration, ideally bootstrapped from the same schema/migrations the API's own tests already use, so drift between the two is caught.
- Get a CI workflow running the fast, DB-free tests as soon as possible
Acceptance Criteria
Confirm creation
Is this the right issue type?
Summary
app/airflowcurrently has no tests at all. No pytest dependency, no test/ directory, no CI job running anything. Every DAG (scan report processing, auto mapping, rules export) has no automated verification.Most of the actual logic isn't Python we can unit test, it's raw SQL run directly against tables that are owned and migrated by django app.
Shadow of the API's schema with nothing to catch migration changes.
There is a pure Python layer untested: the Excel-parsing helpers in
SR_processing/helpers.py, the parameter validation functions inlibs/utils.py, and (after separating fetch from shaping) the CSV/JSON-building logic inrules_export/file_services.py. None of this needs a database and all of it is easy to unit test as-is.Proposed direction:
DagBagand assert every DAG parses with no import errors. Catches broken DAGs before deploy, no infra needed.Acceptance Criteria
Confirm creation