Red test on main: test_schema_expansion — and tests/test_workbench.py (29 tests) never runs under make test

[usathyan]

Discovered while working #28. Three related facts, in increasing order of importance.

1. test_schema_expansion fails on main

At 286650f, clean tree:

tests/test_workbench.py::test_schema_expansion
    assert 'COMMITTEE' in ['PARTY', 'CONTRACT', 'OBLIGATION', 'DEADLINE', 'COST', 'VENUE', ...]
FAILED

The test asserts that the contracts schema contains entity types COMMITTEE, PERSON, EVENT, STAGE, ROOM and relation types CHAIRED_BY, RESPONSIBLE_FOR, HOSTED_AT, REQUIRES, SCHEDULED, "per D-20".

domains/contracts/domain.yaml actually has:

• entity_types (9): PARTY, CONTRACT, OBLIGATION, DEADLINE, COST, VENUE, SERVICE, INSURANCE, PENALTY
• relation_types (9): OBLIGATED_TO, HAS_DEADLINE, COSTS, SIGNED_BY, PROVIDES_SERVICE, HELD_AT, REQUIRES_INSURANCE, CROSS_REFERENCES, PENALIZES

So either the D-20 expansion was never applied, or it was reverted and the test was left behind. Someone who knows the D-20 intent should decide whether to apply the schema change or delete the stale assertions — I did not want to guess which.

2. An entire test file never runs under make test

Makefile:test is python -m pytest tests/ -m unit -v. Marker selection across the workbench tests:

$ pytest tests/ -m unit --collect-only | grep test_workbench
tests/test_workbench_security.py::test_xss_sanitization
tests/test_workbench_security.py::test_path_traversal_blocked
tests/test_workbench_security.py::test_role_validation
tests/test_workbench_security.py::test_sri_hashes_present
tests/test_workbench_security.py::test_cors_restricted
tests/test_workbench_security.py::test_sidebar_xss_dom_api

tests/test_workbench_security.py is marked. tests/test_workbench.py is not — none of its 29 tests carry a marker, so make test selects zero of them.

That is the actual reason a red test sat on main unnoticed. It also means the 29 workbench API tests provide no signal in the project's default test command.

Worth auditing every test file for the same gap, not just this one.

3. Docs drift on the contracts domain

CLAUDE.md states:

contracts — 11 entity types, 11 relation types for event/vendor contract analysis

The schema has 9 and 9 (listed above). Either the doc predates a trim, or it describes the unapplied D-20 state. Same root question as item 1.

Suggested resolution order

1. Decide the D-20 question — apply the expansion, or delete the stale assertions.
2. Mark tests/test_workbench.py (and audit the other test files) so make test actually covers them.
3. Reconcile the CLAUDE.md counts with whatever the schema ends up being.

Item 2 is the one that prevents recurrence.

[usathyan]

Found a second red test on main while working #31, plus a sharper number for the marker gap.

Second failing test

tests/test_telegram_bot.py::test_format_welcome_with_starters also fails on clean main:

>       assert "STA" in msg, "Welcome should contain template title"
E       AssertionError: Welcome should contain template title
E       assert 'STA' in 'Welcome to Sample Contract Analysis Workbench!\n\nTry asking:\n...'

The assertion looks stale rather than the code being wrong — the welcome message does contain the template title ("Sample Contract Analysis Workbench"); the test is checking for the literal substring "STA", which nothing produces. Likely a leftover from an earlier title or an acronym scheme that was dropped.

Same call as the D-20 question in the issue body: someone who knows the intent should decide whether the assertion or the code is wrong. Verified pre-existing by stashing and re-running against the base commit.

The marker gap is bigger than one file

$ pytest tests/ -m unit --collect-only
183/347 tests collected (164 deselected)

make test runs 183 of 347 tests — 47% of the suite is deselected. tests/test_telegram_bot.py is also unmarked (0 selected), alongside tests/test_workbench.py.

So this is not "one file was forgotten" — marker coverage is partial across the suite, and both red tests on main happen to sit in unmarked files. That is not a coincidence: unmarked files get no CI signal, so nothing was there to catch them going red.

Reinforces item 2 in the issue body — auditing markers across every test file is the fix that prevents recurrence, and it is worth doing before the D-20 and welcome-title questions, since those are one-line decisions once someone confirms intent.

[chrisdavidson]

Re-verified all three claims against main at a362da7. All still hold. Item 2's audit request is now filed separately as #43 — test_workbench.py turned out to be 1 of 6 affected files, 145 unmarked tests in total.

Two things worth adding to this issue.

The failure depends on the directory you run pytest from

$ pytest tests/test_workbench.py                  # from repo root
1 failed, 30 passed

$ cd docs && pytest ../tests/test_workbench.py
29 passed, 2 skipped                              # green

tests/test_workbench.py:389 builds a CWD-relative path:

schema_path = Path("skills/contract-extraction/domain.yaml")
if not schema_path.exists():
    pytest.skip("contract-extraction domain.yaml not yet created (Plan 01)")

From anywhere but the repo root that exists() returns False, the skip fires, and the failure disappears. Line 378 (domains/contracts/workbench/template.yaml) has the same defect — that file does exist, which is the second skip in the run above.

The skip message also misdiagnoses: the file is present and tracked, it just wasn't found from that CWD. test_workbench.py never imports PROJECT_ROOT, unlike tests/test_unit.py.

Worth fixing independently of the D-20 decision below — it's what makes the test report honestly either way.

Nothing runs these automatically

There is no .github/workflows/, no other CI config in the tree, and GET /repos/usathyan/epistract/commits/a362da7/check-runs returns total_count: 0.

So surfacing this failure needs three things to coincide:

1. test_workbench.py is unmarked, so make test (-m unit) selects zero of its 31 tests — it must be bare pytest or make test-all
2. invoked from the repo root, per the above
3. someone actually looking at the output

That is the real reason a red test sat unnoticed — the marker gap is necessary but not sufficient. It also means #43's suggested collection-time check can't help until something executes it.

D-20: one branch is ruled out

The issue offers "either the D-20 expansion was never applied, or it was reverted and the test was left behind." It's the first:

$ git log --oneline -- domains/contracts/domain.yaml
cd226de v2.0 — Cross-Domain Knowledge Graph Framework (#3)

One commit, the original v2.0 import. The file has never been modified since, so there was no revert. D-20 appears nowhere in the tracked tree except that test's docstring.

The remaining decision is unchanged and still needs someone with the D-20 intent: apply the expansion, or delete the stale assertions.

Minor corrections

• test_workbench.py is 31 tests now, not 29.
• The test reads skills/contract-extraction/domain.yaml while this issue discusses domains/contracts/domain.yaml. Same file — skills/contract-extraction is a tracked symlink (mode 120000) to ../domains/contracts. The attribution is right; noting the indirection so nobody re-derives it.
• Item 3 is still open. PR docs: correct stale pharmacovigilance domain entries #41 corrected the pharmacovigilance domain entries but left the contracts counts. CLAUDE.md states 11 entity / 11 relation types in three places (lines 12, 70, 188); the schema has 9 and 9.

[chrisdavidson]

Splitting this up as its parts get resolved:

• Part 1 (test_schema_expansion red on main) — needs a product decision on D-20, now tracked in its own issue: Decide D-20: apply the contracts schema expansion, or delete test_schema_expansion's assertions #50. It is currently the only red test in the suite.
• Part 3 (marker coverage) — tracked as 145 of 355 tests carry no tier marker — invisible to make test / test-integration / test-e2e #43.

Two other long-standing red tests, not part of this issue, turned out to be stale assertions left behind by renames and are fixed on fix/stale-test-assertions: test_kg_provenance::test_akka_party_referenced_in_response (grepped for akka; the fixture says STA) and test_telegram_bot::test_format_welcome_with_starters (asserted "STA" against a title renamed to "Sample Contract Analysis Workbench"). Full suite went 3 failed → 1 failed.

That leaves part 2 — tests/test_workbench.py (29 tests) never running under make test — as this issue's remaining unique scope.