blog: Test-driven Rego policy generation — catching semantic errors before they ship

[sonupreetam]

Topic

ComplyPack now generates human-reviewable test cases before writing any Rego policy. This addresses a fundamental problem with LLM-assisted policy generation: models produce structurally valid policies that enforce the wrong conditions entirely.

Suggested Angle

The hallucination problem in compliance policy generation. Current validation gates (validate_policy, test_policy) catch syntax errors and wrong input.* paths, but a policy that checks the wrong field or inverts a condition passes all structural checks. The test-driven approach moves human review to the test case layer — practitioners evaluate scenarios in domain terms ("containers must not run as root") rather than Rego syntax.

Key Points

• Why structural validation is necessary but insufficient for semantic correctness
• The test-first pattern: generate test cases from MCP data → human approval → policy generation → tests as specification
• How parameter values from get_assessment_requirements replace hallucinated thresholds
• Single mode (interactive, one requirement at a time) vs. batch mode (all at once)
• The "never modify approved tests" invariant — if tests fail, the policy is revised
• Unique package namespaces enable per-requirement evaluation via conftest --namespace

Reference

• ADR 018: Test-Driven Rego Policy Generation (docs/adr/018-test-driven-rego-generation.md)
• skills/build-assessment/SKILL.md — the orchestrator skill
• Branch: fix/unique-rego-namespaces on complytime/complypack