PR-03: Rules as data (scanner)

dsgai_scanner_tool/CHANGES_v0.3.md

@@ -17,6 +17,13 @@ dates are ISO-8601. The previous line is recorded in [`CHANGES_v0.2.md`](CHANGES
   internal-link check), a deterministic markdown internal-link checker
   (`scripts/check_md_links.py`), and a `.gitattributes` forcing LF on scripts/YAML so
   Windows checkouts can't ship CRLF that breaks Linux CI. (PR-02)
+- **Rules as data**: all 106 detection patterns extracted verbatim from the skill's
+  Step 2 into `rules/dsgai-rules.yaml` (source of truth), validated by
+  `rules/rules.schema.json`, compiled to `rules/dsgai-rules.json` by
+  `build/build_rules_json.py`. Compound logic (`subtract`, `requires_nearby`,
+  `exclude_globs`, `gated_on`) and per-rule `classification`/`signal`/`confidence` are
+  encoded. `rules/README.md` documents the format. Skill Step 2 now marks the YAML as
+  canonical. (PR-03)
 
 ### Changed
 - `DSGAI-samplereport.png` compressed from ~5.0 MB to ~0.35 MB (14×) as an interim fix;

dsgai_scanner_tool/CONTRIBUTING.md

@@ -28,11 +28,11 @@ make the tool measurably better — a good repro is the contribution.
 
 ## Contributing a rule
 
-Detection rules are moving from prose in `dsgai_scanner_tool.md` into data at
-`rules/dsgai-rules.yaml`, validated by `rules/rules.schema.json`. **(Landing soon —
-tracked by PR-03.)** Until that lands, describe your rule in a
-[new-rule issue](../.github/ISSUE_TEMPLATE/scanner-new-rule.yml) using the format below;
-once the YAML rule format ships, this section will point at `rules/README.md`.
+Detection rules live as data in [`rules/dsgai-rules.yaml`](rules/dsgai-rules.yaml),
+validated by `rules/rules.schema.json` and compiled to `rules/dsgai-rules.json`. See
+[`rules/README.md`](rules/README.md) for the full field reference. You can also propose
+a rule without writing YAML via a
+[new-rule issue](../.github/ISSUE_TEMPLATE/scanner-new-rule.yml).
 
 Every rule needs:
 

dsgai_scanner_tool/build/build_rules_json.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+"""Build rules/dsgai-rules.json from rules/dsgai-rules.yaml.
+
+This is the repeatable build step (unlike the one-time generate_rules.py). The
+deterministic CLI loads the JSON with the standard library only, so the
+curl-one-file install story needs no PyYAML at runtime. PyYAML is required only
+to edit rules and to run this build / the self-test that asserts JSON == YAML.
+
+Validates against rules/rules.schema.json before writing. Output is
+deterministic (sorted keys, stable rule order, trailing newline) so the PR-06
+self-test can assert the checked-in JSON matches a fresh build.
+
+Usage:
+  python build/build_rules_json.py            # write rules/dsgai-rules.json
+  python build/build_rules_json.py --check     # verify checked-in JSON is current (CI)
+"""
+import json
+import sys
+from pathlib import Path
+
+RULES_DIR = Path(__file__).resolve().parent.parent / "rules"
+YAML_PATH = RULES_DIR / "dsgai-rules.yaml"
+JSON_PATH = RULES_DIR / "dsgai-rules.json"
+SCHEMA_PATH = RULES_DIR / "rules.schema.json"
+
+
+def build():
+    import yaml  # only needed for build/edit, not at CLI runtime
+    data = yaml.safe_load(YAML_PATH.read_text(encoding="utf-8"))
+    try:
+        import jsonschema
+        schema = json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))
+        jsonschema.validate(data, schema)
+    except ImportError:
+        sys.stderr.write("warning: jsonschema not installed; skipping validation\n")
+    return json.dumps(data, indent=2, sort_keys=True, ensure_ascii=False) + "\n"
+
+
+def main(argv):
+    rendered = build()
+    if "--check" in argv:
+        current = JSON_PATH.read_text(encoding="utf-8") if JSON_PATH.exists() else ""
+        if current != rendered:
+            sys.stderr.write(
+                "rules/dsgai-rules.json is out of date. Run: "
+                "python build/build_rules_json.py\n")
+            return 1
+        print("rules/dsgai-rules.json is up to date.")
+        return 0
+    JSON_PATH.write_text(rendered, encoding="utf-8", newline="\n")
+    print(f"Wrote {JSON_PATH} ({rendered.count(chr(10))} lines).")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

dsgai_scanner_tool/build/generate_rules.py

@@ -0,0 +1,233 @@
+#!/usr/bin/env python3
+"""One-time bootstrap: extract DSGAI scanner patterns from dsgai_scanner_tool.md
+Step 2 into rules/dsgai-rules.yaml.
+
+Patterns are copied VERBATIM (bug-for-bug faithful — fixes come in PR-11 as
+reviewable diffs). Classification, signal, confidence, and compound logic
+(subtract / requires_nearby / exclude_globs / gated_on / notes) are augmented
+from the skill's prose and the improvement plan's Appendix B seeds.
+
+After this bootstrap runs, rules/dsgai-rules.yaml is the hand-maintained source
+of truth; the skill prose becomes descriptive. Re-running would overwrite manual
+edits, so it is kept only for provenance / audit.
+
+Usage: python build/generate_rules.py > rules/dsgai-rules.yaml
+"""
+import re
+import sys
+from pathlib import Path
+
+SKILL = Path(__file__).resolve().parent.parent / "dsgai_scanner_tool.md"
+VALUE_BEARING_CONTROLS = {2, 13, 14, 15}
+FRAMEWORK = "dsgai-2026-v1.0"
+RULESET_VERSION = "0.3.0"
+
+HEADER_RE = re.compile(r'^### DSGAI(\d{2}) Scan .*\[(STRUCTURAL|VALUE-BEARING)')
+FILES_RE = re.compile(r'^Files:\s*(.+)$')
+# Pattern line: ID  <desc up to first colon>: <pcre to EOL>.  ':' separator is
+# the FIRST colon — no rule description contains an internal colon.
+PAT_RE = re.compile(r'^(P\d{2}\.\d+)\s+(.+?):\s+(.+?)\s*$')
+GLOB_RE = re.compile(r'`([^`]+)`')
+
+# Signal overrides for rules whose FAIL/PASS status comes from prose, not a
+# marker in the description text.
+SIGNAL_OVERRIDE = {
+    "P02.1": "fail", "P02.2": "fail", "P02.3": "fail",
+    "P02.4": "fail", "P02.5": "fail",
+}
+
+# Compound logic and gating, transcribed from the Step 2 prose notes.
+SUBTRACT = {"P04.1": ["P04.2"]}
+REQUIRES_NEARBY = {
+    "P05.1": {"rules": ["P05.2", "P05.3"], "scope": "module"},
+    "P06.5": {"rule": "P06.2", "scope": "module", "absent": True},
+    "P11.1": {"rule": "P11.2", "lines": 15},
+    "P18.4": {"rule": "P18.5", "lines": 10},
+    "P20.5": {"rules": ["P20.1", "P20.2"], "lines": 15},
+}
+EXCLUDE_GLOBS = {
+    "P12.6": ["**/migrations/**", "**/fixtures/**", "**/tests/**", "**/test/**"],
+}
+GATED_ON = {
+    "P09.1": "multimodal", "P09.2": "multimodal", "P09.3": "multimodal",
+    "P09.4": "multimodal", "P09.5": "multimodal",
+    "P10.1": "synthetic_data", "P10.2": "synthetic_data", "P10.3": "synthetic_data",
+    "P10.4": "synthetic_data", "P10.5": "synthetic_data",
+    "P19.1": "labeling", "P19.2": "labeling", "P19.3": "labeling", "P19.4": "labeling",
+}
+# Free-text prose that resists full formalization at import time (refined later).
+NOTES = {
+    "P07.1": "Absence of P07.1-P07.4 in a multi-tenant or PII-handling repo = WARN.",
+    "P08.1": "Absence of P08.1-P08.6 in a production GenAI service = WARN; "
+             "absence in a high-risk EU AI Act use case = FAIL.",
+    "P09.5": "Absence = note only (advanced control); P09.1-P09.4 absence in a "
+             "multimodal pipeline = WARN.",
+    "P10.1": "Synthetic data pipeline without any of P10.1/P10.2/P10.4 = FAIL.",
+    "P12.6": "DDL in migrations/fixtures/tests is benign — excluded from FAIL.",
+    "P14.4": "May contain inline PII in the format string — treated as VALUE-BEARING.",
+    "P16.1": "Filename-existence check (not content grep). Absence in a repo with "
+             ".env or secrets/ = WARN.",
+    "P16.2": "Matched inside any AI-ignore file.",
+    "P17.1": "LLM-calling module with none of P17.1-P17.5 = WARN.",
+    "P21.2": "P21.2 in an agent module without P21.3 = WARN.",
+}
+# P16 rules carry a mode prefix ('filename match:' / 'in any ignore file:') in
+# the pcre slot; strip it to the bare regex.
+MODE_PREFIX_RE = re.compile(r'^(filename match|in any ignore file):\s*')
+
+
+def slugify(text):
+    text = re.sub(r'\([^)]*\)', '', text)          # drop (FAIL)/(PASS)/... markers
+    text = text.lower().strip()
+    text = re.sub(r'[^a-z0-9]+', '-', text).strip('-')
+    return text or "rule"
+
+
+def derive_signal(pid, desc):
+    if pid in SIGNAL_OVERRIDE:
+        return SIGNAL_OVERRIDE[pid]
+    d = desc.lower()
+    if "fail" in d:
+        return "fail"
+    if "warn" in d:
+        return "warn"
+    if "pass" in d:
+        return "pass_signal"
+    if "count" in d:
+        return "count"
+    return "info"
+
+
+def derive_confidence(pid, control, signal, pcre):
+    # Value-bearing credential-literal FAILs are the high-confidence anchors.
+    if control in VALUE_BEARING_CONTROLS and signal == "fail":
+        return "high"
+    if signal == "warn":
+        return "low"          # heuristic / absence-adjacent — weak evidence
+    if signal in ("pass_signal", "count"):
+        return "medium"       # an import is not proof of correct use
+    if signal == "fail":
+        return "medium"       # structural heuristic FAILs (e.g. P12.1, P17.6)
+    return "low"              # bare detection/info
+
+
+def yaml_scalar(s):
+    """Single-quote a scalar for YAML, escaping embedded single quotes."""
+    return "'" + s.replace("'", "''") + "'"
+
+
+def emit_list(vals):
+    return "[" + ", ".join(yaml_scalar(v) for v in vals) + "]"
+
+
+def main():
+    # Ensure UTF-8 output regardless of the platform console codepage (Windows
+    # defaults to cp1252, which corrupts em-dashes / non-ASCII on redirect).
+    try:
+        sys.stdout.reconfigure(encoding="utf-8", newline="\n")
+    except AttributeError:
+        pass
+    lines = SKILL.read_text(encoding="utf-8").splitlines()
+    control = None
+    classification = None
+    file_globs = []
+    in_fence = False
+    rules = []
+    unparsed = []
+
+    # Only scan Step 2 (between its header and Step 3).
+    start = next(i for i, l in enumerate(lines) if l.startswith("## Step 2:"))
+    end = next(i for i, l in enumerate(lines) if l.startswith("## Step 3:"))
+
+    for raw in lines[start:end]:
+        line = raw.rstrip("\n")
+        m = HEADER_RE.match(line)
+        if m:
+            control = int(m.group(1))
+            classification = "value_bearing" if control in VALUE_BEARING_CONTROLS else "structural"
+            file_globs = []
+            in_fence = False
+            continue
+        fm = FILES_RE.match(line)
+        if fm:
+            file_globs = GLOB_RE.findall(fm.group(1))
+            continue
+        if line.strip().startswith("```"):
+            in_fence = not in_fence
+            continue
+        if in_fence and control is not None:
+            pm = PAT_RE.match(line)
+            if not pm:
+                if line.strip():
+                    unparsed.append(line)
+                continue
+            pid, desc, pcre = pm.group(1), pm.group(2).strip(), pm.group(3)
+            pcre = MODE_PREFIX_RE.sub("", pcre).strip()
+            signal = derive_signal(pid, desc)
+            rules.append({
+                "id": pid,
+                "control": f"DSGAI{control:02d}",
+                "name": slugify(desc),
+                "classification": classification,
+                "signal": signal,
+                "confidence": derive_confidence(pid, control, signal, pcre),
+                "pcre": pcre,
+                "file_globs": list(file_globs),
+                "exclude_globs": EXCLUDE_GLOBS.get(pid, []),
+                "framework": FRAMEWORK,
+                "description": desc,
+                "subtract": SUBTRACT.get(pid),
+                "requires_nearby": REQUIRES_NEARBY.get(pid),
+                "gated_on": GATED_ON.get(pid),
+                "notes": NOTES.get(pid),
+            })
+
+    if unparsed:
+        sys.stderr.write("UNPARSED LINES:\n" + "\n".join(unparsed) + "\n")
+
+    # Emit YAML by hand (deterministic ordering, faithful quoting of PCREs).
+    out = []
+    out.append("# DSGAI scanner detection rules — source of truth.")
+    out.append("# Generated once from dsgai_scanner_tool.md Step 2 by build/generate_rules.py,")
+    out.append("# then hand-maintained. Validated by rules/rules.schema.json (see rules/README.md).")
+    out.append(f"ruleset_version: '{RULESET_VERSION}'")
+    out.append(f"framework: '{FRAMEWORK}'")
+    out.append("rules:")
+    for r in rules:
+        out.append(f"  - id: {r['id']}")
+        out.append(f"    control: {r['control']}")
+        out.append(f"    name: {r['name']}")
+        out.append(f"    classification: {r['classification']}")
+        out.append(f"    signal: {r['signal']}")
+        out.append(f"    confidence: {r['confidence']}")
+        out.append(f"    pcre: {yaml_scalar(r['pcre'])}")
+        out.append(f"    file_globs: {emit_list(r['file_globs'])}")
+        out.append(f"    exclude_globs: {emit_list(r['exclude_globs'])}")
+        out.append(f"    framework: '{r['framework']}'")
+        out.append(f"    description: {yaml_scalar(r['description'])}")
+        if r["subtract"]:
+            out.append(f"    subtract: {emit_list(r['subtract'])}")
+        if r["requires_nearby"]:
+            rn = r["requires_nearby"]
+            parts = []
+            if "rule" in rn:
+                parts.append(f"rule: {rn['rule']}")
+            if "rules" in rn:
+                parts.append("rules: " + emit_list(rn["rules"]))
+            if "lines" in rn:
+                parts.append(f"lines: {rn['lines']}")
+            if "scope" in rn:
+                parts.append(f"scope: {rn['scope']}")
+            if rn.get("absent"):
+                parts.append("absent: true")
+            out.append("    requires_nearby: {" + ", ".join(parts) + "}")
+        if r["gated_on"]:
+            out.append(f"    gated_on: {r['gated_on']}")
+        if r["notes"]:
+            out.append(f"    notes: {yaml_scalar(r['notes'])}")
+    sys.stdout.write("\n".join(out) + "\n")
+    sys.stderr.write(f"\nExtracted {len(rules)} rules.\n")
+
+
+if __name__ == "__main__":
+    main()

dsgai_scanner_tool/dsgai_scanner_tool.md

@@ -564,6 +564,8 @@ Also identify:
 
 ## Step 2: Scan for DSGAI Issues
 
+> **Canonical rule definitions live in [`rules/dsgai-rules.yaml`](rules/dsgai-rules.yaml)** (validated by `rules/rules.schema.json`, compiled to `rules/dsgai-rules.json`). The pattern listings in this Step are descriptive — the YAML is authoritative and is what the deterministic CLI executes. When they disagree, the YAML wins. (Full skill rewrite to CLI-first orchestration is PR-07.)
+
 ### Search Engine Prerequisite
 
 All patterns below use **PCRE / Perl-compatible regex syntax** — `\s`, `{n,m}`, character classes inside groups, alternation. Inside Claude Code, the Grep tool (ripgrep) supports this natively. Outside Claude Code, use `rg` (ripgrep) or `grep -P` (GNU grep with PCRE). Plain POSIX BRE/ERE will *not* match `\s`, `\d`, or `{n,m}` correctly and will produce false negatives.

dsgai_scanner_tool/rules/README.md

@@ -0,0 +1,61 @@
+# DSGAI scanner rules
+
+`dsgai-rules.yaml` is the **source of truth** for every detection pattern. It is
+validated by `rules.schema.json` and compiled to `dsgai-rules.json` (which the
+deterministic CLI loads with the standard library only — no PyYAML at runtime).
+
+- **Edit** `dsgai-rules.yaml`, then rebuild the JSON:
+  `python build/build_rules_json.py`
+- **Never edit** `dsgai-rules.json` by hand — it is generated. CI checks it is
+  in sync (`python build/build_rules_json.py --check`).
+
+## Rule schema
+
+```yaml
+- id: P02.1                      # P<control>.<n>, unique
+  control: DSGAI02               # DSGAI01 .. DSGAI21
+  name: hardcoded-openai-api-key # kebab-case slug
+  classification: value_bearing  # structural | value_bearing
+  signal: fail                   # fail | warn | pass_signal | count | info
+  confidence: high               # high | medium | low
+  pcre: '(?i)(OPENAI_API_KEY|...)\s*[:=]\s*["'']?sk-[A-Za-z0-9_\-]{20,}'
+  file_globs: ['*.py', '*.env*'] # which files the rule runs against
+  exclude_globs: []              # paths excluded from this rule
+  framework: dsgai-2026-v1.0     # framework version binding
+  description: 'Hardcoded OpenAI API key assignment'
+```
+
+### Field reference
+
+| Field | Required | Meaning |
+|---|---|---|
+| `id` | yes | `P<NN>.<n>`, matches the control number |
+| `control` | yes | `DSGAI01`–`DSGAI21` |
+| `name` | yes | kebab-case identifier |
+| `classification` | yes | `structural` (match may be shown) or `value_bearing` (match content is a secret/PII — never shown; located in `--replace ''` mode) |
+| `signal` | yes | how a hit is weighted: `fail`, `warn`, `pass_signal`, `count`, `info` |
+| `confidence` | yes | `high` / `medium` / `low` — feeds SARIF `level` and report rendering |
+| `pcre` | yes | PCRE2 pattern; must compile under `rg --pcre2` (checked in CI, not by the schema) |
+| `file_globs` | yes | globs the rule scans |
+| `exclude_globs` | no | globs excluded from the rule (e.g. migrations/tests) |
+| `framework` | yes | `dsgai-YYYY-vX.Y` binding |
+| `description` | yes | human-readable summary |
+| `remediation` | no | fix guidance |
+| `references` | no | CVE IDs / links |
+| `subtract` | no | rule IDs whose match on the same line cancels this hit (e.g. `torch.load(` minus `weights_only=True`) |
+| `requires_nearby` | no | compound proximity logic: `{rule\|rules, lines\|scope, absent}` |
+| `gated_on` | no | only evaluated when the stack is detected: `multimodal`, `synthetic_data`, `labeling` |
+| `notes` | no | control-level prose that isn't yet fully formalized |
+
+## Classification: STRUCTURAL vs VALUE-BEARING
+
+See [`../CONTRIBUTING.md`](../CONTRIBUTING.md#structural-vs-value-bearing). In
+short: if a rule can match a line whose *content is* a secret or PII, it is
+`value_bearing` and runs in location-only mode so the value never leaves
+ripgrep. All rules under DSGAI02/13/14/15 are value-bearing.
+
+## Provenance
+
+The initial ruleset was extracted verbatim from `dsgai_scanner_tool.md` Step 2 by
+`build/generate_rules.py` (a one-time bootstrap, kept for audit). Pattern *fixes*
+land as reviewable diffs against this baseline starting in PR-11.