diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..8cd084d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,10 @@ +blank_issues_enabled: true +contact_links: + # TODO(maintainer): replace the OWASP Slack invite URL below with the direct + # invite link to the initiative's channel (e.g. #genai-data-security) once known. + - name: OWASP GenAI Slack (DSGAI scanner discussion) + url: https://owasp.org/slack/invite + about: >- + Questions, scan reports, or general discussion? Join the OWASP Slack and find + the GenAI Data Security Initiative channel. For wrong scanner results, please + still file a [scanner] false-positive / false-negative issue so it becomes a test case. diff --git a/.github/ISSUE_TEMPLATE/scanner-bug.yml b/.github/ISSUE_TEMPLATE/scanner-bug.yml new file mode 100644 index 0000000..c869a6f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/scanner-bug.yml @@ -0,0 +1,53 @@ +name: "[scanner] Bug" +description: A general behavior bug in the DSGAI scanner (report rendering, CI, flags, CLI). +title: "[scanner] Bug: " +labels: ["scanner", "bug"] +body: + - type: markdown + attributes: + value: | + For wrong findings, prefer the false-positive / false-negative templates. + Use this one for everything else: crashes, rendering, CI, flag handling. + - type: textarea + id: what-happened + attributes: + label: What happened + description: Describe the bug and what you expected instead. + validations: + required: true + - type: textarea + id: repro + attributes: + label: Steps to reproduce + description: Exact commands / actions. MUST NOT contain real secrets. + render: text + validations: + required: true + - type: dropdown + id: area + attributes: + label: Area + options: + - Report rendering (HTML) + - CLI (dsgai_scan.py) + - Claude Code skill + - GitHub Action / CI + - Flags / arguments + - Other + validations: + required: true + - type: input + id: version + attributes: + label: Skill / CLI version + placeholder: "0.3.0" + validations: + required: true + - type: textarea + id: context + attributes: + label: Logs / additional context + description: Paste relevant output. Redact anything sensitive. + render: text + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/scanner-false-negative.yml b/.github/ISSUE_TEMPLATE/scanner-false-negative.yml new file mode 100644 index 0000000..5658bb0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/scanner-false-negative.yml @@ -0,0 +1,56 @@ +name: "[scanner] False negative" +description: The DSGAI scanner missed something it should have flagged. +title: "[scanner] False negative: " +labels: ["scanner", "false-negative"] +body: + - type: markdown + attributes: + value: | + Thanks for improving the DSGAI scanner's coverage. Every accepted false-negative + report becomes a permanent positive test case in the public fixture corpus, with credit. + - type: textarea + id: what-missed + attributes: + label: What should have been flagged + description: Describe the issue the scanner failed to detect. + validations: + required: true + - type: textarea + id: snippet + attributes: + label: Minimal reproducing snippet + description: The smallest code that should have triggered a finding. MUST NOT contain real secrets — use obviously fake values. + render: text + validations: + required: true + - type: input + id: expected-control + attributes: + label: Expected control + description: Which DSGAI control should this map to, e.g. `DSGAI02`. + placeholder: DSGAI02 + validations: + required: true + - type: dropdown + id: expected-severity + attributes: + label: Expected severity + options: + - FAIL + - WARN + - PASS signal + validations: + required: true + - type: input + id: version + attributes: + label: Skill / CLI version + description: e.g. `0.3.0`. See the report header or `dsgai_scan.py --version`. + placeholder: "0.3.0" + validations: + required: true + - type: markdown + attributes: + value: | + > Reproducing snippets are added to the public test fixture corpus under the repo license. + > Never paste real credentials — use obviously fake values. diff --git a/.github/ISSUE_TEMPLATE/scanner-false-positive.yml b/.github/ISSUE_TEMPLATE/scanner-false-positive.yml new file mode 100644 index 0000000..2e2e353 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/scanner-false-positive.yml @@ -0,0 +1,73 @@ +name: "[scanner] False positive" +description: The DSGAI scanner flagged something that is not actually a problem. +title: "[scanner] False positive: " +labels: ["scanner", "false-positive"] +body: + - type: markdown + attributes: + value: | + Thanks for improving the DSGAI scanner's precision. Every accepted false-positive + report becomes a permanent negative test case in the public fixture corpus, with credit. + - type: input + id: pattern-id + attributes: + label: Pattern ID + description: The rule that fired, e.g. `P12.1`. Find it in the report next to the finding. + placeholder: P12.1 + validations: + required: true + - type: input + id: control + attributes: + label: DSGAI control + description: The control the rule maps to, e.g. `DSGAI12`. + placeholder: DSGAI12 + validations: + required: false + - type: textarea + id: snippet + attributes: + label: Minimal reproducing snippet + description: The smallest code that triggers the false positive. MUST NOT contain real secrets — use obviously fake values. + render: text + validations: + required: true + - type: textarea + id: reported + attributes: + label: What the scanner reported + description: Paste the finding line(s) from the report or SARIF. + validations: + required: true + - type: textarea + id: why-wrong + attributes: + label: Why it's wrong + description: Explain why this code is safe / why the finding is incorrect. + validations: + required: true + - type: dropdown + id: environment + attributes: + label: Environment + description: How did you run the scanner? + options: + - Claude Code skill (dsgai_scanner_tool.md) + - Deterministic CLI (cli/dsgai_scan.py) + - GitHub Action + - Other tool (prompt variant) + validations: + required: true + - type: input + id: version + attributes: + label: Skill / CLI version + description: e.g. `0.3.0`. See the report header or `dsgai_scan.py --version`. + placeholder: "0.3.0" + validations: + required: true + - type: markdown + attributes: + value: | + > Reproducing snippets are added to the public test fixture corpus under the repo license. + > Never paste real credentials — use obviously fake values. diff --git a/.github/ISSUE_TEMPLATE/scanner-new-rule.yml b/.github/ISSUE_TEMPLATE/scanner-new-rule.yml new file mode 100644 index 0000000..2b9eeaa --- /dev/null +++ b/.github/ISSUE_TEMPLATE/scanner-new-rule.yml @@ -0,0 +1,58 @@ +name: "[scanner] Propose a new rule" +description: Propose a new detection rule for a DSGAI control. +title: "[scanner] New rule: " +labels: ["scanner", "rules"] +body: + - type: markdown + attributes: + value: | + Thanks for extending DSGAI coverage. Accepted rules ship with your test snippets + as permanent fixture cases, with credit. Rules cannot be merged without at least + one positive and one negative test case. + - type: input + id: control + attributes: + label: Proposed control mapping + description: Which DSGAI control does this rule serve, e.g. `DSGAI17`. + placeholder: DSGAI17 + validations: + required: true + - type: textarea + id: pcre + attributes: + label: Proposed PCRE pattern + description: The regex (PCRE2, ripgrep-compatible). Give it as a code block, not a table cell. + render: text + validations: + required: true + - type: dropdown + id: classification + attributes: + label: Classification + description: STRUCTURAL rules match code shape; VALUE-BEARING rules match around secret material and run in location-only mode. + options: + - STRUCTURAL + - VALUE-BEARING + validations: + required: true + - type: textarea + id: positive + attributes: + label: Positive test snippet(s) + description: Code that MUST match. Fake values only. Goes straight into the fixture app on acceptance. + render: text + validations: + required: true + - type: textarea + id: negative + attributes: + label: Negative test snippet(s) + description: Similar-looking code that MUST NOT match. Fake values only. + render: text + validations: + required: true + - type: markdown + attributes: + value: | + > Test snippets are added to the public test fixture corpus under the repo license. + > Never paste real credentials — use obviously fake values. diff --git a/dsgai_scanner_tool/CHANGES_v0.3.md b/dsgai_scanner_tool/CHANGES_v0.3.md new file mode 100644 index 0000000..83867a2 --- /dev/null +++ b/dsgai_scanner_tool/CHANGES_v0.3.md @@ -0,0 +1,20 @@ +# Changelog — DSGAI Scanner Tool v0.3 + +All notable changes for the v0.3 line. This file is appended by every merged PR. +v0.3 releases when Phases 0–2 of the improvement plan are complete. + +Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) loosely; +dates are ISO-8601. The previous line is recorded in [`CHANGES_v0.2.md`](CHANGES_v0.2.md). + +## [Unreleased] + +### Added +- Contributor infrastructure: `[scanner]` GitHub issue-form templates (false-positive, + false-negative, new-rule, bug), scanner `CONTRIBUTING.md`, public `ROADMAP.md`, and + this changelog scaffold. (PR-01) + +### Changed +- _nothing yet_ + +### Fixed +- _nothing yet_ diff --git a/dsgai_scanner_tool/CONTRIBUTING.md b/dsgai_scanner_tool/CONTRIBUTING.md new file mode 100644 index 0000000..09615f8 --- /dev/null +++ b/dsgai_scanner_tool/CONTRIBUTING.md @@ -0,0 +1,94 @@ +# Contributing to the DSGAI Scanner Tool + +The DSGAI scanner is a Claude Code skill (plus a deterministic CLI, landing across +v0.3) that audits GenAI applications against the 21 [OWASP GenAI Data Security +controls](https://genai.owasp.org/resource/owasp-genai-data-security-risks-mitigations-2026/). +It gets more useful every time someone runs it on a real repo and tells us what it +got wrong. That is the single most valuable thing you can do here. + +> This file covers the **scanner subproject** (`dsgai_scanner_tool/`). For the wider +> initiative, see the [repository-root CONTRIBUTING](../CONTRIBUTING.md). + +## Fastest way to help + +1. Run the scan on one of your own GenAI repos. +2. When it flags something wrong, or misses something real, open an issue with the + [scanner templates](../.github/ISSUE_TEMPLATE/): + - **[scanner] False positive** — it flagged safe code. + - **[scanner] False negative** — it missed a real problem. + - **[scanner] Propose a new rule** — a control needs coverage it doesn't have. + - **[scanner] Bug** — anything else (rendering, CLI, CI, flags). + +**Every accepted false-positive / false-negative report becomes a permanent test case +in the public fixture corpus, credited to you.** You don't have to write any code to +make the tool measurably better — a good repro is the contribution. + +> Never paste real credentials into an issue or a fixture. Use obviously fake, +> canonical values (e.g. `sk-proj-FAKE00000000000000000000000000`). + +## Contributing a rule + +Detection rules are moving from prose in `dsgai_scanner_tool.md` into data at +`rules/dsgai-rules.yaml`, validated by `rules/rules.schema.json`. **(Landing soon — +tracked by PR-03.)** Until that lands, describe your rule in a +[new-rule issue](../.github/ISSUE_TEMPLATE/scanner-new-rule.yml) using the format below; +once the YAML rule format ships, this section will point at `rules/README.md`. + +Every rule needs: + +- **A control mapping** — one of `DSGAI01`–`DSGAI21`. +- **A PCRE pattern** — PCRE2 syntax, runs under `rg --pcre2`. Give it in a fenced code + block, never a markdown table cell (`\|` becomes a literal pipe when the file is read + back and silently breaks alternation). +- **A classification** (see below). +- **At least one positive and one negative test snippet.** No exceptions — a rule + without a negative case has no defined precision. + +### STRUCTURAL vs VALUE-BEARING + +This classification controls *how* a rule executes and *what may appear in a report*. + +| Class | The match is… | Report may show the matched line? | Execution | +|---|---|---|---| +| **STRUCTURAL** | a code shape — a call, a missing import, a missing decorator, an architectural gap. Contains no runtime secret. | Yes | normal `rg` match | +| **VALUE-BEARING ⚠️** | a line whose *content is* a credential, key, secret, connection string, or PII. | **Never** — not in the report, checkpoint, or any persisted tool call, even if the value looks fake. | location-only: `rg -n -o --replace '' --pcre2` so ripgrep erases the match before emitting `path:line:` | + +The four VALUE-BEARING controls today are **DSGAI02, DSGAI13, DSGAI14, DSGAI15**. If +your rule matches lines that could contain secret material, it is VALUE-BEARING — when +in doubt, classify up. + +### Running the self-test locally + +Once the test infrastructure lands (PR-04/PR-05/PR-06): + +```bash +pip install pyyaml pytest +pytest dsgai_scanner_tool/tests/ -q +``` + +The suite compiles every rule's PCRE, scans the fixture app, and checks the result +against `tests/expected-findings.yaml`. **Rule PRs are only safely reviewable once this +infrastructure exists** — if you draft a rule PR before then, that's welcome, just +expect it to sit until the fixtures land (see the `blocked-on-phase-1` label). + +## Code contributions + +- **Branch naming:** `improve/pr-NN-short-slug` for planned work, or a short descriptive + slug otherwise. Never commit to `main`. +- **Conventional commits:** `fix:`, `feat:`, `docs:`, `ci:`, `test:`, `refactor:`. +- **Shell scripts** must be `shellcheck` clean and portable (bash 3.2 / macOS included — + no `mapfile`, `readarray`, `${var,,}`, or GNU-only `grep` flags). +- **CI must be green** — the scanner has path-filtered lint and self-test workflows. + +## What we will NOT merge + +The scanner has deliberate limits that keep it maintainable and trustworthy. See the +[Non-goals section of the README](README.md#non-goals) — please read it before +proposing large additions, so your effort lands somewhere we can accept it. + +## Recognition + +Contributors are credited by handle in every release note. **Recurring contributors are +invited to become co-maintainers** — this project explicitly wants to grow its +maintainer base, and consistent, high-quality contributors are how that happens. If you +land a few accepted issues or PRs, expect an invitation. diff --git a/dsgai_scanner_tool/ROADMAP.md b/dsgai_scanner_tool/ROADMAP.md new file mode 100644 index 0000000..1632360 --- /dev/null +++ b/dsgai_scanner_tool/ROADMAP.md @@ -0,0 +1,68 @@ +# DSGAI Scanner Tool — Roadmap + +This is the public roadmap for the DSGAI scanner. It turns the internal improvement +plan into work anyone can pick up. + +> **Want one of these? Comment on the tracking issue and claim it.** Each unstarted +> item below links to a GitHub issue. Rule and precision work is labelled +> `blocked-on-phase-1` until the test corpus and self-test CI land — you can draft it +> now, it just can't merge until then. + +Status legend: ✅ done · 🚧 in progress · ⬜ not started + +## Phase 1 — Trust foundation (determinism, tests, schemas) + +A compliance report that changes run-to-run on identical input is an opinion, not +evidence. This phase makes pattern matching deterministic and testable, so everything +after it is verifiable. + +- ⬜ **Rules as data** — extract every detection pattern into `rules/dsgai-rules.yaml` + with a JSON Schema, so rules are reviewable data instead of prose. +- ⬜ **Fixture app + known-answer sheet** — a small, intentionally vulnerable + multi-language GenAI app that is the test corpus, the demo, and the contributor + on-ramp. +- ⬜ **Deterministic runner** — a stdlib Python CLI that runs the rules via ripgrep and + emits findings JSON + SARIF, identically every run. +- ⬜ **Checkpoint schema + self-test CI** — a formal, redaction-checkable output schema + and the CI gate that makes external rule PRs safe to merge. + +## Phase 2 — Secure the pipeline itself + +A security scanner that is itself a prompt-injection vector or a secret-leak channel is +a liability. This phase closes those. + +- ⬜ **Skill rewrite** — the LLM becomes the orchestrator; the deterministic engine owns + pattern matching. Adds an untrusted-content trust preamble, structural secret + redaction, stable file IDs, and honest language about what the report guarantees. +- ⬜ **Harden the GitHub Action** — split scanning (no secrets, runs on forks) from + narration (restricted tools), pin actions by SHA, and fix the push-gate behaviour. +- ⬜ **README truth pass + lighter sample image** — every claim matches reality; the + ~5 MB sample screenshot is replaced with a small one from the public fixture app. +- ⬜ **Pre-commit: gitleaks rule pack** — ship a battle-tested gitleaks pack, keep a + portable no-dependency fallback script. + +## Phase 3 — Precision + +- ⬜ **Pattern precision wave 1** — fix confirmed false positives/negatives (unquoted + `.env` keys, the innocent-webhook SQL false positive), add per-rule confidence levels. +- ⬜ **CVE pipeline rework** — move CVE fetching into the CLI (no hallucinated CVEs), + add caching, inline suppressions with reasons, a baseline for CI gating, and + incremental `--diff` scans. + +## Phase 4 — Professional polish + +- ⬜ **License split + SPDX headers** — content stays CC BY-SA 4.0; executable code + moves to Apache-2.0 (pending OWASP leadership sign-off). +- ⬜ **Single-source variants + templated report + static ATLAS map** — generate the + tool-neutral prompt variant from the skill, render reports deterministically from a + template, and ship a static MITRE ATLAS technique map. +- ⬜ **Ecosystem expansion** — C#/NuGet, Rust, Ruby coverage, plus Semgrep and gitleaks + rule-pack exports so incumbent toolchains carry the DSGAI framework. +- ⬜ **Benchmark + published precision report** — run against public GenAI repos, + hand-label, and publish per-rule precision. This is what separates "interesting + project" from "tool AppSec teams adopt". + +--- + +*The roadmap follows the initiative's improvement plan. Checkboxes are updated as work +merges; each item's tracking issue carries the detail and the discussion.*