Skip to content

quality_and_standards.md

Latest commit

 

History

History
170 lines (123 loc) · 12.4 KB

quality_and_standards.md

File metadata and controls

170 lines (123 loc) · 12.4 KB
Raw

Engineering Standards & Quality Gates

Authoritative Operational Directives & Software Engineering Discipline
Repository: graft | Platform: Extensible Detection-as-Code Platform
Target Python: >= 3.13 | Package Manager: uv (Native uv_build)
Discipline: Strict TDD, Hexagonal Architecture, Stdlib-First, High Signal-to-Noise

1. How Hexagonal Architecture, AST Gates, and TDD Work Together

In many projects, architectural rules ("don't import engine code in Core", "don't add unapproved dependencies", "keep schemas and dataclasses in sync") exist only in documentation and gradually erode during fast iteration. Graft turns its core design principles into executable, sub-second unit tests that run automatically inside the Test-Driven Development (TDD) loop:

flowchart LR
    TDD["<b>1. Strict TDD Loop</b><br/>Red → Green → Refactor<br/>(tests/unit/ &amp; tests/engines/)"]
    AST["<b>2. AST Boundary &amp; Supply-Chain Gate</b><br/>Python <code>ast</code> parses every .py file<br/>Enforces Hexagonal &amp; Stdlib-First imports"]
    COMP["<b>3. Completeness &amp; Round-Trip Gate</b><br/>JSON Schema ↔ Frozen Dataclass<br/>Lossless load → dump → load"]
    ADV["<b>4. Adversarial Input Gate</b><br/>Malformed YAML, bad UTF-8 &amp; CLI errors<br/>Typed exceptions &amp; strict --json stdout"]

    TDD --> AST --> COMP --> ADV
Loading

1. What Is AST (ast) Static Analysis & Why Use It?

An Abstract Syntax Tree (AST) is the structured tree representation of Python source code produced by Python's built-in compiler before execution. Using the standard library ast module (ast.parse() and ast.walk()), Graft's test suite inspects every .py file under src/graft/ in milliseconds—without executing the code or relying on brittle regular expressions that can be tricked by comments, docstrings, or multi-line strings.

In test_architecture_layers.py, the AST walker extracts every import x (ast.Import) and from x import y (ast.ImportFrom) statement to enforce two critical invariants:

  • Hexagonal Import Boundaries (test_hexagonal_import_boundaries): Hexagonal Architecture (Ports & Adapters) requires dependencies to point strictly inward toward the domain core. The AST gate verifies that:
    • Domain Models (src/graft/core/models/): Import only Python stdlib and sibling models (pure @dataclass(frozen=True) value objects).
    • Port Protocols (src/graft/core/ports/): Import only Python stdlib, models, and sibling ports (typing.Protocol interfaces).
    • Driving Core (src/graft/core/): Never imports graft.cli, graft.engines, or HTTP clients (urllib.request, http.client, socket, except mitre_validator.py for explicit STIX updates). This guarantees Core remains 100% engine-agnostic and offline-testable.
    • Driven Engine Adapters (src/graft/engines/<engine>/): Never import graft.cli or another engine adapter (e.g., secops can never couple to a future sentinel or splunk package).
    • Driving CLI (src/graft/cli/): Never statically imports graft.engines.<engine>.*; all engine interaction routes dynamically through EngineRegistry and EngineAdapter.
  • Stdlib-First & Supply-Chain Gate (test_stdlib_first_runtime_imports): Compares every root module imported in src/graft/ against Python's sys.stdlib_module_names and the approved runtime dependencies declared in pyproject.toml. Even if an extra third-party library is installed in a local virtual environment, importing it in src/graft/ without maintainer approval and test-gate alignment fails pytest immediately.

2. Schema-to-Model & Round-Trip Completeness (test_architecture_layers.py)

Detection-as-Code relies on translating between YAML files on disk, frozen Python dataclasses in memory, and engine-native queries (such as YARA-L). If a developer adds a field to a JSON schema but forgets to add it to RuleEnvelope or rule_to_dict(), data is silently dropped during graft <engine> pull or graft lint.

  • test_schema_and_dataclass_completeness mechanically verifies that every property in src/graft/core/schemas/*.schema.json matches the fields on RuleEnvelope / DatasetEnvelope and the keys emitted by rule_to_dict() / dataset_to_dict().
  • test_repository_corpora_lossless_round_trip loads every active dataset in datasets/ and rule in rulesets/, serializes it back to YAML, re-parses it, and asserts load(dump(load(file))) == load(file).

3. Never-Crash & Strict --json Error Gate (test_adversarial_inputs.py)

Because both human engineers and AI agents drive Graft via the CLI, hostile or malformed inputs (truncated YAML, invalid UTF-8 bytes, wrong types, or missing file paths) must never produce raw Python tracebacks (KeyError, AttributeError, UnicodeDecodeError).

  • test_adversarial_inputs.py feeds corrupt payloads into Core/Engine loaders and CLI commands, verifying that parsers always raise typed domain exceptions (RuleLoadError, DatasetLoadError, ManagedManifestLoadError, EngineError, ScaffoldError) and that passing --json always outputs valid JSON ({"success": false, "error": "..."}) on stdout.

2. Strict Stdlib-First Policy

Graft enforces an uncompromising standard: solve problems using Python standard libraries first.

1. Approved Runtime Dependencies & Governance

Runtime dependencies are locked in pyproject.toml and verified by test_architecture_layers.py. Currently, only YAML parsing (pyyaml) and JSON Schema Draft 2020-12 validation (jsonschema and its $ref registry companion referencing) are used at runtime because Python's standard library does not include a YAML parser or JSON Schema validator. Adding any new runtime dependency or OS tool requires explicit maintainer approval.

2. Mandatory Standard Library Replacements

All other functionality must be implemented using Python >= 3.13 standard libraries:

  • HTTP Client: Use urllib.request and urllib.error (no requests, no httpx).
  • CLI Parsing: Use argparse (no click, no typer).
  • Data Models: Use dataclasses (no pydantic in core).
  • VCS & Subprocesses: Use subprocess and shutil (no gitpython).
  • Data Formats: Use json, csv, hashlib, uuid.

3. Isolation of Development Dependencies

Development tools (pytest, mypy, ruff) are isolated strictly to [dependency-groups.dev] in pyproject.toml. They must never be imported in production source code (src/graft/).

3. Test-Driven Development (TDD) Conventions

Graft enforces strict Red-Green-Refactor test-driven development:

flowchart LR
    RED["<b>1. Red</b><br/>Write failing test in tests/<br/>Verify expected failure"]
    GREEN["<b>2. Green</b><br/>Write minimal code in src/<br/>Verify test passes"]
    REFACTOR["<b>3. Refactor</b><br/>Clean code, enforce types,<br/>ensure 0 linter warnings"]

    RED --> GREEN --> REFACTOR --> RED
Loading

Partitioned Test Hierarchy

Test Layer Directory Location Execution Speed Description
Unit Tests tests/unit/ <1 second total 100% mock-isolated. Never makes real network or disk calls outside temporary test directories.
Engine Tests tests/engines/<engine>/ Sub-second Mock-transport contract tests verifying API payload encoding and protocol conformance.
Integration Tests tests/integration/ Variable Live API tests targeting dedicated staging tenants, strictly gated behind @pytest.mark.integration and --run-integration.

Graceful Degradation for Replay Tests

If a detection rule defines a tests: block, but staging credentials (GRAFT_SECOPS_STAGING_*) are absent from the environment:

  • Emit a structured warning: [WARNING] Replay test skipped: staging credentials not configured.
  • Exit with code 0. Never break local developer workflows.
  • Enforce failure (exit code 1) only when --require-staging is explicitly supplied in CI.

4. Static Quality Gates & Verification Commands

Before declaring any phase or pull request complete, the following four commands must return 0 errors:

# Gate 1: Enforce consistent code formatting
uv run ruff format --check .

# Gate 2: Strict static linter (E, F, B, I, UP, S, RUF, SIM, T20)
uv run ruff check .

# Gate 3: Strict static type checking (zero untyped definitions, zero Any)
uv run mypy --strict src tests

# Gate 4: Fast unit and engine test suite
uv run pytest

Caution

The T20 linter rule strictly bans print() calls in production source code (src/graft/). Use sys.stdout.write(), sys.stderr.write(), or standard library logging.

Logging & Operational Diagnostics Standards

Operational logs are a first-class engineering contract across CLI, Core, and Driven Adapters:

  • UTC Timestamps (When): Root logging formats every line as %(asctime)s [%(levelname)s] %(message)s with datefmt="%Y-%m-%dT%H:%M:%SZ" and logging.Formatter.converter = time.gmtime.
  • Log Level Taxonomy:
    • DEBUG: Low-level adapter sub-steps (e.g., step 1 rule creation vs. step 2 deployment toggle), enabled via graft --verbose.
    • INFO: High-level rule/exclusion CRUD lifecycle actions emitted by Core reconcilers (graft.reconciler) and CLI controllers (graft.cli). Adapters must not emit duplicate INFO CRUD logs.
    • WARNING: Transient HTTP retry backoffs (429/503), two-stage partial mutation divergence (when step 1 succeeds on the tenant before step 2 fails), and graceful degradation (e.g., skipped staging replay when credentials are absent).
    • ERROR: Rule/exclusion failure context (What), adapter API error details (Where: HTTP status, vendor status code, HTTP method, and endpoint path), and partial-progress batch abort summaries (Progress: X/Y applied [...], 1 failed [...], Z pending [...]).

5. Code Commenting Directives (High Signal-to-Noise)

  • Only add comments when strictly necessary to explain why, non-obvious constraints, or security rationale.
  • Strictly Banned:
    • Zero obvious comments (e.g., # set name to rule name, # loop over rules, # return the result).
    • Zero boilerplate function docstrings repeating type signatures or parameter lists already captured by type hints.
    • Zero commented-out dead code. Delete obsolete code immediately.
    • Zero promotional trailers or LLM signatures (# Generated by..., # Created with AI).

6. Documentation, Markdown & Diagram Directives

  • Markdown List Style: All unordered markdown lists must strictly use hyphens (-), never asterisks (*).
  • Mermaid Everywhere: All architectural diagrams, workflows, execution sequences, state machines, and system topologies must be expressed using standard GitHub Markdown Mermaid fenced code blocks (```mermaid).
  • Strictly Banned:
    • Zero ASCII art box-drawing diagrams in markdown documentation.
    • Zero static binary images for architectural or procedural flows where Mermaid can represent the structure.
  • Mermaid Syntax Standards:
    • Always quote node labels containing special characters, parentheses, brackets, or code tags (e.g., ID["Label (Detail)"]).
    • Use directed flowcharts (flowchart TD or flowchart LR) with clean semantic naming.

7. Version Control, Branching & Scoped Commits

  • Always Work in a Feature Branch: Never commit directly to main. Always branch from main (git checkout -b <branch-name>) before making changes. Feature branches make rollback trivial, isolate experimental work from working code, and ensure every change runs through Pull Request verification gates (pr-validation.yml) before merging.

All commits must follow Scoped Commits:

<scope>: <description>
  • Scope First: Scope is lowercase and specific to the affected subsystem (e.g., secops:, core:, cli:, docs:, registry:).
  • Imperative Mood: One-sentence imperative description explaining the rationale.
  • Banned Prefixes: Never use generic prefixes like feat:, fix:, chore:, or update:.
  • Multi-Area Scopes: Use comma-separated scopes for multi-area changes (core,cli: ...), or treewide for repository-wide changes.
  • Clean History: Never add advertisements or promotional trailers to commit messages.