Skip to content

internal/sanitize: control-char stripping and Markdown escaping #7

Description

@Saber5656

Title

internal/sanitize: control-char stripping and Markdown escaping

Summary

Implement the text-hygiene primitives applied to all untrusted source
content: control-character stripping (terminal-escape defense), UTF-8
repair, length capping, single-line flattening, and Markdown escaping for
table cells and content lines (DESIGN.md §12.1 B6, §12.2).

Context

History lines, commit subjects, and agent session titles are untrusted bytes
(invariant I5). If rendered raw they can smuggle ANSI escape sequences into
terminals or break/forge Markdown structure in reports. Aggregation (18) and
renderers (20/21) call these helpers; parsers use the UTF-8/caps primitives.

Scope

  • internal/sanitize/sanitize.go + tests. Stdlib only.

Detailed Requirements

  1. Clean(s string) string:
    • Repair invalid UTF-8 (strings.ToValidUTF8(s, "�")).
    • Remove all C0 controls except \n and \t; remove \r (normalize
      CRLF→LF first); remove DEL (0x7F) and C1 controls (U+0080–U+009F).
    • Remove Unicode line/paragraph separators (U+2028, U+2029) and BOM.
    • Leaves all other printable Unicode intact (Japanese text must pass
      through unchanged — test with CJK + emoji).
  2. Line(s string, max int) stringClean, then replace \n/\t runs
    with a single space, trim, and cap to max runes appending when
    truncated (max ≤ 0 = no cap). Truncation must not split a rune.
  3. Cap(s string, max int) string — rune-safe cap with , preserving
    newlines (used for Body/narrative).
  4. EscapeCell(s string) string — for Markdown table cells: apply Line
    semantics (no cap), then escape | as \|, backtick as \`, and
    HTML-escape < and & (&lt;, &amp;).
  5. EscapeText(s string) string — for non-table content lines: Clean,
    HTML-escape < and &, and neutralize a leading #, -, *, >, or
    digit-dot list marker by prefixing \ (prevents structure forgery when a
    subject starts a line).
  6. Idempotence contract (documented in doc comments and asserted in tests
    over the corpus): Clean, Line, and Cap are idempotent
    (f(f(x)) == f(x)). EscapeCell and EscapeText are deliberately NOT
    idempotent
    (escaping & twice yields &amp;amp;): they are
    single-application functions applied exactly once, at render time, by the
    renderer — never by parsers or aggregation. Their doc comments must state
    this.

Acceptance Criteria

  • ANSI corpus test: strings containing \x1b]0;evil\x07,
    \x1b[31mred\x1b[0m, \x9b31m, raw \x07 come out with all escape
    bytes removed (assert no byte < 0x20 except \n/\t remains, no
    0x7F, no C1).
  • CJK/emoji passthrough: Japanese sentences and emoji unchanged by
    Clean.
  • Rune-safe truncation: capping inside a multi-byte rune never yields
    invalid UTF-8 (fuzz-style table over Japanese strings with varying
    caps).
  • EscapeCell("a|bc<d&e")` produces the exact documented escapes; a
    rendered table row containing it still parses as one row (golden
    string assert).
  • EscapeText("# not a heading") and EscapeText("- not a bullet")
    neutralize the marker; mid-line #/- untouched.
  • Idempotence asserted for Clean/Line/Cap over the whole corpus;
    for EscapeCell/EscapeText a test documents the non-idempotence
    (double application visibly differs) so callers cannot miss it.
  • ≥ 95% coverage; stdlib-only.

Validation

go test -race -cover ./internal/sanitize/ output in PR. Add one Go fuzz
target FuzzClean (seeded with the corpus) run for ≥ 30s locally:
output must always satisfy the "no forbidden bytes + valid UTF-8" property.

Dependencies

Non-goals

Redaction (16/17 — secrets are a separate pass), zsh unmetafy (09 — that is
format decoding, not sanitizing), full HTML sanitization (reports are
Markdown; only </& neutralization is required).

Design References

  • docs/DESIGN.md §12.1 (B6), §12.2 (terminal escape / markdown injection
    rows), §6.2 (Event field hygiene)

Source of truth: docs/issues/06-sanitize-package.md (PR #1, branch docs/v1-design). If this issue and the repo docs disagree, the docs win. Execution order and dependencies: docs/ISSUE_PLAN.md (this is issue 06 of 33).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions