diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000..c4057db --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,686 @@ +# worklog — v1 Design + +Status: v1 design, approved requirements baseline (2026-07-10). +Owner: design by Fable (requirements manager); implementation delegated via +`docs/issues/*.md`. +Canonical: this file is the source of truth for v1 behavior. GitHub Issues are +derived artifacts of `docs/ISSUE_PLAN.md` + `docs/issues/*.md`. + +--- + +## 1. Product definition + +`worklog` is a **local-only, read-only CLI** that generates a daily work +report (日報) and a standup report from activity traces already present on the +machine: + +- zsh command history, +- git commit history across local repositories, +- AI coding agent session logs (Claude Code, Codex CLI). + +Reports are rendered as Markdown to stdout (default) or to a file. An +**optional local LLM** (OpenAI-compatible endpoint on loopback) turns the +deterministic facts into a natural-language narrative; when the LLM is +unavailable or disabled, the deterministic report is the complete product, not +a degraded one. + +Written in Go, distributed as a single static binary, intended for public +open-source release. + +### 1.1 Requirements baseline (user-approved 2026-07-10) + +| Topic | Decision | +|---|---| +| Summarization | Local LLM shipped in v1, **opt-in via config** (`llm.enabled`, default false — per the approved "オプトイン" wording); deterministic rendering is the base and fallback | +| Sources (v1) | zsh history + git + Claude Code + Codex CLI; other agents = extension interface only | +| LLM connectivity | OpenAI-compatible API, **loopback-only, enforced in code**; non-loopback unsupported in v1 | +| Report types | `daily` + `standup` | +| Output | stdout default; optional file output | +| Secrets | Redaction ON by default, applied before rendering and before any data reaches the LLM; agent sessions contribute metadata by default, raw prompts opt-in | +| Language | Go; minimal dependencies | +| Report languages | Japanese and English templates in v1 | + +### 1.2 Hard product invariants + +These are non-negotiable properties. Every issue that touches them carries +matching acceptance criteria; CI enforces them where mechanically possible. + +- **I1 — No external network I/O.** The process never opens a network + connection to a non-loopback address. The only network feature is the local + LLM client, restricted to loopback by a custom dialer. Proxy environment + variables are ignored. No telemetry, no update checks, no DNS lookups + except for the LLM endpoint host (whose resolved IPs must all be loopback). +- **I2 — Sources are read-only.** Files and directories being read are never + created, modified, locked, or deleted. Git is invoked only with read-only + commands and `--no-optional-locks` (plus `GIT_OPTIONAL_LOCKS=0`). No + `git status` (it may refresh the index). Files are opened `O_RDONLY`. +- **I3 — Writes are confined** to (a) an explicit user-given output path or + the configured output directory, (b) the worklog config file written by + `worklog init` only. v1 keeps **no state, no cache, no log files**. +- **I4 — Redaction by default.** Secret masking is applied to all source + content before aggregation, rendering, and LLM submission. Disabling it + requires an explicit flag/config and prints a warning banner. +- **I5 — Untrusted input discipline.** All source file content is untrusted: + parsers are tolerant (skip-and-count, never crash), bounded (line/file + caps), and content is sanitized (control characters stripped) before it can + reach a terminal, a file, or the LLM. + +## 2. v1 scope + +### 2.1 In scope + +- Subcommands: `daily`, `standup`, `init`, `doctor`, `sources`, `version`. +- Sources: zsh (extended-format timestamps required for date filtering), git + (multi-repo discovery + commit extraction), Claude Code sessions, Codex + sessions. +- Redaction engine with built-in ruleset + user-extensible patterns. +- Deterministic Markdown rendering, Japanese + English templates. +- Optional loopback LLM narrative with graceful fallback. +- File output with safe-write semantics. +- macOS + Linux. CI on both. + +### 2.2 v1 non-goals (explicit) + +- Windows support. +- bash/fish history parsing. +- Web/browser enrichment ("browser search" extension idea) — recorded as a + v2+ extension direction; v1 core must not depend on it. +- Custom user templates, HTML/PDF output. +- Incremental state, caching, daemon/watch mode, scheduling (use cron/launchd + externally). +- Reporting uncommitted changes (`git status` conflicts with I2). +- Multi-machine aggregation. +- Plugin execution (subprocess source providers) — interface boundary is + designed (§6.1) but no dynamic loading/exec ships in v1. +- Holiday-calendar awareness for standup (weekend skip only). +- Auto-detected "blockers" content (placeholder section + optional LLM + suggestions only). + +### 2.3 Deferred v2 ideas + +Weekly/monthly rollups; bash/fish; Gemini CLI and other agents; subprocess +plugin protocol for third-party sources; template overrides; Obsidian +frontmatter/properties mode; entropy-based redaction; holiday calendars; +Homebrew tap; signed releases (Sigstore); `~/.claude/history.jsonl` and +`~/.codex/history.jsonl` as lightweight prompt sources; browser-search +enrichment as an optional, clearly network-labeled extension. + +## 3. Architecture overview + +### 3.1 Pipeline + +``` + +-----------+ +----------+ +--------+ +-----------+ + sources → | collect | → | sanitize | → | redact | → | aggregate | + (parallel)| (Provider)| | (bounds, | | (rules)| | (Report) | + +-----------+ | ctrl-ch)| +--------+ +-----------+ + +----------+ | + +--------------------+ + ▼ + +---------------------+ + | render (determin.) | + | ja/en templates | + +---------------------+ + | + (llm enabled?) ▼ + +---------------------+ fail → fallback + | llm narrative | ────────────────┐ + | (loopback only) | │ + +---------------------+ │ + ▼ ▼ + +---------------------+ deterministic + | output (stdout/file)| ←────── report + +---------------------+ +``` + +Key properties: + +- Providers run concurrently with a per-source timeout; a failing source + yields warnings, never a failed report. +- Redaction happens **before** aggregation so no unredacted string exists in + the report model. +- The LLM writes only the narrative section; all tables/facts are always + deterministic output. A hallucinating model cannot alter facts, only prose. + +### 3.2 Module layout + +``` +cmd/worklog/main.go # wiring only, no logic +internal/cli/ # dispatch, flags, usage, exit codes + cli.go daily.go standup.go init.go doctor.go sources.go version.go +internal/config/ # TOML schema, defaults, load/validate +internal/timeutil/ # report range resolution (dates, tz, workdays) +internal/model/ # Event, Report, Warning, enums +internal/sanitize/ # control-char strip, caps, markdown escape +internal/redact/ # engine + built-in ruleset +internal/source/ # Provider interface, registry, collector + jsonlutil/ # shared tolerant JSONL scanner + zsh/ # zsh history parser + provider + gitsrc/ # repo discovery + git log extraction + claudecode/ # Claude Code session provider + codex/ # Codex session provider +internal/aggregate/ # Report assembly, grouping, standup derivation +internal/render/ # embedded templates (ja/en × daily/standup) +internal/llm/ # loopback client + prompt builder + narrative +internal/output/ # stdout/file writers, safe-write rules +internal/version/ # version metadata (ldflags) +``` + +Dependency rule (CI-enforced): only `internal/llm` may import `net/http`/ +`net`; `os/exec` may be imported only by `internal/source/gitsrc`. + +### 3.3 External dependencies policy + +Standard library first. Allowed third-party modules in v1: + +| Module | Why | +|---|---| +| `github.com/BurntSushi/toml` | config parsing; small, stable, no transitive deps | + +Everything else (CLI flags, templates, HTTP, JSON) uses the standard library. +Adding a dependency requires an ADR. See ADR-001. + +## 4. CLI specification + +``` +worklog [flags] + +Subcommands: + daily Generate a daily work report + standup Generate a standup report (yesterday / today / blockers) + init Write a starter config file + doctor Diagnose configuration, sources, and LLM connectivity + sources List sources and per-source event counts for a date + version Print version information + +Global flags (accepted by daily/standup/doctor/sources): + --config PATH Config file (default: $XDG_CONFIG_HOME/worklog/config.toml, + fallback ~/.config/worklog/config.toml) + --date YYYY-MM-DD Report date (default: today in configured timezone) + --lang ja|en Report language (default: config, default ja) + --verbose Debug details to stderr +Flags for daily/standup: + --out PATH Write to file or directory (see §10); default stdout + --save Write into [output].directory from config + --force Allow overwriting an existing output file + --no-llm Skip LLM narrative for this run + --redact on|off Override redaction mode (off prints a warning banner) + --source LIST Comma-separated subset of enabled sources for this run + (e.g. --source git,codex) +Flags for init: + --config PATH Target path (default: the default config path) + --force Overwrite an existing config file +``` + +Exit codes: `0` success (including empty reports and degraded-source runs); +`1` runtime failure (e.g. output write failed); `2` usage/config error. +`doctor`: `0` all checks pass or warn, `1` at least one FAIL. + +Notes: + +- Flag parsing: standard library `flag` with one `FlagSet` per subcommand. + Unknown subcommand → usage to stderr, exit 2. +- `--date` accepts only `YYYY-MM-DD`. `standup` interprets `--date` as "the + standup happens on this morning" (see §9.2). +- All human-facing diagnostics go to **stderr**; only the report goes to + stdout (pipe-safe). Exceptions: `doctor` and `sources` are diagnostic + commands whose check/table output IS the product and goes to stdout, in + English only (`--lang` parses but does not affect them; `--date` parses + and is ignored by `doctor`). + +## 5. Configuration + +Location: `$XDG_CONFIG_HOME/worklog/config.toml`, fallback +`~/.config/worklog/config.toml`. Missing file = all defaults (zero-config +should produce a sensible report). Unknown keys → warning (typo detection), +not error. All relative/`~` paths are tilde-expanded; environment variable +expansion is NOT performed (predictability), except the zsh `HISTFILE` +special case below. + +```toml +[general] +language = "ja" # "ja" | "en" +timezone = "" # IANA name; "" = system local time + +[output] +directory = "" # used by --save; must be set for --save to work + +[sources] +enabled = ["zsh", "git", "claude-code", "codex"] + +[sources.zsh] +history_file = "" # "" → $HISTFILE if set, else ~/.zsh_history +exclude_patterns = [] # RE2 regexes; matching commands are dropped at parse time + +[sources.git] +roots = ["~/dev"] # scanned for repos (depth-limited walk) +max_depth = 3 # directory depth below each root +exclude_dirs = ["node_modules", ".cache", "vendor", "00_Archive"] +repos = [] # explicit extra repo paths (bypass discovery) +authors = [] # author emails counted as "me"; [] → global git config user.email + +[sources.claude_code] +projects_dir = "~/.claude/projects" +include_prompts = false # true → session titles may use first user prompt (redacted) + +[sources.codex] +sessions_dir = "~/.codex/sessions" +session_index = "~/.codex/session_index.jsonl" +include_archived = false # also scan ~/.codex/archived_sessions +archived_dir = "~/.codex/archived_sessions" +include_prompts = false +exclude_originators = [] # session_meta.originator exact matches to skip + +[llm] +enabled = false # opt-in: set true AND set model (see doctor) +endpoint = "http://127.0.0.1:11434/v1" # loopback enforced regardless of value +model = "" # required when enabled; doctor lists available ids +api_key = "" # optional bearer for local servers that want one +timeout_seconds = 120 +max_input_chars = 24000 +max_output_chars = 4000 +temperature = 0.2 + +[redaction] +mode = "on" # "on" | "off" +extra_patterns = [] # RE2 regexes, masked entirely +allowlist = [] # RE2 regexes; matches are exempted from masking + +[standup] +previous_workday = true # Monday standup looks back to Friday +``` + +Validation rules (exit 2 with all errors listed, not just the first): +`language ∈ {ja,en}`; `timezone` loadable via `time.LoadLocation` when set; +regexes compile as RE2; `llm.enabled && model == ""` → error advising +`worklog doctor` (this can only arise from an explicit config file — the +default is `enabled = false`, so zero-config runs always validate); +`llm.endpoint` must be `http`/`https` URL with explicit host (port optional; +scheme defaults apply); +numeric fields within sane bounds (`timeout_seconds` 1–600, `max_input_chars` +1000–1_000_000, `max_output_chars` 100–100_000, `temperature` 0–2, +`max_depth` 1–6). Loopback validation of the endpoint host happens both at +config validation (static check for obvious non-loopback hosts) and at dial +time (authoritative, per resolved IP). + +## 6. Data model + +### 6.1 Source provider boundary + +```go +package source + +type Provider interface { + // ID returns the stable source identifier ("zsh", "git", ...). + ID() model.SourceID + // Collect returns events overlapping rng. It must honor ctx cancellation, + // never write to disk, and return partial results with warnings rather + // than failing wholesale where possible. + Collect(ctx context.Context, rng timeutil.Range) ([]model.Event, []model.Warning, error) +} +``` + +Providers are constructed with their typed config at wiring time. The +collector runs all enabled providers concurrently (per-source timeout, +default 30s; on timeout: warning + zero events). This interface is the future +plugin boundary; v1 links all providers statically. + +### 6.2 Event (canonical activity record) + +```go +package model + +type SourceID string // "zsh" | "git" | "claude-code" | "codex" +type EventKind string // "command" | "commit" | "agent-session" + +type Event struct { + Source SourceID + Kind EventKind + Start time.Time // required, non-zero + End time.Time // zero for instantaneous events + Project string // normalized project key ("" = unattributed) + Ref string // commit hash / session id / "" + Title string // single line; sanitized + redacted upstream + Body string // optional detail; sanitized + redacted + Meta map[string]string // source-specific, values sanitized + redacted +} +``` + +Project normalization: for git, the repository directory base name; for agent +sessions, the base name of `cwd` if it is under a configured git root or an +explicit repo path — matched by path prefix — else the full cleaned `cwd` +(tilde-abbreviated for display). Shell commands have `Project == ""` in v1. + +### 6.3 Report model + +```go +type Report struct { + Kind ReportKind // "daily" | "standup" + Date string // YYYY-MM-DD (report date) + Range timeutil.Range // resolved instants + Lang string + Projects []ProjectActivity + Shell ShellActivity // unattributed command groups + Totals Totals + Narrative Narrative // LLM output or empty + Warnings []Warning + Gen GenerationMeta // version, sources used, llm model/none, redaction mode + Standup *StandupData // standup only: yesterday/today buckets, blockers placeholder +} + +type ProjectActivity struct { + Name string + Commits []Commit // Hash(short), Subject, When, FilesChanged, Insertions, Deletions, Repo + Sessions []AgentSession // Agent, Title, Start, End, UserMsgs, AgentMsgs, Tasks, Ref +} + +type ShellActivity struct { + Groups []CommandGroup // Head (first token+subcommand), Count, First, Last, Examples []string (≤3, redacted) + Total int +} +``` + +Deterministic ordering everywhere (stable output = testable output): projects +by activity score (`3*commits + 2*sessions`; shell commands are unattributed +in v1 and never contribute) descending, then name; +commits by time; sessions by start; command groups by count desc, then head. + +## 7. Source specifications + +Per-source parsing details live in `docs/research/*.md` (formats) and the +issue files (procedures). This section fixes the behavioral contract. + +### 7.1 zsh (`internal/source/zsh`) + +- History file resolution: config → `$HISTFILE` → `~/.zsh_history`. +- Supports simple and extended formats; **only extended-format entries + (with epoch timestamps) are date-filterable**. A file with no extended + entries yields 0 events + warning `zsh_no_timestamps` (doctor explains the + fix: `setopt EXTENDED_HISTORY INC_APPEND_HISTORY`). +- Unmetafy → UTF-8 sanitize → multiline join → header parse → range filter → + `exclude_patterns` drop → Event{Kind: command, Title: command line (first + 200 chars), Meta: duration}. +- Caps: 64 KiB/line, 128 MiB/file (stop + warn `zsh_file_truncated`). + +### 7.2 git (`internal/source/gitsrc`) + +Discovery: + +- Walk each root to `max_depth`, skipping `exclude_dirs` by base name and all + dot-directories except `.git` itself; symlinked directories are not + followed (cycle prevention). +- A repo = directory containing `.git` (dir **or** file — worktrees have a + `.git` file). Do not descend into a repo once found. +- Deduplicate worktrees: `git rev-parse --path-format=absolute --git-common-dir` + per candidate; canonical repo key = parent of the common dir. One logical + repo per common dir; project name = base name of the canonical work dir + (for bare-ish edge cases fall back to common-dir parent name). +- Explicit `repos` entries join the same dedup pass. + +Extraction (per canonical repo, one `git log` invocation): + +- Invocation: `git --no-optional-locks -C log --all --no-merges + --since= --until= --date=iso-strict + --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%s%x1e --shortstat` + with env `GIT_OPTIONAL_LOCKS=0`, `GIT_TERMINAL_PROMPT=0`, `LC_ALL=C`, + `GIT_CONFIG_PARAMETERS` unset. `os/exec` with argument slice — no shell. +- The ±48h widening exists because `--since/--until` filter committer date; + precise filtering happens in Go on the **author date** (`%aI`), which is the + "when the work happened" semantic (stable across rebases). +- Author filter in Go: case-insensitive exact match of `%ae` against + `authors` (or the single global `user.email` when unset; if that is also + empty → warning `git_no_author_identity`, source disabled for the run). +- Output parsing: records split on `0x1e`, fields on `0x1f`; `--shortstat` + lines are associated with the preceding record; missing shortstat (empty + commits) tolerated. +- Per-repo timeout 10s; repos failing to parse yield warning `git_repo_failed` + (repo path, first stderr line) and are skipped. +- Event mapping: Kind `commit`, Start = author date, Project = repo name, + Ref = short hash (12), Title = subject, Meta = `files`, `insertions`, + `deletions`, `repo_path`. + +### 7.3 Claude Code (`internal/source/claudecode`) — contract per research doc + +mtime prefilter (± 48h of range), streaming tolerant JSONL, session assembly +(min/max in-range timestamps, modal cwd, sidechain counted separately), +title precedence: latest `summary` line → first user prompt (only when +`include_prompts`) → `claude-code session `. Caps: 4 MiB/line. +Warnings: `claudecode_dir_missing`, `claudecode_malformed_lines` (>20% of a +file), `claudecode_file_skipped`. + +### 7.4 Codex (`internal/source/codex`) — contract per research doc + +Date-dir pruning (range days ±1) with recursive-walk fallback, session_index +title map, `event_msg` counting (`user_message`, `agent_message`, +`task_started`/`task_complete`), `exclude_originators`, optional archived dir. +Same caps/warnings pattern (`codex_*`). + +## 8. Redaction (`internal/redact`) + +Engine: + +- `Redact(s string) (string, int)` — applies the ordered ruleset; returns the + masked string and the number of maskings. Applied by the pipeline to every + Event `Title`, `Body`, and `Meta` value, and re-applied by the LLM prompt + builder to the final prompt as defense in depth. +- Replacement token: `[REDACTED:]`. +- Rule order: block rules (private key blocks) → specific token formats → + contextual assignments/flags → URL userinfo. `allowlist` regexes exempt a + match; `extra_patterns` run last, masking the entire match. +- Rules are RE2 (no backtracking DoS). Multiline block masking is bounded + (max 100 lines per block). + +Built-in ruleset (each with true/false-positive test corpus; ids are stable +public API for allowlisting): + +| id | Targets | +|---|---| +| `private-key-block` | `-----BEGIN … PRIVATE KEY-----` blocks | +| `aws-access-key-id` | `\b(AKIA|ASIA)[0-9A-Z]{16}\b` | +| `github-token` | `\bgh[pousr]_[A-Za-z0-9]{36,255}\b`, `\bgithub_pat_[A-Za-z0-9_]{22,255}\b` | +| `slack-token` | `\bxox[baprs]-[A-Za-z0-9-]{10,}\b` | +| `google-api-key` | `\bAIza[0-9A-Za-z_-]{35}\b` | +| `openai-anthropic-key` | `\bsk-[A-Za-z0-9_-]{20,}\b` (covers `sk-ant-…`, `sk-proj-…`) | +| `jwt` | `\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{5,}\b` | +| `authorization-header` | `(?i)\bauthorization\s*:\s*(basic|bearer|token)\s+\S+` (value part) | +| `env-assignment` | `(?i)\b[A-Z0-9_]*(TOKEN|SECRET|PASSWORD|PASSWD|API_?KEY|CREDENTIALS?|PRIVATE_KEY)[A-Z0-9_]*\s*=\s*("[^"]*"|'[^']*'|\S+)` (value part; name kept) | +| `cli-secret-flag` | `(?i)(--?(password|passwd|token|secret|api-?key|access-?key|auth))([= ])("[^"]*"|'[^']*'|\S+)` (value part) | +| `url-userinfo` | `[a-zA-Z][a-zA-Z0-9+.-]*://[^/\s:@]+:[^@\s/]+@` (password part) | + +Guarantees: git short hashes (12 hex) and commit subjects without secrets are +never masked by the built-in set (regression corpus enforces this). Entropy +heuristics are v2 (false-positive risk). + +Redaction OFF (`--redact off` or config): stderr banner +`WARNING: redaction disabled — output may contain secrets` and +`Gen.RedactionMode = "off"` printed in the report footer. + +## 9. Aggregation & report semantics + +### 9.1 daily + +- Range: `[00:00, 24:00)` of `--date` in the configured timezone. +- Sections (fixed order): Header (date, tz) → Summary line (totals) → + Narrative (if any) → Per-project (commits table, sessions table) → Shell + activity (top 20 groups) → Warnings → Generation footer. +- Empty day: all-zero totals render an explicit "no recorded activity" + sentence (exit 0). + +### 9.2 standup + +- `--date D` = the morning the standup happens. +- Yesterday bucket: previous workday of `D` (Mon→Fri) when + `standup.previous_workday`, else literal `D−1`. Full-day range. +- Today bucket: half-open `[00:00 of D, now)` when `D` is today; for a + past/future `D` (retro-generation) the whole day of `D`. +- Blockers: static placeholder `- (none recorded — fill in manually)`; when + the LLM narrative is enabled, model-suggested blockers render under + `Suggested by local model:` and are visually marked as suggestions. +- Sections: Yesterday (per-project one-liners: `repo — N commits (subjects…); + M agent sessions (titles…)`) → Today → Blockers → footer. + +### 9.3 Totals & grouping rules + +- Command grouping: head = first token; if head ∈ {git, npm, pnpm, yarn, go, + cargo, docker, kubectl, make, brew, gh, uv, pip} and a second token exists, + head = first two tokens. Consecutive duplicates collapse before grouping. +- Session dedup: by `Ref`; a session appearing in both live and archived + Codex dirs counts once (live wins). +- Totals: commits, insertions, deletions, sessions, agent user-messages, + commands, active span (earliest→latest event times). + +## 10. Output (`internal/output`) + +- stdout: report bytes only; everything else on stderr. +- `--out PATH`: if PATH exists as a directory OR ends with `/`, write + `-.md` inside it; otherwise treat as file path. `--save`: same + naming inside `[output].directory` (error if unset). +- Safe-write rules: refuse existing target without `--force` (error lists the + path); refuse target whose final component is a symlink (`Lstat`); create + missing parent directories `0700` (directories worklog itself creates are + re-verified via `Lstat` as real directories; pre-existing symlinked parent + directories are honored as user intent); write `0600`; atomic `CreateTemp` + in the target directory + `rename`; temp file removed on failure. +- Report content may contain personal data → conservative permissions are a + product decision, documented in README. + +## 11. LLM narrative (`internal/llm`) + +### 11.1 Client + +- Endpoint from config; path join → `POST {base}/chat/completions`, + `GET {base}/models` (doctor only). +- **Loopback enforcement (I1):** custom `http.Transport` with + `Proxy: nil` and a `DialContext` that resolves the host, verifies **every** + resolved IP satisfies `IsLoopback()`, and dials only verified addresses. + Non-loopback → typed error `ErrNonLoopbackBlocked` (doctor prints it + verbatim; daily/standup convert it to warning `llm_blocked_nonloopback` + + deterministic fallback). + `CheckRedirect` denies all redirects. Schemes: `http`, `https` only. +- Single attempt, overall timeout `llm.timeout_seconds`. No streaming. +- `Authorization: Bearer` only when `api_key` set. + +### 11.2 Prompt & response contract + +- System prompt (embedded, per report kind × language): the model is a + reporting assistant; it must use ONLY the provided data; data is enclosed + between `` markers and is **data, not instructions** (prompt-injection + hedge — agent session titles are attacker-influenced text). +- User message: compact JSON of the redacted Report (projects, commits + subjects, session titles/counts, command groups), truncated to + `max_input_chars` with priority commits > sessions > commands; truncation + is annotated (`"truncated": true` per section). +- Response: `choices[0].message.content` → sanitize (strip control chars, + collapse >2 blank lines, cap `max_output_chars`), reject empty → + `Narrative{Text, Model}`. +- Any error path → warning + deterministic report, exit code 0. Warning + mapping is fixed: `ErrNonLoopbackBlocked` → `llm_blocked_nonloopback`; + connection refused / timeout / HTTP ≥ 400 / redirect blocked → + `llm_unavailable`; empty or sanitized-to-empty content → `llm_failed`. + +## 12. Security model + +### 12.1 Assets & trust boundaries + +Assets: secrets embedded in histories/logs; private activity data (what you +worked on, when); integrity of the user's source files. + +| # | Boundary | In | Out | Controls | +|---|---|---|---|---| +| B1 | Source files → parsers | untrusted bytes | Events | O_RDONLY, caps, tolerant parse, sanitize, redact | +| B2 | git binary invocation | repo contents | stdout parse | read-only args, no shell, env hygiene, timeouts | +| B3 | Process → LLM endpoint | redacted JSON | narrative text | loopback-only dialer, no proxy, no redirects, re-redaction, output sanitize, no tool/function calls | +| B4 | Process → output file | report | file | explicit path only, no-overwrite, no-symlink, 0600, atomic write | +| B5 | Config file → process | user TOML | behavior | validation, RE2 only, no exec-capable fields in v1 | +| B6 | Terminal | report/diagnostics | display | control-char strip renders escape-sequence injection inert | + +### 12.2 Threats & mitigations + +| Threat | Mitigation | Enforced by | +|---|---|---| +| Data exfiltration (malicious PR, compromised dep) | I1: loopback-only dialer, `Proxy: nil`; import-boundary CI check (only `internal/llm` imports net); minimal deps + `govulncheck` + pinned `go.sum` | issues 02, 23, 30 | +| Secret leakage into shared reports | I4 default-on redaction; metadata-first agent content; prompts opt-in; 0600 outputs; footer discloses redaction mode | issues 16, 17, 22 | +| Source corruption / lock contention | I2: O_RDONLY, `--no-optional-locks`, no `git status`, no SQLite opens, stateless runs | issues 09–15 | +| Terminal escape injection from history/log content | sanitize strips C0/C1 except `\n\t` | issue 06 | +| Markdown/table injection breaking report structure | cell escaping (`|`, backticks, leading `#`/`-`), raw HTML escaped | issues 06, 20, 21 | +| Prompt injection via session titles/commit subjects | data-marker framing, model has no tools, output sanitized + length-capped, narrative visually attributed to model | issue 24 | +| Path traversal / symlink abuse on write | cleaned absolute paths, Lstat final component, no-follow, no-overwrite default | issue 22 | +| DoS via huge/malformed sources | line/file caps, per-source + per-repo timeouts, RE2 | issues 08–15, 16 | +| Malicious repo names / project names in output | same sanitize+escape path as all content | issue 06 | +| Supply chain (build/release) | stdlib-first, one vetted dep, tidy-check, `govulncheck` in CI, release checksums | issues 02, 32 | +| LLM endpoint hijack via DNS name resolving publicly | per-IP loopback verification at dial time (not string comparison) | issue 23 | + +### 12.3 Privacy stance (README-facing) + +Everything runs locally; nothing leaves the machine except to the loopback +LLM endpoint you configure; no telemetry, no crash reporting, no update +checks; reports are written only where you ask; redaction is on by default +and its mode is disclosed in every report footer. SECURITY.md defines +vulnerability reporting (GitHub private advisories). + +## 13. Error & warning taxonomy + +`model.Warning{Source SourceID, Code string, Message string}` — stable `Code` +strings (documented in README): `zsh_no_timestamps`, `zsh_file_truncated`, +`zsh_history_missing`, `zsh_lines_skipped`, `git_no_author_identity`, +`git_repo_failed`, `git_binary_missing`, `git_root_missing`, +`claudecode_dir_missing`, `claudecode_malformed_lines`, +`claudecode_file_skipped`, `codex_dir_missing`, `codex_malformed_lines`, +`codex_index_unreadable`, `codex_file_skipped`, `source_timeout`, +`llm_unavailable`, `llm_failed`, `llm_blocked_nonloopback`, +`redaction_disabled`. Warnings render in the +report footer and on stderr; they never fail the run. Hard errors (exit 2): +invalid flags, unparseable/invalid config, `--save` without configured +directory. Hard errors (exit 1): output write failure only. + +## 14. Testing & validation strategy + +- **Unit**: every package, table-driven; RE2 corpus for redaction (true and + false positives); unmetafy byte-level tests; git record parser fed crafted + `%x1f/%x1e` streams. +- **Fixtures**: `testdata/home/` synthetic tree — zsh histories (simple, + extended, metafied Japanese, multiline, torn tail, oversized line), + Claude/Codex JSONL files (schema-matching, plus malicious lines: ANSI + escapes, fake AWS/GitHub keys, prompt-injection strings, 5 MiB line), + codex `session_index.jsonl`. Git fixture repos are **built by test helpers** + at test time (`git init` in `t.TempDir()` with fixed + `GIT_AUTHOR_DATE/GIT_COMMITTER_DATE`, incl. one worktree) — no binary repo + fixtures in the tree. +- **Golden files**: renderer and end-to-end CLI outputs (ja/en × daily/standup + × llm-on/off), byte-exact, `-update` flag regenerates. +- **E2E**: built binary + fixture HOME (`HOME`, `XDG_CONFIG_HOME` overridden) + → golden reports; LLM path served by `httptest` loopback server; a canned + non-loopback endpoint must produce `llm_blocked_nonloopback`. +- **Invariant tests**: dialer rejects public IP, private-range IP, and + hostnames resolving to them (fake resolver); proxy env vars set during test + must not leak (`Proxy: nil`); redirect to non-loopback rejected; + no file under fixture HOME is modified by a full run (hash tree before/ + after); output safe-write matrix (exists/symlink/missing-parents/--force). +- **CI** (GitHub Actions, ubuntu + macos): build, `gofmt -l`, `go vet`, + `golangci-lint`, `go test -race ./...`, `govulncheck`, + `go mod tidy -diff`, `scripts/check_net_imports.sh` (import boundary), E2E. + +Whole-product acceptance: see `docs/ISSUE_PLAN.md` §"v1 completion +statement" — completing all issues and their validations constitutes v1. + +## 15. Known unknowns + +| # | Unknown | Handling | +|---|---|---| +| U1 | Claude Code / Codex schema drift across CLI versions | tolerant parsers, fixtures per observed version, doctor surfaces malformed-line ratios | +| U2 | `originator`-based Codex automation filtering adequacy | `worklog sources` prints per-originator counts; config exclude list; revisit post-v1 | +| U3 | zsh EXTENDED_HISTORY disabled on many machines (incl. reference machine) | doctor guidance; zsh source degraded-but-explicit; consider `fc`-based import v2 | +| U4 | LLM narrative quality across small local models | prompt iteration expected; deterministic facts unaffected; goldens use fake server | +| U5 | Go toolchain absent on the reference dev machine | issue 01 prerequisite documents `brew install go` (Go ≥ 1.23) | +| U6 | Worktree/bare/submodule discovery edge cases | common-dir dedup + fixtures; bare repos out of v1 scope (no worktree checkout = no daily work) | +| U7 | Windows paths/console | out of scope v1 (non-goal) | + +## 16. Naming, versioning, licensing + +- Module path `github.com/Saber5656/worklog`; binary `worklog`. +- SemVer, `v0.x` during v1 development; `v1.0.0` at issue-plan completion. +- License: MIT proposed (ADR-005). Note: the repository is **already + public** (confirmed 2026-07-10) with no LICENSE file, i.e. currently + all-rights-reserved. User confirmation of MIT vs Apache-2.0 is the open + gate before the LICENSE file lands via issue 01. +- README: English primary + `docs/README.ja.md` Japanese (issue 31). diff --git a/docs/ISSUE_PLAN.md b/docs/ISSUE_PLAN.md new file mode 100644 index 0000000..203282d --- /dev/null +++ b/docs/ISSUE_PLAN.md @@ -0,0 +1,189 @@ +# worklog — v1 Issue Plan + +Status: derived from `docs/DESIGN.md` (2026-07-10). GitHub Issues are created +from `docs/issues/NN-*.md`; when they disagree, these files win. + +## v1 completion statement + +When issues **01–33** are all completed and each issue's Validation section +passes, worklog v1 is complete as specified by `docs/DESIGN.md`: + +> A user on macOS or Linux can install a single `worklog` binary, run +> `worklog init`, `worklog doctor`, `worklog daily`, and `worklog standup`, +> and obtain Japanese or English Markdown reports built from zsh history, git +> commits, and Claude Code / Codex session logs — with no external network +> I/O (loopback LLM only), read-only source access, confined writes, and +> redaction on by default (invariants I1–I5), verified by the CI suite. + +One human gate remains outside the issue plan by design: confirming the +license (ADR-005). Note the repository is **already public**, so every push +is a publication — the standing rule is to scan content for personal data +and secrets before each push (ADR-005 §5), not a one-time go-public gate. + +## Issue list (recommended execution order) + +| # | File | Title (GitHub) | Wave | +|---|---|---|---| +| 01 | `issues/01-repo-scaffolding.md` | Repository scaffolding: Go module, layout, Makefile, LICENSE | 0 | +| 02 | `issues/02-ci-workflow.md` | CI workflow: build, lint, test, vulnerability and boundary checks | 0 | +| 04 | `issues/04-timeutil-package.md` | internal/timeutil: report range resolution | 0 | +| 03 | `issues/03-model-package.md` | internal/model: Event, Report, Warning types and ordering (after 04: uses `timeutil.Range`) | 0 | +| 05 | `issues/05-config-package.md` | internal/config: TOML schema, defaults, validation | 0 | +| 06 | `issues/06-sanitize-package.md` | internal/sanitize: control-char stripping and Markdown escaping | 0 | +| 07 | `issues/07-cli-skeleton.md` | internal/cli: subcommand dispatch, flags, exit codes | 0 | +| 08 | `issues/08-source-provider-collector.md` | internal/source: Provider interface and concurrent collector | 1 | +| 09 | `issues/09-zsh-history-parser.md` | zsh history parser: formats, unmetafy, multiline, caps | 1 | +| 10 | `issues/10-zsh-provider.md` | zsh source provider: resolution, filtering, warnings | 1 | +| 11 | `issues/11-git-repo-discovery.md` | git repo discovery: root walk, worktree dedup | 1 | +| 12 | `issues/12-git-commit-extraction.md` | git commit extraction: read-only log invocation and parsing | 1 | +| 13 | `issues/13-jsonl-scanner-util.md` | internal/source/jsonlutil: tolerant capped JSONL scanner | 1 | +| 14 | `issues/14-claude-code-provider.md` | Claude Code session provider | 1 | +| 15 | `issues/15-codex-provider.md` | Codex session provider | 1 | +| 16 | `issues/16-redaction-engine.md` | internal/redact: rule engine, allowlist, config integration | 2 | +| 17 | `issues/17-redaction-ruleset.md` | Built-in redaction ruleset and test corpus | 2 | +| 18 | `issues/18-aggregation.md` | internal/aggregate: sanitize/redact stage and Report assembly | 2 | +| 19 | `issues/19-standup-derivation.md` | Standup derivation: workday logic, buckets, blockers | 2 | +| 20 | `issues/20-render-daily.md` | Daily report renderer (ja/en) with golden tests | 3 | +| 21 | `issues/21-render-standup.md` | Standup report renderer (ja/en) with golden tests | 3 | +| 22 | `issues/22-output-writer.md` | internal/output: stdout/file writers with safe-write rules | 3 | +| 23 | `issues/23-llm-client.md` | internal/llm: loopback-enforced OpenAI-compatible client | 4 | +| 24 | `issues/24-llm-narrative.md` | LLM narrative: prompt builder, response handling, fallback | 4 | +| 25 | `issues/25-cmd-daily.md` | `worklog daily`: end-to-end wiring and integration tests | 5 | +| 26 | `issues/26-cmd-standup.md` | `worklog standup`: wiring and integration tests | 5 | +| 27 | `issues/27-cmd-init.md` | `worklog init`: starter config writer | 5 | +| 28 | `issues/28-cmd-doctor.md` | `worklog doctor`: environment and connectivity diagnostics | 5 | +| 29 | `issues/29-cmd-sources.md` | `worklog sources`: source listing and event counts | 5 | +| 30 | `issues/30-network-invariant-enforcement.md` | Network/read-only invariant enforcement suite | 6 | +| 31 | `issues/31-readme-and-user-docs.md` | README and user documentation (en + ja) | 6 | +| 32 | `issues/32-release-workflow.md` | Release workflow: cross-compile, checksums, tag automation | 6 | +| 33 | `issues/33-security-docs.md` | SECURITY.md and security model documentation | 6 | + +## Dependency table + +`A ← B` means B must be completed before A starts. + +| Issue | Depends on | +|---|---| +| 01 | — | +| 02 | 01 | +| 03 | 01, 04 | +| 04 | 01 | +| 05 | 01 | +| 06 | 01 | +| 07 | 01, 05 | +| 08 | 03, 04, 05 | +| 09 | 01, 06 | +| 10 | 08, 09 | +| 11 | 01, 03 | +| 12 | 03, 04, 11 | +| 13 | 01 | +| 14 | 04, 08, 13 | +| 15 | 04, 08, 13 | +| 16 | 01, 05 | +| 17 | 16 | +| 18 | 03, 04, 06, 16, 17 | +| 19 | 04, 18 | +| 20 | 06, 18 | +| 21 | 06, 18, 19 | +| 22 | 01, 03 | +| 23 | 01, 05 | +| 24 | 05, 16, 17, 18, 23 | +| 25 | 07, 10, 12, 14, 15, 18, 20, 22, 24 | +| 26 | 19, 21, 25 | +| 27 | 05, 07, 22 | +| 28 | 07, 09, 11, 14, 15, 23 | +| 29 | 07, 08, 10, 12, 14, 15 | +| 30 | 02, 23, 25, 26, 27, 28, 29 | +| 31 | 25, 26, 27, 28, 29, 30, 32 | +| 32 | 01, 02 | +| 33 | 01, 30, 31 | + +Within a wave, issues with disjoint dependencies can proceed in parallel +(e.g. 09/11/13 are independent; 16 can start any time after 05). + +## Implementation waves + +| Wave | Theme | Issues | Exit criterion | +|---|---|---|---| +| 0 | Foundation | 01–07 | `worklog version` builds and runs; CI green on empty-but-wired packages | +| 1 | Sources | 08–15 | Each provider returns correct Events from fixtures; read-only verified per source | +| 2 | Processing | 16–19 | Redacted, aggregated `Report` built from fixture events; standup buckets correct | +| 3 | Presentation | 20–22 | Golden Markdown (ja/en × daily/standup) renders; safe file writes proven | +| 4 | LLM | 23–24 | Narrative from `httptest` loopback fake; non-loopback blocked; fallback clean | +| 5 | Commands | 25–29 | All six subcommands work end-to-end against fixture HOME | +| 6 | Hardening & release | 30–33 | Invariant suite in CI; docs complete; tag produces release artifacts | + +## Coverage: DESIGN.md sections → issues + +| DESIGN.md section | Issues | +|---|---| +| §1.2 Invariants I1–I5 | cross-cutting AC in all issues; enforced by 02, 30 | +| §3.2 Module layout / §3.3 deps policy | 01, 02 | +| §4 CLI specification | 07, 25, 26, 27, 28, 29 | +| §5 Configuration | 05, 27 | +| §6.1 Provider boundary | 08 | +| §6.2–6.3 Data model | 03, 18 | +| §7.1 zsh | 09, 10 | +| §7.2 git | 11, 12 | +| §7.3 Claude Code | 13, 14 | +| §7.4 Codex | 13, 15 | +| §8 Redaction | 16, 17 | +| §9 Aggregation & standup | 18, 19 | +| §10 Output | 22 | +| §11 LLM | 23, 24 | +| §12 Security model | 06, 16, 17, 22, 23, 30, 33 (+ per-issue AC) | +| §13 Warning taxonomy | 03, 09–15, 23, 24 | +| §14 Testing strategy | 02, 25, 26, 30 (+ every issue's Validation) | +| §16 Naming/versioning/licensing | 01, 31, 32 (ADR-005) | + +Every DESIGN.md behavior is owned by at least one issue; no v1 behavior lives +only in prose. + +## Whole-product validation strategy + +1. **Per-issue gates**: each issue's Validation section (unit tests, lint, + race detector) must pass in CI before the issue closes. +2. **Fixture realism**: parser fixtures are derived from the formats recorded + in `docs/research/*.md`; every observed hazard (metafied bytes, torn + lines, sidechains, missing session_meta, ANSI injection, fake secrets) has + a fixture line and an assertion. +3. **Integration goldens** (issues 25, 26): byte-exact CLI outputs for + ja/en × daily/standup × llm-on/off against a fixture HOME. +4. **Invariant suite** (issue 30): no-network (dialer attack tests, proxy + ignorance, import boundary script), read-only (fixture tree hash + before/after), write confinement (safe-write matrix). +5. **Release rehearsal** (issue 32): dry-run tag build produces all four + platform binaries + checksums; `go install` path verified. +6. **Docs acceptance** (issues 31, 33): README quickstart reproduced verbatim + on a clean macOS and Linux environment; SECURITY.md consistent with + DESIGN §12. + +## Deferred v2 items + +Weekly/monthly rollups; bash/fish histories; Gemini CLI and other agents; +subprocess plugin protocol for sources; template overrides; Obsidian +frontmatter mode; entropy-based redaction; holiday calendars; Homebrew tap; +artifact signing (Sigstore); `history.jsonl` lightweight prompt sources; +browser-search enrichment extension (explicitly network-labeled, opt-in); +LAN (non-loopback) LLM endpoints via a loud opt-in ADR; incremental state. + +## Known unknowns (may create additional issues) + +| # | Unknown | Likely follow-up | +|---|---|---| +| U1 | Claude/Codex schema drift across CLI versions | new fixtures + parser patch issues | +| U2 | `originator` adequacy for Codex automation filtering | heuristic/filter issue after real-world use | +| U3 | zsh EXTENDED_HISTORY off on many machines | `fc`-import or first-run guidance issue | +| U4 | Small-model narrative quality | prompt-tuning issue(s) post-v1 | +| U5 | Go toolchain absent on the reference machine | handled as issue 01 prerequisite; no code | +| U6 | Worktree/bare/submodule discovery edge cases | targeted fixture + fix issues | +| U7 | LM Studio auth quirks across versions | client compatibility patch issue | + +## Conventions for implementers + +- One issue = one PR = one working branch (`issue/NN-short-title`). +- Do not start an issue whose dependencies are not merged. +- Every PR must keep `go test ./...`, lint, and the CI invariant checks + green; issues 02/30 define those gates. +- If implementation reveals a conflict with DESIGN.md, stop and update + DESIGN.md first (docs are canonical), then continue. diff --git a/docs/decisions/ADR-001-go-and-dependency-policy.md b/docs/decisions/ADR-001-go-and-dependency-policy.md new file mode 100644 index 0000000..fd5045d --- /dev/null +++ b/docs/decisions/ADR-001-go-and-dependency-policy.md @@ -0,0 +1,39 @@ +# ADR-001: Go with a stdlib-first dependency policy + +Status: Accepted (user-approved language choice, 2026-07-10) + +## Context + +worklog is a security-sensitive local CLI for public OSS release: it parses +private files (shell history, agent logs), promises "no external network +I/O", and will be implemented largely by delegated lower-capability agents. +The user selected Go over Rust/TypeScript/Python. + +## Decision + +- Implementation language: **Go** (minimum toolchain 1.23). +- **Standard library first.** The only third-party module allowed in v1 is + `github.com/BurntSushi/toml` (config parsing). +- CLI flags: stdlib `flag` with per-subcommand FlagSets (no cobra). + Templates: `text/template`. HTTP: `net/http`. JSON: `encoding/json`. +- Any new dependency requires a new ADR stating why the stdlib cannot serve. + +## Consequences + +- Single static binary; easy audit of the no-network invariant (a tiny + dependency graph makes `grep`-level import auditing meaningful, enforced by + `scripts/check_net_imports.sh` in CI). +- Minimal npm/cargo-style supply-chain exposure; `govulncheck` + `go.sum` + pinning cover the remaining surface. +- Some conveniences (cobra help formatting, TUI-grade output) are given up; + acceptable for a report generator. +- TOML chosen over JSON/YAML for the config: comments + human editing, one + small dependency, no YAML parser complexity. + +## Alternatives considered + +- **Rust**: strongest guarantees, higher implementation/review cost for the + delegated-agent workflow. +- **TypeScript/Node**: fastest iteration, but runtime dependency and npm + supply-chain risk contradict the audit story. +- **Python**: distribution weight (pipx/venv) hurts OSS adoption for a CLI. diff --git a/docs/decisions/ADR-002-network-boundary.md b/docs/decisions/ADR-002-network-boundary.md new file mode 100644 index 0000000..a49f0da --- /dev/null +++ b/docs/decisions/ADR-002-network-boundary.md @@ -0,0 +1,45 @@ +# ADR-002: Zero external network; loopback-only LLM endpoint + +Status: Accepted (user requirement + user-approved connectivity choice, +2026-07-10) + +## Context + +The user's core requirement: the tool must never communicate externally +("必ずローカルで外部通信しない"). The user also chose to include local-LLM +summarization in v1, connecting via an OpenAI-compatible API (Ollama, +LM Studio, llama.cpp server) — which is HTTP, i.e. technically "network". + +## Decision + +1. **Invariant I1**: the process performs no network I/O except HTTP(S) to a + **loopback** address for the optional LLM feature. +2. Loopback is enforced **at dial time, per resolved IP** (`net.IP.IsLoopback` + for every address the hostname resolves to), not by string-matching the + URL. A custom `http.Transport` sets `Proxy: nil` (proxy env vars ignored) + and denies all redirects. +3. Non-loopback endpoints are a **hard unsupported case in v1** — no config + escape hatch, no `--allow-remote` flag. Requests are refused with + `ErrNonLoopbackBlocked`; report generation falls back to deterministic + rendering. +4. Only `internal/llm` may import `net`/`net/http`; CI enforces the import + boundary (`scripts/check_net_imports.sh`). +5. No telemetry, no update checks, no crash reporting — ever (documented in + README + SECURITY.md). + +## Consequences + +- The privacy claim is testable and reviewable: unit tests attack the dialer + with public IPs, private-range IPs, and hostnames resolving to them; E2E + runs prove `HTTP_PROXY` is ignored. +- Users with LLMs on another machine (LAN) are not served in v1. Relaxing + this (e.g. explicit allowlist of private addresses) would be a new ADR and + a loud, opt-in flag — deliberately deferred. +- The "browser search enrichment" idea the user floated is confirmed as an + extension-level v2+ concept that must not weaken the v1 core invariant. + +## Alternatives considered + +- Subprocess llama.cpp execution (zero sockets): strongest isolation but poor + model management UX and heavy integration burden. +- Ollama-native API only: narrower compatibility for the same risk profile. diff --git a/docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md b/docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md new file mode 100644 index 0000000..a723722 --- /dev/null +++ b/docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md @@ -0,0 +1,46 @@ +# ADR-003: Read-only source access and stateless runs + +Status: Accepted (user requirement, 2026-07-10) + +## Context + +The user's second core requirement: source files are read, never written +("各種ファイルは読み取りしかしないこと"). Sources include live files (a +running shell appends to `~/.zsh_history`; agent CLIs append to session +JSONL; git repos are in active use). Naive implementations can violate +read-only in non-obvious ways: `git status` refreshes the index, opening +SQLite files creates `-wal`/`-shm` files, incremental scanners keep state. + +## Decision + +1. **Invariant I2**: all source access is read-only. Concretely: + - files opened `O_RDONLY`; no locks taken; torn tails tolerated; + - git invoked only with read-only subcommands (`log`, `rev-parse`) plus + `--no-optional-locks` and `GIT_OPTIONAL_LOCKS=0`; `git status` is + forbidden in the codebase; + - SQLite databases (e.g. `~/.codex/*.sqlite`) are **not opened at all** + (driverless read is unsafe; drivers create sidecar files); + - symlinked directories are not followed during discovery walks. +2. **Invariant I3 / stateless v1**: no cache, no state directory, no log + files. Each run scans fresh. Writes are limited to (a) the explicit output + target, (b) the config file written by `worklog init`. +3. Reporting uncommitted changes is a v1 non-goal because it would require + `git status`/`diff` against the worktree with index-refresh risk. + +## Consequences + +- A full run is verifiable: E2E test hashes the fixture HOME tree before and + after and asserts zero modifications. +- Re-scans cost more than incremental state would, but v1 data volumes + (one day of history/logs, date-sharded Codex dirs, mtime prefilters) keep + runs comfortably fast; incremental state is a v2 option with its own + security review. +- Some data is out of reach (SQLite-only Codex metadata); acceptable — JSONL + covers the needed facts. + +## Alternatives considered + +- go-git (pure-Go read) instead of the git binary: attractive (no exec) but a + large dependency contradicting ADR-001; revisit if exec proves brittle. +- State file for "since last report": deferred; contradicts stateless v1 and + adds a write surface. diff --git a/docs/decisions/ADR-004-redaction-default-on.md b/docs/decisions/ADR-004-redaction-default-on.md new file mode 100644 index 0000000..aa81882 --- /dev/null +++ b/docs/decisions/ADR-004-redaction-default-on.md @@ -0,0 +1,41 @@ +# ADR-004: Redaction on by default; agent logs contribute metadata first + +Status: Accepted (user-approved, 2026-07-10) + +## Context + +Shell history and agent session logs routinely contain live credentials +(`export TOKEN=…`, `curl -H 'Authorization: …'`, pasted keys). worklog's +outputs are precisely the artifacts people paste into team channels (standup +reports), so leaked secrets would propagate. The user chose redaction +default-ON with configurable relaxation. + +## Decision + +1. **Invariant I4**: a built-in redaction ruleset (DESIGN.md §8) runs over + every event's title/body/meta **before aggregation**, and the LLM prompt + builder re-redacts its final payload (defense in depth: nothing unredacted + ever reaches the report model or the LLM). +2. Disabling requires explicit `--redact off` or `redaction.mode = "off"`, + prints a stderr warning banner, and is disclosed in the report footer. +3. Agent sessions are **metadata-first**: titles/counts/projects by default; + raw first-prompt text requires `include_prompts = true` per source, and is + still redacted. +4. Rules are RE2 with stable ids; users can extend (`extra_patterns`) and + exempt false positives (`allowlist`). Entropy-based detection is v2 + (false-positive cost). + +## Consequences + +- Safe-by-default sharing; power users can relax per run or per pattern. +- False positives are possible (a masked non-secret) — mitigated by the + allowlist and by keeping variable *names* visible (only values masked). +- Redaction cannot be perfect; README must say so plainly (defense in depth, + not a guarantee), and the regression corpus grows with reported misses. + +## Alternatives considered + +- Off-by-default (raw fidelity): rejected — unsafe default for the primary + sharing use case. +- Always-strict (titles only, no relaxation): rejected — guts report utility + with no user recourse. diff --git a/docs/decisions/ADR-005-licensing-and-distribution.md b/docs/decisions/ADR-005-licensing-and-distribution.md new file mode 100644 index 0000000..5230cb6 --- /dev/null +++ b/docs/decisions/ADR-005-licensing-and-distribution.md @@ -0,0 +1,52 @@ +# ADR-005: Licensing and distribution posture + +Status: Proposed — license choice **requires explicit user confirmation** +(all other points accepted). Urgency note: the repository was found to be +**already public** on 2026-07-10. + +## Context + +The repository is a public OSS project **as of design time** — it is already +visible on GitHub with no LICENSE file, which legally means +all-rights-reserved: the public can read but not lawfully reuse the code. +Everything pushed to it is immediately public, so the design docs were +scanned for personal data and secrets before the first docs push (clean; the +pre-existing history is a two-line README only). The user has not yet stated +a license preference. + +## Decision + +1. **License: MIT (proposed default).** Rationale: maximal adoption for a + small CLI tool, matches ecosystem norms. Open alternative: Apache-2.0 + (explicit patent grant, better for larger corporate reuse). The LICENSE + file lands via issue 01 **only after the user confirms the choice**; + until then the public repo remains all-rights-reserved. +2. **Distribution v1**: `go install github.com/Saber5656/worklog/cmd/worklog@latest` + plus GitHub Releases with cross-compiled binaries + (darwin/arm64, darwin/amd64, linux/amd64, linux/arm64) and a + `SHA256SUMS` file. Homebrew tap: v2. +3. **Release integrity**: tag-triggered GitHub Actions workflow with pinned + action SHAs, minimal `permissions:` (contents: write only for the release + job), reproducible `-trimpath -ldflags` builds, checksums published. + Artifact signing (Sigstore/cosign): v2. +4. **No telemetry ever** is part of the public product promise (ADR-002). +5. Ongoing requirement (from the user's global policy, adapted to the + already-public reality): every push is a publication — scan content for + personal data and secrets **before each push**, not as a one-time + go-public gate. History to date: initial README commit + docs commits, + verified clean. + +## Consequences + +- v1 issues include LICENSE placement (01), release workflow (32), and + security/privacy documentation (31, 33). +- Because the repo is already public, the LICENSE decision blocks issue 01's + LICENSE file but nothing else; code contributions before the LICENSE lands + would be legally murky for outside contributors, so confirming the license + is the most urgent open decision. +- If the user chooses Apache-2.0 instead, only the LICENSE file and README + badge change; no code impact. + +## Open question for the user + +- Confirm MIT vs Apache-2.0 (tracked in the design PR description). diff --git a/docs/issues/01-repo-scaffolding.md b/docs/issues/01-repo-scaffolding.md new file mode 100644 index 0000000..18ff904 --- /dev/null +++ b/docs/issues/01-repo-scaffolding.md @@ -0,0 +1,99 @@ +# Title + +Repository scaffolding: Go module, layout, Makefile, LICENSE + +# Summary + +Initialize the Go module and the fixed directory skeleton from DESIGN.md +§3.2, add a Makefile with the standard developer targets, a `.gitignore`, an +`.editorconfig`, and the LICENSE file, so that every later issue lands into a +stable layout with working `make build` / `make test`. + +# Context + +The repository currently contains only `README.md`. All 32 later issues +assume the module path, directory layout, and Make targets defined here. +DESIGN.md §3.3 fixes the dependency policy (stdlib + `BurntSushi/toml` only). + +Prerequisite for the implementing agent (environment setup, NOT part of this +issue's deliverables): a Go toolchain ≥ 1.23 must be available +(`go version`). On the reference macOS machine Go is NOT yet installed; +installing it (e.g. `brew install go`) is explicitly permitted setup work, +performed before starting, with the installed version recorded in the PR +description. The repository-scoped acceptance criteria below apply to the +issue's deliverables only, not to toolchain installation. + +# Scope + +- `go.mod` — module `github.com/Saber5656/worklog`, `go 1.23`. +- Directory skeleton with placeholder `doc.go` files (package comment only) + for: `cmd/worklog`, `internal/cli`, `internal/config`, `internal/timeutil`, + `internal/model`, `internal/sanitize`, `internal/redact`, + `internal/source`, `internal/source/jsonlutil`, `internal/source/zsh`, + `internal/source/gitsrc`, `internal/source/claudecode`, + `internal/source/codex`, `internal/aggregate`, `internal/render`, + `internal/llm`, `internal/output`, `internal/version`. +- `cmd/worklog/main.go` — prints `worklog: not yet implemented` to stderr and + exits 2 (replaced by issue 07). +- `internal/version/version.go` — `var Version = "dev"`, `var Commit = ""` + (populated via `-ldflags` later). +- `Makefile` — targets: `build` (`go build -trimpath -o bin/worklog + ./cmd/worklog`), `test` (`go test -race ./...`), `lint` + (`gofmt -l . && go vet ./...`; golangci-lint added by issue 02), `clean`. +- `.gitignore` — `bin/`, coverage files, `.DS_Store`. +- `.editorconfig` — tabs for `.go`, spaces(2) for `.yml`/`.md`, final newline. +- `LICENSE` — MIT, copyright `2026 Saber5656` (per ADR-005; flagged as + pending final user confirmation before the repo goes public — do NOT + publish the repository in this issue). + +# Detailed Requirements + +1. `go.mod` must not declare any `require` yet (BurntSushi/toml is added by + issue 05 when first imported). +2. Every `internal/...` placeholder package must contain exactly one + `doc.go` with a one-sentence package comment matching its DESIGN.md §3.2 + role, so `go build ./...` compiles the full tree from day one. +3. `main.go` must contain no logic besides the stderr message and + `os.Exit(2)`. +4. Makefile must work on macOS (BSD make compatibility: no GNU-only + features) and print the binary path on `make build` success. +5. Do not create `docs/` content, CI files (issue 02), or any parser code. + +# Acceptance Criteria + +- [ ] `go build ./...` and `make build` succeed on a clean checkout. +- [ ] `./bin/worklog` exits with code 2 and prints the placeholder line to + stderr, nothing to stdout. +- [ ] `make test` passes (no tests yet ⇒ passes trivially). +- [ ] `gofmt -l .` prints nothing. +- [ ] `go.mod` has zero dependencies; module path is exactly + `github.com/Saber5656/worklog`. +- [ ] LICENSE file is valid MIT text with the correct year/holder — ONLY if + the user has confirmed MIT by then (ADR-005 open question; the repo + is already public, so pushing an unconfirmed license would publish a + wrong legal statement. If unconfirmed, skip the LICENSE file and note + it in the PR). +- [ ] No file outside the repository is created or modified by this issue's + deliverables (toolchain installation done as a prerequisite is exempt). + +# Validation + +Run and paste into the PR: `go version`, `make build && ./bin/worklog; +echo "exit=$?"`, `make test`, `gofmt -l .`. Verify the directory tree matches +DESIGN.md §3.2 with `find . -name doc.go | sort`. + +# Dependencies + +None (first issue). + +# Non-goals + +CI (02), real CLI dispatch (07), any parsing/rendering logic, README rewrite +(31), release tooling (32), changing repository visibility (the repo is +already public). + +# Design References + +- `docs/DESIGN.md` §3.2 (module layout), §3.3 (dependency policy), §16 +- `docs/decisions/ADR-001-go-and-dependency-policy.md` +- `docs/decisions/ADR-005-licensing-and-distribution.md` diff --git a/docs/issues/02-ci-workflow.md b/docs/issues/02-ci-workflow.md new file mode 100644 index 0000000..7c771a1 --- /dev/null +++ b/docs/issues/02-ci-workflow.md @@ -0,0 +1,93 @@ +# Title + +CI workflow: build, lint, test, vulnerability and boundary checks + +# Summary + +Add the GitHub Actions CI pipeline that every later PR must keep green: +build + `gofmt` + `go vet` + `golangci-lint` + `go test -race` + +`govulncheck` + `go mod tidy` drift check + the network import-boundary +script, on ubuntu and macos runners. + +# Context + +DESIGN.md §14 defines CI as the enforcement point for the testing strategy, +and §3.2 fixes an import boundary (only `internal/llm` may import +`net`/`net/http`; only `internal/source/gitsrc` may import `os/exec`) that +makes invariants I1/I2 mechanically auditable. The boundary script created +here is extended into a full invariant suite by issue 30. + +# Scope + +- `.github/workflows/ci.yml` +- `scripts/check_net_imports.sh` +- `.golangci.yml` + +# Detailed Requirements + +1. Workflow `ci.yml`: + - Triggers: `pull_request`, and `push` to `main`. + - Top-level `permissions: contents: read` (least privilege). + - Matrix: `ubuntu-latest`, `macos-latest`; Go from `go.mod` + (`actions/setup-go` with `go-version-file: go.mod`, caching on). + - All third-party actions pinned to a full commit SHA (not a tag), with + the tag noted in a comment (per ADR-005 release-integrity posture). + - Steps in order: checkout → setup-go → `gofmt -l .` (fail if output + non-empty) → `go vet ./...` → golangci-lint → `go build ./...` → + `go test -race ./...` → `govulncheck ./...` → tidy check → + `shellcheck scripts/*.sh` (ubuntu job only; shellcheck is preinstalled + on ubuntu runners) → `scripts/check_net_imports.sh`. + - Tidy check: `go mod tidy && git diff --exit-code go.mod go.sum` (tolerate + absent `go.sum` while there are no deps). + - `govulncheck`: install with a pinned version + (`golang.org/x/vuln/cmd/govulncheck@vX.Y.Z`, latest at implementation + time; record the chosen version in the PR). +2. `scripts/check_net_imports.sh` (POSIX sh, no bashisms): + - Uses `go list -deps -json ./...` or `go list -f` over `./internal/...` + and `./cmd/...` to assert: + - packages importing `net` or `net/http` ⊆ {`internal/llm`}, + - packages importing `os/exec` ⊆ {`internal/source/gitsrc`}, + - no package imports `net/rpc`, `net/smtp`, or `net/mail`. + - Prints each violation as `VIOLATION imports ` and + exits 1 on any; exits 0 silently otherwise. + - Must pass on the issue-01 skeleton (no such imports exist yet). +3. `.golangci.yml`: enable at minimum `govet`, `staticcheck`, `errcheck`, + `ineffassign`, `unused`, `misspell`, `gosec`; exclude `testdata/`; + golangci-lint invoked via its official action (SHA-pinned) with a pinned + lint version. +4. Badge/README edits are out of scope (issue 31). + +# Acceptance Criteria + +- [ ] CI runs on a PR touching any file and completes green on both OSes + against the issue-01 tree. +- [ ] Introducing `import "net/http"` into `internal/model` in a scratch + commit makes `check_net_imports.sh` (and thus CI) fail — demonstrated + in the PR description with the failing output, then reverted. +- [ ] All actions referenced by full commit SHA; workflow has explicit least + `permissions`. +- [ ] `scripts/check_net_imports.sh` is executable, POSIX-sh clean + (`sh -n`), and passes `shellcheck` with no errors. +- [ ] CI completes in under 10 minutes per OS on the skeleton tree. + +# Validation + +Open a draft PR to trigger the workflow; paste the green run URL. Include +the demonstrated red run URL (or log excerpt) for the boundary violation +test. Run `shellcheck scripts/check_net_imports.sh` locally. + +# Dependencies + +01. + +# Non-goals + +Release workflow (32), E2E/invariant suite content (30), coverage upload, +CodeQL/Dependabot configuration (repo-hardening is handled outside the issue +plan by the repository owner). + +# Design References + +- `docs/DESIGN.md` §3.2 (import boundary), §12.2 (supply chain row), §14 (CI) +- `docs/decisions/ADR-002-network-boundary.md` (boundary rationale) +- `docs/decisions/ADR-005-licensing-and-distribution.md` (pinned actions) diff --git a/docs/issues/03-model-package.md b/docs/issues/03-model-package.md new file mode 100644 index 0000000..08c2400 --- /dev/null +++ b/docs/issues/03-model-package.md @@ -0,0 +1,97 @@ +# Title + +internal/model: Event, Report, Warning types and ordering + +# Summary + +Implement the canonical data types every other package exchanges: `Event` +(one activity record), the `Report` tree, `Warning`, the enums, and the +deterministic sort/dedup helpers, exactly as specified in DESIGN.md §6.2–§6.3 +and §13. + +# Context + +Providers (issues 09–15) emit `[]Event`; aggregation (18) consumes them and +produces `Report`; renderers (20–21) and the LLM prompt builder (24) consume +`Report`. Deterministic ordering here is what makes golden-file testing +possible everywhere downstream. + +# Scope + +- `internal/model/event.go`, `report.go`, `warning.go`, plus `_test.go` + files. No I/O, no dependencies outside stdlib. + +# Detailed Requirements + +1. Types exactly per DESIGN.md §6.2/§6.3: + - `type SourceID string` with constants `SourceZsh("zsh")`, + `SourceGit("git")`, `SourceClaudeCode("claude-code")`, + `SourceCodex("codex")`. + - `type EventKind string` with `KindCommand`, `KindCommit`, + `KindAgentSession` (values `"command"`, `"commit"`, `"agent-session"`). + - `Event{Source, Kind, Start, End, Project, Ref, Title, Body, Meta}`. + - `Warning{Source SourceID, Code, Message string}` — `Code` values are the + stable strings from DESIGN §13; define them as constants in this + package (e.g. `WarnZshNoTimestamps = "zsh_no_timestamps"`), one constant + per code listed there. + - `Report`, `ReportKind` (`"daily"`, `"standup"`), `ProjectActivity`, + `AgentSession`, `Commit`, `ShellActivity`, `CommandGroup`, `Totals`, + `Narrative{Text, Model string}`, `GenerationMeta{Version string, + Sources []SourceID, LLMModel string, RedactionMode string}`, + `StandupData{YesterdayDate string, Yesterday []ProjectActivity, + Today []ProjectActivity, YesterdayShell, TodayShell ShellActivity, + YesterdayTotals, TodayTotals Totals}`. + - `Commit{Repo, Hash, Subject string, When time.Time, Files, Insertions, + Deletions int}`; `AgentSession{Agent, Title, Ref string, Start, End + time.Time, UserMsgs, AgentMsgs, Tasks int}`; `CommandGroup{Head string, + Count int, First, Last time.Time, Examples []string}`; + `Totals{Commits, Insertions, Deletions, Sessions, AgentUserMsgs, + Commands int, SpanStart, SpanEnd time.Time}`. +2. Ordering helpers (pure functions, stable): + - `SortEvents([]Event)` — by Start, then Source, then Ref, then Title. + - `SortProjects([]ProjectActivity)` — activity score desc + (`3*len(Commits) + 2*len(Sessions)`), then name asc — exactly the + DESIGN §6.3 rule (shell commands are unattributed in v1 and never + contribute to the score). + - Within `ProjectActivity`: commits by `When` asc then Hash; sessions by + `Start` asc then Ref; command groups by Count desc then Head asc. +3. `Event.Validate() error` — non-zero Start, known Source/Kind, End zero or + ≥ Start. +4. Dedup helper `DedupSessions([]Event) []Event` — for `KindAgentSession` + events with non-empty `Ref`, keep the first occurrence per (Source, Ref) + pair; events with empty `Ref` are never deduplicated (unrelated sessions + must not collapse); preserve order. +5. No JSON tags on `Report` yet except where needed by the LLM prompt + builder later — add lower_snake_case JSON tags to all Report-tree structs + now (they are also useful for tests). + +# Acceptance Criteria + +- [ ] All types and constants above exist with the exact names/values. +- [ ] Sort helpers are deterministic under permutation (property-style table + tests shuffle inputs and assert identical output). +- [ ] `DedupSessions` collapses two events with identical (Source, Ref) to + one (the live-vs-archived Codex case) and never collapses events with + empty Ref (two empty-Ref sessions stay two). +- [ ] `Event.Validate` rejects: zero Start, unknown Source, unknown Kind, + `End < Start` — one test case each. +- [ ] Package has zero non-stdlib imports; ≥ 90% statement coverage. + +# Validation + +`go test -race -cover ./internal/model/` output pasted into the PR; +`scripts/check_net_imports.sh` still green. + +# Dependencies + +01, 04 (`Report.Range` is a `timeutil.Range`; timeutil has no model +dependency, so no cycle). + +# Non-goals + +Aggregation logic (18), warning *emission* (sources do that), rendering, +serialization formats beyond struct tags. + +# Design References + +- `docs/DESIGN.md` §6.2, §6.3, §9.3 (ordering), §13 (warning codes) diff --git a/docs/issues/04-timeutil-package.md b/docs/issues/04-timeutil-package.md new file mode 100644 index 0000000..0b7a36a --- /dev/null +++ b/docs/issues/04-timeutil-package.md @@ -0,0 +1,81 @@ +# Title + +internal/timeutil: report range resolution + +# Summary + +Implement the date/time helpers that turn CLI inputs (`--date`, configured +timezone) into concrete half-open time ranges, including the +previous-workday logic used by standup (DESIGN.md §9.1–§9.2). + +# Context + +Every provider filters events by a `timeutil.Range`; `daily` needs "that day +in the report timezone"; `standup` needs a yesterday bucket (previous +workday) and a today bucket (midnight → now). Getting timezone and DST +boundaries right here prevents a whole class of off-by-one-day bugs. + +# Scope + +- `internal/timeutil/timeutil.go` + tests. Stdlib only. + +# Detailed Requirements + +1. `type Range struct { Start, End time.Time }` — half-open `[Start, End)`, + with `Contains(t time.Time) bool` and `Overlaps(start, end time.Time) + bool` (for span events: `end` may be zero ⇒ treat as instantaneous). +2. `LoadLocation(name string) (*time.Location, error)` — `""` → `time.Local`; + otherwise `time.LoadLocation` (errors propagate for config validation). +3. `ParseDate(s string, loc *time.Location) (Day, error)` — strict + `YYYY-MM-DD` (`time.ParseInLocation("2006-01-02")`; reject anything else, + including `2026-7-1`). `type Day struct { Y int; M time.Month; D int; + Loc *time.Location }` with `String() "2026-07-10"`. +4. `Today(now time.Time, loc *time.Location) Day`. +5. `DayRange(d Day) Range` — midnight to next midnight via + `time.Date(...).AddDate(0,0,1)` (DST-safe: a 23h/25h day is whatever the + location says, never a fixed 24h addition). +6. `PrevWorkday(d Day) Day` — subtract one day; while result is Saturday or + Sunday keep subtracting. `PrevDay(d Day) Day` — literal minus one. +7. `StandupBuckets(d Day, now time.Time, previousWorkday bool) + (yesterday Range, today Range, yesterdayDay Day)`: + - `yesterday` = full `DayRange` of `PrevWorkday(d)` (or `PrevDay` when + `previousWorkday == false`). + - `today` = `[midnight(d), now)` when `d` is the same calendar day as + `now` in `d.Loc`; otherwise full `DayRange(d)` (retro-generation, per + DESIGN §9.2). +8. All functions pure; `now` always injected (no `time.Now()` inside the + package) so tests are deterministic. + +# Acceptance Criteria + +- [ ] Table tests cover: normal weekday; Monday→Friday lookback (with + `previousWorkday=true`) and Monday→Sunday (false); Sunday and Saturday + standup dates; year boundary (Jan 1); DST spring-forward and fall-back + days in `America/New_York` (23h and 25h day lengths asserted via + `End.Sub(Start)`); `Asia/Tokyo` no-DST sanity. +- [ ] `ParseDate` rejects: `2026-7-01`, `2026/07/01`, `20260701`, + `2026-13-01`, `2026-02-30`, empty string. +- [ ] `Range.Overlaps` correct for: instant inside, instant at End + (excluded), span straddling Start, span fully before/after. +- [ ] No `time.Now()` call inside the package (grep asserted in test or + review). +- [ ] ≥ 95% statement coverage; stdlib-only imports. + +# Validation + +`go test -race -cover ./internal/timeutil/` pasted into PR. Reviewer spot +check: `TZ=America/New_York go test ./internal/timeutil/` also green +(host-TZ independence). + +# Dependencies + +01. + +# Non-goals + +Holiday calendars (v1 non-goal), week/month ranges (v2 rollups), parsing +times from source files (each parser owns its own timestamp parsing). + +# Design References + +- `docs/DESIGN.md` §4 (`--date`), §9.1, §9.2 diff --git a/docs/issues/05-config-package.md b/docs/issues/05-config-package.md new file mode 100644 index 0000000..c62143b --- /dev/null +++ b/docs/issues/05-config-package.md @@ -0,0 +1,102 @@ +# Title + +internal/config: TOML schema, defaults, validation + +# Summary + +Implement configuration loading exactly per DESIGN.md §5: TOML file at the +XDG path, zero-config defaults, tilde expansion, strict validation with all +errors reported at once, and unknown-key warnings. + +# Context + +Every subcommand starts by loading config; `worklog init` (27) writes the +starter file; `doctor` (28) re-uses validation. This issue introduces the +project's single third-party dependency (`github.com/BurntSushi/toml`, +ADR-001). + +# Scope + +- `internal/config/config.go`, `defaults.go`, `validate.go`, tests, and + `testdata/*.toml` fixtures. Adds the toml dependency to `go.mod`. + +# Detailed Requirements + +1. Structs mirroring DESIGN §5's TOML exactly (`General`, `Output`, + `Sources`, `SourcesZsh`, `SourcesGit`, `SourcesClaudeCode`, + `SourcesCodex`, `LLM`, `Redaction`, `Standup`), field tags matching the + TOML key names shown there (`toml:"history_file"` etc.). +2. `Default() *Config` returns the documented defaults verbatim (language + "ja", git root `~/dev`, **`llm.enabled` false (opt-in)**, llm endpoint + `http://127.0.0.1:11434/v1`, `timeout_seconds` 120, `max_input_chars` + 24000, `max_output_chars` 4000, `temperature` 0.2, redaction "on", + `standup.previous_workday` true, sources enabled: all four, `max_depth` + 3, exclude_dirs per DESIGN §5). +3. `DefaultPath() string` — `$XDG_CONFIG_HOME/worklog/config.toml` when + `XDG_CONFIG_HOME` is set and absolute, else `~/.config/worklog/config.toml`. +4. `Load(path string) (*Config, []string, error)`: + - `path == ""` → `DefaultPath()`; missing file → defaults, no error. + - Decode with `toml.DecodeFile` using `md.Undecoded()` to produce one + warning string per unknown key (typo detection), returned as the second + value — never an error. + - After decode, merge onto defaults (decode into a copy of `Default()` so + absent keys keep defaults). +5. `ExpandPath(p string) string` — `~` and `~/` expansion via + `os.UserHomeDir`; no environment-variable expansion (predictability rule, + DESIGN §5). Applied by accessor methods (`c.ZshHistoryFile()`, + `c.GitRoots()`, `c.ClaudeProjectsDir()`, `c.CodexSessionsDir()`, + `c.CodexSessionIndex()`, `c.CodexArchivedDir()`, `c.OutputDirectory()`) + rather than mutating stored values (round-trip fidelity for `init`). + `c.ZshHistoryFile()` implements the resolution order: configured value → + `$HISTFILE` (absolute paths only) → `~/.zsh_history`. +6. `Validate() []error` collecting ALL violations, per DESIGN §5 rules: + language ∈ {ja,en}; timezone loads; every regex in + `sources.zsh.exclude_patterns`, `redaction.extra_patterns`, + `redaction.allowlist` compiles as RE2; `sources.enabled` values ∈ the four + known ids (unknown id = error, not warning); `llm.enabled && model==""` → + error text mentioning `worklog doctor` (reachable only via an explicit + config file since the default is `enabled=false` — zero-config always + validates); endpoint URL: scheme http/https + non-empty host (port + optional); static loopback pre-check: if the host is an IP literal it + must be loopback (hostnames are checked at dial time, DESIGN §5/§11); + numeric bounds exactly as listed (`timeout_seconds` 1–600, + `max_input_chars` 1000–1000000, `max_output_chars` 100–100000, + `temperature` 0–2, `max_depth` 1–6); `redaction.mode` ∈ {on,off}. +7. `Starter() string` returns the commented starter TOML used by + `worklog init` (content = DESIGN §5 block with explanatory comments; + `[llm]` section ships `enabled = false` with the comment + `# set enabled = true and model = "..." after running worklog doctor`). + Keep it in a `const`; issue 27 writes it to disk. + +# Acceptance Criteria + +- [ ] Zero-config: `Load("")` with no file present returns defaults with no + warnings/errors. +- [ ] Fixture matrix: full valid file; file with unknown key (warning lists + the exact dotted key); each validation rule violated once (one fixture + or table case per rule) with `Validate()` returning ALL errors + together, not just the first. +- [ ] `ZshHistoryFile()` order proven with tests manipulating `HISTFILE` + (set-absolute, set-relative→ignored, unset). +- [ ] Starter TOML from `Starter()` round-trips: decodes with zero unknown + keys and equals `Default()` after load. +- [ ] Only new dependency in `go.mod` is `github.com/BurntSushi/toml`. + +# Validation + +`go test -race -cover ./internal/config/` in PR; `go mod tidy` diff empty; +boundary script green. + +# Dependencies + +01. + +# Non-goals + +Writing the config file (27), flag handling (07), dial-time loopback +enforcement (23). + +# Design References + +- `docs/DESIGN.md` §5, §11.1 +- `docs/decisions/ADR-001-go-and-dependency-policy.md` diff --git a/docs/issues/06-sanitize-package.md b/docs/issues/06-sanitize-package.md new file mode 100644 index 0000000..afc1198 --- /dev/null +++ b/docs/issues/06-sanitize-package.md @@ -0,0 +1,92 @@ +# Title + +internal/sanitize: control-char stripping and Markdown escaping + +# Summary + +Implement the text-hygiene primitives applied to all untrusted source +content: control-character stripping (terminal-escape defense), UTF-8 +repair, length capping, single-line flattening, and Markdown escaping for +table cells and content lines (DESIGN.md §12.1 B6, §12.2). + +# Context + +History lines, commit subjects, and agent session titles are untrusted bytes +(invariant I5). If rendered raw they can smuggle ANSI escape sequences into +terminals or break/forge Markdown structure in reports. Aggregation (18) and +renderers (20/21) call these helpers; parsers use the UTF-8/caps primitives. + +# Scope + +- `internal/sanitize/sanitize.go` + tests. Stdlib only. + +# Detailed Requirements + +1. `Clean(s string) string`: + - Repair invalid UTF-8 (`strings.ToValidUTF8(s, "�")`). + - Remove all C0 controls except `\n` and `\t`; remove `\r` (normalize + CRLF→LF first); remove DEL (0x7F) and C1 controls (U+0080–U+009F). + - Remove Unicode line/paragraph separators (U+2028, U+2029) and BOM. + - Leaves all other printable Unicode intact (Japanese text must pass + through unchanged — test with CJK + emoji). +2. `Line(s string, max int) string` — `Clean`, then replace `\n`/`\t` runs + with a single space, trim, and cap to `max` runes appending `…` when + truncated (`max ≤ 0` = no cap). Truncation must not split a rune. +3. `Cap(s string, max int) string` — rune-safe cap with `…`, preserving + newlines (used for `Body`/narrative). +4. `EscapeCell(s string) string` — for Markdown table cells: apply `Line` + semantics (no cap), then escape `|` as `\|`, backtick as `` \` ``, and + HTML-escape `<` and `&` (`<`, `&`). +5. `EscapeText(s string) string` — for non-table content lines: `Clean`, + HTML-escape `<` and `&`, and neutralize a leading `#`, `-`, `*`, `>`, or + digit-dot list marker by prefixing `\` (prevents structure forgery when a + subject starts a line). +6. Idempotence contract (documented in doc comments and asserted in tests + over the corpus): `Clean`, `Line`, and `Cap` are idempotent + (`f(f(x)) == f(x)`). `EscapeCell` and `EscapeText` are **deliberately NOT + idempotent** (escaping `&` twice yields `&amp;`): they are + single-application functions applied exactly once, at render time, by the + renderer — never by parsers or aggregation. Their doc comments must state + this. + +# Acceptance Criteria + +- [ ] ANSI corpus test: strings containing `\x1b]0;evil\x07`, + `\x1b[31mred\x1b[0m`, `\x9b31m`, raw `\x07` come out with all escape + bytes removed (assert no byte < 0x20 except `\n`/`\t` remains, no + 0x7F, no C1). +- [ ] CJK/emoji passthrough: Japanese sentences and emoji unchanged by + `Clean`. +- [ ] Rune-safe truncation: capping inside a multi-byte rune never yields + invalid UTF-8 (fuzz-style table over Japanese strings with varying + caps). +- [ ] `EscapeCell("a|b`c`, + run `Validate()` — any error → all errors to stderr, exit 2. Then parse + `--date` (via timeutil, in the configured timezone) — invalid → exit 2. + Then call the stub, which returns an error → message to stderr, exit 1. + (Stub exit-1 is a TEMPORARY scaffolding exception to the DESIGN §13 + exit-code contract, removed as issues 25/26/28/29 replace the stubs; + mark each stub with a `// TODO(issue NN)` comment.) + - `version`: prints `worklog ()` (from + `internal/version`, values "dev"/"none" when unset) to stdout, exit 0. +4. `--lang` overrides `general.language`; `--redact` overrides + `redaction.mode`; `--no-llm` overrides `llm.enabled`; overrides are + applied to the in-memory config copy handed to subcommands. +5. Diagnostics discipline (DESIGN §4): stubs and all harness messages write + only to stderr; stdout is reserved for reports/`version` output. +6. `--verbose` stores a flag on the context struct passed to subcommands + (used later); no logging framework. + +# Acceptance Criteria + +- [ ] Exit-code matrix test via `cli.Main`: no args→2, `--help`→0, unknown + subcommand→2, `daily --date bad`→2, `daily` (stub)→1, `version`→0, + `daily --redact maybe`→2, `daily --source nope`→2, `daily --save` + without configured output directory→2 (checked here in the harness, + per DESIGN §13). +- [ ] `version` prints exactly one line to stdout; nothing to stderr. +- [ ] Config warnings/errors surface as specified (fixture config with typo + key + one invalid value; assert stderr contents and exit code). +- [ ] Stubs write nothing to stdout. +- [ ] `gofmt`/vet/lint green; no new dependencies. + +# Validation + +`go test -race -cover ./internal/cli/` plus a manual transcript in the PR: +`worklog`, `worklog help`, `worklog version`, `worklog daily`, +`worklog daily --date 2026-07-10` against an empty HOME (expected stub +error, exit 1). + +# Dependencies + +01, 05 (config), 04 (date parsing). + +# Non-goals + +Real subcommand behavior (25–29), output writing (22), doctor checks (28). + +# Design References + +- `docs/DESIGN.md` §4, §13 diff --git a/docs/issues/08-source-provider-collector.md b/docs/issues/08-source-provider-collector.md new file mode 100644 index 0000000..eabebb6 --- /dev/null +++ b/docs/issues/08-source-provider-collector.md @@ -0,0 +1,90 @@ +# Title + +internal/source: Provider interface and concurrent collector + +# Summary + +Define the `Provider` interface (the future plugin boundary) and implement +the collector that runs enabled providers concurrently with per-source +timeouts, gathering events and warnings without letting any single source +fail the run (DESIGN.md §3.1, §6.1). + +# Context + +Four providers (10, 12→via gitsrc provider, 14, 15) implement this +interface; `daily`/`standup`/`sources` commands call the collector. The +"sources degrade, reports never fail" property lives here. + +# Scope + +- `internal/source/provider.go`, `collector.go`, tests with fake providers. + +# Detailed Requirements + +1. Interface exactly per DESIGN §6.1: + + ```go + type Provider interface { + ID() model.SourceID + Collect(ctx context.Context, rng timeutil.Range) ([]model.Event, []model.Warning, error) + } + ``` + +2. `Collect(ctx, providers []Provider, rng timeutil.Range, opts Options) + Result`: + - `Options{PerSourceTimeout time.Duration}` (default 30s when zero); + `Result{Events []model.Event, Warnings []model.Warning}`. + - Runs each provider in its own goroutine with + `context.WithTimeout(ctx, PerSourceTimeout)`. + - Timeout or context cancellation → warning + `model.WarnSourceTimeout` (`source_timeout`) carrying the provider id; + provider results arriving after timeout are discarded. + - Provider error → converted to a warning + (`Code = string(id)+"_failed"` unless the provider already returned + typed warnings; the error message becomes the warning message); + partial events returned alongside an error are kept. + - Panic in a provider is recovered → warning (`_panic`), run + continues (a malformed source file must never crash the CLI — + invariant I5). + - Output ordering is deterministic: events sorted with + `model.SortEvents`, warnings sorted by (Source, Code, Message) — + regardless of goroutine completion order. +3. `FilterByIDs(providers []Provider, ids []model.SourceID) []Provider` — + supports the `--source` flag (order-preserving; unknown ids are the CLI's + problem, already validated in 07). +4. Providers receive their config at construction (each provider issue owns + its constructor); the collector knows nothing about config. +5. Zero providers → empty Result, no warnings (the CLI decides what an + empty report looks like). + +# Acceptance Criteria + +- [ ] Fake-provider tests: fast+slow (slow exceeds timeout → its events + dropped, `source_timeout` warning present, fast source unaffected); + erroring provider (warning, others unaffected); panicking provider + (recovered, warning, others unaffected); partial-events-with-error + kept. +- [ ] Determinism: run the same mixed set 50× and assert byte-identical + marshaled Result (ordering rule works under scheduling variance). +- [ ] Context cancellation from the caller stops in-flight providers + (fake provider observes ctx.Done within its loop; asserted). +- [ ] `-race` clean. +- [ ] Package imports: stdlib + internal/model + internal/timeutil only. + +# Validation + +`go test -race -count=20 ./internal/source/` (repeat count shakes out +ordering flakes) pasted into the PR. + +# Dependencies + +03, 04, 05 (types only — providers take typed config structs defined in 05). + +# Non-goals + +Any real source parsing (09–15), sanitize/redact stages (18 owns them), +subprocess plugin execution (v2, ADR/DESIGN non-goal). + +# Design References + +- `docs/DESIGN.md` §3.1 (pipeline), §6.1 (interface), §13 (`source_timeout`) diff --git a/docs/issues/09-zsh-history-parser.md b/docs/issues/09-zsh-history-parser.md new file mode 100644 index 0000000..a571dc6 --- /dev/null +++ b/docs/issues/09-zsh-history-parser.md @@ -0,0 +1,91 @@ +# Title + +zsh history parser: formats, unmetafy, multiline, caps + +# Summary + +Implement the pure parsing layer for zsh history files: unmetafication, +simple + extended format detection, multiline entry joining, and the +size/line caps — exactly per `docs/research/zsh-history-format.md`. + +# Context + +This is parsing only (bytes → entries); provider glue (date filtering, +config, warnings) is issue 10. The reference machine's history is +simple-format (no timestamps), and oh-my-zsh machines are extended-format; +both must parse. Metafied bytes (0x83 escapes) corrupt Japanese commands if +skipped — this is the most commonly missed zsh detail. + +# Scope + +- `internal/source/zsh/parser.go` + tests + `testdata/` fixture files. + +# Detailed Requirements + +1. `type Entry struct { Start time.Time; Duration time.Duration; Command + string; HasTimestamp bool }`. +2. `Unmetafy([]byte) []byte` — exact algorithm from the research doc + (0x83 marker, next byte XOR 0x20; trailing lone 0x83 dropped). Operates + on raw bytes BEFORE any string conversion. +3. `ParseReader(r io.Reader, caps Caps) (entries []Entry, stats Stats)`: + - `Caps{MaxLineBytes int (default 64*1024), MaxFileBytes int64 (default + 128*1024*1024)}`; `Stats{Lines, Oversized, Malformed int; Truncated + bool; ExtendedCount, SimpleCount int}`. + - Streaming scan (`bufio.Reader`, manual line assembly — `bufio.Scanner` + token limits must not abort the file: an oversized line is consumed, + counted in `Oversized`, and skipped). + - Per physical line: unmetafy → UTF-8 repair (`sanitize.Clean` is NOT + applied here; only `strings.ToValidUTF8` — commands keep their tabs + etc.; display hygiene happens downstream) → multiline joining: while + the line ends with an odd number of `\`, strip the final `\`, append + `\n`, read next physical line (a line starting with `": "` while in + extended mode terminates a malformed continuation; count `Malformed`). + - Entry header: `^: (\d{10,}):(\d+);(.*)$` → extended entry + (`HasTimestamp=true`, Start from epoch, Duration seconds); otherwise + the whole line is a simple entry (`HasTimestamp=false`). + - Stop reading at `MaxFileBytes` (set `Truncated`). + - Empty lines are skipped (not Malformed). +4. No file I/O in this file beyond the `io.Reader` (testability); no + goroutines. + +# Acceptance Criteria + +- [ ] Fixture `extended.hist`: ≥ 5 extended entries incl. one with + duration > 0; timestamps parse to the exact epochs. +- [ ] Fixture `simple.hist`: plain commands; all `HasTimestamp=false`. +- [ ] Fixture `metafied.hist`: contains a Japanese command stored with real + 0x83 metafication (fixture built by a small Go generator in + `testdata/gen/main.go`, committed with its output); parsed Command + equals the original Japanese string. +- [ ] Fixture `multiline.hist`: an extended entry whose command spans 3 + physical lines via trailing backslashes → one Entry with two `\n`. +- [ ] Fixture `torn.hist`: file ending mid-entry (no trailing newline, + dangling continuation) → parsed without error; tail counted Malformed + or yielded as-is (documented choice: count Malformed, drop). +- [ ] Oversized line fixture (>64 KiB) → skipped, `Oversized==1`, following + entries still parse. +- [ ] Mixed file (simple + extended lines) → both counted in Stats. +- [ ] `FuzzParseReader` fuzz target (seeded with all fixtures) runs 30s + locally with no panics and no invalid-UTF-8 Command output. +- [ ] ≥ 90% coverage. + +# Validation + +`go test -race -cover ./internal/source/zsh/` and 30s fuzz run output in the +PR. Reviewer check: `Unmetafy` table includes the byte pairs +`0x83 0xC3 → 0xE3` (start of Japanese UTF-8) and lone-trailing-0x83. + +# Dependencies + +01, 06 (only for shared constants if any; parser itself uses +`strings.ToValidUTF8` — keep sanitize out of the parse path). + +# Non-goals + +Date filtering, HISTFILE resolution, warnings, Events (all issue 10); +bash/fish formats (v2). + +# Design References + +- `docs/research/zsh-history-format.md` (normative format spec) +- `docs/DESIGN.md` §7.1 diff --git a/docs/issues/10-zsh-provider.md b/docs/issues/10-zsh-provider.md new file mode 100644 index 0000000..c7b767e --- /dev/null +++ b/docs/issues/10-zsh-provider.md @@ -0,0 +1,92 @@ +# Title + +zsh source provider: resolution, filtering, warnings + +# Summary + +Wrap the zsh parser (09) in a `source.Provider`: resolve the history file +path, open it read-only, filter entries to the report range, apply +`exclude_patterns`, and emit Events and the documented warnings +(DESIGN.md §7.1). + +# Context + +This provider is where the "simple-format history cannot be date-filtered" +reality becomes a clean user-facing behavior: zero events + a +`zsh_no_timestamps` warning that `doctor` (28) later explains. + +# Scope + +- `internal/source/zsh/provider.go` + tests. + +# Detailed Requirements + +1. `New(cfg config.SourcesZsh, resolve func() string) *Provider` — `resolve` + is the already-config-aware path resolver from 05 + (`config.ZshHistoryFile()`); provider stores the resolved path lazily at + Collect time (HISTFILE may differ between construction and run in tests). +2. `ID()` returns `model.SourceZsh`. +3. `Collect`: + - Missing file → warning `zsh_history_missing` (message includes the + resolved path), zero events, nil error. + - Open `os.Open` (read-only); never create/lock. Parse via + `ParseReader` with default caps. + - `Stats.Truncated` → warning `zsh_file_truncated`. + - `Stats.Oversized > 0 || Stats.Malformed > 0` → warning + `zsh_lines_skipped` with both counts in the message (surfaces the + parser's skip-and-count so degradation is never silent). + - If `ExtendedCount == 0 && SimpleCount > 0` → warning + `zsh_no_timestamps` (message: entry count + one-line fix hint), return + zero events. + - Range filter: keep entries with `HasTimestamp && rng.Contains(Start)`. + - `exclude_patterns` (pre-compiled at construction; compile errors were + already rejected by config validation): a match on the raw command + drops the entry silently (privacy feature — no per-entry warning, no + count leak; document in code comment). + - Event mapping per DESIGN §7.1: `Kind=KindCommand`, `Start`, `End = + Start + Duration` when Duration > 0, `Project=""`, `Title =` first 200 + runes of the command's first line (raw; sanitize/redact happen in the + 18 pipeline stage), `Meta = {"duration_s": }` when Duration > 0. + - Respect `ctx`: check `ctx.Err()` at least every 1000 entries; on + cancellation return what was collected so far with nil error (collector + already adds the timeout warning). +4. No sorting here (collector sorts globally). + +# Acceptance Criteria + +- [ ] Extended fixture + range covering only day D → exactly the day-D + entries become Events with correct Start/End and duration Meta. +- [ ] Simple fixture → 0 events + `zsh_no_timestamps` warning (and no + others). +- [ ] Missing path → `zsh_history_missing`, nil error. +- [ ] `exclude_patterns=["^secretcmd"]` drops matching entries; count + difference asserted; no warning emitted for drops. +- [ ] Fixture with one oversized and one malformed line → single + `zsh_lines_skipped` warning whose message contains both counts; + clean fixture → no such warning. +- [ ] Range boundaries: entry at exactly 00:00 included, at 24:00 excluded + (half-open contract). +- [ ] Title capped at 200 runes; multiline command flattened to first line + in Title (assert with the multiline fixture). +- [ ] Read-only proof: fixture file mtime + content hash identical + before/after Collect (test helper asserts). +- [ ] `-race` clean; provider satisfies `source.Provider` (compile-time + `var _ source.Provider = (*Provider)(nil)`). + +# Validation + +`go test -race -cover ./internal/source/zsh/` in PR. + +# Dependencies + +08, 09. + +# Non-goals + +doctor formatting of the fix hint (28), aggregation grouping (18), +redaction (16–18). + +# Design References + +- `docs/DESIGN.md` §7.1, §13 +- `docs/research/zsh-history-format.md` diff --git a/docs/issues/11-git-repo-discovery.md b/docs/issues/11-git-repo-discovery.md new file mode 100644 index 0000000..c20e216 --- /dev/null +++ b/docs/issues/11-git-repo-discovery.md @@ -0,0 +1,100 @@ +# Title + +git repo discovery: root walk, worktree dedup + +# Summary + +Implement repository discovery: depth-limited walk of configured roots, +`.git` dir/file detection, exclusion rules, symlink safety, and worktree +deduplication via `git rev-parse --git-common-dir` (DESIGN.md §7.2 +"Discovery"). + +# Context + +The user's environment keeps repos under `~/dev` and uses git worktrees +heavily (1 task = 1 worktree policy). Naive discovery would list a main repo +and each of its worktrees as separate repos and duplicate every commit in +reports. The common-dir dedup rule here is what prevents that. + +# Scope + +- `internal/source/gitsrc/discover.go` + tests (fixture repos built in + `t.TempDir()` by test helpers). + +# Detailed Requirements + +1. `type Repo struct { WorkDir string; CommonDir string; Name string }` — + `Name` = base name of the canonical work dir. +2. `Discover(ctx context.Context, cfg config.SourcesGit, runner Runner) + ([]Repo, []model.Warning)`: + - Walk each root from `cfg.GitRoots()` (tilde-expanded by config + accessors): breadth-first, max depth `cfg.MaxDepth` (root itself = + depth 0). + - Skip (do not descend into): directories whose base name is in + `exclude_dirs`; all dot-directories (`.foo`) except that a `.git` + entry marks its parent as a repo; symlinked directories (`Lstat` + check — never follow, cycle prevention). + - Repo detection: directory containing `.git` (dir or file). On + detection, record candidate and do NOT descend further into it. + - Missing/non-directory root → warning `git_root_missing` (already part + of the canonical DESIGN §13 taxonomy; the constant exists in + `internal/model` from issue 03). + - Explicit `cfg.Repos` paths join the candidate list (missing path → + `git_root_missing` warning naming it). +3. Dedup: for each candidate run + `git --no-optional-locks -C rev-parse --path-format=absolute + --git-common-dir` via the injected `Runner` (interface + `Run(ctx, dir string, args ...string) (stdout string, err error)` — real + impl in issue 12, test impl fake). Canonical key = cleaned common-dir + path. First candidate wins per key; candidates whose rev-parse fails → + warning `git_repo_failed` (skip). `WorkDir` of the kept entry: if the + common dir is `/.git`, use `X`; else (detached/odd layouts) keep the + candidate dir. +4. Deterministic output: sort Repos by Name then WorkDir. +5. `git` binary absence is issue 12's `Runner` concern: `Discover` surfaces + it as a single `git_binary_missing` warning and returns zero repos + (detected on first Run error of type ErrGitNotFound). +6. Walk must be allocation-sane on big trees: use `os.ReadDir`, check + `ctx.Err()` per directory. + +# Acceptance Criteria + +- [ ] Fixture tree: root with 2 normal repos at depth 1–2, one repo below + max_depth (excluded), one under `node_modules` (excluded), one + symlinked dir (not followed), one plain dir (not a repo) → exactly the + 2 expected repos. +- [ ] Worktree fixture: main repo + `git worktree add` sibling → ONE Repo, + `Name` = main repo dir name (helper builds real worktrees with the + system git). +- [ ] Explicit `repos` entry pointing at a worktree of an already-discovered + main repo → still one Repo (dedup across discovery+explicit). +- [ ] Missing root and missing explicit repo → `git_root_missing` warnings, + others unaffected. +- [ ] Fake runner returning error for one candidate → that candidate + skipped with `git_repo_failed`, others kept. +- [ ] Determinism: shuffled directory creation order still yields identical + sorted output. + +# Validation + +`go test -race -cover ./internal/source/gitsrc/` in PR (helper-built +fixtures; requires system git on the test machine — CI has it). + +# Dependencies + +01, 03. + +# Non-goals + +Commit extraction (12). Submodules: because the walk never descends past a +detected repo boundary, submodules inside a discovered repo are NOT +enumerated (their commits belong to their own repos; users who want them add +the submodule path to `repos` explicitly). A submodule checkout sitting +directly under a root (unusual) is treated as an ordinary repo candidate. +Bare repos: not supported in v1 (no worktree = no daily-work relevance); +rev-parse-based dedup keeps them from duplicating anything. + +# Design References + +- `docs/DESIGN.md` §7.2 (Discovery), §13 +- `docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md` diff --git a/docs/issues/12-git-commit-extraction.md b/docs/issues/12-git-commit-extraction.md new file mode 100644 index 0000000..b7e788f --- /dev/null +++ b/docs/issues/12-git-commit-extraction.md @@ -0,0 +1,127 @@ +# Title + +git commit extraction: read-only log invocation and parsing + +# Summary + +Implement the git `Runner` (safe subprocess execution) and the commit +extraction pipeline: one read-only `git log` per repo with control-char +field separators, `--shortstat` association, author-date precision +filtering, author-email matching, and the gitsrc `source.Provider` +(DESIGN.md §7.2 "Extraction"). + +# Context + +This is the only place in the codebase allowed to spawn subprocesses +(CI-enforced boundary, issue 02). Read-only discipline (invariant I2) and +injection-proof invocation are the security core of this issue. + +# Scope + +- `internal/source/gitsrc/runner.go` (real `Runner`), `log.go` (invocation + + parser), `provider.go` (`source.Provider`), tests. + +# Detailed Requirements + +1. `Runner` (real implementation of the 11-defined interface + `Run(ctx context.Context, dir string, args ...string) (stdout string, + err error)`): + - `dir != ""` → git runs with `--no-optional-locks -C ` prepended; + `dir == ""` → git runs with `--no-optional-locks` only (no `-C`), used + for repo-independent invocations like the global + `git config --get user.email`. + - `exec.CommandContext` with argument slices only — never a shell string. + - Env for every invocation: inherited PATH plus `GIT_OPTIONAL_LOCKS=0`, + `GIT_TERMINAL_PROMPT=0`, `LC_ALL=C`, and `GIT_CONFIG_PARAMETERS` + removed from the child env. + - `git` resolved via `exec.LookPath` once; not found → typed + `ErrGitNotFound`. + - Per-invocation timeout: caller-supplied ctx (provider uses 10s/repo). + - Stdout capped at 64 MiB, stderr capped at 64 KiB (both via + `io.LimitReader`; hitting the stdout cap = error for that repo, not a + crash). Warning messages use at most the first line of captured + stderr, truncated to 200 runes (hostile git wrappers with huge stderr + must not flood reports — I5). +2. Log invocation exactly per DESIGN §7.2: + + ``` + git --no-optional-locks -C log --all --no-merges + --since= --until= + --date=iso-strict + --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%s%x1e + --shortstat + ``` + +3. Parser: + - Records split on `0x1e`; within a record: 5 `0x1f` fields; the bytes + after the subject up to the record end may contain a blank line + one + shortstat line (` N files changed, M insertions(+), K deletions(-)`, + any subset of the three clauses) — parse with a tolerant regex; + absent shortstat ⇒ zeros (empty commits). + - `%aI` parsed with `time.Parse(time.RFC3339)`; unparseable date ⇒ skip + commit, count malformed. + - Precise filter in Go: keep commits whose **author date** ∈ `rng` + (the ±48h widening exists only because `--since/--until` filter + committer date). + - Author filter: case-insensitive exact match of `%ae` against the + configured author set. +4. Author set resolution (provider construction): + - `cfg.Authors` non-empty → use as-is (lowercased). + - Empty → `Run(ctx, "", "config", "--get", "user.email")` once (the + `dir==""` global form); + empty/error → provider emits `git_no_author_identity` warning at + Collect and returns zero events (source disabled for the run, + DESIGN §7.2). +5. Provider `Collect`: `Discover` (11) → for each repo (sequentially; + collector-level timeout still applies; per-repo ctx timeout 10s) run + log+parse → map to Events per DESIGN §7.2 (Kind `commit`, Start=author + date, Project=repo Name, Ref=hash[:12], Title=subject raw, Meta: + `files`, `insertions`, `deletions`, `repo_path`). Repo failure → + `git_repo_failed` warning (path + first stderr line ≤ 200 chars), + continue with remaining repos. +6. Read-only guard rails: the only git subcommands in the whole package are + `log`, `rev-parse`, `config --get` — assert with a package test that + greps the package source for `"status"`, `"fetch"`, `"pull"` etc. + (deny-list test), and document the rule in the package doc.go. + +# Acceptance Criteria + +- [ ] Fixture repo (helper-built, fixed `GIT_AUTHOR_DATE`/ + `GIT_COMMITTER_DATE`, two authors): only configured-author commits in + range become Events; hash/subject/stats fields exact. +- [ ] Author-date vs committer-date divergence case (helper commits with + committer date outside the range, author date inside, and vice versa) + filters on author date. +- [ ] Commit subjects containing `|`, backticks, a fake ANSI sequence, and + Japanese text arrive byte-faithful in `Title` (downstream sanitizes). +- [ ] Empty-commit (no shortstat) parses with zero stats. +- [ ] Merge commits absent (`--no-merges` honored) — fixture includes one. +- [ ] Repo with corrupted `.git` (helper truncates HEAD) → `git_repo_failed` + warning, other repos still reported. +- [ ] Warning message from a repo whose git invocation emits multi-KB + stderr is capped to one line ≤ 200 runes (fake Runner or PATH-shimmed + git script fixture). +- [ ] No author identity anywhere → `git_no_author_identity`, zero events. +- [ ] Read-only proof: SHA-256 over every file under the fixture repos + (incl. `.git/index`) identical before/after Collect. +- [ ] Deny-list source test passes; boundary script still green (only + gitsrc imports os/exec). + +# Validation + +`go test -race -cover ./internal/source/gitsrc/` in PR; run the read-only +hash test 5× (`-count=5`) to catch index-refresh flakes. + +# Dependencies + +03, 04, 11. + +# Non-goals + +Uncommitted changes (v1 non-goal, ADR-003), branch attribution per commit, +remote operations of any kind, go-git migration (ADR-003 alternative). + +# Design References + +- `docs/DESIGN.md` §7.2, §12.1 (B2), §13 +- `docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md` diff --git a/docs/issues/13-jsonl-scanner-util.md b/docs/issues/13-jsonl-scanner-util.md new file mode 100644 index 0000000..ee555db --- /dev/null +++ b/docs/issues/13-jsonl-scanner-util.md @@ -0,0 +1,99 @@ +# Title + +internal/source/jsonlutil: tolerant capped JSONL scanner + +# Summary + +Implement the shared streaming JSONL reader used by both agent-log providers +(14, 15): line-by-line scanning with a hard line cap, decode-or-skip +semantics, and malformed/oversized counters — so both parsers get identical +robustness behavior from one tested implementation. + +# Context + +Agent session files are unofficial formats written by live processes: lines +can be huge (embedded base64), torn (concurrent append), or of unknown shape +(schema drift). DESIGN.md I5 requires skip-and-count, never crash. Claude +Code and Codex files share the "JSON object per line" envelope, differing +only in fields — hence one utility. + +# Scope + +- `internal/source/jsonlutil/jsonlutil.go` + tests + fixtures. + +# Detailed Requirements + +1. API: + + ```go + type Stats struct { Lines, Decoded, Malformed, Oversized int } + + // ForEach streams r line by line. For each physical line ≤ maxLineBytes + // it attempts json.Unmarshal into a fresh map[string]json.RawMessage and + // calls fn. fn returning false stops the scan early (no error). + // Oversized lines are consumed fully but never buffered whole. + func ForEach(ctx context.Context, r io.Reader, maxLineBytes int, + fn func(line map[string]json.RawMessage) bool) (Stats, error) + ``` + + - Default cap constant `DefaultMaxLineBytes = 4 << 20` (4 MiB, DESIGN + §7.3/§7.4). + - Implementation: `bufio.Reader.ReadSlice('\n')` loop; when a line + exceeds the cap, continue consuming until newline/EOF without + allocating (count `Oversized`, discard). + - Empty/whitespace-only lines: counted in `Lines`, not `Malformed`. + - Invalid JSON / non-object JSON → `Malformed`. + - `ctx.Err()` checked every 256 lines → returns `ctx.Err()`. + - Torn final line (no trailing newline): still processed if valid JSON; + torn mid-JSON counts `Malformed`. Never an error. + - Returned `error` is non-nil only for read errors from `r` or context + cancellation — never for content. +2. Field helpers (shared by 14/15, all nil-safe on missing keys): + - `Str(m, key) (string, bool)`, `Bool(m, key) (bool, bool)`, + `Time(m, key) (time.Time, bool)` — RFC3339 with/without fractional + seconds and offset (`time.RFC3339Nano` then `RFC3339` fallback), + - `Obj(m, key) (map[string]json.RawMessage, bool)` — one nesting step + (for `payload`/`message`). +3. `MalformedRatio(s Stats) float64` — used by providers for the >20% + warning rule. +4. No goroutines; allocation-conscious (reuse decode buffer where safe). + +# Acceptance Criteria + +- [ ] Happy path: 1000-line fixture decodes with `Decoded==1000`. +- [ ] Oversized: fixture with one 5 MiB line between valid lines → that line + skipped (`Oversized==1`), neighbors decoded; peak memory stays bounded + (no 5 MiB allocation — asserted via `testing.AllocsPerRun` style bound + or a 64 KiB read-chunk implementation detail test). +- [ ] Malformed: truncated JSON, `[]` array line, bare string line → each + counts `Malformed`, scan continues. +- [ ] Torn tail: valid-JSON-no-newline processed; half-written JSON tail + counted malformed; both without error. +- [ ] Early stop: `fn` returning false stops with partial Stats, nil error. +- [ ] Context cancellation mid-file returns `ctx.Err()`. +- [ ] `Time` helper parses both `2026-07-10T12:34:56.789Z` and + `2026-07-10T21:34:56+09:00`. +- [ ] `FuzzForEach` (seeded with fixtures) 30s: no panics, Stats counters + always sum consistently (`Decoded+Malformed+Oversized ≤ Lines`; + the difference is the count of empty/whitespace-only lines, which + have no dedicated counter). +- [ ] ≥ 95% coverage; stdlib-only. + +# Validation + +`go test -race -cover ./internal/source/jsonlutil/` + fuzz output in PR. + +# Dependencies + +01. + +# Non-goals + +Schema knowledge of either agent format (14/15), gzip/rotation handling +(not observed in the wild for these files), sanitize/redact. + +# Design References + +- `docs/DESIGN.md` §7.3, §7.4 (caps + tolerance), §1.2 (I5) +- `docs/research/claude-code-session-format.md`, + `docs/research/codex-session-format.md` (robustness tables) diff --git a/docs/issues/14-claude-code-provider.md b/docs/issues/14-claude-code-provider.md new file mode 100644 index 0000000..95e4236 --- /dev/null +++ b/docs/issues/14-claude-code-provider.md @@ -0,0 +1,113 @@ +# Title + +Claude Code session provider + +# Summary + +Implement the Claude Code source: discover session JSONL files under the +projects directory with an mtime prefilter, assemble per-session facts +(span, project, counts, title) with the tolerant scanner (13), and emit +`agent-session` Events per DESIGN.md §7.3 and the research doc. + +# Context + +`~/.claude/projects//.jsonl` is an unofficial, +version-drifting format. The research doc +(`docs/research/claude-code-session-format.md`) is the normative field +contract: `cwd` (not the lossy directory name) attributes the project; +`summary` lines beat first-prompt titles; sidechain lines are counted +separately; prompts are opt-in (`include_prompts`, ADR-004). + +# Scope + +- `internal/source/claudecode/provider.go`, `session.go`, tests, synthetic + fixtures under `testdata/`. + +# Detailed Requirements + +1. `New(cfg config.SourcesClaudeCode) *Provider`; `ID()` = + `model.SourceClaudeCode`. +2. Discovery: `/*/*.jsonl` via `os.ReadDir` two levels (no + deeper recursion); skip files with mtime < `rng.Start − 48h` + (files still being written have fresh mtimes; a session ending after + `rng.End` still matters for spanning, so no upper-bound skip). Missing + projects dir → `claudecode_dir_missing` warning, zero events, nil error. + Symlinked project dirs are not followed. +3. Per file, one streaming `jsonlutil.ForEach` pass building: + - `first`, `last` — min/max `timestamp` over lines with a parseable + timestamp **that fall inside `rng`** (per research doc: do not trust + line order); also track any-line-in-range boolean. + - counts (computed over **in-range lines only** — a midnight-spanning + session reports per-day counts, per the research doc): `user` lines + with `isSidechain != true` → `UserMsgs`; `assistant` non-sidechain → + `AgentMsgs`; any user/assistant with `isSidechain == true` → + `SidechainMsgs`. + - modal `cwd` and modal `gitBranch` over in-range user/assistant lines. + - title candidates: latest `summary`-type line's `summary` string + (regardless of range — a summary describes the whole session); first + in-range user line's text when `include_prompts` (content as a plain + string, OR the concatenation of all `text`-type blocks when content is + a block array — per the research doc; then take the first line and cap + at 120 runes). + - `version` (last seen), sessionId (any line; fallback filename stem). + - Early stop: none (need min/max over whole file) — but skip files fast + when the first 50 parseable timestamps are all > `rng.End` AND mtime + indicates no rewrite risk? NO — keep it simple and correct: full scan + per file that passes the mtime prefilter (documented decision). +4. Emit one Event per session with ≥ 1 in-range user/assistant line: + Kind `KindAgentSession`, Start=first, End=last, Project = modal cwd + (raw path; normalization happens in 18), Ref=sessionId, + Title = summary → prompt (opt-in) → `claude-code session `, + Meta: `agent="claude-code"`, `user_messages`, `assistant_messages`, + `sidechain_messages`, `git_branch`, `cli_version` (omit empty values). +5. Per-file malformed handling: `MalformedRatio > 0.20 && Lines ≥ 10` → + `claudecode_malformed_lines` warning naming the file (basename only — + full paths of other projects would leak into shared reports via the + warnings footer; DESIGN §13 messages must use basenames). Unreadable + file (permissions) → `claudecode_file_skipped` warning, continue. +6. `ctx` honored between files and via ForEach. + +# Acceptance Criteria + +- [ ] Fixture session with summary line + user/assistant/sidechain mix → + one Event with exact counts, summary-based Title, modal cwd Project. +- [ ] `include_prompts=false` (default): no prompt text anywhere in Event + (Title falls back when no summary); `=true`: first-prompt Title, + first line only, ≤ 120 runes. +- [ ] `message.content` as array-of-blocks fixture parses (text blocks + concatenated). +- [ ] Midnight-spanning fixture: session included for both adjacent days; + Start/End clamp to in-range min/max per day AND the per-day message + counts differ according to which lines fall in each day (two Collect + calls asserted, counts hand-computed in the test). +- [ ] Out-of-range session file (old mtime) never opened (assert via a + fixture with unreadable permissions that would warn if opened — no + warning appears when mtime is old). +- [ ] Malformed fixture (30% bad lines) → warning with basename, session + still emitted from good lines. +- [ ] Fixture with ANSI escapes + fake `ghp_…` token in title text passes + through raw here (sanitize/redact are 18's job) — asserted so nobody + "helpfully" sanitizes in the parser and breaks layering. +- [ ] Unknown line types (fixture includes `queue-operation`, + `last-prompt`, an invented `type`) silently ignored. +- [ ] Read-only proof: fixture tree hash identical before/after. + +# Validation + +`go test -race -cover ./internal/source/claudecode/` in PR. + +# Dependencies + +04, 08, 13. + +# Non-goals + +`~/.claude/history.jsonl` (v2 candidate), automation-noise filtering beyond +sidechain separation (known unknown U2-adjacent), project-name +normalization (18). + +# Design References + +- `docs/research/claude-code-session-format.md` (normative) +- `docs/DESIGN.md` §7.3, §13 +- `docs/decisions/ADR-004-redaction-default-on.md` (prompts opt-in) diff --git a/docs/issues/15-codex-provider.md b/docs/issues/15-codex-provider.md new file mode 100644 index 0000000..b68fe84 --- /dev/null +++ b/docs/issues/15-codex-provider.md @@ -0,0 +1,114 @@ +# Title + +Codex session provider + +# Summary + +Implement the Codex CLI source: date-directory-pruned discovery of rollout +files (with recursive fallback), `session_index.jsonl` title mapping, +event/task counting, originator excludes, optional archived-sessions scan — +per DESIGN.md §7.4 and the research doc. + +# Context + +`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` (envelope +`{timestamp, type, payload}`) is unofficial and version-drifting; +`docs/research/codex-session-format.md` is the normative contract. Date +sharding gives a cheap fast path; automation noise (orchestrators, cron) is +a real problem on the reference machine, handled via +`exclude_originators` + per-originator visibility in `worklog sources` (29). + +# Scope + +- `internal/source/codex/provider.go`, `session.go`, `index.go`, tests, + synthetic fixtures. + +# Detailed Requirements + +1. `New(cfg config.SourcesCodex) *Provider`; `ID()` = `model.SourceCodex`. +2. Discovery fast path: for each calendar day D touching + `[rng.Start − 24h, rng.End + 24h]`, list + `/YYYY/MM/DD/rollout-*.jsonl`. Fallback: whenever the fast + path finds **zero files** for the whole padded range (and + `` exists), run a recursive walk capped at depth 4 with an + mtime prefilter (≥ rng.Start − 48h) — this covers any future layout + drift, including trees that still look date-sharded but moved the files. Missing sessions dir → `codex_dir_missing`, zero + events, nil error. `include_archived` adds the same scan over + `archived_dir`. +3. Title index: single tolerant pass over `cfg.CodexSessionIndex()` building + `map[id]thread_name` (jsonlutil; missing/unreadable file → warning + `codex_index_unreadable`, continue without titles). Cap: 1_000_000 index + lines (beyond → stop + same warning; degenerate file defense). +4. Per rollout file (jsonlutil pass): + - envelope: `timestamp`, `type`, `payload` (via `Obj`). + - `session_meta.payload`: `id`, `cwd`, `cli_version`, `originator`. + Missing session_meta (torn file) → per research doc: id/start from + filename `rollout--.jsonl` (regex + `^rollout-(\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2})-([0-9a-f-]{36})\.jsonl$`), + cwd from modal `turn_context.payload.cwd`, else `Project=""`. + - counts from `event_msg.payload.type`, computed over **in-range lines + only** (midnight-spanning sessions report per-day counts, mirroring + the Claude Code provider contract): `user_message` → UserMsgs, + `agent_message` → AgentMsgs, `task_started` → Tasks (count starts; + ignore `task_complete` for the count, simpler and stable); + `token_count` ignored. + - span: min/max envelope timestamps of in-range lines; include session + iff ≥ 1 in-range line. + - title precedence: index[id] → first in-range `user_message` text + (`include_prompts` only, first line ≤ 120 runes) → + `codex session `. +5. `exclude_originators`: exact string match against + `session_meta.originator` → skip the whole file silently (privacy-style + silent drop, mirroring zsh excludes; per-originator counts surface in + issue 29 instead). +6. Dedup live-vs-archived by session id: live wins + (`model.DedupSessions` handles final dedup, but avoid double work: track + seen ids during this Collect). +7. Event mapping: Kind `KindAgentSession`, Project = cwd raw, + Ref = session id, Meta: `agent="codex"`, `user_messages`, + `agent_messages`, `tasks`, `originator`, `cli_version` (omit empties). +8. Warnings mirror 14: `codex_malformed_lines` (>20% & ≥10 lines, basename + only), unreadable file → `codex_file_skipped` (both are part of the + canonical DESIGN §13 taxonomy; constants exist from issue 03). + +# Acceptance Criteria + +- [ ] Fixture day-sharded tree: sessions on D−1/D/D+1; range=D returns + exactly D's sessions plus a midnight-spanning D−1 session (fixture + line at 00:10 of D). +- [ ] session_meta-less torn fixture: id/start recovered from filename; + session emitted. +- [ ] Index present → thread_name Title; index missing → warning + fallback + titles; `include_prompts=true` path asserted. +- [ ] `exclude_originators=["auto-orchestrator"]` fixture skipped entirely; + non-matching originators kept. +- [ ] Archived dedup: same id in live and archived → one Event, live data. +- [ ] Fallback walk: fixture with non-sharded layout still found via mtime + walk; fixture with a date-sharded-looking tree whose files moved one + level deeper also found (zero-fast-path-hits trigger asserted). +- [ ] Midnight-spanning fixture: per-day counts differ per the in-range + rule (two Collect calls, counts hand-computed). +- [ ] Robustness fixtures (oversized line, 30% malformed, unknown types, + ANSI + fake token in message text) behave per 13/14 patterns (raw + passthrough asserted). +- [ ] Read-only proof: fixture tree hash identical before/after. + +# Validation + +`go test -race -cover ./internal/source/codex/` in PR. + +# Dependencies + +04, 08, 13. + +# Non-goals + +SQLite files in `~/.codex` (ADR-003: never opened), `~/.codex/history.jsonl` +(v2), token accounting (`token_count` shape unconfirmed, research doc), +project normalization (18). + +# Design References + +- `docs/research/codex-session-format.md` (normative) +- `docs/DESIGN.md` §7.4, §13 +- `docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md` diff --git a/docs/issues/16-redaction-engine.md b/docs/issues/16-redaction-engine.md new file mode 100644 index 0000000..f471a23 --- /dev/null +++ b/docs/issues/16-redaction-engine.md @@ -0,0 +1,105 @@ +# Title + +internal/redact: rule engine, allowlist, config integration + +# Summary + +Implement the redaction engine mechanics: ordered rule application with +value-group masking, allowlist exemption, user extra patterns, bounded +block masking, and the `Redactor` construction from config — per DESIGN.md +§8. The built-in ruleset content ships separately (17). + +# Context + +Invariant I4 makes this engine the chokepoint between raw source content and +everything user-visible (report, LLM). It must be RE2-safe (no backtracking +DoS from adversarial history lines), deterministic, and cheap enough to run +twice (aggregation stage + LLM prompt defense-in-depth). + +# Scope + +- `internal/redact/redact.go`, `rule.go`, tests. Rules *content* minimal + here (two placeholder rules for tests); real corpus in 17. + +# Detailed Requirements + +1. Types: + + ```go + type Rule struct { + ID string // stable, kebab-case (public API for allowlist docs) + Pattern *regexp.Regexp // RE2 + Group int // 0 = mask whole match; N = mask only group N + Block bool // multiline block rule (see 4) + } + type Redactor struct { /* rules, allowlist, counters */ } + ``` + +2. `New(rules []Rule, allowlist []*regexp.Regexp, extra []*regexp.Regexp) + (*Redactor, error)` — validates: unique ids, Group within pattern's group + count. `FromConfig(cfg config.Redaction, builtin []Rule)` compiles + `extra_patterns` (ids `extra-1…`, Group 0) and `allowlist`; when + `cfg.Mode == "off"` returns a no-op Redactor whose `Enabled() == false`. +3. `Redact(s string) (string, int)`: + - Apply rules in slice order. For each match: if any allowlist regex + matches the **entire matched text** (group-0 span), skip masking that + match. + - Replacement: group-targeted splice `[REDACTED:]` replacing only the + group span (regexp `FindAllStringSubmatchIndex` + manual rebuild — + `ReplaceAllString` cannot target groups). + - Count total maskings; deterministic single pass per rule (no re-scan of + already-masked output for the same rule; later rules do scan the + partially-masked string — order matters and is fixed by the rule + slice). +4. Block rules (`Block: true`, e.g. private-key blocks): pattern matches the + opening line; masking extends from the opening match through the matching + `-----END …-----` line or at most 100 lines, whichever first + (DESIGN §8), replaced by one `[REDACTED:]`. +5. `RedactEvent(e model.Event) (model.Event, int)` helper — applies + `Redact` to Title, Body, Project, and every Meta value; returns the + modified copy and the total masking count across all fields (used by 18; + LLM (24) uses plain `Redact` on the final prompt string). +6. Performance guard: engine must process a 1 MiB adversarial string + (`aaaa…` + near-miss token prefixes) in < 100ms with the 17 ruleset — + benchmark included (`BenchmarkRedactWorstCase`), asserted loosely in a + test (< 1s) to catch accidental catastrophic patterns. +7. No logging of matched content anywhere (the engine must never leak what + it masked — code-review AC). + +# Acceptance Criteria + +- [ ] Group masking: rule with Group=2 masks only the value span, preserving + prefix/suffix (table test with env-var style placeholder rule). +- [ ] Whole-match masking (Group=0) verified. +- [ ] Allowlist: exact-match exemption works; non-matching allowlist leaves + masking intact; allowlist never *adds* content. +- [ ] Block rule: 3-line PEM-style fixture collapses to one token; runaway + block without END masks exactly 100 lines then stops. +- [ ] `extra_patterns` masked with `extra-N` ids; `mode="off"` returns input + verbatim with count 0 and `Enabled()==false`. +- [ ] Idempotence: `Redact(Redact(x)) == Redact(x)` over the test corpus + (replacement tokens must not re-match any rule — asserted). +- [ ] Multiple matches of multiple rules in one string all masked; overlap + resolution: earlier-rule match wins, later rules see the masked text + (asserted with crafted overlap). +- [ ] Benchmark exists; worst-case test < 1s. +- [ ] ≥ 95% coverage; stdlib-only. + +# Validation + +`go test -race -cover ./internal/redact/` + `go test -bench RedactWorstCase +-benchtime 1x` output in PR. + +# Dependencies + +01, 05 (config types), 03 (Event for the helper). + +# Non-goals + +The real ruleset + corpus (17), entropy detection (v2, ADR-004), masking in +parsers (layering: parsers stay raw). + +# Design References + +- `docs/DESIGN.md` §8, §1.2 (I4) +- `docs/decisions/ADR-004-redaction-default-on.md` diff --git a/docs/issues/17-redaction-ruleset.md b/docs/issues/17-redaction-ruleset.md new file mode 100644 index 0000000..96b9a5d --- /dev/null +++ b/docs/issues/17-redaction-ruleset.md @@ -0,0 +1,99 @@ +# Title + +Built-in redaction ruleset and test corpus + +# Summary + +Implement the eleven built-in redaction rules from DESIGN.md §8 with their +exact ids and patterns, plus the true-positive / false-positive regression +corpus that pins their behavior. + +# Context + +The ruleset table in DESIGN §8 is the contract: rule ids are public API +(users reference them in `allowlist` docs), and the "commit hashes and +ordinary text are never masked" guarantee is what keeps reports readable. +This issue is deliberately separate from the engine (16) because the corpus +is large and the patterns need focused review. + +# Scope + +- `internal/redact/builtin.go` (`func Builtin() []Rule`), + `builtin_test.go`, `testdata/corpus.json` — a JSON array of cases: + `{"name": "", "rule": "", "input": "", + "want": ""}`. JSON strings carry multiline inputs + (PEM blocks) losslessly and `want` pins the exact masking result. + +# Detailed Requirements + +1. Implement exactly the rules of DESIGN §8's table, in that order, with + those ids: `private-key-block` (Block rule), `aws-access-key-id`, + `github-token`, `slack-token`, `google-api-key`, `openai-anthropic-key`, + `jwt`, `authorization-header` (mask value group), + `env-assignment` (mask value group, keep name), `cli-secret-flag` + (mask value group), `url-userinfo` (mask password group). +2. Pattern refinements (allowed to tighten, never loosen, relative to the + table; document any deviation in code comments AND update DESIGN §8 in + the same PR if a pattern must change): + - `env-assignment` must handle `export NAME=…`, quoted values, and stop + at whitespace for unquoted values. + - `cli-secret-flag` must handle `--token=x`, `--token x`, `-p=x` forms + listed in the pattern. + - `url-userinfo` masks only the password span (`user:****@host`). +3. Corpus — minimum content: + - Positives (each rule ≥ 5 lines): realistic fakes (e.g. + `AKIA` + 16 uppercase, `ghp_` + 36 alnum, quoted/unquoted env + assignments, `curl -H "Authorization: Bearer …"`, a 3-line PEM block, + JWT with three segments, `https://user:pass@host/path`, + `--api-key=sk-…` which must double-mask under two rules without + corruption). + - Negatives (must NOT mask, ≥ 20 lines): 12/40-char git hashes; commit + subject `fix: rotate token handling`; `TOKEN_COUNT=5` (name not in the + secret-name list — verify the pattern's name list is anchored); + `--tokenizer bert`; URL without userinfo; `skiing sk-lift schedule` + (no, `sk-lift` — 7 chars — must not match `sk-[A-Za-z0-9_-]{20,}`); + Japanese text; `password:` prose without assignment; base64 blob of 30 + chars in plain prose (entropy is v2 — must pass through). +4. Corpus runner test: for every case, assert `Redact(input) == want` + byte-for-byte. Derived assertions: cases with `rule != "NONE"` must have + `want` containing `[REDACTED:]` and must differ from `input` + (self-checking corpus — a `want` equal to `input` on a positive case + fails the suite); cases with `rule == "NONE"` must have + `want == input` (asserted at load time so corpus typos are caught). +5. A `doc.go` table listing rule ids + one-line descriptions (renders in + pkg docs; README links here in 31). + +# Acceptance Criteria + +- [ ] All 11 rules present, ids exactly as specified, order preserved. +- [ ] Corpus passes (`Redact(input) == want` for all cases); negatives + byte-identical; the multiline PEM positive is present as a JSON case + and collapses to one `[REDACTED:private-key-block]` token. +- [ ] Double-match line (`--api-key=sk-…`) yields a stable, parseable result + (no nested/garbled tokens) — exact expected string in the test. +- [ ] Engine idempotence test (16) re-run against the full builtin set still + green (`[REDACTED:…]` tokens match no builtin rule). +- [ ] Worst-case benchmark from 16 with full ruleset still < 1s. +- [ ] Every pattern compiles as RE2 and contains no lookahead/backreference + (compile-time guarantee via `regexp.MustCompile` — build fails + otherwise). + +# Validation + +`go test -race -cover ./internal/redact/` in PR, plus paste 5 sample +positive→masked lines (fake secrets only) demonstrating output shape. + +# Dependencies + +16. + +# Non-goals + +Entropy heuristics, cloud-provider-exhaustive token catalogs (the 11 rules +are v1 scope; additions are post-v1 issues driven by real misses), +locale-specific secrets. + +# Design References + +- `docs/DESIGN.md` §8 (normative rule table) +- `docs/decisions/ADR-004-redaction-default-on.md` diff --git a/docs/issues/18-aggregation.md b/docs/issues/18-aggregation.md new file mode 100644 index 0000000..73fdb49 --- /dev/null +++ b/docs/issues/18-aggregation.md @@ -0,0 +1,124 @@ +# Title + +internal/aggregate: sanitize/redact stage and Report assembly + +# Summary + +Implement the pipeline stage that turns raw collected Events into the final +`Report` tree: apply sanitize + redact to every event (the single +enforcement point of invariants I4/I5 for report content), normalize project +names, group and sort everything per DESIGN.md §6.2, §9.1, §9.3. + +# Context + +Parsers deliberately emit raw content (asserted by their tests); this stage +is where hygiene is centrally enforced so no code path can reach the +renderer or LLM with unsanitized/unredacted text. It also owns the grouping +semantics that make reports readable (command grouping, project ordering, +totals). + +# Scope + +- `internal/aggregate/aggregate.go`, `project.go`, `commands.go`, tests. + +# Detailed Requirements + +1. Entry point: + + ```go + type Options struct { + Kind model.ReportKind + Date timeutil.Day + Range timeutil.Range + Lang string + Redactor *redact.Redactor + GitRoots []string // expanded, for project normalization + ExplicitRepos []string + Version string // GenerationMeta + SourcesUsed []model.SourceID + LLMModel string // "" until 24 fills narrative + TopCommands int // default 20 + } + func Build(events []model.Event, warnings []model.Warning, o Options) *model.Report + ``` + +2. Stage order (fixed): dedup sessions (`model.DedupSessions`) → per event: + `sanitize.Line(Title, 200)`, `sanitize.Cap(Clean(Body), 4000)`, + `sanitize.Line(Project, 300)`, sanitize each Meta value (`Line(v, 500)`) + → `redact.RedactEvent` (which also covers the Project field — extend + 16's helper contract accordingly: Project is redacted like Title) → + project normalization → grouping. Project names are thereby + sanitized+redacted BEFORE they become headings/keys; Markdown escaping + of project names remains the renderer's job (20/21). +3. Project normalization (DESIGN §6.2): for `agent-session` events whose + raw Project (cwd) equals or is a subpath of `/` for any + configured git root (or matches an explicit repo path), the project key + is `` (first path element under the root). A cwd equal to a root + itself → key = base name of cwd. Otherwise: cleaned absolute path with + home directory abbreviated to `~` (display-safe). Commit events already + carry repo-name projects (12). Command events keep `Project == ""`. +4. Grouping: + - `ProjectActivity` per distinct project key: commits (map Event → + `model.Commit`: Repo=Project, Hash=Ref, Subject=Title, When=Start, + stats from Meta ints), sessions (Event → `model.AgentSession`: + Agent=Meta["agent"], Title, Ref, Start, End, counts from Meta ints). + - `ShellActivity` from command events: consecutive-duplicate collapse + (same Title adjacent after sort by Start) → group by head per DESIGN + §9.3 (first token; two tokens when first ∈ the fixed multi-tool set + {git,npm,pnpm,yarn,go,cargo,docker,kubectl,make,brew,gh,uv,pip}); + per group: Count, First, Last, up to 3 example commands + (already-redacted full titles, deduped, first-seen order). Keep top + `TopCommands` groups by (Count desc, Head asc); `Total` = command + count before top-N truncation. + - Totals per DESIGN §9.3 incl. SpanStart/SpanEnd = min/max event + Start/End (End=zero treated as Start). +5. Sorting: delegate to `model.Sort*` helpers everywhere; output must be + deterministic under input permutation. +6. `GenerationMeta`: Version, SourcesUsed (sorted), LLMModel (as passed), + RedactionMode from `Redactor.Enabled()` (`"on"`/`"off"`). +7. Warnings: pass through (sorted by model rules), plus append + `redaction_disabled` warning when `!Redactor.Enabled()`. +8. Standup assembly is issue 19; `Build` handles the single-range case and + exposes the internals 19 needs (grouping funcs exported within the + package, not the module). + +# Acceptance Criteria + +- [ ] Layering proof: input Event with ANSI escape + fake `ghp_` token in + Title → Report contains neither (sanitized AND `[REDACTED:github-token]` + present). +- [ ] Malicious project name (raw cwd containing an ANSI escape and a fake + token) arrives in the Report sanitized and redacted. +- [ ] Project normalization table: cwd `~/dev/foo/sub/dir` with root + `~/dev` → `foo`; cwd `~/dev` → `dev`; cwd `~/elsewhere/x` → + `~/elsewhere/x`; explicit repo path match; worktree-style cwd + `~/dev/foo/.worktrees/bar` → `foo`. +- [ ] Command grouping: `git commit`/`git push` distinct heads; `ls` single + token; consecutive duplicates collapsed before counting; top-N + truncation with `Total` preserving the full count; ≤ 3 examples. +- [ ] Totals exact on a crafted 3-project fixture (numbers hand-computed in + the test). +- [ ] Determinism: shuffled input events (50 permutations) → byte-identical + JSON-marshaled Report. +- [ ] `redaction_disabled` warning appears exactly when redactor is off. +- [ ] Empty input → Report with zero totals, empty slices (not nil — JSON + `[]`), no panic. +- [ ] ≥ 90% coverage. + +# Validation + +`go test -race -cover ./internal/aggregate/` in PR. + +# Dependencies + +03, 04, 06, 16, 17 (the built-in ruleset must exist for the default-on +redaction path this stage enforces). + +# Non-goals + +Standup buckets (19), rendering, LLM narrative injection (24 sets +`Report.Narrative`), command→project attribution heuristics (v2). + +# Design References + +- `docs/DESIGN.md` §3.1 (stage order), §6.2 (normalization), §9.1, §9.3 diff --git a/docs/issues/19-standup-derivation.md b/docs/issues/19-standup-derivation.md new file mode 100644 index 0000000..042681b --- /dev/null +++ b/docs/issues/19-standup-derivation.md @@ -0,0 +1,87 @@ +# Title + +Standup derivation: workday logic, buckets, blockers + +# Summary + +Implement standup report assembly: collect over two ranges (yesterday = +previous workday, today = midnight→now), build the two-bucket +`StandupData`, and the blockers placeholder — per DESIGN.md §9.2. + +# Context + +`daily` is one range; `standup` is two. The bucket semantics (Monday looks +back to Friday; retro-generation for past dates) live in timeutil (04); this +issue wires them into aggregation so the standup command (26) only renders. + +# Scope + +- `internal/aggregate/standup.go` + tests. + +# Detailed Requirements + +1. Entry point: + + ```go + type StandupInput struct { + YesterdayEvents []model.Event + TodayEvents []model.Event + Warnings []model.Warning + YesterdayRange timeutil.Range // for display clamping + TodayRange timeutil.Range // for display clamping + YesterdayDay timeutil.Day + } + func BuildStandup(in StandupInput, o Options) *model.Report + ``` + + - Produces `Report{Kind: "standup"}` whose `Standup` field carries: + `YesterdayDate` (string), `Yesterday`/`Today` `[]ProjectActivity`, + `YesterdayShell`/`TodayShell`, `YesterdayTotals`/`TodayTotals`. + - Reuses `Build`'s internal stages (sanitize→redact→normalize→group) on + each bucket independently; warnings merged once (dedup identical + (Source, Code, Message) triples). + - Top-level `Projects`/`Shell`/`Totals` on the Report are set from the + **yesterday** bucket (the primary content of a standup; renderer uses + `Standup.*` fields explicitly, but tools reading the generic fields see + the meaningful bucket). +2. Caller contract (command 26 does the collection): yesterday events come + from a Collect over `StandupBuckets().yesterday`, today events over + `.today`, and those same two Ranges are passed into `StandupInput` — + this function does not collect. Document in the function comment. +3. Blockers: v1 renders a placeholder (the renderer's job, 21); this issue + only guarantees `StandupData` exists even when both buckets are empty + (never nil when Kind==standup). +4. Session spanning both buckets (e.g. a session running 23:30–00:30) + appears in both buckets — this falls out of per-bucket collection; the + displayed Start/End of each bucket's copy is clamped to that bucket's + Range from `StandupInput` (`Start = max(Start, rng.Start)`, + `End = min(End, rng.End)`; zero End treated as Start). Assert rather + than "fix" the double appearance (documented behavior). + +# Acceptance Criteria + +- [ ] Both-buckets fixture: distinct events in each bucket land in the + right `Standup` fields with correct totals. +- [ ] Warning dedup: same warning from both collects appears once. +- [ ] Empty buckets → non-nil `StandupData`, zero totals, no panic. +- [ ] Spanning-session fixture appears in both buckets (clamped) — + documented-behavior test. +- [ ] Determinism under permutation (same 50-shuffle test pattern as 18). +- [ ] ≥ 90% coverage. + +# Validation + +`go test -race -cover ./internal/aggregate/` in PR. + +# Dependencies + +04, 18. + +# Non-goals + +Rendering (21), collection/wiring (26), automatic blocker detection (v1 +non-goal; DESIGN §2.2). + +# Design References + +- `docs/DESIGN.md` §9.2 diff --git a/docs/issues/20-render-daily.md b/docs/issues/20-render-daily.md new file mode 100644 index 0000000..fbc433d --- /dev/null +++ b/docs/issues/20-render-daily.md @@ -0,0 +1,114 @@ +# Title + +Daily report renderer (ja/en) with golden tests + +# Summary + +Implement the deterministic Markdown renderer for daily reports: embedded +`text/template` templates for Japanese and English, the section structure of +DESIGN.md §9.1, escaping via internal/sanitize, and byte-exact golden +tests. + +# Context + +The deterministic report IS the product (the LLM only adds a narrative +paragraph). Golden tests here become the regression net for every upstream +change, so template output must be stable, timezone-explicit, and +locale-independent. + +# Scope + +- `internal/render/render.go`, `funcs.go`, + `templates/daily.ja.md.tmpl`, `templates/daily.en.md.tmpl` (embedded via + `embed.FS`), `testdata/golden/*.md`, tests. + +# Detailed Requirements + +1. `func Daily(r *model.Report) (string, error)` — selects template by + `r.Lang` (`ja`/`en` only; anything else = error, config already + validated). +2. Section structure exactly per DESIGN §9.1, in order: + 1. `# 日報 2026-07-10 (木)` / `# Daily report 2026-07-10 (Thu)` — date + + weekday; timezone shown when not system-local? Simpler fixed rule: + always show IANA zone in the header line's suffix + (`— Asia/Tokyo`). + 2. Summary line: totals sentence (commits, sessions, commands, active + span `HH:MM–HH:MM`); zero-activity day renders the explicit + no-activity sentence instead (ja: `記録された活動はありません。` / + en: `No recorded activity.`) and skips sections 3–5. + 3. Narrative: rendered only when `r.Narrative.Text != ""`, under heading + `## サマリー（ローカルLLM: ）` / `## Summary (local model: + )`. Allowed narrative surface: plain paragraphs only — the + renderer applies `sanitize.EscapeText` per line (control chars were + already stripped by 24; this pass additionally HTML-escapes `<`/`&` + and neutralizes leading list/heading markers), so LLM output cannot + inject headings, HTML, or list structure into the report. + 4. Per-project sections `## ` — the project name is rendered + through `sanitize.EscapeText` (heading position is text context; + neutralizes structure forgery; the name was already + sanitized+redacted in aggregation): commits table + (`| time | hash | subject | +/− |`, hash in backticks, subject via + `sanitize.EscapeCell`), sessions table + (`| agent | span | msgs | tasks | title |`, title via `EscapeCell`). + Omit an empty table entirely (no headers over nothing). + 5. `## シェル操作` / `## Shell activity`: `| command | count | first–last |` + top groups + total line; omitted when zero commands. + 6. Warnings (when any): `## 注記` / `## Notes` — bullet per warning + (`- [source] code: message`, message via `EscapeText`). + 7. Footer (always): one line — + `Generated by worklog · sources: zsh,git,… · LLM: · redaction: on|off`. +3. Formatting rules: times `15:04` in report tz; dates `2006-01-02`; + weekday names: ja `(月火水木金土日)`, en `Mon…` — via lookup tables, NOT + locale (`time.Format` weekday is English-only; map it). + Insertions/deletions cell: `+12/−3` (U+2212 not needed — ASCII hyphen). + All numbers via `strconv` (no locale grouping). +4. Template funcs registered from internal/sanitize only (`escCell`, + `escText`) plus tiny format helpers; NO business logic in templates + (tables built from prepared row structs in Go). +5. Output ends with exactly one trailing newline; no trailing spaces on any + line (golden-enforced; makes diffs clean). +6. Golden tests: fixtures for (ja, en) × (full report incl. narrative + + warnings, empty day, no-narrative day, adversarial content report — + subjects with `|`, backticks, `&`/`<` (ANSI already stripped upstream), + long Japanese titles, a malicious project name like + `# fake | heading`, and a narrative containing `## injected` + + `