diff --git a/CHANGELOG.md b/CHANGELOG.md index c9f8b1c..d0f1c35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ## [Unreleased] +### Added +- **User search preferences (`/preferences`, `~/.superhuman/preferences.md`).** `/repo-finder`'s filters were constants baked into the agent prompt (`language:java`, `language:python`), and no command accepted a preference — ask for Go, get Python. They are now sourced from a file the **user** owns. `## Filters` (`languages`, `topics`, `stars`) is mechanical: parsed by `scripts/lib/preferences.sh`, compiled 1:1 into GitHub search qualifiers by `scripts/repo-finder/build_queries.sh`. `## Notes` is prose the agent applies at tie-breaks and issue selection only — never to a numeric score, and disclosed in the shortlist's `notes` field when it does. `topic[:min_stars]` gives one topic its own floor; `any` is reserved and compiles to a query with no `topic:` qualifier; `stars` is a floor and there is no ceiling (a range is a loud error, not a quiet cap). `/repo-finder` gains `--lang` / `--topic` / `--min-stars` overrides that shape one run and never persist. No preferences file means `DEFAULT_PROFILE`, byte-identical to the queries the agent carried inline, so `/contribute` on a fresh machine cannot regress. Covered by `tests/scripts/test_parse_preferences.sh` and `test_build_queries.sh` (network-free, bash 3.2-clean). + +### Changed +- **Query shape follows the qualifier asymmetry.** Repeated `language:` unions; repeated `topic:` **intersects**. So languages collapse into one query and each topic gets its own — query count is `|topics|`, and adding a language costs zero queries. Emitting two `topic:` qualifiers on one line silently returns only repos tagged with *every* topic, often none, with exit 0 and no error; `build_queries.sh` refuses to emit one. Same class of silent failure: a typo'd qualifier key and a parenthesized `OR` both return `total_count: 0` with exit 0, so both are rejected at emit time rather than trusted. + +### Removed +- **`repo-finder`'s Category Preference Order and its `3e` category bonus (10%).** With filtering server-side, every candidate matches the user's filters by construction, so a bonus for matching is a constant added to every row and ranks nothing. Its weight went to signals that discriminate: responsiveness 30 → **35%**, outside-contributor track 20 → **25%**. + ## [0.6.4] — 2026-07-13 ## [0.6.3] — 2026-07-12 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d94ca47..d6b99cf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,9 +14,10 @@ Thanks for your interest. This is a [Claude Code](https://claude.com/claude-code superhuman/ ├── .claude-plugin/ # plugin.json + marketplace.json manifests ├── agents/ # subagent prompts (the behavioral logic) + SHARED_STATE.md -├── commands/ # slash commands (/contribute, /repo-finder, …) +├── commands/ # slash commands (/contribute, /repo-finder, /preferences, …) ├── scripts/ # shell extracted from agent prompts, organised by agent -│ ├── lib/ # shared helpers: state.sh, mistakes.sh, flake.sh, delim.sh +│ ├── lib/ # shared helpers: state.sh, preferences.sh, mistakes.sh, flake.sh, delim.sh +│ ├── repo-finder/ # build_queries.sh — compiles preferences.md into search queries │ ├── profiler/ scorer/ orchestrator/ builder/ ├── schemas/ # JSON Schema (draft 2020-12) for every shared-state file ├── tests/scripts/ # bash unit tests for scripts/ (one per script/schema) @@ -29,6 +30,7 @@ Behavior is split deliberately: - **Agent prompts (`agents/*.md`)** hold the reasoning, decisions, and safety prose. Things a model needs to *judge* stay here. - **Scripts (`scripts/*.sh`)** hold the deterministic shell — JSON reads/writes, state transitions, gates. Things a computer should *execute the same way every time* live here. +- **User config (`~/.superhuman/preferences.md`)** holds what the *user* wants. Everything else under `~/.superhuman/` is machine-written state the plugin owns; this one file is theirs, and only `/preferences` writes it on their behalf. When you find yourself writing more than a couple of lines of `bash`/`jq` inside an agent prompt, extract it to `scripts/` and call it from the prompt. That keeps prompts short (cheaper to load per phase) and makes the logic testable. diff --git a/README.md b/README.md index 995dee5..2a378bf 100644 --- a/README.md +++ b/README.md @@ -176,9 +176,52 @@ Codex does not have slash commands. Use these prompts to invoke the same workflo | `/contribution-fleet [N]` | Not supported in Codex — fleet runs require parallel subagent dispatch. Run sequential loops instead. | | `/contribution-dashboard [owner/repo]` | `Use the superhuman skill to show the contribution dashboard for owner/repo (or all repos if omitted).` | | `/repo-finder [N]` | `Use the superhuman skill to refresh my open-source repo shortlist with up to N candidates.` | +| `/preferences` | `Use the superhuman skill to set my repo search preferences.` | +| `/preferences --show` | `Use the superhuman skill to show my repo search preferences and the queries they compile to.` | ## Usage +**Tell it what you want to work on** — run `/preferences` once. Without it, the scan uses a default profile (AI/ML, Java, Python, dev-tools, >20K stars), which is fine but is not *yours*: + +```text +/preferences +``` + +It asks four things — languages, topics, a star floor, and anything else in your own words — then writes `~/.superhuman/preferences.md`: + +```markdown +## Filters +languages: go, rust +topics: backend, infrastructure, llm:5000, any +stars: 2000 + +## Notes +Prefer small focused libraries over sprawling frameworks. +I'd rather fix bugs than add features — feature work invites bikeshedding. +Nothing that needs a GPU to run the test suite. +``` + +The two halves do different jobs, and the split is deliberate: + +- **`## Filters` is mechanical.** It compiles 1:1 into GitHub search qualifiers. "Only Go and Rust" means only Go and Rust. `topic[:min_stars]` gives one topic its own floor (`llm:5000`); `any` is reserved and means "…plus repos carrying none of these tags"; `stars` is a **floor**, and there is no ceiling. +- **`## Notes` is prose.** Nothing can hard-filter on "focused rather than sprawling", so the agent applies your notes where judgment actually happens — breaking ties between repos, and choosing which issue to fix — and it tells you in the shortlist's `notes` field when your prose drove the pick. It never touches a score. + +You can edit the file by hand; it is yours. `/preferences --show` prints it back along with the exact queries it compiles to, so a filter is never opaque: + +```text +Your next /repo-finder will search: + q='language:go language:rust topic:backend stars:>2000 archived:false' + q='language:go language:rust topic:infrastructure stars:>2000 archived:false' + q='language:go language:rust topic:llm stars:>5000 archived:false' + q='language:go language:rust stars:>2000 archived:false' +``` + +One query per topic — languages are free (they union inside a single query, so adding one costs nothing). To deviate for a single run without touching the saved file: + +```text +/repo-finder 5 --lang rust --topic cli +``` + **Single run** — let the orchestrator pick everything: ``` @@ -231,7 +274,8 @@ Orchestrator + 10 specialists + 1 shared-state contract document. | Command | Purpose | |---|---| | `/contribute [owner/repo] [issue#]` | One full end-to-end contribution. Loopable. | -| `/repo-finder [N]` | Refresh `repo-shortlist.json` with up to N candidate repos (default 10, max 25). | +| `/preferences` | Set what `/repo-finder` searches for — languages, topics, star floor, free-form notes. Writes `~/.superhuman/preferences.md`. `/preferences --show` prints the current file and the queries it compiles to. | +| `/repo-finder [N] [--lang X] [--topic Y] [--min-stars Z]` | Refresh `repo-shortlist.json` with up to N candidate repos (default 10, max 25). The flags override your saved preferences **for that run only** — they never write to disk. | | `/contribute-loop [N]` | Run N sequential contributions (default 3, max 20). Stops on `suspicious_halt` or `crash`. | | `/contribution-fleet [N \| owner/repo ...]` | Launch N parallel contributor runs. | | `/contribution-dashboard [owner/repo]` | Read-only view of active run, score history, plateaued dimensions, iteration cap, recent merge outcomes, latest loop. | @@ -265,9 +309,11 @@ superhuman/ │ ├── contribute-loop.md │ ├── contribution-dashboard.md │ ├── contribution-fleet.md +│ ├── preferences.md │ └── repo-finder.md ├── scripts/ # Versioned shell extracted from agent prompts (v0.5.0+) -│ ├── lib/ # Shared helpers: state.sh, mistakes.sh, flake.sh, delim.sh, lesson_checks.sh +│ ├── lib/ # Shared helpers: state.sh, preferences.sh, mistakes.sh, flake.sh, delim.sh, lesson_checks.sh +│ ├── repo-finder/ # build_queries.sh — compiles preferences.md into GitHub search queries │ ├── profiler/ # repo-profiler steps + scan_structure/write_repo_scan/dossier_fresh │ ├── scorer/ # merge-probability-scorer step implementations │ ├── orchestrator/ # opensource-contributor + reputation_gate.sh + write_run_summary.sh @@ -286,6 +332,7 @@ All persistent state lives under `~/.superhuman/`. Per-repo state is keyed by `< ``` ~/.superhuman/ +├── preferences.md # YOURS. Written by /preferences, read by repo-finder. ├── repos/ │ └── apache-airflow/ │ ├── repo_profile.json @@ -317,6 +364,8 @@ All persistent state lives under `~/.superhuman/`. Per-repo state is keyed by `< File ownership (sole-writer + readers) is documented in `agents/SHARED_STATE.md`. Every shared-state file has a matching JSON Schema (draft 2020-12) under `schemas/`, validated at write time. +`preferences.md` is the exception, and its location says so: `global/` holds cross-repo *state* (mostly agent-written, with user-editable exceptions like `repo_blocklist.json`), while `preferences.md` sits at the root because it is not state at all — it is **input**. No agent writes it; `/preferences` writes it on your behalf. + ## Safety rails - **Impact audits before refactors.** `builder` invokes `impact-auditor` before applying any reviewer-suggested refactor to a shared function. Blocks the class of bug where "just read `self.app.config` instead of calling `conf.get()`" is correct at Flask request time and fatal at FastAPI startup. @@ -340,6 +389,12 @@ See [SECURITY.md](./SECURITY.md) for the full safety model and how to report a v **What is superhuman?** superhuman is an autonomous open-source contribution agent for [Claude Code](https://claude.com/claude-code) and Codex. It's a multi-agent harness that picks an open issue in a repository, writes the fix, scores its own pull request on a 10-dimension merge-probability rubric, and iterates on the weakest dimension until a maintainer would merge it. +**How do I make it look for the kind of repos I actually care about?** +Run `/preferences`. It asks for languages, topics, a star floor, and anything else in your own words, then writes `~/.superhuman/preferences.md` — a file you own and can hand-edit. Filters are hard (only Go and Rust means only Go and Rust); the notes are advisory and steer which issue gets picked. `/preferences --show` prints the exact GitHub queries your answers compile to, so you can see the filter rather than trust it. Without a preferences file, the scan uses a sensible default profile. + +**Why does `/repo-finder` return fewer repos than I asked for?** +Because your filters matched that many. It reports honestly ("12 of 63 candidates matched; showing 12") and will not widen a query to pad the list — filler is what made the shortlist feel random in the first place. Loosen the filters (drop a topic, lower `stars`, add `any`) if you want more. + **How is it different from Copilot, Cursor, Devin, or a plain coding agent?** Most AI coding tools are *open-loop*: they generate a change once and hope it's good. superhuman is *closed-loop*. It grades its own pull request before opening it, spends each iteration fixing the single weakest dimension, re-scores, and stops only when it converges — gradient descent on one objective: *will a maintainer merge this?* diff --git a/agents/SHARED_STATE.md b/agents/SHARED_STATE.md index 43e3fa7..e23fd1e 100644 --- a/agents/SHARED_STATE.md +++ b/agents/SHARED_STATE.md @@ -47,6 +47,26 @@ Repo-agnostic: `~/.superhuman/global/` | `lessons_global.jsonl` | lesson-distiller | planner, builder, merge-probability-scorer | | `lesson_regressions.jsonl` | lesson-distiller | merge-probability-scorer, future `/contribution-dashboard` | +User-owned: `~/.superhuman/` + +| File | Owner | Readers | +|------|-------|---------| +| `preferences.md` | **user** (form: `/preferences`) | repo-finder | + +`preferences.md` sits at the root of `~/.superhuman/`, not under `global/`, and +the location is the point: `global/` is **cross-repo state**, mostly written by +agents (`repo-shortlist.json`, `repo_cooldown.json`, `merge_outcomes.jsonl`) with +a couple of user-edited exceptions (`repo_blocklist.json`). `preferences.md` is +not state at all — it is *input*, the only file here that exists purely to tell +the plugin what you want. No agent writes it; `/preferences` writes it on your +behalf, and repo-finder only reads it. + +Its `## Filters` block is mechanical (parsed by `scripts/lib/preferences.sh`, +compiled to search qualifiers by `scripts/repo-finder/build_queries.sh`). Its +`## Notes` block is prose the agent applies at tie-breaks and issue selection +only — never to a numeric score, and always disclosed in the shortlist's `notes` +field when it did. + `` is formed as `-` (single hyphen; slash replaced). Example: `apache/airflow` → `apache-airflow`. diff --git a/agents/builder.md b/agents/builder.md index 21a4921..16ef861 100644 --- a/agents/builder.md +++ b/agents/builder.md @@ -71,7 +71,11 @@ edit, check whether its path is present in `entries`. ```bash GEN="$STATE_DIR/generated_files.json" if [ -f "$GEN" ]; then - for f in $PLAN_EDIT_FILES; do + # One path per line. `for f in $PLAN_EDIT_FILES` would need the shell to split an + # unquoted expansion; zsh does not, so every path glues into one string, matches + # no entry, and this guard silently passes. + while IFS= read -r f; do + [ -n "$f" ] || continue MATCH=$(jq --arg p "$f" '.entries[] | select(.path == $p)' "$GEN") if [ -n "$MATCH" ]; then RC=$(jq -r '.regenerate_cmd // "null"' <<<"$MATCH") @@ -95,7 +99,9 @@ EOF git -C "$WORKDIR" add "$f" fi fi - done + done </dev/null) # Changed-files list = the plan's touch list, one path per line. CF_TMP="$STATE_DIR/.build_changed.$$" : > "$CF_TMP" -for f in $PLAN_EDIT_FILES; do +while IFS= read -r f; do [ -n "$f" ] && printf '%s\n' "$f" >> "$CF_TMP" -done +done <&2; exit 10; } # Feed the loop by heredoc, not `echo ... | while`. A piped while runs in a # subshell, where `exit` terminates only the subshell and the scan continues past @@ -95,11 +116,28 @@ $(printf '%s' "$RESP" | jq -r '.total_count') matched: $q" >&2 archived, open_issues_count, default_branch, description}') " done < -> file body on stdout, empty if absent curl -sfL "https://raw.githubusercontent.com/OWNER/REPO/$DEFAULT_BRANCH/$1" || true } +# HARD SKIP: curated list, not a code repo. Reads only fields already in +# $CANDIDATE — no API call. Exit 1 = list. Exit 10 = filter misconfigured: abort, +# by the same rule as the reputation gate, because a config error applies to every +# repo equally and skipping on it silently empties the shortlist. +# +# Scoring cannot do this job. Lists answer issues fast and merge outside PRs +# readily, so they score *well* on responsiveness (35%) and outside-contributor +# track (25%) — 60% of the weight — while offering nothing to contribute. A list +# is not a repo we rank low; it is a repo we do not rank at all. +CLF_RC=0 +printf '%s' "$CANDIDATE" \ + | "${CLAUDE_PLUGIN_ROOT}/scripts/repo-finder/curated_list_filter.sh" || CLF_RC=$? +if [ "$CLF_RC" -eq 10 ]; then + echo "FATAL: curated-list filter is misconfigured — aborting scan." >&2 + exit 10 +fi +[ "$CLF_RC" -eq 1 ] && echo "SKIP: curated list / book / roadmap" + # HARD SKIP: PR template requires pre-approval. These phrases mean external PRs # get bot-closed unless the author is pre-assigned. Racing for assignment is not # a viable autonomous strategy. @@ -186,18 +242,52 @@ fi # List workflows with one tree call, then read each body via raw(). WORKFLOWS=$(gh api "repos/OWNER/REPO/git/trees/$DEFAULT_BRANCH?recursive=1" \ --jq '.tree[] | select(.path | startswith(".github/workflows/")) | .path' 2>/dev/null) -for wf in $WORKFLOWS; do +while IFS= read -r wf; do + [ -n "$wf" ] || continue if raw "$wf" | grep -qiE "require-issue-link|require-assigned|auto-close.*unassigned|close.*unassigned"; then echo "SKIP: repo has auto-close-unassigned workflow" fi -done +done < "$ISSUES" +FIELDS=number,title,labels,comments,createdAt,assignees,body,isPinned + +# Three fetches, unioned and deduped — never a bare `--limit 30`. +# +# `--limit 30` returns the 30 NEWEST issues, and Stage A then drops everything +# under 24h old. On a high-velocity repo all 30 are under 24h, so the repo yields +# zero candidates and is cut for "no clear contribution path" — even when 3b's +# server-side count credits it with dozens of open good-first-issues. +# +# The two label queries filter SERVER-side, so they see every open issue rather +# than a page of the newest. A repo without those labels returns an empty list. +{ + gh issue list --repo OWNER/REPO --state open --limit 30 \ + --label "good first issue" --json "$FIELDS" 2>/dev/null || echo '[]' + gh issue list --repo OWNER/REPO --state open --limit 30 \ + --label "help wanted" --json "$FIELDS" 2>/dev/null || echo '[]' + gh issue list --repo OWNER/REPO --state open --limit 60 --json "$FIELDS" +} | jq -s 'add | unique_by(.number)' > "$ISSUES" ``` Filter in two stages. Stage A runs on `$ISSUES` alone and costs nothing. Stage B @@ -481,6 +621,10 @@ jq -c 'select(.verdict == "KEEP") | del(.verdict)' "$SCRATCH/stage_a_all.jsonl" > "$SCRATCH/stage_a.jsonl" ``` +A KEEP row carries the issue payload — `title`, `labels`, `body`, `createdAt` +alongside the maintainer flags. **Score the rubric below off this row.** Do not +re-join against `$ISSUES` and do not re-fetch. + **Stage B — one batched call.** The timeline is the only thing Stage A cannot answer, and it serves two filters at once. Fetch it for the survivors in a single aliased GraphQL query rather than two paginated REST calls per issue. @@ -507,9 +651,14 @@ MAINT_WINDOW_DAYS="${MAINT_WINDOW_DAYS:-45}" # Build one query with an alias per issue. 20 per request keeps the response # under GraphQL's node limit; chunk if more survive. -build_query() { # build_query +# Numbers arrive on stdin, one per line — NOT as arguments. `build_query $NUMS` +# would need the shell to split an unquoted expansion; zsh does not, so every +# number glues into one argument, the query goes out as `i4120\n4107\n…`, GitHub +# answers `Expected NAME, actual: INT`, and the guard below aborts the scan. +build_query() { # issue numbers on stdin, one per line printf 'query { repository(owner: "OWNER", name: "REPO") {\n' - for n in "$@"; do + while IFS= read -r n; do + [ -n "$n" ] || continue printf ' i%s: issue(number: %s) { number timelineItems(last: 100, itemTypes: [CROSS_REFERENCED_EVENT, LABELED_EVENT, ASSIGNED_EVENT, MILESTONED_EVENT, RENAMED_TITLE_EVENT, REOPENED_EVENT]) { nodes {\n' "$n" "$n" printf ' __typename\n' printf ' ... on CrossReferencedEvent { source { ... on PullRequest { number state merged closedAt } } }\n' @@ -526,14 +675,27 @@ build_query() { # build_query NUMS=$(jq -r '.number' "$SCRATCH/stage_a.jsonl") [ -z "$NUMS" ] && { echo "no issue survived Stage A"; return; } -# Abort on failure — an errored fetch must never arrive at a filter as "no -# events", which reads as both "abandoned" and "no competing PR" at once. -if ! TL=$(gh api graphql -f query="$(build_query $NUMS)" 2>&1); then - echo "FATAL: GitHub GraphQL failed — aborting scan." >&2 - echo " $TL" >&2 - exit 10 -fi -printf '%s' "$TL" | jq '[.data.repository[] | select(. != null)]' > "$SCRATCH/timelines.json" +# Chunk at 20 aliases per request. A busy repo routinely leaves 30-50 issues alive +# after Stage A, and one query carrying 50 `issue(...)` aliases each asking for +# `timelineItems(last: 100)` exceeds GraphQL's node limit. GitHub answers that with +# an ERROR, not a truncated result, so the abort below kills the whole scan. +CHUNK_DIR="$SCRATCH/chunks"; mkdir -p "$CHUNK_DIR" +PARTS="$SCRATCH/timelines.parts.jsonl"; : > "$PARTS" +printf '%s\n' "$NUMS" | split -l 20 - "$CHUNK_DIR/nums." + +for part in "$CHUNK_DIR"/nums.*; do + [ -s "$part" ] || continue + # Abort on failure — an errored fetch must never arrive at a filter as "no + # events", which reads as both "abandoned" and "no competing PR" at once. + if ! TL=$(gh api graphql -f query="$(build_query < "$part")" 2>&1); then + echo "FATAL: GitHub GraphQL failed — aborting scan." >&2 + echo " $TL" >&2 + exit 10 + fi + printf '%s' "$TL" | jq -c '.data.repository[] | select(. != null)' >> "$PARTS" +done + +jq -s '.' "$PARTS" > "$SCRATCH/timelines.json" # Join Stage A with the timelines and apply both Stage B filters. jq -r --rawfile m "$MAINTAINERS" \ @@ -581,7 +743,11 @@ jq -r --rawfile m "$MAINTAINERS" \ equally mean they gave up or were rejected, which would make the issue more available, not less. Revisit if merge outcomes suggest it over-skips. -Pick the issue that scores highest on: +Pick the issue that scores highest on the list below — and this is the second +place `## Notes` binds. "I'd rather fix bugs than add features" should visibly +change which issue wins; "nothing that needs a GPU to run the test suite" should +visibly drop one. Apply the prose here, then say in `notes` that it did. + - **Is a bug** (not a feature request): +3 - **Has reproduction steps**: +2 - **Labeled good-first-issue or help-wanted**: +2 @@ -637,14 +803,32 @@ Shortlist payload shape. The row key is `repo`, not `full_name`: map Step 1's `full_name` across on write, or the orchestrator's `.repos[0].repo` binds null. `topics[]` comes from the Step 1 projection; cite it in the `notes` rationale. +Record the **resolved** filters in `criteria` — the ones that actually ran, from +`parse_preferences`, not the ones you meant to run. Every shortlist then says what +produced it, and a scan that searched Python when you asked for Go is visible in +the file rather than three scans later. + +Record `coverage` too: `candidates` (unique after dedup), `survivors` (passed the +Step 2 fast filter), `scored` (survivors you actually scored in Step 3), and +`returned` (rows in this shortlist). `scored` should equal `survivors`; a gap +means the safety ceiling or a rate/time limit truncated the scan, and the reader +needs that number to know the list is partial rather than the field being thin. + ```json { "generated_at": "ISO8601", "criteria": { - "min_stars": 20000, - "categories": ["ai-ml", "java", "python", "springboot", "fastapi", "tools", "other"], + "profile": "user", + "languages": ["go", "rust"], + "topics": [{"topic": "backend"}, {"topic": "llm", "min_stars": 5000}], + "min_stars": 2000, + "queries": [ + "language:go language:rust topic:backend stars:>2000 archived:false", + "language:go language:rust topic:llm stars:>5000 archived:false" + ], "min_score": 60 }, + "coverage": {"candidates": 102, "survivors": 78, "scored": 78, "returned": 5}, "repos": [ { "rank": 1, @@ -659,7 +843,6 @@ Shortlist payload shape. The row key is `repo`, not `full_name`: map Step 1's "opportunity_quality": 8, "outside_contributor_track": 7, "ai_friendliness": 9, - "category_bonus": 10, "final": 87 }, "responsiveness_detail": { @@ -719,6 +902,8 @@ To start contributing: - **Responsiveness is king, but only maintainer-to-external responsiveness** — measuring maintainer self-merge speed is useless. What matters is how fast external contributors get merged, which 3c captures. - **One issue per repo** — don't list 5 issues per repo. Find the single best one. The contributor agent will re-evaluate anyway. - **Freshness matters** — repos that haven't merged a PR in 30 days are stale regardless of star count -- **Rate limit budget** — 5,000 REST requests/hour. Search phase ~20 queries. Per-repo scoring ~8: one `repos/` call, two cached list fetches, three for the maintainer union, one tree call. File bodies come from raw.githubusercontent and cost nothing. Step 4 adds ~2 per repo: one bulk issue list, one batched GraphQL timeline query over the Stage-A survivors. Scoring 50 repos and running Step 4 on the top 15 costs ~500 requests. **Never issue a per-issue call for data the bulk `gh issue list` already returned** — its `comments` array carries `authorAssociation`, `author.login` and `createdAt`. +- **Never widen a filter to fill the list** — if the user's filters yield 12 repos, the answer is 12 repos. Filler is what made this scan feel random. +- **Never let prose move a number** — `## Notes` applies at tie-breaks and issue selection only, and it discloses itself in `notes` when it does. A rubric prose can silently reweight is one nobody can explain. +- **Rate limit budget** — 5,000 REST requests/hour, but **search is a separate, far tighter pool: 30 requests/minute**. The search phase now costs one query per topic in the user's profile (five on the default profile), so it is nowhere near that cap — but the cap is why `/preferences` limits topics to 20 and does not limit languages at all: languages OR inside a single query and cost nothing. Per-repo scoring ~8: one `repos/` call, two cached list fetches, three for the maintainer union, one tree call. File bodies come from raw.githubusercontent and cost nothing. Step 4 adds ~2 per repo: one bulk issue list, one batched GraphQL timeline query over the Stage-A survivors. Scoring 50 repos and running Step 4 on the top 15 costs ~500 requests. **Never issue a per-issue call for data the bulk `gh issue list` already returned** — its `comments` array carries `authorAssociation`, `author.login` and `createdAt`. - **Cache aggressively** — save the shortlist with a timestamp. If the user runs this again within 24 hours, load the cache and only re-check the top 5 for freshness. - **Be honest about uncertainty** — if a repo looks promising but you couldn't verify AI-friendliness or responsiveness, say so in the notes field. Don't inflate scores. diff --git a/commands/contribution-fleet.md b/commands/contribution-fleet.md index 18fa8b8..eba9863 100644 --- a/commands/contribution-fleet.md +++ b/commands/contribution-fleet.md @@ -38,14 +38,19 @@ if [ -z "$ARG" ]; then elif [[ "$ARG" =~ ^[0-9]+$ ]]; then N="$ARG" else - # Explicit slugs - for tok in $ARG; do + # Explicit slugs, one per line. `for tok in $ARG` would need the shell to split an + # unquoted expansion; zsh does not, so every slug glues into one token and the + # owner/repo regex rejects a valid invocation. + while IFS= read -r tok; do + [ -n "$tok" ] || continue if [[ "$tok" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]]; then TARGETS+=("$tok") else echo "SKIP malformed slug: $tok" fi - done + done <20K stars**, with `llm` at +**>5K** (LLM repos are younger; a 20K floor erases the category). + +## The file + +```markdown +# My superhuman preferences + +## Filters +languages: go, rust +topics: backend, infrastructure, llm:5000, any +stars: 2000 + +## Notes +Prefer small focused libraries over sprawling frameworks. +I'd rather fix bugs than add features — feature work invites bikeshedding. +Nothing that needs a GPU to run the test suite. +``` + +The two halves promise different things, and being explicit about which one binds +is the whole point: + +- **`## Filters` is mechanical.** Parsed by `scripts/lib/preferences.sh`, compiled + 1:1 into GitHub search qualifiers. "Only Go and Rust" means only Go and Rust. +- **`## Notes` is advisory.** No code can hard-filter on "focused rather than + sprawling". The agent applies it at tie-breaks and issue selection, and it must + disclose in the shortlist's `notes` field when it did. Prose is never allowed to + move a numeric score. + +Treating prose as if it were a filter is exactly how the old version ended up +ignoring its own `stars:>20000`. + +## Syntax + +| Line | Meaning | +|---|---| +| `languages: go, rust` | Language ∈ {go, rust}. **Languages are free** — they OR inside one query, so adding one costs zero queries. | +| `topics: backend, llm:5000` | **One query per topic.** `topic[:min_stars]` — a per-entry floor overrides the global one for that topic only. | +| `topics: … , any` | `any` is **reserved**: it compiles to a query with *no* `topic:` qualifier, for repos carrying none of your tags. It never means GitHub's real `any` topic. | +| *(no `topics:` line)* | Exactly `topics: any`. | +| `stars: 2000` | A **floor**. There is no ceiling — no ranges, no maximum. | + +**Why no star ceiling.** The only argument for one is "mega-repos don't merge +outsider PRs", and `repo-finder` already measures that per repo from real data +(median response time, median merge time, external-merge ratio, pre-approval PR +templates). A ceiling is a guess about those measurements; the scorer has the +answer. Worse, it fails asymmetrically: `ollama` (~176K), `ragflow` (~85K) and +`netdata` (~80K) all sit above any plausible cap, so a ceiling silently deletes +the category the default profile exists to find. A range in the file is a **loud +error**, not a quietly honored cap. + +## Workflow + +### Step 1: `--show` + +If `$ARGUMENTS` is `--show`, print the current file and the queries it compiles +to, then stop: + +```bash +"${CLAUDE_PLUGIN_ROOT}/scripts/lib/usage_event.sh" --command preferences >/dev/null 2>&1 & disown 2>/dev/null || true + +PREFS="$HOME/.superhuman/preferences.md" +if [ ! -f "$PREFS" ]; then + echo "No preferences yet — /repo-finder is using the default profile." +else + cat "$PREFS" +fi +echo +echo "Your next /repo-finder will search:" +"${CLAUDE_PLUGIN_ROOT}/scripts/repo-finder/build_queries.sh" --no-nudge | sed 's/^/ q=/' +``` + +### Step 2: Ask four things + +1. **Languages** — free text, comma-separated. Empty means any language. +2. **Topics** — offer these well-populated GitHub topics as a multi-select, and + accept free text for anything else: `backend`, `infrastructure`, `devops`, + `cli`, `database`, `security`, `frontend`, `machine-learning`. The list is a + convenience, not a gate — **any GitHub topic is valid**. Mention `any` as the + option for "…plus untagged repos I might like". +3. **Star floor** — default `2000`. Say why: smaller repos merge outsider PRs + faster. Offer **no** ceiling. +4. **Anything else, in your own words** — free text, lands in `## Notes`. + +### Step 3: Count the compiled queries live + +A typo'd topic is not an error at GitHub — it is a filter that silently matches +nothing, and the scan reads as a thin week for open source. + +**Count the query the scan will actually run, not the bare topic.** Two failures +hide behind a bare `topic:` check, and both look like a healthy filter: + +- **A typo other people also made.** `topic:bakcend` matches 26 real repos — humans + misspell their own topics — so any `total_count > 0` test passes it, and the scan + still returns nothing once the *other* qualifiers apply. +- **A correct topic that is empty in context.** `topic:backend` has ~42K repos, but + `language:go language:python topic:backend stars:>20000` matches **2**. Nothing is + misspelled; the profile is too narrow to rank. + +Only the compiled query answers either question, and it costs the same one call per +topic. + +```bash +# Compile the candidate profile, then count what it emits. Call build_queries.sh +# rather than re-deriving the queries here: it is the single compiler, and a second +# one would drift from the scan this check is meant to predict. +# mktemp, not $$-predictable names: `>` onto a guessable path in a world-writable +# /tmp is open to stale-file reuse and symlink redirection. Both temps live and die +# inside this block, so an EXIT trap is the right cleanup here. +CHECK=$(mktemp "${TMPDIR:-/tmp}/superhuman-prefs-check.XXXXXX") || { echo "mktemp failed"; exit 1; } +CHECKQ=$(mktemp "${TMPDIR:-/tmp}/superhuman-prefs-checkq.XXXXXX") || { rm -f "$CHECK"; echo "mktemp failed"; exit 1; } +trap 'rm -f "$CHECK" "$CHECKQ"' EXIT +{ + echo "## Filters" + [ -n "$LANGUAGES" ] && echo "languages: $LANGUAGES" + [ -n "$TOPICS" ] && echo "topics: $TOPICS" + echo "stars: $STARS" +} > "$CHECK" + +# A malformed profile exits 10 here — before anything reaches disk. +"${CLAUDE_PLUGIN_ROOT}/scripts/repo-finder/build_queries.sh" --file "$CHECK" --no-nudge \ + > "$CHECKQ" || { echo "Fix the filters above and try again."; exit 10; } + +while IFS= read -r q; do + [ -n "$q" ] || continue + # total_count, not the items array: a MALFORMED query also returns an empty + # array, so counting rows cannot tell a typo from a genuinely empty result. + if ! n=$(gh api -X GET search/repositories -f q="$q" -f per_page=1 --jq '.total_count' 2>&1); then + printf ' %s\n -> API call failed (rate limit / auth?): %s\n' "$q" "$n" + continue + fi + printf ' %s\n -> %s repos\n' "$q" "${n:-0}" + if [ "${n:-0}" -eq 0 ]; then + echo " WARNING: matches NOTHING. Check the topic for a typo, or lower the star floor." + elif [ "${n:-0}" -lt 10 ]; then + echo " WARNING: only ${n} repos — too few to rank. Lower the star floor, or drop a language." + fi +done < "$CHECKQ" +``` + +`any` needs no special handling: it is reserved, and `build_queries.sh` already +compiles it to a query with no `topic:` qualifier — so what gets counted is that +query, which is the thing that matters. + +This lives here and nowhere else. The unit suite is network-free by design, so a +live check belongs in the interactive command, which always has a network. + +**Cap topics at 20.** Query count *is* topic count, and GitHub's search API allows +30 requests/minute. Do not cap languages — they cost nothing. + +### Step 4: Compile prose, and show the compile + +The user may answer in a sentence. **You may compile a sentence into `## Filters` +— here, at authoring, with the user confirming.** Show them the result before you +write it: + +```text +You said: "I need llm repos having >5k stars, and Go backend stuff" + +Writing to ~/.superhuman/preferences.md: + languages: go + topics: llm:5000, backend + stars: 2000 + +Your next /repo-finder will search: + q='language:go topic:llm stars:>5000 archived:false' + q='language:go topic:backend stars:>2000 archived:false' + 2 queries — one per topic. [y/n] +``` + +Anything that cannot become a qualifier — *"prefer small focused libraries"*, +*"rather fix bugs than add features"* — goes to `## Notes` and stays advisory. + +**The compile happens once, at authoring.** What lands on disk is mechanical, and +`build_queries.sh` never sees a sentence: an LLM in the scan path costs +determinism (same file, same queries, every run), testability (CI is network-free, +so a prose-driven generator cannot be tested by the gate that protects it), and +fail-loudness (a misparsed sentence scans confidently wrong with exit 0). + +### Step 5: Write, then echo the queries + +Atomic temp-rename, same as every other writer: + +```bash +PREFS="$HOME/.superhuman/preferences.md" +mkdir -p "$HOME/.superhuman" +# mktemp in the destination dir, not a $$-predictable sibling: keeps the rename +# atomic (same filesystem) while closing the stale-file/symlink window. +TMP=$(mktemp "$PREFS.tmp.XXXXXX") || { echo "mktemp failed"; exit 1; } +cat > "$TMP" </dev/null \ + || { echo "FATAL: refusing to write preferences the parser rejects" >&2; rm -f "$TMP"; exit 10; } +mv "$TMP" "$PREFS" + +echo "Your next /repo-finder will search:" +"${CLAUDE_PLUGIN_ROOT}/scripts/repo-finder/build_queries.sh" --no-nudge | sed 's/^/ q=/' +``` + +Echoing the compiled queries is not decoration: it makes the filter +**non-opaque**. Ask for Go, see `language:python`, catch it in ten seconds instead +of three scans later. It also makes the AND/OR asymmetry visible — two `topic:` +qualifiers on one line is a bug the user can spot themselves. + +## Rules + +- **Never write a preferences file the parser rejects.** Round-trip through + `parse_preferences` before the rename. +- **`stars:` is a floor.** Never offer a ceiling, never write a range. +- **Filters are always hard.** No `strict` flag, no per-axis softening — soft + nuance lives in `## Notes`, which is where it can be honest about being soft. +- **`any` is reserved** and compiles to the absence of a `topic:` qualifier. Say + so when the user picks it. +- **Cap topics at 20; never cap languages.** +- **Never infer preferences from the local filesystem.** A repo finder that + rummages through `~/projects` looking for `go.mod` is not what anyone expects it + to do, and this plugin ships to other people. +- **macOS bash 3.2 safe.** No bash 4 features. diff --git a/commands/repo-finder.md b/commands/repo-finder.md index 35c517e..c17efcb 100644 --- a/commands/repo-finder.md +++ b/commands/repo-finder.md @@ -1,6 +1,6 @@ --- name: repo-finder -description: Refresh repo-shortlist.json with up to N candidate repos. Wraps the repo-finder agent. Usage — `/repo-finder` (default N=10) or `/repo-finder 25` (request 25, capped at 25). +description: Refresh repo-shortlist.json with up to N candidate repos. Wraps the repo-finder agent. Usage — `/repo-finder` (default N=10), `/repo-finder 25` (request 25, capped at 25), or `/repo-finder 10 --lang rust --topic cli` (override this run's filters only). --- You are a thin wrapper around the `repo-finder` agent. You validate @@ -20,34 +20,150 @@ the agent name. # never block or fail the command. See scripts/lib/usage_event.sh, TELEMETRY.md. "${CLAUDE_PLUGIN_ROOT}/scripts/lib/usage_event.sh" --command repo-finder >/dev/null 2>&1 & disown 2>/dev/null || true -ARG="$ARGUMENTS" N=10 -if [ -n "$ARG" ]; then - if [[ ! "$ARG" =~ ^[0-9]+$ ]]; then - echo "Usage: /repo-finder [N] (positive integer, max 25)" - exit 1 +LANGS="" TOPICS="" MIN_STARS="" + +# NOTHING IN THIS FILE MAY READ $1..$9. They are slash-command template +# placeholders: the harness substitutes them at load time, before bash runs. A +# `case "$1"` argv parser therefore arrives as `case ""`, every token falls +# through to the usage arm, and `/repo-finder 5` exits 1 — the shell never sees +# the positional parameter that was written here. Parse the token stream with a +# pending-flag state variable and named locals instead; read no positional. +# +# Tokens are split on space/tab by hand because zsh — the macOS default, and the +# shell this block runs in — does not word-split unquoted expansions: an +# unquoted $ARGUMENTS arrives glued into one argument. Splitting explicitly also +# subsumes what `set -f` guarded: no token is left unquoted, so `--topic c*` +# cannot glob against the cwd. +# +# A `while ... done <` onto a guessable path in a world-writable + # /tmp is open to stale-file reuse and symlink redirection. The file must outlive + # this block (the agent reads it in Step 2), so it is removed explicitly on the + # error path below and after the agent returns — not via an EXIT trap that would + # fire the moment this block ends. + OVERRIDE=$(mktemp "${TMPDIR:-/tmp}/superhuman-prefs.XXXXXX") || { echo "mktemp failed"; exit 1; } + + # Inherit each unnamed axis from the saved profile, so --lang rust on a + # backend-topics profile still searches backend. Read only the `## Filters` block: + # a `## Notes` line that happens to begin `languages:`/`topics:`/`stars:` (copied + # prose) must never be mistaken for a hard filter. + FILTERS="" + [ -f "$SAVED" ] && FILTERS=$(awk '/^## Filters/{f=1;next} /^## /{f=0} f' "$SAVED") + if [ -z "$LANGS" ]; then + LANGS=$(printf '%s\n' "$FILTERS" | sed -n 's/^[[:space:]]*languages:[[:space:]]*//p' | head -1) + fi + if [ -z "$TOPICS" ]; then + TOPICS=$(printf '%s\n' "$FILTERS" | sed -n 's/^[[:space:]]*topics:[[:space:]]*//p' | head -1) + fi + if [ -z "$MIN_STARS" ]; then + MIN_STARS=$(printf '%s\n' "$FILTERS" | sed -n 's/^[[:space:]]*stars:[[:space:]]*//p' | head -1) + fi + [ -n "$MIN_STARS" ] || MIN_STARS=2000 + { + echo "# Per-run override — not saved." + echo + echo "## Filters" + [ -n "$LANGS" ] && echo "languages: $LANGS" + [ -n "$TOPICS" ] && echo "topics: $TOPICS" + echo "stars: $MIN_STARS" + } > "$OVERRIDE" + + # Fail before the scan, not during it: a malformed override must not silently + # fall through to the saved profile or the default one. + "${CLAUDE_PLUGIN_ROOT}/scripts/repo-finder/build_queries.sh" --file "$OVERRIDE" --no-nudge \ + | sed 's/^/ q=/' \ + || { echo "Invalid override."; rm -f "$OVERRIDE"; exit 1; } + PREFS_ARG="$OVERRIDE" +fi ``` + +### Step 2: Dispatch the agent + +Pass `SUPERHUMAN_PREFS` only when an override is in play — otherwise the agent +reads the saved profile, or falls back to the default one. + +```text Agent( subagent_type: "repo-finder", description: "refresh repo shortlist (N=)", prompt: | N= MODE=cli + SUPERHUMAN_PREFS= ) ``` +Delete the override file once the agent returns: `rm -f "$OVERRIDE"`. + ### Step 3: Render the new shortlist ```bash @@ -58,10 +174,25 @@ if [ ! -f "$SHORTLIST" ]; then fi echo "Top $N candidates:" -jq -r '.repos[] | "\(.scores.final)\t\(.repo)\t\(.notes // "")"' "$SHORTLIST" \ + +# Coverage line: make a partial scan visible here, not only in the agent's reply. +# `scored` < `survivors` means repos were never evaluated — the list is short +# because the scan stopped, not because the field is thin. Absent on older +# shortlists written before coverage existed, so print only when present. +jq -e '.coverage' "$SHORTLIST" >/dev/null 2>&1 && \ + jq -r '.coverage | " coverage: scored \(.scored // "?")/\(.survivors // "?") survivors, returned \(.returned // "?") of \(.candidates // "?") candidates"' "$SHORTLIST" + +# No awk here, for the same reason the parser above reads no positional: awk's +# field refs are spelled $1/$2/$3, and the harness eats them exactly as it eats +# a shell positional. `printf "%-6s %-32s %s\n",$1,$2,$3` reaches bash as +# `printf "%-6s %-32s %s\n",,,` — an awk syntax error, or worse, three blank +# columns. Format with shell printf over a jq @tsv stream instead. +printf "%-6s %-32s %s\n" score repo notes +jq -r '.repos[] | [(.scores.final|tostring), .repo, (.notes // "")] | @tsv' "$SHORTLIST" \ | head -"$N" \ - | awk -F'\t' 'BEGIN{printf "%-6s %-32s %s\n","score","repo","notes"} - {printf "%-6s %-32s %s\n",$1,$2,$3}' + | while IFS="$(printf '\t')" read -r f_score f_repo f_notes; do + printf "%-6s %-32s %s\n" "$f_score" "$f_repo" "$f_notes" + done ``` ## Rules @@ -73,6 +204,14 @@ jq -r '.repos[] | "\(.scores.final)\t\(.repo)\t\(.notes // "")"' "$SHORTLIST" \ candidate scoring) and `opensource-contributor` Phase 0 (at dispatch time) re-check blocklist + cooldown + active-lock. This wrapper does not gate. -- **`$ARGUMENTS` is optional.** Empty defaults to N=10. Anything other - than a positive integer is rejected with a usage line. +- **`$ARGUMENTS` is optional.** Empty defaults to N=10 and the saved + preferences (or the default profile when none exist). Anything that is + not a positive integer or a known override flag is rejected with a + usage line. +- **Overrides never persist.** `--lang` / `--topic` / `--min-stars` shape + one run. Only `/preferences` writes `~/.superhuman/preferences.md`. +- **There is no `--max-stars`.** `stars` is a floor. A ceiling is a guess + about what the responsiveness scorer already measures per repo, and it + silently deletes the biggest, most active repos rather than letting the + scorer judge them. - **macOS bash 3.2 safe.** No bash 4 features. diff --git a/install.sh b/install.sh index 7b961c8..5dbf39c 100755 --- a/install.sh +++ b/install.sh @@ -93,6 +93,32 @@ check_prereqs() { warn "gh is not authenticated — run 'gh auth login' before contributing" fi fi + + # jsonschema (the Python package) is what makes validate_json in + # scripts/lib/state.sh actually validate: without it the check degrades to a + # top-level required-key test that cannot see inside repos[]/scores/best_issue, + # so a shortlist with a null repo or a missing score validates clean. Install it + # here so the strict validator is the real runtime path, matching CI. PEP 668 + # (externally-managed Python — the macOS default) blocks a plain install, so + # escalate plain -> --user -> --break-system-packages, then verify. + if command -v python3 >/dev/null 2>&1; then + if python3 -c 'import jsonschema' 2>/dev/null; then + ok "python jsonschema" + else + info "Installing python 'jsonschema' (strict schema validation)" + python3 -m pip install --quiet jsonschema 2>/dev/null \ + || python3 -m pip install --quiet --user jsonschema 2>/dev/null \ + || python3 -m pip install --quiet --break-system-packages jsonschema 2>/dev/null \ + || true + if python3 -c 'import jsonschema' 2>/dev/null; then + ok "python jsonschema" + else + warn "python 'jsonschema' not installed — schema validation degrades to a top-level-key check. Install it: python3 -m pip install jsonschema" + missing=1 + fi + fi + fi + [ "$missing" = "0" ] || warn "Some tools are missing; install them from the README before running the workflow." } diff --git a/schemas/repo_shortlist.schema.json b/schemas/repo_shortlist.schema.json index 900cc34..e6a727e 100644 --- a/schemas/repo_shortlist.schema.json +++ b/schemas/repo_shortlist.schema.json @@ -11,7 +11,37 @@ "criteria": { "type": "object", "additionalProperties": true, - "description": "the filter that produced this list, e.g. {\"min_stars\":20000,\"min_score\":60}" + "description": "the RESOLVED filter that produced this list — what actually ran, from parse_preferences, not what was intended. A scan that searched Python when the user asked for Go is then visible in the file rather than three scans later.", + "properties": { + "profile": {"type": "string", "enum": ["user", "default", "override"], "description": "user = ~/.superhuman/preferences.md; default = DEFAULT_PROFILE (no preferences file); override = per-run --lang/--topic/--min-stars"}, + "languages": {"type": "array", "items": {"type": "string"}, "description": "empty means any language. Languages OR inside one query, so they never add queries."}, + "topics": { + "type": "array", + "description": "one search query per entry. `any` is reserved and compiles to a query with no topic: qualifier.", + "items": { + "type": "object", + "required": ["topic"], + "properties": { + "topic": {"type": "string"}, + "min_stars": {"type": "integer", "minimum": 0, "description": "per-entry star floor; overrides min_stars for this topic only"} + } + } + }, + "min_stars": {"type": "integer", "minimum": 0, "description": "a floor. There is no ceiling: no ranges, no maximum, anywhere."}, + "queries": {"type": "array", "items": {"type": "string"}, "description": "the literal q= strings sent to search/repositories"}, + "min_score": {"type": "number"} + } + }, + "coverage": { + "type": "object", + "additionalProperties": true, + "description": "How much of the candidate space was actually scored, so a partial scan is visible in the artifact rather than only in the agent's prose. `scored` much smaller than `survivors` means repos were never evaluated and a good lower-ranked repo may be missing — a fact a downstream consumer, or the user, can only act on if it is written here.", + "properties": { + "candidates": {"type": "integer", "minimum": 0, "description": "unique repos after dedup across all queries"}, + "survivors": {"type": "integer", "minimum": 0, "description": "candidates that passed the Step 2 fast filter"}, + "scored": {"type": "integer", "minimum": 0, "description": "survivors actually scored in Step 3; must equal survivors unless the deterministic Step 1 cap truncated the set"}, + "returned": {"type": "integer", "minimum": 0, "description": "repos in this shortlist"} + } }, "repos": { "type": "array", @@ -37,7 +67,6 @@ "opportunity_quality": {"type": "number"}, "outside_contributor_track": {"type": "number"}, "ai_friendliness": {"type": "number"}, - "category_bonus": {"type": "number"}, "final": {"type": "number"} } }, diff --git a/scripts/lib/preferences.sh b/scripts/lib/preferences.sh new file mode 100644 index 0000000..3910fed --- /dev/null +++ b/scripts/lib/preferences.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +# User search preferences (~/.superhuman/preferences.md). +# +# The file has two halves and they promise different things: +# +# ## Filters mechanical. Parsed here, compiled 1:1 into GitHub search +# qualifiers by scripts/repo-finder/build_queries.sh. +# ## Notes prose. Never parsed. Read by the agent at tie-breaks and issue +# selection only; it may not touch a numeric score or a query. +# +# Only the Filters block is read here. Everything else in the file is prose and +# is legal by construction — a Notes sentence like "languages are fun" is not a +# filter and must not be mistaken for one. +# +# No yq: scripts/profiler/parse_workflows.sh already depends on it and it is not +# installed on a stock macOS box. CI installs it, so a second dependency would go +# green in CI and dead on the user's laptop. awk + jq are hard requirements. +# +# Exit 10 = malformed preferences. Never a silent skip: a mistyped `langauges: go` +# that parses to "no language filter" produces a confidently wrong scan with +# exit 0, which is the failure class this file exists to prevent. + +PREFS_MAX_TOPICS=20 + +prefs_path() { + echo "${SUPERHUMAN_PREFS:-$HOME/.superhuman/preferences.md}" +} + +# Emit "keyvalue" for each line of the `## Filters` block. A line in that +# block that is not `key: value` is an error, not prose — prose belongs in Notes. +_prefs_filter_lines() { + awk ' + /^##[ \t]+/ { inblk = ($0 ~ /^##[ \t]*[Ff]ilters[ \t]*$/); next } + !inblk { next } + /^[ \t]*$/ { next } + /^[ \t]*#/ { next } + { + line = $0 + gsub(/^[ \t]+|[ \t]+$/, "", line) + i = index(line, ":") + if (i == 0) { print "ERR\t" line; exit } + key = substr(line, 1, i - 1) + val = substr(line, i + 1) + gsub(/^[ \t]+|[ \t]+$/, "", key) + gsub(/^[ \t]+|[ \t]+$/, "", val) + print key "\t" val + } + ' "$1" +} + +# Comma-separated list -> one trimmed entry per line. Whitespace inside an entry +# is preserved on purpose; the charset guards below reject it. +_prefs_split() { + printf '%s' "$1" | tr ',' '\n' | sed 's/^[[:space:]]*//; s/[[:space:]]*$//' | grep -v '^$' || true +} + +# parse_preferences -> JSON on stdout +# +# {"languages": ["go","rust"], +# "topics": [{"topic":"backend"}, {"topic":"llm","min_stars":5000}, {"topic":"any"}], +# "stars": {"min": 2000}} +# +# `any` is reserved: it compiles to a query with no `topic:` qualifier, never to +# a literal `topic:any` (GitHub has a real `any` topic; this will never mean it). +# An omitted `topics:` line is exactly `topics: any` — one mechanism, not two. +parse_preferences() { + local file="${1:-$(prefs_path)}" + [ -f "$file" ] || { echo "FATAL: preferences file not found: $file" >&2; return 10; } + + local key val langs="" topics="" stars="" + local seen_langs="" seen_topics="" seen_stars="" + local tab + tab=$(printf '\t') + + while IFS="$tab" read -r key val; do + [ -n "$key" ] || continue + case "$key" in + ERR) + echo "FATAL: not a 'key: value' line in ## Filters: '$val'" >&2 + echo " Prose belongs under ## Notes." >&2 + return 10 ;; + languages) + [ -z "$seen_langs" ] || { echo "FATAL: duplicate 'languages:' key in ## Filters" >&2; return 10; } + seen_langs=1; langs="$val" ;; + topics) + [ -z "$seen_topics" ] || { echo "FATAL: duplicate 'topics:' key in ## Filters" >&2; return 10; } + seen_topics=1; topics="$val" ;; + stars) + [ -z "$seen_stars" ] || { echo "FATAL: duplicate 'stars:' key in ## Filters" >&2; return 10; } + seen_stars=1; stars="$val" ;; + *) + echo "FATAL: unknown key in ## Filters: '$key' (expected: languages, topics, stars)" >&2 + return 10 ;; + esac + done <&2 + echo " There is no ceiling: no ranges (2000-60000), no '>' prefix." >&2 + return 10 ;; + esac + else + stars=2000 + fi + + # Entries are comma-separated, and the split keeps whitespace INSIDE an entry. + # That is what catches `languages: go OR rust`: the operator stays glued to its + # entry, trips the charset guard, and exits 10. Split on whitespace too and + # `OR` becomes a third language — a query GitHub answers with total_count: 0 + # and exit 0. + local lang_list="" tok + while IFS= read -r tok; do + [ -n "$tok" ] || continue + case "$tok" in + [A-Za-z0-9]*) ;; + *) echo "FATAL: invalid language: '$tok'" >&2; return 10 ;; + esac + case "$tok" in + *[!A-Za-z0-9+#._-]*) + echo "FATAL: invalid character in language: '$tok' (one language per comma)" >&2 + return 10 ;; + esac + lang_list="$lang_list $tok" + done <&2; return 10 ;; + esac + case "$name" in + *[!a-z0-9._-]*) + echo "FATAL: invalid character in topic: '$name' (one topic per comma)" >&2 + return 10 ;; + esac + case "$tok" in + *:*) + case "$floor" in + ''|*[!0-9]*) + echo "FATAL: topic star floor must be an integer, got '$tok' (want e.g. llm:5000)" >&2 + return 10 ;; + esac ;; + esac + + n=$((n + 1)) + if [ "$n" -gt "$PREFS_MAX_TOPICS" ]; then + echo "WARN: more than $PREFS_MAX_TOPICS topics; using the first $PREFS_MAX_TOPICS." >&2 + echo " Query count is topic count, and GitHub search allows 30/minute." >&2 + break + fi + topic_list="$topic_list ${name}${floor:+:$floor}" + done <&2 + echo " Add a 'languages:' line, or drop 'any' and name the topics you want." >&2 + return 10 + fi + + jq -nc \ + --arg langs "$lang_list" \ + --arg topics "$topic_list" \ + --argjson stars "$stars" ' + { + languages: ($langs | split(" ") | map(select(length > 0))), + topics: ($topics | split(" ") | map(select(length > 0)) | map( + if test(":") + then (split(":") | {topic: .[0], min_stars: (.[1] | tonumber)}) + else {topic: .} + end)), + stars: {min: $stars} + }' +} + +# One line at the top of every scan, so the active filter is never a mystery. +prefs_summary() { + local file="${1:-$(prefs_path)}" + if [ ! -f "$file" ]; then + # The floors are not uniform: llm sits at >5000 because LLM repos are younger + # and a 20K floor erases the category. Say so rather than round it off — a + # summary that misstates the active filter is worse than none. + echo "filters: default profile (machine-learning, artificial-intelligence, developer-tools >20000 · llm >5000 · java, python >20000)" + return 0 + fi + local json + json=$(parse_preferences "$file") || return 10 + printf '%s' "$json" | jq -r ' + "filters: languages=" + (if (.languages | length) == 0 then "any" else (.languages | join(",")) end) + + " · topics=" + (.topics | map(.topic + (if .min_stars then ">\(.min_stars)" else "" end)) | join(",")) + + " · stars >\(.stars.min)"' +} diff --git a/scripts/lib/state.sh b/scripts/lib/state.sh index bd208ac..77adf0f 100644 --- a/scripts/lib/state.sh +++ b/scripts/lib/state.sh @@ -41,13 +41,31 @@ jsonschema.validate(data, schema) PY return $? fi - local req + # Announce the degradation. This fallback checks TOP-LEVEL required keys only and + # cannot see inside an array, so every nested constraint goes unchecked: a + # shortlist whose `repos[]` rows are missing `repo` or `scores.final` validates + # clean here. Callers must assert their own row invariants, and can only know to + # do so if the degradation is visible. Silence makes an unvalidated document look + # validated. + echo "validate_json: WARN: python3 jsonschema not available — falling back to a" >&2 + echo " top-level required-key check. Nested constraints (array rows, sub-objects)" >&2 + echo " are NOT validated. Install it with: python3 -m pip install jsonschema" >&2 + + # Read one field per line. `for k in $req` would need the shell to split an + # unquoted expansion; zsh does not, and this file is SOURCED, so it runs in the + # caller's shell — zsh on macOS, where the whole field list arrives as one word + # and a valid document is rejected. This is the fallback path whenever + # `python3 -m jsonschema` is absent. + local req k req=$(jq -r '.required // [] | .[]' "$schema") - for k in $req; do + while IFS= read -r k; do + [ -n "$k" ] || continue jq -e --arg k "$k" 'has($k)' "$data" >/dev/null || { echo "validate_json: missing required field: $k" >&2 return 1 } - done + done < ../.. is the plugin root. if [ -z "${CLAUDE_PLUGIN_ROOT:-}" ]; then - echo "reputation_gate.sh: CONFIG ERROR: CLAUDE_PLUGIN_ROOT is unset." >&2 - echo " This is NOT a verdict about the repo. Export it and re-run." >&2 - exit "$EX_CONFIG" + CLAUDE_PLUGIN_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) || { + echo "reputation_gate.sh: CONFIG ERROR: cannot resolve plugin root from \$0=$0" >&2 + echo " This is NOT a verdict about the repo." >&2 + exit "$EX_CONFIG" + } fi if [ ! -r "${CLAUDE_PLUGIN_ROOT}/scripts/lib/state.sh" ]; then echo "reputation_gate.sh: CONFIG ERROR: cannot read" \ diff --git a/scripts/orchestrator/triage_filter.sh b/scripts/orchestrator/triage_filter.sh index fa1a69b..efddb43 100755 --- a/scripts/orchestrator/triage_filter.sh +++ b/scripts/orchestrator/triage_filter.sh @@ -5,10 +5,14 @@ # which issues are worth ranking. Emits, per issue that clears the age and # triage-signal gates: # -# {"verdict":"KEEP","number":N,"maintainer_commented":bool, +# {"verdict":"KEEP","number":N,"title":"…","labels":["bug"],"body":"…", +# "createdAt":"ISO8601","maintainer_commented":bool, # "maintainer_comment_assoc":bool,"last_maintainer_comment":"ISO8601"|null} # {"verdict":"SKIP","number":N,"reason":"..."} # +# A KEEP row carries the issue payload: the Step-4 rubric scores +# title/labels/body/createdAt and may not re-fetch them. +# # Issues that fail those two gates produce no output at all — they are not # candidates. Everything past them earns an explicit SKIP with a reason, because # the decline/claimed tests match prose and a heuristic you cannot watch misfire @@ -56,13 +60,21 @@ jq -c --rawfile m "$MAINTAINERS" --argjson now "$NOW" ' # Parens are load-bearing: `A + B as $x | rest` binds as `A + (B as $x | rest)`, # so an unparenthesized concat spills the string into the downstream object # stream and jq aborts ("string and object cannot be added") on every issue. + # A refusal is also a decline: "we cannot fix bugs on v4" closes the door as + # firmly as "wontfix". | ("by design|working as intended|works as intended|not a bug|isn.t a bug" + "|don.t think (that )?this is a bug|expected behaviou?r|this is expected" - + "|won.t fix|wontfix|out of scope|not something we|we don.t plan") as $declined + + "|won.t fix|wontfix|out of scope|not something we|we don.t plan" + + "|(can.t|cannot|can not) fix|not (fixing|supporting) (this|that|it)" + + "|no longer (supported|maintained)|we do not support") as $declined - # A maintainer who already holds the patch will land theirs, not ours. + # A maintainer who already holds the patch will land theirs, not ours. The fix is + # as often announced as promised, so match both: "should have fixed it in + # tokenizers v0.23.1" names a dead issue. | ("patch locally|i have a (fix|patch)|i.ll (fix|push|open|submit)" - + "|i.m working on|working on (a fix|this)|already fixed in" + + "|i.m working on|working on (a fix|this)" + + "|(already |should have )?fixed (it |this )?in [a-z0-9.@/#-]" + + "|(fix|patch) (has )?landed|landed in|resolved (in|by) (#|[a-z0-9])" + "|(fix|pr) (is )?incoming") as $claimed # Announcements and containers wear defect labels. A pinned "the project moved" @@ -128,7 +140,14 @@ jq -c --rawfile m "$MAINTAINERS" --argjson now "$NOW" ' {verdict: "SKIP", number: $i.number, reason: "maintainer is already fixing it"} else + # Carry the issue payload, not just a verdict on a number. The Step-4 rubric + # scores title, labels, body and createdAt, and the agent contract forbids + # re-fetching them — emit them here or the caller has no legal source. {verdict: "KEEP", number: $i.number, + title: $i.title, + labels: $L, + body: ($i.body // ""), + createdAt: $i.createdAt, maintainer_commented: ($mc | length > 0), maintainer_comment_assoc: ($mca | length > 0), last_maintainer_comment: ($mc | map(.createdAt) | max)} diff --git a/scripts/repo-finder/build_queries.sh b/scripts/repo-finder/build_queries.sh new file mode 100755 index 0000000..747521f --- /dev/null +++ b/scripts/repo-finder/build_queries.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# Compile ~/.superhuman/preferences.md into GitHub search queries (repo-finder +# Step 1). Prints one `q=` qualifier string per line; the caller feeds each to +# `gh api search/repositories`. +# +# The qualifier asymmetry is the whole design, and it is measured, not assumed: +# +# language: repeated -> OR `language:go language:rust` returns Go ∪ Rust +# topic: repeated -> AND `topic:cli topic:terminal` returns only repos +# tagged BOTH — a smaller or empty set, exit 0, +# no error anywhere. +# +# Therefore: all languages collapse into one query, and each topic needs its own. +# Query count is |topics|. Adding a language adds zero queries. Emitting two +# `topic:` qualifiers on one line is the bug every guard in this file exists to +# prevent. +# +# No LLM in this path. Same file in, same bytes out, every run — a model in the +# query path cannot promise that, cannot be tested by a network-free CI, and +# turns a misparse into a confidently wrong scan with exit 0. Prose compiles to +# filters in /preferences, at authoring, with a human confirming, or not at all. +# +# Exit: 0 ok, 10 malformed preferences (from parse_preferences) or a query that +# fails its own guards. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=../lib/preferences.sh +. "$HERE/../lib/preferences.sh" + +die() { echo "FATAL: $*" >&2; exit 10; } + +# Today's queries, lifted verbatim from the agent's DEFAULT_QUERIES. A fresh +# machine with no preferences.md must scan exactly as it does today, so +# /contribute, /contribute-loop and /contribution-fleet keep working untouched. +# +# Union semantics on purpose: the topic queries carry no language and the +# language query carries no topic. Generating this from the strict +# `language AND topic` shape would silently drop a Rust ML repo and an untagged +# Java repo from a fresh machine's candidate set — a composition change for users +# who never asked for one. The strict shape is opt-in: it applies the moment +# someone writes a preferences.md, and not before. +DEFAULT_PROFILE="topic:machine-learning stars:>20000 archived:false +topic:llm stars:>5000 archived:false +topic:artificial-intelligence stars:>20000 archived:false +topic:developer-tools stars:>20000 archived:false +language:java language:python stars:>20000 archived:false" + +PREFS="" +NUDGE=1 +while [ $# -gt 0 ]; do + case "$1" in + --file) [ $# -ge 2 ] || die "--file needs a value"; PREFS="$2"; shift 2 ;; + --no-nudge) NUDGE=0; shift ;; + *) die "unknown argument: $1" ;; + esac +done +[ -n "$PREFS" ] || PREFS="$(prefs_path)" +command -v jq >/dev/null 2>&1 || die "jq is required" + +# Every emitted line passes this. The guards are here, not only in the tests, +# because each failure they catch is silent at the API: a typo'd qualifier key +# and a parenthesized OR both return total_count: 0 with exit 0, and a second +# `topic:` returns a quietly intersected set. Nothing downstream can tell any of +# them apart from a thin week for open source. +assert_query() { + local q="$1" tok key topics=0 langs=0 + + case "$q" in + *" OR "*|*" AND "*|*"("*|*")"*) + die "boolean operator or paren in query: '$q' (GitHub reads these as free text)" ;; + *"-language:"*|*"-topic:"*) + die "negated qualifier in query: '$q' (exclusion was considered and rejected)" ;; + *".."*|*"stars:<"*) + die "star ceiling in query: '$q' (stars is a floor; there is no ceiling)" ;; + esac + + for tok in $q; do + case "$tok" in + *:*) key="${tok%%:*}" ;; + *) die "bare word in query: '$tok' in '$q' (free text is not a filter)" ;; + esac + case "$key" in + language) langs=$((langs + 1)) ;; + topic) topics=$((topics + 1)) ;; + stars|archived) ;; + *) die "unknown qualifier key '$key' in '$q' (allowed: language, topic, stars, archived)" ;; + esac + done + + # A second topic: intersects rather than unions. + [ "$topics" -le 1 ] || die "more than one topic: qualifier in '$q' — topics AND, they do not OR" + # A query with neither is a catch-all: it admits repos matching no criterion. + [ $((langs + topics)) -ge 1 ] || die "catch-all query (no language:, no topic:): '$q'" +} + +emit() { + local q="$1" + assert_query "$q" + printf '%s\n' "$q" +} + +# Shown once per install, not every run: a nag you see ten times is a nag you +# stop reading. last_version.json already exists for exactly this class of +# once-per-install lifecycle state, so no new file. +nudge_once() { + [ "$NUDGE" -eq 1 ] || return 0 + local dir="$HOME/.superhuman/global" f tmp + f="$dir/last_version.json" + if [ -f "$f" ] && [ "$(jq -r '.preferences_nudged // false' "$f" 2>/dev/null)" = "true" ]; then + return 0 + fi + echo "Using default filters (ai-ml, java, python, devtools · >20K stars; llm >5K)." >&2 + echo "Run /preferences to set your own — takes about a minute." >&2 + mkdir -p "$dir" + tmp="$f.tmp.$$" + if [ -f "$f" ]; then + jq '.preferences_nudged = true' "$f" > "$tmp" 2>/dev/null && mv "$tmp" "$f" || rm -f "$tmp" + else + jq -n '{preferences_nudged: true}' > "$tmp" && mv "$tmp" "$f" || rm -f "$tmp" + fi +} + +if [ ! -f "$PREFS" ]; then + nudge_once + while IFS= read -r q; do + [ -n "$q" ] && emit "$q" + done <\t". `any` +# is reserved and compiles to the ABSENCE of a topic: qualifier — never to a +# literal topic:any. The topic goes last precisely because it can be empty: as a +# leading space-separated field it is swallowed by word splitting, and the `any` +# query vanishes with no error. +# +# A single loop, no nested loops, no `declare -A`, no `mapfile`: bash 3.2 is what +# ships on macOS, and those pass on the ubuntu runner and die there. +TAB=$(printf '\t') +TOPIC_SPEC=$(printf '%s' "$PARSED" | jq -r --argjson g "$GLOBAL_MIN" ' + .topics[] | "\(.min_stars // $g)\t\(if .topic == "any" then "" else .topic end)"') + +while IFS="$TAB" read -r min topic; do + [ -n "$min" ] || continue + Q="$LANG_PREFIX" + if [ -n "$topic" ]; then Q="${Q:+$Q }topic:$topic"; fi + Q="${Q:+$Q }stars:>$min archived:false" + emit "$Q" +done </dev/null 2>&1 || { + echo "curated_list_filter.sh: CONFIG ERROR: jq not found on PATH." >&2 + exit "$EX_CONFIG" +} + +CANDIDATE=$(cat) +[ -n "$CANDIDATE" ] || { + echo "curated_list_filter.sh: CONFIG ERROR: empty candidate on stdin." >&2 + exit "$EX_CONFIG" +} +printf '%s' "$CANDIDATE" | jq -e . >/dev/null 2>&1 || { + echo "curated_list_filter.sh: CONFIG ERROR: candidate is not valid JSON." >&2 + exit "$EX_CONFIG" +} + +NAME=$(printf '%s' "$CANDIDATE" \ + | jq -r '((.name // (.full_name // "" | split("/") | last)) // "") | ascii_downcase') +LANG=$(printf '%s' "$CANDIDATE" | jq -r '(.language // "") | ascii_downcase') +TOPICS=$(printf '%s' "$CANDIDATE" | jq -r '(.topics // [])[] | ascii_downcase') + +# Sufficient alone. +# +# Pedagogy topics are strong, not weak: a repo tagged `tutorial` or `course` exists +# to be read, not patched, and a course carries exactly one such tag, so a two-weak +# bar never fires on it. No real library on a go/python scan carries any of them. +STRONG_TOPICS='awesome awesome-list awesome-lists curated-list cheatsheet cheatsheets interview-questions roadmap free-programming-books tutorial tutorials course courses educational teaching learning-path study-plan' +# Each of these also tags real code, so two are required. +WEAK_TOPICS='list lists resources resource books book learning education guide guides collection from-scratch' + +strong="" +weak=0 + +# A repo with no primary language has no code to contribute to: GitHub reports +# `language: null` for a tree of Markdown, which is precisely what a list is. +# +# Dead whenever the profile pins `languages:` — every candidate then arrives from a +# query carrying `language:go language:python`, so `.language` cannot be null by +# construction. It still fires on a profile with no `languages:` line. Do not treat +# it as this filter's backbone: on a language-pinned scan the topic signals are. +case "$LANG" in + ''|null|markdown) strong="language=${LANG:-null}" ;; +esac + +case "$NAME" in + awesome|awesome-*|awesome_*) strong="${strong:-name=$NAME}" ;; +esac + +# Read topics line-by-line: zsh does not word-split an unquoted expansion, so +# `for t in $TOPICS` would iterate once over the whole blob. +while IFS= read -r t; do + [ -n "$t" ] || continue + for s in $STRONG_TOPICS; do + [ "$t" = "$s" ] && strong="${strong:-topic=$t}" + done + for w in $WEAK_TOPICS; do + [ "$t" = "$w" ] && weak=$((weak + 1)) + done +done <&2 + exit 1 +fi +if [ "$weak" -ge 2 ]; then + echo "SKIP: curated list, not a code repo ($weak weak list signals in topics)" >&2 + exit 1 +fi + +exit 0 diff --git a/skills/superhuman/SKILL.md b/skills/superhuman/SKILL.md index 65ae216..f3dfee6 100644 --- a/skills/superhuman/SKILL.md +++ b/skills/superhuman/SKILL.md @@ -33,13 +33,16 @@ Entry points: - Full contribution: read `agents/opensource-contributor.md` and execute the phase sequence inline. -- Repo discovery: read `agents/repo-finder.md`. +- Repo discovery: read `agents/repo-finder.md`. Its search filters come from + `~/.superhuman/preferences.md`, compiled by `scripts/repo-finder/build_queries.sh` + — never hardcode a language or topic into a query. - Issue selection, profiling, planning, building, scoring, review dispatch, and comment resolution: read the corresponding file in `agents/`. - Claude Code slash command requests such as `/contribute`, `/repo-finder`, - `/contribute-loop`, `/contribution-fleet`, and `/contribution-dashboard`: - read the matching file under `commands/`, then translate any `Agent(...)` - calls into inline execution of the referenced agent contract. + `/preferences`, `/contribute-loop`, `/contribution-fleet`, and + `/contribution-dashboard`: read the matching file under `commands/`, then + translate any `Agent(...)` calls into inline execution of the referenced agent + contract. ## State And Safety diff --git a/tests/fixtures/default_queries.txt b/tests/fixtures/default_queries.txt new file mode 100644 index 0000000..aa0262f --- /dev/null +++ b/tests/fixtures/default_queries.txt @@ -0,0 +1,5 @@ +topic:machine-learning stars:>20000 archived:false +topic:llm stars:>5000 archived:false +topic:artificial-intelligence stars:>20000 archived:false +topic:developer-tools stars:>20000 archived:false +language:java language:python stars:>20000 archived:false diff --git a/tests/scripts/test_build_queries.sh b/tests/scripts/test_build_queries.sh new file mode 100644 index 0000000..20f1adf --- /dev/null +++ b/tests/scripts/test_build_queries.sh @@ -0,0 +1,233 @@ +#!/usr/bin/env bash +# build_queries.sh: preferences -> GitHub search queries. +# +# Every assertion here guards a failure that GitHub reports as success. A second +# `topic:` silently intersects; a typo'd qualifier key becomes free text; a +# parenthesized OR returns total_count: 0. All of them exit 0 and read as a thin +# week for open source. The API will never tell you — these tests are the only +# thing that will. +set -uo pipefail +CLAUDE_PLUGIN_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +export CLAUDE_PLUGIN_ROOT +BUILD="$CLAUDE_PLUGIN_ROOT/scripts/repo-finder/build_queries.sh" +GOLDEN="$CLAUDE_PLUGIN_ROOT/tests/fixtures/default_queries.txt" + +tmpdir=$(mktemp -d) +trap 'rm -rf "$tmpdir"' EXIT + +# Fake HOME: the default path writes the once-per-install nudge flag into +# ~/.superhuman/global/last_version.json, and a test must never touch the real one. +HOME="$tmpdir/home"; export HOME +mkdir -p "$HOME" + +fail() { echo "FAIL: $*"; exit 1; } + +write() { cat > "$tmpdir/$1.md"; } +run() { bash "$BUILD" --file "$tmpdir/$1.md" --no-nudge 2>/dev/null; } +count() { run "$1" | grep -c . ; } + +# Never `run x | grep -q`: under `set -o pipefail`, grep -q exits on the first +# match, SIGPIPEs the builder, and the pipeline reports 141 — a MATCH reads as a +# failure. Match against a captured string instead. +has() { grep -q "$2" < "$tmpdir/cross.txt" +while IFS= read -r q; do + n=$(printf '%s\n' "$q" | tr ' ' '\n' | grep -c '^topic:') + [ "$n" -le 1 ] || fail "$n topic: qualifiers in one query: $q" +done < "$tmpdir/cross.txt" + +# 4. Query count == |topics|. Adding a language must add ZERO queries — if the +# count scales with language count, the OR collapse was lost and the cross +# product is back. +write cross3 <<'EOF' +## Filters +languages: go, rust, python +topics: backend, infrastructure +stars: 2000 +EOF +[ "$(count cross3)" = "2" ] \ + || fail "a third language added queries: $(count cross3) (languages are free)" + +# 5. Every query carries the full language union. +run cross3 > "$tmpdir/cross3.txt" +while IFS= read -r q; do + for l in go rust python; do + case "$q" in *"language:$l"*) ;; *) fail "language:$l missing from: $q" ;; esac + done +done < "$tmpdir/cross3.txt" + +# 9c. `any` emits a TOPIC-LESS query, never a literal topic:any (GitHub has a +# real `any` topic; topics: any will never mean it). +write anytopic <<'EOF' +## Filters +languages: go +topics: backend, any +stars: 2000 +EOF +[ "$(count anytopic)" = "2" ] || fail "any: want 2 queries, got $(count anytopic)" +A=$(run anytopic) +has "$A" '^language:go stars:>2000 archived:false$' \ + || fail "any must emit a query with no topic: qualifier" +hasnt "$A" 'topic:any' || fail "literal topic:any emitted" + +# 9b. An omitted topics: line IS topics: any — one mechanism, not two rules. +write omitted <<'EOF' +## Filters +languages: go +stars: 2000 +EOF +write explicit <<'EOF' +## Filters +languages: go +topics: any +stars: 2000 +EOF +[ "$(run omitted)" = "$(run explicit)" ] \ + || fail "omitted topics: must be byte-identical to 'topics: any'" +[ "$(count omitted)" = "1" ] || fail "omitted topics: want 1 query, got $(count omitted)" + +# 9d. A per-entry floor overrides the global floor, for its own entry and no other. +write floors <<'EOF' +## Filters +languages: go +topics: backend, llm:5000 +stars: 20000 +EOF +F=$(run floors) +has "$F" '^language:go topic:backend stars:>20000 archived:false$' \ + || fail "global floor not applied to backend" +has "$F" '^language:go topic:llm stars:>5000 archived:false$' \ + || fail "per-entry floor not applied to llm" + +# 9e. No LLM in the scan path. A ## Notes sentence must not alter a single query — +# that compile happens in /preferences, at authoring, with a human confirming. +write prose <<'EOF' +## Filters +languages: go +topics: backend +stars: 2000 + +## Notes +I need llm repos having >5k stars, and rust too. +EOF +[ "$(run prose)" = "$(run go_backend)" ] \ + || fail "a Notes sentence changed the emitted queries" + +# --------------------------------------------------------------------------- +# Invariants that hold for every emitted query, on every profile. +# --------------------------------------------------------------------------- +ALL="$tmpdir/all.txt" +{ run cross3; run anytopic; run floors + bash "$BUILD" --file "$tmpdir/none.md" --no-nudge 2>/dev/null; } > "$ALL" + +# 6. Qualifier-key allowlist. A typo'd key (langauge:go) is parsed as free text +# and returns total_count: 0 with exit 0 — an allowlist is the only thing that +# catches this class. +while IFS= read -r q; do + for tok in $q; do + case "${tok%%:*}" in + language|topic|stars|archived) ;; + *) fail "qualifier key outside the allowlist: '$tok' in: $q" ;; + esac + done +done < "$ALL" + +# 7. No catch-all: every query carries a language: or a topic:. +while IFS= read -r q; do + case "$q" in + *language:*|*topic:*) ;; + *) fail "catch-all query (no language:, no topic:): $q" ;; + esac +done < "$ALL" + +# 8. No boolean operators. Repeated `language:` is NOT an operator — it is the +# sanctioned union mechanism and must not trip this check. +grep -qE ' (OR|AND) |[()]' "$ALL" && fail "boolean operator or paren in an emitted query" + +# 9f. No negation. Exclusion was considered and rejected; this pins it. +grep -qE '\-(language|topic):' "$ALL" && fail "negated qualifier emitted" + +# 14. Stars are floor-only. No ceiling exists anywhere: no `..`, no `stars:<`. +grep -qE '\.\.|stars:<' "$ALL" && fail "star ceiling emitted" +grep -q 'stars:>' "$ALL" || fail "no star floor emitted at all" + +# --------------------------------------------------------------------------- +# The default path is a pure refactor. +# --------------------------------------------------------------------------- + +# 9. No preferences file => DEFAULT_PROFILE emits byte-identical queries to the +# five the agent carried before this change. /contribute on a fresh machine +# cannot regress, because the queries are literally the same bytes. +bash "$BUILD" --file "$tmpdir/none.md" --no-nudge > "$tmpdir/default.txt" 2>/dev/null +diff -u "$GOLDEN" "$tmpdir/default.txt" || fail "default queries drifted from the golden list" + +# 9a. The default keeps UNION semantics: no default query carries both a +# language: and a topic:. If one does, the strict shape has leaked into the +# default path and fresh-machine behavior has silently changed. +while IFS= read -r q; do + case "$q" in + *language:*) case "$q" in *topic:*) fail "strict shape leaked into the default: $q" ;; esac ;; + esac +done < "$tmpdir/default.txt" + +# 10. The nudge fires once per install, not every run. A nag you see ten times is +# a nag you stop reading. +rm -rf "$HOME/.superhuman" +N1=$(bash "$BUILD" --file "$tmpdir/none.md" 2>&1 >/dev/null) +N2=$(bash "$BUILD" --file "$tmpdir/none.md" 2>&1 >/dev/null) +case "$N1" in *"/preferences"*) ;; *) fail "first default run did not nudge" ;; esac +[ -z "$N2" ] || fail "the nudge re-fired on a second run: $N2" +[ "$(jq -r '.preferences_nudged' "$HOME/.superhuman/global/last_version.json")" = "true" ] \ + || fail "nudge flag not persisted to last_version.json" + +# The nudge must never contaminate the query list: it goes to stderr. +bash "$BUILD" --file "$tmpdir/none.md" 2>/dev/null | diff -q - "$GOLDEN" >/dev/null \ + || fail "nudge text leaked into stdout" + +# --------------------------------------------------------------------------- +# Malformed preferences abort. They never degrade to an unfiltered scan. +# --------------------------------------------------------------------------- +write bad <<'EOF' +## Filters +langauges: go +EOF +bash "$BUILD" --file "$tmpdir/bad.md" --no-nudge >/dev/null 2>&1 +rc=$? +[ "$rc" = "10" ] || fail "malformed preferences must exit 10, got $rc (never fall back to defaults)" + +# 11. Determinism: same profile, two runs, byte-identical output. +[ "$(run cross3)" = "$(run cross3)" ] || fail "build is not deterministic" + +echo "OK test_build_queries.sh" diff --git a/tests/scripts/test_curated_list_filter.sh b/tests/scripts/test_curated_list_filter.sh new file mode 100755 index 0000000..513013f --- /dev/null +++ b/tests/scripts/test_curated_list_filter.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Network-free, bash 3.2-clean. Gates on ubuntu and macOS alike. +set -euo pipefail + +ROOT="${CLAUDE_PLUGIN_ROOT:-$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd)}" +F="$ROOT/scripts/repo-finder/curated_list_filter.sh" + +keep() { # keep