perf: repo-finder burns ~59-109 API calls per scan for data it already fetched

agents/repo-finder.md

@@ -29,35 +29,81 @@ Search in this priority order. Spend more effort on higher-priority categories:
 
 ### Step 1: Build Search Queries
 
-Run GitHub searches across each category. Use `gh` CLI for all API access.
-
-The repo slug field is `fullName`. `gh search repos` rejects `nameWithOwner` with
-`Unknown JSON field` — that field belongs to `gh repo` and the GraphQL schema, not
-to the search command. Passing it fails every query in this step, so the scan ends
-with zero candidates and writes an empty shortlist that reads as a clean run.
+Use `gh api search/repositories`. Do not use `gh search repos`: it cannot return
+`topics[]`, which the shortlist requires. Every other required field ships inline
+in the same response.
+
+#### Query shape rules
+
+Repeated qualifiers do not behave alike:
+
+| Qualifier | Repeated behavior |
+|---|---|
+| `language:` | OR — `language:go language:rust` returns Go ∪ Rust |
+| `topic:` | AND — `topic:cli topic:terminal` returns only repos tagged both |
+
+- Union languages into a single query. Multi-language costs nothing.
+- **Emit at most one `topic:` per query.** A second `topic:` intersects rather
+  than unions, returning a smaller or empty set with no error.
+- Query count is `|topics|`, not `|langs| × |topics|`.
+- Every `key:` token must be one of `{language, topic, stars, archived}`. An
+  unrecognized qualifier (`langauge:go`) is parsed as free text and returns
+  `total_count: 0` with exit 0.
+- Every query must carry a `language:` or a `topic:`. A query with neither is a
+  catch-all that admits repos matching no criterion.
+- Sort by `stars`, not `updated`. `updated` reorders hourly, making the per-query
+  cutoff non-deterministic.
+- Pass `archived:false` as a qualifier so archived repos never consume a result
+  slot.
 
 ```bash
-# Category 1: AI/ML
-gh search repos --topic machine-learning --stars ">20000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
-gh search repos --topic llm --stars ">5000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
-gh search repos --topic artificial-intelligence --stars ">20000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
-
-# Category 2: Language-specific
-gh search repos --language java --stars ">20000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
-gh search repos --language python --stars ">20000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
-
-# Category 3: Framework-specific
-gh search repos "spring boot" --stars ">10000" --sort updated --limit 10 --json fullName,stargazersCount,updatedAt,description
-gh search repos "fastapi" --stars ">5000" --sort updated --limit 10 --json fullName,stargazersCount,updatedAt,description
-
-# Category 4: Famous tools
-gh search repos --topic developer-tools --stars ">20000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
+DEFAULT_QUERIES="
+topic:machine-learning        stars:>20000 archived:false
+topic:llm                     stars:>5000  archived:false
+topic:artificial-intelligence stars:>20000 archived:false
+topic:developer-tools         stars:>20000 archived:false
+language:java language:python stars:>20000 archived:false
+"
+
+# Feed the loop by heredoc, not `echo ... | while`. A piped while runs in a
+# subshell, where `exit` terminates only the subshell and the scan continues past
+# a fatal error.
+CANDIDATES=""
+while IFS= read -r q; do
+  [ -z "$q" ] && continue
+
+  # An error is not a verdict. A failed search must abort, never contribute zero
+  # rows to a scan that then reads as clean.
+  RESP=$(gh api -X GET search/repositories \
+    -f q="$q" -f sort=stars -f order=desc -f per_page=50) \
+    || { echo "FATAL: search failed for query: $q" >&2; exit 1; }
+
+  # Field-name drift against the API yields null, not an error. Abort; do not skip
+  # the row. A partial candidate set is indistinguishable from a clean scan.
+  if printf '%s' "$RESP" | jq -e '.items | any(.full_name == null)' >/dev/null; then
+    echo "FATAL: null full_name in search response — projection drifted: $q" >&2
+    exit 1
+  fi
 
-# Category 5: Other active repos (catch-all)
-gh search repos --stars ">20000" --sort updated --limit 20 --json fullName,stargazersCount,updatedAt,description
+  # Report matched-of-total from the same response. Never widen a query to fill
+  # the list.
+  echo "$(printf '%s' "$RESP" | jq -r '.items | length') of \
+$(printf '%s' "$RESP" | jq -r '.total_count') matched: $q" >&2
+
+  CANDIDATES="$CANDIDATES$(printf '%s' "$RESP" | jq -c '.items[] |
+    {full_name, language, topics, stargazers_count, pushed_at,
+     archived, open_issues_count, default_branch, description}')
+"
+done <<EOF
+$DEFAULT_QUERIES
+EOF
 ```
 
-Deduplicate across all queries. You should have 50-100 unique repos after dedup.
+Deduplicate across all queries. Expect 50-100 unique repos after dedup.
+
+Carry the full projected record forward. `topics[]` reaches the shortlist row;
+`pushed_at`, `archived`, `open_issues_count`, and `default_branch` are read by
+Step 2 from the record in hand. Step 2 does not re-fetch them.
 
 ### Step 2: Fast Filter (eliminate obvious non-starters)
 
@@ -96,15 +142,25 @@ candidate in the loop below. Skipped repos do not consume API rate limit
 on the rest of this step.
 
 For each repo that passes the reputation gate, run these quick checks.
-Skip repos that fail any:
+`$CANDIDATE` is that repo's projected record from Step 1 — the full JSON object,
+not just the slug. Skip repos that fail any check:
 
 ```bash
-# One repo call, not three.
-read -r PUSHED_AT OPEN_ISSUES ARCHIVED DEFAULT_BRANCH < <(
-  gh api repos/OWNER/REPO --jq '[.pushed_at, .open_issues_count, .archived, .default_branch] | @tsv')
-# Skip if pushed_at older than 30 days, open_issues_count is 0, or archived.
-
-# Skip if no merged PRs in the last 30 days.
+# Read from the Step 1 record. Do not re-fetch: `gh api repos/OWNER/REPO` here
+# costs one request per candidate, 50-100 per scan, for fields already in hand.
+PUSHED_AT=$(printf '%s' "$CANDIDATE" | jq -r '.pushed_at')
+OPEN_ISSUES=$(printf '%s' "$CANDIDATE" | jq -r '.open_issues_count')
+DEFAULT_BRANCH=$(printf '%s' "$CANDIDATE" | jq -r '.default_branch')
+
+# Skip if pushed_at is older than 30 days, or open_issues_count is 0.
+# No archived check: `archived:false` is a Step 1 qualifier, so archived repos
+# never enter the candidate set.
+# The search index lags the repo endpoint by minutes to hours. That is within
+# tolerance for a 30-day gate. A check needing to-the-minute accuracy must make
+# its own call.
+
+# Skip if no merged PRs in the last 30 days. No search result carries this signal,
+# so it keeps its own call: one per candidate.
 #
 # Ask the search index directly (`merged:>=YYYY-MM-DD`). Do NOT fetch `--state
 # merged --limit N` and filter on mergedAt: that list is ordered by CREATION
@@ -545,11 +601,30 @@ If no issue scores 8+ after applying the hard filter, mark the repo as "no clear
 Save results to `~/.superhuman/global/repo-shortlist.json`
 (create the directory if missing; atomic temp-rename write):
 
+Validate against `schemas/repo_shortlist.schema.json` before the rename. A
+malformed shortlist must never reach the orchestrator.
+
 ```bash
+source "${CLAUDE_PLUGIN_ROOT}/scripts/lib/state.sh"
+
 GLOBAL_DIR="$HOME/.superhuman/global"
 mkdir -p "$GLOBAL_DIR"
 TMP="$GLOBAL_DIR/repo-shortlist.json.tmp.$$"
-printf '%s' "$SHORTLIST" | jq . > "$TMP" && mv "$TMP" "$GLOBAL_DIR/repo-shortlist.json"
+printf '%s' "$SHORTLIST" | jq . > "$TMP" \
+  || { echo "FATAL: shortlist is not valid JSON" >&2; rm -f "$TMP"; exit 1; }
+
+validate_json "${CLAUDE_PLUGIN_ROOT}/schemas/repo_shortlist.schema.json" "$TMP" \
+  || { echo "FATAL: shortlist failed schema validation" >&2; rm -f "$TMP"; exit 1; }
+
+# Row invariant, asserted separately: validate_json degrades to a top-level
+# required-key check when python jsonschema is absent (stock macOS runner), and
+# `repo`/`scores.final` are nested inside repos[]. A null `repo` (dropped
+# full_name -> repo mapping) or null score (written to `.score`) survives that
+# fallback and propagates downstream as a valid-looking value.
+jq -e '.repos | length > 0 and all(.repo != null and .scores.final != null)' "$TMP" >/dev/null \
+  || { echo "FATAL: shortlist row missing repo or scores.final" >&2; rm -f "$TMP"; exit 1; }
+
+mv "$TMP" "$GLOBAL_DIR/repo-shortlist.json"
 ```
 
 The orchestrator binds the top result via:
@@ -558,7 +633,9 @@ The orchestrator binds the top result via:
 REPO=$(jq -r '.repos[0].repo' "$GLOBAL_DIR/repo-shortlist.json")
 ```
 
-Shortlist payload shape:
+Shortlist payload shape. The row key is `repo`, not `full_name`: map Step 1's
+`full_name` across on write, or the orchestrator's `.repos[0].repo` binds null.
+`topics[]` comes from the Step 1 projection; cite it in the `notes` rationale.
 
 ```json
 {
@@ -575,6 +652,7 @@ Shortlist payload shape:
       "stars": 45000,
       "category": "ai-ml",
       "language": "Python",
+      "topics": ["machine-learning", "llm", "python"],
       "description": "repo description",
       "scores": {
         "responsiveness": 9,

commands/repo-finder.md

@@ -58,7 +58,7 @@ if [ ! -f "$SHORTLIST" ]; then
 fi
 
 echo "Top $N candidates:"
-jq -r '.repos[] | "\(.score)\t\(.repo)\t\(.notes // "")"' "$SHORTLIST" \
+jq -r '.repos[] | "\(.scores.final)\t\(.repo)\t\(.notes // "")"' "$SHORTLIST" \
   | head -"$N" \
   | awk -F'\t' 'BEGIN{printf "%-6s %-32s %s\n","score","repo","notes"}
                 {printf "%-6s %-32s %s\n",$1,$2,$3}'

schemas/repo_shortlist.schema.json

@@ -0,0 +1,64 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/gaurav0107/superhuman/schemas/repo_shortlist.schema.json",
+  "title": "repo_shortlist",
+  "description": "Ranked repos worth contributing to, with the single best issue candidate per repo. Consumed by opensource-contributor and the contribute/contribution-fleet commands, which bind a target via .repos[0].repo. Owner (sole writer): repo-finder.",
+  "type": "object",
+  "additionalProperties": true,
+  "required": ["generated_at", "repos"],
+  "properties": {
+    "generated_at": {"type": "string", "format": "date-time"},
+    "criteria": {
+      "type": "object",
+      "additionalProperties": true,
+      "description": "the filter that produced this list, e.g. {\"min_stars\":20000,\"min_score\":60}"
+    },
+    "repos": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "additionalProperties": true,
+        "required": ["repo", "scores"],
+        "properties": {
+          "rank":        {"type": "integer", "minimum": 1},
+          "repo":        {"type": "string", "pattern": "^[^/]+/[^/]+$", "description": "owner/repo slug. Search returns this field as full_name; map it across on write or .repos[0].repo binds null downstream."},
+          "stars":       {"type": "integer", "minimum": 0},
+          "category":    {"type": "string"},
+          "language":    {"type": ["string", "null"], "description": "null for repos GitHub cannot classify"},
+          "topics":      {"type": "array", "items": {"type": "string"}, "description": "from the Step 1 search projection; cited by the notes rationale"},
+          "description": {"type": ["string", "null"]},
+          "scores": {
+            "type": "object",
+            "additionalProperties": true,
+            "required": ["final"],
+            "description": "commands/repo-finder.md renders scores.final. A score written to a bare .score key renders as null on every row.",
+            "properties": {
+              "responsiveness":            {"type": "number"},
+              "opportunity_quality":       {"type": "number"},
+              "outside_contributor_track": {"type": "number"},
+              "ai_friendliness":           {"type": "number"},
+              "category_bonus":            {"type": "number"},
+              "final":                     {"type": "number"}
+            }
+          },
+          "responsiveness_detail": {"type": "object", "additionalProperties": true},
+          "best_issue": {
+            "type": "object",
+            "additionalProperties": true,
+            "required": ["number", "url"],
+            "properties": {
+              "number":                  {"type": "integer"},
+              "title":                   {"type": "string"},
+              "type":                    {"type": "string"},
+              "labels":                  {"type": "array", "items": {"type": "string"}},
+              "has_maintainer_approval": {"type": "boolean"},
+              "issue_score":             {"type": "number"},
+              "url":                     {"type": "string"}
+            }
+          },
+          "notes": {"type": "string"}
+        }
+      }
+    }
+  }
+}