-
Notifications
You must be signed in to change notification settings - Fork 1
perf: repo-finder burns ~59-109 API calls per scan for data it already fetched #28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
gaurav0107
merged 4 commits into
gaurav0107:main
from
imkp1:feat/repo-finder-query-shape
Jul 13, 2026
+336
−34
Merged
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
9098f3b
perf(repo-finder): cut per-scan API calls from ~59-109 to 5 + 1/candi…
imkp1 dd9ae6b
fix(repo-finder): implement the Step 1 guards the prose already promised
imkp1 7f103ee
test(repo-finder): execute the Step 1 guards instead of grepping for …
imkp1 0b05f34
test(repo-finder): close two holes in the Step 1 regression checks
imkp1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "$id": "https://github.com/gaurav0107/superhuman/schemas/repo_shortlist.schema.json", | ||
| "title": "repo_shortlist", | ||
| "description": "Ranked repos worth contributing to, with the single best issue candidate per repo. Consumed by opensource-contributor and the contribute/contribution-fleet commands, which bind a target via .repos[0].repo. Owner (sole writer): repo-finder.", | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "required": ["generated_at", "repos"], | ||
| "properties": { | ||
| "generated_at": {"type": "string", "format": "date-time"}, | ||
| "criteria": { | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "description": "the filter that produced this list, e.g. {\"min_stars\":20000,\"min_score\":60}" | ||
| }, | ||
| "repos": { | ||
| "type": "array", | ||
| "items": { | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "required": ["repo", "scores"], | ||
| "properties": { | ||
| "rank": {"type": "integer", "minimum": 1}, | ||
| "repo": {"type": "string", "pattern": "^[^/]+/[^/]+$", "description": "owner/repo slug. Search returns this field as full_name; map it across on write or .repos[0].repo binds null downstream."}, | ||
| "stars": {"type": "integer", "minimum": 0}, | ||
| "category": {"type": "string"}, | ||
| "language": {"type": ["string", "null"], "description": "null for repos GitHub cannot classify"}, | ||
| "topics": {"type": "array", "items": {"type": "string"}, "description": "from the Step 1 search projection; cited by the notes rationale"}, | ||
| "description": {"type": ["string", "null"]}, | ||
| "scores": { | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "required": ["final"], | ||
| "description": "commands/repo-finder.md renders scores.final. A score written to a bare .score key renders as null on every row.", | ||
| "properties": { | ||
| "responsiveness": {"type": "number"}, | ||
| "opportunity_quality": {"type": "number"}, | ||
| "outside_contributor_track": {"type": "number"}, | ||
| "ai_friendliness": {"type": "number"}, | ||
| "category_bonus": {"type": "number"}, | ||
| "final": {"type": "number"} | ||
| } | ||
| }, | ||
| "responsiveness_detail": {"type": "object", "additionalProperties": true}, | ||
| "best_issue": { | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "required": ["number", "url"], | ||
| "properties": { | ||
| "number": {"type": "integer"}, | ||
| "title": {"type": "string"}, | ||
| "type": {"type": "string"}, | ||
| "labels": {"type": "array", "items": {"type": "string"}}, | ||
| "has_maintainer_approval": {"type": "boolean"}, | ||
| "issue_score": {"type": "number"}, | ||
| "url": {"type": "string"} | ||
| } | ||
| }, | ||
| "notes": {"type": "string"} | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| CLAUDE_PLUGIN_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" | ||
| export CLAUDE_PLUGIN_ROOT | ||
| source "$CLAUDE_PLUGIN_ROOT/scripts/lib/state.sh" | ||
| SCHEMA="$CLAUDE_PLUGIN_ROOT/schemas/repo_shortlist.schema.json" | ||
| tmpdir=$(mktemp -d) | ||
| trap 'rm -rf "$tmpdir"' EXIT | ||
|
|
||
| cat > "$tmpdir/ok.json" <<'EOF' | ||
| { | ||
| "generated_at": "2026-07-13T00:00:00Z", | ||
| "criteria": {"min_stars": 20000, "min_score": 60}, | ||
| "repos": [ | ||
| { | ||
| "rank": 1, | ||
| "repo": "owner/repo", | ||
| "stars": 45000, | ||
| "category": "ai-ml", | ||
| "language": "Python", | ||
| "topics": ["machine-learning", "llm"], | ||
| "description": "a repo", | ||
| "scores": {"responsiveness": 9, "final": 87}, | ||
| "best_issue": {"number": 1234, "url": "https://github.com/owner/repo/issues/1234"}, | ||
| "notes": "tagged llm, responsive maintainers" | ||
| } | ||
| ] | ||
| } | ||
| EOF | ||
| validate_json "$SCHEMA" "$tmpdir/ok.json" || { echo "FAIL valid repo_shortlist rejected"; exit 1; } | ||
|
|
||
| echo '{"generated_at":"2026-07-13T00:00:00Z"}' > "$tmpdir/no_repos.json" | ||
| validate_json "$SCHEMA" "$tmpdir/no_repos.json" 2>/dev/null && { | ||
| echo "FAIL repo_shortlist missing repos accepted"; exit 1; } | ||
|
|
||
| # validate_json degrades to a top-level required-key check when python jsonschema | ||
| # is absent, and repo/scores.final are nested inside repos[]. agents/repo-finder.md | ||
| # Step 5 asserts them with jq at write time. Keep this guard in lockstep with it. | ||
| row_guard() { # row_guard <file> -> 0 if every row has repo + scores.final | ||
| jq -e '.repos | length > 0 and all(.repo != null and .scores.final != null)' "$1" >/dev/null | ||
| } | ||
|
|
||
| row_guard "$tmpdir/ok.json" || { echo "FAIL row guard rejected a valid shortlist"; exit 1; } | ||
|
|
||
| # full_name -> repo mapping dropped on write. | ||
| cat > "$tmpdir/null_repo.json" <<'EOF' | ||
| {"generated_at":"2026-07-13T00:00:00Z","repos":[{"repo":null,"scores":{"final":87}}]} | ||
| EOF | ||
| row_guard "$tmpdir/null_repo.json" 2>/dev/null && { echo "FAIL null repo accepted"; exit 1; } | ||
|
|
||
| # Score written to a bare .score key instead of .scores.final. | ||
| cat > "$tmpdir/bare_score.json" <<'EOF' | ||
| {"generated_at":"2026-07-13T00:00:00Z","repos":[{"repo":"owner/repo","score":87,"scores":{}}]} | ||
| EOF | ||
| row_guard "$tmpdir/bare_score.json" 2>/dev/null && { echo "FAIL bare .score accepted"; exit 1; } | ||
|
|
||
| # commands/repo-finder.md must render scores.final, not score. | ||
| RENDERED=$(jq -r '.repos[] | "\(.scores.final)\t\(.repo)"' "$tmpdir/ok.json") | ||
| case "$RENDERED" in | ||
| "87 owner/repo") ;; | ||
| *) echo "FAIL render projection: got '$RENDERED'"; exit 1 ;; | ||
| esac | ||
| grep -q 'scores\.final' "$CLAUDE_PLUGIN_ROOT/commands/repo-finder.md" || { | ||
| echo "FAIL commands/repo-finder.md does not render .scores.final"; exit 1; } | ||
|
|
||
| AGENT="$CLAUDE_PLUGIN_ROOT/agents/repo-finder.md" | ||
|
|
||
| # Step 1's guards are executed, not grepped: a literal-text assertion is satisfied | ||
| # by prose or a comment and proves nothing about behavior. Extract the Step 1 | ||
| # block and run it against a stubbed gh. No network. | ||
| STEP1=$(awk ' | ||
| /^```bash$/ {inb=1; buf=""; next} | ||
| /^```$/ {if (inb && buf ~ /DEFAULT_QUERIES=/) {printf "%s", buf; exit} inb=0; buf=""; next} | ||
| inb {buf = buf $0 "\n"} | ||
| ' "$AGENT") | ||
| [ -n "$STEP1" ] || { echo "FAIL could not extract the Step 1 query block"; exit 1; } | ||
| printf '%s' "$STEP1" > "$tmpdir/step1.sh" | ||
| bash -n "$tmpdir/step1.sh" || { echo "FAIL Step 1 block is not valid bash"; exit 1; } | ||
|
|
||
| mkdir -p "$tmpdir/bin" | ||
| cat > "$tmpdir/bin/gh" <<'EOF' | ||
| #!/usr/bin/env bash | ||
| case "${GH_STUB_MODE:-ok}" in | ||
| fail) echo "gh: HTTP 403 rate limit exceeded" >&2; exit 1 ;; | ||
| null) echo '{"total_count":2,"items":[{"full_name":null,"language":"Go"}]}' ;; | ||
| *) echo '{"total_count":63,"items":[{"full_name":"owner/repo","language":"Go","topics":["cli"],"stargazers_count":1,"pushed_at":"2026-07-01T00:00:00Z","archived":false,"open_issues_count":3,"default_branch":"main","description":"d"}]}' ;; | ||
| esac | ||
| EOF | ||
| chmod +x "$tmpdir/bin/gh" | ||
|
|
||
| run_step1() { # run_step1 <mode> -> Step 1's exit code; stdout/stderr in $tmpdir/out | ||
| GH_STUB_MODE="$1" PATH="$tmpdir/bin:$PATH" \ | ||
| bash "$tmpdir/step1.sh" > "$tmpdir/out" 2>&1 | ||
| } | ||
|
|
||
| # A failed search must abort the scan. Contributing zero rows instead leaves a | ||
| # partial candidate set that reads as a clean run. | ||
| run_step1 fail && { echo "FAIL Step 1 did not abort on a failed search"; exit 1; } | ||
| grep -q 'FATAL' "$tmpdir/out" || { echo "FAIL failed search did not report FATAL"; exit 1; } | ||
|
|
||
| # Field-name drift against the API yields null, not an error. | ||
| run_step1 null && { echo "FAIL Step 1 did not abort on null full_name"; exit 1; } | ||
| grep -q 'FATAL' "$tmpdir/out" || { echo "FAIL null full_name did not report FATAL"; exit 1; } | ||
|
|
||
| # Happy path: still returns rows, and reports matched-of-total from the response. | ||
| run_step1 ok || { echo "FAIL Step 1 aborted on a valid response"; exit 1; } | ||
| grep -q 'of 63 matched' "$tmpdir/out" || { | ||
| echo "FAIL Step 1 does not report matched-of-total"; cat "$tmpdir/out"; exit 1; } | ||
|
coderabbitai[bot] marked this conversation as resolved.
Outdated
|
||
|
|
||
| # Both aborts above only terminate the scan if the loop runs in the current shell. | ||
| # A piped `while` runs in a subshell, where `exit` kills the subshell and the scan | ||
| # continues — the guards would pass review and do nothing. Checked on code with | ||
| # comments stripped, so a comment cannot satisfy or trip it. | ||
| STEP1_CODE=$(printf '%s' "$STEP1" | sed 's/#.*//') | ||
| printf '%s' "$STEP1_CODE" | grep -qE '\|[[:space:]]*while' && { | ||
| echo "FAIL Step 1 query loop is piped; its aborts cannot exit the scan"; exit 1; } | ||
| printf '%s' "$STEP1_CODE" | grep -q 'done <<' || { | ||
| echo "FAIL Step 1 query loop is not heredoc-fed"; exit 1; } | ||
|
|
||
| # The per-candidate repo call must stay deleted. Matched on the resurrection | ||
| # signature — a repos/ call carrying the fields Step 1 already projects — not on | ||
| # `gh api repos/` wholesale: the workflow-tree, contributors, comments, and pulls | ||
| # calls are legitimate and must keep working. | ||
| sed 's/#.*//' "$AGENT" \ | ||
| | grep -E 'gh api .*repos/' \ | ||
| | grep -qE 'pushed_at|open_issues_count|default_branch' && { | ||
| echo "FAIL Step 2 re-fetches fields already carried by the Step 1 projection"; exit 1; } | ||
|
coderabbitai[bot] marked this conversation as resolved.
Outdated
|
||
|
|
||
| echo "OK test_repo_shortlist_schema.sh" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.