From caf1db59d6dd881b553d0908373125199b10098d Mon Sep 17 00:00:00 2001 From: Amit Arora Date: Wed, 14 Oct 2026 22:12:00 +0000 Subject: [PATCH 1/4] benchmark(haiku-5-5, sonnet-5-5, opus-5-5): Claude 5.5 on Bedrock via omp/swe3, cli/cli dataset Set A of the cross-repo check in #221: the three Claude 5.5 models on the 21 cli-cli-v1 tasks, omp/swe3 on Amazon Bedrock, judged by openai.gpt-5.6-sol at high reasoning effort. Only run-summary.json, metrics.json and eval.json are committed, as for v2. claude-opus-5-5 85.62 $2.18/task 21/21 scored claude-sonnet-5-5 82.50 $0.91/task 21/21 scored claude-haiku-5-5 80.82 $0.09/task 19/21 scored Haiku 5.5 failed two high-tier tasks by writing its artifacts outside the output directory the prompt named, as it did once on v2. auth-clipboard-config-default went to claude-code/cli/ on both attempts; skill-install-nested-discovery went to cli-v1/, and its retry wrote nothing. Both count as failures and were not judged. Their artifacts are kept under do-not-include/claude-haiku-5-5-misplaced-artifacts/, outside the harness scan, and the README there records what happened. cli/cli results stay in their own output scope and do not feed the v2 frontier, charts or vended models.json. --- .../auth-clipboard-config-default/eval.json | 60 + .../metrics.json | 153 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../extension-uninstall-alias/eval.json | 64 + .../extension-uninstall-alias/metrics.json | 156 ++ .../issue-view-comments-with-json/eval.json | 64 + .../metrics.json | 156 ++ .../pr-checks-all-cancelled-summary/eval.json | 64 + .../metrics.json | 156 ++ .../pr-comment-attach-ghu-tokens/eval.json | 64 + .../pr-comment-attach-ghu-tokens/metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../pr-view-json-number-on-issue/eval.json | 64 + .../pr-view-json-number-on-issue/metrics.json | 156 ++ .../project-item-add-output-non-tty/eval.json | 64 + .../metrics.json | 156 ++ .../project-item-edit-by-name/eval.json | 64 + .../project-item-edit-by-name/metrics.json | 155 ++ .../release-create-404-scope-blame/eval.json | 64 + .../metrics.json | 156 ++ .../release-verify-sha1-prefix/eval.json | 64 + .../release-verify-sha1-prefix/metrics.json | 156 ++ .../repo-clone-ghe-ssh-ca-url/eval.json | 64 + .../repo-clone-ghe-ssh-ca-url/metrics.json | 158 ++ .../repo-create-reject-empty-name/eval.json | 64 + .../metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../repo-sync-checked-out-worktree/eval.json | 64 + .../metrics.json | 155 ++ .../omp/swe3/cli-cli-v1/run-summary.json | 1436 ++++++++++++++++ .../secret-set-app-agents/eval.json | 64 + .../secret-set-app-agents/metrics.json | 155 ++ .../skill-install-nested-discovery/eval.json | 60 + .../metrics.json | 153 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../auth-clipboard-config-default/eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../extension-uninstall-alias/eval.json | 64 + .../extension-uninstall-alias/metrics.json | 156 ++ .../issue-view-comments-with-json/eval.json | 64 + .../metrics.json | 156 ++ .../pr-checks-all-cancelled-summary/eval.json | 64 + .../metrics.json | 156 ++ .../pr-comment-attach-ghu-tokens/eval.json | 64 + .../pr-comment-attach-ghu-tokens/metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../pr-view-json-number-on-issue/eval.json | 64 + .../pr-view-json-number-on-issue/metrics.json | 156 ++ .../project-item-add-output-non-tty/eval.json | 64 + .../metrics.json | 156 ++ .../project-item-edit-by-name/eval.json | 64 + .../project-item-edit-by-name/metrics.json | 155 ++ .../release-create-404-scope-blame/eval.json | 64 + .../metrics.json | 156 ++ .../release-verify-sha1-prefix/eval.json | 64 + .../release-verify-sha1-prefix/metrics.json | 156 ++ .../repo-clone-ghe-ssh-ca-url/eval.json | 64 + .../repo-clone-ghe-ssh-ca-url/metrics.json | 157 ++ .../repo-create-reject-empty-name/eval.json | 64 + .../metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../repo-sync-checked-out-worktree/eval.json | 64 + .../metrics.json | 155 ++ .../omp/swe3/cli-cli-v1/run-summary.json | 1437 +++++++++++++++++ .../secret-set-app-agents/eval.json | 64 + .../secret-set-app-agents/metrics.json | 155 ++ .../skill-install-nested-discovery/eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../auth-clipboard-config-default/eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../extension-uninstall-alias/eval.json | 64 + .../extension-uninstall-alias/metrics.json | 156 ++ .../issue-view-comments-with-json/eval.json | 64 + .../metrics.json | 156 ++ .../pr-checks-all-cancelled-summary/eval.json | 64 + .../metrics.json | 156 ++ .../pr-comment-attach-ghu-tokens/eval.json | 64 + .../pr-comment-attach-ghu-tokens/metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../pr-view-json-number-on-issue/eval.json | 64 + .../pr-view-json-number-on-issue/metrics.json | 156 ++ .../project-item-add-output-non-tty/eval.json | 64 + .../metrics.json | 156 ++ .../project-item-edit-by-name/eval.json | 64 + .../project-item-edit-by-name/metrics.json | 155 ++ .../release-create-404-scope-blame/eval.json | 64 + .../metrics.json | 156 ++ .../release-verify-sha1-prefix/eval.json | 64 + .../release-verify-sha1-prefix/metrics.json | 156 ++ .../repo-clone-ghe-ssh-ca-url/eval.json | 64 + .../repo-clone-ghe-ssh-ca-url/metrics.json | 157 ++ .../repo-create-reject-empty-name/eval.json | 64 + .../metrics.json | 156 ++ .../eval.json | 64 + .../metrics.json | 156 ++ .../repo-sync-checked-out-worktree/eval.json | 64 + .../metrics.json | 155 ++ .../omp/swe3/cli-cli-v1/run-summary.json | 1437 +++++++++++++++++ .../secret-set-app-agents/eval.json | 64 + .../secret-set-app-agents/metrics.json | 155 ++ .../skill-install-nested-discovery/eval.json | 64 + .../metrics.json | 155 ++ .../eval.json | 64 + .../metrics.json | 155 ++ .../do-not-include/README.md | 9 + .../github-issue.md | 45 + .../implementation.md | 36 + .../auth-clipboard-config-default/lld.md | 192 +++ .../auth-clipboard-config-default/patch.diff | 304 ++++ .../auth-clipboard-config-default/review.md | 31 + .../auth-clipboard-config-default/testing.md | 37 + .../github-issue.md | 34 + .../implementation.md | 35 + .../skill-install-nested-discovery/lld.md | 75 + .../skill-install-nested-discovery/patch.diff | 133 ++ .../skill-install-nested-discovery/review.md | 33 + .../skill-install-nested-discovery/testing.md | 59 + 142 files changed, 19161 insertions(+) create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/run-summary.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/run-summary.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/run-summary.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json create mode 100644 benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/github-issue.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/implementation.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/lld.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/patch.diff create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/review.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/testing.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/github-issue.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/implementation.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/lld.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/patch.diff create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/review.md create mode 100644 benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/testing.md diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json new file mode 100644 index 00000000..b4286481 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json @@ -0,0 +1,60 @@ +{ + "task": "auth-clipboard-config-default", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "lld": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "review": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "testing": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "implementation": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + } + }, + "task_score": 0.0, + "verdict": "MODEL FAILURE: the candidate run did not produce the required artifact(s): github-issue.md, lld.md, review.md, testing.md. Scored 0.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": false, + "scored_zero_missing_artifacts": [ + "github-issue.md", + "lld.md", + "review.md", + "testing.md" + ], + "evaluated_at": "2026-10-14T13:47:44.669482Z" + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json new file mode 100644 index 00000000..fffa3108 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json @@ -0,0 +1,153 @@ +{ + "task": "auth-clipboard-config-default", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "high", + "tags": [ + "auth", + "go", + "cli", + "config", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 0, + "artifacts_expected": 6, + "generation_tokens_per_sec": 160.3, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 222, + "output_tokens": 153667, + "cache_read_tokens": 6522967, + "cache_write_tokens": 617732, + "total_tokens": 7294588, + "total_cost_usd": 0.241232057, + "latency_seconds": 1005.0, + "num_turns": 107, + "generation_tokens_per_sec": 160.3, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 222, + "output_tokens": 153667, + "latency_seconds": 1005.0, + "num_turns": 107, + "total_cost_usd": 0.241232057, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 6522967, + "cache_creation_tokens": 617732, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T13:16:04.194954Z", + "run_ended_at": "2026-10-14T13:22:23.501565Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 222, + "output_tokens": 153667, + "cache_read_tokens": 6522967, + "cache_write_tokens": 617732, + "total_tokens": 7294588, + "total_cost_usd": 0.241232057, + "latency_seconds": 1005.0, + "num_turns": 107, + "generation_tokens_per_sec": 160.3, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "agent_invocations": 2, + "topped_up_artifacts": [], + "evaluation": { + "task": "auth-clipboard-config-default", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "lld": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "review": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "testing": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "implementation": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + } + }, + "task_score": 0.0, + "verdict": "MODEL FAILURE: the candidate run did not produce the required artifact(s): github-issue.md, lld.md, review.md, testing.md. Scored 0.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": false, + "scored_zero_missing_artifacts": [ + "github-issue.md", + "lld.md", + "review.md", + "testing.md" + ], + "evaluated_at": "2026-10-14T13:47:44.669482Z" + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json new file mode 100644 index 00000000..4bdade9f --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json @@ -0,0 +1,64 @@ +{ + "task": "auth-refresh-hint-on-expired-token", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 24, + "risk_awareness": 20, + "total": 84, + "notes": "The issue provides unusually testable host-mapping, token-source, fallback, and non-401 acceptance criteria, with clear non-goals. Its largest weakness is broadening the upstream request from tokens generated by `gh auth login` to every gh-stored writable token, without addressing stored user-supplied PATs. The submitted old-side diff confirms the cited `internal/ghcmd/cmd.go` hardcoded 401 message, while the upstream issue states the narrower generated-token requirement. ([github.com](https://github.com/cli/cli/issues/13066)) Independent local source inspection was unavailable because the read-only command sandbox failed before executing commands." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 24, + "risk_awareness": 20, + "total": 83, + "notes": "The LLD commits to concrete functions, integration points, fallback behavior, host conversion, and preservation of the existing SSO/scope branch ordering. Its largest design weakness is treating `AuthTokenWriteable` as proof that a token is the OAuth credential targeted by the upstream request; writeability establishes storage provenance, not how the token was generated. The proposed `HasActiveToken` guard correctly addresses the documented empty-token behavior, and the nil URL and configuration-error paths are explicit. Patch applicability and the cited source line numbers could not be independently verified because local shell inspection was unavailable." + }, + "review": { + "completeness": 20, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 80, + "notes": "The review finds three concrete defects with consequences and records specific revisions, especially the empty-token false positive and required `-h` argument. It does not challenge the central writeable-versus-generated-token assumption, so its no-blockers verdict is more confident than the evidence supports. Its claim that the existing generic login hint already reveals stored-token existence is also incorrect because only the new differential message provides that signal. The findings reference exact symbols such as `AuthTokenWriteable`, `HasActiveToken`, and the refresh hostname validation, although those references could not be independently opened locally." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 23, + "risk_awareness": 18, + "total": 80, + "notes": "The plan supplies concrete table cases, setup, commands, and exact string oracles for host mapping and helper behavior. Its largest gap is the absence of an automated integration assertion that `Main` actually uses the helper while preserving stderr and exit behavior; unchanged `Test_printError` cases cannot establish that wiring. It also promises both keyring and `oauth_token` behavior but only specifies `Login(..., secureStorage=true)`, and it lacks a stored user-supplied-token case. The manual E2E section acknowledges that its exit-code expectation was not verified." + }, + "implementation": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 18, + "total": 78, + "notes": "The patch implements the submitted LLD end to end, including known API-host conversion, stored-token checks, safe fallbacks, and focused unit tests. Its largest material risk is that writeability alone also selects writable user-supplied tokens, exceeding the upstream requirement for tokens generated by `gh auth login`. The tests omit plain-config `oauth_token` storage and direct `Main` integration, and `assert.NoError` for `Login` violates the repository instruction requiring `require` for error checks. The old-side contexts are internally consistent with the cited files, but patch application and claimed test executions could not be independently reproduced because repository command execution failed at sandbox initialization." + } + }, + "task_score": 81.0, + "verdict": "Strong, concrete work with good fallback coverage, but it fails to distinguish refresh-target OAuth credentials from all writable stored tokens and under-tests the actual `Main` integration.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T13:51:02.365509Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 148724, + "cached_input_tokens": 84594, + "cache_write_input_tokens": 26726, + "output_tokens": 13045, + "reasoning_output_tokens": 11383 + }, + "duration_ms": 194501 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json new file mode 100644 index 00000000..30a64446 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "auth-refresh-hint-on-expired-token", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "auth", + "go", + "cli", + "error-messages", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 170.1, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 114, + "output_tokens": 90180, + "cache_read_tokens": 3231063, + "cache_write_tokens": 264034, + "total_tokens": 3585391, + "total_cost_usd": 0.12145790800000002, + "latency_seconds": 530.1, + "num_turns": 55, + "generation_tokens_per_sec": 170.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 114, + "output_tokens": 90180, + "latency_seconds": 530.1, + "num_turns": 55, + "total_cost_usd": 0.12145790800000002, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3231063, + "cache_creation_tokens": 264034, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:40:11.625998Z", + "run_ended_at": "2026-10-14T12:49:01.799714Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 114, + "output_tokens": 90180, + "cache_read_tokens": 3231063, + "cache_write_tokens": 264034, + "total_tokens": 3585391, + "total_cost_usd": 0.12145790800000002, + "latency_seconds": 530.1, + "num_turns": 55, + "generation_tokens_per_sec": 170.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "auth-refresh-hint-on-expired-token", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 24, + "risk_awareness": 20, + "total": 84, + "notes": "The issue provides unusually testable host-mapping, token-source, fallback, and non-401 acceptance criteria, with clear non-goals. Its largest weakness is broadening the upstream request from tokens generated by `gh auth login` to every gh-stored writable token, without addressing stored user-supplied PATs. The submitted old-side diff confirms the cited `internal/ghcmd/cmd.go` hardcoded 401 message, while the upstream issue states the narrower generated-token requirement. ([github.com](https://github.com/cli/cli/issues/13066)) Independent local source inspection was unavailable because the read-only command sandbox failed before executing commands." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 24, + "risk_awareness": 20, + "total": 83, + "notes": "The LLD commits to concrete functions, integration points, fallback behavior, host conversion, and preservation of the existing SSO/scope branch ordering. Its largest design weakness is treating `AuthTokenWriteable` as proof that a token is the OAuth credential targeted by the upstream request; writeability establishes storage provenance, not how the token was generated. The proposed `HasActiveToken` guard correctly addresses the documented empty-token behavior, and the nil URL and configuration-error paths are explicit. Patch applicability and the cited source line numbers could not be independently verified because local shell inspection was unavailable." + }, + "review": { + "completeness": 20, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 80, + "notes": "The review finds three concrete defects with consequences and records specific revisions, especially the empty-token false positive and required `-h` argument. It does not challenge the central writeable-versus-generated-token assumption, so its no-blockers verdict is more confident than the evidence supports. Its claim that the existing generic login hint already reveals stored-token existence is also incorrect because only the new differential message provides that signal. The findings reference exact symbols such as `AuthTokenWriteable`, `HasActiveToken`, and the refresh hostname validation, although those references could not be independently opened locally." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 23, + "risk_awareness": 18, + "total": 80, + "notes": "The plan supplies concrete table cases, setup, commands, and exact string oracles for host mapping and helper behavior. Its largest gap is the absence of an automated integration assertion that `Main` actually uses the helper while preserving stderr and exit behavior; unchanged `Test_printError` cases cannot establish that wiring. It also promises both keyring and `oauth_token` behavior but only specifies `Login(..., secureStorage=true)`, and it lacks a stored user-supplied-token case. The manual E2E section acknowledges that its exit-code expectation was not verified." + }, + "implementation": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 18, + "total": 78, + "notes": "The patch implements the submitted LLD end to end, including known API-host conversion, stored-token checks, safe fallbacks, and focused unit tests. Its largest material risk is that writeability alone also selects writable user-supplied tokens, exceeding the upstream requirement for tokens generated by `gh auth login`. The tests omit plain-config `oauth_token` storage and direct `Main` integration, and `assert.NoError` for `Login` violates the repository instruction requiring `require` for error checks. The old-side contexts are internally consistent with the cited files, but patch application and claimed test executions could not be independently reproduced because repository command execution failed at sandbox initialization." + } + }, + "task_score": 81.0, + "verdict": "Strong, concrete work with good fallback coverage, but it fails to distinguish refresh-target OAuth credentials from all writable stored tokens and under-tests the actual `Main` integration.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T13:51:02.365509Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 148724, + "cached_input_tokens": 84594, + "cache_write_input_tokens": 26726, + "output_tokens": 13045, + "reasoning_output_tokens": 11383 + }, + "duration_ms": 194501 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json new file mode 100644 index 00000000..25793c42 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json @@ -0,0 +1,64 @@ +{ + "task": "codespace-ports-forward-concurrent-map", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible command, concrete symptoms, bounded scope, non-goals, and testable compatibility criteria. Its paths and concurrency diagnosis agree with the reported stack and upstream fix. The largest factual defect is the reference to issue #13339; the matching report is #13399, fixed by PR #13313. ([github.com](https://github.com/cli/cli/issues/13399), [github.com](https://github.com/cli/cli/pull/13313)) No independent task statement establishes the additional constraints, so those remain internally reasonable but externally unverifiable." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The LLD is strongly grounded in real symbols and clearly explains why copying pointer-bearing connections does not isolate mutable tunnel state. Its pointer conversion and connection-owned mutex match the core upstream solution. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest defect is releasing the mutex between the get/check/delete phases of UpdatePortVisibility; upstream explicitly corrected this TOCTOU gap and also adopted deferred unlocking for robustness. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) It also claims close-error behavior is preserved, although moving safeClose from a closure-local err to the outer named result can make close failures newly observable." + }, + "review": { + "completeness": 18, + "correctness": 17, + "specificity": 21, + "risk_awareness": 17, + "total": 73, + "notes": "The review's strongest work is its concrete analysis of race-detector dependence, mock-server map behavior, lock ordering, and log-order changes. It correctly validates the shared-pointer and mutex approach used by the upstream fix. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest miss is approving separate locks around UpdatePortVisibility operations despite the real TOCTOU hazard later fixed upstream. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) The no-blockers verdict is therefore too confident, and it also accepts the untested close-once lifecycle change without examining its error-semantic consequence." + }, + "testing": { + "completeness": 17, + "correctness": 15, + "specificity": 20, + "risk_awareness": 17, + "total": 69, + "notes": "The plan supplies useful race-enabled package commands, negative scenarios, compatibility checks, and explicit expected outcomes. Its largest defect is that F1 names TestForwardPortConcurrentlyForwardsMultiplePorts while the patch defines TestForwardPortConcurrentlyOnSharedConnection, so the targeted -run command can succeed while running no regression test. The broader E1 command remains useful, and a concurrent race-detector test is the correct basic strategy. ([github.com](https://github.com/cli/cli/pull/13313)) No automated test exercises ForwardPorts, close-once behavior, listener forwarding, or the final port set, while the live checks and claimed failure on the unpatched base remain explicitly unexecuted." + }, + "implementation": { + "completeness": 18, + "correctness": 16, + "specificity": 20, + "risk_awareness": 16, + "total": 70, + "notes": "The patch implements the central fix end to end: it retains the connection pointer, adds a shared mutex, guards manager operations, and adds concurrent coverage, closely matching the upstream resolution. ([github.com](https://github.com/cli/cli/pull/13313)) The diff targets real symbols and plausible base context, but git apply --check could not be independently executed because local shell access failed. Its largest code defect is the non-atomic get/check/delete sequence in UpdatePortVisibility, a TOCTOU issue the upstream review explicitly corrected; manual unlocks are also less robust than the accepted deferred-unlock helper. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) The test violates the repository's require/testify convention and does not cover the added ForwardPorts lifecycle change, while the summary incorrectly calls error behavior unchanged after moving safeClose to the outer named return." + } + }, + "task_score": 75.4, + "verdict": "Strong, repository-specific work that fixes the primary race, but it is not merge-ready because the visibility-update TOCTOU, mismatched targeted test command, and untested close-lifecycle change remain material.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T13:55:10.432518Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 142839, + "cached_input_tokens": 73039, + "cache_write_input_tokens": 25242, + "output_tokens": 15578, + "reasoning_output_tokens": 13374 + }, + "duration_ms": 245043 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json new file mode 100644 index 00000000..a7374ea0 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "codespace-ports-forward-concurrent-map", + "repo": "https://github.com/cli/cli", + "ref": "v2.95.0", + "complexity": "medium", + "tags": [ + "codespace", + "go", + "cli", + "concurrency", + "race-condition", + "fixed-in-2.96.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 184.4, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 88, + "output_tokens": 84133, + "cache_read_tokens": 2445739, + "cache_write_tokens": 213718, + "total_tokens": 2743678, + "total_cost_usd": 0.102572184, + "latency_seconds": 456.2, + "num_turns": 42, + "generation_tokens_per_sec": 184.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 88, + "output_tokens": 84133, + "latency_seconds": 456.2, + "num_turns": 42, + "total_cost_usd": 0.102572184, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2445739, + "cache_creation_tokens": 213718, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:18:58.906889Z", + "run_ended_at": "2026-10-14T12:26:35.118653Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 88, + "output_tokens": 84133, + "cache_read_tokens": 2445739, + "cache_write_tokens": 213718, + "total_tokens": 2743678, + "total_cost_usd": 0.102572184, + "latency_seconds": 456.2, + "num_turns": 42, + "generation_tokens_per_sec": 184.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "codespace-ports-forward-concurrent-map", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible command, concrete symptoms, bounded scope, non-goals, and testable compatibility criteria. Its paths and concurrency diagnosis agree with the reported stack and upstream fix. The largest factual defect is the reference to issue #13339; the matching report is #13399, fixed by PR #13313. ([github.com](https://github.com/cli/cli/issues/13399), [github.com](https://github.com/cli/cli/pull/13313)) No independent task statement establishes the additional constraints, so those remain internally reasonable but externally unverifiable." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The LLD is strongly grounded in real symbols and clearly explains why copying pointer-bearing connections does not isolate mutable tunnel state. Its pointer conversion and connection-owned mutex match the core upstream solution. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest defect is releasing the mutex between the get/check/delete phases of UpdatePortVisibility; upstream explicitly corrected this TOCTOU gap and also adopted deferred unlocking for robustness. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) It also claims close-error behavior is preserved, although moving safeClose from a closure-local err to the outer named result can make close failures newly observable." + }, + "review": { + "completeness": 18, + "correctness": 17, + "specificity": 21, + "risk_awareness": 17, + "total": 73, + "notes": "The review's strongest work is its concrete analysis of race-detector dependence, mock-server map behavior, lock ordering, and log-order changes. It correctly validates the shared-pointer and mutex approach used by the upstream fix. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest miss is approving separate locks around UpdatePortVisibility operations despite the real TOCTOU hazard later fixed upstream. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) The no-blockers verdict is therefore too confident, and it also accepts the untested close-once lifecycle change without examining its error-semantic consequence." + }, + "testing": { + "completeness": 17, + "correctness": 15, + "specificity": 20, + "risk_awareness": 17, + "total": 69, + "notes": "The plan supplies useful race-enabled package commands, negative scenarios, compatibility checks, and explicit expected outcomes. Its largest defect is that F1 names TestForwardPortConcurrentlyForwardsMultiplePorts while the patch defines TestForwardPortConcurrentlyOnSharedConnection, so the targeted -run command can succeed while running no regression test. The broader E1 command remains useful, and a concurrent race-detector test is the correct basic strategy. ([github.com](https://github.com/cli/cli/pull/13313)) No automated test exercises ForwardPorts, close-once behavior, listener forwarding, or the final port set, while the live checks and claimed failure on the unpatched base remain explicitly unexecuted." + }, + "implementation": { + "completeness": 18, + "correctness": 16, + "specificity": 20, + "risk_awareness": 16, + "total": 70, + "notes": "The patch implements the central fix end to end: it retains the connection pointer, adds a shared mutex, guards manager operations, and adds concurrent coverage, closely matching the upstream resolution. ([github.com](https://github.com/cli/cli/pull/13313)) The diff targets real symbols and plausible base context, but git apply --check could not be independently executed because local shell access failed. Its largest code defect is the non-atomic get/check/delete sequence in UpdatePortVisibility, a TOCTOU issue the upstream review explicitly corrected; manual unlocks are also less robust than the accepted deferred-unlock helper. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) The test violates the repository's require/testify convention and does not cover the added ForwardPorts lifecycle change, while the summary incorrectly calls error behavior unchanged after moving safeClose to the outer named return." + } + }, + "task_score": 75.4, + "verdict": "Strong, repository-specific work that fixes the primary race, but it is not merge-ready because the visibility-update TOCTOU, mismatched targeted test command, and untested close-lifecycle change remain material.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T13:55:10.432518Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 142839, + "cached_input_tokens": 73039, + "cache_write_input_tokens": 25242, + "output_tokens": 15578, + "reasoning_output_tokens": 13374 + }, + "duration_ms": 245043 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json new file mode 100644 index 00000000..d718434e --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json @@ -0,0 +1,64 @@ +{ + "task": "copilot-not-installed-trailing-newline", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue precisely defines the interactive declined-install behavior, reproduction, exclusions, and exact stderr oracle. Its acceptance criteria explicitly preserve cmdutil.SilentError and prohibit invoking the download path. The largest limitation is evidentiary: no independent task statement establishes whether excluding the sibling non-TTY warning is the intended upstream scope. The named pkg/cmd/copilot paths and runCopilot behavior agree with the submitted patch context, but the referenced issue could not be independently validated." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The LLD commits to a minimal format-string change and gives an implementation-ready test setup using CanPrompt, PrompterMock, findCopilotBinaryFunc, and an explicit HttpClient failure stub. It addresses output bytes, control flow, compatibility, color stability, rollback, and the intentionally unchanged non-TTY branch. Its main weakness is excessive confidence in exact line and API claims that could not all be independently checked because repository shell access failed. The submitted diff nevertheless matches the described runCopilot branch and test integration points without an apparent architectural inconsistency." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review identifies a concrete weakness in the initial test design: relying on a nil HttpClient would not clearly assert that download setup was skipped. It also examines the residual non-TTY defect and byte-exact stderr compatibility, then records explicit resolutions and a defensible verdict. The largest deficiency is that several reviewer sections repeat the same findings, while the proposed stdout assertion is answered affirmatively but neither triaged nor implemented. Some UX claims about wrapper behavior and the prompt's rendering were not independently verifiable from the available source evidence." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 21, + "risk_awareness": 19, + "total": 77, + "notes": "The central unit test has meaningful oracles for the exact newline, SilentError, and absence of HttpClient use, and the plan states the expected pre-fix failure. It also includes focused package commands, a regression run, and a manual terminal check. The largest gap is claiming that the accepted-install and non-TTY paths are covered by the existing suite without identifying a test that actually exercises the latter. It also omits the repository-mandated final go test ./... and make lint checks and uses an environment-specific REPO_DIR that was not verified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The patch implements the requested behavior directly by adding one newline and adds a focused regression test that configures both TTY flags, disables color, restores findCopilotBinaryFunc, and rejects HttpClient use. The summary accurately describes the two touched files, the residual non-TTY behavior, and the one-byte compatibility change. No evident logic, API, security, or caller regression appears in the submitted hunks. However, local shell failure prevented an independent git apply --check or compilation check, so clean applicability and exact surrounding source remain an evidence gap rather than a demonstrated defect." + } + }, + "task_score": 87.6, + "verdict": "This is a strong, narrowly targeted submission whose largest limitation is incomplete independent source verification and a testing plan that overstates coverage of unchanged branches.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T13:57:48.639298Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 180384, + "cached_input_tokens": 100139, + "cache_write_input_tokens": 27651, + "output_tokens": 10432, + "reasoning_output_tokens": 9174 + }, + "duration_ms": 154930 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json new file mode 100644 index 00000000..5387c6cd --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "copilot-not-installed-trailing-newline", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "trivial", + "tags": [ + "copilot", + "go", + "cli", + "terminal-output", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 192.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 50, + "output_tokens": 37854, + "cache_read_tokens": 1468743, + "cache_write_tokens": 109874, + "total_tokens": 1616521, + "total_cost_usd": 0.05208904800000001, + "latency_seconds": 196.4, + "num_turns": 24, + "generation_tokens_per_sec": 192.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 50, + "output_tokens": 37854, + "latency_seconds": 196.4, + "num_turns": 24, + "total_cost_usd": 0.05208904800000001, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1468743, + "cache_creation_tokens": 109874, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:31:31.501529Z", + "run_ended_at": "2026-10-14T11:34:47.935941Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 50, + "output_tokens": 37854, + "cache_read_tokens": 1468743, + "cache_write_tokens": 109874, + "total_tokens": 1616521, + "total_cost_usd": 0.05208904800000001, + "latency_seconds": 196.4, + "num_turns": 24, + "generation_tokens_per_sec": 192.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "copilot-not-installed-trailing-newline", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue precisely defines the interactive declined-install behavior, reproduction, exclusions, and exact stderr oracle. Its acceptance criteria explicitly preserve cmdutil.SilentError and prohibit invoking the download path. The largest limitation is evidentiary: no independent task statement establishes whether excluding the sibling non-TTY warning is the intended upstream scope. The named pkg/cmd/copilot paths and runCopilot behavior agree with the submitted patch context, but the referenced issue could not be independently validated." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The LLD commits to a minimal format-string change and gives an implementation-ready test setup using CanPrompt, PrompterMock, findCopilotBinaryFunc, and an explicit HttpClient failure stub. It addresses output bytes, control flow, compatibility, color stability, rollback, and the intentionally unchanged non-TTY branch. Its main weakness is excessive confidence in exact line and API claims that could not all be independently checked because repository shell access failed. The submitted diff nevertheless matches the described runCopilot branch and test integration points without an apparent architectural inconsistency." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review identifies a concrete weakness in the initial test design: relying on a nil HttpClient would not clearly assert that download setup was skipped. It also examines the residual non-TTY defect and byte-exact stderr compatibility, then records explicit resolutions and a defensible verdict. The largest deficiency is that several reviewer sections repeat the same findings, while the proposed stdout assertion is answered affirmatively but neither triaged nor implemented. Some UX claims about wrapper behavior and the prompt's rendering were not independently verifiable from the available source evidence." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 21, + "risk_awareness": 19, + "total": 77, + "notes": "The central unit test has meaningful oracles for the exact newline, SilentError, and absence of HttpClient use, and the plan states the expected pre-fix failure. It also includes focused package commands, a regression run, and a manual terminal check. The largest gap is claiming that the accepted-install and non-TTY paths are covered by the existing suite without identifying a test that actually exercises the latter. It also omits the repository-mandated final go test ./... and make lint checks and uses an environment-specific REPO_DIR that was not verified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The patch implements the requested behavior directly by adding one newline and adds a focused regression test that configures both TTY flags, disables color, restores findCopilotBinaryFunc, and rejects HttpClient use. The summary accurately describes the two touched files, the residual non-TTY behavior, and the one-byte compatibility change. No evident logic, API, security, or caller regression appears in the submitted hunks. However, local shell failure prevented an independent git apply --check or compilation check, so clean applicability and exact surrounding source remain an evidence gap rather than a demonstrated defect." + } + }, + "task_score": 87.6, + "verdict": "This is a strong, narrowly targeted submission whose largest limitation is incomplete independent source verification and a testing plan that overstates coverage of unchanged branches.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T13:57:48.639298Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 180384, + "cached_input_tokens": 100139, + "cache_write_input_tokens": 27651, + "output_tokens": 10432, + "reasoning_output_tokens": 9174 + }, + "duration_ms": 154930 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json new file mode 100644 index 00000000..cce88f0b --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json @@ -0,0 +1,64 @@ +{ + "task": "extension-uninstall-alias", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 19, + "correctness": 12, + "specificity": 22, + "risk_awareness": 16, + "total": 69, + "notes": "The issue clearly defines user-visible behavior, compatibility, concrete acceptance criteria, and non-goals. Its largest defect is converting an alias task into a separate discoverable command based on an unsupported help-listing requirement. The published manual presents `gh extension uninstall` under the aliases of `gh extension remove`, contradicting that central choice. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) With no independent task statement, the help requirement and related issue #13598 could not be verified." + }, + "lld": { + "completeness": 20, + "correctness": 12, + "specificity": 22, + "risk_awareness": 14, + "total": 68, + "notes": "The LLD names concrete integration points such as `NewCmdExtension`, `normalizeExtensionSelector`, `ExtensionManager.Remove`, and the relevant command tests. Its central decision is nevertheless wrong for an alias task: it extracts a handler and registers a second command instead of adding Cobra alias metadata. Existing repository evidence shows `NewCmdExtension` already uses Cobra `Aliases`, while the published manual identifies uninstall specifically as a remove alias. ([github.com](https://github.com/cli/cli/pull/5974/files/a6596762701e0b1299675d9071c8b8f06af8d523), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The claimed need for a live fixture and exact baseline line contexts could not be independently verified because the local read-only command runner failed before execution." + }, + "review": { + "completeness": 13, + "correctness": 9, + "specificity": 15, + "risk_awareness": 10, + "total": 47, + "notes": "The review does identify a concrete acceptance-test gap and the duplicated help description. Most reviewer sections merely approve the design, however, and none challenges the load-bearing decision to create a distinct command. That endorsement conflicts with the published CLI interface, where uninstall is documented as an alias of remove. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The assertion that acceptance coverage cannot be added because it cannot run in the sandbox is not a sufficient design justification, and the live-fixture requirement was not independently verified." + }, + "testing": { + "completeness": 17, + "correctness": 12, + "specificity": 21, + "risk_awareness": 13, + "total": 63, + "notes": "The plan provides concrete commands and meaningful oracles for selector normalization, TTY output, manager errors, help output, and remove regression behavior. Its main deficiency is that it tests and enshrines a separate `uninstall` command rather than verifying alias metadata or canonical-command behavior. The implementation tests also use `assert.Equal` for call count and then unconditionally index `calls[0]`, allowing a failed count assertion to become a panic. The commands were not executed, and the claimed necessity of a live acceptance fixture could not be independently confirmed." + }, + "implementation": { + "completeness": 17, + "correctness": 11, + "specificity": 20, + "risk_awareness": 12, + "total": 60, + "notes": "The patch functionally wires `uninstall` through the same normalization, removal, error, and TTY-output path and adds focused tests. Its largest defect is implementing a sibling command rather than the requested alias, creating unnecessary handler extraction, duplicate command metadata, and different generated help/documentation semantics. The official manual lists `gh extension uninstall` in the alias section for `gh extension remove`, which does not match this patch's command model. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The submitted hunks are internally coherent, but patch applicability and the claimed baseline SHA verification could not be independently repeated because the repository command runner was unavailable." + } + }, + "task_score": 61.4, + "verdict": "The artifacts are concrete and functionally plausible, but they consistently implement and defend a separate command instead of the repository's intended uninstall alias.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:00:37.912949Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 173074, + "cached_input_tokens": 112877, + "cache_write_input_tokens": 25027, + "output_tokens": 11175, + "reasoning_output_tokens": 8910 + }, + "duration_ms": 166325 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json new file mode 100644 index 00000000..4546c609 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "extension-uninstall-alias", + "repo": "https://github.com/cli/cli", + "ref": "v2.93.0", + "complexity": "trivial", + "tags": [ + "extension", + "go", + "cli", + "cobra", + "ux", + "fixed-in-2.94.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 179.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 44, + "output_tokens": 33074, + "cache_read_tokens": 1228975, + "cache_write_tokens": 101125, + "total_tokens": 1363218, + "total_cost_usd": 0.045618952500000004, + "latency_seconds": 184.0, + "num_turns": 21, + "generation_tokens_per_sec": 179.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 44, + "output_tokens": 33074, + "latency_seconds": 184.0, + "num_turns": 21, + "total_cost_usd": 0.045618952500000004, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1228975, + "cache_creation_tokens": 101125, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:43:12.121641Z", + "run_ended_at": "2026-10-14T11:46:16.091877Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 44, + "output_tokens": 33074, + "cache_read_tokens": 1228975, + "cache_write_tokens": 101125, + "total_tokens": 1363218, + "total_cost_usd": 0.045618952500000004, + "latency_seconds": 184.0, + "num_turns": 21, + "generation_tokens_per_sec": 179.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "extension-uninstall-alias", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 19, + "correctness": 12, + "specificity": 22, + "risk_awareness": 16, + "total": 69, + "notes": "The issue clearly defines user-visible behavior, compatibility, concrete acceptance criteria, and non-goals. Its largest defect is converting an alias task into a separate discoverable command based on an unsupported help-listing requirement. The published manual presents `gh extension uninstall` under the aliases of `gh extension remove`, contradicting that central choice. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) With no independent task statement, the help requirement and related issue #13598 could not be verified." + }, + "lld": { + "completeness": 20, + "correctness": 12, + "specificity": 22, + "risk_awareness": 14, + "total": 68, + "notes": "The LLD names concrete integration points such as `NewCmdExtension`, `normalizeExtensionSelector`, `ExtensionManager.Remove`, and the relevant command tests. Its central decision is nevertheless wrong for an alias task: it extracts a handler and registers a second command instead of adding Cobra alias metadata. Existing repository evidence shows `NewCmdExtension` already uses Cobra `Aliases`, while the published manual identifies uninstall specifically as a remove alias. ([github.com](https://github.com/cli/cli/pull/5974/files/a6596762701e0b1299675d9071c8b8f06af8d523), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The claimed need for a live fixture and exact baseline line contexts could not be independently verified because the local read-only command runner failed before execution." + }, + "review": { + "completeness": 13, + "correctness": 9, + "specificity": 15, + "risk_awareness": 10, + "total": 47, + "notes": "The review does identify a concrete acceptance-test gap and the duplicated help description. Most reviewer sections merely approve the design, however, and none challenges the load-bearing decision to create a distinct command. That endorsement conflicts with the published CLI interface, where uninstall is documented as an alias of remove. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The assertion that acceptance coverage cannot be added because it cannot run in the sandbox is not a sufficient design justification, and the live-fixture requirement was not independently verified." + }, + "testing": { + "completeness": 17, + "correctness": 12, + "specificity": 21, + "risk_awareness": 13, + "total": 63, + "notes": "The plan provides concrete commands and meaningful oracles for selector normalization, TTY output, manager errors, help output, and remove regression behavior. Its main deficiency is that it tests and enshrines a separate `uninstall` command rather than verifying alias metadata or canonical-command behavior. The implementation tests also use `assert.Equal` for call count and then unconditionally index `calls[0]`, allowing a failed count assertion to become a panic. The commands were not executed, and the claimed necessity of a live acceptance fixture could not be independently confirmed." + }, + "implementation": { + "completeness": 17, + "correctness": 11, + "specificity": 20, + "risk_awareness": 12, + "total": 60, + "notes": "The patch functionally wires `uninstall` through the same normalization, removal, error, and TTY-output path and adds focused tests. Its largest defect is implementing a sibling command rather than the requested alias, creating unnecessary handler extraction, duplicate command metadata, and different generated help/documentation semantics. The official manual lists `gh extension uninstall` in the alias section for `gh extension remove`, which does not match this patch's command model. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The submitted hunks are internally coherent, but patch applicability and the claimed baseline SHA verification could not be independently repeated because the repository command runner was unavailable." + } + }, + "task_score": 61.4, + "verdict": "The artifacts are concrete and functionally plausible, but they consistently implement and defend a separate command instead of the repository's intended uninstall alias.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:00:37.912949Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 173074, + "cached_input_tokens": 112877, + "cache_write_input_tokens": 25027, + "output_tokens": 11175, + "reasoning_output_tokens": 8910 + }, + "duration_ms": 166325 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json new file mode 100644 index 00000000..3d392f94 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json @@ -0,0 +1,64 @@ +{ + "task": "issue-view-comments-with-json", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the silent flag interaction, covers issue, PR, and discussion views, and gives directly testable acceptance criteria for rejected and preserved combinations. Its described command structure agrees with the submitted view-command patches and the repository's JSON exporter pattern. The largest evidence gap is that the independent upstream issue text was not supplied, so the assertion that either behavior and the comments-field exception are accepted requirements could not be verified. Compatibility impact, no-request behavior, and explicit non-goals are handled well, with only release communication left implicit." + }, + "lld": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The design commits to an exact predicate, helper signature, error type, call ordering, and eight concrete file changes rather than leaving implementation decisions open. The diff confirms the named constructors, Exporter.Fields usage, RunE integration points, and shared cmdutil location exist as described. Its main weakness is limited treatment of error-precedence interactions such as discussion comment URLs combined with JSON, and the exported command-specific helper is not compared against existing validation abstractions beyond rejecting inline duplication. The compatibility break and release-note need are identified, but rollback and validation guidance remain lightweight." + }, + "review": { + "completeness": 18, + "correctness": 19, + "specificity": 18, + "risk_awareness": 17, + "total": 72, + "notes": "The review checks relevant details including Cobra ordering, nil-exporter behavior, request timing, error UX, and the intentional exit-code change. These findings agree with the patch's placement of ValidateCommentsWithJSON before runF in all three constructors. The review is predominantly a confirmation pass and misses the concrete discrepancy that implementation.md and testing.md claim five helper-test rows while patch.diff contains four. It also offers little challenge to the comments-field exception or command-level accepted-path coverage, and the unexplained Step 7.5 reference is unsupported by the supplied artifacts." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan names exact test functions, arguments, expected errors, option state, and a meaningful no-HTTP oracle, with coverage across all three commands and the shared validator. The listed paths and test additions match patch.diff, and the compatibility and manual checks target the stated regression. However, it describes five TestValidateCommentsWithJSON cases while the implementation contains four, omitting the separate comments-false/exporter-nil row. Accepted command wiring is tested directly only for discussion, and package test commands plus stronger explicit checks for preserved issue and PR combinations would improve regression confidence." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the design end to end: one validator checks exporter fields, all three RunE paths invoke it before execution, and unit tests cover the rejected path plus shared acceptance logic. The submitted contexts and symbols are internally consistent with the repository structure, and the code follows Go and repository conventions with a godoc comment, FlagErrorf, table testing, require for errors, and no unrelated churn. The largest defect is the summary's false claim of five helper-test rows when the diff has four; issue and PR also lack direct accepted-combination constructor tests. The unexecuted compiler and test status is disclosed honestly, while the claimed local reverse-apply check could not be independently verified from the available execution environment." + } + }, + "task_score": 83.0, + "verdict": "This is a strong, concrete design and implementation, limited mainly by a rubber-stamp review and a small mismatch between claimed and actual test coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:03:39.171641Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 184881, + "cached_input_tokens": 104153, + "cache_write_input_tokens": 26959, + "output_tokens": 11798, + "reasoning_output_tokens": 10257 + }, + "duration_ms": 181245 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json new file mode 100644 index 00000000..4123cbf8 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "issue-view-comments-with-json", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "medium", + "tags": [ + "issue", + "go", + "cli", + "json-output", + "flags", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 157.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 72, + "output_tokens": 46501, + "cache_read_tokens": 2000597, + "cache_write_tokens": 153007, + "total_tokens": 2200177, + "total_cost_usd": 0.0686284995, + "latency_seconds": 295.3, + "num_turns": 34, + "generation_tokens_per_sec": 157.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 72, + "output_tokens": 46501, + "latency_seconds": 295.3, + "num_turns": 34, + "total_cost_usd": 0.0686284995, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2000597, + "cache_creation_tokens": 153007, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:26:38.498278Z", + "run_ended_at": "2026-10-14T12:31:33.790206Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 72, + "output_tokens": 46501, + "cache_read_tokens": 2000597, + "cache_write_tokens": 153007, + "total_tokens": 2200177, + "total_cost_usd": 0.0686284995, + "latency_seconds": 295.3, + "num_turns": 34, + "generation_tokens_per_sec": 157.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "issue-view-comments-with-json", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the silent flag interaction, covers issue, PR, and discussion views, and gives directly testable acceptance criteria for rejected and preserved combinations. Its described command structure agrees with the submitted view-command patches and the repository's JSON exporter pattern. The largest evidence gap is that the independent upstream issue text was not supplied, so the assertion that either behavior and the comments-field exception are accepted requirements could not be verified. Compatibility impact, no-request behavior, and explicit non-goals are handled well, with only release communication left implicit." + }, + "lld": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The design commits to an exact predicate, helper signature, error type, call ordering, and eight concrete file changes rather than leaving implementation decisions open. The diff confirms the named constructors, Exporter.Fields usage, RunE integration points, and shared cmdutil location exist as described. Its main weakness is limited treatment of error-precedence interactions such as discussion comment URLs combined with JSON, and the exported command-specific helper is not compared against existing validation abstractions beyond rejecting inline duplication. The compatibility break and release-note need are identified, but rollback and validation guidance remain lightweight." + }, + "review": { + "completeness": 18, + "correctness": 19, + "specificity": 18, + "risk_awareness": 17, + "total": 72, + "notes": "The review checks relevant details including Cobra ordering, nil-exporter behavior, request timing, error UX, and the intentional exit-code change. These findings agree with the patch's placement of ValidateCommentsWithJSON before runF in all three constructors. The review is predominantly a confirmation pass and misses the concrete discrepancy that implementation.md and testing.md claim five helper-test rows while patch.diff contains four. It also offers little challenge to the comments-field exception or command-level accepted-path coverage, and the unexplained Step 7.5 reference is unsupported by the supplied artifacts." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan names exact test functions, arguments, expected errors, option state, and a meaningful no-HTTP oracle, with coverage across all three commands and the shared validator. The listed paths and test additions match patch.diff, and the compatibility and manual checks target the stated regression. However, it describes five TestValidateCommentsWithJSON cases while the implementation contains four, omitting the separate comments-false/exporter-nil row. Accepted command wiring is tested directly only for discussion, and package test commands plus stronger explicit checks for preserved issue and PR combinations would improve regression confidence." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the design end to end: one validator checks exporter fields, all three RunE paths invoke it before execution, and unit tests cover the rejected path plus shared acceptance logic. The submitted contexts and symbols are internally consistent with the repository structure, and the code follows Go and repository conventions with a godoc comment, FlagErrorf, table testing, require for errors, and no unrelated churn. The largest defect is the summary's false claim of five helper-test rows when the diff has four; issue and PR also lack direct accepted-combination constructor tests. The unexecuted compiler and test status is disclosed honestly, while the claimed local reverse-apply check could not be independently verified from the available execution environment." + } + }, + "task_score": 83.0, + "verdict": "This is a strong, concrete design and implementation, limited mainly by a rubber-stamp review and a small mismatch between claimed and actual test coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:03:39.171641Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 184881, + "cached_input_tokens": 104153, + "cache_write_input_tokens": 26959, + "output_tokens": 11798, + "reasoning_output_tokens": 10257 + }, + "duration_ms": 181245 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json new file mode 100644 index 00000000..d533ab5b --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-checks-all-cancelled-summary", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 23, + "risk_awareness": 21, + "total": 91, + "notes": "The issue clearly defines the cancelled-only failure, expected headline and tally, regression requirement, and explicit non-goals. The cited guard and existing Canceled branch in pkg/cmd/pr/checks/output.go match the submitted patch context and the documented upstream defect. Its largest gap is that preserving the zero-check case is only implied by the broad requirement that all other combinations remain unchanged. No independent task statement was supplied, but the task identifier and repository evidence substantiate the central requirement." + }, + "lld": { + "completeness": 22, + "correctness": 17, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The LLD is unusually concrete about output.go, aggregate.go, checkCounts.Canceled, fixture structure, precedence, and the one-line implementation decision. Its main code change is correct, but several repository claims are inconsistent: it names TestChecksRun although the patch context shows Test_checksRun, and it says printSummary is called only in watch mode even though its proposed default-options TTY test exercises it outside watch mode. It also alternately claims the old behavior writes an empty line and that the tally was not printed, leaving the precise pre-fix output unclear. Compatibility reasoning is otherwise strong, particularly the proof that adding Canceled changes only cancelled-only inputs while preserving the zero-count guard." + }, + "review": { + "completeness": 18, + "correctness": 16, + "specificity": 19, + "risk_awareness": 17, + "total": 70, + "notes": "The backend section provides the strongest analysis by tracing the count predicate and showing why mixed cancelled-and-passing cases masked the defect. However, every reviewer approves without catching the LLD's incorrect test symbol, contradictory tally description, or false assertion that printSummary is only called in watch mode. The recommendation for a Bandit-style review is also inappropriate for this Go repository and suggests generic security-review language rather than a repository-specific check. It identifies sensible compatibility and terminal-output risks, but does not challenge the lack of explicit zero-check and complete summary-branch coverage." + }, + "testing": { + "completeness": 20, + "correctness": 13, + "specificity": 22, + "risk_awareness": 18, + "total": 73, + "notes": "The plan supplies concrete fixture state, exact output oracles, a before-fix negative check, manual CLI behavior, exit-status expectations, and compatibility coverage. Its focused commands use TestChecksRun, while the actual test function shown by the patch is Test_checksRun, so both -run commands can report success while executing no matching tests. The full-package command would still execute the suite, preventing the plan from being unusable, but the principal targeted verification is materially defective. It also omits an explicit zero-check oracle, the adjacent behavior most directly exposed to changes in this guard." + }, + "implementation": { + "completeness": 22, + "correctness": 23, + "specificity": 22, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the required behavior end to end: it adds counts.Canceled to the existing guard, supplies a valid-looking cancelled-only GraphQL fixture, and adds a full-output TTY regression case in the actual Test_checksRun table. The output.go context and symbols match the repository defect, and the one-line logic preserves precedence and the zero-check condition. The main limitation is test breadth: the new test covers only the regression through the heavier command harness rather than directly pinning every printSummary branch, especially the no-check case. The summary accurately reports that tests were not run, although its statement that the function previously printed nothing is imprecise relative to the observed blank-headline behavior." + } + }, + "task_score": 80.0, + "verdict": "The submission contains a strong, likely correct implementation and issue, but its design review misses factual inconsistencies and its targeted test commands match the wrong Go test symbol.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:06:19.408521Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 181237, + "cached_input_tokens": 114649, + "cache_write_input_tokens": 25381, + "output_tokens": 10714, + "reasoning_output_tokens": 8974 + }, + "duration_ms": 160223 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json new file mode 100644 index 00000000..88c00fdd --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-checks-all-cancelled-summary", + "repo": "https://github.com/cli/cli", + "ref": "v2.95.0", + "complexity": "trivial", + "tags": [ + "pr", + "go", + "cli", + "checks", + "terminal-output", + "fixed-in-2.96.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 172.2, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 52, + "output_tokens": 27930, + "cache_read_tokens": 1503994, + "cache_write_tokens": 88330, + "total_tokens": 1620306, + "total_cost_usd": 0.044056529000000004, + "latency_seconds": 162.2, + "num_turns": 25, + "generation_tokens_per_sec": 172.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 52, + "output_tokens": 27930, + "latency_seconds": 162.2, + "num_turns": 25, + "total_cost_usd": 0.044056529000000004, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1503994, + "cache_creation_tokens": 88330, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:40:26.504957Z", + "run_ended_at": "2026-10-14T11:43:08.685786Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 52, + "output_tokens": 27930, + "cache_read_tokens": 1503994, + "cache_write_tokens": 88330, + "total_tokens": 1620306, + "total_cost_usd": 0.044056529000000004, + "latency_seconds": 162.2, + "num_turns": 25, + "generation_tokens_per_sec": 172.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-checks-all-cancelled-summary", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 23, + "risk_awareness": 21, + "total": 91, + "notes": "The issue clearly defines the cancelled-only failure, expected headline and tally, regression requirement, and explicit non-goals. The cited guard and existing Canceled branch in pkg/cmd/pr/checks/output.go match the submitted patch context and the documented upstream defect. Its largest gap is that preserving the zero-check case is only implied by the broad requirement that all other combinations remain unchanged. No independent task statement was supplied, but the task identifier and repository evidence substantiate the central requirement." + }, + "lld": { + "completeness": 22, + "correctness": 17, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The LLD is unusually concrete about output.go, aggregate.go, checkCounts.Canceled, fixture structure, precedence, and the one-line implementation decision. Its main code change is correct, but several repository claims are inconsistent: it names TestChecksRun although the patch context shows Test_checksRun, and it says printSummary is called only in watch mode even though its proposed default-options TTY test exercises it outside watch mode. It also alternately claims the old behavior writes an empty line and that the tally was not printed, leaving the precise pre-fix output unclear. Compatibility reasoning is otherwise strong, particularly the proof that adding Canceled changes only cancelled-only inputs while preserving the zero-count guard." + }, + "review": { + "completeness": 18, + "correctness": 16, + "specificity": 19, + "risk_awareness": 17, + "total": 70, + "notes": "The backend section provides the strongest analysis by tracing the count predicate and showing why mixed cancelled-and-passing cases masked the defect. However, every reviewer approves without catching the LLD's incorrect test symbol, contradictory tally description, or false assertion that printSummary is only called in watch mode. The recommendation for a Bandit-style review is also inappropriate for this Go repository and suggests generic security-review language rather than a repository-specific check. It identifies sensible compatibility and terminal-output risks, but does not challenge the lack of explicit zero-check and complete summary-branch coverage." + }, + "testing": { + "completeness": 20, + "correctness": 13, + "specificity": 22, + "risk_awareness": 18, + "total": 73, + "notes": "The plan supplies concrete fixture state, exact output oracles, a before-fix negative check, manual CLI behavior, exit-status expectations, and compatibility coverage. Its focused commands use TestChecksRun, while the actual test function shown by the patch is Test_checksRun, so both -run commands can report success while executing no matching tests. The full-package command would still execute the suite, preventing the plan from being unusable, but the principal targeted verification is materially defective. It also omits an explicit zero-check oracle, the adjacent behavior most directly exposed to changes in this guard." + }, + "implementation": { + "completeness": 22, + "correctness": 23, + "specificity": 22, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the required behavior end to end: it adds counts.Canceled to the existing guard, supplies a valid-looking cancelled-only GraphQL fixture, and adds a full-output TTY regression case in the actual Test_checksRun table. The output.go context and symbols match the repository defect, and the one-line logic preserves precedence and the zero-check condition. The main limitation is test breadth: the new test covers only the regression through the heavier command harness rather than directly pinning every printSummary branch, especially the no-check case. The summary accurately reports that tests were not run, although its statement that the function previously printed nothing is imprecise relative to the observed blank-headline behavior." + } + }, + "task_score": 80.0, + "verdict": "The submission contains a strong, likely correct implementation and issue, but its design review misses factual inconsistencies and its targeted test commands match the wrong Go test symbol.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:06:19.408521Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 181237, + "cached_input_tokens": 114649, + "cache_write_input_tokens": 25381, + "output_tokens": 10714, + "reasoning_output_tokens": 8974 + }, + "duration_ms": 160223 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json new file mode 100644 index 00000000..89f3265f --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-comment-attach-ghu-tokens", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 19, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The issue provides six testable acceptance criteria, explicit exclusions, reproduction steps, and compatibility expectations. Its largest defect is the repeated v2.10.0 reference, which conflicts with the submitted repository context of v2.101.0 and makes the stated base identity unreliable. The patch context corroborates the central symbols in internal/attachments/client.go and the existing user-to-server rejection row in client_test.go. The referenced issue and reported curl HTTP 201 result could not be independently verified, so server acceptance remains an asserted premise." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The design commits to a minimal allowlist change and precisely traces token classification, upload authorization, error propagation, and shared command integration. Its largest deficiency is unreliable version metadata plus a promised uploadTokenTypes comment rewrite that the implementation omits. The submitted diff contains the named uploadTokenTypes, NewUploader, uploadError, and commentable test integration points and is consistent with the main design. Claims that documentation contains no relevant text and that every create and edit path shares this gate could not be independently checked." + }, + "review": { + "completeness": 20, + "correctness": 17, + "specificity": 22, + "risk_awareness": 18, + "total": 77, + "notes": "The review identifies concrete test defects, including the contradictory rejection row, missing command coverage, authorization-header oracle, and fragile error-string assertion. Its largest correctness problem is the SRE claim that there is \"no new request,\" which directly contradicts the feature's purpose of allowing previously blocked ghu_ uploads to send a request. The findings map to specific patch changes in client_test.go and commentable_test.go rather than generic reviewer role-play. It does not flag the erroneous repository version or the implementation's omission of the LLD's promised comment update." + }, + "testing": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The plan has strong behavioral oracles for the returned URL, exact Authorization header, upload count, comment body, stdout, rejection status, and parsed server message. Its largest gap is the absence of executable affected-package test commands, while the manual procedure invokes gh rather than the repository-required built binary. The implementation contains the planned accepted and refused upload subtests and the shared comment table row with concrete expected output. The plan honestly reports that compilation, regression tests, and live server behavior were not verified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The patch implements the requested behavior through the narrow uploadTokenTypes change and adds unit, failure-path, authorization-header, and command-level coverage. The diff specifically changes internal/attachments/client.go, flips the existing TestNewUploader row, checks uploadError fields for HTTP 401, and exercises the shared comment path. Its largest deviation is omitting the LLD's promised comment rewrite, and its implementation metadata repeats the incorrect v2.10.0 base reference. Local source inspection and git apply checking were unavailable because the read-only shell failed before command execution, leaving exact applicability unverified but revealing no concrete defect in the submitted diff." + } + }, + "task_score": 84.8, + "verdict": "This is a strong, narrowly scoped solution with unusually concrete tests, but its repeated base-version mismatch and the review's false claim that no new request is introduced reduce confidence.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:09:13.178131Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 232323, + "cached_input_tokens": 152728, + "cache_write_input_tokens": 28739, + "output_tokens": 10806, + "reasoning_output_tokens": 9289 + }, + "duration_ms": 169980 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json new file mode 100644 index 00000000..f90de755 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-comment-attach-ghu-tokens", + "repo": "https://github.com/cli/cli", + "ref": "v2.101.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "auth", + "attachments", + "fixed-in-2.102.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 178.0, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 64, + "output_tokens": 55874, + "cache_read_tokens": 1871696, + "cache_write_tokens": 167990, + "total_tokens": 2095624, + "total_cost_usd": 0.074425021, + "latency_seconds": 313.9, + "num_turns": 31, + "generation_tokens_per_sec": 178.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 64, + "output_tokens": 55874, + "latency_seconds": 313.9, + "num_turns": 31, + "total_cost_usd": 0.074425021, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1871696, + "cache_creation_tokens": 167990, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:02:42.322344Z", + "run_ended_at": "2026-10-14T12:07:56.203877Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 64, + "output_tokens": 55874, + "cache_read_tokens": 1871696, + "cache_write_tokens": 167990, + "total_tokens": 2095624, + "total_cost_usd": 0.074425021, + "latency_seconds": 313.9, + "num_turns": 31, + "generation_tokens_per_sec": 178.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-comment-attach-ghu-tokens", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 19, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The issue provides six testable acceptance criteria, explicit exclusions, reproduction steps, and compatibility expectations. Its largest defect is the repeated v2.10.0 reference, which conflicts with the submitted repository context of v2.101.0 and makes the stated base identity unreliable. The patch context corroborates the central symbols in internal/attachments/client.go and the existing user-to-server rejection row in client_test.go. The referenced issue and reported curl HTTP 201 result could not be independently verified, so server acceptance remains an asserted premise." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The design commits to a minimal allowlist change and precisely traces token classification, upload authorization, error propagation, and shared command integration. Its largest deficiency is unreliable version metadata plus a promised uploadTokenTypes comment rewrite that the implementation omits. The submitted diff contains the named uploadTokenTypes, NewUploader, uploadError, and commentable test integration points and is consistent with the main design. Claims that documentation contains no relevant text and that every create and edit path shares this gate could not be independently checked." + }, + "review": { + "completeness": 20, + "correctness": 17, + "specificity": 22, + "risk_awareness": 18, + "total": 77, + "notes": "The review identifies concrete test defects, including the contradictory rejection row, missing command coverage, authorization-header oracle, and fragile error-string assertion. Its largest correctness problem is the SRE claim that there is \"no new request,\" which directly contradicts the feature's purpose of allowing previously blocked ghu_ uploads to send a request. The findings map to specific patch changes in client_test.go and commentable_test.go rather than generic reviewer role-play. It does not flag the erroneous repository version or the implementation's omission of the LLD's promised comment update." + }, + "testing": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The plan has strong behavioral oracles for the returned URL, exact Authorization header, upload count, comment body, stdout, rejection status, and parsed server message. Its largest gap is the absence of executable affected-package test commands, while the manual procedure invokes gh rather than the repository-required built binary. The implementation contains the planned accepted and refused upload subtests and the shared comment table row with concrete expected output. The plan honestly reports that compilation, regression tests, and live server behavior were not verified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The patch implements the requested behavior through the narrow uploadTokenTypes change and adds unit, failure-path, authorization-header, and command-level coverage. The diff specifically changes internal/attachments/client.go, flips the existing TestNewUploader row, checks uploadError fields for HTTP 401, and exercises the shared comment path. Its largest deviation is omitting the LLD's promised comment rewrite, and its implementation metadata repeats the incorrect v2.10.0 base reference. Local source inspection and git apply checking were unavailable because the read-only shell failed before command execution, leaving exact applicability unverified but revealing no concrete defect in the submitted diff." + } + }, + "task_score": 84.8, + "verdict": "This is a strong, narrowly scoped solution with unusually concrete tests, but its repeated base-version mismatch and the review's false claim that no new request is introduced reduce confidence.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:09:13.178131Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 232323, + "cached_input_tokens": 152728, + "cache_write_input_tokens": 28739, + "output_tokens": 10806, + "reasoning_output_tokens": 9289 + }, + "duration_ms": 169980 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json new file mode 100644 index 00000000..0bdd3f52 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-merge-delete-branch-owner-case", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 19, + "specificity": 23, + "risk_awareness": 20, + "total": 84, + "notes": "The issue crisply identifies the case-sensitive crossRepoPR comparison, user-visible consequence, fork non-goal, and testable regression boundary. The cited guard and observed absence of the DELETE are consistent with v2.100.0 evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest defect is that acceptance requires a lowercase repos/owner/REPO DELETE while the LLD and patch intentionally require canonical repos/OWNER/REPO, and the output criterion does not specify the TTY condition. No independent task statement was supplied, so broader requirement coverage cannot be verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is strongly grounded in merge.go, merge_test.go, finder.go, ghrepo.IsSame, and BranchDeleteRemote, with an implementation-ready data flow and explicit alternatives. Its line-level account of the comparison, early-return guard, omitted isCrossRepository field, and existing EqualFold convention agrees with available v2.100.0 evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/pull/5915/files)) The canonical DELETE reconstruction is technically plausible but justified by an explicitly unverified hypothetical REST 404 rather than demonstrated repository behavior. It otherwise handles compatibility, fork behavior, empty owners, errors, rollout, and scope proportionately, with only minor non-TTY observability nuance omitted." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 21, + "total": 83, + "notes": "The strongest review work is its concrete identification that the default mock owner masks the regression and that ignored 404/422 responses could produce false success. The referenced comparison and deletion guard are supported by available source evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest weakness is elevating an unverified mixed-case REST-path rejection to MUST-FIX while several persona sections contribute little substantive stress testing. It also misses the issue's canonical-path contradiction and does not challenge the acceptance demand covering all three merge methods." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 89, + "notes": "The proposed regression test has concrete fixtures, HTTP and git stubs, exact output, and a canonical-path oracle that would fail both the old guard and a partial comparison-only fix. It also reuses the existing fork and same-repository tests for compatibility coverage, matching the documented failure path. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The main gap is that only --merge is exercised despite acceptance explicitly naming squash and rebase, and the unused REPO environment path is repository-external noise. Test execution was not required or claimed, but the unavailable local command runner prevented an independent compile or execution check." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the revised design end to end: EqualFold removes the false fork classification, the DELETE uses the API-returned owner, and the focused regression test verifies both behavior and request casing. The submitted contexts and symbols align with available v2.100.0 evidence for crossRepoPR, the early deletion guard, and the existing case-insensitive repository convention. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/pull/5915/files)) Its largest limitation is the additional canonical-repository construction being based on an unverified API-path concern, while the summary does not acknowledge that it contradicts the issue's lowercase-path acceptance criterion. The local sandbox failure prevented git apply --check, but the diff is internally coherent, gofmt-shaped, narrowly scoped, and its stated 58-insertion/2-deletion statistic matches the patch." + } + }, + "task_score": 87.0, + "verdict": "This is strong, implementation-ready work whose largest limitation is an unresolved cross-artifact DELETE-path/output contract combined with the absence of an independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:11:54.811502Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 171086, + "cached_input_tokens": 98511, + "cache_write_input_tokens": 27183, + "output_tokens": 10574, + "reasoning_output_tokens": 8671 + }, + "duration_ms": 161620 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json new file mode 100644 index 00000000..6e3d02ea --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-merge-delete-branch-owner-case", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "git", + "remotes", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 199.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 50, + "output_tokens": 53016, + "cache_read_tokens": 1444439, + "cache_write_tokens": 165500, + "total_tokens": 1663005, + "total_cost_usd": 0.067809379, + "latency_seconds": 265.5, + "num_turns": 24, + "generation_tokens_per_sec": 199.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 50, + "output_tokens": 53016, + "latency_seconds": 265.5, + "num_turns": 24, + "total_cost_usd": 0.067809379, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1444439, + "cache_creation_tokens": 165500, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:46:19.457270Z", + "run_ended_at": "2026-10-14T11:50:45.003039Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 50, + "output_tokens": 53016, + "cache_read_tokens": 1444439, + "cache_write_tokens": 165500, + "total_tokens": 1663005, + "total_cost_usd": 0.067809379, + "latency_seconds": 265.5, + "num_turns": 24, + "generation_tokens_per_sec": 199.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-merge-delete-branch-owner-case", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 19, + "specificity": 23, + "risk_awareness": 20, + "total": 84, + "notes": "The issue crisply identifies the case-sensitive crossRepoPR comparison, user-visible consequence, fork non-goal, and testable regression boundary. The cited guard and observed absence of the DELETE are consistent with v2.100.0 evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest defect is that acceptance requires a lowercase repos/owner/REPO DELETE while the LLD and patch intentionally require canonical repos/OWNER/REPO, and the output criterion does not specify the TTY condition. No independent task statement was supplied, so broader requirement coverage cannot be verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is strongly grounded in merge.go, merge_test.go, finder.go, ghrepo.IsSame, and BranchDeleteRemote, with an implementation-ready data flow and explicit alternatives. Its line-level account of the comparison, early-return guard, omitted isCrossRepository field, and existing EqualFold convention agrees with available v2.100.0 evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/pull/5915/files)) The canonical DELETE reconstruction is technically plausible but justified by an explicitly unverified hypothetical REST 404 rather than demonstrated repository behavior. It otherwise handles compatibility, fork behavior, empty owners, errors, rollout, and scope proportionately, with only minor non-TTY observability nuance omitted." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 21, + "total": 83, + "notes": "The strongest review work is its concrete identification that the default mock owner masks the regression and that ignored 404/422 responses could produce false success. The referenced comparison and deletion guard are supported by available source evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest weakness is elevating an unverified mixed-case REST-path rejection to MUST-FIX while several persona sections contribute little substantive stress testing. It also misses the issue's canonical-path contradiction and does not challenge the acceptance demand covering all three merge methods." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 89, + "notes": "The proposed regression test has concrete fixtures, HTTP and git stubs, exact output, and a canonical-path oracle that would fail both the old guard and a partial comparison-only fix. It also reuses the existing fork and same-repository tests for compatibility coverage, matching the documented failure path. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The main gap is that only --merge is exercised despite acceptance explicitly naming squash and rebase, and the unused REPO environment path is repository-external noise. Test execution was not required or claimed, but the unavailable local command runner prevented an independent compile or execution check." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the revised design end to end: EqualFold removes the false fork classification, the DELETE uses the API-returned owner, and the focused regression test verifies both behavior and request casing. The submitted contexts and symbols align with available v2.100.0 evidence for crossRepoPR, the early deletion guard, and the existing case-insensitive repository convention. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/pull/5915/files)) Its largest limitation is the additional canonical-repository construction being based on an unverified API-path concern, while the summary does not acknowledge that it contradicts the issue's lowercase-path acceptance criterion. The local sandbox failure prevented git apply --check, but the diff is internally coherent, gofmt-shaped, narrowly scoped, and its stated 58-insertion/2-deletion statistic matches the patch." + } + }, + "task_score": 87.0, + "verdict": "This is strong, implementation-ready work whose largest limitation is an unresolved cross-artifact DELETE-path/output contract combined with the absence of an independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:11:54.811502Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 171086, + "cached_input_tokens": 98511, + "cache_write_input_tokens": 27183, + "output_tokens": 10574, + "reasoning_output_tokens": 8671 + }, + "duration_ms": 161620 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json new file mode 100644 index 00000000..a0b98c67 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-view-json-number-on-issue", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 16, + "total": 81, + "notes": "The issue precisely identifies the number-only shortcut and supplies concrete, testable acceptance criteria. Its cited `finder.Find` and `findByNumber` integration points match the repository structure. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest deficiency is omitting the finder-wide effects on `pr checks --web`, `pr diff`, and commands that now require network access. No independent task statement was supplied, and the exact live GitHub error wording could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The LLD is implementation-ready, naming the exact finder branch, GraphQL operation, tests, callers, failure behavior, and rejected alternatives. The repository confirms that numeric lookups flow through `findByNumber`, while `diffRun` requests only `number` before fetching the REST diff. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) Its main flaw is claiming the empty-selector `refs/pull/N/head` route is unaffected, although removing the shortcut also adds validation and network failure exposure there. The document appropriately labels GitHub's exact error text as an unverified inference." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 24, + "total": 90, + "notes": "The review finds a real design blocker: the shared fix changes `checks --web`, and it also identifies the earlier `pr diff` failure point and new network dependency. Its `pr diff` analysis agrees with the actual sequence from finder lookup to REST fetch. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) The largest correctness blemish is the security claim that only digit-parsed selectors reach `findByNumber`; URL and branch-configuration paths can reach it too, and `strconv.Atoi` is not strictly digit-only. The recommendations are prioritized, actionable, and followed by an explicit blocker resolution." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 19, + "total": 81, + "notes": "The strongest portion is the exact finder test setup, including operation matchers, response bodies, concrete oracles, and explanations of why both cases fail before the fix. The unit command and `gh pr view` invocation are consistent with the repository's command surface. ([cli.github.com](https://cli.github.com/manual/gh_pr_view)) The main deficiency is that manual checks invoke an unspecified installed `gh` rather than the patched binary, and the promised `pr diff` behavioral coverage is absent. Collateral command behavior and network failures are discussed but not directly asserted, and live behavior was not verified." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 23, + "risk_awareness": 22, + "total": 93, + "notes": "The patch directly removes the erroneous shortcut, preserves the existing numeric lookup path, and adds both successful and issue-number GraphQL tests. Its target files, symbols, and surrounding control flow match the repository, including the existing `findByNumber` API and the number-only `diffRun` caller. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) The largest limitation is that the summary overstates its selected package list as all finder callers and supplies no direct regression test for the documented command-level side effects. Patch application and claimed test execution could not be independently repeated, but no source-level defect or design deviation is apparent." + } + }, + "task_score": 87.4, + "verdict": "This is a strong and likely correct fix, with the largest limitation being incomplete issue and test coverage of the finder-wide network and command-behavior changes amid the missing independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:14:18.859608Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 134559, + "cached_input_tokens": 70609, + "cache_write_input_tokens": 24005, + "output_tokens": 9977, + "reasoning_output_tokens": 8416 + }, + "duration_ms": 144035 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json new file mode 100644 index 00000000..2a0ae380 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-view-json-number-on-issue", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "json-output", + "api", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 154.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 66, + "output_tokens": 42948, + "cache_read_tokens": 1933529, + "cache_write_tokens": 149911, + "total_tokens": 2126454, + "total_cost_usd": 0.06551024150000001, + "latency_seconds": 277.4, + "num_turns": 32, + "generation_tokens_per_sec": 154.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 66, + "output_tokens": 42948, + "latency_seconds": 277.4, + "num_turns": 32, + "total_cost_usd": 0.06551024150000001, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1933529, + "cache_creation_tokens": 149911, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:50:48.434133Z", + "run_ended_at": "2026-10-14T11:55:25.860094Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 66, + "output_tokens": 42948, + "cache_read_tokens": 1933529, + "cache_write_tokens": 149911, + "total_tokens": 2126454, + "total_cost_usd": 0.06551024150000001, + "latency_seconds": 277.4, + "num_turns": 32, + "generation_tokens_per_sec": 154.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-view-json-number-on-issue", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 16, + "total": 81, + "notes": "The issue precisely identifies the number-only shortcut and supplies concrete, testable acceptance criteria. Its cited `finder.Find` and `findByNumber` integration points match the repository structure. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest deficiency is omitting the finder-wide effects on `pr checks --web`, `pr diff`, and commands that now require network access. No independent task statement was supplied, and the exact live GitHub error wording could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The LLD is implementation-ready, naming the exact finder branch, GraphQL operation, tests, callers, failure behavior, and rejected alternatives. The repository confirms that numeric lookups flow through `findByNumber`, while `diffRun` requests only `number` before fetching the REST diff. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) Its main flaw is claiming the empty-selector `refs/pull/N/head` route is unaffected, although removing the shortcut also adds validation and network failure exposure there. The document appropriately labels GitHub's exact error text as an unverified inference." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 24, + "total": 90, + "notes": "The review finds a real design blocker: the shared fix changes `checks --web`, and it also identifies the earlier `pr diff` failure point and new network dependency. Its `pr diff` analysis agrees with the actual sequence from finder lookup to REST fetch. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) The largest correctness blemish is the security claim that only digit-parsed selectors reach `findByNumber`; URL and branch-configuration paths can reach it too, and `strconv.Atoi` is not strictly digit-only. The recommendations are prioritized, actionable, and followed by an explicit blocker resolution." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 19, + "total": 81, + "notes": "The strongest portion is the exact finder test setup, including operation matchers, response bodies, concrete oracles, and explanations of why both cases fail before the fix. The unit command and `gh pr view` invocation are consistent with the repository's command surface. ([cli.github.com](https://cli.github.com/manual/gh_pr_view)) The main deficiency is that manual checks invoke an unspecified installed `gh` rather than the patched binary, and the promised `pr diff` behavioral coverage is absent. Collateral command behavior and network failures are discussed but not directly asserted, and live behavior was not verified." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 23, + "risk_awareness": 22, + "total": 93, + "notes": "The patch directly removes the erroneous shortcut, preserves the existing numeric lookup path, and adds both successful and issue-number GraphQL tests. Its target files, symbols, and surrounding control flow match the repository, including the existing `findByNumber` API and the number-only `diffRun` caller. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) The largest limitation is that the summary overstates its selected package list as all finder callers and supplies no direct regression test for the documented command-level side effects. Patch application and claimed test execution could not be independently repeated, but no source-level defect or design deviation is apparent." + } + }, + "task_score": 87.4, + "verdict": "This is a strong and likely correct fix, with the largest limitation being incomplete issue and test coverage of the finder-wide network and command-behavior changes amid the missing independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:14:18.859608Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 134559, + "cached_input_tokens": 70609, + "cache_write_input_tokens": 24005, + "output_tokens": 9977, + "reasoning_output_tokens": 8416 + }, + "duration_ms": 144035 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json new file mode 100644 index 00000000..57ca2eef --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json @@ -0,0 +1,64 @@ +{ + "task": "project-item-add-output-non-tty", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly defines outputs for non-TTY, TTY, and JSON modes, distinguishes the project-item ID from the content ID, and sets useful non-goals. The documented command options exist, and the v2.98.0 release identifies this exact non-TTY fix. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [github.com](https://github.com/cli/cli/releases)) Its largest weakness is the unsupported 2-to-9 retry statistic and an overstated claim that callers cannot recognize success despite the exit status. No independent task statement establishes whether the precise wording or parsing contract was originally required." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The design commits to a minimal data flow, exact byte-level outputs, exporter precedence, mutation-response ID selection, and a concrete two-file implementation. It also addresses compatibility, post-mutation write failures, data exposure, rollout, and deferred sibling commands proportionately. Its broad claim that every project printResults implementation is TTY-gated is not substantiated, and the non-null ID guarantee is appropriately admitted to be unverified. The release record corroborates the overall change, but local sandbox failure prevented independent symbol-by-symbol source verification. ([github.com](https://github.com/cli/cli/releases))" + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 22, + "total": 85, + "notes": "The review's strongest work is identifying the stdout compatibility break, output-contract requirement, empty-ID assumption, and non-zero exit after a post-mutation write failure. Findings are prioritized and paired with concrete resolutions rather than generic approvals. Its largest deficiency is unclear chronology: several reviewers criticize omissions already present in the submitted LLD, reducing confidence that it independently stressed the final design. Claims that the ID is not logged elsewhere and that sibling commands share identical behavior were not independently demonstrated." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The plan provides strong oracles, especially the distinct content ID and mutation item ID that detect printing the wrong identifier. Its command names and documented flags are valid, and the stated project token scope is correct. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [cli.github.com](https://cli.github.com/manual/gh_project)) The largest gaps are no explicit TTY-plus-JSON test and no mocked unit failure case; the optional live mutation also lacks cleanup or rerun isolation. Tests were not executed, so compilation and the claimed existing JSON fixture remain unverified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the design end to end: the production hunk preserves TTY output, prints item.Id for non-TTY output, and adds a test whose mutation ID differs from the content ID. The official v2.98.0 release records the same fix through PR 14056, strongly corroborating the target and behavior. ([github.com](https://github.com/cli/cli/releases)) The test uses assert.NoError despite the repository instruction requiring require for error checks, and the summary's claimed +5/-2 production diffstat does not match the displayed +2/-1 hunk. Independent git apply checking was unavailable because local command execution could not initialize its sandbox, so the candidate's applicability claim could not be fully verified." + } + }, + "task_score": 87.2, + "verdict": "This is a strong, implementation-ready submission whose main limitation is incomplete independent verification and a few test and repository-convention gaps.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:17:08.955578Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 210765, + "cached_input_tokens": 123417, + "cache_write_input_tokens": 29946, + "output_tokens": 11120, + "reasoning_output_tokens": 9731 + }, + "duration_ms": 167111 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json new file mode 100644 index 00000000..2bce143a --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "project-item-add-output-non-tty", + "repo": "https://github.com/cli/cli", + "ref": "v2.97.0", + "complexity": "low", + "tags": [ + "project", + "go", + "cli", + "tty", + "automation", + "fixed-in-2.98.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 184.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 40, + "output_tokens": 33027, + "cache_read_tokens": 1118508, + "cache_write_tokens": 105341, + "total_tokens": 1256916, + "total_cost_usd": 0.0449572255, + "latency_seconds": 179.0, + "num_turns": 19, + "generation_tokens_per_sec": 184.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 40, + "output_tokens": 33027, + "latency_seconds": 179.0, + "num_turns": 19, + "total_cost_usd": 0.0449572255, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1118508, + "cache_creation_tokens": 105341, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:55:29.259893Z", + "run_ended_at": "2026-10-14T11:58:28.304470Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 40, + "output_tokens": 33027, + "cache_read_tokens": 1118508, + "cache_write_tokens": 105341, + "total_tokens": 1256916, + "total_cost_usd": 0.0449572255, + "latency_seconds": 179.0, + "num_turns": 19, + "generation_tokens_per_sec": 184.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "project-item-add-output-non-tty", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly defines outputs for non-TTY, TTY, and JSON modes, distinguishes the project-item ID from the content ID, and sets useful non-goals. The documented command options exist, and the v2.98.0 release identifies this exact non-TTY fix. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [github.com](https://github.com/cli/cli/releases)) Its largest weakness is the unsupported 2-to-9 retry statistic and an overstated claim that callers cannot recognize success despite the exit status. No independent task statement establishes whether the precise wording or parsing contract was originally required." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The design commits to a minimal data flow, exact byte-level outputs, exporter precedence, mutation-response ID selection, and a concrete two-file implementation. It also addresses compatibility, post-mutation write failures, data exposure, rollout, and deferred sibling commands proportionately. Its broad claim that every project printResults implementation is TTY-gated is not substantiated, and the non-null ID guarantee is appropriately admitted to be unverified. The release record corroborates the overall change, but local sandbox failure prevented independent symbol-by-symbol source verification. ([github.com](https://github.com/cli/cli/releases))" + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 22, + "total": 85, + "notes": "The review's strongest work is identifying the stdout compatibility break, output-contract requirement, empty-ID assumption, and non-zero exit after a post-mutation write failure. Findings are prioritized and paired with concrete resolutions rather than generic approvals. Its largest deficiency is unclear chronology: several reviewers criticize omissions already present in the submitted LLD, reducing confidence that it independently stressed the final design. Claims that the ID is not logged elsewhere and that sibling commands share identical behavior were not independently demonstrated." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The plan provides strong oracles, especially the distinct content ID and mutation item ID that detect printing the wrong identifier. Its command names and documented flags are valid, and the stated project token scope is correct. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [cli.github.com](https://cli.github.com/manual/gh_project)) The largest gaps are no explicit TTY-plus-JSON test and no mocked unit failure case; the optional live mutation also lacks cleanup or rerun isolation. Tests were not executed, so compilation and the claimed existing JSON fixture remain unverified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the design end to end: the production hunk preserves TTY output, prints item.Id for non-TTY output, and adds a test whose mutation ID differs from the content ID. The official v2.98.0 release records the same fix through PR 14056, strongly corroborating the target and behavior. ([github.com](https://github.com/cli/cli/releases)) The test uses assert.NoError despite the repository instruction requiring require for error checks, and the summary's claimed +5/-2 production diffstat does not match the displayed +2/-1 hunk. Independent git apply checking was unavailable because local command execution could not initialize its sandbox, so the candidate's applicability claim could not be fully verified." + } + }, + "task_score": 87.2, + "verdict": "This is a strong, implementation-ready submission whose main limitation is incomplete independent verification and a few test and repository-convention gaps.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:17:08.955578Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 210765, + "cached_input_tokens": 123417, + "cache_write_input_tokens": 29946, + "output_tokens": 11120, + "reasoning_output_tokens": 9731 + }, + "duration_ms": 167111 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json new file mode 100644 index 00000000..be69b4e0 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json @@ -0,0 +1,64 @@ +{ + "task": "project-item-edit-by-name", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 15, + "specificity": 23, + "risk_awareness": 19, + "total": 79, + "notes": "The issue defines a concrete name-based interface, typed value behavior, compatibility expectations, ambiguity handling, and useful non-goals. Its largest defect is the incorrect claim that `gh project item-edit --id X` should report that `id` is missing; the existing `MarkFlagRequired(\"id\")` only produces that error when `--id` is absent. It also requires invalid numbers to fail before any request, although the field type must first be learned through `ProjectFields`. No independent task statement was supplied, so requirements beyond the identifier and internally consistent artifacts could not be verified." + }, + "lld": { + "completeness": 20, + "correctness": 16, + "specificity": 23, + "risk_awareness": 20, + "total": 79, + "notes": "The design is unusually concrete about `item_edit.go`, `queries.go`, `NewOwner`, `ProjectFields`, `ProjectField.Options`, validation order, mutations, and data-type conversion. Its proposed raw-string error `required flag(s) \\\"id\\\" not set` would preserve literal backslashes rather than the existing Cobra message, and value parsing occurs after project queries rather than before every request as the issue demands. It explicitly recognizes pagination, duplicate names, finite-number validation, unsupported field types, the 100-membership limit, and uncertain archived-item behavior. The live API behavior for archived memberships remains unresolved, and the design accepts a potentially false non-membership result instead of providing complete lookup pagination." + }, + "review": { + "completeness": 17, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review catches two real design defects: the 30-field default limit in `ProjectFields` and misleading handling when exactly 100 project memberships are returned. It also gives concrete security and request-cost analysis tied to GraphQL variables, `resource(url:)`, and the existing `item-add` flow. However, it misses the malformed legacy error string, the contradiction over pre-request number validation, and the requirement for conflict errors to identify the actual supplied flags. It provides blocker status but no clear overall verdict, and it accepts the membership truncation and archived-item uncertainty without a robust fallback." + }, + "testing": { + "completeness": 21, + "correctness": 17, + "specificity": 22, + "risk_awareness": 20, + "total": 80, + "notes": "The plan provides concrete cases and oracles for matching, type conversion, flag validation, GraphQL query shape, pagination, request counts, compatibility, help output, and manual live checks. One command expectation is inconsistent with the patch: the plan expects `invalid number: abc`, while the implementation returns `invalid project number \"abc\": must be a positive integer`; it also confusingly describes the positional project number as missing `--number`. It does not include an integrated oracle proving invalid typed values cause no requests, despite that explicit issue requirement. Archived-item behavior is appropriately identified as unverified, but most backward-compatibility coverage is stated broadly rather than mapped to concrete cases." + }, + "implementation": { + "completeness": 16, + "correctness": 14, + "specificity": 20, + "risk_awareness": 15, + "total": 65, + "notes": "The patch implements the principal flow end to end: new flags, owner and project resolution, case-insensitive field and option lookup, typed conversion, URL membership lookup, and reuse of `updateItemValues`. The legacy missing-ID error is incorrectly constructed as a raw string containing escaped quotes, and the generic conflict error does not identify which supplied flags conflict as required. Tests omit command parsing, `applyNamedValue`, field resolution, GraphQL lookup, mutation behavior, and legacy regression coverage; they also pin complete error strings and use `assert` for error checks contrary to the submitted review and repository instructions. The summary honestly discloses the missing tests and 100-membership limit, while archived-item behavior and successful patch compilation remain unverified." + } + }, + "task_score": 75.4, + "verdict": "The submission offers a strong, repository-specific design and testing blueprint, but the implementation is under-tested and contains an explicit legacy-error regression plus unresolved membership limitations.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:20:04.698939Z", + "repo_ref": "v2.96.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 148674, + "cached_input_tokens": 81161, + "cache_write_input_tokens": 27990, + "output_tokens": 11380, + "reasoning_output_tokens": 9276 + }, + "duration_ms": 172635 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json new file mode 100644 index 00000000..b747a90a --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "project-item-edit-by-name", + "repo": "https://github.com/cli/cli", + "ref": "v2.96.0", + "complexity": "high", + "tags": [ + "project", + "go", + "cli", + "graphql", + "fixed-in-2.97.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 186.4, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 134, + "output_tokens": 120947, + "cache_read_tokens": 3674860, + "cache_write_tokens": 274494, + "total_tokens": 4070435, + "total_cost_usd": 0.14470197499999995, + "latency_seconds": 648.7, + "num_turns": 64, + "generation_tokens_per_sec": 186.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 134, + "output_tokens": 120947, + "latency_seconds": 648.7, + "num_turns": 64, + "total_cost_usd": 0.14470197499999995, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3674860, + "cache_creation_tokens": 274494, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T13:22:26.933303Z", + "run_ended_at": "2026-10-14T13:33:15.630849Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 134, + "output_tokens": 120947, + "cache_read_tokens": 3674860, + "cache_write_tokens": 274494, + "total_tokens": 4070435, + "total_cost_usd": 0.14470197499999995, + "latency_seconds": 648.7, + "num_turns": 64, + "generation_tokens_per_sec": 186.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "project-item-edit-by-name", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 15, + "specificity": 23, + "risk_awareness": 19, + "total": 79, + "notes": "The issue defines a concrete name-based interface, typed value behavior, compatibility expectations, ambiguity handling, and useful non-goals. Its largest defect is the incorrect claim that `gh project item-edit --id X` should report that `id` is missing; the existing `MarkFlagRequired(\"id\")` only produces that error when `--id` is absent. It also requires invalid numbers to fail before any request, although the field type must first be learned through `ProjectFields`. No independent task statement was supplied, so requirements beyond the identifier and internally consistent artifacts could not be verified." + }, + "lld": { + "completeness": 20, + "correctness": 16, + "specificity": 23, + "risk_awareness": 20, + "total": 79, + "notes": "The design is unusually concrete about `item_edit.go`, `queries.go`, `NewOwner`, `ProjectFields`, `ProjectField.Options`, validation order, mutations, and data-type conversion. Its proposed raw-string error `required flag(s) \\\"id\\\" not set` would preserve literal backslashes rather than the existing Cobra message, and value parsing occurs after project queries rather than before every request as the issue demands. It explicitly recognizes pagination, duplicate names, finite-number validation, unsupported field types, the 100-membership limit, and uncertain archived-item behavior. The live API behavior for archived memberships remains unresolved, and the design accepts a potentially false non-membership result instead of providing complete lookup pagination." + }, + "review": { + "completeness": 17, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review catches two real design defects: the 30-field default limit in `ProjectFields` and misleading handling when exactly 100 project memberships are returned. It also gives concrete security and request-cost analysis tied to GraphQL variables, `resource(url:)`, and the existing `item-add` flow. However, it misses the malformed legacy error string, the contradiction over pre-request number validation, and the requirement for conflict errors to identify the actual supplied flags. It provides blocker status but no clear overall verdict, and it accepts the membership truncation and archived-item uncertainty without a robust fallback." + }, + "testing": { + "completeness": 21, + "correctness": 17, + "specificity": 22, + "risk_awareness": 20, + "total": 80, + "notes": "The plan provides concrete cases and oracles for matching, type conversion, flag validation, GraphQL query shape, pagination, request counts, compatibility, help output, and manual live checks. One command expectation is inconsistent with the patch: the plan expects `invalid number: abc`, while the implementation returns `invalid project number \"abc\": must be a positive integer`; it also confusingly describes the positional project number as missing `--number`. It does not include an integrated oracle proving invalid typed values cause no requests, despite that explicit issue requirement. Archived-item behavior is appropriately identified as unverified, but most backward-compatibility coverage is stated broadly rather than mapped to concrete cases." + }, + "implementation": { + "completeness": 16, + "correctness": 14, + "specificity": 20, + "risk_awareness": 15, + "total": 65, + "notes": "The patch implements the principal flow end to end: new flags, owner and project resolution, case-insensitive field and option lookup, typed conversion, URL membership lookup, and reuse of `updateItemValues`. The legacy missing-ID error is incorrectly constructed as a raw string containing escaped quotes, and the generic conflict error does not identify which supplied flags conflict as required. Tests omit command parsing, `applyNamedValue`, field resolution, GraphQL lookup, mutation behavior, and legacy regression coverage; they also pin complete error strings and use `assert` for error checks contrary to the submitted review and repository instructions. The summary honestly discloses the missing tests and 100-membership limit, while archived-item behavior and successful patch compilation remain unverified." + } + }, + "task_score": 75.4, + "verdict": "The submission offers a strong, repository-specific design and testing blueprint, but the implementation is under-tested and contains an explicit legacy-error regression plus unresolved membership limitations.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:20:04.698939Z", + "repo_ref": "v2.96.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 148674, + "cached_input_tokens": 81161, + "cache_write_input_tokens": 27990, + "output_tokens": 11380, + "reasoning_output_tokens": 9276 + }, + "duration_ms": 172635 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json new file mode 100644 index 00000000..8335f190 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json @@ -0,0 +1,64 @@ +{ + "task": "release-create-404-scope-blame", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue precisely defines the malformed scope comparison, observable outcomes, compatibility cases, and explicit non-goals. Its acceptance criteria align with GitHub's documented workflow-scope 404 behavior and the repository's subsequent fix. ([docs.github.com](https://docs.github.com/pt/rest/releases/releases), [github.com](https://github.com/cli/cli/releases)) The largest weakness is the unsupported overstatement that this affects every 404 or almost every OAuth token. With no independent task statement, broader requirement coverage and the claimed live reproduction remain unverifiable." + }, + "lld": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD names concrete functions, files, branches, outputs, and test inputs, leaving no major implementation decision unresolved. The proposed split-and-trim comparison is technically sound and matches the fix recorded for v2.98.0. ([github.com](https://github.com/cli/cli/releases)) Its largest factual defect is claiming the release code is the repository's only scope splitter, despite established comma splitting and TrimSpace handling in the API scope-suggestion code. ([github.com](https://github.com/cli/cli/issues/9403)) Exact line references and clean patch applicability could not be independently checked because the local command sandbox failed." + }, + "review": { + "completeness": 20, + "correctness": 19, + "specificity": 21, + "risk_awareness": 20, + "total": 80, + "notes": "The review concretely identifies why the existing comma-only test misses the regression and recommends spaced positive, negative, and exact-match cases. It also evaluates the relevant UX, scripting, and authorization consequences rather than treating the header as an access-control decision. Its largest weakness is that it mostly validates its own design, misses the LLD's false claim about repository-wide parsing conventions, and labels explicitly out-of-scope 404 guidance as a should-fix before deferring it. Exact source-line and import claims could not be locally reverified, although the central fix is corroborated by the release record. ([github.com](https://github.com/cli/cli/releases))" + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 82, + "notes": "The plan supplies concrete commands, exact headers, expected errors and stderr, compatibility cases, and a regression check that would fail against the old parser. Its test names and oracles agree with the submitted patch, including absent-header and workflowish cases. The largest omission is the repository-required go test ./... and make lint validation; package-only go vet is not an equivalent substitute. Manual switching between two same-host tokens is underspecified, and the final checklist inconsistently describes which live-token scenario requires workflow scope." + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The patch implements the complete requested behavior by trimming each comma-delimited scope before an exact workflow comparison while preserving the empty-header branch. It targets tokenHasWorkflowScope and adds meaningful parser and command-level tests for both 404 outcomes; the repository's v2.98.0 notes independently confirm this exact fix area. ([github.com](https://github.com/cli/cli/releases)) The added tests are somewhat redundant for a one-line correction but remain relevant and contain concrete oracles, with no apparent production-code defect. The claimed git apply --check and exact baseline context could not be rerun because the local shell sandbox was unavailable, while the summary accurately discloses that tests and lint were not executed." + } + }, + "task_score": 87.0, + "verdict": "This is a strong, likely correct submission whose main limitations are an inaccurate LLD claim about existing scope parsing and a testing plan that omits the repository-mandated full test and lint commands.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:22:49.745964Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 160704, + "cached_input_tokens": 79801, + "cache_write_input_tokens": 29034, + "output_tokens": 11125, + "reasoning_output_tokens": 9405 + }, + "duration_ms": 165034 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json new file mode 100644 index 00000000..afb3f613 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "release-create-404-scope-blame", + "repo": "https://github.com/cli/cli", + "ref": "v2.97.0", + "complexity": "low", + "tags": [ + "release", + "go", + "cli", + "auth", + "error-handling", + "fixed-in-2.98.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 175.4, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 43349, + "cache_read_tokens": 1647521, + "cache_write_tokens": 146796, + "total_tokens": 1837724, + "total_cost_usd": 0.06215551100000002, + "latency_seconds": 247.2, + "num_turns": 27, + "generation_tokens_per_sec": 175.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 58, + "output_tokens": 43349, + "latency_seconds": 247.2, + "num_turns": 27, + "total_cost_usd": 0.06215551100000002, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1647521, + "cache_creation_tokens": 146796, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:58:31.731834Z", + "run_ended_at": "2026-10-14T12:02:38.947330Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 43349, + "cache_read_tokens": 1647521, + "cache_write_tokens": 146796, + "total_tokens": 1837724, + "total_cost_usd": 0.06215551100000002, + "latency_seconds": 247.2, + "num_turns": 27, + "generation_tokens_per_sec": 175.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "release-create-404-scope-blame", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue precisely defines the malformed scope comparison, observable outcomes, compatibility cases, and explicit non-goals. Its acceptance criteria align with GitHub's documented workflow-scope 404 behavior and the repository's subsequent fix. ([docs.github.com](https://docs.github.com/pt/rest/releases/releases), [github.com](https://github.com/cli/cli/releases)) The largest weakness is the unsupported overstatement that this affects every 404 or almost every OAuth token. With no independent task statement, broader requirement coverage and the claimed live reproduction remain unverifiable." + }, + "lld": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD names concrete functions, files, branches, outputs, and test inputs, leaving no major implementation decision unresolved. The proposed split-and-trim comparison is technically sound and matches the fix recorded for v2.98.0. ([github.com](https://github.com/cli/cli/releases)) Its largest factual defect is claiming the release code is the repository's only scope splitter, despite established comma splitting and TrimSpace handling in the API scope-suggestion code. ([github.com](https://github.com/cli/cli/issues/9403)) Exact line references and clean patch applicability could not be independently checked because the local command sandbox failed." + }, + "review": { + "completeness": 20, + "correctness": 19, + "specificity": 21, + "risk_awareness": 20, + "total": 80, + "notes": "The review concretely identifies why the existing comma-only test misses the regression and recommends spaced positive, negative, and exact-match cases. It also evaluates the relevant UX, scripting, and authorization consequences rather than treating the header as an access-control decision. Its largest weakness is that it mostly validates its own design, misses the LLD's false claim about repository-wide parsing conventions, and labels explicitly out-of-scope 404 guidance as a should-fix before deferring it. Exact source-line and import claims could not be locally reverified, although the central fix is corroborated by the release record. ([github.com](https://github.com/cli/cli/releases))" + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 82, + "notes": "The plan supplies concrete commands, exact headers, expected errors and stderr, compatibility cases, and a regression check that would fail against the old parser. Its test names and oracles agree with the submitted patch, including absent-header and workflowish cases. The largest omission is the repository-required go test ./... and make lint validation; package-only go vet is not an equivalent substitute. Manual switching between two same-host tokens is underspecified, and the final checklist inconsistently describes which live-token scenario requires workflow scope." + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The patch implements the complete requested behavior by trimming each comma-delimited scope before an exact workflow comparison while preserving the empty-header branch. It targets tokenHasWorkflowScope and adds meaningful parser and command-level tests for both 404 outcomes; the repository's v2.98.0 notes independently confirm this exact fix area. ([github.com](https://github.com/cli/cli/releases)) The added tests are somewhat redundant for a one-line correction but remain relevant and contain concrete oracles, with no apparent production-code defect. The claimed git apply --check and exact baseline context could not be rerun because the local shell sandbox was unavailable, while the summary accurately discloses that tests and lint were not executed." + } + }, + "task_score": 87.0, + "verdict": "This is a strong, likely correct submission whose main limitations are an inaccurate LLD claim about existing scope parsing and a testing plan that omits the repository-mandated full test and lint commands.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:22:49.745964Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 160704, + "cached_input_tokens": 79801, + "cache_write_input_tokens": 29034, + "output_tokens": 11125, + "reasoning_output_tokens": 9405 + }, + "duration_ms": 165034 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json new file mode 100644 index 00000000..f1c7009a --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json @@ -0,0 +1,64 @@ +{ + "task": "release-verify-sha1-prefix", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The issue clearly defines the SHA-1/SHA-256 behavior for both release commands, with testable acceptance criteria and useful non-goals. The submitted patch contexts confirm the hard-coded `sha1` constructor calls in `verify.go` and `verify_asset.go`. Its main limitation is that no independent task statement establishes every claimed requirement, particularly malformed-ref rejection and GitHub's SHA-256 rollout behavior. It addresses compatibility and invalid input well, but attestation-service support for SHA-256 remains unverified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is unusually concrete about data flow, constructor changes, error handling, and the two command integration points. Its cited `NewDigestedArtifactForRelease` call sites and test seams agree with the submitted source diff. The largest deficiency is minor internal inconsistency: the file-change table proposes `release_test.go`, while the testing and implementation artifacts create `artifact_test.go`; the claimed repository-wide caller search could not be independently rerun. It responsibly identifies malformed refs, backward compatibility, policy coverage, rollout, and unknown server indexing." + }, + "review": { + "completeness": 19, + "correctness": 19, + "specificity": 20, + "risk_awareness": 20, + "total": 78, + "notes": "The strongest review finding is the concrete demand for length-plus-hex validation and explicit coverage of `FetchParams.Digest` and the verifier-policy gap. It names actionable symbols such as `releaseRefAlgorithm`, `BuildDigestPolicyOption`, and `api.MockClient.OnGetByDigest`. Much of the multi-role review repeats approval rather than independently challenging the constructor API change, output coverage, or the test-file inconsistency. Its security framing overstates malformed API data as a query-injection concern, while the genuinely material SHA-256 service-indexing question remains unresolved." + }, + "testing": { + "completeness": 19, + "correctness": 19, + "specificity": 23, + "risk_awareness": 20, + "total": 81, + "notes": "The plan gives concrete table inputs, test names, mock setup, and meaningful oracles for both digest algorithms and rejected refs. The implementation diff contains the described `OnGetByDigest` captures and assertions that malformed refs avoid the attestation call. Its largest gap is failure to directly assert the user-visible resolved-tag output or capture the artifact passed to the verifier, and it omits the repository-prescribed `go test ./...` and `make lint` checks. It also presents scoped package tests as final verification even though implementation.md says the full package trees were not rerun after the last test-table edit." + }, + "implementation": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 20, + "total": 84, + "notes": "The patch implements the central behavior in both commands, derives the algorithm once, validates malformed refs, and adds focused SHA-1/SHA-256 and negative tests. Its diff contexts consistently target the existing constructor and command call sites, and the error handling follows the surrounding Go pattern. The main coverage deficiency is that no test observes the artifact received by `AttVerifier` or directly checks the resolved-tag output, although the shared object makes the intended propagation plausible. The summary incorrectly says the old constructor behavior was hard-coded `sha256` rather than `sha1`, and independent `git apply --check` execution was unavailable because the local command sandbox could not initialize." + } + }, + "task_score": 84.2, + "verdict": "This is a strong, implementation-ready submission whose largest limitation is incomplete verification of the user-visible and verifier-policy propagation paths.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:25:16.931281Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 134875, + "cached_input_tokens": 82010, + "cache_write_input_tokens": 29605, + "output_tokens": 9055, + "reasoning_output_tokens": 7686 + }, + "duration_ms": 147172 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json new file mode 100644 index 00000000..761d550e --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "release-verify-sha1-prefix", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "release", + "go", + "cli", + "attestation", + "sha256", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 148.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 136, + "output_tokens": 75849, + "cache_read_tokens": 4076105, + "cache_write_tokens": 274038, + "total_tokens": 4426128, + "total_cost_usd": 0.12424928999999997, + "latency_seconds": 510.9, + "num_turns": 65, + "generation_tokens_per_sec": 148.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 136, + "output_tokens": 75849, + "latency_seconds": 510.9, + "num_turns": 65, + "total_cost_usd": 0.12424928999999997, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4076105, + "cache_creation_tokens": 274038, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:31:37.249354Z", + "run_ended_at": "2026-10-14T12:40:08.193050Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 136, + "output_tokens": 75849, + "cache_read_tokens": 4076105, + "cache_write_tokens": 274038, + "total_tokens": 4426128, + "total_cost_usd": 0.12424928999999997, + "latency_seconds": 510.9, + "num_turns": 65, + "generation_tokens_per_sec": 148.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "release-verify-sha1-prefix", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The issue clearly defines the SHA-1/SHA-256 behavior for both release commands, with testable acceptance criteria and useful non-goals. The submitted patch contexts confirm the hard-coded `sha1` constructor calls in `verify.go` and `verify_asset.go`. Its main limitation is that no independent task statement establishes every claimed requirement, particularly malformed-ref rejection and GitHub's SHA-256 rollout behavior. It addresses compatibility and invalid input well, but attestation-service support for SHA-256 remains unverified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is unusually concrete about data flow, constructor changes, error handling, and the two command integration points. Its cited `NewDigestedArtifactForRelease` call sites and test seams agree with the submitted source diff. The largest deficiency is minor internal inconsistency: the file-change table proposes `release_test.go`, while the testing and implementation artifacts create `artifact_test.go`; the claimed repository-wide caller search could not be independently rerun. It responsibly identifies malformed refs, backward compatibility, policy coverage, rollout, and unknown server indexing." + }, + "review": { + "completeness": 19, + "correctness": 19, + "specificity": 20, + "risk_awareness": 20, + "total": 78, + "notes": "The strongest review finding is the concrete demand for length-plus-hex validation and explicit coverage of `FetchParams.Digest` and the verifier-policy gap. It names actionable symbols such as `releaseRefAlgorithm`, `BuildDigestPolicyOption`, and `api.MockClient.OnGetByDigest`. Much of the multi-role review repeats approval rather than independently challenging the constructor API change, output coverage, or the test-file inconsistency. Its security framing overstates malformed API data as a query-injection concern, while the genuinely material SHA-256 service-indexing question remains unresolved." + }, + "testing": { + "completeness": 19, + "correctness": 19, + "specificity": 23, + "risk_awareness": 20, + "total": 81, + "notes": "The plan gives concrete table inputs, test names, mock setup, and meaningful oracles for both digest algorithms and rejected refs. The implementation diff contains the described `OnGetByDigest` captures and assertions that malformed refs avoid the attestation call. Its largest gap is failure to directly assert the user-visible resolved-tag output or capture the artifact passed to the verifier, and it omits the repository-prescribed `go test ./...` and `make lint` checks. It also presents scoped package tests as final verification even though implementation.md says the full package trees were not rerun after the last test-table edit." + }, + "implementation": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 20, + "total": 84, + "notes": "The patch implements the central behavior in both commands, derives the algorithm once, validates malformed refs, and adds focused SHA-1/SHA-256 and negative tests. Its diff contexts consistently target the existing constructor and command call sites, and the error handling follows the surrounding Go pattern. The main coverage deficiency is that no test observes the artifact received by `AttVerifier` or directly checks the resolved-tag output, although the shared object makes the intended propagation plausible. The summary incorrectly says the old constructor behavior was hard-coded `sha256` rather than `sha1`, and independent `git apply --check` execution was unavailable because the local command sandbox could not initialize." + } + }, + "task_score": 84.2, + "verdict": "This is a strong, implementation-ready submission whose largest limitation is incomplete verification of the user-visible and verifier-policy propagation paths.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:25:16.931281Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 134875, + "cached_input_tokens": 82010, + "cache_write_input_tokens": 29605, + "output_tokens": 9055, + "reasoning_output_tokens": 7686 + }, + "duration_ms": 147172 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json new file mode 100644 index 00000000..ba0fa397 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-clone-ghe-ssh-ca-url", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The issue provides a strong, testable contract covering API-provided SSH URLs, fork upstreams, HTTPS compatibility, fallback behavior, and wiki clones. Its named symbols and root cause match the submitted diff context and the public issue, including `GitHubRepo`, `FormatRemoteURL`, and `cloneRun`. ([github.com](https://github.com/cli/cli/issues/14344), [github.com](https://github.com/cli/cli/blob/v2.54.0/internal/ghrepo/repo.go)) The explicit exclusions keep the change bounded. Its main limitation is the absent independent task statement and limited treatment of malformed API URL risks." + }, + "lld": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The design commits to an exact data flow, helper contract, two integration points, fallback semantics, and wiki transformation. The submitted source context supports the cited files and interfaces, while the public issue confirms that the repository model already supports `SSHURL`. ([github.com](https://github.com/cli/cli/issues/14344)) Its largest gap is that the proposed implementation verification covers only the primary clone path, not the parent URL or leading-dash branch. The older-GHES fallback discussion also assumes the GraphQL field exists in the server schema; an unsupported field would fail the query rather than decode empty." + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 21, + "risk_awareness": 18, + "total": 77, + "notes": "The strongest review finding is the concrete leading-dash option-injection case, with a narrowly scoped guard and clear consequence. It also examines pointer interface satisfaction, shared parent fragments, request count, fallback behavior, and command output. Its largest deficiency is approving the patch without noticing that the new mock does not prove `sshUrl` was requested and that no automated test covers the required parent-upstream behavior. The comparison to existing HTTPS trust is imprecise because `FormatRemoteURL` synthesizes that URL rather than accepting an arbitrary API URL." + }, + "testing": { + "completeness": 14, + "correctness": 13, + "specificity": 19, + "risk_awareness": 15, + "total": 61, + "notes": "The plan gives concrete commands, fixture data, command regexes, compatibility cases, and realistic manual GHE.com steps. Its critical oracle flaw is that the GraphQL mock returns `sshUrl` regardless of the submitted query, so the new test could pass if the `queries_repo.go` change were removed. It explicitly leaves fork-upstream and leading-dash behavior untested, and existing HTTPS tests without `sshUrl` do not exercise the claimed negative case with that field present. Wiki behavior with an API SSH URL and query-shape verification are also missing." + }, + "implementation": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 16, + "total": 77, + "notes": "The focused patch implements the stated data flow by requesting `sshUrl`, selecting it for SSH at both clone sites, preserving fallback behavior, and retaining the existing wiki transformation. The diff context and public issue support the target symbols and underlying defect, but local forward applicability, formatting, and compilation could not be independently rechecked because the repository shell failed before executing commands. ([github.com](https://github.com/cli/cli/issues/14344), [github.com](https://github.com/cli/cli/blob/v2.54.0/internal/ghrepo/repo.go)) The largest defect is test coverage: the test does not verify the query field and omits parent, malformed URL, and API-backed wiki cases; it also uses `t.Fatalf` instead of the repository-mandated `require` error assertion. The summary accurately discloses unexecuted tests, although its claim that a skill forbids running them is unsupported and conflicts with the repository guidance." + } + }, + "task_score": 77.8, + "verdict": "The submission presents a technically sound and well-scoped fix, but its implementation confidence is materially limited by a GraphQL mock that cannot catch omission of the new field and by missing automated coverage for the upstream path.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:28:18.057776Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 134641, + "cached_input_tokens": 61746, + "cache_write_input_tokens": 23395, + "output_tokens": 11812, + "reasoning_output_tokens": 9819 + }, + "duration_ms": 181113 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json new file mode 100644 index 00000000..d01a8c7f --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json @@ -0,0 +1,158 @@ +{ + "task": "repo-clone-ghe-ssh-ca-url", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "medium", + "tags": [ + "repo", + "go", + "cli", + "git", + "ssh", + "ghe", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 174.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 154, + "output_tokens": 109073, + "cache_read_tokens": 4333998, + "cache_write_tokens": 355108, + "total_tokens": 4798333, + "total_cost_usd": 0.15650841799999998, + "latency_seconds": 652.5, + "num_turns": 73, + "generation_tokens_per_sec": 174.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 154, + "output_tokens": 109073, + "latency_seconds": 652.5, + "num_turns": 73, + "total_cost_usd": 0.15650841799999998, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4333998, + "cache_creation_tokens": 355108, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:14:18.203141Z", + "run_ended_at": "2026-10-14T12:18:55.577371Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 154, + "output_tokens": 109073, + "cache_read_tokens": 4333998, + "cache_write_tokens": 355108, + "total_tokens": 4798333, + "total_cost_usd": 0.15650841799999998, + "latency_seconds": 652.5, + "num_turns": 73, + "generation_tokens_per_sec": 174.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "agent_invocations": 2, + "evaluation": { + "task": "repo-clone-ghe-ssh-ca-url", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The issue provides a strong, testable contract covering API-provided SSH URLs, fork upstreams, HTTPS compatibility, fallback behavior, and wiki clones. Its named symbols and root cause match the submitted diff context and the public issue, including `GitHubRepo`, `FormatRemoteURL`, and `cloneRun`. ([github.com](https://github.com/cli/cli/issues/14344), [github.com](https://github.com/cli/cli/blob/v2.54.0/internal/ghrepo/repo.go)) The explicit exclusions keep the change bounded. Its main limitation is the absent independent task statement and limited treatment of malformed API URL risks." + }, + "lld": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The design commits to an exact data flow, helper contract, two integration points, fallback semantics, and wiki transformation. The submitted source context supports the cited files and interfaces, while the public issue confirms that the repository model already supports `SSHURL`. ([github.com](https://github.com/cli/cli/issues/14344)) Its largest gap is that the proposed implementation verification covers only the primary clone path, not the parent URL or leading-dash branch. The older-GHES fallback discussion also assumes the GraphQL field exists in the server schema; an unsupported field would fail the query rather than decode empty." + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 21, + "risk_awareness": 18, + "total": 77, + "notes": "The strongest review finding is the concrete leading-dash option-injection case, with a narrowly scoped guard and clear consequence. It also examines pointer interface satisfaction, shared parent fragments, request count, fallback behavior, and command output. Its largest deficiency is approving the patch without noticing that the new mock does not prove `sshUrl` was requested and that no automated test covers the required parent-upstream behavior. The comparison to existing HTTPS trust is imprecise because `FormatRemoteURL` synthesizes that URL rather than accepting an arbitrary API URL." + }, + "testing": { + "completeness": 14, + "correctness": 13, + "specificity": 19, + "risk_awareness": 15, + "total": 61, + "notes": "The plan gives concrete commands, fixture data, command regexes, compatibility cases, and realistic manual GHE.com steps. Its critical oracle flaw is that the GraphQL mock returns `sshUrl` regardless of the submitted query, so the new test could pass if the `queries_repo.go` change were removed. It explicitly leaves fork-upstream and leading-dash behavior untested, and existing HTTPS tests without `sshUrl` do not exercise the claimed negative case with that field present. Wiki behavior with an API SSH URL and query-shape verification are also missing." + }, + "implementation": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 16, + "total": 77, + "notes": "The focused patch implements the stated data flow by requesting `sshUrl`, selecting it for SSH at both clone sites, preserving fallback behavior, and retaining the existing wiki transformation. The diff context and public issue support the target symbols and underlying defect, but local forward applicability, formatting, and compilation could not be independently rechecked because the repository shell failed before executing commands. ([github.com](https://github.com/cli/cli/issues/14344), [github.com](https://github.com/cli/cli/blob/v2.54.0/internal/ghrepo/repo.go)) The largest defect is test coverage: the test does not verify the query field and omits parent, malformed URL, and API-backed wiki cases; it also uses `t.Fatalf` instead of the repository-mandated `require` error assertion. The summary accurately discloses unexecuted tests, although its claim that a skill forbids running them is unsupported and conflicts with the repository guidance." + } + }, + "task_score": 77.8, + "verdict": "The submission presents a technically sound and well-scoped fix, but its implementation confidence is materially limited by a GraphQL mock that cannot catch omission of the new field and by missing automated coverage for the upstream path.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:28:18.057776Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 134641, + "cached_input_tokens": 61746, + "cache_write_input_tokens": 23395, + "output_tokens": 11812, + "reasoning_output_tokens": 9819 + }, + "duration_ms": 181113 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json new file mode 100644 index 00000000..96e6b64c --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-create-reject-empty-name", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the delayed failure, intended early-exit behavior, affected interactive flows, and explicit non-goals. Its root-cause account names interactiveRepoNameAndOwner, splitNameAndOwner, interactiveRepoInfo, and createFromScratch consistently with the submitted source context. The largest limitation is that no independent task statement establishes whitespace-only input or all three flows as requirements, so those remain internally derived scope. It nevertheless addresses the relevant compatibility and UX risks without broadening the production change." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD commits to a minimal check immediately after prompter.Input and before splitNameAndOwner, with concrete propagation through interactiveRepoInfo to the three callers. It accurately preserves nonblank input, avoids additional dependencies, and explicitly treats non-interactive behavior, trimming, and re-prompting as non-goals. Its largest gap is accepting only from-scratch regression tests while claiming behavior across template and local flows, leaving that integration contract unpinned. The exact baseline SHA and cited line numbers are not independently established by the supplied task statement, though the patch context supports the named symbols and locations." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 19, + "total": 80, + "notes": "The review is concrete about helper placement, whitespace semantics, non-interactive behavior, and the missing template/local tests rather than merely approving the design. It correctly identifies that strict prompt stubs make later prompts observable and that the early return prevents userAndOrgs from issuing an HTTP request. Its largest weakness is dismissing the acknowledged cross-flow test gap without proposing a small direct helper test or another durable contract check. Several SRE and security sections are proportionate to this trivial change but add little analysis beyond confirming the absence of new surfaces." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 82, + "notes": "The plan gives valid Go package commands, exact Test_createRun row names, concrete inputs and error oracles, a full-package regression run, and a useful negative control. Strict unexpected-prompt branches and absent HTTP stubs provide meaningful evidence that failure occurs before later interaction or network access. The largest gap is the explicitly untested template and local flows, plus no new automated valid owner-prefixed case despite discussing that compatibility requirement. The negative-control prediction is also overly specific because unchanged code may first fail in userAndOrgs on an unregistered HTTP request rather than at a later prompt." + }, + "implementation": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The production hunk adds the required blank-name check directly after prompter.Input and before splitNameAndOwner, implementing the LLD while leaving valid values unchanged. The two Test_createRun rows cover empty and whitespace-only values and use strict prompt behavior to catch failure to exit early; the referenced strings and errors imports are already expected by the surrounding source. The largest limitation is the absence of dedicated template and local regression rows, although shared helper placement makes the production behavior consistent across those callers. The summary honestly discloses that gap and the unexecuted verification, and no material deviation from the design is apparent in the diff." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, repository-specific submission with a correct minimal implementation; its main limitation is that template and local interactive flows are covered only by shared-code reasoning rather than dedicated regression tests.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:30:54.243257Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 127805, + "cached_input_tokens": 70566, + "cache_write_input_tokens": 24077, + "output_tokens": 9363, + "reasoning_output_tokens": 7792 + }, + "duration_ms": 156172 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json new file mode 100644 index 00000000..ccfb63be --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "repo-create-reject-empty-name", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "trivial", + "tags": [ + "repo", + "go", + "cli", + "interactive-prompts", + "validation", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 168.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 52, + "output_tokens": 37924, + "cache_read_tokens": 1422640, + "cache_write_tokens": 133614, + "total_tokens": 1594230, + "total_cost_usd": 0.05488488499999999, + "latency_seconds": 224.8, + "num_turns": 24, + "generation_tokens_per_sec": 168.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 52, + "output_tokens": 37924, + "latency_seconds": 224.8, + "num_turns": 24, + "total_cost_usd": 0.05488488499999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1422640, + "cache_creation_tokens": 133614, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:27:43.313123Z", + "run_ended_at": "2026-10-14T11:31:28.103354Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 52, + "output_tokens": 37924, + "cache_read_tokens": 1422640, + "cache_write_tokens": 133614, + "total_tokens": 1594230, + "total_cost_usd": 0.05488488499999999, + "latency_seconds": 224.8, + "num_turns": 24, + "generation_tokens_per_sec": 168.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-create-reject-empty-name", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the delayed failure, intended early-exit behavior, affected interactive flows, and explicit non-goals. Its root-cause account names interactiveRepoNameAndOwner, splitNameAndOwner, interactiveRepoInfo, and createFromScratch consistently with the submitted source context. The largest limitation is that no independent task statement establishes whitespace-only input or all three flows as requirements, so those remain internally derived scope. It nevertheless addresses the relevant compatibility and UX risks without broadening the production change." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD commits to a minimal check immediately after prompter.Input and before splitNameAndOwner, with concrete propagation through interactiveRepoInfo to the three callers. It accurately preserves nonblank input, avoids additional dependencies, and explicitly treats non-interactive behavior, trimming, and re-prompting as non-goals. Its largest gap is accepting only from-scratch regression tests while claiming behavior across template and local flows, leaving that integration contract unpinned. The exact baseline SHA and cited line numbers are not independently established by the supplied task statement, though the patch context supports the named symbols and locations." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 19, + "total": 80, + "notes": "The review is concrete about helper placement, whitespace semantics, non-interactive behavior, and the missing template/local tests rather than merely approving the design. It correctly identifies that strict prompt stubs make later prompts observable and that the early return prevents userAndOrgs from issuing an HTTP request. Its largest weakness is dismissing the acknowledged cross-flow test gap without proposing a small direct helper test or another durable contract check. Several SRE and security sections are proportionate to this trivial change but add little analysis beyond confirming the absence of new surfaces." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 82, + "notes": "The plan gives valid Go package commands, exact Test_createRun row names, concrete inputs and error oracles, a full-package regression run, and a useful negative control. Strict unexpected-prompt branches and absent HTTP stubs provide meaningful evidence that failure occurs before later interaction or network access. The largest gap is the explicitly untested template and local flows, plus no new automated valid owner-prefixed case despite discussing that compatibility requirement. The negative-control prediction is also overly specific because unchanged code may first fail in userAndOrgs on an unregistered HTTP request rather than at a later prompt." + }, + "implementation": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The production hunk adds the required blank-name check directly after prompter.Input and before splitNameAndOwner, implementing the LLD while leaving valid values unchanged. The two Test_createRun rows cover empty and whitespace-only values and use strict prompt behavior to catch failure to exit early; the referenced strings and errors imports are already expected by the surrounding source. The largest limitation is the absence of dedicated template and local regression rows, although shared helper placement makes the production behavior consistent across those callers. The summary honestly discloses that gap and the unexecuted verification, and no material deviation from the design is apparent in the diff." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, repository-specific submission with a correct minimal implementation; its main limitation is that template and local interactive flows are covered only by shared-code reasoning rather than dedicated regression tests.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:30:54.243257Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 127805, + "cached_input_tokens": 70566, + "cache_write_input_tokens": 24077, + "output_tokens": 9363, + "reasoning_output_tokens": 7792 + }, + "duration_ms": 156172 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json new file mode 100644 index 00000000..1835381d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-set-default-unset-without-repo", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue gives a reproducible command, exact current error, narrow desired behavior, and testable acceptance criteria. The submitted patch corroborates the named guard, flags, and preserved error text. Its strongest scope decision is explicitly deferring the ignored positional argument while preserving existing behavior. The main evidence gap is the absent independent task statement and inability to verify the linked issue's current labels or acceptance criteria." + }, + "lld": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is grounded in concrete files and symbols, including NewCmdSetDefault, CanPrompt, UnsetRemoteResolution, and the exact guard change. Its behavior table, data flow, test seam, and handling of valid versus invalid ignored arguments are implementation-ready and consistent with the patch. It identifies the realistic local configuration integrity risk and explains why argument rejection is deferred. The main gap is that it proposes a live acceptance-test change without documenting the repository's authorization and credential constraints." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 22, + "total": 85, + "notes": "The review identifies a concrete wrong-target configuration risk from ignored positional arguments and traces it to argument resolution preceding the guard. It prioritizes the issue, explains the compatibility tradeoff, and records an actionable deferral rather than expanding scope. Some findings claim the LLD does not document silence or argument risk even though the submitted revised LLD does, and the statement about default GH_* environments is unsupported and immaterial. Despite that process-version ambiguity, the final resolutions and verdict are specific and defensible." + }, + "testing": { + "completeness": 19, + "correctness": 11, + "specificity": 20, + "risk_awareness": 9, + "total": 59, + "notes": "The plan provides precise command-level cases, expected option values, retained-error oracles, run-path coverage, and an end-to-end scenario. Its acceptance command omits the required -tags=acceptance flag, uses an unexplained GH_ACCEPTANCE_GROUP variable, and incorrectly describes the suite as fixture-server based even though repository documentation requires credentials for a real GitHub instance. ([github.com](https://github.com/cli/cli/blob/trunk/acceptance/README.md)) The backward-compatibility table also claims non-TTY repository arguments and ignored --unset arguments are covered while describing only TTY or intentionally absent tests. The optional smoke test runs inside the source checkout and can alter its .git/config, contrary to the repository's isolation requirement." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the central behavior end to end: it exempts UnsetMode from the non-interactive repository guard and adds focused regression and preservation tests. The condition, non-TTY setup, option assertions, and acceptance sequence are internally consistent with the surrounding context shown in the diff. Its largest defect is omitting the LLD's plain no-argument acceptance assertion while claiming no deviation; the verification account also displays an acceptance command without the required build tag, although it honestly states that the txtar scenario was unexecuted. Independent git apply and source checks could not be completed because the read-only command sandbox failed before execution, which is an evidence limitation rather than a candidate defect." + } + }, + "task_score": 82.0, + "verdict": "The submission delivers a strong, narrowly correct implementation and well-grounded design, but its testing plan contains materially incorrect acceptance-test instructions and an unsafe manual validation step.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:34:03.327603Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 153047, + "cached_input_tokens": 91022, + "cache_write_input_tokens": 24080, + "output_tokens": 12278, + "reasoning_output_tokens": 10438 + }, + "duration_ms": 189072 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json new file mode 100644 index 00000000..a0548494 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "repo-set-default-unset-without-repo", + "repo": "https://github.com/cli/cli", + "ref": "v2.101.0", + "complexity": "trivial", + "tags": [ + "repo", + "go", + "cli", + "tty", + "flags", + "fixed-in-2.102.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 167.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 82, + "output_tokens": 55685, + "cache_read_tokens": 2098063, + "cache_write_tokens": 150485, + "total_tokens": 2304315, + "total_cost_usd": 0.07440615049999999, + "latency_seconds": 331.9, + "num_turns": 38, + "generation_tokens_per_sec": 167.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 82, + "output_tokens": 55685, + "latency_seconds": 331.9, + "num_turns": 38, + "total_cost_usd": 0.07440615049999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2098063, + "cache_creation_tokens": 150485, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T11:34:51.300235Z", + "run_ended_at": "2026-10-14T11:40:23.250359Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 82, + "output_tokens": 55685, + "cache_read_tokens": 2098063, + "cache_write_tokens": 150485, + "total_tokens": 2304315, + "total_cost_usd": 0.07440615049999999, + "latency_seconds": 331.9, + "num_turns": 38, + "generation_tokens_per_sec": 167.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-set-default-unset-without-repo", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue gives a reproducible command, exact current error, narrow desired behavior, and testable acceptance criteria. The submitted patch corroborates the named guard, flags, and preserved error text. Its strongest scope decision is explicitly deferring the ignored positional argument while preserving existing behavior. The main evidence gap is the absent independent task statement and inability to verify the linked issue's current labels or acceptance criteria." + }, + "lld": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is grounded in concrete files and symbols, including NewCmdSetDefault, CanPrompt, UnsetRemoteResolution, and the exact guard change. Its behavior table, data flow, test seam, and handling of valid versus invalid ignored arguments are implementation-ready and consistent with the patch. It identifies the realistic local configuration integrity risk and explains why argument rejection is deferred. The main gap is that it proposes a live acceptance-test change without documenting the repository's authorization and credential constraints." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 22, + "total": 85, + "notes": "The review identifies a concrete wrong-target configuration risk from ignored positional arguments and traces it to argument resolution preceding the guard. It prioritizes the issue, explains the compatibility tradeoff, and records an actionable deferral rather than expanding scope. Some findings claim the LLD does not document silence or argument risk even though the submitted revised LLD does, and the statement about default GH_* environments is unsupported and immaterial. Despite that process-version ambiguity, the final resolutions and verdict are specific and defensible." + }, + "testing": { + "completeness": 19, + "correctness": 11, + "specificity": 20, + "risk_awareness": 9, + "total": 59, + "notes": "The plan provides precise command-level cases, expected option values, retained-error oracles, run-path coverage, and an end-to-end scenario. Its acceptance command omits the required -tags=acceptance flag, uses an unexplained GH_ACCEPTANCE_GROUP variable, and incorrectly describes the suite as fixture-server based even though repository documentation requires credentials for a real GitHub instance. ([github.com](https://github.com/cli/cli/blob/trunk/acceptance/README.md)) The backward-compatibility table also claims non-TTY repository arguments and ignored --unset arguments are covered while describing only TTY or intentionally absent tests. The optional smoke test runs inside the source checkout and can alter its .git/config, contrary to the repository's isolation requirement." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the central behavior end to end: it exempts UnsetMode from the non-interactive repository guard and adds focused regression and preservation tests. The condition, non-TTY setup, option assertions, and acceptance sequence are internally consistent with the surrounding context shown in the diff. Its largest defect is omitting the LLD's plain no-argument acceptance assertion while claiming no deviation; the verification account also displays an acceptance command without the required build tag, although it honestly states that the txtar scenario was unexecuted. Independent git apply and source checks could not be completed because the read-only command sandbox failed before execution, which is an evidence limitation rather than a candidate defect." + } + }, + "task_score": 82.0, + "verdict": "The submission delivers a strong, narrowly correct implementation and well-grounded design, but its testing plan contains materially incorrect acceptance-test instructions and an unsafe manual validation step.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:34:03.327603Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 153047, + "cached_input_tokens": 91022, + "cache_write_input_tokens": 24080, + "output_tokens": 12278, + "reasoning_output_tokens": 10438 + }, + "duration_ms": 189072 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json new file mode 100644 index 00000000..a141ad36 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-sync-checked-out-worktree", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The issue gives unusually testable acceptance criteria for clean, dirty, divergent, forced, current-worktree, and unchecked-out branch cases. Its proposed fast-forward and forced-reset behavior agrees with the documented `gh repo sync` semantics. ([cli.github.com](https://cli.github.com/manual/gh_repo_sync)) The largest gaps are the unmentioned force-mode TOCTOU risk and the unusable unquoted stash hint when a worktree path contains spaces or shell metacharacters. The claimed scratch reproduction could not be independently verified, and no separate task statement was supplied." + }, + "lld": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The design commits to concrete integration points in `executeLocalRepoSync`, `git.Client`, `gitExecuter`, and `mockGitClient`, including the crucial decision to resolve per-worktree `FETCH_HEAD` before crossing worktrees. Its largest technical weakness is parsing newline-delimited porcelain instead of using `-z`, while also emitting an unquoted `git -C ` recovery command despite claiming paths with spaces are supported. It also treats locked worktrees as expected `git -C` failures without establishing that behavior, and the review itself leaves `Client.Command` use of `RepoDir` unverified. The official release record confirms the asserted v2.98.0 base and commit prefix, but direct source and scratch-path verification were unavailable. ([github.com](https://github.com/cli/cli/releases))" + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 86, + "notes": "The review's strongest contribution is identifying the per-worktree `FETCH_HEAD` defect and preventing unnecessary SHA resolution on the unchanged `UpdateBranch` path. It also gives concrete consequences and resolutions for dirty worktrees, parser matching, contextual errors, and the force-mode race. Its largest deficiency is claiming that a deleted-worktree test was added when `testing.md` contains no such scenario, while the security review overlooks that the displayed stash command is unsafe or invalid without path quoting. The load-bearing assertion that `RepoDir` scopes commands remains a question rather than a repository-verified conclusion." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 79, + "notes": "The real-git fixture has strong behavioral oracles: exact branch SHAs, clean status, file contents, retained current branch, refusal text, and unchanged refs. The largest coverage gaps are the explicitly required untracked-file case and the deleted-worktree case promised by the review; only a tracked modification exercises dirtiness. The proposed `gofmt -l git pkg/cmd/repo/sync` command incorrectly passes directories, and the verification list omits the repository-required `go test ./...` and `make lint`. In the submitted tests, error checks repeatedly use `assert` rather than the repository-mandated `require`, allowing execution to continue after prerequisite failures." + }, + "implementation": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 19, + "total": 80, + "notes": "The patch implements the central design end to end: it discovers the owning worktree, resolves `FETCH_HEAD` locally, runs dirty checks and merge/reset through a copied client, and preserves `UpdateBranch` for unchecked-out branches. The official release evidence confirms that the claimed base is v2.98.0 at `a255baf`, but the stated `git apply --check` result and compilation could not be independently reproduced because direct checkout execution was unavailable. ([github.com](https://github.com/cli/cli/releases)) The largest implementation defects are the unquoted worktree path in the copyable stash hint and extensive `assert` use for error prerequisites contrary to repository conventions; coverage also omits an actual untracked-file case. The summary additionally overstates error wrapping because `WorktreeForBranch` and `ResolveRef` failures are returned unchanged, although the core branch-update flow is internally consistent with the LLD." + } + }, + "task_score": 84.8, + "verdict": "This is a strong, implementation-ready submission whose main limitations are incomplete robustness testing, unsafe recovery-command formatting, and several repository-convention violations in the tests.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:37:46.397805Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 308280, + "cached_input_tokens": 216076, + "cache_write_input_tokens": 39732, + "output_tokens": 15010, + "reasoning_output_tokens": 12528 + }, + "duration_ms": 223057 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json new file mode 100644 index 00000000..65e5ebfd --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "repo-sync-checked-out-worktree", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "high", + "tags": [ + "repo", + "go", + "git", + "worktree", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 187.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 110, + "output_tokens": 116417, + "cache_read_tokens": 3017645, + "cache_write_tokens": 271095, + "total_tokens": 3405267, + "total_cost_usd": 0.1345111075, + "latency_seconds": 621.0, + "num_turns": 53, + "generation_tokens_per_sec": 187.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 110, + "output_tokens": 116417, + "latency_seconds": 621.0, + "num_turns": 53, + "total_cost_usd": 0.1345111075, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3017645, + "cache_creation_tokens": 271095, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:55:10.669267Z", + "run_ended_at": "2026-10-14T13:05:31.740154Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 110, + "output_tokens": 116417, + "cache_read_tokens": 3017645, + "cache_write_tokens": 271095, + "total_tokens": 3405267, + "total_cost_usd": 0.1345111075, + "latency_seconds": 621.0, + "num_turns": 53, + "generation_tokens_per_sec": 187.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-sync-checked-out-worktree", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The issue gives unusually testable acceptance criteria for clean, dirty, divergent, forced, current-worktree, and unchecked-out branch cases. Its proposed fast-forward and forced-reset behavior agrees with the documented `gh repo sync` semantics. ([cli.github.com](https://cli.github.com/manual/gh_repo_sync)) The largest gaps are the unmentioned force-mode TOCTOU risk and the unusable unquoted stash hint when a worktree path contains spaces or shell metacharacters. The claimed scratch reproduction could not be independently verified, and no separate task statement was supplied." + }, + "lld": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The design commits to concrete integration points in `executeLocalRepoSync`, `git.Client`, `gitExecuter`, and `mockGitClient`, including the crucial decision to resolve per-worktree `FETCH_HEAD` before crossing worktrees. Its largest technical weakness is parsing newline-delimited porcelain instead of using `-z`, while also emitting an unquoted `git -C ` recovery command despite claiming paths with spaces are supported. It also treats locked worktrees as expected `git -C` failures without establishing that behavior, and the review itself leaves `Client.Command` use of `RepoDir` unverified. The official release record confirms the asserted v2.98.0 base and commit prefix, but direct source and scratch-path verification were unavailable. ([github.com](https://github.com/cli/cli/releases))" + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 86, + "notes": "The review's strongest contribution is identifying the per-worktree `FETCH_HEAD` defect and preventing unnecessary SHA resolution on the unchanged `UpdateBranch` path. It also gives concrete consequences and resolutions for dirty worktrees, parser matching, contextual errors, and the force-mode race. Its largest deficiency is claiming that a deleted-worktree test was added when `testing.md` contains no such scenario, while the security review overlooks that the displayed stash command is unsafe or invalid without path quoting. The load-bearing assertion that `RepoDir` scopes commands remains a question rather than a repository-verified conclusion." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 79, + "notes": "The real-git fixture has strong behavioral oracles: exact branch SHAs, clean status, file contents, retained current branch, refusal text, and unchanged refs. The largest coverage gaps are the explicitly required untracked-file case and the deleted-worktree case promised by the review; only a tracked modification exercises dirtiness. The proposed `gofmt -l git pkg/cmd/repo/sync` command incorrectly passes directories, and the verification list omits the repository-required `go test ./...` and `make lint`. In the submitted tests, error checks repeatedly use `assert` rather than the repository-mandated `require`, allowing execution to continue after prerequisite failures." + }, + "implementation": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 19, + "total": 80, + "notes": "The patch implements the central design end to end: it discovers the owning worktree, resolves `FETCH_HEAD` locally, runs dirty checks and merge/reset through a copied client, and preserves `UpdateBranch` for unchecked-out branches. The official release evidence confirms that the claimed base is v2.98.0 at `a255baf`, but the stated `git apply --check` result and compilation could not be independently reproduced because direct checkout execution was unavailable. ([github.com](https://github.com/cli/cli/releases)) The largest implementation defects are the unquoted worktree path in the copyable stash hint and extensive `assert` use for error prerequisites contrary to repository conventions; coverage also omits an actual untracked-file case. The summary additionally overstates error wrapping because `WorktreeForBranch` and `ResolveRef` failures are returned unchanged, although the core branch-update flow is internally consistent with the LLD." + } + }, + "task_score": 84.8, + "verdict": "This is a strong, implementation-ready submission whose main limitations are incomplete robustness testing, unsafe recovery-command formatting, and several repository-convention violations in the tests.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:37:46.397805Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 308280, + "cached_input_tokens": 216076, + "cache_write_input_tokens": 39732, + "output_tokens": 15010, + "reasoning_output_tokens": 12528 + }, + "duration_ms": 223057 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/run-summary.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/run-summary.json new file mode 100644 index 00000000..da726199 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/run-summary.json @@ -0,0 +1,1436 @@ +{ + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "scope": "cli-cli-v1", + "provider": "bedrock", + "ref": null, + "refs": [ + "v2.100.0", + "v2.101.0", + "v2.92.0", + "v2.93.0", + "v2.94.0", + "v2.95.0", + "v2.96.0", + "v2.97.0", + "v2.98.0" + ], + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": false, + "scored_zero_missing_artifacts": [ + "github-issue.md", + "lld.md", + "review.md", + "testing.md" + ], + "evaluated_at": "2026-10-14T13:47:44.669482Z" + }, + "num_tasks": 21, + "num_scored": 19, + "num_failed": 2, + "failed_tasks": [ + "auth-clipboard-config-default", + "skill-install-nested-discovery" + ], + "mean_task_score_excl_failed": 80.82, + "mean_cost_usd_excl_failed": 0.09, + "tasks": [ + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 24, + "input_tokens": 50, + "output_tokens": 37854, + "total_tokens": 1616521, + "latency_seconds": 196.4, + "total_cost_usd": 0.05208904800000001, + "result_subtype": "success", + "task_score": 87.6, + "cache_read_tokens": 1468743, + "cache_write_tokens": 109874, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 192.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue precisely defines the interactive declined-install behavior, reproduction, exclusions, and exact stderr oracle. Its acceptance criteria explicitly preserve cmdutil.SilentError and prohibit invoking the download path. The largest limitation is evidentiary: no independent task statement establishes whether excluding the sibling non-TTY warning is the intended upstream scope. The named pkg/cmd/copilot paths and runCopilot behavior agree with the submitted patch context, but the referenced issue could not be independently validated." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The LLD commits to a minimal format-string change and gives an implementation-ready test setup using CanPrompt, PrompterMock, findCopilotBinaryFunc, and an explicit HttpClient failure stub. It addresses output bytes, control flow, compatibility, color stability, rollback, and the intentionally unchanged non-TTY branch. Its main weakness is excessive confidence in exact line and API claims that could not all be independently checked because repository shell access failed. The submitted diff nevertheless matches the described runCopilot branch and test integration points without an apparent architectural inconsistency." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review identifies a concrete weakness in the initial test design: relying on a nil HttpClient would not clearly assert that download setup was skipped. It also examines the residual non-TTY defect and byte-exact stderr compatibility, then records explicit resolutions and a defensible verdict. The largest deficiency is that several reviewer sections repeat the same findings, while the proposed stdout assertion is answered affirmatively but neither triaged nor implemented. Some UX claims about wrapper behavior and the prompt's rendering were not independently verifiable from the available source evidence." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 21, + "risk_awareness": 19, + "total": 77, + "notes": "The central unit test has meaningful oracles for the exact newline, SilentError, and absence of HttpClient use, and the plan states the expected pre-fix failure. It also includes focused package commands, a regression run, and a manual terminal check. The largest gap is claiming that the accepted-install and non-TTY paths are covered by the existing suite without identifying a test that actually exercises the latter. It also omits the repository-mandated final go test ./... and make lint checks and uses an environment-specific REPO_DIR that was not verified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The patch implements the requested behavior directly by adding one newline and adds a focused regression test that configures both TTY flags, disables color, restores findCopilotBinaryFunc, and rejects HttpClient use. The summary accurately describes the two touched files, the residual non-TTY behavior, and the one-byte compatibility change. No evident logic, API, security, or caller regression appears in the submitted hunks. However, local shell failure prevented an independent git apply --check or compilation check, so clean applicability and exact surrounding source remain an evidence gap rather than a demonstrated defect." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "low", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 32, + "input_tokens": 66, + "output_tokens": 42948, + "total_tokens": 2126454, + "latency_seconds": 277.4, + "total_cost_usd": 0.06551024150000001, + "result_subtype": "success", + "task_score": 87.4, + "cache_read_tokens": 1933529, + "cache_write_tokens": 149911, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 154.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 16, + "total": 81, + "notes": "The issue precisely identifies the number-only shortcut and supplies concrete, testable acceptance criteria. Its cited `finder.Find` and `findByNumber` integration points match the repository structure. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest deficiency is omitting the finder-wide effects on `pr checks --web`, `pr diff`, and commands that now require network access. No independent task statement was supplied, and the exact live GitHub error wording could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The LLD is implementation-ready, naming the exact finder branch, GraphQL operation, tests, callers, failure behavior, and rejected alternatives. The repository confirms that numeric lookups flow through `findByNumber`, while `diffRun` requests only `number` before fetching the REST diff. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) Its main flaw is claiming the empty-selector `refs/pull/N/head` route is unaffected, although removing the shortcut also adds validation and network failure exposure there. The document appropriately labels GitHub's exact error text as an unverified inference." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 24, + "total": 90, + "notes": "The review finds a real design blocker: the shared fix changes `checks --web`, and it also identifies the earlier `pr diff` failure point and new network dependency. Its `pr diff` analysis agrees with the actual sequence from finder lookup to REST fetch. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) The largest correctness blemish is the security claim that only digit-parsed selectors reach `findByNumber`; URL and branch-configuration paths can reach it too, and `strconv.Atoi` is not strictly digit-only. The recommendations are prioritized, actionable, and followed by an explicit blocker resolution." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 19, + "total": 81, + "notes": "The strongest portion is the exact finder test setup, including operation matchers, response bodies, concrete oracles, and explanations of why both cases fail before the fix. The unit command and `gh pr view` invocation are consistent with the repository's command surface. ([cli.github.com](https://cli.github.com/manual/gh_pr_view)) The main deficiency is that manual checks invoke an unspecified installed `gh` rather than the patched binary, and the promised `pr diff` behavioral coverage is absent. Collateral command behavior and network failures are discussed but not directly asserted, and live behavior was not verified." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 23, + "risk_awareness": 22, + "total": 93, + "notes": "The patch directly removes the erroneous shortcut, preserves the existing numeric lookup path, and adds both successful and issue-number GraphQL tests. Its target files, symbols, and surrounding control flow match the repository, including the existing `findByNumber` API and the number-only `diffRun` caller. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) The largest limitation is that the summary overstates its selected package list as all finder callers and supplies no direct regression test for the documented command-level side effects. Patch application and claimed test execution could not be independently repeated, but no source-level defect or design deviation is apparent." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "ref": "v2.97.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 19, + "input_tokens": 40, + "output_tokens": 33027, + "total_tokens": 1256916, + "latency_seconds": 179.0, + "total_cost_usd": 0.0449572255, + "result_subtype": "success", + "task_score": 87.2, + "cache_read_tokens": 1118508, + "cache_write_tokens": 105341, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 184.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly defines outputs for non-TTY, TTY, and JSON modes, distinguishes the project-item ID from the content ID, and sets useful non-goals. The documented command options exist, and the v2.98.0 release identifies this exact non-TTY fix. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [github.com](https://github.com/cli/cli/releases)) Its largest weakness is the unsupported 2-to-9 retry statistic and an overstated claim that callers cannot recognize success despite the exit status. No independent task statement establishes whether the precise wording or parsing contract was originally required." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The design commits to a minimal data flow, exact byte-level outputs, exporter precedence, mutation-response ID selection, and a concrete two-file implementation. It also addresses compatibility, post-mutation write failures, data exposure, rollout, and deferred sibling commands proportionately. Its broad claim that every project printResults implementation is TTY-gated is not substantiated, and the non-null ID guarantee is appropriately admitted to be unverified. The release record corroborates the overall change, but local sandbox failure prevented independent symbol-by-symbol source verification. ([github.com](https://github.com/cli/cli/releases))" + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 22, + "total": 85, + "notes": "The review's strongest work is identifying the stdout compatibility break, output-contract requirement, empty-ID assumption, and non-zero exit after a post-mutation write failure. Findings are prioritized and paired with concrete resolutions rather than generic approvals. Its largest deficiency is unclear chronology: several reviewers criticize omissions already present in the submitted LLD, reducing confidence that it independently stressed the final design. Claims that the ID is not logged elsewhere and that sibling commands share identical behavior were not independently demonstrated." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The plan provides strong oracles, especially the distinct content ID and mutation item ID that detect printing the wrong identifier. Its command names and documented flags are valid, and the stated project token scope is correct. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [cli.github.com](https://cli.github.com/manual/gh_project)) The largest gaps are no explicit TTY-plus-JSON test and no mocked unit failure case; the optional live mutation also lacks cleanup or rerun isolation. Tests were not executed, so compilation and the claimed existing JSON fixture remain unverified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the design end to end: the production hunk preserves TTY output, prints item.Id for non-TTY output, and adds a test whose mutation ID differs from the content ID. The official v2.98.0 release records the same fix through PR 14056, strongly corroborating the target and behavior. ([github.com](https://github.com/cli/cli/releases)) The test uses assert.NoError despite the repository instruction requiring require for error checks, and the summary's claimed +5/-2 production diffstat does not match the displayed +2/-1 hunk. Independent git apply checking was unavailable because local command execution could not initialize its sandbox, so the candidate's applicability claim could not be fully verified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 24, + "input_tokens": 50, + "output_tokens": 53016, + "total_tokens": 1663005, + "latency_seconds": 265.5, + "total_cost_usd": 0.067809379, + "result_subtype": "success", + "task_score": 87.0, + "cache_read_tokens": 1444439, + "cache_write_tokens": 165500, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 199.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 19, + "specificity": 23, + "risk_awareness": 20, + "total": 84, + "notes": "The issue crisply identifies the case-sensitive crossRepoPR comparison, user-visible consequence, fork non-goal, and testable regression boundary. The cited guard and observed absence of the DELETE are consistent with v2.100.0 evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest defect is that acceptance requires a lowercase repos/owner/REPO DELETE while the LLD and patch intentionally require canonical repos/OWNER/REPO, and the output criterion does not specify the TTY condition. No independent task statement was supplied, so broader requirement coverage cannot be verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is strongly grounded in merge.go, merge_test.go, finder.go, ghrepo.IsSame, and BranchDeleteRemote, with an implementation-ready data flow and explicit alternatives. Its line-level account of the comparison, early-return guard, omitted isCrossRepository field, and existing EqualFold convention agrees with available v2.100.0 evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/pull/5915/files)) The canonical DELETE reconstruction is technically plausible but justified by an explicitly unverified hypothetical REST 404 rather than demonstrated repository behavior. It otherwise handles compatibility, fork behavior, empty owners, errors, rollout, and scope proportionately, with only minor non-TTY observability nuance omitted." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 21, + "total": 83, + "notes": "The strongest review work is its concrete identification that the default mock owner masks the regression and that ignored 404/422 responses could produce false success. The referenced comparison and deletion guard are supported by available source evidence. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest weakness is elevating an unverified mixed-case REST-path rejection to MUST-FIX while several persona sections contribute little substantive stress testing. It also misses the issue's canonical-path contradiction and does not challenge the acceptance demand covering all three merge methods." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 89, + "notes": "The proposed regression test has concrete fixtures, HTTP and git stubs, exact output, and a canonical-path oracle that would fail both the old guard and a partial comparison-only fix. It also reuses the existing fork and same-repository tests for compatibility coverage, matching the documented failure path. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The main gap is that only --merge is exercised despite acceptance explicitly naming squash and rebase, and the unused REPO environment path is repository-external noise. Test execution was not required or claimed, but the unavailable local command runner prevented an independent compile or execution check." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the revised design end to end: EqualFold removes the false fork classification, the DELETE uses the API-returned owner, and the focused regression test verifies both behavior and request casing. The submitted contexts and symbols align with available v2.100.0 evidence for crossRepoPR, the early deletion guard, and the existing case-insensitive repository convention. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/pull/5915/files)) Its largest limitation is the additional canonical-repository construction being based on an unverified API-path concern, while the summary does not acknowledge that it contradicts the issue's lowercase-path acceptance criterion. The local sandbox failure prevented git apply --check, but the diff is internally coherent, gofmt-shaped, narrowly scoped, and its stated 58-insertion/2-deletion statistic matches the patch." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "release-create-404-scope-blame", + "complexity": "low", + "ref": "v2.97.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 27, + "input_tokens": 58, + "output_tokens": 43349, + "total_tokens": 1837724, + "latency_seconds": 247.2, + "total_cost_usd": 0.06215551100000002, + "result_subtype": "success", + "task_score": 87.0, + "cache_read_tokens": 1647521, + "cache_write_tokens": 146796, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 175.4, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue precisely defines the malformed scope comparison, observable outcomes, compatibility cases, and explicit non-goals. Its acceptance criteria align with GitHub's documented workflow-scope 404 behavior and the repository's subsequent fix. ([docs.github.com](https://docs.github.com/pt/rest/releases/releases), [github.com](https://github.com/cli/cli/releases)) The largest weakness is the unsupported overstatement that this affects every 404 or almost every OAuth token. With no independent task statement, broader requirement coverage and the claimed live reproduction remain unverifiable." + }, + "lld": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD names concrete functions, files, branches, outputs, and test inputs, leaving no major implementation decision unresolved. The proposed split-and-trim comparison is technically sound and matches the fix recorded for v2.98.0. ([github.com](https://github.com/cli/cli/releases)) Its largest factual defect is claiming the release code is the repository's only scope splitter, despite established comma splitting and TrimSpace handling in the API scope-suggestion code. ([github.com](https://github.com/cli/cli/issues/9403)) Exact line references and clean patch applicability could not be independently checked because the local command sandbox failed." + }, + "review": { + "completeness": 20, + "correctness": 19, + "specificity": 21, + "risk_awareness": 20, + "total": 80, + "notes": "The review concretely identifies why the existing comma-only test misses the regression and recommends spaced positive, negative, and exact-match cases. It also evaluates the relevant UX, scripting, and authorization consequences rather than treating the header as an access-control decision. Its largest weakness is that it mostly validates its own design, misses the LLD's false claim about repository-wide parsing conventions, and labels explicitly out-of-scope 404 guidance as a should-fix before deferring it. Exact source-line and import claims could not be locally reverified, although the central fix is corroborated by the release record. ([github.com](https://github.com/cli/cli/releases))" + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 82, + "notes": "The plan supplies concrete commands, exact headers, expected errors and stderr, compatibility cases, and a regression check that would fail against the old parser. Its test names and oracles agree with the submitted patch, including absent-header and workflowish cases. The largest omission is the repository-required go test ./... and make lint validation; package-only go vet is not an equivalent substitute. Manual switching between two same-host tokens is underspecified, and the final checklist inconsistently describes which live-token scenario requires workflow scope." + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The patch implements the complete requested behavior by trimming each comma-delimited scope before an exact workflow comparison while preserving the empty-header branch. It targets tokenHasWorkflowScope and adds meaningful parser and command-level tests for both 404 outcomes; the repository's v2.98.0 notes independently confirm this exact fix area. ([github.com](https://github.com/cli/cli/releases)) The added tests are somewhat redundant for a one-line correction but remain relevant and contain concrete oracles, with no apparent production-code defect. The claimed git apply --check and exact baseline context could not be rerun because the local shell sandbox was unavailable, while the summary accurately discloses that tests and lint were not executed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-create-reject-empty-name", + "complexity": "trivial", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 24, + "input_tokens": 52, + "output_tokens": 37924, + "total_tokens": 1594230, + "latency_seconds": 224.8, + "total_cost_usd": 0.05488488499999999, + "result_subtype": "success", + "task_score": 85.6, + "cache_read_tokens": 1422640, + "cache_write_tokens": 133614, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 168.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the delayed failure, intended early-exit behavior, affected interactive flows, and explicit non-goals. Its root-cause account names interactiveRepoNameAndOwner, splitNameAndOwner, interactiveRepoInfo, and createFromScratch consistently with the submitted source context. The largest limitation is that no independent task statement establishes whitespace-only input or all three flows as requirements, so those remain internally derived scope. It nevertheless addresses the relevant compatibility and UX risks without broadening the production change." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD commits to a minimal check immediately after prompter.Input and before splitNameAndOwner, with concrete propagation through interactiveRepoInfo to the three callers. It accurately preserves nonblank input, avoids additional dependencies, and explicitly treats non-interactive behavior, trimming, and re-prompting as non-goals. Its largest gap is accepting only from-scratch regression tests while claiming behavior across template and local flows, leaving that integration contract unpinned. The exact baseline SHA and cited line numbers are not independently established by the supplied task statement, though the patch context supports the named symbols and locations." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 19, + "total": 80, + "notes": "The review is concrete about helper placement, whitespace semantics, non-interactive behavior, and the missing template/local tests rather than merely approving the design. It correctly identifies that strict prompt stubs make later prompts observable and that the early return prevents userAndOrgs from issuing an HTTP request. Its largest weakness is dismissing the acknowledged cross-flow test gap without proposing a small direct helper test or another durable contract check. Several SRE and security sections are proportionate to this trivial change but add little analysis beyond confirming the absence of new surfaces." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 82, + "notes": "The plan gives valid Go package commands, exact Test_createRun row names, concrete inputs and error oracles, a full-package regression run, and a useful negative control. Strict unexpected-prompt branches and absent HTTP stubs provide meaningful evidence that failure occurs before later interaction or network access. The largest gap is the explicitly untested template and local flows, plus no new automated valid owner-prefixed case despite discussing that compatibility requirement. The negative-control prediction is also overly specific because unchanged code may first fail in userAndOrgs on an unregistered HTTP request rather than at a later prompt." + }, + "implementation": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The production hunk adds the required blank-name check directly after prompter.Input and before splitNameAndOwner, implementing the LLD while leaving valid values unchanged. The two Test_createRun rows cover empty and whitespace-only values and use strict prompt behavior to catch failure to exit early; the referenced strings and errors imports are already expected by the surrounding source. The largest limitation is the absence of dedicated template and local regression rows, although shared helper placement makes the production behavior consistent across those callers. The summary honestly discloses that gap and the unexecuted verification, and no material deviation from the design is apparent in the diff." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "low", + "ref": "v2.101.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 31, + "input_tokens": 64, + "output_tokens": 55874, + "total_tokens": 2095624, + "latency_seconds": 313.9, + "total_cost_usd": 0.074425021, + "result_subtype": "success", + "task_score": 84.8, + "cache_read_tokens": 1871696, + "cache_write_tokens": 167990, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 178.0, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 19, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The issue provides six testable acceptance criteria, explicit exclusions, reproduction steps, and compatibility expectations. Its largest defect is the repeated v2.10.0 reference, which conflicts with the submitted repository context of v2.101.0 and makes the stated base identity unreliable. The patch context corroborates the central symbols in internal/attachments/client.go and the existing user-to-server rejection row in client_test.go. The referenced issue and reported curl HTTP 201 result could not be independently verified, so server acceptance remains an asserted premise." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The design commits to a minimal allowlist change and precisely traces token classification, upload authorization, error propagation, and shared command integration. Its largest deficiency is unreliable version metadata plus a promised uploadTokenTypes comment rewrite that the implementation omits. The submitted diff contains the named uploadTokenTypes, NewUploader, uploadError, and commentable test integration points and is consistent with the main design. Claims that documentation contains no relevant text and that every create and edit path shares this gate could not be independently checked." + }, + "review": { + "completeness": 20, + "correctness": 17, + "specificity": 22, + "risk_awareness": 18, + "total": 77, + "notes": "The review identifies concrete test defects, including the contradictory rejection row, missing command coverage, authorization-header oracle, and fragile error-string assertion. Its largest correctness problem is the SRE claim that there is \"no new request,\" which directly contradicts the feature's purpose of allowing previously blocked ghu_ uploads to send a request. The findings map to specific patch changes in client_test.go and commentable_test.go rather than generic reviewer role-play. It does not flag the erroneous repository version or the implementation's omission of the LLD's promised comment update." + }, + "testing": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The plan has strong behavioral oracles for the returned URL, exact Authorization header, upload count, comment body, stdout, rejection status, and parsed server message. Its largest gap is the absence of executable affected-package test commands, while the manual procedure invokes gh rather than the repository-required built binary. The implementation contains the planned accepted and refused upload subtests and the shared comment table row with concrete expected output. The plan honestly reports that compilation, regression tests, and live server behavior were not verified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The patch implements the requested behavior through the narrow uploadTokenTypes change and adds unit, failure-path, authorization-header, and command-level coverage. The diff specifically changes internal/attachments/client.go, flips the existing TestNewUploader row, checks uploadError fields for HTTP 401, and exercises the shared comment path. Its largest deviation is omitting the LLD's promised comment rewrite, and its implementation metadata repeats the incorrect v2.10.0 base reference. Local source inspection and git apply checking were unavailable because the read-only shell failed before command execution, leaving exact applicability unverified but revealing no concrete defect in the submitted diff." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "high", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 53, + "input_tokens": 110, + "output_tokens": 116417, + "total_tokens": 3405267, + "latency_seconds": 621.0, + "total_cost_usd": 0.1345111075, + "result_subtype": "success", + "task_score": 84.8, + "cache_read_tokens": 3017645, + "cache_write_tokens": 271095, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 187.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The issue gives unusually testable acceptance criteria for clean, dirty, divergent, forced, current-worktree, and unchecked-out branch cases. Its proposed fast-forward and forced-reset behavior agrees with the documented `gh repo sync` semantics. ([cli.github.com](https://cli.github.com/manual/gh_repo_sync)) The largest gaps are the unmentioned force-mode TOCTOU risk and the unusable unquoted stash hint when a worktree path contains spaces or shell metacharacters. The claimed scratch reproduction could not be independently verified, and no separate task statement was supplied." + }, + "lld": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The design commits to concrete integration points in `executeLocalRepoSync`, `git.Client`, `gitExecuter`, and `mockGitClient`, including the crucial decision to resolve per-worktree `FETCH_HEAD` before crossing worktrees. Its largest technical weakness is parsing newline-delimited porcelain instead of using `-z`, while also emitting an unquoted `git -C ` recovery command despite claiming paths with spaces are supported. It also treats locked worktrees as expected `git -C` failures without establishing that behavior, and the review itself leaves `Client.Command` use of `RepoDir` unverified. The official release record confirms the asserted v2.98.0 base and commit prefix, but direct source and scratch-path verification were unavailable. ([github.com](https://github.com/cli/cli/releases))" + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 86, + "notes": "The review's strongest contribution is identifying the per-worktree `FETCH_HEAD` defect and preventing unnecessary SHA resolution on the unchanged `UpdateBranch` path. It also gives concrete consequences and resolutions for dirty worktrees, parser matching, contextual errors, and the force-mode race. Its largest deficiency is claiming that a deleted-worktree test was added when `testing.md` contains no such scenario, while the security review overlooks that the displayed stash command is unsafe or invalid without path quoting. The load-bearing assertion that `RepoDir` scopes commands remains a question rather than a repository-verified conclusion." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 79, + "notes": "The real-git fixture has strong behavioral oracles: exact branch SHAs, clean status, file contents, retained current branch, refusal text, and unchanged refs. The largest coverage gaps are the explicitly required untracked-file case and the deleted-worktree case promised by the review; only a tracked modification exercises dirtiness. The proposed `gofmt -l git pkg/cmd/repo/sync` command incorrectly passes directories, and the verification list omits the repository-required `go test ./...` and `make lint`. In the submitted tests, error checks repeatedly use `assert` rather than the repository-mandated `require`, allowing execution to continue after prerequisite failures." + }, + "implementation": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 19, + "total": 80, + "notes": "The patch implements the central design end to end: it discovers the owning worktree, resolves `FETCH_HEAD` locally, runs dirty checks and merge/reset through a copied client, and preserves `UpdateBranch` for unchecked-out branches. The official release evidence confirms that the claimed base is v2.98.0 at `a255baf`, but the stated `git apply --check` result and compilation could not be independently reproduced because direct checkout execution was unavailable. ([github.com](https://github.com/cli/cli/releases)) The largest implementation defects are the unquoted worktree path in the copyable stash hint and extensive `assert` use for error prerequisites contrary to repository conventions; coverage also omits an actual untracked-file case. The summary additionally overstates error wrapping because `WorktreeForBranch` and `ResolveRef` failures are returned unchanged, although the core branch-update flow is internally consistent with the LLD." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 65, + "input_tokens": 136, + "output_tokens": 75849, + "total_tokens": 4426128, + "latency_seconds": 510.9, + "total_cost_usd": 0.12424928999999997, + "result_subtype": "success", + "task_score": 84.2, + "cache_read_tokens": 4076105, + "cache_write_tokens": 274038, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 148.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The issue clearly defines the SHA-1/SHA-256 behavior for both release commands, with testable acceptance criteria and useful non-goals. The submitted patch contexts confirm the hard-coded `sha1` constructor calls in `verify.go` and `verify_asset.go`. Its main limitation is that no independent task statement establishes every claimed requirement, particularly malformed-ref rejection and GitHub's SHA-256 rollout behavior. It addresses compatibility and invalid input well, but attestation-service support for SHA-256 remains unverified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is unusually concrete about data flow, constructor changes, error handling, and the two command integration points. Its cited `NewDigestedArtifactForRelease` call sites and test seams agree with the submitted source diff. The largest deficiency is minor internal inconsistency: the file-change table proposes `release_test.go`, while the testing and implementation artifacts create `artifact_test.go`; the claimed repository-wide caller search could not be independently rerun. It responsibly identifies malformed refs, backward compatibility, policy coverage, rollout, and unknown server indexing." + }, + "review": { + "completeness": 19, + "correctness": 19, + "specificity": 20, + "risk_awareness": 20, + "total": 78, + "notes": "The strongest review finding is the concrete demand for length-plus-hex validation and explicit coverage of `FetchParams.Digest` and the verifier-policy gap. It names actionable symbols such as `releaseRefAlgorithm`, `BuildDigestPolicyOption`, and `api.MockClient.OnGetByDigest`. Much of the multi-role review repeats approval rather than independently challenging the constructor API change, output coverage, or the test-file inconsistency. Its security framing overstates malformed API data as a query-injection concern, while the genuinely material SHA-256 service-indexing question remains unresolved." + }, + "testing": { + "completeness": 19, + "correctness": 19, + "specificity": 23, + "risk_awareness": 20, + "total": 81, + "notes": "The plan gives concrete table inputs, test names, mock setup, and meaningful oracles for both digest algorithms and rejected refs. The implementation diff contains the described `OnGetByDigest` captures and assertions that malformed refs avoid the attestation call. Its largest gap is failure to directly assert the user-visible resolved-tag output or capture the artifact passed to the verifier, and it omits the repository-prescribed `go test ./...` and `make lint` checks. It also presents scoped package tests as final verification even though implementation.md says the full package trees were not rerun after the last test-table edit." + }, + "implementation": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 20, + "total": 84, + "notes": "The patch implements the central behavior in both commands, derives the algorithm once, validates malformed refs, and adds focused SHA-1/SHA-256 and negative tests. Its diff contexts consistently target the existing constructor and command call sites, and the error handling follows the surrounding Go pattern. The main coverage deficiency is that no test observes the artifact received by `AttVerifier` or directly checks the resolved-tag output, although the shared object makes the intended propagation plausible. The summary incorrectly says the old constructor behavior was hard-coded `sha256` rather than `sha1`, and independent `git apply --check` execution was unavailable because the local command sandbox could not initialize." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 34, + "input_tokens": 72, + "output_tokens": 46501, + "total_tokens": 2200177, + "latency_seconds": 295.3, + "total_cost_usd": 0.0686284995, + "result_subtype": "success", + "task_score": 83.0, + "cache_read_tokens": 2000597, + "cache_write_tokens": 153007, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 157.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the silent flag interaction, covers issue, PR, and discussion views, and gives directly testable acceptance criteria for rejected and preserved combinations. Its described command structure agrees with the submitted view-command patches and the repository's JSON exporter pattern. The largest evidence gap is that the independent upstream issue text was not supplied, so the assertion that either behavior and the comments-field exception are accepted requirements could not be verified. Compatibility impact, no-request behavior, and explicit non-goals are handled well, with only release communication left implicit." + }, + "lld": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The design commits to an exact predicate, helper signature, error type, call ordering, and eight concrete file changes rather than leaving implementation decisions open. The diff confirms the named constructors, Exporter.Fields usage, RunE integration points, and shared cmdutil location exist as described. Its main weakness is limited treatment of error-precedence interactions such as discussion comment URLs combined with JSON, and the exported command-specific helper is not compared against existing validation abstractions beyond rejecting inline duplication. The compatibility break and release-note need are identified, but rollback and validation guidance remain lightweight." + }, + "review": { + "completeness": 18, + "correctness": 19, + "specificity": 18, + "risk_awareness": 17, + "total": 72, + "notes": "The review checks relevant details including Cobra ordering, nil-exporter behavior, request timing, error UX, and the intentional exit-code change. These findings agree with the patch's placement of ValidateCommentsWithJSON before runF in all three constructors. The review is predominantly a confirmation pass and misses the concrete discrepancy that implementation.md and testing.md claim five helper-test rows while patch.diff contains four. It also offers little challenge to the comments-field exception or command-level accepted-path coverage, and the unexplained Step 7.5 reference is unsupported by the supplied artifacts." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan names exact test functions, arguments, expected errors, option state, and a meaningful no-HTTP oracle, with coverage across all three commands and the shared validator. The listed paths and test additions match patch.diff, and the compatibility and manual checks target the stated regression. However, it describes five TestValidateCommentsWithJSON cases while the implementation contains four, omitting the separate comments-false/exporter-nil row. Accepted command wiring is tested directly only for discussion, and package test commands plus stronger explicit checks for preserved issue and PR combinations would improve regression confidence." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the design end to end: one validator checks exporter fields, all three RunE paths invoke it before execution, and unit tests cover the rejected path plus shared acceptance logic. The submitted contexts and symbols are internally consistent with the repository structure, and the code follows Go and repository conventions with a godoc comment, FlagErrorf, table testing, require for errors, and no unrelated churn. The largest defect is the summary's false claim of five helper-test rows when the diff has four; issue and PR also lack direct accepted-combination constructor tests. The unexecuted compiler and test status is disclosed honestly, while the claimed local reverse-apply check could not be independently verified from the available execution environment." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-set-default-unset-without-repo", + "complexity": "trivial", + "ref": "v2.101.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 38, + "input_tokens": 82, + "output_tokens": 55685, + "total_tokens": 2304315, + "latency_seconds": 331.9, + "total_cost_usd": 0.07440615049999999, + "result_subtype": "success", + "task_score": 82.0, + "cache_read_tokens": 2098063, + "cache_write_tokens": 150485, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 167.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue gives a reproducible command, exact current error, narrow desired behavior, and testable acceptance criteria. The submitted patch corroborates the named guard, flags, and preserved error text. Its strongest scope decision is explicitly deferring the ignored positional argument while preserving existing behavior. The main evidence gap is the absent independent task statement and inability to verify the linked issue's current labels or acceptance criteria." + }, + "lld": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is grounded in concrete files and symbols, including NewCmdSetDefault, CanPrompt, UnsetRemoteResolution, and the exact guard change. Its behavior table, data flow, test seam, and handling of valid versus invalid ignored arguments are implementation-ready and consistent with the patch. It identifies the realistic local configuration integrity risk and explains why argument rejection is deferred. The main gap is that it proposes a live acceptance-test change without documenting the repository's authorization and credential constraints." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 22, + "total": 85, + "notes": "The review identifies a concrete wrong-target configuration risk from ignored positional arguments and traces it to argument resolution preceding the guard. It prioritizes the issue, explains the compatibility tradeoff, and records an actionable deferral rather than expanding scope. Some findings claim the LLD does not document silence or argument risk even though the submitted revised LLD does, and the statement about default GH_* environments is unsupported and immaterial. Despite that process-version ambiguity, the final resolutions and verdict are specific and defensible." + }, + "testing": { + "completeness": 19, + "correctness": 11, + "specificity": 20, + "risk_awareness": 9, + "total": 59, + "notes": "The plan provides precise command-level cases, expected option values, retained-error oracles, run-path coverage, and an end-to-end scenario. Its acceptance command omits the required -tags=acceptance flag, uses an unexplained GH_ACCEPTANCE_GROUP variable, and incorrectly describes the suite as fixture-server based even though repository documentation requires credentials for a real GitHub instance. ([github.com](https://github.com/cli/cli/blob/trunk/acceptance/README.md)) The backward-compatibility table also claims non-TTY repository arguments and ignored --unset arguments are covered while describing only TTY or intentionally absent tests. The optional smoke test runs inside the source checkout and can alter its .git/config, contrary to the repository's isolation requirement." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the central behavior end to end: it exempts UnsetMode from the non-interactive repository guard and adds focused regression and preservation tests. The condition, non-TTY setup, option assertions, and acceptance sequence are internally consistent with the surrounding context shown in the diff. Its largest defect is omitting the LLD's plain no-argument acceptance assertion while claiming no deviation; the verification account also displays an acceptance command without the required build tag, although it honestly states that the txtar scenario was unexecuted. Independent git apply and source checks could not be completed because the read-only command sandbox failed before execution, which is an evidence limitation rather than a candidate defect." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 51, + "input_tokens": 110, + "output_tokens": 53128, + "total_tokens": 3480768, + "latency_seconds": 362.0, + "total_cost_usd": 0.10172599249999999, + "result_subtype": "success", + "task_score": 81.6, + "cache_read_tokens": 3152505, + "cache_write_tokens": 275025, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 146.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 20, + "total": 90, + "notes": "The issue precisely defines repository and organization behavior for set, list, and delete, preserves defaults, and gives testable negative and compatibility criteria. The documented agents endpoints and integer selected_repository_ids match its API claims. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets)) Its main limitation is that no independent task statement establishes the intended breadth, although the linked upstream issue and merged PR support expanding the change across secret commands. ([github.com](https://github.com/cli/cli/issues/13419), [github.com](https://github.com/cli/cli/pull/13421)) The assertion that the existing API-version handling needs no change is not demonstrated within the artifact, but upstream live smoke testing supports feasibility. ([github.com](https://github.com/cli/cli/pull/13421))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 20, + "total": 89, + "notes": "The LLD is strongly grounded in concrete symbols such as GetSecretApp, IsSupportedSecretEntity, SecretPayload, and the existing interpolated request paths; the upstream implementation confirms this integration approach. ([github.com](https://github.com/cli/cli/pull/13421), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/secret/set/set.go)) Its endpoint and integer-ID payload decisions are correct according to the REST contract. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets)) The largest weakness is internal inconsistency: it repeatedly labels the payload contract unverified while later claiming the documented contract requires integers, and it says changes are confined to four packages while also requiring pkg/cmd/secret/secret.go. Exact base line numbers and patch applicability could not be independently checked because repository command execution failed during sandbox initialization." + }, + "review": { + "completeness": 17, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review's strongest contribution is the concrete identification of the list-test path rewrite and the explanation of why agents should use the generic payload branch. It misses two clear discrepancies: the patch omits the LLD's pkg/cmd/secret/secret.go umbrella-help update and lacks the acceptance criterion's organization-delete runtime test. The upstream merged change explicitly included umbrella documentation and repository-plus-organization list/delete runtime coverage, making the no-blocker verdict too generous. ([github.com](https://github.com/cli/cli/pull/13421), [cli.github.com](https://cli.github.com/manual/gh_secret)) Its API-contract concern is responsibly disclosed, although the official documentation directly verifies the assumed paths and integer IDs. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets))" + }, + "testing": { + "completeness": 17, + "correctness": 19, + "specificity": 21, + "risk_awareness": 17, + "total": 74, + "notes": "The plan names real test functions, concrete HTTP behavior, selected-repository payload values, and exact output modes rather than relying on generic assertions. Its largest omission is organization-level delete coverage, despite the issue requiring repository and organization tests for every command; it also substitutes the shared support-matrix test for command-level environment and user rejection checks. The package test, vet, and formatting commands are plausible, but cd cli is not appropriate when already running from the submitted repository root. The claimed observed test results could not be independently verified because shell execution failed before commands ran, and no captured output was supplied." + }, + "implementation": { + "completeness": 20, + "correctness": 22, + "specificity": 21, + "risk_awareness": 18, + "total": 81, + "notes": "The patch implements the core feature coherently by adding Agents parsing and support, exposing it in all three command enums, and relying on the existing generic path and encryption flow; this matches the upstream design and documented API. ([github.com](https://github.com/cli/cli/pull/13421), [docs.github.com](https://docs.github.com/en/rest/agents/secrets)) The largest defects are omission of the promised pkg/cmd/secret/secret.go umbrella help change and absence of an organization-delete runtime test, both of which were present in the upstream merged solution. ([github.com](https://github.com/cli/cli/pull/13421), [cli.github.com](https://cli.github.com/manual/gh_secret)) Consequently, the implementation summary's claim of no LLD deviations is inaccurate, and its known-gaps section overlooks those omissions. Exact application of the diff and the reported test execution could not be independently confirmed because the local command sandbox failed at initialization." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 55, + "input_tokens": 114, + "output_tokens": 90180, + "total_tokens": 3585391, + "latency_seconds": 530.1, + "total_cost_usd": 0.12145790800000002, + "result_subtype": "success", + "task_score": 81.0, + "cache_read_tokens": 3231063, + "cache_write_tokens": 264034, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 170.1, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 24, + "risk_awareness": 20, + "total": 84, + "notes": "The issue provides unusually testable host-mapping, token-source, fallback, and non-401 acceptance criteria, with clear non-goals. Its largest weakness is broadening the upstream request from tokens generated by `gh auth login` to every gh-stored writable token, without addressing stored user-supplied PATs. The submitted old-side diff confirms the cited `internal/ghcmd/cmd.go` hardcoded 401 message, while the upstream issue states the narrower generated-token requirement. ([github.com](https://github.com/cli/cli/issues/13066)) Independent local source inspection was unavailable because the read-only command sandbox failed before executing commands." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 24, + "risk_awareness": 20, + "total": 83, + "notes": "The LLD commits to concrete functions, integration points, fallback behavior, host conversion, and preservation of the existing SSO/scope branch ordering. Its largest design weakness is treating `AuthTokenWriteable` as proof that a token is the OAuth credential targeted by the upstream request; writeability establishes storage provenance, not how the token was generated. The proposed `HasActiveToken` guard correctly addresses the documented empty-token behavior, and the nil URL and configuration-error paths are explicit. Patch applicability and the cited source line numbers could not be independently verified because local shell inspection was unavailable." + }, + "review": { + "completeness": 20, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 80, + "notes": "The review finds three concrete defects with consequences and records specific revisions, especially the empty-token false positive and required `-h` argument. It does not challenge the central writeable-versus-generated-token assumption, so its no-blockers verdict is more confident than the evidence supports. Its claim that the existing generic login hint already reveals stored-token existence is also incorrect because only the new differential message provides that signal. The findings reference exact symbols such as `AuthTokenWriteable`, `HasActiveToken`, and the refresh hostname validation, although those references could not be independently opened locally." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 23, + "risk_awareness": 18, + "total": 80, + "notes": "The plan supplies concrete table cases, setup, commands, and exact string oracles for host mapping and helper behavior. Its largest gap is the absence of an automated integration assertion that `Main` actually uses the helper while preserving stderr and exit behavior; unchanged `Test_printError` cases cannot establish that wiring. It also promises both keyring and `oauth_token` behavior but only specifies `Login(..., secureStorage=true)`, and it lacks a stored user-supplied-token case. The manual E2E section acknowledges that its exit-code expectation was not verified." + }, + "implementation": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 18, + "total": 78, + "notes": "The patch implements the submitted LLD end to end, including known API-host conversion, stored-token checks, safe fallbacks, and focused unit tests. Its largest material risk is that writeability alone also selects writable user-supplied tokens, exceeding the upstream requirement for tokens generated by `gh auth login`. The tests omit plain-config `oauth_token` storage and direct `Main` integration, and `assert.NoError` for `Login` violates the repository instruction requiring `require` for error checks. The old-side contexts are internally consistent with the cited files, but patch application and claimed test executions could not be independently reproduced because repository command execution failed at sandbox initialization." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "trivial", + "ref": "v2.95.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 25, + "input_tokens": 52, + "output_tokens": 27930, + "total_tokens": 1620306, + "latency_seconds": 162.2, + "total_cost_usd": 0.044056529000000004, + "result_subtype": "success", + "task_score": 80.0, + "cache_read_tokens": 1503994, + "cache_write_tokens": 88330, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 172.2, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 23, + "risk_awareness": 21, + "total": 91, + "notes": "The issue clearly defines the cancelled-only failure, expected headline and tally, regression requirement, and explicit non-goals. The cited guard and existing Canceled branch in pkg/cmd/pr/checks/output.go match the submitted patch context and the documented upstream defect. Its largest gap is that preserving the zero-check case is only implied by the broad requirement that all other combinations remain unchanged. No independent task statement was supplied, but the task identifier and repository evidence substantiate the central requirement." + }, + "lld": { + "completeness": 22, + "correctness": 17, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The LLD is unusually concrete about output.go, aggregate.go, checkCounts.Canceled, fixture structure, precedence, and the one-line implementation decision. Its main code change is correct, but several repository claims are inconsistent: it names TestChecksRun although the patch context shows Test_checksRun, and it says printSummary is called only in watch mode even though its proposed default-options TTY test exercises it outside watch mode. It also alternately claims the old behavior writes an empty line and that the tally was not printed, leaving the precise pre-fix output unclear. Compatibility reasoning is otherwise strong, particularly the proof that adding Canceled changes only cancelled-only inputs while preserving the zero-count guard." + }, + "review": { + "completeness": 18, + "correctness": 16, + "specificity": 19, + "risk_awareness": 17, + "total": 70, + "notes": "The backend section provides the strongest analysis by tracing the count predicate and showing why mixed cancelled-and-passing cases masked the defect. However, every reviewer approves without catching the LLD's incorrect test symbol, contradictory tally description, or false assertion that printSummary is only called in watch mode. The recommendation for a Bandit-style review is also inappropriate for this Go repository and suggests generic security-review language rather than a repository-specific check. It identifies sensible compatibility and terminal-output risks, but does not challenge the lack of explicit zero-check and complete summary-branch coverage." + }, + "testing": { + "completeness": 20, + "correctness": 13, + "specificity": 22, + "risk_awareness": 18, + "total": 73, + "notes": "The plan supplies concrete fixture state, exact output oracles, a before-fix negative check, manual CLI behavior, exit-status expectations, and compatibility coverage. Its focused commands use TestChecksRun, while the actual test function shown by the patch is Test_checksRun, so both -run commands can report success while executing no matching tests. The full-package command would still execute the suite, preventing the plan from being unusable, but the principal targeted verification is materially defective. It also omits an explicit zero-check oracle, the adjacent behavior most directly exposed to changes in this guard." + }, + "implementation": { + "completeness": 22, + "correctness": 23, + "specificity": 22, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the required behavior end to end: it adds counts.Canceled to the existing guard, supplies a valid-looking cancelled-only GraphQL fixture, and adds a full-output TTY regression case in the actual Test_checksRun table. The output.go context and symbols match the repository defect, and the one-line logic preserves precedence and the zero-check condition. The main limitation is test breadth: the new test covers only the regression through the heavier command harness rather than directly pinning every printSummary branch, especially the no-check case. The summary accurately reports that tests were not run, although its statement that the function previously printed nothing is imprecise relative to the observed blank-headline behavior." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 73, + "input_tokens": 154, + "output_tokens": 109073, + "total_tokens": 4798333, + "latency_seconds": 652.5, + "total_cost_usd": 0.15650841799999998, + "result_subtype": "success", + "task_score": 77.8, + "cache_read_tokens": 4333998, + "cache_write_tokens": 355108, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 174.7, + "kv_cache_usage": null, + "agent_invocations": 2, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The issue provides a strong, testable contract covering API-provided SSH URLs, fork upstreams, HTTPS compatibility, fallback behavior, and wiki clones. Its named symbols and root cause match the submitted diff context and the public issue, including `GitHubRepo`, `FormatRemoteURL`, and `cloneRun`. ([github.com](https://github.com/cli/cli/issues/14344), [github.com](https://github.com/cli/cli/blob/v2.54.0/internal/ghrepo/repo.go)) The explicit exclusions keep the change bounded. Its main limitation is the absent independent task statement and limited treatment of malformed API URL risks." + }, + "lld": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The design commits to an exact data flow, helper contract, two integration points, fallback semantics, and wiki transformation. The submitted source context supports the cited files and interfaces, while the public issue confirms that the repository model already supports `SSHURL`. ([github.com](https://github.com/cli/cli/issues/14344)) Its largest gap is that the proposed implementation verification covers only the primary clone path, not the parent URL or leading-dash branch. The older-GHES fallback discussion also assumes the GraphQL field exists in the server schema; an unsupported field would fail the query rather than decode empty." + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 21, + "risk_awareness": 18, + "total": 77, + "notes": "The strongest review finding is the concrete leading-dash option-injection case, with a narrowly scoped guard and clear consequence. It also examines pointer interface satisfaction, shared parent fragments, request count, fallback behavior, and command output. Its largest deficiency is approving the patch without noticing that the new mock does not prove `sshUrl` was requested and that no automated test covers the required parent-upstream behavior. The comparison to existing HTTPS trust is imprecise because `FormatRemoteURL` synthesizes that URL rather than accepting an arbitrary API URL." + }, + "testing": { + "completeness": 14, + "correctness": 13, + "specificity": 19, + "risk_awareness": 15, + "total": 61, + "notes": "The plan gives concrete commands, fixture data, command regexes, compatibility cases, and realistic manual GHE.com steps. Its critical oracle flaw is that the GraphQL mock returns `sshUrl` regardless of the submitted query, so the new test could pass if the `queries_repo.go` change were removed. It explicitly leaves fork-upstream and leading-dash behavior untested, and existing HTTPS tests without `sshUrl` do not exercise the claimed negative case with that field present. Wiki behavior with an API SSH URL and query-shape verification are also missing." + }, + "implementation": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 16, + "total": 77, + "notes": "The focused patch implements the stated data flow by requesting `sshUrl`, selecting it for SSH at both clone sites, preserving fallback behavior, and retaining the existing wiki transformation. The diff context and public issue support the target symbols and underlying defect, but local forward applicability, formatting, and compilation could not be independently rechecked because the repository shell failed before executing commands. ([github.com](https://github.com/cli/cli/issues/14344), [github.com](https://github.com/cli/cli/blob/v2.54.0/internal/ghrepo/repo.go)) The largest defect is test coverage: the test does not verify the query field and omits parent, malformed URL, and API-backed wiki cases; it also uses `t.Fatalf` instead of the repository-mandated `require` error assertion. The summary accurately discloses unexecuted tests, although its claim that a skill forbids running them is unsupported and conflicts with the repository guidance." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "medium", + "ref": "v2.95.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 42, + "input_tokens": 88, + "output_tokens": 84133, + "total_tokens": 2743678, + "latency_seconds": 456.2, + "total_cost_usd": 0.102572184, + "result_subtype": "success", + "task_score": 75.4, + "cache_read_tokens": 2445739, + "cache_write_tokens": 213718, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 184.4, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible command, concrete symptoms, bounded scope, non-goals, and testable compatibility criteria. Its paths and concurrency diagnosis agree with the reported stack and upstream fix. The largest factual defect is the reference to issue #13339; the matching report is #13399, fixed by PR #13313. ([github.com](https://github.com/cli/cli/issues/13399), [github.com](https://github.com/cli/cli/pull/13313)) No independent task statement establishes the additional constraints, so those remain internally reasonable but externally unverifiable." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The LLD is strongly grounded in real symbols and clearly explains why copying pointer-bearing connections does not isolate mutable tunnel state. Its pointer conversion and connection-owned mutex match the core upstream solution. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest defect is releasing the mutex between the get/check/delete phases of UpdatePortVisibility; upstream explicitly corrected this TOCTOU gap and also adopted deferred unlocking for robustness. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) It also claims close-error behavior is preserved, although moving safeClose from a closure-local err to the outer named result can make close failures newly observable." + }, + "review": { + "completeness": 18, + "correctness": 17, + "specificity": 21, + "risk_awareness": 17, + "total": 73, + "notes": "The review's strongest work is its concrete analysis of race-detector dependence, mock-server map behavior, lock ordering, and log-order changes. It correctly validates the shared-pointer and mutex approach used by the upstream fix. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest miss is approving separate locks around UpdatePortVisibility operations despite the real TOCTOU hazard later fixed upstream. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) The no-blockers verdict is therefore too confident, and it also accepts the untested close-once lifecycle change without examining its error-semantic consequence." + }, + "testing": { + "completeness": 17, + "correctness": 15, + "specificity": 20, + "risk_awareness": 17, + "total": 69, + "notes": "The plan supplies useful race-enabled package commands, negative scenarios, compatibility checks, and explicit expected outcomes. Its largest defect is that F1 names TestForwardPortConcurrentlyForwardsMultiplePorts while the patch defines TestForwardPortConcurrentlyOnSharedConnection, so the targeted -run command can succeed while running no regression test. The broader E1 command remains useful, and a concurrent race-detector test is the correct basic strategy. ([github.com](https://github.com/cli/cli/pull/13313)) No automated test exercises ForwardPorts, close-once behavior, listener forwarding, or the final port set, while the live checks and claimed failure on the unpatched base remain explicitly unexecuted." + }, + "implementation": { + "completeness": 18, + "correctness": 16, + "specificity": 20, + "risk_awareness": 16, + "total": 70, + "notes": "The patch implements the central fix end to end: it retains the connection pointer, adds a shared mutex, guards manager operations, and adds concurrent coverage, closely matching the upstream resolution. ([github.com](https://github.com/cli/cli/pull/13313)) The diff targets real symbols and plausible base context, but git apply --check could not be independently executed because local shell access failed. Its largest code defect is the non-atomic get/check/delete sequence in UpdatePortVisibility, a TOCTOU issue the upstream review explicitly corrected; manual unlocks are also less robust than the accepted deferred-unlock helper. ([github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313), [github.com](https://github.com/cli/cli/pull/13313)) The test violates the repository's require/testify convention and does not cover the added ForwardPorts lifecycle change, while the summary incorrectly calls error behavior unchanged after moving safeClose to the outer named return." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "high", + "ref": "v2.96.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 64, + "input_tokens": 134, + "output_tokens": 120947, + "total_tokens": 4070435, + "latency_seconds": 648.7, + "total_cost_usd": 0.14470197499999995, + "result_subtype": "success", + "task_score": 75.4, + "cache_read_tokens": 3674860, + "cache_write_tokens": 274494, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 186.4, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 15, + "specificity": 23, + "risk_awareness": 19, + "total": 79, + "notes": "The issue defines a concrete name-based interface, typed value behavior, compatibility expectations, ambiguity handling, and useful non-goals. Its largest defect is the incorrect claim that `gh project item-edit --id X` should report that `id` is missing; the existing `MarkFlagRequired(\"id\")` only produces that error when `--id` is absent. It also requires invalid numbers to fail before any request, although the field type must first be learned through `ProjectFields`. No independent task statement was supplied, so requirements beyond the identifier and internally consistent artifacts could not be verified." + }, + "lld": { + "completeness": 20, + "correctness": 16, + "specificity": 23, + "risk_awareness": 20, + "total": 79, + "notes": "The design is unusually concrete about `item_edit.go`, `queries.go`, `NewOwner`, `ProjectFields`, `ProjectField.Options`, validation order, mutations, and data-type conversion. Its proposed raw-string error `required flag(s) \\\"id\\\" not set` would preserve literal backslashes rather than the existing Cobra message, and value parsing occurs after project queries rather than before every request as the issue demands. It explicitly recognizes pagination, duplicate names, finite-number validation, unsupported field types, the 100-membership limit, and uncertain archived-item behavior. The live API behavior for archived memberships remains unresolved, and the design accepts a potentially false non-membership result instead of providing complete lookup pagination." + }, + "review": { + "completeness": 17, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review catches two real design defects: the 30-field default limit in `ProjectFields` and misleading handling when exactly 100 project memberships are returned. It also gives concrete security and request-cost analysis tied to GraphQL variables, `resource(url:)`, and the existing `item-add` flow. However, it misses the malformed legacy error string, the contradiction over pre-request number validation, and the requirement for conflict errors to identify the actual supplied flags. It provides blocker status but no clear overall verdict, and it accepts the membership truncation and archived-item uncertainty without a robust fallback." + }, + "testing": { + "completeness": 21, + "correctness": 17, + "specificity": 22, + "risk_awareness": 20, + "total": 80, + "notes": "The plan provides concrete cases and oracles for matching, type conversion, flag validation, GraphQL query shape, pagination, request counts, compatibility, help output, and manual live checks. One command expectation is inconsistent with the patch: the plan expects `invalid number: abc`, while the implementation returns `invalid project number \"abc\": must be a positive integer`; it also confusingly describes the positional project number as missing `--number`. It does not include an integrated oracle proving invalid typed values cause no requests, despite that explicit issue requirement. Archived-item behavior is appropriately identified as unverified, but most backward-compatibility coverage is stated broadly rather than mapped to concrete cases." + }, + "implementation": { + "completeness": 16, + "correctness": 14, + "specificity": 20, + "risk_awareness": 15, + "total": 65, + "notes": "The patch implements the principal flow end to end: new flags, owner and project resolution, case-insensitive field and option lookup, typed conversion, URL membership lookup, and reuse of `updateItemValues`. The legacy missing-ID error is incorrectly constructed as a raw string containing escaped quotes, and the generic conflict error does not identify which supplied flags conflict as required. Tests omit command parsing, `applyNamedValue`, field resolution, GraphQL lookup, mutation behavior, and legacy regression coverage; they also pin complete error strings and use `assert` for error checks contrary to the submitted review and repository instructions. The summary honestly discloses the missing tests and 100-membership limit, while archived-item behavior and successful patch compilation remain unverified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "high", + "ref": "v2.94.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 31, + "input_tokens": 66, + "output_tokens": 59478, + "total_tokens": 2132381, + "latency_seconds": 338.8, + "total_cost_usd": 0.07760447200000001, + "result_subtype": "success", + "task_score": 62.4, + "cache_read_tokens": 1898267, + "cache_write_tokens": 174570, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 175.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 18, + "total": 82, + "notes": "The issue gives a reproducible command sequence, exact old and desired paths, scoped files, testable acceptance criteria, and explicit exclusions. Its relocation diagnosis matches the repository's documented defect involving the two-level base calculation and subsequent RemoveAll. ([github.com](https://github.com/cli/cli/pull/13449)) The largest gap is that the broad promise that every failed update preserves the install is narrowed to fetch failures, leaving write and replacement failures unresolved. No independent task statement was supplied, so requirements beyond the identifier and repository evidence remain unverifiable." + }, + "lld": { + "completeness": 15, + "correctness": 13, + "specificity": 20, + "risk_awareness": 14, + "total": 62, + "notes": "The strongest element is the concrete Dir-plus-leaf-name calculation, which directly targets the discovered installation path and is straightforward to implement. The design knowingly permits phase-two write failures to leave mixed old and new content, contradicting its issue-level guarantee that a failed update leaves the installation intact. The repository's fuller solution stages in a sibling directory, swaps entries with rollback, and removes stale files, decisions this LLD omits. ([github.com](https://github.com/cli/cli/pull/13449)) Its invariant that updates write only inside u.local.dir is also overstated because its own compatibility section acknowledges RecordInstall side effects, while the claimed inertness of all removed option fields could not be locally compiled or tested." + }, + "review": { + "completeness": 12, + "correctness": 10, + "specificity": 17, + "risk_awareness": 11, + "total": 50, + "notes": "The review names concrete symbols, tests, and the partial-write residual instead of offering only generic approval. Its largest defect is approving the design while treating mid-write corruption as acceptable even though the issue promises preservation on failed updates. It also invents an AGENTS.md 30-50-line guideline absent from the supplied repository instructions and misses that the existing error-case harness skipped its verify block, a defect identified by the repository's corresponding fix. ([github.com](https://github.com/cli/cli/pull/13449)) The review candidly marks help-text inspection and test execution as unverified, but those gaps further weaken its approve verdict." + }, + "testing": { + "completeness": 15, + "correctness": 12, + "specificity": 19, + "risk_awareness": 12, + "total": 58, + "notes": "The plan provides concrete fixtures and oracles for namespaced placement, flat placement, output, and a later-blob fetch failure. Its claim that the existing first-file failure case enforces the Original content assertion is incorrect because that harness skipped verify whenever wantErr was set. ([github.com](https://github.com/cli/cli/pull/13449)) It lacks coverage for write-phase rollback, stale-file removal, directory identity, and a repository-level acceptance test, all of which distinguish a genuinely safe in-place update. The targeted Go commands are plausible, but neither they nor the end-to-end scenario were executed, so compilation and behavior remain unverified." + }, + "implementation": { + "completeness": 15, + "correctness": 14, + "specificity": 19, + "risk_awareness": 12, + "total": 60, + "notes": "The patch implements the central path fix by deriving Dir and Name from u.local.dir, removes relocation cleanup, and adds a meaningful later-fetch regression test. Its largest defect is replacing files directly after buffering every blob: a write failure still produces mixed content, stale upstream-deleted files remain, and the global installer now incurs aggregate memory use. The repository's robust fix instead stages beside the destination, swaps entries with rollback, preserves directory identity, and tests stale cleanup and the previously skipped error verifier. ([github.com](https://github.com/cli/cli/pull/13449)) Local shell inspection was blocked by the environment, so the claimed reverse-apply check could not be independently repeated, and the submission explicitly leaves build, vet, and tests unverified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "extension-uninstall-alias", + "complexity": "trivial", + "ref": "v2.93.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 21, + "input_tokens": 44, + "output_tokens": 33074, + "total_tokens": 1363218, + "latency_seconds": 184.0, + "total_cost_usd": 0.045618952500000004, + "result_subtype": "success", + "task_score": 61.4, + "cache_read_tokens": 1228975, + "cache_write_tokens": 101125, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 179.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 19, + "correctness": 12, + "specificity": 22, + "risk_awareness": 16, + "total": 69, + "notes": "The issue clearly defines user-visible behavior, compatibility, concrete acceptance criteria, and non-goals. Its largest defect is converting an alias task into a separate discoverable command based on an unsupported help-listing requirement. The published manual presents `gh extension uninstall` under the aliases of `gh extension remove`, contradicting that central choice. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) With no independent task statement, the help requirement and related issue #13598 could not be verified." + }, + "lld": { + "completeness": 20, + "correctness": 12, + "specificity": 22, + "risk_awareness": 14, + "total": 68, + "notes": "The LLD names concrete integration points such as `NewCmdExtension`, `normalizeExtensionSelector`, `ExtensionManager.Remove`, and the relevant command tests. Its central decision is nevertheless wrong for an alias task: it extracts a handler and registers a second command instead of adding Cobra alias metadata. Existing repository evidence shows `NewCmdExtension` already uses Cobra `Aliases`, while the published manual identifies uninstall specifically as a remove alias. ([github.com](https://github.com/cli/cli/pull/5974/files/a6596762701e0b1299675d9071c8b8f06af8d523), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The claimed need for a live fixture and exact baseline line contexts could not be independently verified because the local read-only command runner failed before execution." + }, + "review": { + "completeness": 13, + "correctness": 9, + "specificity": 15, + "risk_awareness": 10, + "total": 47, + "notes": "The review does identify a concrete acceptance-test gap and the duplicated help description. Most reviewer sections merely approve the design, however, and none challenges the load-bearing decision to create a distinct command. That endorsement conflicts with the published CLI interface, where uninstall is documented as an alias of remove. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The assertion that acceptance coverage cannot be added because it cannot run in the sandbox is not a sufficient design justification, and the live-fixture requirement was not independently verified." + }, + "testing": { + "completeness": 17, + "correctness": 12, + "specificity": 21, + "risk_awareness": 13, + "total": 63, + "notes": "The plan provides concrete commands and meaningful oracles for selector normalization, TTY output, manager errors, help output, and remove regression behavior. Its main deficiency is that it tests and enshrines a separate `uninstall` command rather than verifying alias metadata or canonical-command behavior. The implementation tests also use `assert.Equal` for call count and then unconditionally index `calls[0]`, allowing a failed count assertion to become a panic. The commands were not executed, and the claimed necessity of a live acceptance fixture could not be independently confirmed." + }, + "implementation": { + "completeness": 17, + "correctness": 11, + "specificity": 20, + "risk_awareness": 12, + "total": 60, + "notes": "The patch functionally wires `uninstall` through the same normalization, removal, error, and TTY-output path and adds focused tests. Its largest defect is implementing a sibling command rather than the requested alias, creating unnecessary handler extraction, duplicate command metadata, and different generated help/documentation semantics. The official manual lists `gh extension uninstall` in the alias section for `gh extension remove`, which does not match this patch's command model. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The submitted hunks are internally coherent, but patch applicability and the claimed baseline SHA verification could not be independently repeated because the repository command runner was unavailable." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "auth-clipboard-config-default", + "complexity": "high", + "ref": "v2.100.0", + "artifacts_produced": 0, + "artifacts_expected": 6, + "num_turns": 107, + "input_tokens": 222, + "output_tokens": 153667, + "total_tokens": 7294588, + "latency_seconds": 1005.0, + "total_cost_usd": 0.241232057, + "result_subtype": "success", + "task_score": 0.0, + "cache_read_tokens": 6522967, + "cache_write_tokens": 617732, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 160.3, + "kv_cache_usage": null, + "agent_invocations": 2, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "lld": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "review": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "testing": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "implementation": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + } + }, + "is_error": false, + "failed": true + }, + { + "task": "skill-install-nested-discovery", + "complexity": "high", + "ref": "v2.93.0", + "artifacts_produced": 0, + "artifacts_expected": 6, + "num_turns": 54, + "input_tokens": 114, + "output_tokens": 84584, + "total_tokens": 3498514, + "latency_seconds": 519.5999999999999, + "total_cost_usd": 0.129655317, + "result_subtype": "success", + "task_score": 0.0, + "cache_read_tokens": 3053582, + "cache_write_tokens": 360234, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 156.8, + "kv_cache_usage": null, + "agent_invocations": 2, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "lld": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "review": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "testing": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "implementation": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + } + }, + "is_error": false, + "failed": true + } + ], + "run_date": "2026-10-14" +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json new file mode 100644 index 00000000..e9c17b29 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json @@ -0,0 +1,64 @@ +{ + "task": "secret-set-app-agents", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 20, + "total": 90, + "notes": "The issue precisely defines repository and organization behavior for set, list, and delete, preserves defaults, and gives testable negative and compatibility criteria. The documented agents endpoints and integer selected_repository_ids match its API claims. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets)) Its main limitation is that no independent task statement establishes the intended breadth, although the linked upstream issue and merged PR support expanding the change across secret commands. ([github.com](https://github.com/cli/cli/issues/13419), [github.com](https://github.com/cli/cli/pull/13421)) The assertion that the existing API-version handling needs no change is not demonstrated within the artifact, but upstream live smoke testing supports feasibility. ([github.com](https://github.com/cli/cli/pull/13421))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 20, + "total": 89, + "notes": "The LLD is strongly grounded in concrete symbols such as GetSecretApp, IsSupportedSecretEntity, SecretPayload, and the existing interpolated request paths; the upstream implementation confirms this integration approach. ([github.com](https://github.com/cli/cli/pull/13421), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/secret/set/set.go)) Its endpoint and integer-ID payload decisions are correct according to the REST contract. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets)) The largest weakness is internal inconsistency: it repeatedly labels the payload contract unverified while later claiming the documented contract requires integers, and it says changes are confined to four packages while also requiring pkg/cmd/secret/secret.go. Exact base line numbers and patch applicability could not be independently checked because repository command execution failed during sandbox initialization." + }, + "review": { + "completeness": 17, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review's strongest contribution is the concrete identification of the list-test path rewrite and the explanation of why agents should use the generic payload branch. It misses two clear discrepancies: the patch omits the LLD's pkg/cmd/secret/secret.go umbrella-help update and lacks the acceptance criterion's organization-delete runtime test. The upstream merged change explicitly included umbrella documentation and repository-plus-organization list/delete runtime coverage, making the no-blocker verdict too generous. ([github.com](https://github.com/cli/cli/pull/13421), [cli.github.com](https://cli.github.com/manual/gh_secret)) Its API-contract concern is responsibly disclosed, although the official documentation directly verifies the assumed paths and integer IDs. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets))" + }, + "testing": { + "completeness": 17, + "correctness": 19, + "specificity": 21, + "risk_awareness": 17, + "total": 74, + "notes": "The plan names real test functions, concrete HTTP behavior, selected-repository payload values, and exact output modes rather than relying on generic assertions. Its largest omission is organization-level delete coverage, despite the issue requiring repository and organization tests for every command; it also substitutes the shared support-matrix test for command-level environment and user rejection checks. The package test, vet, and formatting commands are plausible, but cd cli is not appropriate when already running from the submitted repository root. The claimed observed test results could not be independently verified because shell execution failed before commands ran, and no captured output was supplied." + }, + "implementation": { + "completeness": 20, + "correctness": 22, + "specificity": 21, + "risk_awareness": 18, + "total": 81, + "notes": "The patch implements the core feature coherently by adding Agents parsing and support, exposing it in all three command enums, and relying on the existing generic path and encryption flow; this matches the upstream design and documented API. ([github.com](https://github.com/cli/cli/pull/13421), [docs.github.com](https://docs.github.com/en/rest/agents/secrets)) The largest defects are omission of the promised pkg/cmd/secret/secret.go umbrella help change and absence of an organization-delete runtime test, both of which were present in the upstream merged solution. ([github.com](https://github.com/cli/cli/pull/13421), [cli.github.com](https://cli.github.com/manual/gh_secret)) Consequently, the implementation summary's claim of no LLD deviations is inaccurate, and its known-gaps section overlooks those omissions. Exact application of the diff and the reported test execution could not be independently confirmed because the local command sandbox failed at initialization." + } + }, + "task_score": 81.6, + "verdict": "Strong, repository-specific design and a mostly correct implementation are limited by review and testing blind spots that allowed the promised umbrella help update and organization-delete runtime coverage to be omitted.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:39:39.631138Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 137559, + "cached_input_tokens": 77622, + "cache_write_input_tokens": 26709, + "output_tokens": 7929, + "reasoning_output_tokens": 5882 + }, + "duration_ms": 113220 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json new file mode 100644 index 00000000..cf40eb5b --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "secret-set-app-agents", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "secret", + "go", + "cli", + "github-api", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 146.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 110, + "output_tokens": 53128, + "cache_read_tokens": 3152505, + "cache_write_tokens": 275025, + "total_tokens": 3480768, + "total_cost_usd": 0.10172599249999999, + "latency_seconds": 362.0, + "num_turns": 51, + "generation_tokens_per_sec": 146.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 110, + "output_tokens": 53128, + "latency_seconds": 362.0, + "num_turns": 51, + "total_cost_usd": 0.10172599249999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3152505, + "cache_creation_tokens": 275025, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T12:49:05.213263Z", + "run_ended_at": "2026-10-14T12:55:07.201610Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 110, + "output_tokens": 53128, + "cache_read_tokens": 3152505, + "cache_write_tokens": 275025, + "total_tokens": 3480768, + "total_cost_usd": 0.10172599249999999, + "latency_seconds": 362.0, + "num_turns": 51, + "generation_tokens_per_sec": 146.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "secret-set-app-agents", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 20, + "total": 90, + "notes": "The issue precisely defines repository and organization behavior for set, list, and delete, preserves defaults, and gives testable negative and compatibility criteria. The documented agents endpoints and integer selected_repository_ids match its API claims. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets)) Its main limitation is that no independent task statement establishes the intended breadth, although the linked upstream issue and merged PR support expanding the change across secret commands. ([github.com](https://github.com/cli/cli/issues/13419), [github.com](https://github.com/cli/cli/pull/13421)) The assertion that the existing API-version handling needs no change is not demonstrated within the artifact, but upstream live smoke testing supports feasibility. ([github.com](https://github.com/cli/cli/pull/13421))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 20, + "total": 89, + "notes": "The LLD is strongly grounded in concrete symbols such as GetSecretApp, IsSupportedSecretEntity, SecretPayload, and the existing interpolated request paths; the upstream implementation confirms this integration approach. ([github.com](https://github.com/cli/cli/pull/13421), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/secret/set/set.go)) Its endpoint and integer-ID payload decisions are correct according to the REST contract. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets)) The largest weakness is internal inconsistency: it repeatedly labels the payload contract unverified while later claiming the documented contract requires integers, and it says changes are confined to four packages while also requiring pkg/cmd/secret/secret.go. Exact base line numbers and patch applicability could not be independently checked because repository command execution failed during sandbox initialization." + }, + "review": { + "completeness": 17, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review's strongest contribution is the concrete identification of the list-test path rewrite and the explanation of why agents should use the generic payload branch. It misses two clear discrepancies: the patch omits the LLD's pkg/cmd/secret/secret.go umbrella-help update and lacks the acceptance criterion's organization-delete runtime test. The upstream merged change explicitly included umbrella documentation and repository-plus-organization list/delete runtime coverage, making the no-blocker verdict too generous. ([github.com](https://github.com/cli/cli/pull/13421), [cli.github.com](https://cli.github.com/manual/gh_secret)) Its API-contract concern is responsibly disclosed, although the official documentation directly verifies the assumed paths and integer IDs. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets))" + }, + "testing": { + "completeness": 17, + "correctness": 19, + "specificity": 21, + "risk_awareness": 17, + "total": 74, + "notes": "The plan names real test functions, concrete HTTP behavior, selected-repository payload values, and exact output modes rather than relying on generic assertions. Its largest omission is organization-level delete coverage, despite the issue requiring repository and organization tests for every command; it also substitutes the shared support-matrix test for command-level environment and user rejection checks. The package test, vet, and formatting commands are plausible, but cd cli is not appropriate when already running from the submitted repository root. The claimed observed test results could not be independently verified because shell execution failed before commands ran, and no captured output was supplied." + }, + "implementation": { + "completeness": 20, + "correctness": 22, + "specificity": 21, + "risk_awareness": 18, + "total": 81, + "notes": "The patch implements the core feature coherently by adding Agents parsing and support, exposing it in all three command enums, and relying on the existing generic path and encryption flow; this matches the upstream design and documented API. ([github.com](https://github.com/cli/cli/pull/13421), [docs.github.com](https://docs.github.com/en/rest/agents/secrets)) The largest defects are omission of the promised pkg/cmd/secret/secret.go umbrella help change and absence of an organization-delete runtime test, both of which were present in the upstream merged solution. ([github.com](https://github.com/cli/cli/pull/13421), [cli.github.com](https://cli.github.com/manual/gh_secret)) Consequently, the implementation summary's claim of no LLD deviations is inaccurate, and its known-gaps section overlooks those omissions. Exact application of the diff and the reported test execution could not be independently confirmed because the local command sandbox failed at initialization." + } + }, + "task_score": 81.6, + "verdict": "Strong, repository-specific design and a mostly correct implementation are limited by review and testing blind spots that allowed the promised umbrella help update and organization-delete runtime coverage to be omitted.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:39:39.631138Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 137559, + "cached_input_tokens": 77622, + "cache_write_input_tokens": 26709, + "output_tokens": 7929, + "reasoning_output_tokens": 5882 + }, + "duration_ms": 113220 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json new file mode 100644 index 00000000..4a1771d8 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json @@ -0,0 +1,60 @@ +{ + "task": "skill-install-nested-discovery", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "lld": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "review": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "testing": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "implementation": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + } + }, + "task_score": 0.0, + "verdict": "MODEL FAILURE: the candidate run did not produce the required artifact(s): github-issue.md, lld.md, review.md, testing.md. Scored 0.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": false, + "scored_zero_missing_artifacts": [ + "github-issue.md", + "lld.md", + "review.md", + "testing.md" + ], + "evaluated_at": "2026-10-14T14:39:39.641030Z" + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json new file mode 100644 index 00000000..b7c327c1 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json @@ -0,0 +1,153 @@ +{ + "task": "skill-install-nested-discovery", + "repo": "https://github.com/cli/cli", + "ref": "v2.93.0", + "complexity": "high", + "tags": [ + "skill", + "go", + "cli", + "discovery", + "fixed-in-2.94.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 0, + "artifacts_expected": 6, + "generation_tokens_per_sec": 156.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 114, + "output_tokens": 84584, + "cache_read_tokens": 3053582, + "cache_write_tokens": 360234, + "total_tokens": 3498514, + "total_cost_usd": 0.129655317, + "latency_seconds": 519.5999999999999, + "num_turns": 54, + "generation_tokens_per_sec": 156.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 114, + "output_tokens": 84584, + "latency_seconds": 519.5999999999999, + "num_turns": 54, + "total_cost_usd": 0.129655317, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3053582, + "cache_creation_tokens": 360234, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T13:44:22.013827Z", + "run_ended_at": "2026-10-14T13:47:44.174617Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 114, + "output_tokens": 84584, + "cache_read_tokens": 3053582, + "cache_write_tokens": 360234, + "total_tokens": 3498514, + "total_cost_usd": 0.129655317, + "latency_seconds": 519.5999999999999, + "num_turns": 54, + "generation_tokens_per_sec": 156.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "agent_invocations": 2, + "topped_up_artifacts": [], + "evaluation": { + "task": "skill-install-nested-discovery", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "lld": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "review": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "testing": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + }, + "implementation": { + "completeness": 0, + "correctness": 0, + "specificity": 0, + "risk_awareness": 0, + "total": 0, + "notes": "Artifact not produced by the candidate run." + } + }, + "task_score": 0.0, + "verdict": "MODEL FAILURE: the candidate run did not produce the required artifact(s): github-issue.md, lld.md, review.md, testing.md. Scored 0.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": false, + "scored_zero_missing_artifacts": [ + "github-issue.md", + "lld.md", + "review.md", + "testing.md" + ], + "evaluated_at": "2026-10-14T14:39:39.641030Z" + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json new file mode 100644 index 00000000..788fefb1 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json @@ -0,0 +1,64 @@ +{ + "task": "skill-update-dir-keeps-install-path", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 18, + "total": 82, + "notes": "The issue gives a reproducible command sequence, exact old and desired paths, scoped files, testable acceptance criteria, and explicit exclusions. Its relocation diagnosis matches the repository's documented defect involving the two-level base calculation and subsequent RemoveAll. ([github.com](https://github.com/cli/cli/pull/13449)) The largest gap is that the broad promise that every failed update preserves the install is narrowed to fetch failures, leaving write and replacement failures unresolved. No independent task statement was supplied, so requirements beyond the identifier and repository evidence remain unverifiable." + }, + "lld": { + "completeness": 15, + "correctness": 13, + "specificity": 20, + "risk_awareness": 14, + "total": 62, + "notes": "The strongest element is the concrete Dir-plus-leaf-name calculation, which directly targets the discovered installation path and is straightforward to implement. The design knowingly permits phase-two write failures to leave mixed old and new content, contradicting its issue-level guarantee that a failed update leaves the installation intact. The repository's fuller solution stages in a sibling directory, swaps entries with rollback, and removes stale files, decisions this LLD omits. ([github.com](https://github.com/cli/cli/pull/13449)) Its invariant that updates write only inside u.local.dir is also overstated because its own compatibility section acknowledges RecordInstall side effects, while the claimed inertness of all removed option fields could not be locally compiled or tested." + }, + "review": { + "completeness": 12, + "correctness": 10, + "specificity": 17, + "risk_awareness": 11, + "total": 50, + "notes": "The review names concrete symbols, tests, and the partial-write residual instead of offering only generic approval. Its largest defect is approving the design while treating mid-write corruption as acceptable even though the issue promises preservation on failed updates. It also invents an AGENTS.md 30-50-line guideline absent from the supplied repository instructions and misses that the existing error-case harness skipped its verify block, a defect identified by the repository's corresponding fix. ([github.com](https://github.com/cli/cli/pull/13449)) The review candidly marks help-text inspection and test execution as unverified, but those gaps further weaken its approve verdict." + }, + "testing": { + "completeness": 15, + "correctness": 12, + "specificity": 19, + "risk_awareness": 12, + "total": 58, + "notes": "The plan provides concrete fixtures and oracles for namespaced placement, flat placement, output, and a later-blob fetch failure. Its claim that the existing first-file failure case enforces the Original content assertion is incorrect because that harness skipped verify whenever wantErr was set. ([github.com](https://github.com/cli/cli/pull/13449)) It lacks coverage for write-phase rollback, stale-file removal, directory identity, and a repository-level acceptance test, all of which distinguish a genuinely safe in-place update. The targeted Go commands are plausible, but neither they nor the end-to-end scenario were executed, so compilation and behavior remain unverified." + }, + "implementation": { + "completeness": 15, + "correctness": 14, + "specificity": 19, + "risk_awareness": 12, + "total": 60, + "notes": "The patch implements the central path fix by deriving Dir and Name from u.local.dir, removes relocation cleanup, and adds a meaningful later-fetch regression test. Its largest defect is replacing files directly after buffering every blob: a write failure still produces mixed content, stale upstream-deleted files remain, and the global installer now incurs aggregate memory use. The repository's robust fix instead stages beside the destination, swaps entries with rollback, preserves directory identity, and tests stale cleanup and the previously skipped error verifier. ([github.com](https://github.com/cli/cli/pull/13449)) Local shell inspection was blocked by the environment, so the claimed reverse-apply check could not be independently repeated, and the submission explicitly leaves build, vet, and tests unverified." + } + }, + "task_score": 62.4, + "verdict": "The submission correctly identifies and addresses path relocation, but its non-transactional direct-write implementation does not satisfy the promised failure-safety guarantee, and the review and tests miss important repository-proven gaps.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:42:19.929643Z", + "repo_ref": "v2.94.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 127839, + "cached_input_tokens": 63638, + "cache_write_input_tokens": 22059, + "output_tokens": 10076, + "reasoning_output_tokens": 7978 + }, + "duration_ms": 157154 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json new file mode 100644 index 00000000..af3b50bb --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-haiku-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "skill-update-dir-keeps-install-path", + "repo": "https://github.com/cli/cli", + "ref": "v2.94.0", + "complexity": "high", + "tags": [ + "skill", + "go", + "cli", + "filesystem", + "fixed-in-2.95.0" + ], + "model": "us.anthropic.claude-haiku-5-5", + "model_slug": "claude-haiku-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 175.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 66, + "output_tokens": 59478, + "cache_read_tokens": 1898267, + "cache_write_tokens": 174570, + "total_tokens": 2132381, + "total_cost_usd": 0.07760447200000001, + "latency_seconds": 338.8, + "num_turns": 31, + "generation_tokens_per_sec": 175.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 66, + "output_tokens": 59478, + "latency_seconds": 338.8, + "num_turns": 31, + "total_cost_usd": 0.07760447200000001, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1898267, + "cache_creation_tokens": 174570, + "models_used": [ + "us.anthropic.claude-haiku-5-5" + ], + "run_started_at": "2026-10-14T13:33:18.998949Z", + "run_ended_at": "2026-10-14T13:38:57.794745Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 66, + "output_tokens": 59478, + "cache_read_tokens": 1898267, + "cache_write_tokens": 174570, + "total_tokens": 2132381, + "total_cost_usd": 0.07760447200000001, + "latency_seconds": 338.8, + "num_turns": 31, + "generation_tokens_per_sec": 175.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "skill-update-dir-keeps-install-path", + "model": "us.anthropic.claude-haiku-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 18, + "total": 82, + "notes": "The issue gives a reproducible command sequence, exact old and desired paths, scoped files, testable acceptance criteria, and explicit exclusions. Its relocation diagnosis matches the repository's documented defect involving the two-level base calculation and subsequent RemoveAll. ([github.com](https://github.com/cli/cli/pull/13449)) The largest gap is that the broad promise that every failed update preserves the install is narrowed to fetch failures, leaving write and replacement failures unresolved. No independent task statement was supplied, so requirements beyond the identifier and repository evidence remain unverifiable." + }, + "lld": { + "completeness": 15, + "correctness": 13, + "specificity": 20, + "risk_awareness": 14, + "total": 62, + "notes": "The strongest element is the concrete Dir-plus-leaf-name calculation, which directly targets the discovered installation path and is straightforward to implement. The design knowingly permits phase-two write failures to leave mixed old and new content, contradicting its issue-level guarantee that a failed update leaves the installation intact. The repository's fuller solution stages in a sibling directory, swaps entries with rollback, and removes stale files, decisions this LLD omits. ([github.com](https://github.com/cli/cli/pull/13449)) Its invariant that updates write only inside u.local.dir is also overstated because its own compatibility section acknowledges RecordInstall side effects, while the claimed inertness of all removed option fields could not be locally compiled or tested." + }, + "review": { + "completeness": 12, + "correctness": 10, + "specificity": 17, + "risk_awareness": 11, + "total": 50, + "notes": "The review names concrete symbols, tests, and the partial-write residual instead of offering only generic approval. Its largest defect is approving the design while treating mid-write corruption as acceptable even though the issue promises preservation on failed updates. It also invents an AGENTS.md 30-50-line guideline absent from the supplied repository instructions and misses that the existing error-case harness skipped its verify block, a defect identified by the repository's corresponding fix. ([github.com](https://github.com/cli/cli/pull/13449)) The review candidly marks help-text inspection and test execution as unverified, but those gaps further weaken its approve verdict." + }, + "testing": { + "completeness": 15, + "correctness": 12, + "specificity": 19, + "risk_awareness": 12, + "total": 58, + "notes": "The plan provides concrete fixtures and oracles for namespaced placement, flat placement, output, and a later-blob fetch failure. Its claim that the existing first-file failure case enforces the Original content assertion is incorrect because that harness skipped verify whenever wantErr was set. ([github.com](https://github.com/cli/cli/pull/13449)) It lacks coverage for write-phase rollback, stale-file removal, directory identity, and a repository-level acceptance test, all of which distinguish a genuinely safe in-place update. The targeted Go commands are plausible, but neither they nor the end-to-end scenario were executed, so compilation and behavior remain unverified." + }, + "implementation": { + "completeness": 15, + "correctness": 14, + "specificity": 19, + "risk_awareness": 12, + "total": 60, + "notes": "The patch implements the central path fix by deriving Dir and Name from u.local.dir, removes relocation cleanup, and adds a meaningful later-fetch regression test. Its largest defect is replacing files directly after buffering every blob: a write failure still produces mixed content, stale upstream-deleted files remain, and the global installer now incurs aggregate memory use. The repository's robust fix instead stages beside the destination, swaps entries with rollback, preserves directory identity, and tests stale cleanup and the previously skipped error verifier. ([github.com](https://github.com/cli/cli/pull/13449)) Local shell inspection was blocked by the environment, so the claimed reverse-apply check could not be independently repeated, and the submission explicitly leaves build, vet, and tests unverified." + } + }, + "task_score": 62.4, + "verdict": "The submission correctly identifies and addresses path relocation, but its non-transactional direct-write implementation does not satisfy the promised failure-safety guarantee, and the review and tests miss important repository-proven gaps.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T14:42:19.929643Z", + "repo_ref": "v2.94.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 127839, + "cached_input_tokens": 63638, + "cache_write_input_tokens": 22059, + "output_tokens": 10076, + "reasoning_output_tokens": 7978 + }, + "duration_ms": 157154 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json new file mode 100644 index 00000000..1a2a40b5 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json @@ -0,0 +1,64 @@ +{ + "task": "auth-clipboard-config-default", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 17, + "correctness": 11, + "specificity": 23, + "risk_awareness": 18, + "total": 69, + "notes": "The issue provides unusually concrete precedence rules, acceptance criteria, user stories, and exclusions. Its central requirement is wrong: it specifies a disabled default and unchanged behavior, while v2.101.0 explicitly introduced copying by default with persistent opt-out via `clipboard disabled`. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) Issue #13153 supports the motivation and proposed configuration key, and the CLI manuals confirm per-host configuration support. ([github.com](https://github.com/cli/cli/issues/13153), [cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get)) No independent task statement was supplied, so exact diagnostic wording could not be established separately from the candidate's assertions." + }, + "lld": { + "completeness": 19, + "correctness": 13, + "specificity": 23, + "risk_awareness": 19, + "total": 74, + "notes": "The LLD is strongly grounded in named repository files, interfaces, option structs, mock generation, hostname resolution, and the existing auth-flow boundary. Its load-bearing decision is nevertheless incorrect: `DefaultValue: \"disabled\"` and the claimed no-change rollout contradict the released requirement that clipboard copying default to enabled. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) The host-scoped design is feasible because `gh config set`, `get`, and `list` expose per-host operation. ([cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get), [cli.github.com](https://cli.github.com/manual/gh_config_list)) It honestly records the untested `loginRun` wiring, but leaves that regression path unmitigated, and the unavailable local shell prevented complete independent validation of every line-level repository claim." + }, + "review": { + "completeness": 18, + "correctness": 15, + "specificity": 22, + "risk_awareness": 20, + "total": 75, + "notes": "The strongest finding is Byte's concrete identification of the existing `Test_refreshRun` case that must set `ClipboardProvided`, including the exact behavioral consequence. The review never challenges the design's disabled default, despite the corresponding release explicitly defaulting clipboard copying to enabled, so it misses the most important product-level defect. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) It also says there is “no new file” while the reviewed design adds `clipboard.go` and `clipboard_test.go`, an internal factual inconsistency. Its SSH/headless clipboard warning, generated-mock drift concern, discoverability feedback, and login wiring gap are otherwise specific and actionable." + }, + "testing": { + "completeness": 20, + "correctness": 12, + "specificity": 23, + "risk_awareness": 20, + "total": 75, + "notes": "The plan covers config validation, host precedence, both auth commands, explicit true and false flags, help, rollback, headless failures, and detailed output oracles. Its central compatibility and default assertions are backwards: it expects fresh configuration not to copy, whereas v2.101.0 intentionally copies by default and uses `clipboard disabled` as the opt-out. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) The proposed `go generate` drift check is also unsound immediately after applying an uncommitted patch because `git diff --exit-code internal/gh/mock/config.go` detects the intended patch itself. Most individual tests are precise, but no automated test observes the value passed by `loginRun`, leaving an acknowledged wiring regression uncovered." + }, + "implementation": { + "completeness": 18, + "correctness": 12, + "specificity": 22, + "risk_awareness": 17, + "total": 69, + "notes": "The patch coherently adds the config accessor and interface method, generated mock support, flag-provided tracking, a shared precedence resolver, login and refresh wiring, help text, and focused tests. However, `DefaultValue: \"disabled\"` and `clipboard: disabled` preserve the old default, directly failing the defining v2.101.0 behavior of copying authentication codes by default. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) Explicit flag precedence and host scoping are internally consistent, and per-host config operation is supported by the CLI. ([cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get)) Local `git apply --check` and compilation could not be independently run because the repository shell sandbox failed before command execution; additionally, the summary's claim that the mock was regenerated is unclear because it also says moq could not read the toolchain export data." + } + }, + "task_score": 72.4, + "verdict": "The submission is detailed and mostly repository-shaped, but every artifact preserves a disabled default and therefore misses the task's defining requirement to copy authentication codes by default. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0))", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T19:57:58.576789Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 232108, + "cached_input_tokens": 148468, + "cache_write_input_tokens": 39210, + "output_tokens": 10216, + "reasoning_output_tokens": 8460 + }, + "duration_ms": 181417 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json new file mode 100644 index 00000000..770151c4 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "auth-clipboard-config-default", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "high", + "tags": [ + "auth", + "go", + "cli", + "config", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 62.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 144, + "output_tokens": 51908, + "cache_read_tokens": 9289448, + "cache_write_tokens": 168269, + "total_tokens": 9509769, + "total_cost_usd": 4.111767660000001, + "latency_seconds": 830.4, + "num_turns": 69, + "generation_tokens_per_sec": 62.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 144, + "output_tokens": 51908, + "latency_seconds": 830.4, + "num_turns": 69, + "total_cost_usd": 4.111767660000001, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 9289448, + "cache_creation_tokens": 168269, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T18:54:15.707309Z", + "run_ended_at": "2026-10-14T19:08:06.113346Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 144, + "output_tokens": 51908, + "cache_read_tokens": 9289448, + "cache_write_tokens": 168269, + "total_tokens": 9509769, + "total_cost_usd": 4.111767660000001, + "latency_seconds": 830.4, + "num_turns": 69, + "generation_tokens_per_sec": 62.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "auth-clipboard-config-default", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 17, + "correctness": 11, + "specificity": 23, + "risk_awareness": 18, + "total": 69, + "notes": "The issue provides unusually concrete precedence rules, acceptance criteria, user stories, and exclusions. Its central requirement is wrong: it specifies a disabled default and unchanged behavior, while v2.101.0 explicitly introduced copying by default with persistent opt-out via `clipboard disabled`. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) Issue #13153 supports the motivation and proposed configuration key, and the CLI manuals confirm per-host configuration support. ([github.com](https://github.com/cli/cli/issues/13153), [cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get)) No independent task statement was supplied, so exact diagnostic wording could not be established separately from the candidate's assertions." + }, + "lld": { + "completeness": 19, + "correctness": 13, + "specificity": 23, + "risk_awareness": 19, + "total": 74, + "notes": "The LLD is strongly grounded in named repository files, interfaces, option structs, mock generation, hostname resolution, and the existing auth-flow boundary. Its load-bearing decision is nevertheless incorrect: `DefaultValue: \"disabled\"` and the claimed no-change rollout contradict the released requirement that clipboard copying default to enabled. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) The host-scoped design is feasible because `gh config set`, `get`, and `list` expose per-host operation. ([cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get), [cli.github.com](https://cli.github.com/manual/gh_config_list)) It honestly records the untested `loginRun` wiring, but leaves that regression path unmitigated, and the unavailable local shell prevented complete independent validation of every line-level repository claim." + }, + "review": { + "completeness": 18, + "correctness": 15, + "specificity": 22, + "risk_awareness": 20, + "total": 75, + "notes": "The strongest finding is Byte's concrete identification of the existing `Test_refreshRun` case that must set `ClipboardProvided`, including the exact behavioral consequence. The review never challenges the design's disabled default, despite the corresponding release explicitly defaulting clipboard copying to enabled, so it misses the most important product-level defect. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) It also says there is “no new file” while the reviewed design adds `clipboard.go` and `clipboard_test.go`, an internal factual inconsistency. Its SSH/headless clipboard warning, generated-mock drift concern, discoverability feedback, and login wiring gap are otherwise specific and actionable." + }, + "testing": { + "completeness": 20, + "correctness": 12, + "specificity": 23, + "risk_awareness": 20, + "total": 75, + "notes": "The plan covers config validation, host precedence, both auth commands, explicit true and false flags, help, rollback, headless failures, and detailed output oracles. Its central compatibility and default assertions are backwards: it expects fresh configuration not to copy, whereas v2.101.0 intentionally copies by default and uses `clipboard disabled` as the opt-out. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) The proposed `go generate` drift check is also unsound immediately after applying an uncommitted patch because `git diff --exit-code internal/gh/mock/config.go` detects the intended patch itself. Most individual tests are precise, but no automated test observes the value passed by `loginRun`, leaving an acknowledged wiring regression uncovered." + }, + "implementation": { + "completeness": 18, + "correctness": 12, + "specificity": 22, + "risk_awareness": 17, + "total": 69, + "notes": "The patch coherently adds the config accessor and interface method, generated mock support, flag-provided tracking, a shared precedence resolver, login and refresh wiring, help text, and focused tests. However, `DefaultValue: \"disabled\"` and `clipboard: disabled` preserve the old default, directly failing the defining v2.101.0 behavior of copying authentication codes by default. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) Explicit flag precedence and host scoping are internally consistent, and per-host config operation is supported by the CLI. ([cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get)) Local `git apply --check` and compilation could not be independently run because the repository shell sandbox failed before command execution; additionally, the summary's claim that the mock was regenerated is unclear because it also says moq could not read the toolchain export data." + } + }, + "task_score": 72.4, + "verdict": "The submission is detailed and mostly repository-shaped, but every artifact preserves a disabled default and therefore misses the task's defining requirement to copy authentication codes by default. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0))", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T19:57:58.576789Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 232108, + "cached_input_tokens": 148468, + "cache_write_input_tokens": 39210, + "output_tokens": 10216, + "reasoning_output_tokens": 8460 + }, + "duration_ms": 181417 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json new file mode 100644 index 00000000..5515e1e6 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json @@ -0,0 +1,64 @@ +{ + "task": "auth-refresh-hint-on-expired-token", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The issue provides exact messages, token classifications, hostname mappings, testable acceptance criteria, and explicit exclusions. Its core problem matches upstream issue #13066, while the merged PR also corroborates the status-output scope. ([github.com](https://github.com/cli/cli/pull/13068), [github.com](https://github.com/cli/cli/issues/13066)) The largest deficiency is overbroad wording: an explicit Authorization header can make the failing token differ from ActiveToken, and invalid environment-token status entries do not necessarily print any reauthentication line. No independent task statement was supplied, so requirements beyond the identifier and corroborated upstream issue remain an evidence gap." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest aspect is its repository-specific data flow through internal/ghcmd/cmd.go, ActiveToken, NormalizeHostname, shared.IsRefreshableToken, and status.buildEntry. The submitted patch context supports those symbols and the design explicitly handles nil configuration, nil RequestURL, environment tokens, PATs, inactive accounts, and JSON stability. Its main weakness is reliance on duplicated private source strings plus the acknowledged header-override mismatch, with no mitigation beyond preserving the old fallback behavior. The claim about scope behavior after a rejected token could not be independently verified; the public manual only states that a normal refresh without scope flags maintains prior scopes. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review identifies concrete defects rather than merely approving: missing keyring coverage, brittle source literals, and misleading advice after an Authorization-header override. Those findings directly correspond to the proposed predicate, status active-account gate, and request-host lookup in the patch. Its largest omission is failure to challenge the issue's overbroad status criterion or the testing plan's keyring-isolation assumption. The active-account limitation is independently consistent with the gh auth refresh manual. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The plan has precise oracles for stored OAuth tokens, PATs, environment tokens, hostname normalization, inactive accounts, nil inputs, SAML behavior, missing scopes, JSON output, and recovery. Its automated paths, test names, httpmock style, and package commands agree with the submitted patch and repository conventions. The largest deficiency is that GH_CONFIG_DIR does not isolate credentials placed in the system keyring, so the manual setup can affect or leave real keyring state; it also tests authHint directly without an integration assertion through Main and omits the prescribed make lint command. The live revocation and recovery scenarios were not executed, so their operational assumptions remain unverified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements both output paths end to end, centralizes classification in IsRefreshableToken, preserves existing fallbacks, and adds focused positive and negative tests. Its file contexts and symbols are internally consistent with the baseline diff, and the official merged PR confirms the same feature shape and successful checks. ([github.com](https://github.com/cli/cli/pull/13068)) The largest residual risk is that the source-plus-prefix heuristic can describe a stored token rather than the token actually sent, while no test exercises the complete Main error-printing path. Local git apply and test execution could not be independently performed because the provided shell sandbox failed before command execution, leaving exact applicability as a harness evidence gap rather than a demonstrated candidate defect." + } + }, + "task_score": 88.4, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is imperfect identification of the token that actually caused a 401, compounded by incomplete end-to-end and keyring-safe verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:01:42.867660Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 203491, + "cached_input_tokens": 125948, + "cache_write_input_tokens": 32764, + "output_tokens": 13275, + "reasoning_output_tokens": 11136 + }, + "duration_ms": 224278 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json new file mode 100644 index 00000000..6aa639dc --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "auth-refresh-hint-on-expired-token", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "auth", + "go", + "cli", + "error-messages", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 62.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 100, + "output_tokens": 44974, + "cache_read_tokens": 4785039, + "cache_write_tokens": 123714, + "total_tokens": 4953827, + "total_cost_usd": 2.72300358, + "latency_seconds": 717.1, + "num_turns": 48, + "generation_tokens_per_sec": 62.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 100, + "output_tokens": 44974, + "latency_seconds": 717.1, + "num_turns": 48, + "total_cost_usd": 2.72300358, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4785039, + "cache_creation_tokens": 123714, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T18:23:01.415759Z", + "run_ended_at": "2026-10-14T18:34:58.531838Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 100, + "output_tokens": 44974, + "cache_read_tokens": 4785039, + "cache_write_tokens": 123714, + "total_tokens": 4953827, + "total_cost_usd": 2.72300358, + "latency_seconds": 717.1, + "num_turns": 48, + "generation_tokens_per_sec": 62.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "auth-refresh-hint-on-expired-token", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The issue provides exact messages, token classifications, hostname mappings, testable acceptance criteria, and explicit exclusions. Its core problem matches upstream issue #13066, while the merged PR also corroborates the status-output scope. ([github.com](https://github.com/cli/cli/pull/13068), [github.com](https://github.com/cli/cli/issues/13066)) The largest deficiency is overbroad wording: an explicit Authorization header can make the failing token differ from ActiveToken, and invalid environment-token status entries do not necessarily print any reauthentication line. No independent task statement was supplied, so requirements beyond the identifier and corroborated upstream issue remain an evidence gap." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest aspect is its repository-specific data flow through internal/ghcmd/cmd.go, ActiveToken, NormalizeHostname, shared.IsRefreshableToken, and status.buildEntry. The submitted patch context supports those symbols and the design explicitly handles nil configuration, nil RequestURL, environment tokens, PATs, inactive accounts, and JSON stability. Its main weakness is reliance on duplicated private source strings plus the acknowledged header-override mismatch, with no mitigation beyond preserving the old fallback behavior. The claim about scope behavior after a rejected token could not be independently verified; the public manual only states that a normal refresh without scope flags maintains prior scopes. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review identifies concrete defects rather than merely approving: missing keyring coverage, brittle source literals, and misleading advice after an Authorization-header override. Those findings directly correspond to the proposed predicate, status active-account gate, and request-host lookup in the patch. Its largest omission is failure to challenge the issue's overbroad status criterion or the testing plan's keyring-isolation assumption. The active-account limitation is independently consistent with the gh auth refresh manual. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The plan has precise oracles for stored OAuth tokens, PATs, environment tokens, hostname normalization, inactive accounts, nil inputs, SAML behavior, missing scopes, JSON output, and recovery. Its automated paths, test names, httpmock style, and package commands agree with the submitted patch and repository conventions. The largest deficiency is that GH_CONFIG_DIR does not isolate credentials placed in the system keyring, so the manual setup can affect or leave real keyring state; it also tests authHint directly without an integration assertion through Main and omits the prescribed make lint command. The live revocation and recovery scenarios were not executed, so their operational assumptions remain unverified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements both output paths end to end, centralizes classification in IsRefreshableToken, preserves existing fallbacks, and adds focused positive and negative tests. Its file contexts and symbols are internally consistent with the baseline diff, and the official merged PR confirms the same feature shape and successful checks. ([github.com](https://github.com/cli/cli/pull/13068)) The largest residual risk is that the source-plus-prefix heuristic can describe a stored token rather than the token actually sent, while no test exercises the complete Main error-printing path. Local git apply and test execution could not be independently performed because the provided shell sandbox failed before command execution, leaving exact applicability as a harness evidence gap rather than a demonstrated candidate defect." + } + }, + "task_score": 88.4, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is imperfect identification of the token that actually caused a 401, compounded by incomplete end-to-end and keyring-safe verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:01:42.867660Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 203491, + "cached_input_tokens": 125948, + "cache_write_input_tokens": 32764, + "output_tokens": 13275, + "reasoning_output_tokens": 11136 + }, + "duration_ms": 224278 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json new file mode 100644 index 00000000..c1fc976e --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json @@ -0,0 +1,64 @@ +{ + "task": "codespace-ports-forward-concurrent-map", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The issue clearly identifies the command, concurrency path, observed stack, user impact, exclusions, and testable acceptance criteria; issue #13399 confirms the command and `CreateTunnelPort` failure at manager.go:452. ([github.com](https://github.com/cli/cli/issues/13399)) Its strongest aspect is distinguishing the shared options-map race, tunnel mutation, and notification behavior instead of treating every reported error as one cause. The largest limitation is the absent independent task statement: the leaked-header explanation for HTTP 400 and lost-notification hang are technically reasoned extensions rather than established requirements, although the dependency's nonblocking single-recipient channel supports the latter. ([github.com](https://github.com/microsoft/dev-tunnels/blob/v0.0.7/go/tunnels/remote_forwarded_ports.go)) The compatibility criterion covering many unrelated codespace commands is also broader than necessary to define this bug." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is implementation-ready: it names the real connection and forwarder files, specifies mutex ownership, option cloning, pointer semantics, method signatures, polling behavior, mock synchronization, and rollout checks. Baseline source confirms that `CodespaceConnection` owns exported manager, options, and tunnel pointers while `NewPortForwarder` copies the connection value, making the selected lock ownership and pointer conversion appropriate. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go)) Its largest deficiency is that the latency estimate is explicitly unmeasured and it does not discuss context-unaware waiting to acquire `portsMu` during a slow management request. The exact v0.1.19 line-level mutation claims could not be independently retrieved from the local checkout, though the submitted issue stack and available dependency notification source corroborate the critical behavior." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 23, + "total": 90, + "notes": "The review performs real design stress rather than ceremonial approval, especially by finding the concurrent mock-map access, the lock-defeating connection copy, and the ability to bypass synchronization through exported fields. The value-copy and exported-field findings agree with the baseline `CodespaceConnection` and `NewPortForwarder` definitions. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go)) Recommendations are prioritized, actionable, and explicitly resolved in the revised LLD, while cancellation, serialization cost, security exposure, and testability receive proportionate treatment. The largest limitation is that the mock handler's precise line-level race could not be independently verified after local shell access failed, and several reviewer sections add little beyond approving unchanged behavior." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 82, + "notes": "The plan has concrete race-test commands, repeated scheduling runs, exact helper outcomes, manual multi-port checks, cancellation coverage, compatibility checks, and valid repository surfaces such as `make bin/gh` and codespace JSON fields. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile), [cli.github.com](https://cli.github.com/manual/gh_codespace_ports)) Its largest deficiency is that the acceptance criterion that stored request options remain unchanged is assigned only to code review rather than a test that invokes create and then inspects the original headers. The negative check requires manually reverting and adapting files, so it is not a stable reproducible test recipe, and the manual `org` visibility scenario depends on codespace policy. The concurrent test also omits the LLD's promised protocol oracle, so it can pass while creating requested ports with an incorrect protocol." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 22, + "risk_awareness": 20, + "total": 86, + "notes": "The patch implements the central design end to end: connection-wide serialization, per-call header-map cloning, shared pointer ownership, guarded mock state, bounded forwarded-port waits, and focused concurrency and cancellation tests. Its target symbols and surrounding contexts match the established baseline structure, and the submitted baseline hash corresponds to release v2.95.0. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go), [github.com](https://github.com/cli/cli/releases/tag/v2.95.0)) The largest defect is test coverage: no test proves `conn.options.AdditionalHeaders` remains unmodified, and `TestForwardPortConcurrently` verifies presence but not the promised HTTP protocol. Local sandbox initialization failed, so `git apply --check`, compilation, and repository-wide symbol searches could not be independently executed; no source-level contradiction was found through the available exact issue, release, and baseline evidence." + } + }, + "task_score": 87.2, + "verdict": "This is a strong, repository-specific solution whose main limitation is incomplete automated proof of request-option immutability and the absence of an independent task statement defining the extra notification scope.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:05:04.130328Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 161243, + "cached_input_tokens": 93851, + "cache_write_input_tokens": 31752, + "output_tokens": 12007, + "reasoning_output_tokens": 10116 + }, + "duration_ms": 201251 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json new file mode 100644 index 00000000..7b7a4f90 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "codespace-ports-forward-concurrent-map", + "repo": "https://github.com/cli/cli", + "ref": "v2.95.0", + "complexity": "medium", + "tags": [ + "codespace", + "go", + "cli", + "concurrency", + "race-condition", + "fixed-in-2.96.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 62.2, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 98, + "output_tokens": 50550, + "cache_read_tokens": 5097072, + "cache_write_tokens": 132017, + "total_tokens": 5279737, + "total_cost_usd": 2.95998054, + "latency_seconds": 812.8, + "num_turns": 48, + "generation_tokens_per_sec": 62.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 98, + "output_tokens": 50550, + "latency_seconds": 812.8, + "num_turns": 48, + "total_cost_usd": 2.95998054, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 5097072, + "cache_creation_tokens": 132017, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:54:29.470209Z", + "run_ended_at": "2026-10-14T18:08:02.313659Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 98, + "output_tokens": 50550, + "cache_read_tokens": 5097072, + "cache_write_tokens": 132017, + "total_tokens": 5279737, + "total_cost_usd": 2.95998054, + "latency_seconds": 812.8, + "num_turns": 48, + "generation_tokens_per_sec": 62.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "codespace-ports-forward-concurrent-map", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The issue clearly identifies the command, concurrency path, observed stack, user impact, exclusions, and testable acceptance criteria; issue #13399 confirms the command and `CreateTunnelPort` failure at manager.go:452. ([github.com](https://github.com/cli/cli/issues/13399)) Its strongest aspect is distinguishing the shared options-map race, tunnel mutation, and notification behavior instead of treating every reported error as one cause. The largest limitation is the absent independent task statement: the leaked-header explanation for HTTP 400 and lost-notification hang are technically reasoned extensions rather than established requirements, although the dependency's nonblocking single-recipient channel supports the latter. ([github.com](https://github.com/microsoft/dev-tunnels/blob/v0.0.7/go/tunnels/remote_forwarded_ports.go)) The compatibility criterion covering many unrelated codespace commands is also broader than necessary to define this bug." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is implementation-ready: it names the real connection and forwarder files, specifies mutex ownership, option cloning, pointer semantics, method signatures, polling behavior, mock synchronization, and rollout checks. Baseline source confirms that `CodespaceConnection` owns exported manager, options, and tunnel pointers while `NewPortForwarder` copies the connection value, making the selected lock ownership and pointer conversion appropriate. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go)) Its largest deficiency is that the latency estimate is explicitly unmeasured and it does not discuss context-unaware waiting to acquire `portsMu` during a slow management request. The exact v0.1.19 line-level mutation claims could not be independently retrieved from the local checkout, though the submitted issue stack and available dependency notification source corroborate the critical behavior." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 23, + "total": 90, + "notes": "The review performs real design stress rather than ceremonial approval, especially by finding the concurrent mock-map access, the lock-defeating connection copy, and the ability to bypass synchronization through exported fields. The value-copy and exported-field findings agree with the baseline `CodespaceConnection` and `NewPortForwarder` definitions. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go)) Recommendations are prioritized, actionable, and explicitly resolved in the revised LLD, while cancellation, serialization cost, security exposure, and testability receive proportionate treatment. The largest limitation is that the mock handler's precise line-level race could not be independently verified after local shell access failed, and several reviewer sections add little beyond approving unchanged behavior." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 82, + "notes": "The plan has concrete race-test commands, repeated scheduling runs, exact helper outcomes, manual multi-port checks, cancellation coverage, compatibility checks, and valid repository surfaces such as `make bin/gh` and codespace JSON fields. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile), [cli.github.com](https://cli.github.com/manual/gh_codespace_ports)) Its largest deficiency is that the acceptance criterion that stored request options remain unchanged is assigned only to code review rather than a test that invokes create and then inspects the original headers. The negative check requires manually reverting and adapting files, so it is not a stable reproducible test recipe, and the manual `org` visibility scenario depends on codespace policy. The concurrent test also omits the LLD's promised protocol oracle, so it can pass while creating requested ports with an incorrect protocol." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 22, + "risk_awareness": 20, + "total": 86, + "notes": "The patch implements the central design end to end: connection-wide serialization, per-call header-map cloning, shared pointer ownership, guarded mock state, bounded forwarded-port waits, and focused concurrency and cancellation tests. Its target symbols and surrounding contexts match the established baseline structure, and the submitted baseline hash corresponds to release v2.95.0. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go), [github.com](https://github.com/cli/cli/releases/tag/v2.95.0)) The largest defect is test coverage: no test proves `conn.options.AdditionalHeaders` remains unmodified, and `TestForwardPortConcurrently` verifies presence but not the promised HTTP protocol. Local sandbox initialization failed, so `git apply --check`, compilation, and repository-wide symbol searches could not be independently executed; no source-level contradiction was found through the available exact issue, release, and baseline evidence." + } + }, + "task_score": 87.2, + "verdict": "This is a strong, repository-specific solution whose main limitation is incomplete automated proof of request-option immutability and the absence of an independent task statement defining the extra notification scope.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:05:04.130328Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 161243, + "cached_input_tokens": 93851, + "cache_write_input_tokens": 31752, + "output_tokens": 12007, + "reasoning_output_tokens": 10116 + }, + "duration_ms": 201251 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json new file mode 100644 index 00000000..35301b9b --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json @@ -0,0 +1,64 @@ +{ + "task": "copilot-not-installed-trailing-newline", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue's strongest aspect is its byte-exact acceptance criteria covering stderr, stdout, return value, and prohibited side effects. The submitted source context corroborates the identified fmt.Fprintf call and cmdutil.SilentError path. Its largest deficiency is declaring the sibling non-interactive warning out of scope despite the absence of an independent task statement establishing that boundary. The external issue reference and exact repository line numbers could not be independently verified through the unavailable local shell." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD is implementation-ready for this small change, naming runCopilot, CanPrompt, PrompterMock, findCopilotBinaryFunc, and runExternalCmdFunc with concrete data flow and test behavior. Its proposed code and integration points agree with the submitted patch context. The main weakness is overstating regression risk as nonexistent and treating the non-interactive branch exclusion as task-mandated when the independent task statement is missing. Claims about IOStreams internals, exact line numbers, and existing test seams were plausible but could not all be independently read from the local repository." + }, + "review": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 23, + "total": 91, + "notes": "The review's strongest work is identifying concrete test defects: environment-dependent binary lookup, accidental download or execution, and an assertion that would pass without the newline. The implementation visibly adopts its recommendations through exact stderr equality, fatal side-effect guards, and cleanup of package-level seams. Its largest deficiency is that several reviewer sections add little scrutiny, while the scope decision for the sibling warning is accepted without resolving the missing task evidence. The assertion that the survey prompt itself supplies the preceding newline is inferred from reported output rather than independently verified." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The plan has a strong regression oracle: exact stderr bytes, empty stdout, SilentError, prompt parameters, and explicit download and execution guards. The package paths and focused go test command agree with the submitted implementation, and the negative mutation check would prove the test detects the original defect. Its main gaps are omitting the repository-required make lint check and presenting POSIX commands as applicable to PowerShell without adaptation. The commands and manual PTY behavior were not executed, so their platform portability remains unverified." + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The patch implements the design end to end: it adds the newline at the declined-install write and adds a focused test covering output, return value, prompt routing, and prohibited side effects. Its imports and referenced symbols are internally consistent with the surrounding submitted diff, and the official v2.99.0 release independently lists this exact declined-install newline fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The largest residual risk is mutation of package-level test seams, although avoiding t.Parallel and restoring them with t.Cleanup appropriately mitigates it. The claimed git apply --check result and compilation could not be independently rerun because the local read-only shell failed before executing repository commands." + } + }, + "task_score": 91.6, + "verdict": "This is a strong, precise submission with a complete regression patch, limited mainly by the missing independent task statement and unavailable independent local patch verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:07:55.548268Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 157129, + "cached_input_tokens": 93520, + "cache_write_input_tokens": 26444, + "output_tokens": 9383, + "reasoning_output_tokens": 7582 + }, + "duration_ms": 171405 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json new file mode 100644 index 00000000..279ed76c --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "copilot-not-installed-trailing-newline", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "trivial", + "tags": [ + "copilot", + "go", + "cli", + "terminal-output", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 69.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 44, + "output_tokens": 18749, + "cache_read_tokens": 1320280, + "cache_write_tokens": 52795, + "total_tokens": 1391868, + "total_cost_usd": 0.9935056999999998, + "latency_seconds": 268.9, + "num_turns": 21, + "generation_tokens_per_sec": 69.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 44, + "output_tokens": 18749, + "latency_seconds": 268.9, + "num_turns": 21, + "total_cost_usd": 0.9935056999999998, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1320280, + "cache_creation_tokens": 52795, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T16:58:57.588811Z", + "run_ended_at": "2026-10-14T17:03:26.448809Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 44, + "output_tokens": 18749, + "cache_read_tokens": 1320280, + "cache_write_tokens": 52795, + "total_tokens": 1391868, + "total_cost_usd": 0.9935056999999998, + "latency_seconds": 268.9, + "num_turns": 21, + "generation_tokens_per_sec": 69.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "copilot-not-installed-trailing-newline", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue's strongest aspect is its byte-exact acceptance criteria covering stderr, stdout, return value, and prohibited side effects. The submitted source context corroborates the identified fmt.Fprintf call and cmdutil.SilentError path. Its largest deficiency is declaring the sibling non-interactive warning out of scope despite the absence of an independent task statement establishing that boundary. The external issue reference and exact repository line numbers could not be independently verified through the unavailable local shell." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD is implementation-ready for this small change, naming runCopilot, CanPrompt, PrompterMock, findCopilotBinaryFunc, and runExternalCmdFunc with concrete data flow and test behavior. Its proposed code and integration points agree with the submitted patch context. The main weakness is overstating regression risk as nonexistent and treating the non-interactive branch exclusion as task-mandated when the independent task statement is missing. Claims about IOStreams internals, exact line numbers, and existing test seams were plausible but could not all be independently read from the local repository." + }, + "review": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 23, + "total": 91, + "notes": "The review's strongest work is identifying concrete test defects: environment-dependent binary lookup, accidental download or execution, and an assertion that would pass without the newline. The implementation visibly adopts its recommendations through exact stderr equality, fatal side-effect guards, and cleanup of package-level seams. Its largest deficiency is that several reviewer sections add little scrutiny, while the scope decision for the sibling warning is accepted without resolving the missing task evidence. The assertion that the survey prompt itself supplies the preceding newline is inferred from reported output rather than independently verified." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The plan has a strong regression oracle: exact stderr bytes, empty stdout, SilentError, prompt parameters, and explicit download and execution guards. The package paths and focused go test command agree with the submitted implementation, and the negative mutation check would prove the test detects the original defect. Its main gaps are omitting the repository-required make lint check and presenting POSIX commands as applicable to PowerShell without adaptation. The commands and manual PTY behavior were not executed, so their platform portability remains unverified." + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The patch implements the design end to end: it adds the newline at the declined-install write and adds a focused test covering output, return value, prompt routing, and prohibited side effects. Its imports and referenced symbols are internally consistent with the surrounding submitted diff, and the official v2.99.0 release independently lists this exact declined-install newline fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The largest residual risk is mutation of package-level test seams, although avoiding t.Parallel and restoring them with t.Cleanup appropriately mitigates it. The claimed git apply --check result and compilation could not be independently rerun because the local read-only shell failed before executing repository commands." + } + }, + "task_score": 91.6, + "verdict": "This is a strong, precise submission with a complete regression patch, limited mainly by the missing independent task statement and unavailable independent local patch verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:07:55.548268Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 157129, + "cached_input_tokens": 93520, + "cache_write_input_tokens": 26444, + "output_tokens": 9383, + "reasoning_output_tokens": 7582 + }, + "duration_ms": 171405 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json new file mode 100644 index 00000000..939ab81e --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json @@ -0,0 +1,64 @@ +{ + "task": "extension-uninstall-alias", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue crisply defines the alias behavior, compatibility guarantee, testable acceptance criteria, and narrow non-goals. Upstream PR #13599 confirms that adding Cobra's Aliases field is the intended solution and keeps remove canonical. ([github.com](https://github.com/cli/cli/pull/13599)) The current manual also confirms that uninstall is exposed through all three parent command forms. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The main evidence gap is the absent independent task statement, although the task identifier and upstream release entry substantiate the proposed scope." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD is unusually well grounded for a one-line change, naming the command literal, test table, help renderer, documentation generator, dispatch path, and backward-compatibility boundary. Repository evidence supports the root-level extension dispatch and the documented gen-docs flags. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/root/root.go), [github.com](https://github.com/cli/cli/blob/trunk/cmd/gen-docs/main.go)) Its largest factual defect is the asserted exact ALIASES ordering, which disagrees with both the upstream PR's manual check and the generated manual. ([github.com](https://github.com/cli/cli/pull/13599), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The telemetry behavior is plausible but was not directly verified against the v2.93.0 implementation." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 22, + "risk_awareness": 21, + "total": 87, + "notes": "The review identifies a concrete weakness in the original test proposal and converts it into an actionable missing-argument test that appears in the patch. Its collision analysis is supported by the root command's registration of installed extensions at the top level, outside gh extension. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/root/root.go)) The largest omission is that no reviewer challenges the absent direct help assertion or notices that the proposed exact alias ordering is wrong. The role breakdown adds some ceremony, but most conclusions remain specific and proportionate to this low-risk change." + }, + "testing": { + "completeness": 23, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 85, + "notes": "The plan covers alias dispatch, normalization, argument failure, backward compatibility, parent aliases, generated documentation, and a full install/remove lifecycle with concrete commands and oracles. The unit-test and documentation-generation commands agree with the repository's command structure and gen-docs flags. ([github.com](https://github.com/cli/cli/blob/trunk/cmd/gen-docs/main.go)) Its exact ALIASES oracle is incorrect: neither the upstream manual check nor the generated manual uses the stated ordering. ([github.com](https://github.com/cli/cli/pull/13599), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) It also expects usage output for the missing argument despite the root command silencing Cobra usage, and several live checks depend on mutable network state." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the requested behavior directly by adding Aliases to the existing remove command, exactly matching the mechanism merged upstream and listed in the v2.94.0 release. ([github.com](https://github.com/cli/cli/pull/13599), [github.com](https://github.com/cli/cli/releases/tag/v2.94.0)) The added table rows exercise real Cobra resolution, TTY output, full-name normalization, and inherited argument validation without changing removal logic. Its main limitation is the lack of an automated assertion for help output or parent aliases, although the declarative alias still implements those behaviors through shared infrastructure. Local git apply verification could not run because the provided sandbox failed to initialize, but the target symbols and source context align with the verified upstream change." + } + }, + "task_score": 88.8, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is a pair of inaccurate testing oracles, compounded by the absence of an independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:10:42.561693Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 149504, + "cached_input_tokens": 64816, + "cache_write_input_tokens": 22945, + "output_tokens": 11275, + "reasoning_output_tokens": 8265 + }, + "duration_ms": 167001 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json new file mode 100644 index 00000000..b0df22b3 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "extension-uninstall-alias", + "repo": "https://github.com/cli/cli", + "ref": "v2.93.0", + "complexity": "trivial", + "tags": [ + "extension", + "go", + "cli", + "cobra", + "ux", + "fixed-in-2.94.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 71.4, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 26, + "output_tokens": 15109, + "cache_read_tokens": 652343, + "cache_write_tokens": 47010, + "total_tokens": 714488, + "total_cost_usd": 0.7345828600000002, + "latency_seconds": 211.7, + "num_turns": 12, + "generation_tokens_per_sec": 71.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 26, + "output_tokens": 15109, + "latency_seconds": 211.7, + "num_turns": 12, + "total_cost_usd": 0.7345828600000002, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 652343, + "cache_creation_tokens": 47010, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:12:30.609081Z", + "run_ended_at": "2026-10-14T17:16:02.360734Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 26, + "output_tokens": 15109, + "cache_read_tokens": 652343, + "cache_write_tokens": 47010, + "total_tokens": 714488, + "total_cost_usd": 0.7345828600000002, + "latency_seconds": 211.7, + "num_turns": 12, + "generation_tokens_per_sec": 71.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "extension-uninstall-alias", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue crisply defines the alias behavior, compatibility guarantee, testable acceptance criteria, and narrow non-goals. Upstream PR #13599 confirms that adding Cobra's Aliases field is the intended solution and keeps remove canonical. ([github.com](https://github.com/cli/cli/pull/13599)) The current manual also confirms that uninstall is exposed through all three parent command forms. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The main evidence gap is the absent independent task statement, although the task identifier and upstream release entry substantiate the proposed scope." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD is unusually well grounded for a one-line change, naming the command literal, test table, help renderer, documentation generator, dispatch path, and backward-compatibility boundary. Repository evidence supports the root-level extension dispatch and the documented gen-docs flags. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/root/root.go), [github.com](https://github.com/cli/cli/blob/trunk/cmd/gen-docs/main.go)) Its largest factual defect is the asserted exact ALIASES ordering, which disagrees with both the upstream PR's manual check and the generated manual. ([github.com](https://github.com/cli/cli/pull/13599), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The telemetry behavior is plausible but was not directly verified against the v2.93.0 implementation." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 22, + "risk_awareness": 21, + "total": 87, + "notes": "The review identifies a concrete weakness in the original test proposal and converts it into an actionable missing-argument test that appears in the patch. Its collision analysis is supported by the root command's registration of installed extensions at the top level, outside gh extension. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/root/root.go)) The largest omission is that no reviewer challenges the absent direct help assertion or notices that the proposed exact alias ordering is wrong. The role breakdown adds some ceremony, but most conclusions remain specific and proportionate to this low-risk change." + }, + "testing": { + "completeness": 23, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 85, + "notes": "The plan covers alias dispatch, normalization, argument failure, backward compatibility, parent aliases, generated documentation, and a full install/remove lifecycle with concrete commands and oracles. The unit-test and documentation-generation commands agree with the repository's command structure and gen-docs flags. ([github.com](https://github.com/cli/cli/blob/trunk/cmd/gen-docs/main.go)) Its exact ALIASES oracle is incorrect: neither the upstream manual check nor the generated manual uses the stated ordering. ([github.com](https://github.com/cli/cli/pull/13599), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) It also expects usage output for the missing argument despite the root command silencing Cobra usage, and several live checks depend on mutable network state." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the requested behavior directly by adding Aliases to the existing remove command, exactly matching the mechanism merged upstream and listed in the v2.94.0 release. ([github.com](https://github.com/cli/cli/pull/13599), [github.com](https://github.com/cli/cli/releases/tag/v2.94.0)) The added table rows exercise real Cobra resolution, TTY output, full-name normalization, and inherited argument validation without changing removal logic. Its main limitation is the lack of an automated assertion for help output or parent aliases, although the declarative alias still implements those behaviors through shared infrastructure. Local git apply verification could not run because the provided sandbox failed to initialize, but the target symbols and source context align with the verified upstream change." + } + }, + "task_score": 88.8, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is a pair of inaccurate testing oracles, compounded by the absence of an independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:10:42.561693Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 149504, + "cached_input_tokens": 64816, + "cache_write_input_tokens": 22945, + "output_tokens": 11275, + "reasoning_output_tokens": 8265 + }, + "duration_ms": 167001 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json new file mode 100644 index 00000000..72e049d3 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json @@ -0,0 +1,64 @@ +{ + "task": "issue-view-comments-with-json", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 12, + "specificity": 23, + "risk_awareness": 18, + "total": 74, + "notes": "The issue gives concrete reproductions, unchanged cases, exclusions, and testable exit-code and no-request criteria. Its largest defect is requiring `--comments --json comments` to remain valid; the official v2.99.0 release describes the shipped fix as rejecting `--comments` with `--json`, and later usage evidence describes outright rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) It usefully covers scripts, sibling commands, and unrelated search qualifiers. With no independent task statement and unavailable local shell access, the exact issue #14214 wording and claimed acceptance-test usages could not be verified." + }, + "lld": { + "completeness": 22, + "correctness": 11, + "specificity": 23, + "risk_awareness": 19, + "total": 75, + "notes": "The LLD is highly specific about command constructors, exporter lifecycle, validation ordering, error handling, files, and tests. Official manuals confirm that the three commands expose the relevant `--comments` and JSON surfaces. ([cli.github.com](https://cli.github.com/manual/gh_issue_view), [cli.github.com](https://cli.github.com/manual/gh_pr_view), [cli.github.com](https://cli.github.com/manual/gh_discussion_view)) Its central conditional-rejection design conflicts with the shipped blanket rejection and leaves PR review parity explicitly unresolved. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Exact line numbers, `PreRunE` behavior, and acceptance-fixture claims could not be independently checked locally." + }, + "review": { + "completeness": 18, + "correctness": 12, + "specificity": 21, + "risk_awareness": 18, + "total": 69, + "notes": "The review finds a concrete discussion error-precedence bug and converts it into an actionable ordering test. It also identifies the important PR-specific fact that human `--comments` output includes reviews, although it permits that mismatch to be deferred. Its largest miss is endorsing the conditional policy without challenging the upstream blanket-rejection contract evidenced by the v2.99.0 fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Claims about exact acceptance fixtures and existing source expressions were not independently verifiable through the unavailable local shell." + }, + "testing": { + "completeness": 19, + "correctness": 11, + "specificity": 21, + "risk_awareness": 17, + "total": 68, + "notes": "The plan provides a broad command matrix, exact error oracles, precedence cases, compatibility checks, and live smoke tests. Its build command is grounded in the repository Makefile, which defines `bin/gh` as a target. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile), [github.com](https://github.com/cli/cli/blob/trunk/Makefile)) The main deficiency is that several tests require the combined flags to succeed when `comments` is selected, contradicting the shipped rejection behavior. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) The HTTP-debug grep is not demonstrated for all commands, and “same output as v2.98.0” is not a concrete regression oracle." + }, + "implementation": { + "completeness": 15, + "correctness": 10, + "specificity": 20, + "risk_awareness": 14, + "total": 59, + "notes": "The diff consistently adds guards and focused constructor tests across issue, PR, and discussion view, including discussion error precedence and prevention of `runF` execution. The stated baseline commit matches the official v2.98.0 release listing. ([github.com](https://github.com/cli/cli/releases)) Its largest defect is implementing conditional rejection rather than the shipped rule that rejects `--comments` with `--json` outright, so a key combined invocation remains accepted. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Local `git apply --check` and compilation could not be repeated because the shell sandbox failed to start, while the summary confirms tests and formatting were not executed." + } + }, + "task_score": 69.0, + "verdict": "The artifacts are detailed and internally coherent, but their central decision to preserve one `--comments`/`--json` combination materially diverges from the upstream fix.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:13:56.648170Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 203891, + "cached_input_tokens": 116734, + "cache_write_input_tokens": 31509, + "output_tokens": 11675, + "reasoning_output_tokens": 9735 + }, + "duration_ms": 194073 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json new file mode 100644 index 00000000..fc00cdf2 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "issue-view-comments-with-json", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "medium", + "tags": [ + "issue", + "go", + "cli", + "json-output", + "flags", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 95.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 80, + "output_tokens": 34133, + "cache_read_tokens": 3316354, + "cache_write_tokens": 92737, + "total_tokens": 3443304, + "total_cost_usd": 1.9909293799999996, + "latency_seconds": 356.3, + "num_turns": 38, + "generation_tokens_per_sec": 95.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 80, + "output_tokens": 34133, + "latency_seconds": 356.3, + "num_turns": 38, + "total_cost_usd": 1.9909293799999996, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3316354, + "cache_creation_tokens": 92737, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T18:08:05.833617Z", + "run_ended_at": "2026-10-14T18:14:02.193825Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 80, + "output_tokens": 34133, + "cache_read_tokens": 3316354, + "cache_write_tokens": 92737, + "total_tokens": 3443304, + "total_cost_usd": 1.9909293799999996, + "latency_seconds": 356.3, + "num_turns": 38, + "generation_tokens_per_sec": 95.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "issue-view-comments-with-json", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 12, + "specificity": 23, + "risk_awareness": 18, + "total": 74, + "notes": "The issue gives concrete reproductions, unchanged cases, exclusions, and testable exit-code and no-request criteria. Its largest defect is requiring `--comments --json comments` to remain valid; the official v2.99.0 release describes the shipped fix as rejecting `--comments` with `--json`, and later usage evidence describes outright rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) It usefully covers scripts, sibling commands, and unrelated search qualifiers. With no independent task statement and unavailable local shell access, the exact issue #14214 wording and claimed acceptance-test usages could not be verified." + }, + "lld": { + "completeness": 22, + "correctness": 11, + "specificity": 23, + "risk_awareness": 19, + "total": 75, + "notes": "The LLD is highly specific about command constructors, exporter lifecycle, validation ordering, error handling, files, and tests. Official manuals confirm that the three commands expose the relevant `--comments` and JSON surfaces. ([cli.github.com](https://cli.github.com/manual/gh_issue_view), [cli.github.com](https://cli.github.com/manual/gh_pr_view), [cli.github.com](https://cli.github.com/manual/gh_discussion_view)) Its central conditional-rejection design conflicts with the shipped blanket rejection and leaves PR review parity explicitly unresolved. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Exact line numbers, `PreRunE` behavior, and acceptance-fixture claims could not be independently checked locally." + }, + "review": { + "completeness": 18, + "correctness": 12, + "specificity": 21, + "risk_awareness": 18, + "total": 69, + "notes": "The review finds a concrete discussion error-precedence bug and converts it into an actionable ordering test. It also identifies the important PR-specific fact that human `--comments` output includes reviews, although it permits that mismatch to be deferred. Its largest miss is endorsing the conditional policy without challenging the upstream blanket-rejection contract evidenced by the v2.99.0 fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Claims about exact acceptance fixtures and existing source expressions were not independently verifiable through the unavailable local shell." + }, + "testing": { + "completeness": 19, + "correctness": 11, + "specificity": 21, + "risk_awareness": 17, + "total": 68, + "notes": "The plan provides a broad command matrix, exact error oracles, precedence cases, compatibility checks, and live smoke tests. Its build command is grounded in the repository Makefile, which defines `bin/gh` as a target. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile), [github.com](https://github.com/cli/cli/blob/trunk/Makefile)) The main deficiency is that several tests require the combined flags to succeed when `comments` is selected, contradicting the shipped rejection behavior. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) The HTTP-debug grep is not demonstrated for all commands, and “same output as v2.98.0” is not a concrete regression oracle." + }, + "implementation": { + "completeness": 15, + "correctness": 10, + "specificity": 20, + "risk_awareness": 14, + "total": 59, + "notes": "The diff consistently adds guards and focused constructor tests across issue, PR, and discussion view, including discussion error precedence and prevention of `runF` execution. The stated baseline commit matches the official v2.98.0 release listing. ([github.com](https://github.com/cli/cli/releases)) Its largest defect is implementing conditional rejection rather than the shipped rule that rejects `--comments` with `--json` outright, so a key combined invocation remains accepted. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Local `git apply --check` and compilation could not be repeated because the shell sandbox failed to start, while the summary confirms tests and formatting were not executed." + } + }, + "task_score": 69.0, + "verdict": "The artifacts are detailed and internally coherent, but their central decision to preserve one `--comments`/`--json` combination materially diverges from the upstream fix.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:13:56.648170Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 203891, + "cached_input_tokens": 116734, + "cache_write_input_tokens": 31509, + "output_tokens": 11675, + "reasoning_output_tokens": 9735 + }, + "duration_ms": 194073 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json new file mode 100644 index 00000000..642fa11e --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-checks-all-cancelled-summary", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The issue precisely identifies the omitted counts.Canceled term, desired TTY behavior, compatibility constraints, and concrete regression oracle. Repository history confirms this root cause and the one-term fix in pkg/cmd/pr/checks/output.go. ([github.com](https://github.com/cli/cli/pull/13679)) Its main limitation is the absent independent task statement, so the requested wording and unchanged exit code are established only by the artifacts and upstream issue context. Explicit non-goals for non-TTY status, JSON, wording, and exit behavior make the scope unusually clear." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The LLD grounds the change in printSummary, checkCounts, aggregateChecks, checksRun, the table-driven tests, and the existing fixture convention. Its monotonicity argument correctly explains why adding Canceled admits only the previously omitted cancelled-only case while preserving headline precedence. The largest deficiency is minor bookkeeping: the approximately 76-line estimate does not match the submitted patch's 59 additions and one deletion, and the command-level fixture approach is heavier than a direct printSummary unit test. It nevertheless addresses TTY, non-TTY, watch-path reuse, zero-check behavior, exit compatibility, rollout, and future field-enumeration fragility." + }, + "review": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest finding is the concrete demand for a non-TTY regression case, including its exact tab-separated oracle, which the revised design and patch adopt. The review also correctly checks headline precedence, exit-code independence, table padding, watch-path reuse, scripted-output compatibility, and the absence of a new security surface. Its main weakness is repetition across reviewer personas and limited challenge to the relatively expensive fixture-based test strategy; it does not suggest the focused printSummary branch table later used upstream. No material repository claim beyond the missing independent task requirements remained unresolved." + }, + "testing": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The plan provides focused Go test commands, exact TTY and non-TTY output oracles, a negative regression check, existing-case coverage, manual GitHub verification, and exit-code checks. The subtest regexes and package paths agree with the submitted Test_checksRun additions, and the expected 1m26s duration follows from the fixture timestamps. Its largest correctness defect is the before/after TTY comparison redirecting both commands to files, which makes stdout non-TTY and therefore cannot compare TTY summaries despite claiming to do so. It also omits the repository-required make lint validation and uses a grep pipeline that does not itself preserve the go test exit status." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The patch implements the complete behavioral fix by adding counts.Canceled to the existing printSummary guard, exactly matching the confirmed upstream correction shipped after v2.95.0. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) Its fixture and two Test_checksRun cases exercise both the newly fixed TTY output and unchanged non-TTY row and success behavior, with hunk contexts and symbols consistent with the submitted baseline. The largest evidence limitation is that the local sandbox prevented an independent git apply --check, while the implementation summary also honestly states that tests were not executed; absence of test execution is not scored as an implementation defect. Apart from lacking an all-cancelled watch-specific test, the change is minimal, convention-consistent, and faithful to the revised LLD." + } + }, + "task_score": 92.4, + "verdict": "This is a precise, repository-grounded fix with strong design and regression coverage; the largest material limitation is the testing plan's invalid redirected TTY comparison and omitted lint step.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:17:05.318608Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 137775, + "cached_input_tokens": 71564, + "cache_write_input_tokens": 27042, + "output_tokens": 11753, + "reasoning_output_tokens": 9536 + }, + "duration_ms": 188658 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json new file mode 100644 index 00000000..ca9051e7 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-checks-all-cancelled-summary", + "repo": "https://github.com/cli/cli", + "ref": "v2.95.0", + "complexity": "trivial", + "tags": [ + "pr", + "go", + "cli", + "checks", + "terminal-output", + "fixed-in-2.96.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 68.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 38, + "output_tokens": 19912, + "cache_read_tokens": 1160944, + "cache_write_tokens": 60247, + "total_tokens": 1241141, + "total_cost_usd": 1.02499738, + "latency_seconds": 290.3, + "num_turns": 18, + "generation_tokens_per_sec": 68.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 38, + "output_tokens": 19912, + "latency_seconds": 290.3, + "num_turns": 18, + "total_cost_usd": 1.02499738, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1160944, + "cache_creation_tokens": 60247, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:07:36.761473Z", + "run_ended_at": "2026-10-14T17:12:27.077881Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 38, + "output_tokens": 19912, + "cache_read_tokens": 1160944, + "cache_write_tokens": 60247, + "total_tokens": 1241141, + "total_cost_usd": 1.02499738, + "latency_seconds": 290.3, + "num_turns": 18, + "generation_tokens_per_sec": 68.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-checks-all-cancelled-summary", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The issue precisely identifies the omitted counts.Canceled term, desired TTY behavior, compatibility constraints, and concrete regression oracle. Repository history confirms this root cause and the one-term fix in pkg/cmd/pr/checks/output.go. ([github.com](https://github.com/cli/cli/pull/13679)) Its main limitation is the absent independent task statement, so the requested wording and unchanged exit code are established only by the artifacts and upstream issue context. Explicit non-goals for non-TTY status, JSON, wording, and exit behavior make the scope unusually clear." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The LLD grounds the change in printSummary, checkCounts, aggregateChecks, checksRun, the table-driven tests, and the existing fixture convention. Its monotonicity argument correctly explains why adding Canceled admits only the previously omitted cancelled-only case while preserving headline precedence. The largest deficiency is minor bookkeeping: the approximately 76-line estimate does not match the submitted patch's 59 additions and one deletion, and the command-level fixture approach is heavier than a direct printSummary unit test. It nevertheless addresses TTY, non-TTY, watch-path reuse, zero-check behavior, exit compatibility, rollout, and future field-enumeration fragility." + }, + "review": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest finding is the concrete demand for a non-TTY regression case, including its exact tab-separated oracle, which the revised design and patch adopt. The review also correctly checks headline precedence, exit-code independence, table padding, watch-path reuse, scripted-output compatibility, and the absence of a new security surface. Its main weakness is repetition across reviewer personas and limited challenge to the relatively expensive fixture-based test strategy; it does not suggest the focused printSummary branch table later used upstream. No material repository claim beyond the missing independent task requirements remained unresolved." + }, + "testing": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The plan provides focused Go test commands, exact TTY and non-TTY output oracles, a negative regression check, existing-case coverage, manual GitHub verification, and exit-code checks. The subtest regexes and package paths agree with the submitted Test_checksRun additions, and the expected 1m26s duration follows from the fixture timestamps. Its largest correctness defect is the before/after TTY comparison redirecting both commands to files, which makes stdout non-TTY and therefore cannot compare TTY summaries despite claiming to do so. It also omits the repository-required make lint validation and uses a grep pipeline that does not itself preserve the go test exit status." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The patch implements the complete behavioral fix by adding counts.Canceled to the existing printSummary guard, exactly matching the confirmed upstream correction shipped after v2.95.0. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) Its fixture and two Test_checksRun cases exercise both the newly fixed TTY output and unchanged non-TTY row and success behavior, with hunk contexts and symbols consistent with the submitted baseline. The largest evidence limitation is that the local sandbox prevented an independent git apply --check, while the implementation summary also honestly states that tests were not executed; absence of test execution is not scored as an implementation defect. Apart from lacking an all-cancelled watch-specific test, the change is minimal, convention-consistent, and faithful to the revised LLD." + } + }, + "task_score": 92.4, + "verdict": "This is a precise, repository-grounded fix with strong design and regression coverage; the largest material limitation is the testing plan's invalid redirected TTY comparison and omitted lint step.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:17:05.318608Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 137775, + "cached_input_tokens": 71564, + "cache_write_input_tokens": 27042, + "output_tokens": 11753, + "reasoning_output_tokens": 9536 + }, + "duration_ms": 188658 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json new file mode 100644 index 00000000..9f4dd8f3 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-comment-attach-ghu-tokens", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 87, + "notes": "The issue provides unusually testable acceptance criteria covering success, refusal, authorization scheme, rejected token classes, and host-check ordering. Repository evidence supports the existing enterprise rejection and token-scheme behavior described. ([github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/13317)) Its largest deficiency is provenance: no independent task statement exists, issue #14495 could not be verified, and the supplied labels omit the required help wanted eligibility signal. The endpoint's ghu_ acceptance, Bearer 404 behavior, and Copilot credential claim therefore remain empirical assertions rather than verified repository facts." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD is implementation-ready, naming the central allowlist, token classifier, callers, error path, request transport, and concrete test changes. Its account of the existing host and permission gates agrees with available repository evidence, while the proposed one-entry allowlist change preserves those controls. ([github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/14302), [github.com](https://github.com/cli/cli/issues/13317)) The main deficiency is that the load-bearing claim that ghu_ tokens receive HTTP 201 is not independently established, and data-residency support is explicitly unresolved. It also does not address the repository's upstream-contribution eligibility gate despite describing a release rollout." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds concrete design defects rather than merely approving: missing JSON Content-Type in StubUpload, brittle full-URL error assertions, and the need to pin host-before-token validation. Those findings directly shape the revised tests and are technically coherent with the patch's use of StatusJSONResponse and substring assertions. Its largest omission is failure to challenge the unverified endpoint premise or the absent help wanted authorization required for an intended upstream contribution. The detailed StubUpload and HandleHTTPError behavior could not be independently checked against local source because repository shell inspection was unavailable." + }, + "testing": { + "completeness": 21, + "correctness": 19, + "specificity": 23, + "risk_awareness": 19, + "total": 82, + "notes": "The strongest coverage is the unit matrix with exact oracles for URL, status, header scheme, request count, unchanged rejection behavior, and comment mutation suppression. The package paths and token-header expectation align with the repository behavior available for verification. ([github.com](https://github.com/cli/cli/issues/13317), [github.com](https://github.com/cli/cli/issues/14309)) The proposed live refusal setup may fail during repository lookup before reaching upload because an app lacking repository access cannot necessarily exercise the intended branch, and the eight-byte PNG fixture is not a valid renderable image. The plan also omits the repository-required go fix and make lint gates and provides no cleanup plan for comments and uploaded assets created by manual tests." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the design end to end: one shared allowlist addition plus focused tests for acceptance, authorization, server refusal, host ordering, and the comment path. Available repository evidence confirms the surrounding enterprise gate, restricted token policy, and token Authorization scheme that the patch relies upon. ([github.com](https://github.com/cli/cli/issues/14309), [github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/13317)) No material code defect is apparent, though the implementation proceeded without establishing the required upstream issue eligibility and its comment states the 201 result more categorically than the permission-dependent design warrants. Exact git apply applicability and imports could not be independently rechecked because the local read-only shell wrapper failed, so that claim remains an evidence gap rather than a scored execution failure." + } + }, + "task_score": 87.0, + "verdict": "Strong, repository-specific work with a likely correct low-risk patch, but the required upstream issue eligibility and the core live-endpoint premise were not independently established.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:20:37.878680Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 212592, + "cached_input_tokens": 134396, + "cache_write_input_tokens": 30580, + "output_tokens": 13027, + "reasoning_output_tokens": 11535 + }, + "duration_ms": 212547 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json new file mode 100644 index 00000000..d99cfc7d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-comment-attach-ghu-tokens", + "repo": "https://github.com/cli/cli", + "ref": "v2.101.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "auth", + "attachments", + "fixed-in-2.102.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 90.0, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 86, + "output_tokens": 34164, + "cache_read_tokens": 4309020, + "cache_write_tokens": 117025, + "total_tokens": 4460295, + "total_cost_usd": 2.3436083, + "latency_seconds": 379.6, + "num_turns": 41, + "generation_tokens_per_sec": 90.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 86, + "output_tokens": 34164, + "latency_seconds": 379.6, + "num_turns": 41, + "total_cost_usd": 2.3436083, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4309020, + "cache_creation_tokens": 117025, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:40:49.899321Z", + "run_ended_at": "2026-10-14T17:47:09.517179Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 86, + "output_tokens": 34164, + "cache_read_tokens": 4309020, + "cache_write_tokens": 117025, + "total_tokens": 4460295, + "total_cost_usd": 2.3436083, + "latency_seconds": 379.6, + "num_turns": 41, + "generation_tokens_per_sec": 90.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-comment-attach-ghu-tokens", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 87, + "notes": "The issue provides unusually testable acceptance criteria covering success, refusal, authorization scheme, rejected token classes, and host-check ordering. Repository evidence supports the existing enterprise rejection and token-scheme behavior described. ([github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/13317)) Its largest deficiency is provenance: no independent task statement exists, issue #14495 could not be verified, and the supplied labels omit the required help wanted eligibility signal. The endpoint's ghu_ acceptance, Bearer 404 behavior, and Copilot credential claim therefore remain empirical assertions rather than verified repository facts." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD is implementation-ready, naming the central allowlist, token classifier, callers, error path, request transport, and concrete test changes. Its account of the existing host and permission gates agrees with available repository evidence, while the proposed one-entry allowlist change preserves those controls. ([github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/14302), [github.com](https://github.com/cli/cli/issues/13317)) The main deficiency is that the load-bearing claim that ghu_ tokens receive HTTP 201 is not independently established, and data-residency support is explicitly unresolved. It also does not address the repository's upstream-contribution eligibility gate despite describing a release rollout." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds concrete design defects rather than merely approving: missing JSON Content-Type in StubUpload, brittle full-URL error assertions, and the need to pin host-before-token validation. Those findings directly shape the revised tests and are technically coherent with the patch's use of StatusJSONResponse and substring assertions. Its largest omission is failure to challenge the unverified endpoint premise or the absent help wanted authorization required for an intended upstream contribution. The detailed StubUpload and HandleHTTPError behavior could not be independently checked against local source because repository shell inspection was unavailable." + }, + "testing": { + "completeness": 21, + "correctness": 19, + "specificity": 23, + "risk_awareness": 19, + "total": 82, + "notes": "The strongest coverage is the unit matrix with exact oracles for URL, status, header scheme, request count, unchanged rejection behavior, and comment mutation suppression. The package paths and token-header expectation align with the repository behavior available for verification. ([github.com](https://github.com/cli/cli/issues/13317), [github.com](https://github.com/cli/cli/issues/14309)) The proposed live refusal setup may fail during repository lookup before reaching upload because an app lacking repository access cannot necessarily exercise the intended branch, and the eight-byte PNG fixture is not a valid renderable image. The plan also omits the repository-required go fix and make lint gates and provides no cleanup plan for comments and uploaded assets created by manual tests." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the design end to end: one shared allowlist addition plus focused tests for acceptance, authorization, server refusal, host ordering, and the comment path. Available repository evidence confirms the surrounding enterprise gate, restricted token policy, and token Authorization scheme that the patch relies upon. ([github.com](https://github.com/cli/cli/issues/14309), [github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/13317)) No material code defect is apparent, though the implementation proceeded without establishing the required upstream issue eligibility and its comment states the 201 result more categorically than the permission-dependent design warrants. Exact git apply applicability and imports could not be independently rechecked because the local read-only shell wrapper failed, so that claim remains an evidence gap rather than a scored execution failure." + } + }, + "task_score": 87.0, + "verdict": "Strong, repository-specific work with a likely correct low-risk patch, but the required upstream issue eligibility and the core live-endpoint premise were not independently established.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:20:37.878680Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 212592, + "cached_input_tokens": 134396, + "cache_write_input_tokens": 30580, + "output_tokens": 13027, + "reasoning_output_tokens": 11535 + }, + "duration_ms": 212547 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json new file mode 100644 index 00000000..cffb83a6 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-merge-delete-branch-owner-case", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The issue precisely connects the casing mismatch to `NewMergeContext`, the `crossRepoPR` guard, and the missing branch-delete request, with reproducible steps and testable same-repository and fork criteria. The pinned v2.100.0 evidence confirms the case-sensitive comparison at `merge.go:592` and early return at `merge.go:519`. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its main deficiency is that the output acceptance criterion does not distinguish TTY from non-TTY behavior, although the baseline suppresses that informational line outside a TTY. No independent task statement was supplied, so intended scope beyond the task identifier and internally consistent artifacts could not be verified." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The LLD is implementation-ready: it names the exact field-list replacement, assignment, consumers, fixture migrations, API call, and interactive behavior. Repository evidence supports both the existing faulty query and the finder behavior that adds `isCrossRepository` and `headRepositoryOwner` during branch lookup. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that the proposed tests do not assert the `FindOptions.Fields` contract, so omission of the new GraphQL field would not be caught by the mocked-PR regressions. The broad GHES compatibility claim is plausible but lacks a supported-version boundary or direct pinned-schema verification." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review finds a concrete deficiency in the original design - missing interactive-prompt coverage - and identifies the two fork fixtures that must gain `IsCrossRepository: true`. Its fixture analysis agrees with the mock finder, which backfills only `HeadRepositoryOwner.Login` and leaves the boolean at its zero value. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) The largest missed concern is that neither proposed regression verifies that `NewMergeContext` actually requests `isCrossRepository`. Some GHES and deleted-head-repository assertions are presented confidently without pinned repository or compatibility evidence." + }, + "testing": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The plan supplies concrete commands, fixtures, HTTP expectations, prompt text, negative behavior, backward-compatibility scenarios, and end-to-end branch-state oracles. The key REST oracle matches the documented `--delete-branch` contract and the baseline failure path. ([cli.github.com](https://cli.github.com/manual/gh_pr_merge), [gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest gap is again the absence of a field-selection assertion, allowing tests to pass with a stubbed `IsCrossRepository` value even if the real GraphQL query stopped requesting it. The manual fork and GHES procedures require substantial external account and server setup, and the GHES expectation is not tied to a stated supported version." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the design end to end by swapping the requested field, assigning `crossRepoPR` from `pr.IsCrossRepository`, updating both fork fixtures, and adding flag-path and prompt regressions. The `merge.go` hunk matches the verified v2.100.0 symbols and line context, and the resulting delete path is consistent with the existing guard and REST behavior. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The largest weakness is that the tests inject the boolean through `StubFinderForRunCommandStyleTests` without asserting the requested field list, leaving the API wiring less protected than the decision logic. No behavioral deviation from the LLD is apparent, but local `git apply --check`, compilation, and formatting could not be independently executed because the provided shell sandbox failed during initialization." + } + }, + "task_score": 91.0, + "verdict": "This is an excellent, repository-grounded fix with focused regression coverage; its largest material limitation is that no test pins the GraphQL field request that supplies the new decision value.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:22:53.123717Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 170770, + "cached_input_tokens": 77720, + "cache_write_input_tokens": 28149, + "output_tokens": 8573, + "reasoning_output_tokens": 7079 + }, + "duration_ms": 135233 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json new file mode 100644 index 00000000..b9113cb4 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-merge-delete-branch-owner-case", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "git", + "remotes", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 66.3, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 29842, + "cache_read_tokens": 2242200, + "cache_write_tokens": 85792, + "total_tokens": 2357892, + "total_cost_usd": 1.6219192, + "latency_seconds": 449.9, + "num_turns": 28, + "generation_tokens_per_sec": 66.3, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 58, + "output_tokens": 29842, + "latency_seconds": 449.9, + "num_turns": 28, + "total_cost_usd": 1.6219192, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2242200, + "cache_creation_tokens": 85792, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:16:05.610550Z", + "run_ended_at": "2026-10-14T17:23:35.476385Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 29842, + "cache_read_tokens": 2242200, + "cache_write_tokens": 85792, + "total_tokens": 2357892, + "total_cost_usd": 1.6219192, + "latency_seconds": 449.9, + "num_turns": 28, + "generation_tokens_per_sec": 66.3, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-merge-delete-branch-owner-case", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The issue precisely connects the casing mismatch to `NewMergeContext`, the `crossRepoPR` guard, and the missing branch-delete request, with reproducible steps and testable same-repository and fork criteria. The pinned v2.100.0 evidence confirms the case-sensitive comparison at `merge.go:592` and early return at `merge.go:519`. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its main deficiency is that the output acceptance criterion does not distinguish TTY from non-TTY behavior, although the baseline suppresses that informational line outside a TTY. No independent task statement was supplied, so intended scope beyond the task identifier and internally consistent artifacts could not be verified." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The LLD is implementation-ready: it names the exact field-list replacement, assignment, consumers, fixture migrations, API call, and interactive behavior. Repository evidence supports both the existing faulty query and the finder behavior that adds `isCrossRepository` and `headRepositoryOwner` during branch lookup. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that the proposed tests do not assert the `FindOptions.Fields` contract, so omission of the new GraphQL field would not be caught by the mocked-PR regressions. The broad GHES compatibility claim is plausible but lacks a supported-version boundary or direct pinned-schema verification." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review finds a concrete deficiency in the original design - missing interactive-prompt coverage - and identifies the two fork fixtures that must gain `IsCrossRepository: true`. Its fixture analysis agrees with the mock finder, which backfills only `HeadRepositoryOwner.Login` and leaves the boolean at its zero value. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) The largest missed concern is that neither proposed regression verifies that `NewMergeContext` actually requests `isCrossRepository`. Some GHES and deleted-head-repository assertions are presented confidently without pinned repository or compatibility evidence." + }, + "testing": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The plan supplies concrete commands, fixtures, HTTP expectations, prompt text, negative behavior, backward-compatibility scenarios, and end-to-end branch-state oracles. The key REST oracle matches the documented `--delete-branch` contract and the baseline failure path. ([cli.github.com](https://cli.github.com/manual/gh_pr_merge), [gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest gap is again the absence of a field-selection assertion, allowing tests to pass with a stubbed `IsCrossRepository` value even if the real GraphQL query stopped requesting it. The manual fork and GHES procedures require substantial external account and server setup, and the GHES expectation is not tied to a stated supported version." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the design end to end by swapping the requested field, assigning `crossRepoPR` from `pr.IsCrossRepository`, updating both fork fixtures, and adding flag-path and prompt regressions. The `merge.go` hunk matches the verified v2.100.0 symbols and line context, and the resulting delete path is consistent with the existing guard and REST behavior. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The largest weakness is that the tests inject the boolean through `StubFinderForRunCommandStyleTests` without asserting the requested field list, leaving the API wiring less protected than the decision logic. No behavioral deviation from the LLD is apparent, but local `git apply --check`, compilation, and formatting could not be independently executed because the provided shell sandbox failed during initialization." + } + }, + "task_score": 91.0, + "verdict": "This is an excellent, repository-grounded fix with focused regression coverage; its largest material limitation is that no test pins the GraphQL field request that supplies the new decision value.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:22:53.123717Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 170770, + "cached_input_tokens": 77720, + "cache_write_input_tokens": 28149, + "output_tokens": 8573, + "reasoning_output_tokens": 7079 + }, + "duration_ms": 135233 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json new file mode 100644 index 00000000..27ed978a --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-view-json-number-on-issue", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 18, + "total": 89, + "notes": "The issue provides a precise reproduction, scoped solution, negative and positive acceptance criteria, and selector coverage. Its diagnosis and remedy match the repository's merged fix for #13319 and the v2.93.0 release record. ([github.com](https://github.com/cli/cli/pull/13327), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The main deficiency is limited discussion of the added network request and behavior changes inherited by other shared-finder callers. No independent task statement was supplied, so requirements beyond the identifier and linked repository fix could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD is unusually well grounded, naming the actual finder, callers, field normalization, merge-ref path, query, and test structures. The repository source confirms that merge refs set `prNumber`, `id` and `number` are added before querying, and known numbers flow through `findByNumber` after the fix. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that loss of the former offline/no-request behavior is not analyzed explicitly, while the fixed one-point cost assertion is not established by repository evidence. Broader requirement coverage remains unverifiable because no independent task statement was provided." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The review finds concrete weaknesses in its own draft, including missing merge-ref coverage, lack of a user-path regression test, and failure to assert the GraphQL variable. Those concerns are technically relevant because the finder source contains the cited merge-ref path and unconditional numbered lookup. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its main correctness blemish is the claim that every caller subsequently talks to GitHub, which conflicts with its own description of JSON export and browser-only checks behavior. Some reviewer-role sections add little beyond approval, and the stated rate-limit figures were not independently verified." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The plan supplies concrete fixtures, actions, exact output and error oracles, selector variants, compatibility checks, and a mutation check that would expose restoration of the shortcut. Its build command is valid because the repository Makefile provides a `bin/gh` target. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile)) The largest omission is repository-wide verification: it specifies only `go test ./pkg/cmd/pr/...` and does not include the required `go test ./...` or `make lint`. Several live checks depend on authentication and mutable repository data, and their exact browser output was not independently verified." + }, + "implementation": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The production change removes exactly the synthetic-number shortcut identified by the task, matching the repository's merged fix, while the added tests cover success, GraphQL failure, URL selection, merge-ref selection, and JSON output. ([github.com](https://github.com/cli/cli/pull/13327), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The stated v2.92.0 baseline commit is consistent with the release record, and the patch context aligns with the documented pre-fix branch and current post-fix finder flow. ([github.com](https://github.com/cli/cli/releases/tag/v2.92.0), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) The largest coverage gap is the absence of an automated `#N` selector case, although that parser path is unchanged and the testing plan covers it manually. Shell sandbox failure prevented an independent `git apply --check` or compilation run, so the summary's apply claim and test compilation could not be directly verified." + } + }, + "task_score": 90.0, + "verdict": "Excellent, coherent work whose largest material limitation is that the verification plan omits the repository-mandated full test suite and lint run.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:26:16.641477Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 238901, + "cached_input_tokens": 136820, + "cache_write_input_tokens": 29503, + "output_tokens": 12302, + "reasoning_output_tokens": 10557 + }, + "duration_ms": 203505 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json new file mode 100644 index 00000000..96fd5234 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-view-json-number-on-issue", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "json-output", + "api", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 61.9, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 76, + "output_tokens": 32418, + "cache_read_tokens": 3115501, + "cache_write_tokens": 89833, + "total_tokens": 3237828, + "total_cost_usd": 1.8930221200000001, + "latency_seconds": 523.4, + "num_turns": 37, + "generation_tokens_per_sec": 61.9, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 76, + "output_tokens": 32418, + "latency_seconds": 523.4, + "num_turns": 37, + "total_cost_usd": 1.8930221200000001, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3115501, + "cache_creation_tokens": 89833, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:23:39.953756Z", + "run_ended_at": "2026-10-14T17:32:23.415531Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 76, + "output_tokens": 32418, + "cache_read_tokens": 3115501, + "cache_write_tokens": 89833, + "total_tokens": 3237828, + "total_cost_usd": 1.8930221200000001, + "latency_seconds": 523.4, + "num_turns": 37, + "generation_tokens_per_sec": 61.9, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-view-json-number-on-issue", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 18, + "total": 89, + "notes": "The issue provides a precise reproduction, scoped solution, negative and positive acceptance criteria, and selector coverage. Its diagnosis and remedy match the repository's merged fix for #13319 and the v2.93.0 release record. ([github.com](https://github.com/cli/cli/pull/13327), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The main deficiency is limited discussion of the added network request and behavior changes inherited by other shared-finder callers. No independent task statement was supplied, so requirements beyond the identifier and linked repository fix could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD is unusually well grounded, naming the actual finder, callers, field normalization, merge-ref path, query, and test structures. The repository source confirms that merge refs set `prNumber`, `id` and `number` are added before querying, and known numbers flow through `findByNumber` after the fix. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that loss of the former offline/no-request behavior is not analyzed explicitly, while the fixed one-point cost assertion is not established by repository evidence. Broader requirement coverage remains unverifiable because no independent task statement was provided." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The review finds concrete weaknesses in its own draft, including missing merge-ref coverage, lack of a user-path regression test, and failure to assert the GraphQL variable. Those concerns are technically relevant because the finder source contains the cited merge-ref path and unconditional numbered lookup. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its main correctness blemish is the claim that every caller subsequently talks to GitHub, which conflicts with its own description of JSON export and browser-only checks behavior. Some reviewer-role sections add little beyond approval, and the stated rate-limit figures were not independently verified." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The plan supplies concrete fixtures, actions, exact output and error oracles, selector variants, compatibility checks, and a mutation check that would expose restoration of the shortcut. Its build command is valid because the repository Makefile provides a `bin/gh` target. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile)) The largest omission is repository-wide verification: it specifies only `go test ./pkg/cmd/pr/...` and does not include the required `go test ./...` or `make lint`. Several live checks depend on authentication and mutable repository data, and their exact browser output was not independently verified." + }, + "implementation": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The production change removes exactly the synthetic-number shortcut identified by the task, matching the repository's merged fix, while the added tests cover success, GraphQL failure, URL selection, merge-ref selection, and JSON output. ([github.com](https://github.com/cli/cli/pull/13327), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The stated v2.92.0 baseline commit is consistent with the release record, and the patch context aligns with the documented pre-fix branch and current post-fix finder flow. ([github.com](https://github.com/cli/cli/releases/tag/v2.92.0), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) The largest coverage gap is the absence of an automated `#N` selector case, although that parser path is unchanged and the testing plan covers it manually. Shell sandbox failure prevented an independent `git apply --check` or compilation run, so the summary's apply claim and test compilation could not be directly verified." + } + }, + "task_score": 90.0, + "verdict": "Excellent, coherent work whose largest material limitation is that the verification plan omits the repository-mandated full test suite and lint run.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:26:16.641477Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 238901, + "cached_input_tokens": 136820, + "cache_write_input_tokens": 29503, + "output_tokens": 12302, + "reasoning_output_tokens": 10557 + }, + "duration_ms": 203505 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json new file mode 100644 index 00000000..7599f1fe --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json @@ -0,0 +1,64 @@ +{ + "task": "project-item-add-output-non-tty", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 20, + "specificity": 23, + "risk_awareness": 17, + "total": 83, + "notes": "The issue provides exact mode-specific output, testable acceptance criteria, and clear non-goals. Its root-cause claim matches `printResults`, which returns silently for non-TTY output, while the exporter path already precedes that function. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go)) The largest weakness is its unsupported retry-count anecdote and the claim that empty stdout prevents detecting success despite the command's zero exit status. No independent task statement was supplied, so the ID and help-text requirements are established only by these internally consistent artifacts." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is implementation-ready, naming the correct command, mutation result, exporter branch, TTY decision, tests, and exact output. The inspected source confirms that `runAddItem` exports the returned `ProjectItem` before calling `printResults`, and that the proposed edit targets the correct function. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go)) It also addresses write errors, compatibility with scripts expecting empty output, rollout, and the absence of extra API calls. The GraphQL non-null assertion and every claimed sibling-command output pattern could not be independently verified from the local tree because the provided shell sandbox failed." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 22, + "risk_awareness": 20, + "total": 85, + "notes": "The strongest review finding is concrete: explicitly force non-TTY mode in the JSON test and use different content and mutation IDs so the new assertion proves the correct data source. Those recommendations directly exercise the exporter precedence visible in `runAddItem` and the ID-based follow-up workflow used by `item-edit`. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go), [github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-edit/item_edit.go)) Its main deficiency is that much of the persona-based review approves the design without challenging the weak exit-status motivation or requesting automated help-text coverage. The security and compatibility conclusions are reasonable, but the submission provides no independent task statement against which to verify broader expectations." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The plan supplies strong oracles for exact stdout, trailing newlines, JSON precedence, failure separation, forced TTY behavior, ID reuse, and help output. Its flags and positional syntax agree with the documented `gh project item-add` interface, while the proposed ID chaining is compatible with `item-edit` and `item-delete`. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-edit/item_edit.go)) The largest weakness is that several live tests repeatedly mutate the same project without systematic cleanup, making the suite stateful and potentially disruptive despite using a test project. Failure and help behavior are only manual checks rather than durable unit tests, and execution was not demonstrated." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the design end to end: it changes the exact silent non-TTY branch to print `item.ID()`, preserves TTY and exporter behavior, documents the output, and adds focused regression coverage. Its context matches the inspected `item_add.go` structure, and the official v2.98.0 release identifies this same non-TTY defect as fixed. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go), [github.com](https://github.com/cli/cli/releases)) The new test correctly distinguishes the content ID from the created project-item ID and explicitly pins JSON execution to non-TTY mode. The main deficiencies are use of `assert.NoError` instead of the repository-mandated `require.NoError`, absence of executed tests or formatting checks, and an apply-check claim that could not be independently repeated because the local shell sandbox failed." + } + }, + "task_score": 86.2, + "verdict": "This is a strong, repository-specific submission with a correct and nearly complete patch, limited mainly by unsupported motivation claims, stateful manual testing, and unexecuted verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:29:28.237742Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 202048, + "cached_input_tokens": 125576, + "cache_write_input_tokens": 27032, + "output_tokens": 10593, + "reasoning_output_tokens": 9059 + }, + "duration_ms": 191583 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json new file mode 100644 index 00000000..30409b7d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "project-item-add-output-non-tty", + "repo": "https://github.com/cli/cli", + "ref": "v2.97.0", + "complexity": "low", + "tags": [ + "project", + "go", + "cli", + "tty", + "automation", + "fixed-in-2.98.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 103.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 44, + "output_tokens": 23419, + "cache_read_tokens": 1420295, + "cache_write_tokens": 61294, + "total_tokens": 1505052, + "total_cost_usd": 1.1649935000000002, + "latency_seconds": 225.9, + "num_turns": 21, + "generation_tokens_per_sec": 103.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 44, + "output_tokens": 23419, + "latency_seconds": 225.9, + "num_turns": 21, + "total_cost_usd": 1.1649935000000002, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1420295, + "cache_creation_tokens": 61294, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:32:27.121757Z", + "run_ended_at": "2026-10-14T17:36:13.041710Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 44, + "output_tokens": 23419, + "cache_read_tokens": 1420295, + "cache_write_tokens": 61294, + "total_tokens": 1505052, + "total_cost_usd": 1.1649935000000002, + "latency_seconds": 225.9, + "num_turns": 21, + "generation_tokens_per_sec": 103.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "project-item-add-output-non-tty", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 20, + "specificity": 23, + "risk_awareness": 17, + "total": 83, + "notes": "The issue provides exact mode-specific output, testable acceptance criteria, and clear non-goals. Its root-cause claim matches `printResults`, which returns silently for non-TTY output, while the exporter path already precedes that function. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go)) The largest weakness is its unsupported retry-count anecdote and the claim that empty stdout prevents detecting success despite the command's zero exit status. No independent task statement was supplied, so the ID and help-text requirements are established only by these internally consistent artifacts." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is implementation-ready, naming the correct command, mutation result, exporter branch, TTY decision, tests, and exact output. The inspected source confirms that `runAddItem` exports the returned `ProjectItem` before calling `printResults`, and that the proposed edit targets the correct function. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go)) It also addresses write errors, compatibility with scripts expecting empty output, rollout, and the absence of extra API calls. The GraphQL non-null assertion and every claimed sibling-command output pattern could not be independently verified from the local tree because the provided shell sandbox failed." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 22, + "risk_awareness": 20, + "total": 85, + "notes": "The strongest review finding is concrete: explicitly force non-TTY mode in the JSON test and use different content and mutation IDs so the new assertion proves the correct data source. Those recommendations directly exercise the exporter precedence visible in `runAddItem` and the ID-based follow-up workflow used by `item-edit`. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go), [github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-edit/item_edit.go)) Its main deficiency is that much of the persona-based review approves the design without challenging the weak exit-status motivation or requesting automated help-text coverage. The security and compatibility conclusions are reasonable, but the submission provides no independent task statement against which to verify broader expectations." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The plan supplies strong oracles for exact stdout, trailing newlines, JSON precedence, failure separation, forced TTY behavior, ID reuse, and help output. Its flags and positional syntax agree with the documented `gh project item-add` interface, while the proposed ID chaining is compatible with `item-edit` and `item-delete`. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-edit/item_edit.go)) The largest weakness is that several live tests repeatedly mutate the same project without systematic cleanup, making the suite stateful and potentially disruptive despite using a test project. Failure and help behavior are only manual checks rather than durable unit tests, and execution was not demonstrated." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the design end to end: it changes the exact silent non-TTY branch to print `item.ID()`, preserves TTY and exporter behavior, documents the output, and adds focused regression coverage. Its context matches the inspected `item_add.go` structure, and the official v2.98.0 release identifies this same non-TTY defect as fixed. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go), [github.com](https://github.com/cli/cli/releases)) The new test correctly distinguishes the content ID from the created project-item ID and explicitly pins JSON execution to non-TTY mode. The main deficiencies are use of `assert.NoError` instead of the repository-mandated `require.NoError`, absence of executed tests or formatting checks, and an apply-check claim that could not be independently repeated because the local shell sandbox failed." + } + }, + "task_score": 86.2, + "verdict": "This is a strong, repository-specific submission with a correct and nearly complete patch, limited mainly by unsupported motivation claims, stateful manual testing, and unexecuted verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:29:28.237742Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 202048, + "cached_input_tokens": 125576, + "cache_write_input_tokens": 27032, + "output_tokens": 10593, + "reasoning_output_tokens": 9059 + }, + "duration_ms": 191583 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json new file mode 100644 index 00000000..fa2227c5 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json @@ -0,0 +1,64 @@ +{ + "task": "project-item-edit-by-name", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The issue provides a crisp workflow problem, detailed acceptance criteria, compatibility requirements, and explicit non-goals. Its baseline claims align with v2.96.0 behavior, while the number/owner/URL interface matches the subsequently released command surface. The largest limitation is that no independent task statement establishes the proposed expansion of --value to text, number, date, and iteration; the public command documentation only specifically describes --value for single-select fields. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) Claims such as the exact opaque error text were not independently substantiated." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The LLD is strongly grounded in real components including item_edit.go, item-add, queries.Client.NewOwner, NewProject, ProjectField, and the existing mutation dispatch. It commits to concrete validation, pagination, matching, conversion, error, compatibility, and rollout behavior rather than leaving major implementation decisions open. Its largest weakness is product-scope uncertainty: generic conversion and iteration support materially exceed the externally verified released interface, and GHES schema compatibility is not analyzed. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) The stated 50-field cap and per-query rate costs are also asserted without repository evidence." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 23, + "total": 89, + "notes": "The review finds concrete defects with real consequences, notably missing projectItems pagination, non-finite numbers, empty values, and unnecessary lookups. It prioritizes them and records specific design resolutions, making this substantially more useful than role-based approval theater. The main omission is failure to challenge whether all field types should acquire new --value semantics when the verified released interface emphasizes single-select option names. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) The rate-cost and host-scoping assertions are plausible but not demonstrated from named repository code." + }, + "testing": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 83, + "notes": "The plan has strong behavior-level oracles for every proposed data type, negative resolution paths, pagination, case matching, compatibility, output, and the no-mutation guarantee. The package path and focused command go test ./pkg/cmd/project/item-edit/... agree with the submitted repository structure. However, the GH_DEBUG pipeline reports grep's exit status rather than gh's, and the legacy --number 0 example reuses FIELD_ID initialized for Notes while claiming it is an Estimate field. It also omits the repository-required full go test ./... and make lint checks, reducing confidence in integration and convention compliance." + }, + "implementation": { + "completeness": 20, + "correctness": 19, + "specificity": 21, + "risk_awareness": 19, + "total": 79, + "notes": "The patch implements its own design end to end: it extends the existing options and validation, resolves names before dispatch, paginates projectItems, converts values, preserves the ID-only mutation path, and adds substantial tests. Its target baseline and public surface are credible: v2.96.0 is commit b300f2e, and v2.97.0 introduced number/owner/URL/field/value addressing. ([github.com](https://github.com/cli/cli/releases/)) The largest concrete quality defect is that new tests repeatedly use assert for error checks despite the repository instruction requiring require, and the pull-request test mocks a PullRequest response for an issue-form URL. Compilation and lint remain unconfirmed because the summary explicitly reports no build, formatting, or test execution, although absence of execution was not itself penalized." + } + }, + "task_score": 85.4, + "verdict": "Strong, unusually concrete artifacts and a mostly coherent implementation are limited by an unverified expansion beyond the externally established single-select name-resolution scope and several test and repository-convention defects.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:33:26.757997Z", + "repo_ref": "v2.96.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 267979, + "cached_input_tokens": 171764, + "cache_write_input_tokens": 45488, + "output_tokens": 14729, + "reasoning_output_tokens": 12774 + }, + "duration_ms": 238507 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json new file mode 100644 index 00000000..a2b1538b --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "project-item-edit-by-name", + "repo": "https://github.com/cli/cli", + "ref": "v2.96.0", + "complexity": "high", + "tags": [ + "project", + "go", + "cli", + "graphql", + "fixed-in-2.97.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 76.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 88, + "output_tokens": 77789, + "cache_read_tokens": 5394193, + "cache_write_tokens": 165946, + "total_tokens": 5638016, + "total_cost_usd": 3.81117066, + "latency_seconds": 1016.8, + "num_turns": 42, + "generation_tokens_per_sec": 76.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 88, + "output_tokens": 77789, + "latency_seconds": 1016.8, + "num_turns": 42, + "total_cost_usd": 3.81117066, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 5394193, + "cache_creation_tokens": 165946, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T19:08:09.395689Z", + "run_ended_at": "2026-10-14T19:25:06.258254Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 88, + "output_tokens": 77789, + "cache_read_tokens": 5394193, + "cache_write_tokens": 165946, + "total_tokens": 5638016, + "total_cost_usd": 3.81117066, + "latency_seconds": 1016.8, + "num_turns": 42, + "generation_tokens_per_sec": 76.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "project-item-edit-by-name", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The issue provides a crisp workflow problem, detailed acceptance criteria, compatibility requirements, and explicit non-goals. Its baseline claims align with v2.96.0 behavior, while the number/owner/URL interface matches the subsequently released command surface. The largest limitation is that no independent task statement establishes the proposed expansion of --value to text, number, date, and iteration; the public command documentation only specifically describes --value for single-select fields. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) Claims such as the exact opaque error text were not independently substantiated." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The LLD is strongly grounded in real components including item_edit.go, item-add, queries.Client.NewOwner, NewProject, ProjectField, and the existing mutation dispatch. It commits to concrete validation, pagination, matching, conversion, error, compatibility, and rollout behavior rather than leaving major implementation decisions open. Its largest weakness is product-scope uncertainty: generic conversion and iteration support materially exceed the externally verified released interface, and GHES schema compatibility is not analyzed. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) The stated 50-field cap and per-query rate costs are also asserted without repository evidence." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 23, + "total": 89, + "notes": "The review finds concrete defects with real consequences, notably missing projectItems pagination, non-finite numbers, empty values, and unnecessary lookups. It prioritizes them and records specific design resolutions, making this substantially more useful than role-based approval theater. The main omission is failure to challenge whether all field types should acquire new --value semantics when the verified released interface emphasizes single-select option names. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) The rate-cost and host-scoping assertions are plausible but not demonstrated from named repository code." + }, + "testing": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 83, + "notes": "The plan has strong behavior-level oracles for every proposed data type, negative resolution paths, pagination, case matching, compatibility, output, and the no-mutation guarantee. The package path and focused command go test ./pkg/cmd/project/item-edit/... agree with the submitted repository structure. However, the GH_DEBUG pipeline reports grep's exit status rather than gh's, and the legacy --number 0 example reuses FIELD_ID initialized for Notes while claiming it is an Estimate field. It also omits the repository-required full go test ./... and make lint checks, reducing confidence in integration and convention compliance." + }, + "implementation": { + "completeness": 20, + "correctness": 19, + "specificity": 21, + "risk_awareness": 19, + "total": 79, + "notes": "The patch implements its own design end to end: it extends the existing options and validation, resolves names before dispatch, paginates projectItems, converts values, preserves the ID-only mutation path, and adds substantial tests. Its target baseline and public surface are credible: v2.96.0 is commit b300f2e, and v2.97.0 introduced number/owner/URL/field/value addressing. ([github.com](https://github.com/cli/cli/releases/)) The largest concrete quality defect is that new tests repeatedly use assert for error checks despite the repository instruction requiring require, and the pull-request test mocks a PullRequest response for an issue-form URL. Compilation and lint remain unconfirmed because the summary explicitly reports no build, formatting, or test execution, although absence of execution was not itself penalized." + } + }, + "task_score": 85.4, + "verdict": "Strong, unusually concrete artifacts and a mostly coherent implementation are limited by an unverified expansion beyond the externally established single-select name-resolution scope and several test and repository-convention defects.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:33:26.757997Z", + "repo_ref": "v2.96.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 267979, + "cached_input_tokens": 171764, + "cache_write_input_tokens": 45488, + "output_tokens": 14729, + "reasoning_output_tokens": 12774 + }, + "duration_ms": 238507 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json new file mode 100644 index 00000000..8bf599f2 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json @@ -0,0 +1,64 @@ +{ + "task": "release-create-404-scope-blame", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue names the affected file and helper, explains the parsing failure, distinguishes all three token cases, and supplies testable acceptance criteria; the v2.98.0 release records this same fix, and project API traces show comma-space scope headers. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0), [github.com](https://github.com/cli/cli/issues/9927)) Its largest limitation is the absent independent task statement, so broader requirements must be inferred, while the claim that GitHub alphabetically sorts scopes was not independently verified. The first/middle/last, look-alike, missing-header, command-error, and retained-hint criteria are unusually specific. It also identifies the remaining 404 heuristic and explicitly excludes broader parser refactoring and cause detection." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD follows the real `createRun` to `createRelease` error path, commits to a trim-and-exact-match loop, identifies the unused `slices` import, and specifies realistic JSON HTTP stubs and exact error output. Its cited repository convention is supported by `generateScopesSuggestion`, which splits the same header and applies `strings.TrimSpace`. ([github.com](https://github.com/cli/cli/issues/9403)) The main gap is that its helper-test matrix does not literally exercise first, middle, and last positions both with and without spaces as promised by the issue. Local line numbers, the asserted go-gh dependency version, and the claimed absence of an existing `http_test.go` could not be independently checked because local command execution failed at the sandbox layer." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 83, + "notes": "The review finds substantive defects rather than merely approving: it catches position-sensitive coverage, the need for a JSON content type to obtain `Not Found`, the unused import, and unsafe substring matching. Its largest error is claiming scripts that grep stderr are unaffected, even though the intended behavior explicitly changes stderr from the hint to the underlying error for normally formatted scoped-token responses. The security discussion correctly focuses on whole-token equality and preserving the empty-header heuristic, and the SRE section recommends release-note communication. The assertion that this package had no existing `http_test.go` was not independently verifiable under the unavailable local execution environment." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 17, + "total": 77, + "notes": "The strongest portion is the automated coverage: it supplies exact commands, concrete boolean and error-string oracles, negative look-alikes, stderr exclusivity, legacy headers, and package/full-suite checks. The major correctness defect is the E2E workflow-change scenario, which creates `wf-change` and a tag only locally but never pushes the branch before passing it to `--target`, even though that option identifies the remote branch or commit used for automatic tag creation. ([cli.github.com](http://cli.github.com/manual/gh_release_create)) The plan also omits the repository-required `make lint` check and mutates authentication scopes and creates releases without a restoration or cleanup procedure. The `--remove-scopes` flag itself is valid, so the problem is test isolation rather than an invented command. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "implementation": { + "completeness": 22, + "correctness": 24, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the design end to end: it removes the sole claimed `slices` use, trims each comma-delimited scope while preserving exact equality, and adds helper and command-level tests for both resulting 404 branches. GitHub's v2.98.0 release independently lists `Trim spaces when parsing X-Oauth-Scopes in gh release create`, strongly corroborating the chosen production fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) The largest limitation is test completeness: the helper cases do not fully cover every first/middle/last and spaced/unspaced combination required by the issue, although the loop itself handles them. The claimed `git apply --check` result and exact working-tree applicability could not be rerun because the local sandbox wrapper failed before command execution; no contradictory context was found in the submitted baseline diff." + } + }, + "task_score": 86.8, + "verdict": "This is a strong, repository-specific submission with a correct minimal implementation, limited chiefly by a flawed manual E2E setup, incomplete test-environment cleanup and lint coverage, and the absence of an independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:37:15.131189Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 219653, + "cached_input_tokens": 130748, + "cache_write_input_tokens": 28584, + "output_tokens": 14460, + "reasoning_output_tokens": 12317 + }, + "duration_ms": 228360 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json new file mode 100644 index 00000000..897a0978 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "release-create-404-scope-blame", + "repo": "https://github.com/cli/cli", + "ref": "v2.97.0", + "complexity": "low", + "tags": [ + "release", + "go", + "cli", + "auth", + "error-handling", + "fixed-in-2.98.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 93.0, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 62, + "output_tokens": 25129, + "cache_read_tokens": 2139595, + "cache_write_tokens": 65982, + "total_tokens": 2230768, + "total_cost_usd": 1.3867227000000004, + "latency_seconds": 270.2, + "num_turns": 30, + "generation_tokens_per_sec": 93.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 62, + "output_tokens": 25129, + "latency_seconds": 270.2, + "num_turns": 30, + "total_cost_usd": 1.3867227000000004, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2139595, + "cache_creation_tokens": 65982, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:36:16.339371Z", + "run_ended_at": "2026-10-14T17:40:46.537342Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 62, + "output_tokens": 25129, + "cache_read_tokens": 2139595, + "cache_write_tokens": 65982, + "total_tokens": 2230768, + "total_cost_usd": 1.3867227000000004, + "latency_seconds": 270.2, + "num_turns": 30, + "generation_tokens_per_sec": 93.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "release-create-404-scope-blame", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue names the affected file and helper, explains the parsing failure, distinguishes all three token cases, and supplies testable acceptance criteria; the v2.98.0 release records this same fix, and project API traces show comma-space scope headers. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0), [github.com](https://github.com/cli/cli/issues/9927)) Its largest limitation is the absent independent task statement, so broader requirements must be inferred, while the claim that GitHub alphabetically sorts scopes was not independently verified. The first/middle/last, look-alike, missing-header, command-error, and retained-hint criteria are unusually specific. It also identifies the remaining 404 heuristic and explicitly excludes broader parser refactoring and cause detection." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD follows the real `createRun` to `createRelease` error path, commits to a trim-and-exact-match loop, identifies the unused `slices` import, and specifies realistic JSON HTTP stubs and exact error output. Its cited repository convention is supported by `generateScopesSuggestion`, which splits the same header and applies `strings.TrimSpace`. ([github.com](https://github.com/cli/cli/issues/9403)) The main gap is that its helper-test matrix does not literally exercise first, middle, and last positions both with and without spaces as promised by the issue. Local line numbers, the asserted go-gh dependency version, and the claimed absence of an existing `http_test.go` could not be independently checked because local command execution failed at the sandbox layer." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 83, + "notes": "The review finds substantive defects rather than merely approving: it catches position-sensitive coverage, the need for a JSON content type to obtain `Not Found`, the unused import, and unsafe substring matching. Its largest error is claiming scripts that grep stderr are unaffected, even though the intended behavior explicitly changes stderr from the hint to the underlying error for normally formatted scoped-token responses. The security discussion correctly focuses on whole-token equality and preserving the empty-header heuristic, and the SRE section recommends release-note communication. The assertion that this package had no existing `http_test.go` was not independently verifiable under the unavailable local execution environment." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 17, + "total": 77, + "notes": "The strongest portion is the automated coverage: it supplies exact commands, concrete boolean and error-string oracles, negative look-alikes, stderr exclusivity, legacy headers, and package/full-suite checks. The major correctness defect is the E2E workflow-change scenario, which creates `wf-change` and a tag only locally but never pushes the branch before passing it to `--target`, even though that option identifies the remote branch or commit used for automatic tag creation. ([cli.github.com](http://cli.github.com/manual/gh_release_create)) The plan also omits the repository-required `make lint` check and mutates authentication scopes and creates releases without a restoration or cleanup procedure. The `--remove-scopes` flag itself is valid, so the problem is test isolation rather than an invented command. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "implementation": { + "completeness": 22, + "correctness": 24, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the design end to end: it removes the sole claimed `slices` use, trims each comma-delimited scope while preserving exact equality, and adds helper and command-level tests for both resulting 404 branches. GitHub's v2.98.0 release independently lists `Trim spaces when parsing X-Oauth-Scopes in gh release create`, strongly corroborating the chosen production fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) The largest limitation is test completeness: the helper cases do not fully cover every first/middle/last and spaced/unspaced combination required by the issue, although the loop itself handles them. The claimed `git apply --check` result and exact working-tree applicability could not be rerun because the local sandbox wrapper failed before command execution; no contradictory context was found in the submitted baseline diff." + } + }, + "task_score": 86.8, + "verdict": "This is a strong, repository-specific submission with a correct minimal implementation, limited chiefly by a flawed manual E2E setup, incomplete test-environment cleanup and lint coverage, and the absence of an independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:37:15.131189Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 219653, + "cached_input_tokens": 130748, + "cache_write_input_tokens": 28584, + "output_tokens": 14460, + "reasoning_output_tokens": 12317 + }, + "duration_ms": 228360 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json new file mode 100644 index 00000000..42b2e524 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json @@ -0,0 +1,64 @@ +{ + "task": "release-verify-sha1-prefix", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The issue clearly traces the algorithm through lookup, output, and verifier policy, with testable acceptance criteria and useful non-goals. Its reproduction syntax for verify-asset incorrectly suggests an asset plus --tag, while the command accepts an optional positional tag followed by the file path. ([cli.github.com](https://cli.github.com/manual/gh_release_verify-asset)) The submitted diff context supports the cited hard-coded constructor locations, although a sandbox bootstrap failure prevented independent checkout inspection. With no independent task statement, the claimed SHA-256 repository rollout and server-side indexing behavior remain unverified." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The LLD is unusually concrete about files, interfaces, data flow, output changes, compatibility, annotated tags, and test seams. Its proposed helper and both call-site changes are internally consistent with the submitted baseline diff and preserve valid 40-hex behavior. The largest unresolved dependency is whether the attestation service actually indexes SHA-256 refs under the proposed key, which the design acknowledges but cannot establish from the supplied task context. It also validates with hex.DecodeString before checking length, allowing an oversized server-controlled hexadecimal value to cause an avoidable allocation and very large error output." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds real, actionable defects: inconsistent error text, case normalization ambiguity, malformed-ref compatibility, an unused import, and a verifier mock that cannot observe the artifact. It records concrete resolutions that are reflected in the revised LLD and patch rather than merely issuing generic approvals. Its largest miss is treating an unbounded ref only as a long-line concern, overlooking the allocation performed by hex.DecodeString before length rejection. The assertion about future attestation-service behavior remains unverifiable because no independent task or service contract was supplied." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 22, + "risk_awareness": 19, + "total": 78, + "notes": "The unit-test matrix has strong oracles for algorithm selection, request digest, output, verifier input, malformed refs, and SHA-1 compatibility. The proposed recording verifier and captured FetchParams.Digest directly test the two important downstream effects rather than merely asserting successful execution. The plan omits the repository-required go test ./... and make lint checks, substituting narrower package tests and go vet. Its default manual success target is cli/cli v2.92.0 even though that repository documents immutable releases beginning with v2.93.0, and the claimed control-character coverage is not actually supplied by the listed z/g non-hex cases. ([github.com](https://github.com/cli/cli/blob/trunk/README.md))" + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the complete proposed path: one shared helper, both command integrations, correct import cleanup, and tests for lookup, display, verifier input, compatibility, and early failure. The unified-diff contexts and referenced symbols are internally consistent with the baseline artifacts, but the environment's shell sandbox failed before execution, so git apply --check and independent source compilation could not be confirmed. The main implementation risk is calling hex.DecodeString before bounding the input length, which needlessly allocates for an oversized hexadecimal API value and then includes the full value in the error. Apart from that bounded-input concern, the implementation matches the revised LLD and its summary accurately describes the behavioral change." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, coherent design and implementation package whose largest material limitation is a testing plan with an unsuitable manual baseline and missing repository-mandated full test and lint commands.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:40:04.592294Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 191774, + "cached_input_tokens": 126214, + "cache_write_input_tokens": 34091, + "output_tokens": 11485, + "reasoning_output_tokens": 9119 + }, + "duration_ms": 169448 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json new file mode 100644 index 00000000..2437b445 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "release-verify-sha1-prefix", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "release", + "go", + "cli", + "attestation", + "sha256", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 70.1, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 64, + "output_tokens": 37335, + "cache_read_tokens": 2984149, + "cache_write_tokens": 105093, + "total_tokens": 3126641, + "total_cost_usd": 2.05617588, + "latency_seconds": 532.3, + "num_turns": 31, + "generation_tokens_per_sec": 70.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 64, + "output_tokens": 37335, + "latency_seconds": 532.3, + "num_turns": 31, + "total_cost_usd": 2.05617588, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2984149, + "cache_creation_tokens": 105093, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T18:14:05.602468Z", + "run_ended_at": "2026-10-14T18:22:57.929285Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 64, + "output_tokens": 37335, + "cache_read_tokens": 2984149, + "cache_write_tokens": 105093, + "total_tokens": 3126641, + "total_cost_usd": 2.05617588, + "latency_seconds": 532.3, + "num_turns": 31, + "generation_tokens_per_sec": 70.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "release-verify-sha1-prefix", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The issue clearly traces the algorithm through lookup, output, and verifier policy, with testable acceptance criteria and useful non-goals. Its reproduction syntax for verify-asset incorrectly suggests an asset plus --tag, while the command accepts an optional positional tag followed by the file path. ([cli.github.com](https://cli.github.com/manual/gh_release_verify-asset)) The submitted diff context supports the cited hard-coded constructor locations, although a sandbox bootstrap failure prevented independent checkout inspection. With no independent task statement, the claimed SHA-256 repository rollout and server-side indexing behavior remain unverified." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The LLD is unusually concrete about files, interfaces, data flow, output changes, compatibility, annotated tags, and test seams. Its proposed helper and both call-site changes are internally consistent with the submitted baseline diff and preserve valid 40-hex behavior. The largest unresolved dependency is whether the attestation service actually indexes SHA-256 refs under the proposed key, which the design acknowledges but cannot establish from the supplied task context. It also validates with hex.DecodeString before checking length, allowing an oversized server-controlled hexadecimal value to cause an avoidable allocation and very large error output." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds real, actionable defects: inconsistent error text, case normalization ambiguity, malformed-ref compatibility, an unused import, and a verifier mock that cannot observe the artifact. It records concrete resolutions that are reflected in the revised LLD and patch rather than merely issuing generic approvals. Its largest miss is treating an unbounded ref only as a long-line concern, overlooking the allocation performed by hex.DecodeString before length rejection. The assertion about future attestation-service behavior remains unverifiable because no independent task or service contract was supplied." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 22, + "risk_awareness": 19, + "total": 78, + "notes": "The unit-test matrix has strong oracles for algorithm selection, request digest, output, verifier input, malformed refs, and SHA-1 compatibility. The proposed recording verifier and captured FetchParams.Digest directly test the two important downstream effects rather than merely asserting successful execution. The plan omits the repository-required go test ./... and make lint checks, substituting narrower package tests and go vet. Its default manual success target is cli/cli v2.92.0 even though that repository documents immutable releases beginning with v2.93.0, and the claimed control-character coverage is not actually supplied by the listed z/g non-hex cases. ([github.com](https://github.com/cli/cli/blob/trunk/README.md))" + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the complete proposed path: one shared helper, both command integrations, correct import cleanup, and tests for lookup, display, verifier input, compatibility, and early failure. The unified-diff contexts and referenced symbols are internally consistent with the baseline artifacts, but the environment's shell sandbox failed before execution, so git apply --check and independent source compilation could not be confirmed. The main implementation risk is calling hex.DecodeString before bounding the input length, which needlessly allocates for an oversized hexadecimal API value and then includes the full value in the error. Apart from that bounded-input concern, the implementation matches the revised LLD and its summary accurately describes the behavioral change." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, coherent design and implementation package whose largest material limitation is a testing plan with an unsuitable manual baseline and missing repository-mandated full test and lint commands.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:40:04.592294Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 191774, + "cached_input_tokens": 126214, + "cache_write_input_tokens": 34091, + "output_tokens": 11485, + "reasoning_output_tokens": 9119 + }, + "duration_ms": 169448 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json new file mode 100644 index 00000000..9f95635c --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-clone-ghe-ssh-ca-url", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The issue crisply defines the authentication failure, fork-upstream impact, wiki behavior, fallbacks, exclusions, and testable acceptance criteria. Its core diagnosis matches the reported repository bug: clone reconstructs the tenant username instead of using API sshUrl. ([github.com](https://github.com/cli/cli/issues/14344)) The largest weakness is that it prescribes same-host validation without establishing that alternate SSH hosts are invalid for every supported enterprise configuration. The assertion that Repository.sshUrl exists on every supported GHES version was not independently verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 23, + "total": 91, + "notes": "The LLD names the real query, model, clone call sites, parent flow, URL parser, fallback, and concrete test structure; the Repository model does contain SSHURL and GitHubRepo uses a shared parent fragment. ([github.com](https://github.com/cli/cli/blob/trunk/api/queries_repo.go)) Its principal deficiency is the mandatory equality between the API hostname and SSH hostname, which can reject a legitimate server-provided alternate SSH endpoint even though the API URL is supposed to be authoritative. The fallback preserves prior behavior and the design explicitly covers leading-option injection, HTTPS compatibility, wiki transformation, and unfixed callers. Exact availability of sshUrl across all historical GHES versions and the claimed local test restriction could not be verified." + }, + "review": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 23, + "total": 88, + "notes": "The review genuinely challenges the design and produces actionable changes for option-like URLs, off-host values, fallback testing, helper signatures, and the repo-fork scope distinction. Its strongest finding is that a leading-dash URL reaches positional git arguments without an option delimiter, with a concrete mitigation and regression test. Its weakest reasoning overstates SSH key or agent disclosure from merely connecting to another host and does not adequately examine legitimate separate SSH hostnames before requiring same-host validation. The review otherwise tracks the submitted source flow and resolves each prioritized finding rather than merely approving the design." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The plan supplies concrete mocked responses and exact git-command oracles for the tenant URL, wiki, fallback, HTTPS, unsafe values, and parent upstream behavior. It also includes compatibility, manual GHE.com, UX, rollback, and end-to-end checks tied to the acceptance criteria. The largest gap is incomplete unit coverage for explicitly promised ssh:// and git+ssh:// arguments, malformed values, and a same-host non-SSH API URL; some piped manual trace commands also do not independently preserve clone failure status. Claims requiring access to an SSH-CA tenant and the exact make target could not be executed in the provided environment." + }, + "implementation": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The patch implements the central flow end to end: it selects sshUrl in the shared repository fragment, routes both clone and parent upstream URLs through one helper, preserves FormatRemoteURL fallback, and adds command-level regression tests. The target symbols and surrounding architecture are consistent with the repository evidence: Repository already exposes SSHURL and Parent hostname initialization, while FormatRemoteURL constructs tenant-based SSH URLs. ([github.com](https://github.com/cli/cli/blob/trunk/api/queries_repo.go), [github.com](https://github.com/cli/cli/blob/trunk/internal/ghrepo/repo.go)) The largest design risk carried into code is rejecting all alternate-host SSH URLs, and tests omit the promised ssh://, git+ssh://, malformed, and same-host wrong-scheme cases. The submitted git apply --check claim and compilation could not be independently rerun because the read-only shell failed before repository commands executed." + } + }, + "task_score": 89.6, + "verdict": "Strong, unusually concrete work that appears to implement the reported fix, with the main limitation being an insufficiently justified same-host restriction and several missing URL-form test cases.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:42:40.949679Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 202499, + "cached_input_tokens": 120176, + "cache_write_input_tokens": 32684, + "output_tokens": 9984, + "reasoning_output_tokens": 7848 + }, + "duration_ms": 156344 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json new file mode 100644 index 00000000..bf8eb436 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json @@ -0,0 +1,157 @@ +{ + "task": "repo-clone-ghe-ssh-ca-url", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "medium", + "tags": [ + "repo", + "go", + "cli", + "git", + "ssh", + "ghe", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 92.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 100, + "output_tokens": 40192, + "cache_read_tokens": 4501496, + "cache_write_tokens": 104530, + "total_tokens": 4646318, + "total_cost_usd": 2.4499081200000012, + "latency_seconds": 433.3, + "num_turns": 49, + "generation_tokens_per_sec": 92.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 100, + "output_tokens": 40192, + "latency_seconds": 433.3, + "num_turns": 49, + "total_cost_usd": 2.4499081200000012, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4501496, + "cache_creation_tokens": 104530, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:47:12.766461Z", + "run_ended_at": "2026-10-14T17:54:26.123709Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 100, + "output_tokens": 40192, + "cache_read_tokens": 4501496, + "cache_write_tokens": 104530, + "total_tokens": 4646318, + "total_cost_usd": 2.4499081200000012, + "latency_seconds": 433.3, + "num_turns": 49, + "generation_tokens_per_sec": 92.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-clone-ghe-ssh-ca-url", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The issue crisply defines the authentication failure, fork-upstream impact, wiki behavior, fallbacks, exclusions, and testable acceptance criteria. Its core diagnosis matches the reported repository bug: clone reconstructs the tenant username instead of using API sshUrl. ([github.com](https://github.com/cli/cli/issues/14344)) The largest weakness is that it prescribes same-host validation without establishing that alternate SSH hosts are invalid for every supported enterprise configuration. The assertion that Repository.sshUrl exists on every supported GHES version was not independently verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 23, + "total": 91, + "notes": "The LLD names the real query, model, clone call sites, parent flow, URL parser, fallback, and concrete test structure; the Repository model does contain SSHURL and GitHubRepo uses a shared parent fragment. ([github.com](https://github.com/cli/cli/blob/trunk/api/queries_repo.go)) Its principal deficiency is the mandatory equality between the API hostname and SSH hostname, which can reject a legitimate server-provided alternate SSH endpoint even though the API URL is supposed to be authoritative. The fallback preserves prior behavior and the design explicitly covers leading-option injection, HTTPS compatibility, wiki transformation, and unfixed callers. Exact availability of sshUrl across all historical GHES versions and the claimed local test restriction could not be verified." + }, + "review": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 23, + "total": 88, + "notes": "The review genuinely challenges the design and produces actionable changes for option-like URLs, off-host values, fallback testing, helper signatures, and the repo-fork scope distinction. Its strongest finding is that a leading-dash URL reaches positional git arguments without an option delimiter, with a concrete mitigation and regression test. Its weakest reasoning overstates SSH key or agent disclosure from merely connecting to another host and does not adequately examine legitimate separate SSH hostnames before requiring same-host validation. The review otherwise tracks the submitted source flow and resolves each prioritized finding rather than merely approving the design." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The plan supplies concrete mocked responses and exact git-command oracles for the tenant URL, wiki, fallback, HTTPS, unsafe values, and parent upstream behavior. It also includes compatibility, manual GHE.com, UX, rollback, and end-to-end checks tied to the acceptance criteria. The largest gap is incomplete unit coverage for explicitly promised ssh:// and git+ssh:// arguments, malformed values, and a same-host non-SSH API URL; some piped manual trace commands also do not independently preserve clone failure status. Claims requiring access to an SSH-CA tenant and the exact make target could not be executed in the provided environment." + }, + "implementation": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The patch implements the central flow end to end: it selects sshUrl in the shared repository fragment, routes both clone and parent upstream URLs through one helper, preserves FormatRemoteURL fallback, and adds command-level regression tests. The target symbols and surrounding architecture are consistent with the repository evidence: Repository already exposes SSHURL and Parent hostname initialization, while FormatRemoteURL constructs tenant-based SSH URLs. ([github.com](https://github.com/cli/cli/blob/trunk/api/queries_repo.go), [github.com](https://github.com/cli/cli/blob/trunk/internal/ghrepo/repo.go)) The largest design risk carried into code is rejecting all alternate-host SSH URLs, and tests omit the promised ssh://, git+ssh://, malformed, and same-host wrong-scheme cases. The submitted git apply --check claim and compilation could not be independently rerun because the read-only shell failed before repository commands executed." + } + }, + "task_score": 89.6, + "verdict": "Strong, unusually concrete work that appears to implement the reported fix, with the main limitation being an insufficiently justified same-host restriction and several missing URL-form test cases.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:42:40.949679Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 202499, + "cached_input_tokens": 120176, + "cache_write_input_tokens": 32684, + "output_tokens": 9984, + "reasoning_output_tokens": 7848 + }, + "duration_ms": 156344 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json new file mode 100644 index 00000000..a63fa7af --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-create-reject-empty-name", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible prompt sequence, explicit error behavior, testable prompt and API-call acceptance criteria, and useful non-goals. Its named integration point, `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`, agrees with the submitted patch context. The main gap is that acceptance criteria explicitly cover only the from-scratch flow even though the description claims template and push-local coverage. No independent task statement establishes whether whitespace-only input or every interactive flow was required." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to an exact validation location, error, data flow, affected callers, and concrete tests in the repository's existing command-test structure. It correctly reasons that validation after `splitNameAndOwner` covers both a wholly blank answer and `OWNER/ ` while remaining before `userAndOrgs`. Its weaker points are reliance on inferred prompter-default and dependency-parser behavior, plus no automated push-local case despite claiming all three flows are protected. The absent independent task statement prevents confirming that the expanded whitespace behavior is part of the original requirement." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds a concrete flaw in its draft design: validating the raw answer misses a whitespace-only repository component after an explicit owner. It gives a precise corrective action, moves validation after `splitNameAndOwner`, and adds a corresponding test case rather than merely approving the design. Several role sections add little scrutiny, and the review does not challenge the missing push-local test or repository-wide verification commands. Claims about accessible-prompter behavior and CI coverage are asserted without directly presented evidence." + }, + "testing": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The plan defines exact inputs and error oracles, and its prompt stubs plus empty HTTP registry would expose later prompts or network requests by changing the observed error. It covers empty, whitespace-only, explicit-owner whitespace, template, valid-name, non-interactive, local-default, and accessible-prompter behavior across automated and manual levels. The main deficiency is omitting the repository-required `go test ./...` and `make lint` checks, while valid-name and push-local compatibility remain manual rather than automated. Some manual cleanup requires additional token scope and the plan does not independently prove zero requests outside the unit-test harness." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The three-line production change implements the revised design at the correct boundary, validating the parsed repository name before owner selection and API work, and the four table rows exercise the important invalid forms and template reuse. The diff targets existing symbols and plausible surrounding contexts, uses already-present `errors` and `strings` imports, and the stated baseline commit matches the official v2.100.0 release; v2.101.0 also lists interactive repository-name validation as PR #14313. ([github.com](https://github.com/cli/cli/releases/tag/v2.100.0), [github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) The largest gap is no automated push-local case despite the summary claiming all three interactive flows, though the shared helper makes the production behavior consistent. Local command execution failed before launch in the harness, so `git apply --check`, compilation, and test success could not be independently rerun." + } + }, + "task_score": 88.0, + "verdict": "This is a strong, repository-specific submission with a small and coherent implementation, limited mainly by incomplete repository-wide verification and untested push-local coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:46:05.901883Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 222401, + "cached_input_tokens": 122429, + "cache_write_input_tokens": 28485, + "output_tokens": 11964, + "reasoning_output_tokens": 10097 + }, + "duration_ms": 204940 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json new file mode 100644 index 00000000..43e7cd59 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "repo-create-reject-empty-name", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "trivial", + "tags": [ + "repo", + "go", + "cli", + "interactive-prompts", + "validation", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 73.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 52, + "output_tokens": 21497, + "cache_read_tokens": 1564148, + "cache_write_tokens": 86868, + "total_tokens": 1672565, + "total_cost_usd": 1.29504936, + "latency_seconds": 292.2, + "num_turns": 24, + "generation_tokens_per_sec": 73.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 52, + "output_tokens": 21497, + "latency_seconds": 292.2, + "num_turns": 24, + "total_cost_usd": 1.29504936, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1564148, + "cache_creation_tokens": 86868, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T16:54:02.066411Z", + "run_ended_at": "2026-10-14T16:58:54.322724Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 52, + "output_tokens": 21497, + "cache_read_tokens": 1564148, + "cache_write_tokens": 86868, + "total_tokens": 1672565, + "total_cost_usd": 1.29504936, + "latency_seconds": 292.2, + "num_turns": 24, + "generation_tokens_per_sec": 73.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-create-reject-empty-name", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible prompt sequence, explicit error behavior, testable prompt and API-call acceptance criteria, and useful non-goals. Its named integration point, `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`, agrees with the submitted patch context. The main gap is that acceptance criteria explicitly cover only the from-scratch flow even though the description claims template and push-local coverage. No independent task statement establishes whether whitespace-only input or every interactive flow was required." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to an exact validation location, error, data flow, affected callers, and concrete tests in the repository's existing command-test structure. It correctly reasons that validation after `splitNameAndOwner` covers both a wholly blank answer and `OWNER/ ` while remaining before `userAndOrgs`. Its weaker points are reliance on inferred prompter-default and dependency-parser behavior, plus no automated push-local case despite claiming all three flows are protected. The absent independent task statement prevents confirming that the expanded whitespace behavior is part of the original requirement." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds a concrete flaw in its draft design: validating the raw answer misses a whitespace-only repository component after an explicit owner. It gives a precise corrective action, moves validation after `splitNameAndOwner`, and adds a corresponding test case rather than merely approving the design. Several role sections add little scrutiny, and the review does not challenge the missing push-local test or repository-wide verification commands. Claims about accessible-prompter behavior and CI coverage are asserted without directly presented evidence." + }, + "testing": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The plan defines exact inputs and error oracles, and its prompt stubs plus empty HTTP registry would expose later prompts or network requests by changing the observed error. It covers empty, whitespace-only, explicit-owner whitespace, template, valid-name, non-interactive, local-default, and accessible-prompter behavior across automated and manual levels. The main deficiency is omitting the repository-required `go test ./...` and `make lint` checks, while valid-name and push-local compatibility remain manual rather than automated. Some manual cleanup requires additional token scope and the plan does not independently prove zero requests outside the unit-test harness." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The three-line production change implements the revised design at the correct boundary, validating the parsed repository name before owner selection and API work, and the four table rows exercise the important invalid forms and template reuse. The diff targets existing symbols and plausible surrounding contexts, uses already-present `errors` and `strings` imports, and the stated baseline commit matches the official v2.100.0 release; v2.101.0 also lists interactive repository-name validation as PR #14313. ([github.com](https://github.com/cli/cli/releases/tag/v2.100.0), [github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) The largest gap is no automated push-local case despite the summary claiming all three interactive flows, though the shared helper makes the production behavior consistent. Local command execution failed before launch in the harness, so `git apply --check`, compilation, and test success could not be independently rerun." + } + }, + "task_score": 88.0, + "verdict": "This is a strong, repository-specific submission with a small and coherent implementation, limited mainly by incomplete repository-wide verification and untested push-local coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:46:05.901883Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 222401, + "cached_input_tokens": 122429, + "cache_write_input_tokens": 28485, + "output_tokens": 11964, + "reasoning_output_tokens": 10097 + }, + "duration_ms": 204940 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json new file mode 100644 index 00000000..f95ae878 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-set-default-unset-without-repo", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The issue precisely distinguishes unset, view, explicit-repository, and interactive behavior, with concrete command-level acceptance criteria and compatibility-focused non-goals. The named command and flags are real, and the repository history records the corresponding fix as \"Allow unset without repository argument.\" ([github.com](https://github.com/cli/cli/commits), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default)) The largest gap is that \"existing interactive behaviour is unchanged\" lacks a specific acceptance oracle. No independent task statement was supplied, and the issue's asserted labels could not be verified." + }, + "lld": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The LLD is unusually well grounded for this small change: it names `NewCmdSetDefault`, `SetDefaultOptions`, `CanPrompt`, `ResolvedRemote`, both target files, and the exact guard modification. Its test design correctly notices that the existing option assertions and forced-TTY setup would otherwise permit false confidence. The main omission is a validation rollout matching repository guidance for `go fix`, `go test ./...`, and `make lint`, rather than only the affected package. Local `git apply --check` could not be rerun because the provided command sandbox failed during initialization, so that submitted claim remains independently unverified." + }, + "review": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The strongest review findings are concrete and consequential: asserting `UnsetMode`, preserving interactive defaults through a `nonTTY` field, and covering an explicit repository without a terminal. Those findings directly correspond to the submitted diff's existing assertion and TTY context, and each is resolved in the revised design. The weaker reviewer sections mostly approve without pressure-testing the testing plan's final cleared-state oracle or repository-wide validation requirements. The no-network security claim is plausible for the described unset branch but could not be independently traced through local source because repository command access was unavailable." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The plan provides meaningful state-based oracles for config removal, exit status, output routing, view behavior, explicit repositories, ignored arguments, and operation outside a Git repository. Its largest defect is the section 5 grep for `No default remote repository has been set` after clearing the default: the underlying `ResolvedRemote()` path reports `no resolved remote found`, making that end-to-end oracle inconsistent with the repository behavior. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/no-resolved-remote-found/), [github.com](https://github.com/malor/gh-cli/blob/trunk/context/remote.go)) It also omits the repository-prescribed `go fix`, full `go test ./...`, and lint gates. The manual checks were not executed, so their exact terminal-dependent output remains unverified." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The patch implements the design end to end: it adds `!opts.UnsetMode` to the existing guard and adds focused command-layer coverage while preserving all existing zero-value test behavior. The target paths, function names, error string, flags, and intended fix align with the repository's documented command and the upstream commit titled `Allow unset without repository argument`. ([github.com](https://github.com/cli/cli/commits), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default)) No material production-code defect or design deviation is apparent; the main limited gap is that unit cases make all streams non-TTY rather than separately pinning stdin-only redirection or prompt-disabled configuration. The summary honestly reports that tests were not run, while its `git apply --check` claim could not be independently repeated because local command execution was unavailable." + } + }, + "task_score": 92.8, + "verdict": "This is an excellent, repository-specific submission with a correct minimal implementation; its largest material limitation is one incorrect cleared-state test oracle and incomplete repository-wide validation coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:49:12.596011Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 181389, + "cached_input_tokens": 93529, + "cache_write_input_tokens": 25433, + "output_tokens": 10982, + "reasoning_output_tokens": 9168 + }, + "duration_ms": 186681 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json new file mode 100644 index 00000000..d5e26298 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "repo-set-default-unset-without-repo", + "repo": "https://github.com/cli/cli", + "ref": "v2.101.0", + "complexity": "trivial", + "tags": [ + "repo", + "go", + "cli", + "tty", + "flags", + "fixed-in-2.102.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 71.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 36, + "output_tokens": 17452, + "cache_read_tokens": 1037986, + "cache_write_tokens": 53282, + "total_tokens": 1108756, + "total_cost_usd": 0.9055103200000001, + "latency_seconds": 243.8, + "num_turns": 17, + "generation_tokens_per_sec": 71.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 36, + "output_tokens": 17452, + "latency_seconds": 243.8, + "num_turns": 17, + "total_cost_usd": 0.9055103200000001, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1037986, + "cache_creation_tokens": 53282, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T17:03:29.815292Z", + "run_ended_at": "2026-10-14T17:07:33.573676Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 36, + "output_tokens": 17452, + "cache_read_tokens": 1037986, + "cache_write_tokens": 53282, + "total_tokens": 1108756, + "total_cost_usd": 0.9055103200000001, + "latency_seconds": 243.8, + "num_turns": 17, + "generation_tokens_per_sec": 71.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-set-default-unset-without-repo", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The issue precisely distinguishes unset, view, explicit-repository, and interactive behavior, with concrete command-level acceptance criteria and compatibility-focused non-goals. The named command and flags are real, and the repository history records the corresponding fix as \"Allow unset without repository argument.\" ([github.com](https://github.com/cli/cli/commits), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default)) The largest gap is that \"existing interactive behaviour is unchanged\" lacks a specific acceptance oracle. No independent task statement was supplied, and the issue's asserted labels could not be verified." + }, + "lld": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The LLD is unusually well grounded for this small change: it names `NewCmdSetDefault`, `SetDefaultOptions`, `CanPrompt`, `ResolvedRemote`, both target files, and the exact guard modification. Its test design correctly notices that the existing option assertions and forced-TTY setup would otherwise permit false confidence. The main omission is a validation rollout matching repository guidance for `go fix`, `go test ./...`, and `make lint`, rather than only the affected package. Local `git apply --check` could not be rerun because the provided command sandbox failed during initialization, so that submitted claim remains independently unverified." + }, + "review": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The strongest review findings are concrete and consequential: asserting `UnsetMode`, preserving interactive defaults through a `nonTTY` field, and covering an explicit repository without a terminal. Those findings directly correspond to the submitted diff's existing assertion and TTY context, and each is resolved in the revised design. The weaker reviewer sections mostly approve without pressure-testing the testing plan's final cleared-state oracle or repository-wide validation requirements. The no-network security claim is plausible for the described unset branch but could not be independently traced through local source because repository command access was unavailable." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The plan provides meaningful state-based oracles for config removal, exit status, output routing, view behavior, explicit repositories, ignored arguments, and operation outside a Git repository. Its largest defect is the section 5 grep for `No default remote repository has been set` after clearing the default: the underlying `ResolvedRemote()` path reports `no resolved remote found`, making that end-to-end oracle inconsistent with the repository behavior. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/no-resolved-remote-found/), [github.com](https://github.com/malor/gh-cli/blob/trunk/context/remote.go)) It also omits the repository-prescribed `go fix`, full `go test ./...`, and lint gates. The manual checks were not executed, so their exact terminal-dependent output remains unverified." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The patch implements the design end to end: it adds `!opts.UnsetMode` to the existing guard and adds focused command-layer coverage while preserving all existing zero-value test behavior. The target paths, function names, error string, flags, and intended fix align with the repository's documented command and the upstream commit titled `Allow unset without repository argument`. ([github.com](https://github.com/cli/cli/commits), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default)) No material production-code defect or design deviation is apparent; the main limited gap is that unit cases make all streams non-TTY rather than separately pinning stdin-only redirection or prompt-disabled configuration. The summary honestly reports that tests were not run, while its `git apply --check` claim could not be independently repeated because local command execution was unavailable." + } + }, + "task_score": 92.8, + "verdict": "This is an excellent, repository-specific submission with a correct minimal implementation; its largest material limitation is one incorrect cleared-state test oracle and incomplete repository-wide validation coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:49:12.596011Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 181389, + "cached_input_tokens": 93529, + "cache_write_input_tokens": 25433, + "output_tokens": 10982, + "reasoning_output_tokens": 9168 + }, + "duration_ms": 186681 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json new file mode 100644 index 00000000..11894e5f --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-sync-checked-out-worktree", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 93, + "notes": "The issue gives a reproducible integrity failure, explicit non-goals, and acceptance criteria covering force, detached HEAD, missing branches, and unchanged behavior. GitHub CLI 2.99.0 independently lists fixes for linked-worktree corruption and its error explanation, supporting the core problem claim. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Its largest weakness is that no independent task statement exists, so scope is largely self-defined, and the asserted minimum supported Git version was not repository-verified. The proposed preflight solution also leaves atomicity and path-normalization decisions for the design." + }, + "lld": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 82, + "notes": "The LLD is concrete about `pkg/cmd/repo/sync/{sync.go,git.go,mocks.go}`, `executeLocalRepoSync`, call ordering, orphan branches, stale worktrees, and user-visible errors. Its strongest decision is placing the guard before both `UpdateBranch` and `CreateBranch`, while explicitly preserving current-branch behavior. The main defect is a time-of-check/time-of-use race between `worktree list` and `update-ref`, so the stated goal of never corrupting another worktree is not guaranteed. It also promises an absolute path without handling Git's supported relative-worktree configuration and understates an O(number of worktrees) scan as effectively constant time. ([git-scm.com](https://git-scm.com/docs/git-worktree), [git-scm.com](https://git-scm.com/docs/git-config))" + }, + "review": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The review identifies substantive defects in its earlier design, especially the unguarded unborn-branch creation path, stale-worktree recovery, and the need for real Git tests. Recommendations are prioritized, actionable, and tied to concrete symbols and behaviors rather than generic reviewer personas. Its largest miss is accepting the non-atomic list-then-update design without discussing concurrent worktree creation, and it does not challenge the absolute-path assumption under relative-worktree configuration. The detached rebase and bisect limitation is honestly documented, although the claimed Git support floor could not be verified." + }, + "testing": { + "completeness": 20, + "correctness": 16, + "specificity": 23, + "risk_awareness": 18, + "total": 77, + "notes": "The plan has unusually concrete setup, actions, and oracles, including unchanged refs, clean porcelain status, force behavior, error propagation, and positive controls. The repository workflow does use Linux, Windows, and macOS and runs the Go test suite, so the cross-platform intent is grounded. ([github.com](https://github.com/cli/cli/blob/d4e33858efe4b8b96dafc13fa2e806ec857886c0/.github/workflows/go.yml)) The most important missing real-Git case is the unborn branch that motivated guarding `CreateBranch`; mocks cannot validate its porcelain representation. The stale manual test incorrectly proposes deleting the main worktree containing the shared Git directory, the missing-branch manual test never creates the remote branch, and the environment setup does not fully isolate global Git configuration." + }, + "implementation": { + "completeness": 20, + "correctness": 13, + "specificity": 21, + "risk_awareness": 14, + "total": 68, + "notes": "The patch implements the described common-case guard before both ref-writing paths and adds focused unit and real-repository tests; the stated baseline commit matches the v2.98.0 release. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) A submitted real-Git assertion uses a raw string containing `\\\"main\\\"`, but the `%q` error contains ordinary quote characters, so that test necessarily fails. Production behavior also remains vulnerable to the list-then-update race and may report a relative rather than absolute worktree path when relative worktree links are configured. Exact patch applicability could not be independently checked because the provided local shell sandbox failed before execution, making the summary's `git apply --check` claim unverifiable." + } + }, + "task_score": 81.4, + "verdict": "The submission is a strong, repository-specific design package, but the implementation is not merge-ready because one test is definitively incorrect and the preflight guard is non-atomic.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:52:44.443550Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 179367, + "cached_input_tokens": 93878, + "cache_write_input_tokens": 31847, + "output_tokens": 12114, + "reasoning_output_tokens": 10208 + }, + "duration_ms": 211835 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json new file mode 100644 index 00000000..18dfc828 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "repo-sync-checked-out-worktree", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "high", + "tags": [ + "repo", + "go", + "git", + "worktree", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 67.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 66, + "output_tokens": 40125, + "cache_read_tokens": 2835341, + "cache_write_tokens": 93336, + "total_tokens": 2968868, + "total_cost_usd": 2.0201634200000003, + "latency_seconds": 593.7, + "num_turns": 32, + "generation_tokens_per_sec": 67.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 66, + "output_tokens": 40125, + "latency_seconds": 593.7, + "num_turns": 32, + "total_cost_usd": 2.0201634200000003, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2835341, + "cache_creation_tokens": 93336, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T18:44:18.650174Z", + "run_ended_at": "2026-10-14T18:54:12.376098Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 66, + "output_tokens": 40125, + "cache_read_tokens": 2835341, + "cache_write_tokens": 93336, + "total_tokens": 2968868, + "total_cost_usd": 2.0201634200000003, + "latency_seconds": 593.7, + "num_turns": 32, + "generation_tokens_per_sec": 67.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-sync-checked-out-worktree", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 93, + "notes": "The issue gives a reproducible integrity failure, explicit non-goals, and acceptance criteria covering force, detached HEAD, missing branches, and unchanged behavior. GitHub CLI 2.99.0 independently lists fixes for linked-worktree corruption and its error explanation, supporting the core problem claim. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Its largest weakness is that no independent task statement exists, so scope is largely self-defined, and the asserted minimum supported Git version was not repository-verified. The proposed preflight solution also leaves atomicity and path-normalization decisions for the design." + }, + "lld": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 82, + "notes": "The LLD is concrete about `pkg/cmd/repo/sync/{sync.go,git.go,mocks.go}`, `executeLocalRepoSync`, call ordering, orphan branches, stale worktrees, and user-visible errors. Its strongest decision is placing the guard before both `UpdateBranch` and `CreateBranch`, while explicitly preserving current-branch behavior. The main defect is a time-of-check/time-of-use race between `worktree list` and `update-ref`, so the stated goal of never corrupting another worktree is not guaranteed. It also promises an absolute path without handling Git's supported relative-worktree configuration and understates an O(number of worktrees) scan as effectively constant time. ([git-scm.com](https://git-scm.com/docs/git-worktree), [git-scm.com](https://git-scm.com/docs/git-config))" + }, + "review": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The review identifies substantive defects in its earlier design, especially the unguarded unborn-branch creation path, stale-worktree recovery, and the need for real Git tests. Recommendations are prioritized, actionable, and tied to concrete symbols and behaviors rather than generic reviewer personas. Its largest miss is accepting the non-atomic list-then-update design without discussing concurrent worktree creation, and it does not challenge the absolute-path assumption under relative-worktree configuration. The detached rebase and bisect limitation is honestly documented, although the claimed Git support floor could not be verified." + }, + "testing": { + "completeness": 20, + "correctness": 16, + "specificity": 23, + "risk_awareness": 18, + "total": 77, + "notes": "The plan has unusually concrete setup, actions, and oracles, including unchanged refs, clean porcelain status, force behavior, error propagation, and positive controls. The repository workflow does use Linux, Windows, and macOS and runs the Go test suite, so the cross-platform intent is grounded. ([github.com](https://github.com/cli/cli/blob/d4e33858efe4b8b96dafc13fa2e806ec857886c0/.github/workflows/go.yml)) The most important missing real-Git case is the unborn branch that motivated guarding `CreateBranch`; mocks cannot validate its porcelain representation. The stale manual test incorrectly proposes deleting the main worktree containing the shared Git directory, the missing-branch manual test never creates the remote branch, and the environment setup does not fully isolate global Git configuration." + }, + "implementation": { + "completeness": 20, + "correctness": 13, + "specificity": 21, + "risk_awareness": 14, + "total": 68, + "notes": "The patch implements the described common-case guard before both ref-writing paths and adds focused unit and real-repository tests; the stated baseline commit matches the v2.98.0 release. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) A submitted real-Git assertion uses a raw string containing `\\\"main\\\"`, but the `%q` error contains ordinary quote characters, so that test necessarily fails. Production behavior also remains vulnerable to the list-then-update race and may report a relative rather than absolute worktree path when relative worktree links are configured. Exact patch applicability could not be independently checked because the provided local shell sandbox failed before execution, making the summary's `git apply --check` claim unverifiable." + } + }, + "task_score": 81.4, + "verdict": "The submission is a strong, repository-specific design package, but the implementation is not merge-ready because one test is definitively incorrect and the preflight guard is non-atomic.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:52:44.443550Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 179367, + "cached_input_tokens": 93878, + "cache_write_input_tokens": 31847, + "output_tokens": 12114, + "reasoning_output_tokens": 10208 + }, + "duration_ms": 211835 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/run-summary.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/run-summary.json new file mode 100644 index 00000000..33d5b5bc --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/run-summary.json @@ -0,0 +1,1437 @@ +{ + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "scope": "cli-cli-v1", + "provider": "bedrock", + "ref": null, + "refs": [ + "v2.100.0", + "v2.101.0", + "v2.92.0", + "v2.93.0", + "v2.94.0", + "v2.95.0", + "v2.96.0", + "v2.97.0", + "v2.98.0" + ], + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T19:57:58.576789Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 232108, + "cached_input_tokens": 148468, + "cache_write_input_tokens": 39210, + "output_tokens": 10216, + "reasoning_output_tokens": 8460 + }, + "duration_ms": 181417 + }, + "num_tasks": 21, + "num_scored": 21, + "num_failed": 0, + "failed_tasks": [], + "mean_task_score_excl_failed": 85.62, + "mean_cost_usd_excl_failed": 2.18, + "tasks": [ + { + "task": "repo-set-default-unset-without-repo", + "complexity": "trivial", + "ref": "v2.101.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 17, + "input_tokens": 36, + "output_tokens": 17452, + "total_tokens": 1108756, + "latency_seconds": 243.8, + "total_cost_usd": 0.9055103200000001, + "result_subtype": "success", + "task_score": 92.8, + "cache_read_tokens": 1037986, + "cache_write_tokens": 53282, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 71.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The issue precisely distinguishes unset, view, explicit-repository, and interactive behavior, with concrete command-level acceptance criteria and compatibility-focused non-goals. The named command and flags are real, and the repository history records the corresponding fix as \"Allow unset without repository argument.\" ([github.com](https://github.com/cli/cli/commits), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default)) The largest gap is that \"existing interactive behaviour is unchanged\" lacks a specific acceptance oracle. No independent task statement was supplied, and the issue's asserted labels could not be verified." + }, + "lld": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The LLD is unusually well grounded for this small change: it names `NewCmdSetDefault`, `SetDefaultOptions`, `CanPrompt`, `ResolvedRemote`, both target files, and the exact guard modification. Its test design correctly notices that the existing option assertions and forced-TTY setup would otherwise permit false confidence. The main omission is a validation rollout matching repository guidance for `go fix`, `go test ./...`, and `make lint`, rather than only the affected package. Local `git apply --check` could not be rerun because the provided command sandbox failed during initialization, so that submitted claim remains independently unverified." + }, + "review": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The strongest review findings are concrete and consequential: asserting `UnsetMode`, preserving interactive defaults through a `nonTTY` field, and covering an explicit repository without a terminal. Those findings directly correspond to the submitted diff's existing assertion and TTY context, and each is resolved in the revised design. The weaker reviewer sections mostly approve without pressure-testing the testing plan's final cleared-state oracle or repository-wide validation requirements. The no-network security claim is plausible for the described unset branch but could not be independently traced through local source because repository command access was unavailable." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The plan provides meaningful state-based oracles for config removal, exit status, output routing, view behavior, explicit repositories, ignored arguments, and operation outside a Git repository. Its largest defect is the section 5 grep for `No default remote repository has been set` after clearing the default: the underlying `ResolvedRemote()` path reports `no resolved remote found`, making that end-to-end oracle inconsistent with the repository behavior. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/no-resolved-remote-found/), [github.com](https://github.com/malor/gh-cli/blob/trunk/context/remote.go)) It also omits the repository-prescribed `go fix`, full `go test ./...`, and lint gates. The manual checks were not executed, so their exact terminal-dependent output remains unverified." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The patch implements the design end to end: it adds `!opts.UnsetMode` to the existing guard and adds focused command-layer coverage while preserving all existing zero-value test behavior. The target paths, function names, error string, flags, and intended fix align with the repository's documented command and the upstream commit titled `Allow unset without repository argument`. ([github.com](https://github.com/cli/cli/commits), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default)) No material production-code defect or design deviation is apparent; the main limited gap is that unit cases make all streams non-TTY rather than separately pinning stdin-only redirection or prompt-disabled configuration. The summary honestly reports that tests were not run, while its `git apply --check` claim could not be independently repeated because local command execution was unavailable." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "trivial", + "ref": "v2.95.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 18, + "input_tokens": 38, + "output_tokens": 19912, + "total_tokens": 1241141, + "latency_seconds": 290.3, + "total_cost_usd": 1.02499738, + "result_subtype": "success", + "task_score": 92.4, + "cache_read_tokens": 1160944, + "cache_write_tokens": 60247, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 68.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The issue precisely identifies the omitted counts.Canceled term, desired TTY behavior, compatibility constraints, and concrete regression oracle. Repository history confirms this root cause and the one-term fix in pkg/cmd/pr/checks/output.go. ([github.com](https://github.com/cli/cli/pull/13679)) Its main limitation is the absent independent task statement, so the requested wording and unchanged exit code are established only by the artifacts and upstream issue context. Explicit non-goals for non-TTY status, JSON, wording, and exit behavior make the scope unusually clear." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The LLD grounds the change in printSummary, checkCounts, aggregateChecks, checksRun, the table-driven tests, and the existing fixture convention. Its monotonicity argument correctly explains why adding Canceled admits only the previously omitted cancelled-only case while preserving headline precedence. The largest deficiency is minor bookkeeping: the approximately 76-line estimate does not match the submitted patch's 59 additions and one deletion, and the command-level fixture approach is heavier than a direct printSummary unit test. It nevertheless addresses TTY, non-TTY, watch-path reuse, zero-check behavior, exit compatibility, rollout, and future field-enumeration fragility." + }, + "review": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest finding is the concrete demand for a non-TTY regression case, including its exact tab-separated oracle, which the revised design and patch adopt. The review also correctly checks headline precedence, exit-code independence, table padding, watch-path reuse, scripted-output compatibility, and the absence of a new security surface. Its main weakness is repetition across reviewer personas and limited challenge to the relatively expensive fixture-based test strategy; it does not suggest the focused printSummary branch table later used upstream. No material repository claim beyond the missing independent task requirements remained unresolved." + }, + "testing": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The plan provides focused Go test commands, exact TTY and non-TTY output oracles, a negative regression check, existing-case coverage, manual GitHub verification, and exit-code checks. The subtest regexes and package paths agree with the submitted Test_checksRun additions, and the expected 1m26s duration follows from the fixture timestamps. Its largest correctness defect is the before/after TTY comparison redirecting both commands to files, which makes stdout non-TTY and therefore cannot compare TTY summaries despite claiming to do so. It also omits the repository-required make lint validation and uses a grep pipeline that does not itself preserve the go test exit status." + }, + "implementation": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The patch implements the complete behavioral fix by adding counts.Canceled to the existing printSummary guard, exactly matching the confirmed upstream correction shipped after v2.95.0. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) Its fixture and two Test_checksRun cases exercise both the newly fixed TTY output and unchanged non-TTY row and success behavior, with hunk contexts and symbols consistent with the submitted baseline. The largest evidence limitation is that the local sandbox prevented an independent git apply --check, while the implementation summary also honestly states that tests were not executed; absence of test execution is not scored as an implementation defect. Apart from lacking an all-cancelled watch-specific test, the change is minimal, convention-consistent, and faithful to the revised LLD." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 21, + "input_tokens": 44, + "output_tokens": 18749, + "total_tokens": 1391868, + "latency_seconds": 268.9, + "total_cost_usd": 0.9935056999999998, + "result_subtype": "success", + "task_score": 91.6, + "cache_read_tokens": 1320280, + "cache_write_tokens": 52795, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 69.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue's strongest aspect is its byte-exact acceptance criteria covering stderr, stdout, return value, and prohibited side effects. The submitted source context corroborates the identified fmt.Fprintf call and cmdutil.SilentError path. Its largest deficiency is declaring the sibling non-interactive warning out of scope despite the absence of an independent task statement establishing that boundary. The external issue reference and exact repository line numbers could not be independently verified through the unavailable local shell." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD is implementation-ready for this small change, naming runCopilot, CanPrompt, PrompterMock, findCopilotBinaryFunc, and runExternalCmdFunc with concrete data flow and test behavior. Its proposed code and integration points agree with the submitted patch context. The main weakness is overstating regression risk as nonexistent and treating the non-interactive branch exclusion as task-mandated when the independent task statement is missing. Claims about IOStreams internals, exact line numbers, and existing test seams were plausible but could not all be independently read from the local repository." + }, + "review": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 23, + "total": 91, + "notes": "The review's strongest work is identifying concrete test defects: environment-dependent binary lookup, accidental download or execution, and an assertion that would pass without the newline. The implementation visibly adopts its recommendations through exact stderr equality, fatal side-effect guards, and cleanup of package-level seams. Its largest deficiency is that several reviewer sections add little scrutiny, while the scope decision for the sibling warning is accepted without resolving the missing task evidence. The assertion that the survey prompt itself supplies the preceding newline is inferred from reported output rather than independently verified." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The plan has a strong regression oracle: exact stderr bytes, empty stdout, SilentError, prompt parameters, and explicit download and execution guards. The package paths and focused go test command agree with the submitted implementation, and the negative mutation check would prove the test detects the original defect. Its main gaps are omitting the repository-required make lint check and presenting POSIX commands as applicable to PowerShell without adaptation. The commands and manual PTY behavior were not executed, so their platform portability remains unverified." + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The patch implements the design end to end: it adds the newline at the declined-install write and adds a focused test covering output, return value, prompt routing, and prohibited side effects. Its imports and referenced symbols are internally consistent with the surrounding submitted diff, and the official v2.99.0 release independently lists this exact declined-install newline fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The largest residual risk is mutation of package-level test seams, although avoiding t.Parallel and restoring them with t.Cleanup appropriately mitigates it. The claimed git apply --check result and compilation could not be independently rerun because the local read-only shell failed before executing repository commands." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 28, + "input_tokens": 58, + "output_tokens": 29842, + "total_tokens": 2357892, + "latency_seconds": 449.9, + "total_cost_usd": 1.6219192, + "result_subtype": "success", + "task_score": 91.0, + "cache_read_tokens": 2242200, + "cache_write_tokens": 85792, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 66.3, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The issue precisely connects the casing mismatch to `NewMergeContext`, the `crossRepoPR` guard, and the missing branch-delete request, with reproducible steps and testable same-repository and fork criteria. The pinned v2.100.0 evidence confirms the case-sensitive comparison at `merge.go:592` and early return at `merge.go:519`. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its main deficiency is that the output acceptance criterion does not distinguish TTY from non-TTY behavior, although the baseline suppresses that informational line outside a TTY. No independent task statement was supplied, so intended scope beyond the task identifier and internally consistent artifacts could not be verified." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The LLD is implementation-ready: it names the exact field-list replacement, assignment, consumers, fixture migrations, API call, and interactive behavior. Repository evidence supports both the existing faulty query and the finder behavior that adds `isCrossRepository` and `headRepositoryOwner` during branch lookup. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that the proposed tests do not assert the `FindOptions.Fields` contract, so omission of the new GraphQL field would not be caught by the mocked-PR regressions. The broad GHES compatibility claim is plausible but lacks a supported-version boundary or direct pinned-schema verification." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review finds a concrete deficiency in the original design - missing interactive-prompt coverage - and identifies the two fork fixtures that must gain `IsCrossRepository: true`. Its fixture analysis agrees with the mock finder, which backfills only `HeadRepositoryOwner.Login` and leaves the boolean at its zero value. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) The largest missed concern is that neither proposed regression verifies that `NewMergeContext` actually requests `isCrossRepository`. Some GHES and deleted-head-repository assertions are presented confidently without pinned repository or compatibility evidence." + }, + "testing": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The plan supplies concrete commands, fixtures, HTTP expectations, prompt text, negative behavior, backward-compatibility scenarios, and end-to-end branch-state oracles. The key REST oracle matches the documented `--delete-branch` contract and the baseline failure path. ([cli.github.com](https://cli.github.com/manual/gh_pr_merge), [gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its largest gap is again the absence of a field-selection assertion, allowing tests to pass with a stubbed `IsCrossRepository` value even if the real GraphQL query stopped requesting it. The manual fork and GHES procedures require substantial external account and server setup, and the GHES expectation is not tied to a stated supported version." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the design end to end by swapping the requested field, assigning `crossRepoPR` from `pr.IsCrossRepository`, updating both fork fixtures, and adding flag-path and prompt regressions. The `merge.go` hunk matches the verified v2.100.0 symbols and line context, and the resulting delete path is consistent with the existing guard and REST behavior. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The largest weakness is that the tests inject the boolean through `StubFinderForRunCommandStyleTests` without asserting the requested field list, leaving the API wiring less protected than the decision logic. No behavioral deviation from the LLD is apparent, but local `git apply --check`, compilation, and formatting could not be independently executed because the provided shell sandbox failed during initialization." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 40, + "input_tokens": 84, + "output_tokens": 39404, + "total_tokens": 4332570, + "latency_seconds": 553.5, + "total_cost_usd": 2.46996684, + "result_subtype": "success", + "task_score": 90.8, + "cache_read_tokens": 4168417, + "cache_write_tokens": 124665, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 71.2, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The strongest aspect is the testable acceptance criteria covering every command, scope, endpoint, default, message, and help surface. The named shared and command packages match the repository architecture, and PR #13421 confirms that list and delete support were required alongside set. The largest deficiency is the absent independent task specification, which prevents fully verifying whether this expanded scope and exact wording were required. GHES behavior is sensibly documented, but support across particular server versions remains externally dependent." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The design is unusually well grounded in real integration points, including shared.GetSecretApp, shared.IsSupportedSecretEntity, putOrgSecret, and the application-interpolated request paths. It correctly commits to integer organization repository IDs and identifies that agents must avoid the Dependabot-specific payload branch. Its largest weakness is disproportionate detail around estimates and diagrams while rollout remains essentially release-and-observe with only passthrough 404 handling. Local patch application could not be independently executed, although the named baseline symbols and surrounding contexts align with repository and merged-change evidence." + }, + "review": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The strongest contribution is finding concrete defects in the initial test design, particularly hard-coded Actions list paths, missing selected-visibility payload coverage, and absent unsupported-entity checks. Those findings are actionable and visibly drive corresponding changes in list_test.go and set_test.go. The security discussion slightly overstates the consequence of moving validation below public-key retrieval because fetching a key alone does not transmit the secret value. GHES limitations and default organization visibility are honestly retained as known tradeoffs rather than falsely declared resolved." + }, + "testing": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 20, + "total": 88, + "notes": "The plan has strong behavioral oracles for endpoint selection, payload visibility, selected-repository counts, JSON output, defaults, case handling, completion, and independence from Actions secrets. Its commands and paths agree with the repository's Go test layout and documentation generator. The largest deficiency is poor cleanup and isolation for live tests: several repository and organization secrets and /tmp/agents.env are left behind, while the expected Actions 404 assumes no preexisting secret with that name. Execution-dependent claims about live permissions, GHES responses, and exact terminal output were not demonstrated." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the feature end to end by extending shared application parsing and entity support, all three command enums, help text, and repository and organization runtime tests while correctly leaving interpolating HTTP helpers unchanged. The diff contexts and symbols match the v2.92.0 baseline architecture, and the same core approach was accepted in PR #13421. The largest concrete defect is the new assert.EqualError usage despite the repository instruction to use require for error checks; additionally, an empty httpmock.Registry does not itself record or prove unmatched requests as the test comment implies. No functional routing defect is apparent, but the claimed apply check and test outcome could not be independently executed, and the implementation summary acknowledges that compilation and tests were not run." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "high", + "ref": "v2.94.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 40, + "input_tokens": 84, + "output_tokens": 45121, + "total_tokens": 4225888, + "latency_seconds": 653.4, + "total_cost_usd": 2.5379866600000005, + "result_subtype": "success", + "task_score": 90.2, + "cache_read_tokens": 4062273, + "cache_write_tokens": 118410, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 69.1, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest aspect is the reproducible nested-directory scenario plus acceptance criteria with exact positive and negative filesystem oracles. Its root-cause account matches the submitted update.go context, while the installer really targets filepath.Join(baseDir, skill.Name). ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The largest deficiency is scope expansion into download atomicity and remote-name mismatch without an independent task statement establishing those requirements. The related issue and the practical likelihood of an upstream name mismatch could not be independently verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 23, + "total": 91, + "notes": "The LLD is unusually well grounded in updateRun, installer.Install, installSkill, installedSkill.dir, discovery.Skill, safepaths, and the existing tests. Its parent-directory target, mismatch guard, cleanup removal, and fetch-before-write flow are implementation-ready and consistent with the installer's real target-selection behavior. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The main correctness blemish is its inconsistent explanation that Skill.Name is path-derived while the review also treats it as frontmatter-derived, plus an unsupported assumption that buffering every skill is always small. It explicitly acknowledges the remaining partial-write risk from local I/O failures, although the absent independent task statement limits verification of its expanded scope." + }, + "review": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 23, + "total": 90, + "notes": "The review's strongest contribution is finding three concrete design defects: base plus skill.Name can select a sibling, host-scanned skills also flatten, and file-by-file fetching can leave mixed versions. Each finding identifies a consequence and a specific revision, rather than merely approving the design. The largest flaw is overstating a remote publisher's ability to change a frontmatter-derived name, contradicting the LLD's own statement that discovery.Skill.Name comes from the path; the local-directory rename case remains valid. Repository evidence confirms the installer joins the selected base with skill.Name, but no independent task statement was available to validate every blocking priority. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/))" + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 88, + "notes": "The plan's strongest aspect is its concrete matrix across flat and two-level layouts, custom and agent scans, mismatch refusal, preserved local files, and second-blob failure, with meaningful path and content oracles. The proposed package paths and HTTP-focused test strategy align with the submitted repository structure, and the global lockfile path is supported by repository evidence. ([github.com](https://github.com/cli/cli/issues/13438), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-create-lock-directory-w/)) Its largest deficiency is omitting the repository-required make lint command, while shorthand such as 'install with --pin' and the non-recursive checksum glob make parts of the manual plan less directly executable. Tests were not executed, and the unavailable independent task statement prevents confirming whether all added scenarios are required." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the submitted design end to end: every update receives the current directory's parent, mismatched names fail per skill, destructive cleanup is removed, and downloads complete before filesystem writes begin. The hunks target existing installSkill and updateRun behavior, and the added tests exercise relocation, preservation, host scanning, mismatch refusal, and fetch failure; the underlying installer and lockfile call shapes match repository evidence. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The largest remaining risk is acknowledged but unresolved atomicity for mkdir or write failures, and buffering all files broadens installer memory usage without an evidenced size bound. The local shell sandbox could not start, so the candidate's git apply --check claim and test compilation could not be independently verified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "low", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 37, + "input_tokens": 76, + "output_tokens": 32418, + "total_tokens": 3237828, + "latency_seconds": 523.4, + "total_cost_usd": 1.8930221200000001, + "result_subtype": "success", + "task_score": 90.0, + "cache_read_tokens": 3115501, + "cache_write_tokens": 89833, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 61.9, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 18, + "total": 89, + "notes": "The issue provides a precise reproduction, scoped solution, negative and positive acceptance criteria, and selector coverage. Its diagnosis and remedy match the repository's merged fix for #13319 and the v2.93.0 release record. ([github.com](https://github.com/cli/cli/pull/13327), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The main deficiency is limited discussion of the added network request and behavior changes inherited by other shared-finder callers. No independent task statement was supplied, so requirements beyond the identifier and linked repository fix could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD is unusually well grounded, naming the actual finder, callers, field normalization, merge-ref path, query, and test structures. The repository source confirms that merge refs set `prNumber`, `id` and `number` are added before querying, and known numbers flow through `findByNumber` after the fix. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that loss of the former offline/no-request behavior is not analyzed explicitly, while the fixed one-point cost assertion is not established by repository evidence. Broader requirement coverage remains unverifiable because no independent task statement was provided." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The review finds concrete weaknesses in its own draft, including missing merge-ref coverage, lack of a user-path regression test, and failure to assert the GraphQL variable. Those concerns are technically relevant because the finder source contains the cited merge-ref path and unconditional numbered lookup. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its main correctness blemish is the claim that every caller subsequently talks to GitHub, which conflicts with its own description of JSON export and browser-only checks behavior. Some reviewer-role sections add little beyond approval, and the stated rate-limit figures were not independently verified." + }, + "testing": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The plan supplies concrete fixtures, actions, exact output and error oracles, selector variants, compatibility checks, and a mutation check that would expose restoration of the shortcut. Its build command is valid because the repository Makefile provides a `bin/gh` target. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile)) The largest omission is repository-wide verification: it specifies only `go test ./pkg/cmd/pr/...` and does not include the required `go test ./...` or `make lint`. Several live checks depend on authentication and mutable repository data, and their exact browser output was not independently verified." + }, + "implementation": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The production change removes exactly the synthetic-number shortcut identified by the task, matching the repository's merged fix, while the added tests cover success, GraphQL failure, URL selection, merge-ref selection, and JSON output. ([github.com](https://github.com/cli/cli/pull/13327), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The stated v2.92.0 baseline commit is consistent with the release record, and the patch context aligns with the documented pre-fix branch and current post-fix finder flow. ([github.com](https://github.com/cli/cli/releases/tag/v2.92.0), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) The largest coverage gap is the absence of an automated `#N` selector case, although that parser path is unchanged and the testing plan covers it manually. Shell sandbox failure prevented an independent `git apply --check` or compilation run, so the summary's apply claim and test compilation could not be directly verified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 49, + "input_tokens": 100, + "output_tokens": 40192, + "total_tokens": 4646318, + "latency_seconds": 433.3, + "total_cost_usd": 2.4499081200000012, + "result_subtype": "success", + "task_score": 89.6, + "cache_read_tokens": 4501496, + "cache_write_tokens": 104530, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 92.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 23, + "total": 94, + "notes": "The issue crisply defines the authentication failure, fork-upstream impact, wiki behavior, fallbacks, exclusions, and testable acceptance criteria. Its core diagnosis matches the reported repository bug: clone reconstructs the tenant username instead of using API sshUrl. ([github.com](https://github.com/cli/cli/issues/14344)) The largest weakness is that it prescribes same-host validation without establishing that alternate SSH hosts are invalid for every supported enterprise configuration. The assertion that Repository.sshUrl exists on every supported GHES version was not independently verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 23, + "total": 91, + "notes": "The LLD names the real query, model, clone call sites, parent flow, URL parser, fallback, and concrete test structure; the Repository model does contain SSHURL and GitHubRepo uses a shared parent fragment. ([github.com](https://github.com/cli/cli/blob/trunk/api/queries_repo.go)) Its principal deficiency is the mandatory equality between the API hostname and SSH hostname, which can reject a legitimate server-provided alternate SSH endpoint even though the API URL is supposed to be authoritative. The fallback preserves prior behavior and the design explicitly covers leading-option injection, HTTPS compatibility, wiki transformation, and unfixed callers. Exact availability of sshUrl across all historical GHES versions and the claimed local test restriction could not be verified." + }, + "review": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 23, + "total": 88, + "notes": "The review genuinely challenges the design and produces actionable changes for option-like URLs, off-host values, fallback testing, helper signatures, and the repo-fork scope distinction. Its strongest finding is that a leading-dash URL reaches positional git arguments without an option delimiter, with a concrete mitigation and regression test. Its weakest reasoning overstates SSH key or agent disclosure from merely connecting to another host and does not adequately examine legitimate separate SSH hostnames before requiring same-host validation. The review otherwise tracks the submitted source flow and resolves each prioritized finding rather than merely approving the design." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The plan supplies concrete mocked responses and exact git-command oracles for the tenant URL, wiki, fallback, HTTPS, unsafe values, and parent upstream behavior. It also includes compatibility, manual GHE.com, UX, rollback, and end-to-end checks tied to the acceptance criteria. The largest gap is incomplete unit coverage for explicitly promised ssh:// and git+ssh:// arguments, malformed values, and a same-host non-SSH API URL; some piped manual trace commands also do not independently preserve clone failure status. Claims requiring access to an SSH-CA tenant and the exact make target could not be executed in the provided environment." + }, + "implementation": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The patch implements the central flow end to end: it selects sshUrl in the shared repository fragment, routes both clone and parent upstream URLs through one helper, preserves FormatRemoteURL fallback, and adds command-level regression tests. The target symbols and surrounding architecture are consistent with the repository evidence: Repository already exposes SSHURL and Parent hostname initialization, while FormatRemoteURL constructs tenant-based SSH URLs. ([github.com](https://github.com/cli/cli/blob/trunk/api/queries_repo.go), [github.com](https://github.com/cli/cli/blob/trunk/internal/ghrepo/repo.go)) The largest design risk carried into code is rejecting all alternate-host SSH URLs, and tests omit the promised ssh://, git+ssh://, malformed, and same-host wrong-scheme cases. The submitted git apply --check claim and compilation could not be independently rerun because the read-only shell failed before repository commands executed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "extension-uninstall-alias", + "complexity": "trivial", + "ref": "v2.93.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 12, + "input_tokens": 26, + "output_tokens": 15109, + "total_tokens": 714488, + "latency_seconds": 211.7, + "total_cost_usd": 0.7345828600000002, + "result_subtype": "success", + "task_score": 88.8, + "cache_read_tokens": 652343, + "cache_write_tokens": 47010, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 71.4, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue crisply defines the alias behavior, compatibility guarantee, testable acceptance criteria, and narrow non-goals. Upstream PR #13599 confirms that adding Cobra's Aliases field is the intended solution and keeps remove canonical. ([github.com](https://github.com/cli/cli/pull/13599)) The current manual also confirms that uninstall is exposed through all three parent command forms. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The main evidence gap is the absent independent task statement, although the task identifier and upstream release entry substantiate the proposed scope." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD is unusually well grounded for a one-line change, naming the command literal, test table, help renderer, documentation generator, dispatch path, and backward-compatibility boundary. Repository evidence supports the root-level extension dispatch and the documented gen-docs flags. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/root/root.go), [github.com](https://github.com/cli/cli/blob/trunk/cmd/gen-docs/main.go)) Its largest factual defect is the asserted exact ALIASES ordering, which disagrees with both the upstream PR's manual check and the generated manual. ([github.com](https://github.com/cli/cli/pull/13599), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The telemetry behavior is plausible but was not directly verified against the v2.93.0 implementation." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 22, + "risk_awareness": 21, + "total": 87, + "notes": "The review identifies a concrete weakness in the original test proposal and converts it into an actionable missing-argument test that appears in the patch. Its collision analysis is supported by the root command's registration of installed extensions at the top level, outside gh extension. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/root/root.go)) The largest omission is that no reviewer challenges the absent direct help assertion or notices that the proposed exact alias ordering is wrong. The role breakdown adds some ceremony, but most conclusions remain specific and proportionate to this low-risk change." + }, + "testing": { + "completeness": 23, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 85, + "notes": "The plan covers alias dispatch, normalization, argument failure, backward compatibility, parent aliases, generated documentation, and a full install/remove lifecycle with concrete commands and oracles. The unit-test and documentation-generation commands agree with the repository's command structure and gen-docs flags. ([github.com](https://github.com/cli/cli/blob/trunk/cmd/gen-docs/main.go)) Its exact ALIASES oracle is incorrect: neither the upstream manual check nor the generated manual uses the stated ordering. ([github.com](https://github.com/cli/cli/pull/13599), [cli.github.com](https://cli.github.com/manual/gh_extension_remove)) It also expects usage output for the missing argument despite the root command silencing Cobra usage, and several live checks depend on mutable network state." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the requested behavior directly by adding Aliases to the existing remove command, exactly matching the mechanism merged upstream and listed in the v2.94.0 release. ([github.com](https://github.com/cli/cli/pull/13599), [github.com](https://github.com/cli/cli/releases/tag/v2.94.0)) The added table rows exercise real Cobra resolution, TTY output, full-name normalization, and inherited argument validation without changing removal logic. Its main limitation is the lack of an automated assertion for help output or parent aliases, although the declarative alias still implements those behaviors through shared infrastructure. Local git apply verification could not run because the provided sandbox failed to initialize, but the target symbols and source context align with the verified upstream change." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 48, + "input_tokens": 100, + "output_tokens": 44974, + "total_tokens": 4953827, + "latency_seconds": 717.1, + "total_cost_usd": 2.72300358, + "result_subtype": "success", + "task_score": 88.4, + "cache_read_tokens": 4785039, + "cache_write_tokens": 123714, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 62.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The issue provides exact messages, token classifications, hostname mappings, testable acceptance criteria, and explicit exclusions. Its core problem matches upstream issue #13066, while the merged PR also corroborates the status-output scope. ([github.com](https://github.com/cli/cli/pull/13068), [github.com](https://github.com/cli/cli/issues/13066)) The largest deficiency is overbroad wording: an explicit Authorization header can make the failing token differ from ActiveToken, and invalid environment-token status entries do not necessarily print any reauthentication line. No independent task statement was supplied, so requirements beyond the identifier and corroborated upstream issue remain an evidence gap." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest aspect is its repository-specific data flow through internal/ghcmd/cmd.go, ActiveToken, NormalizeHostname, shared.IsRefreshableToken, and status.buildEntry. The submitted patch context supports those symbols and the design explicitly handles nil configuration, nil RequestURL, environment tokens, PATs, inactive accounts, and JSON stability. Its main weakness is reliance on duplicated private source strings plus the acknowledged header-override mismatch, with no mitigation beyond preserving the old fallback behavior. The claim about scope behavior after a rejected token could not be independently verified; the public manual only states that a normal refresh without scope flags maintains prior scopes. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 89, + "notes": "The review identifies concrete defects rather than merely approving: missing keyring coverage, brittle source literals, and misleading advice after an Authorization-header override. Those findings directly correspond to the proposed predicate, status active-account gate, and request-host lookup in the patch. Its largest omission is failure to challenge the issue's overbroad status criterion or the testing plan's keyring-isolation assumption. The active-account limitation is independently consistent with the gh auth refresh manual. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The plan has precise oracles for stored OAuth tokens, PATs, environment tokens, hostname normalization, inactive accounts, nil inputs, SAML behavior, missing scopes, JSON output, and recovery. Its automated paths, test names, httpmock style, and package commands agree with the submitted patch and repository conventions. The largest deficiency is that GH_CONFIG_DIR does not isolate credentials placed in the system keyring, so the manual setup can affect or leave real keyring state; it also tests authHint directly without an integration assertion through Main and omits the prescribed make lint command. The live revocation and recovery scenarios were not executed, so their operational assumptions remain unverified." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements both output paths end to end, centralizes classification in IsRefreshableToken, preserves existing fallbacks, and adds focused positive and negative tests. Its file contexts and symbols are internally consistent with the baseline diff, and the official merged PR confirms the same feature shape and successful checks. ([github.com](https://github.com/cli/cli/pull/13068)) The largest residual risk is that the source-plus-prefix heuristic can describe a stored token rather than the token actually sent, while no test exercises the complete Main error-printing path. Local git apply and test execution could not be independently performed because the provided shell sandbox failed before command execution, leaving exact applicability as a harness evidence gap rather than a demonstrated candidate defect." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-create-reject-empty-name", + "complexity": "trivial", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 24, + "input_tokens": 52, + "output_tokens": 21497, + "total_tokens": 1672565, + "latency_seconds": 292.2, + "total_cost_usd": 1.29504936, + "result_subtype": "success", + "task_score": 88.0, + "cache_read_tokens": 1564148, + "cache_write_tokens": 86868, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 73.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible prompt sequence, explicit error behavior, testable prompt and API-call acceptance criteria, and useful non-goals. Its named integration point, `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`, agrees with the submitted patch context. The main gap is that acceptance criteria explicitly cover only the from-scratch flow even though the description claims template and push-local coverage. No independent task statement establishes whether whitespace-only input or every interactive flow was required." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to an exact validation location, error, data flow, affected callers, and concrete tests in the repository's existing command-test structure. It correctly reasons that validation after `splitNameAndOwner` covers both a wholly blank answer and `OWNER/ ` while remaining before `userAndOrgs`. Its weaker points are reliance on inferred prompter-default and dependency-parser behavior, plus no automated push-local case despite claiming all three flows are protected. The absent independent task statement prevents confirming that the expanded whitespace behavior is part of the original requirement." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds a concrete flaw in its draft design: validating the raw answer misses a whitespace-only repository component after an explicit owner. It gives a precise corrective action, moves validation after `splitNameAndOwner`, and adds a corresponding test case rather than merely approving the design. Several role sections add little scrutiny, and the review does not challenge the missing push-local test or repository-wide verification commands. Claims about accessible-prompter behavior and CI coverage are asserted without directly presented evidence." + }, + "testing": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The plan defines exact inputs and error oracles, and its prompt stubs plus empty HTTP registry would expose later prompts or network requests by changing the observed error. It covers empty, whitespace-only, explicit-owner whitespace, template, valid-name, non-interactive, local-default, and accessible-prompter behavior across automated and manual levels. The main deficiency is omitting the repository-required `go test ./...` and `make lint` checks, while valid-name and push-local compatibility remain manual rather than automated. Some manual cleanup requires additional token scope and the plan does not independently prove zero requests outside the unit-test harness." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The three-line production change implements the revised design at the correct boundary, validating the parsed repository name before owner selection and API work, and the four table rows exercise the important invalid forms and template reuse. The diff targets existing symbols and plausible surrounding contexts, uses already-present `errors` and `strings` imports, and the stated baseline commit matches the official v2.100.0 release; v2.101.0 also lists interactive repository-name validation as PR #14313. ([github.com](https://github.com/cli/cli/releases/tag/v2.100.0), [github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) The largest gap is no automated push-local case despite the summary claiming all three interactive flows, though the shared helper makes the production behavior consistent. Local command execution failed before launch in the harness, so `git apply --check`, compilation, and test success could not be independently rerun." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "medium", + "ref": "v2.95.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 48, + "input_tokens": 98, + "output_tokens": 50550, + "total_tokens": 5279737, + "latency_seconds": 812.8, + "total_cost_usd": 2.95998054, + "result_subtype": "success", + "task_score": 87.2, + "cache_read_tokens": 5097072, + "cache_write_tokens": 132017, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 62.2, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 22, + "total": 88, + "notes": "The issue clearly identifies the command, concurrency path, observed stack, user impact, exclusions, and testable acceptance criteria; issue #13399 confirms the command and `CreateTunnelPort` failure at manager.go:452. ([github.com](https://github.com/cli/cli/issues/13399)) Its strongest aspect is distinguishing the shared options-map race, tunnel mutation, and notification behavior instead of treating every reported error as one cause. The largest limitation is the absent independent task statement: the leaked-header explanation for HTTP 400 and lost-notification hang are technically reasoned extensions rather than established requirements, although the dependency's nonblocking single-recipient channel supports the latter. ([github.com](https://github.com/microsoft/dev-tunnels/blob/v0.0.7/go/tunnels/remote_forwarded_ports.go)) The compatibility criterion covering many unrelated codespace commands is also broader than necessary to define this bug." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 22, + "total": 90, + "notes": "The LLD is implementation-ready: it names the real connection and forwarder files, specifies mutex ownership, option cloning, pointer semantics, method signatures, polling behavior, mock synchronization, and rollout checks. Baseline source confirms that `CodespaceConnection` owns exported manager, options, and tunnel pointers while `NewPortForwarder` copies the connection value, making the selected lock ownership and pointer conversion appropriate. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go)) Its largest deficiency is that the latency estimate is explicitly unmeasured and it does not discuss context-unaware waiting to acquire `portsMu` during a slow management request. The exact v0.1.19 line-level mutation claims could not be independently retrieved from the local checkout, though the submitted issue stack and available dependency notification source corroborate the critical behavior." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 23, + "total": 90, + "notes": "The review performs real design stress rather than ceremonial approval, especially by finding the concurrent mock-map access, the lock-defeating connection copy, and the ability to bypass synchronization through exported fields. The value-copy and exported-field findings agree with the baseline `CodespaceConnection` and `NewPortForwarder` definitions. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go)) Recommendations are prioritized, actionable, and explicitly resolved in the revised LLD, while cancellation, serialization cost, security exposure, and testability receive proportionate treatment. The largest limitation is that the mock handler's precise line-level race could not be independently verified after local shell access failed, and several reviewer sections add little beyond approving unchanged behavior." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 82, + "notes": "The plan has concrete race-test commands, repeated scheduling runs, exact helper outcomes, manual multi-port checks, cancellation coverage, compatibility checks, and valid repository surfaces such as `make bin/gh` and codespace JSON fields. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile), [cli.github.com](https://cli.github.com/manual/gh_codespace_ports)) Its largest deficiency is that the acceptance criterion that stored request options remain unchanged is assigned only to code review rather than a test that invokes create and then inspects the original headers. The negative check requires manually reverting and adapting files, so it is not a stable reproducible test recipe, and the manual `org` visibility scenario depends on codespace policy. The concurrent test also omits the LLD's promised protocol oracle, so it can pass while creating requested ports with an incorrect protocol." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 22, + "risk_awareness": 20, + "total": 86, + "notes": "The patch implements the central design end to end: connection-wide serialization, per-call header-map cloning, shared pointer ownership, guarded mock state, bounded forwarded-port waits, and focused concurrency and cancellation tests. Its target symbols and surrounding contexts match the established baseline structure, and the submitted baseline hash corresponds to release v2.95.0. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/internal/codespaces/connection/connection.go), [github.com](https://github.com/malor/gh-cli/blob/trunk/internal/codespaces/portforwarder/port_forwarder.go), [github.com](https://github.com/cli/cli/releases/tag/v2.95.0)) The largest defect is test coverage: no test proves `conn.options.AdditionalHeaders` remains unmodified, and `TestForwardPortConcurrently` verifies presence but not the promised HTTP protocol. Local sandbox initialization failed, so `git apply --check`, compilation, and repository-wide symbol searches could not be independently executed; no source-level contradiction was found through the available exact issue, release, and baseline evidence." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "low", + "ref": "v2.101.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 41, + "input_tokens": 86, + "output_tokens": 34164, + "total_tokens": 4460295, + "latency_seconds": 379.6, + "total_cost_usd": 2.3436083, + "result_subtype": "success", + "task_score": 87.0, + "cache_read_tokens": 4309020, + "cache_write_tokens": 117025, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 90.0, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 21, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 87, + "notes": "The issue provides unusually testable acceptance criteria covering success, refusal, authorization scheme, rejected token classes, and host-check ordering. Repository evidence supports the existing enterprise rejection and token-scheme behavior described. ([github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/13317)) Its largest deficiency is provenance: no independent task statement exists, issue #14495 could not be verified, and the supplied labels omit the required help wanted eligibility signal. The endpoint's ghu_ acceptance, Bearer 404 behavior, and Copilot credential claim therefore remain empirical assertions rather than verified repository facts." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 89, + "notes": "The LLD is implementation-ready, naming the central allowlist, token classifier, callers, error path, request transport, and concrete test changes. Its account of the existing host and permission gates agrees with available repository evidence, while the proposed one-entry allowlist change preserves those controls. ([github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/14302), [github.com](https://github.com/cli/cli/issues/13317)) The main deficiency is that the load-bearing claim that ghu_ tokens receive HTTP 201 is not independently established, and data-residency support is explicitly unresolved. It also does not address the repository's upstream-contribution eligibility gate despite describing a release rollout." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds concrete design defects rather than merely approving: missing JSON Content-Type in StubUpload, brittle full-URL error assertions, and the need to pin host-before-token validation. Those findings directly shape the revised tests and are technically coherent with the patch's use of StatusJSONResponse and substring assertions. Its largest omission is failure to challenge the unverified endpoint premise or the absent help wanted authorization required for an intended upstream contribution. The detailed StubUpload and HandleHTTPError behavior could not be independently checked against local source because repository shell inspection was unavailable." + }, + "testing": { + "completeness": 21, + "correctness": 19, + "specificity": 23, + "risk_awareness": 19, + "total": 82, + "notes": "The strongest coverage is the unit matrix with exact oracles for URL, status, header scheme, request count, unchanged rejection behavior, and comment mutation suppression. The package paths and token-header expectation align with the repository behavior available for verification. ([github.com](https://github.com/cli/cli/issues/13317), [github.com](https://github.com/cli/cli/issues/14309)) The proposed live refusal setup may fail during repository lookup before reaching upload because an app lacking repository access cannot necessarily exercise the intended branch, and the eight-byte PNG fixture is not a valid renderable image. The plan also omits the repository-required go fix and make lint gates and provides no cleanup plan for comments and uploaded assets created by manual tests." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the design end to end: one shared allowlist addition plus focused tests for acceptance, authorization, server refusal, host ordering, and the comment path. Available repository evidence confirms the surrounding enterprise gate, restricted token policy, and token Authorization scheme that the patch relies upon. ([github.com](https://github.com/cli/cli/issues/14309), [github.com](https://github.com/cli/cli/issues/14328), [github.com](https://github.com/cli/cli/issues/13317)) No material code defect is apparent, though the implementation proceeded without establishing the required upstream issue eligibility and its comment states the 201 result more categorically than the permission-dependent design warrants. Exact git apply applicability and imports could not be independently rechecked because the local read-only shell wrapper failed, so that claim remains an evidence gap rather than a scored execution failure." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "release-create-404-scope-blame", + "complexity": "low", + "ref": "v2.97.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 30, + "input_tokens": 62, + "output_tokens": 25129, + "total_tokens": 2230768, + "latency_seconds": 270.2, + "total_cost_usd": 1.3867227000000004, + "result_subtype": "success", + "task_score": 86.8, + "cache_read_tokens": 2139595, + "cache_write_tokens": 65982, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 93.0, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue names the affected file and helper, explains the parsing failure, distinguishes all three token cases, and supplies testable acceptance criteria; the v2.98.0 release records this same fix, and project API traces show comma-space scope headers. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0), [github.com](https://github.com/cli/cli/issues/9927)) Its largest limitation is the absent independent task statement, so broader requirements must be inferred, while the claim that GitHub alphabetically sorts scopes was not independently verified. The first/middle/last, look-alike, missing-header, command-error, and retained-hint criteria are unusually specific. It also identifies the remaining 404 heuristic and explicitly excludes broader parser refactoring and cause detection." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD follows the real `createRun` to `createRelease` error path, commits to a trim-and-exact-match loop, identifies the unused `slices` import, and specifies realistic JSON HTTP stubs and exact error output. Its cited repository convention is supported by `generateScopesSuggestion`, which splits the same header and applies `strings.TrimSpace`. ([github.com](https://github.com/cli/cli/issues/9403)) The main gap is that its helper-test matrix does not literally exercise first, middle, and last positions both with and without spaces as promised by the issue. Local line numbers, the asserted go-gh dependency version, and the claimed absence of an existing `http_test.go` could not be independently checked because local command execution failed at the sandbox layer." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 83, + "notes": "The review finds substantive defects rather than merely approving: it catches position-sensitive coverage, the need for a JSON content type to obtain `Not Found`, the unused import, and unsafe substring matching. Its largest error is claiming scripts that grep stderr are unaffected, even though the intended behavior explicitly changes stderr from the hint to the underlying error for normally formatted scoped-token responses. The security discussion correctly focuses on whole-token equality and preserving the empty-header heuristic, and the SRE section recommends release-note communication. The assertion that this package had no existing `http_test.go` was not independently verifiable under the unavailable local execution environment." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 17, + "total": 77, + "notes": "The strongest portion is the automated coverage: it supplies exact commands, concrete boolean and error-string oracles, negative look-alikes, stderr exclusivity, legacy headers, and package/full-suite checks. The major correctness defect is the E2E workflow-change scenario, which creates `wf-change` and a tag only locally but never pushes the branch before passing it to `--target`, even though that option identifies the remote branch or commit used for automatic tag creation. ([cli.github.com](http://cli.github.com/manual/gh_release_create)) The plan also omits the repository-required `make lint` check and mutates authentication scopes and creates releases without a restoration or cleanup procedure. The `--remove-scopes` flag itself is valid, so the problem is test isolation rather than an invented command. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh))" + }, + "implementation": { + "completeness": 22, + "correctness": 24, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the design end to end: it removes the sole claimed `slices` use, trims each comma-delimited scope while preserving exact equality, and adds helper and command-level tests for both resulting 404 branches. GitHub's v2.98.0 release independently lists `Trim spaces when parsing X-Oauth-Scopes in gh release create`, strongly corroborating the chosen production fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) The largest limitation is test completeness: the helper cases do not fully cover every first/middle/last and spaced/unspaced combination required by the issue, although the loop itself handles them. The claimed `git apply --check` result and exact working-tree applicability could not be rerun because the local sandbox wrapper failed before command execution; no contradictory context was found in the submitted baseline diff." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "ref": "v2.97.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 21, + "input_tokens": 44, + "output_tokens": 23419, + "total_tokens": 1505052, + "latency_seconds": 225.9, + "total_cost_usd": 1.1649935000000002, + "result_subtype": "success", + "task_score": 86.2, + "cache_read_tokens": 1420295, + "cache_write_tokens": 61294, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 103.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 20, + "specificity": 23, + "risk_awareness": 17, + "total": 83, + "notes": "The issue provides exact mode-specific output, testable acceptance criteria, and clear non-goals. Its root-cause claim matches `printResults`, which returns silently for non-TTY output, while the exporter path already precedes that function. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go)) The largest weakness is its unsupported retry-count anecdote and the claim that empty stdout prevents detecting success despite the command's zero exit status. No independent task statement was supplied, so the ID and help-text requirements are established only by these internally consistent artifacts." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is implementation-ready, naming the correct command, mutation result, exporter branch, TTY decision, tests, and exact output. The inspected source confirms that `runAddItem` exports the returned `ProjectItem` before calling `printResults`, and that the proposed edit targets the correct function. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go)) It also addresses write errors, compatibility with scripts expecting empty output, rollout, and the absence of extra API calls. The GraphQL non-null assertion and every claimed sibling-command output pattern could not be independently verified from the local tree because the provided shell sandbox failed." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 22, + "risk_awareness": 20, + "total": 85, + "notes": "The strongest review finding is concrete: explicitly force non-TTY mode in the JSON test and use different content and mutation IDs so the new assertion proves the correct data source. Those recommendations directly exercise the exporter precedence visible in `runAddItem` and the ID-based follow-up workflow used by `item-edit`. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go), [github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-edit/item_edit.go)) Its main deficiency is that much of the persona-based review approves the design without challenging the weak exit-status motivation or requesting automated help-text coverage. The security and compatibility conclusions are reasonable, but the submission provides no independent task statement against which to verify broader expectations." + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The plan supplies strong oracles for exact stdout, trailing newlines, JSON precedence, failure separation, forced TTY behavior, ID reuse, and help output. Its flags and positional syntax agree with the documented `gh project item-add` interface, while the proposed ID chaining is compatible with `item-edit` and `item-delete`. ([cli.github.com](https://cli.github.com/manual/gh_project_item-add), [github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-edit/item_edit.go)) The largest weakness is that several live tests repeatedly mutate the same project without systematic cleanup, making the suite stateful and potentially disruptive despite using a test project. Failure and help behavior are only manual checks rather than durable unit tests, and execution was not demonstrated." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the design end to end: it changes the exact silent non-TTY branch to print `item.ID()`, preserves TTY and exporter behavior, documents the output, and adds focused regression coverage. Its context matches the inspected `item_add.go` structure, and the official v2.98.0 release identifies this same non-TTY defect as fixed. ([github.com](https://github.com/malor/gh-cli/blob/fix-doc/pkg/cmd/project/item-add/item_add.go), [github.com](https://github.com/cli/cli/releases)) The new test correctly distinguishes the content ID from the created project-item ID and explicitly pins JSON execution to non-TTY mode. The main deficiencies are use of `assert.NoError` instead of the repository-mandated `require.NoError`, absence of executed tests or formatting checks, and an apply-check claim that could not be independently repeated because the local shell sandbox failed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 31, + "input_tokens": 64, + "output_tokens": 37335, + "total_tokens": 3126641, + "latency_seconds": 532.3, + "total_cost_usd": 2.05617588, + "result_subtype": "success", + "task_score": 85.6, + "cache_read_tokens": 2984149, + "cache_write_tokens": 105093, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 70.1, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The issue clearly traces the algorithm through lookup, output, and verifier policy, with testable acceptance criteria and useful non-goals. Its reproduction syntax for verify-asset incorrectly suggests an asset plus --tag, while the command accepts an optional positional tag followed by the file path. ([cli.github.com](https://cli.github.com/manual/gh_release_verify-asset)) The submitted diff context supports the cited hard-coded constructor locations, although a sandbox bootstrap failure prevented independent checkout inspection. With no independent task statement, the claimed SHA-256 repository rollout and server-side indexing behavior remain unverified." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The LLD is unusually concrete about files, interfaces, data flow, output changes, compatibility, annotated tags, and test seams. Its proposed helper and both call-site changes are internally consistent with the submitted baseline diff and preserve valid 40-hex behavior. The largest unresolved dependency is whether the attestation service actually indexes SHA-256 refs under the proposed key, which the design acknowledges but cannot establish from the supplied task context. It also validates with hex.DecodeString before checking length, allowing an oversized server-controlled hexadecimal value to cause an avoidable allocation and very large error output." + }, + "review": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 88, + "notes": "The review finds real, actionable defects: inconsistent error text, case normalization ambiguity, malformed-ref compatibility, an unused import, and a verifier mock that cannot observe the artifact. It records concrete resolutions that are reflected in the revised LLD and patch rather than merely issuing generic approvals. Its largest miss is treating an unbounded ref only as a long-line concern, overlooking the allocation performed by hex.DecodeString before length rejection. The assertion about future attestation-service behavior remains unverifiable because no independent task or service contract was supplied." + }, + "testing": { + "completeness": 19, + "correctness": 18, + "specificity": 22, + "risk_awareness": 19, + "total": 78, + "notes": "The unit-test matrix has strong oracles for algorithm selection, request digest, output, verifier input, malformed refs, and SHA-1 compatibility. The proposed recording verifier and captured FetchParams.Digest directly test the two important downstream effects rather than merely asserting successful execution. The plan omits the repository-required go test ./... and make lint checks, substituting narrower package tests and go vet. Its default manual success target is cli/cli v2.92.0 even though that repository documents immutable releases beginning with v2.93.0, and the claimed control-character coverage is not actually supplied by the listed z/g non-hex cases. ([github.com](https://github.com/cli/cli/blob/trunk/README.md))" + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the complete proposed path: one shared helper, both command integrations, correct import cleanup, and tests for lookup, display, verifier input, compatibility, and early failure. The unified-diff contexts and referenced symbols are internally consistent with the baseline artifacts, but the environment's shell sandbox failed before execution, so git apply --check and independent source compilation could not be confirmed. The main implementation risk is calling hex.DecodeString before bounding the input length, which needlessly allocates for an oversized hexadecimal API value and then includes the full value in the error. Apart from that bounded-input concern, the implementation matches the revised LLD and its summary accurately describes the behavioral change." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "high", + "ref": "v2.96.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 42, + "input_tokens": 88, + "output_tokens": 77789, + "total_tokens": 5638016, + "latency_seconds": 1016.8, + "total_cost_usd": 3.81117066, + "result_subtype": "success", + "task_score": 85.4, + "cache_read_tokens": 5394193, + "cache_write_tokens": 165946, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 76.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 22, + "total": 90, + "notes": "The issue provides a crisp workflow problem, detailed acceptance criteria, compatibility requirements, and explicit non-goals. Its baseline claims align with v2.96.0 behavior, while the number/owner/URL interface matches the subsequently released command surface. The largest limitation is that no independent task statement establishes the proposed expansion of --value to text, number, date, and iteration; the public command documentation only specifically describes --value for single-select fields. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) Claims such as the exact opaque error text were not independently substantiated." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 21, + "total": 86, + "notes": "The LLD is strongly grounded in real components including item_edit.go, item-add, queries.Client.NewOwner, NewProject, ProjectField, and the existing mutation dispatch. It commits to concrete validation, pagination, matching, conversion, error, compatibility, and rollout behavior rather than leaving major implementation decisions open. Its largest weakness is product-scope uncertainty: generic conversion and iteration support materially exceed the externally verified released interface, and GHES schema compatibility is not analyzed. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) The stated 50-field cap and per-query rate costs are also asserted without repository evidence." + }, + "review": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 23, + "total": 89, + "notes": "The review finds concrete defects with real consequences, notably missing projectItems pagination, non-finite numbers, empty values, and unnecessary lookups. It prioritizes them and records specific design resolutions, making this substantially more useful than role-based approval theater. The main omission is failure to challenge whether all field types should acquire new --value semantics when the verified released interface emphasizes single-select option names. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) The rate-cost and host-scoping assertions are plausible but not demonstrated from named repository code." + }, + "testing": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 83, + "notes": "The plan has strong behavior-level oracles for every proposed data type, negative resolution paths, pagination, case matching, compatibility, output, and the no-mutation guarantee. The package path and focused command go test ./pkg/cmd/project/item-edit/... agree with the submitted repository structure. However, the GH_DEBUG pipeline reports grep's exit status rather than gh's, and the legacy --number 0 example reuses FIELD_ID initialized for Notes while claiming it is an Estimate field. It also omits the repository-required full go test ./... and make lint checks, reducing confidence in integration and convention compliance." + }, + "implementation": { + "completeness": 20, + "correctness": 19, + "specificity": 21, + "risk_awareness": 19, + "total": 79, + "notes": "The patch implements its own design end to end: it extends the existing options and validation, resolves names before dispatch, paginates projectItems, converts values, preserves the ID-only mutation path, and adds substantial tests. Its target baseline and public surface are credible: v2.96.0 is commit b300f2e, and v2.97.0 introduced number/owner/URL/field/value addressing. ([github.com](https://github.com/cli/cli/releases/)) The largest concrete quality defect is that new tests repeatedly use assert for error checks despite the repository instruction requiring require, and the pull-request test mocks a PullRequest response for an issue-form URL. Compilation and lint remain unconfirmed because the summary explicitly reports no build, formatting, or test execution, although absence of execution was not itself penalized." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "high", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 32, + "input_tokens": 66, + "output_tokens": 40125, + "total_tokens": 2968868, + "latency_seconds": 593.7, + "total_cost_usd": 2.0201634200000003, + "result_subtype": "success", + "task_score": 81.4, + "cache_read_tokens": 2835341, + "cache_write_tokens": 93336, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 67.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 24, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 93, + "notes": "The issue gives a reproducible integrity failure, explicit non-goals, and acceptance criteria covering force, detached HEAD, missing branches, and unchanged behavior. GitHub CLI 2.99.0 independently lists fixes for linked-worktree corruption and its error explanation, supporting the core problem claim. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Its largest weakness is that no independent task statement exists, so scope is largely self-defined, and the asserted minimum supported Git version was not repository-verified. The proposed preflight solution also leaves atomicity and path-normalization decisions for the design." + }, + "lld": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 82, + "notes": "The LLD is concrete about `pkg/cmd/repo/sync/{sync.go,git.go,mocks.go}`, `executeLocalRepoSync`, call ordering, orphan branches, stale worktrees, and user-visible errors. Its strongest decision is placing the guard before both `UpdateBranch` and `CreateBranch`, while explicitly preserving current-branch behavior. The main defect is a time-of-check/time-of-use race between `worktree list` and `update-ref`, so the stated goal of never corrupting another worktree is not guaranteed. It also promises an absolute path without handling Git's supported relative-worktree configuration and understates an O(number of worktrees) scan as effectively constant time. ([git-scm.com](https://git-scm.com/docs/git-worktree), [git-scm.com](https://git-scm.com/docs/git-config))" + }, + "review": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 22, + "total": 87, + "notes": "The review identifies substantive defects in its earlier design, especially the unguarded unborn-branch creation path, stale-worktree recovery, and the need for real Git tests. Recommendations are prioritized, actionable, and tied to concrete symbols and behaviors rather than generic reviewer personas. Its largest miss is accepting the non-atomic list-then-update design without discussing concurrent worktree creation, and it does not challenge the absolute-path assumption under relative-worktree configuration. The detached rebase and bisect limitation is honestly documented, although the claimed Git support floor could not be verified." + }, + "testing": { + "completeness": 20, + "correctness": 16, + "specificity": 23, + "risk_awareness": 18, + "total": 77, + "notes": "The plan has unusually concrete setup, actions, and oracles, including unchanged refs, clean porcelain status, force behavior, error propagation, and positive controls. The repository workflow does use Linux, Windows, and macOS and runs the Go test suite, so the cross-platform intent is grounded. ([github.com](https://github.com/cli/cli/blob/d4e33858efe4b8b96dafc13fa2e806ec857886c0/.github/workflows/go.yml)) The most important missing real-Git case is the unborn branch that motivated guarding `CreateBranch`; mocks cannot validate its porcelain representation. The stale manual test incorrectly proposes deleting the main worktree containing the shared Git directory, the missing-branch manual test never creates the remote branch, and the environment setup does not fully isolate global Git configuration." + }, + "implementation": { + "completeness": 20, + "correctness": 13, + "specificity": 21, + "risk_awareness": 14, + "total": 68, + "notes": "The patch implements the described common-case guard before both ref-writing paths and adds focused unit and real-repository tests; the stated baseline commit matches the v2.98.0 release. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) A submitted real-Git assertion uses a raw string containing `\\\"main\\\"`, but the `%q` error contains ordinary quote characters, so that test necessarily fails. Production behavior also remains vulnerable to the list-then-update race and may report a relative rather than absolute worktree path when relative worktree links are configured. Exact patch applicability could not be independently checked because the provided local shell sandbox failed before execution, making the summary's `git apply --check` claim unverifiable." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "auth-clipboard-config-default", + "complexity": "high", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 69, + "input_tokens": 144, + "output_tokens": 51908, + "total_tokens": 9509769, + "latency_seconds": 830.4, + "total_cost_usd": 4.111767660000001, + "result_subtype": "success", + "task_score": 72.4, + "cache_read_tokens": 9289448, + "cache_write_tokens": 168269, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 62.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 17, + "correctness": 11, + "specificity": 23, + "risk_awareness": 18, + "total": 69, + "notes": "The issue provides unusually concrete precedence rules, acceptance criteria, user stories, and exclusions. Its central requirement is wrong: it specifies a disabled default and unchanged behavior, while v2.101.0 explicitly introduced copying by default with persistent opt-out via `clipboard disabled`. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) Issue #13153 supports the motivation and proposed configuration key, and the CLI manuals confirm per-host configuration support. ([github.com](https://github.com/cli/cli/issues/13153), [cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get)) No independent task statement was supplied, so exact diagnostic wording could not be established separately from the candidate's assertions." + }, + "lld": { + "completeness": 19, + "correctness": 13, + "specificity": 23, + "risk_awareness": 19, + "total": 74, + "notes": "The LLD is strongly grounded in named repository files, interfaces, option structs, mock generation, hostname resolution, and the existing auth-flow boundary. Its load-bearing decision is nevertheless incorrect: `DefaultValue: \"disabled\"` and the claimed no-change rollout contradict the released requirement that clipboard copying default to enabled. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) The host-scoped design is feasible because `gh config set`, `get`, and `list` expose per-host operation. ([cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get), [cli.github.com](https://cli.github.com/manual/gh_config_list)) It honestly records the untested `loginRun` wiring, but leaves that regression path unmitigated, and the unavailable local shell prevented complete independent validation of every line-level repository claim." + }, + "review": { + "completeness": 18, + "correctness": 15, + "specificity": 22, + "risk_awareness": 20, + "total": 75, + "notes": "The strongest finding is Byte's concrete identification of the existing `Test_refreshRun` case that must set `ClipboardProvided`, including the exact behavioral consequence. The review never challenges the design's disabled default, despite the corresponding release explicitly defaulting clipboard copying to enabled, so it misses the most important product-level defect. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) It also says there is \u201cno new file\u201d while the reviewed design adds `clipboard.go` and `clipboard_test.go`, an internal factual inconsistency. Its SSH/headless clipboard warning, generated-mock drift concern, discoverability feedback, and login wiring gap are otherwise specific and actionable." + }, + "testing": { + "completeness": 20, + "correctness": 12, + "specificity": 23, + "risk_awareness": 20, + "total": 75, + "notes": "The plan covers config validation, host precedence, both auth commands, explicit true and false flags, help, rollback, headless failures, and detailed output oracles. Its central compatibility and default assertions are backwards: it expects fresh configuration not to copy, whereas v2.101.0 intentionally copies by default and uses `clipboard disabled` as the opt-out. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) The proposed `go generate` drift check is also unsound immediately after applying an uncommitted patch because `git diff --exit-code internal/gh/mock/config.go` detects the intended patch itself. Most individual tests are precise, but no automated test observes the value passed by `loginRun`, leaving an acknowledged wiring regression uncovered." + }, + "implementation": { + "completeness": 18, + "correctness": 12, + "specificity": 22, + "risk_awareness": 17, + "total": 69, + "notes": "The patch coherently adds the config accessor and interface method, generated mock support, flag-provided tracking, a shared precedence resolver, login and refresh wiring, help text, and focused tests. However, `DefaultValue: \"disabled\"` and `clipboard: disabled` preserve the old default, directly failing the defining v2.101.0 behavior of copying authentication codes by default. ([github.com](https://github.com/cli/cli/releases/tag/v2.101.0), [cli.github.com](https://cli.github.com/manual/gh_config)) Explicit flag precedence and host scoping are internally consistent, and per-host config operation is supported by the CLI. ([cli.github.com](https://cli.github.com/manual/gh_config_set), [cli.github.com](https://cli.github.com/manual/gh_config_get)) Local `git apply --check` and compilation could not be independently run because the repository shell sandbox failed before command execution; additionally, the summary's claim that the mock was regenerated is unclear because it also says moq could not read the toolchain export data." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 38, + "input_tokens": 80, + "output_tokens": 34133, + "total_tokens": 3443304, + "latency_seconds": 356.3, + "total_cost_usd": 1.9909293799999996, + "result_subtype": "success", + "task_score": 69.0, + "cache_read_tokens": 3316354, + "cache_write_tokens": 92737, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 95.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 21, + "correctness": 12, + "specificity": 23, + "risk_awareness": 18, + "total": 74, + "notes": "The issue gives concrete reproductions, unchanged cases, exclusions, and testable exit-code and no-request criteria. Its largest defect is requiring `--comments --json comments` to remain valid; the official v2.99.0 release describes the shipped fix as rejecting `--comments` with `--json`, and later usage evidence describes outright rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) It usefully covers scripts, sibling commands, and unrelated search qualifiers. With no independent task statement and unavailable local shell access, the exact issue #14214 wording and claimed acceptance-test usages could not be verified." + }, + "lld": { + "completeness": 22, + "correctness": 11, + "specificity": 23, + "risk_awareness": 19, + "total": 75, + "notes": "The LLD is highly specific about command constructors, exporter lifecycle, validation ordering, error handling, files, and tests. Official manuals confirm that the three commands expose the relevant `--comments` and JSON surfaces. ([cli.github.com](https://cli.github.com/manual/gh_issue_view), [cli.github.com](https://cli.github.com/manual/gh_pr_view), [cli.github.com](https://cli.github.com/manual/gh_discussion_view)) Its central conditional-rejection design conflicts with the shipped blanket rejection and leaves PR review parity explicitly unresolved. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Exact line numbers, `PreRunE` behavior, and acceptance-fixture claims could not be independently checked locally." + }, + "review": { + "completeness": 18, + "correctness": 12, + "specificity": 21, + "risk_awareness": 18, + "total": 69, + "notes": "The review finds a concrete discussion error-precedence bug and converts it into an actionable ordering test. It also identifies the important PR-specific fact that human `--comments` output includes reviews, although it permits that mismatch to be deferred. Its largest miss is endorsing the conditional policy without challenging the upstream blanket-rejection contract evidenced by the v2.99.0 fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Claims about exact acceptance fixtures and existing source expressions were not independently verifiable through the unavailable local shell." + }, + "testing": { + "completeness": 19, + "correctness": 11, + "specificity": 21, + "risk_awareness": 17, + "total": 68, + "notes": "The plan provides a broad command matrix, exact error oracles, precedence cases, compatibility checks, and live smoke tests. Its build command is grounded in the repository Makefile, which defines `bin/gh` as a target. ([github.com](https://github.com/cli/cli/blob/trunk/Makefile), [github.com](https://github.com/cli/cli/blob/trunk/Makefile)) The main deficiency is that several tests require the combined flags to succeed when `comments` is selected, contradicting the shipped rejection behavior. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) The HTTP-debug grep is not demonstrated for all commands, and \u201csame output as v2.98.0\u201d is not a concrete regression oracle." + }, + "implementation": { + "completeness": 15, + "correctness": 10, + "specificity": 20, + "risk_awareness": 14, + "total": 59, + "notes": "The diff consistently adds guards and focused constructor tests across issue, PR, and discussion view, including discussion error precedence and prevention of `runF` execution. The stated baseline commit matches the official v2.98.0 release listing. ([github.com](https://github.com/cli/cli/releases)) Its largest defect is implementing conditional rejection rather than the shipped rule that rejects `--comments` with `--json` outright, so a key combined invocation remains accepted. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) Local `git apply --check` and compilation could not be repeated because the shell sandbox failed to start, while the summary confirms tests and formatting were not executed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "skill-install-nested-discovery", + "complexity": "high", + "ref": "v2.93.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 65, + "input_tokens": 134, + "output_tokens": 77708, + "total_tokens": 11596183, + "latency_seconds": 1130.2, + "total_cost_usd": 5.384274059999999, + "result_subtype": "success", + "task_score": 63.4, + "cache_read_tokens": 11302418, + "cache_write_tokens": 215923, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 68.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 13, + "correctness": 10, + "specificity": 22, + "risk_awareness": 15, + "total": 60, + "notes": "The issue provides unusually concrete paths, exclusions, and testable acceptance criteria tied to internal/skills/discovery/discovery.go. Its largest deficiency is redefining the task as broad automatic discovery, while the v2.94.0 fix explicitly retained existing publishing conventions and supported non-standard nested layouts only through exact path arguments. ([github.com](https://github.com/cli/cli/pull/13459)) The issue consequently omits the critical packages/agent-skills/code-review case and the rule preserving one-slash selectors as namespaced names, both documented in the resulting CLI behavior. ([cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview)) No independent task statement was supplied, but the fixed-in-2.94.0 repository evidence makes this scope mismatch material." + }, + "lld": { + "completeness": 15, + "correctness": 10, + "specificity": 23, + "risk_awareness": 17, + "total": 65, + "notes": "The LLD is highly specific about real packages and symbols such as matchSkillConventions, matchHiddenDirConventions, DiscoverSkillByPath, and selectSkill, with careful rule-order analysis. Its central architecture is nevertheless wrong for the released fix: it classifies conventional paths after discovery instead of introducing shared path detection and directly fetching arbitrary nested path arguments. ([github.com](https://github.com/cli/cli/pull/13459)) It also declares namespaced layouts below plugins out of scope while its proposed generic hidden matcher would accept a hidden plugins//skills// path. Approximate line locations and the claimed baseline context could not be independently checked locally because repository command execution was unavailable." + }, + "review": { + "completeness": 18, + "correctness": 15, + "specificity": 22, + "risk_awareness": 20, + "total": 75, + "notes": "The review is the strongest artifact because it identifies concrete matcher-order, hidden-gating, compatibility, and path-classification defects and supplies actionable resolutions. It fails to challenge the design's fundamental expansion of automatic search and publish behavior, despite the merged change explicitly rejecting that expansion. ([github.com](https://github.com/cli/cli/pull/13459)) It also misses the contradiction whereby the proposed hidden matcher discovers namespaced plugin layouts that the LLD labels a non-goal. Claims about all uses and persistence of Convention were plausible but could not be fully verified against the local source." + }, + "testing": { + "completeness": 14, + "correctness": 12, + "specificity": 22, + "risk_awareness": 17, + "total": 65, + "notes": "The plan has concrete fixtures, package commands, negative cases, and meaningful output or filesystem oracles for the behavior it chose. It never tests the critical arbitrary non-standard path packages/agent-skills/code-review, shared path detection, or preservation of one-slash namespaced selectors required by the released fix. ([cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview), [github.com](https://github.com/cli/cli/pull/13459)) Some manual checks are weak: the compatibility diff only prints an exit status, and update --all is not directed at the temporary installation directories. The implementation explicitly states that none of these tests or format checks were executed, so their passing status is unverified." + }, + "implementation": { + "completeness": 12, + "correctness": 8, + "specificity": 20, + "risk_awareness": 12, + "total": 52, + "notes": "The patch does implement any-depth hidden-directory matching, preserves the root-hidden ordering case, and adds focused tests and documentation. It does not modify or share isSkillPath logic, and preview merely searches already-discovered skills, so an arbitrary path such as packages/agent-skills/code-review still fails even though that was a principal v2.94.0 requirement. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview)) It additionally broadens automatic search and publish behavior through generalized plugins matching, contrary to the merged product boundary, and its hidden matcher violates its own namespaced-under-plugins exclusion. Exact git apply applicability, compilation, formatting, and test results remain unverified; the summary itself confirms tests and gofmt were not run." + } + }, + "is_error": false, + "failed": false + } + ], + "run_date": "2026-10-14" +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json new file mode 100644 index 00000000..0d5274dd --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json @@ -0,0 +1,64 @@ +{ + "task": "secret-set-app-agents", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The strongest aspect is the testable acceptance criteria covering every command, scope, endpoint, default, message, and help surface. The named shared and command packages match the repository architecture, and PR #13421 confirms that list and delete support were required alongside set. The largest deficiency is the absent independent task specification, which prevents fully verifying whether this expanded scope and exact wording were required. GHES behavior is sensibly documented, but support across particular server versions remains externally dependent." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The design is unusually well grounded in real integration points, including shared.GetSecretApp, shared.IsSupportedSecretEntity, putOrgSecret, and the application-interpolated request paths. It correctly commits to integer organization repository IDs and identifies that agents must avoid the Dependabot-specific payload branch. Its largest weakness is disproportionate detail around estimates and diagrams while rollout remains essentially release-and-observe with only passthrough 404 handling. Local patch application could not be independently executed, although the named baseline symbols and surrounding contexts align with repository and merged-change evidence." + }, + "review": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The strongest contribution is finding concrete defects in the initial test design, particularly hard-coded Actions list paths, missing selected-visibility payload coverage, and absent unsupported-entity checks. Those findings are actionable and visibly drive corresponding changes in list_test.go and set_test.go. The security discussion slightly overstates the consequence of moving validation below public-key retrieval because fetching a key alone does not transmit the secret value. GHES limitations and default organization visibility are honestly retained as known tradeoffs rather than falsely declared resolved." + }, + "testing": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 20, + "total": 88, + "notes": "The plan has strong behavioral oracles for endpoint selection, payload visibility, selected-repository counts, JSON output, defaults, case handling, completion, and independence from Actions secrets. Its commands and paths agree with the repository's Go test layout and documentation generator. The largest deficiency is poor cleanup and isolation for live tests: several repository and organization secrets and /tmp/agents.env are left behind, while the expected Actions 404 assumes no preexisting secret with that name. Execution-dependent claims about live permissions, GHES responses, and exact terminal output were not demonstrated." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the feature end to end by extending shared application parsing and entity support, all three command enums, help text, and repository and organization runtime tests while correctly leaving interpolating HTTP helpers unchanged. The diff contexts and symbols match the v2.92.0 baseline architecture, and the same core approach was accepted in PR #13421. The largest concrete defect is the new assert.EqualError usage despite the repository instruction to use require for error checks; additionally, an empty httpmock.Registry does not itself record or prove unmatched requests as the test comment implies. No functional routing defect is apparent, but the claimed apply check and test outcome could not be independently executed, and the implementation summary acknowledges that compilation and tests were not run." + } + }, + "task_score": 90.8, + "verdict": "This is a strong, repository-specific submission with a functionally complete patch, limited mainly by unexecuted verification, minor test-convention defects, and insufficient cleanup in the live testing plan.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:55:41.039052Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 226297, + "cached_input_tokens": 141369, + "cache_write_input_tokens": 37625, + "output_tokens": 10158, + "reasoning_output_tokens": 8569 + }, + "duration_ms": 176583 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json new file mode 100644 index 00000000..2e314bbd --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "secret-set-app-agents", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "secret", + "go", + "cli", + "github-api", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 71.2, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 84, + "output_tokens": 39404, + "cache_read_tokens": 4168417, + "cache_write_tokens": 124665, + "total_tokens": 4332570, + "total_cost_usd": 2.46996684, + "latency_seconds": 553.5, + "num_turns": 40, + "generation_tokens_per_sec": 71.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 84, + "output_tokens": 39404, + "latency_seconds": 553.5, + "num_turns": 40, + "total_cost_usd": 2.46996684, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4168417, + "cache_creation_tokens": 124665, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T18:35:01.889806Z", + "run_ended_at": "2026-10-14T18:44:15.439308Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 84, + "output_tokens": 39404, + "cache_read_tokens": 4168417, + "cache_write_tokens": 124665, + "total_tokens": 4332570, + "total_cost_usd": 2.46996684, + "latency_seconds": 553.5, + "num_turns": 40, + "generation_tokens_per_sec": 71.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "secret-set-app-agents", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The strongest aspect is the testable acceptance criteria covering every command, scope, endpoint, default, message, and help surface. The named shared and command packages match the repository architecture, and PR #13421 confirms that list and delete support were required alongside set. The largest deficiency is the absent independent task specification, which prevents fully verifying whether this expanded scope and exact wording were required. GHES behavior is sensibly documented, but support across particular server versions remains externally dependent." + }, + "lld": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The design is unusually well grounded in real integration points, including shared.GetSecretApp, shared.IsSupportedSecretEntity, putOrgSecret, and the application-interpolated request paths. It correctly commits to integer organization repository IDs and identifies that agents must avoid the Dependabot-specific payload branch. Its largest weakness is disproportionate detail around estimates and diagrams while rollout remains essentially release-and-observe with only passthrough 404 handling. Local patch application could not be independently executed, although the named baseline symbols and surrounding contexts align with repository and merged-change evidence." + }, + "review": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 23, + "total": 92, + "notes": "The strongest contribution is finding concrete defects in the initial test design, particularly hard-coded Actions list paths, missing selected-visibility payload coverage, and absent unsupported-entity checks. Those findings are actionable and visibly drive corresponding changes in list_test.go and set_test.go. The security discussion slightly overstates the consequence of moving validation below public-key retrieval because fetching a key alone does not transmit the secret value. GHES limitations and default organization visibility are honestly retained as known tradeoffs rather than falsely declared resolved." + }, + "testing": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 20, + "total": 88, + "notes": "The plan has strong behavioral oracles for endpoint selection, payload visibility, selected-repository counts, JSON output, defaults, case handling, completion, and independence from Actions secrets. Its commands and paths agree with the repository's Go test layout and documentation generator. The largest deficiency is poor cleanup and isolation for live tests: several repository and organization secrets and /tmp/agents.env are left behind, while the expected Actions 404 assumes no preexisting secret with that name. Execution-dependent claims about live permissions, GHES responses, and exact terminal output were not demonstrated." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the feature end to end by extending shared application parsing and entity support, all three command enums, help text, and repository and organization runtime tests while correctly leaving interpolating HTTP helpers unchanged. The diff contexts and symbols match the v2.92.0 baseline architecture, and the same core approach was accepted in PR #13421. The largest concrete defect is the new assert.EqualError usage despite the repository instruction to use require for error checks; additionally, an empty httpmock.Registry does not itself record or prove unmatched requests as the test comment implies. No functional routing defect is apparent, but the claimed apply check and test outcome could not be independently executed, and the implementation summary acknowledges that compilation and tests were not run." + } + }, + "task_score": 90.8, + "verdict": "This is a strong, repository-specific submission with a functionally complete patch, limited mainly by unexecuted verification, minor test-convention defects, and insufficient cleanup in the live testing plan.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:55:41.039052Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 226297, + "cached_input_tokens": 141369, + "cache_write_input_tokens": 37625, + "output_tokens": 10158, + "reasoning_output_tokens": 8569 + }, + "duration_ms": 176583 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json new file mode 100644 index 00000000..09ce9ddb --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json @@ -0,0 +1,64 @@ +{ + "task": "skill-install-nested-discovery", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 13, + "correctness": 10, + "specificity": 22, + "risk_awareness": 15, + "total": 60, + "notes": "The issue provides unusually concrete paths, exclusions, and testable acceptance criteria tied to internal/skills/discovery/discovery.go. Its largest deficiency is redefining the task as broad automatic discovery, while the v2.94.0 fix explicitly retained existing publishing conventions and supported non-standard nested layouts only through exact path arguments. ([github.com](https://github.com/cli/cli/pull/13459)) The issue consequently omits the critical packages/agent-skills/code-review case and the rule preserving one-slash selectors as namespaced names, both documented in the resulting CLI behavior. ([cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview)) No independent task statement was supplied, but the fixed-in-2.94.0 repository evidence makes this scope mismatch material." + }, + "lld": { + "completeness": 15, + "correctness": 10, + "specificity": 23, + "risk_awareness": 17, + "total": 65, + "notes": "The LLD is highly specific about real packages and symbols such as matchSkillConventions, matchHiddenDirConventions, DiscoverSkillByPath, and selectSkill, with careful rule-order analysis. Its central architecture is nevertheless wrong for the released fix: it classifies conventional paths after discovery instead of introducing shared path detection and directly fetching arbitrary nested path arguments. ([github.com](https://github.com/cli/cli/pull/13459)) It also declares namespaced layouts below plugins out of scope while its proposed generic hidden matcher would accept a hidden plugins//skills// path. Approximate line locations and the claimed baseline context could not be independently checked locally because repository command execution was unavailable." + }, + "review": { + "completeness": 18, + "correctness": 15, + "specificity": 22, + "risk_awareness": 20, + "total": 75, + "notes": "The review is the strongest artifact because it identifies concrete matcher-order, hidden-gating, compatibility, and path-classification defects and supplies actionable resolutions. It fails to challenge the design's fundamental expansion of automatic search and publish behavior, despite the merged change explicitly rejecting that expansion. ([github.com](https://github.com/cli/cli/pull/13459)) It also misses the contradiction whereby the proposed hidden matcher discovers namespaced plugin layouts that the LLD labels a non-goal. Claims about all uses and persistence of Convention were plausible but could not be fully verified against the local source." + }, + "testing": { + "completeness": 14, + "correctness": 12, + "specificity": 22, + "risk_awareness": 17, + "total": 65, + "notes": "The plan has concrete fixtures, package commands, negative cases, and meaningful output or filesystem oracles for the behavior it chose. It never tests the critical arbitrary non-standard path packages/agent-skills/code-review, shared path detection, or preservation of one-slash namespaced selectors required by the released fix. ([cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview), [github.com](https://github.com/cli/cli/pull/13459)) Some manual checks are weak: the compatibility diff only prints an exit status, and update --all is not directed at the temporary installation directories. The implementation explicitly states that none of these tests or format checks were executed, so their passing status is unverified." + }, + "implementation": { + "completeness": 12, + "correctness": 8, + "specificity": 20, + "risk_awareness": 12, + "total": 52, + "notes": "The patch does implement any-depth hidden-directory matching, preserves the root-hidden ordering case, and adds focused tests and documentation. It does not modify or share isSkillPath logic, and preview merely searches already-discovered skills, so an arbitrary path such as packages/agent-skills/code-review still fails even though that was a principal v2.94.0 requirement. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview)) It additionally broadens automatic search and publish behavior through generalized plugins matching, contrary to the merged product boundary, and its hidden matcher violates its own namespaced-under-plugins exclusion. Exact git apply applicability, compilation, formatting, and test results remain unverified; the summary itself confirms tests and gofmt were not run." + } + }, + "task_score": 63.4, + "verdict": "The submission is detailed and repository-aware, but it designs and implements a broader automatic-discovery feature while missing the released fix's critical arbitrary exact-path handling.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:58:31.481123Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 266457, + "cached_input_tokens": 198989, + "cache_write_input_tokens": 40799, + "output_tokens": 10742, + "reasoning_output_tokens": 8544 + }, + "duration_ms": 170429 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json new file mode 100644 index 00000000..1f9b3fd5 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "skill-install-nested-discovery", + "repo": "https://github.com/cli/cli", + "ref": "v2.93.0", + "complexity": "high", + "tags": [ + "skill", + "go", + "cli", + "discovery", + "fixed-in-2.94.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 68.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 134, + "output_tokens": 77708, + "cache_read_tokens": 11302418, + "cache_write_tokens": 215923, + "total_tokens": 11596183, + "total_cost_usd": 5.384274059999999, + "latency_seconds": 1130.2, + "num_turns": 65, + "generation_tokens_per_sec": 68.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 134, + "output_tokens": 77708, + "latency_seconds": 1130.2, + "num_turns": 65, + "total_cost_usd": 5.384274059999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 11302418, + "cache_creation_tokens": 215923, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T19:36:06.446048Z", + "run_ended_at": "2026-10-14T19:54:56.656848Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 134, + "output_tokens": 77708, + "cache_read_tokens": 11302418, + "cache_write_tokens": 215923, + "total_tokens": 11596183, + "total_cost_usd": 5.384274059999999, + "latency_seconds": 1130.2, + "num_turns": 65, + "generation_tokens_per_sec": 68.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "skill-install-nested-discovery", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 13, + "correctness": 10, + "specificity": 22, + "risk_awareness": 15, + "total": 60, + "notes": "The issue provides unusually concrete paths, exclusions, and testable acceptance criteria tied to internal/skills/discovery/discovery.go. Its largest deficiency is redefining the task as broad automatic discovery, while the v2.94.0 fix explicitly retained existing publishing conventions and supported non-standard nested layouts only through exact path arguments. ([github.com](https://github.com/cli/cli/pull/13459)) The issue consequently omits the critical packages/agent-skills/code-review case and the rule preserving one-slash selectors as namespaced names, both documented in the resulting CLI behavior. ([cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview)) No independent task statement was supplied, but the fixed-in-2.94.0 repository evidence makes this scope mismatch material." + }, + "lld": { + "completeness": 15, + "correctness": 10, + "specificity": 23, + "risk_awareness": 17, + "total": 65, + "notes": "The LLD is highly specific about real packages and symbols such as matchSkillConventions, matchHiddenDirConventions, DiscoverSkillByPath, and selectSkill, with careful rule-order analysis. Its central architecture is nevertheless wrong for the released fix: it classifies conventional paths after discovery instead of introducing shared path detection and directly fetching arbitrary nested path arguments. ([github.com](https://github.com/cli/cli/pull/13459)) It also declares namespaced layouts below plugins out of scope while its proposed generic hidden matcher would accept a hidden plugins//skills// path. Approximate line locations and the claimed baseline context could not be independently checked locally because repository command execution was unavailable." + }, + "review": { + "completeness": 18, + "correctness": 15, + "specificity": 22, + "risk_awareness": 20, + "total": 75, + "notes": "The review is the strongest artifact because it identifies concrete matcher-order, hidden-gating, compatibility, and path-classification defects and supplies actionable resolutions. It fails to challenge the design's fundamental expansion of automatic search and publish behavior, despite the merged change explicitly rejecting that expansion. ([github.com](https://github.com/cli/cli/pull/13459)) It also misses the contradiction whereby the proposed hidden matcher discovers namespaced plugin layouts that the LLD labels a non-goal. Claims about all uses and persistence of Convention were plausible but could not be fully verified against the local source." + }, + "testing": { + "completeness": 14, + "correctness": 12, + "specificity": 22, + "risk_awareness": 17, + "total": 65, + "notes": "The plan has concrete fixtures, package commands, negative cases, and meaningful output or filesystem oracles for the behavior it chose. It never tests the critical arbitrary non-standard path packages/agent-skills/code-review, shared path detection, or preservation of one-slash namespaced selectors required by the released fix. ([cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview), [github.com](https://github.com/cli/cli/pull/13459)) Some manual checks are weak: the compatibility diff only prints an exit status, and update --all is not directed at the temporary installation directories. The implementation explicitly states that none of these tests or format checks were executed, so their passing status is unverified." + }, + "implementation": { + "completeness": 12, + "correctness": 8, + "specificity": 20, + "risk_awareness": 12, + "total": 52, + "notes": "The patch does implement any-depth hidden-directory matching, preserves the root-hidden ordering case, and adds focused tests and documentation. It does not modify or share isSkillPath logic, and preview merely searches already-discovered skills, so an arbitrary path such as packages/agent-skills/code-review still fails even though that was a principal v2.94.0 requirement. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install), [cli.github.com](https://cli.github.com/manual/gh_skill_preview)) It additionally broadens automatic search and publish behavior through generalized plugins matching, contrary to the merged product boundary, and its hidden matcher violates its own namespaced-under-plugins exclusion. Exact git apply applicability, compilation, formatting, and test results remain unverified; the summary itself confirms tests and gofmt were not run." + } + }, + "task_score": 63.4, + "verdict": "The submission is detailed and repository-aware, but it designs and implements a broader automatic-discovery feature while missing the released fix's critical arbitrary exact-path handling.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T20:58:31.481123Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 266457, + "cached_input_tokens": 198989, + "cache_write_input_tokens": 40799, + "output_tokens": 10742, + "reasoning_output_tokens": 8544 + }, + "duration_ms": 170429 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json new file mode 100644 index 00000000..8b02152d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json @@ -0,0 +1,64 @@ +{ + "task": "skill-update-dir-keeps-install-path", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest aspect is the reproducible nested-directory scenario plus acceptance criteria with exact positive and negative filesystem oracles. Its root-cause account matches the submitted update.go context, while the installer really targets filepath.Join(baseDir, skill.Name). ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The largest deficiency is scope expansion into download atomicity and remote-name mismatch without an independent task statement establishing those requirements. The related issue and the practical likelihood of an upstream name mismatch could not be independently verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 23, + "total": 91, + "notes": "The LLD is unusually well grounded in updateRun, installer.Install, installSkill, installedSkill.dir, discovery.Skill, safepaths, and the existing tests. Its parent-directory target, mismatch guard, cleanup removal, and fetch-before-write flow are implementation-ready and consistent with the installer's real target-selection behavior. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The main correctness blemish is its inconsistent explanation that Skill.Name is path-derived while the review also treats it as frontmatter-derived, plus an unsupported assumption that buffering every skill is always small. It explicitly acknowledges the remaining partial-write risk from local I/O failures, although the absent independent task statement limits verification of its expanded scope." + }, + "review": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 23, + "total": 90, + "notes": "The review's strongest contribution is finding three concrete design defects: base plus skill.Name can select a sibling, host-scanned skills also flatten, and file-by-file fetching can leave mixed versions. Each finding identifies a consequence and a specific revision, rather than merely approving the design. The largest flaw is overstating a remote publisher's ability to change a frontmatter-derived name, contradicting the LLD's own statement that discovery.Skill.Name comes from the path; the local-directory rename case remains valid. Repository evidence confirms the installer joins the selected base with skill.Name, but no independent task statement was available to validate every blocking priority. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/))" + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 88, + "notes": "The plan's strongest aspect is its concrete matrix across flat and two-level layouts, custom and agent scans, mismatch refusal, preserved local files, and second-blob failure, with meaningful path and content oracles. The proposed package paths and HTTP-focused test strategy align with the submitted repository structure, and the global lockfile path is supported by repository evidence. ([github.com](https://github.com/cli/cli/issues/13438), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-create-lock-directory-w/)) Its largest deficiency is omitting the repository-required make lint command, while shorthand such as 'install with --pin' and the non-recursive checksum glob make parts of the manual plan less directly executable. Tests were not executed, and the unavailable independent task statement prevents confirming whether all added scenarios are required." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the submitted design end to end: every update receives the current directory's parent, mismatched names fail per skill, destructive cleanup is removed, and downloads complete before filesystem writes begin. The hunks target existing installSkill and updateRun behavior, and the added tests exercise relocation, preservation, host scanning, mismatch refusal, and fetch failure; the underlying installer and lockfile call shapes match repository evidence. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The largest remaining risk is acknowledged but unresolved atomicity for mkdir or write failures, and buffering all files broadens installer memory usage without an evidenced size bound. The local shell sandbox could not start, so the candidate's git apply --check claim and test compilation could not be independently verified." + } + }, + "task_score": 90.2, + "verdict": "This is a strong, repository-specific submission whose largest limitation is the lack of an independent task statement and executable local verification for its broadened failure-atomicity scope.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T21:01:29.329051Z", + "repo_ref": "v2.94.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 184374, + "cached_input_tokens": 105027, + "cache_write_input_tokens": 36788, + "output_tokens": 10990, + "reasoning_output_tokens": 9230 + }, + "duration_ms": 177835 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json new file mode 100644 index 00000000..fdec208f --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "skill-update-dir-keeps-install-path", + "repo": "https://github.com/cli/cli", + "ref": "v2.94.0", + "complexity": "high", + "tags": [ + "skill", + "go", + "cli", + "filesystem", + "fixed-in-2.95.0" + ], + "model": "us.anthropic.claude-opus-5-5", + "model_slug": "claude-opus-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 69.1, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 84, + "output_tokens": 45121, + "cache_read_tokens": 4062273, + "cache_write_tokens": 118410, + "total_tokens": 4225888, + "total_cost_usd": 2.5379866600000005, + "latency_seconds": 653.4, + "num_turns": 40, + "generation_tokens_per_sec": 69.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 84, + "output_tokens": 45121, + "latency_seconds": 653.4, + "num_turns": 40, + "total_cost_usd": 2.5379866600000005, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4062273, + "cache_creation_tokens": 118410, + "models_used": [ + "us.anthropic.claude-opus-5-5" + ], + "run_started_at": "2026-10-14T19:25:09.628710Z", + "run_ended_at": "2026-10-14T19:36:03.021501Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 84, + "output_tokens": 45121, + "cache_read_tokens": 4062273, + "cache_write_tokens": 118410, + "total_tokens": 4225888, + "total_cost_usd": 2.5379866600000005, + "latency_seconds": 653.4, + "num_turns": 40, + "generation_tokens_per_sec": 69.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "skill-update-dir-keeps-install-path", + "model": "us.anthropic.claude-opus-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The strongest aspect is the reproducible nested-directory scenario plus acceptance criteria with exact positive and negative filesystem oracles. Its root-cause account matches the submitted update.go context, while the installer really targets filepath.Join(baseDir, skill.Name). ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The largest deficiency is scope expansion into download atomicity and remote-name mismatch without an independent task statement establishing those requirements. The related issue and the practical likelihood of an upstream name mismatch could not be independently verified." + }, + "lld": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 23, + "total": 91, + "notes": "The LLD is unusually well grounded in updateRun, installer.Install, installSkill, installedSkill.dir, discovery.Skill, safepaths, and the existing tests. Its parent-directory target, mismatch guard, cleanup removal, and fetch-before-write flow are implementation-ready and consistent with the installer's real target-selection behavior. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The main correctness blemish is its inconsistent explanation that Skill.Name is path-derived while the review also treats it as frontmatter-derived, plus an unsupported assumption that buffering every skill is always small. It explicitly acknowledges the remaining partial-write risk from local I/O failures, although the absent independent task statement limits verification of its expanded scope." + }, + "review": { + "completeness": 23, + "correctness": 20, + "specificity": 24, + "risk_awareness": 23, + "total": 90, + "notes": "The review's strongest contribution is finding three concrete design defects: base plus skill.Name can select a sibling, host-scanned skills also flatten, and file-by-file fetching can leave mixed versions. Each finding identifies a consequence and a specific revision, rather than merely approving the design. The largest flaw is overstating a remote publisher's ability to change a frontmatter-derived name, contradicting the LLD's own statement that discovery.Skill.Name comes from the path; the local-directory rename case remains valid. Repository evidence confirms the installer joins the selected base with skill.Name, but no independent task statement was available to validate every blocking priority. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/))" + }, + "testing": { + "completeness": 22, + "correctness": 20, + "specificity": 24, + "risk_awareness": 22, + "total": 88, + "notes": "The plan's strongest aspect is its concrete matrix across flat and two-level layouts, custom and agent scans, mismatch refusal, preserved local files, and second-blob failure, with meaningful path and content oracles. The proposed package paths and HTTP-focused test strategy align with the submitted repository structure, and the global lockfile path is supported by repository evidence. ([github.com](https://github.com/cli/cli/issues/13438), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-create-lock-directory-w/)) Its largest deficiency is omitting the repository-required make lint command, while shorthand such as 'install with --pin' and the non-recursive checksum glob make parts of the manual plan less directly executable. Tests were not executed, and the unavailable independent task statement prevents confirming whether all added scenarios are required." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the submitted design end to end: every update receives the current directory's parent, mismatched names fail per skill, destructive cleanup is removed, and downloads complete before filesystem writes begin. The hunks target existing installSkill and updateRun behavior, and the added tests exercise relocation, preservation, host scanning, mismatch refusal, and fetch failure; the underlying installer and lockfile call shapes match repository evidence. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/could-not-list-skill-files-w/), [errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/failed-to-install-skill-q-w/)) The largest remaining risk is acknowledged but unresolved atomicity for mkdir or write failures, and buffering all files broadens installer memory usage without an evidenced size bound. The local shell sandbox could not start, so the candidate's git apply --check claim and test compilation could not be independently verified." + } + }, + "task_score": 90.2, + "verdict": "This is a strong, repository-specific submission whose largest limitation is the lack of an independent task statement and executable local verification for its broadened failure-atomicity scope.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T21:01:29.329051Z", + "repo_ref": "v2.94.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 184374, + "cached_input_tokens": 105027, + "cache_write_input_tokens": 36788, + "output_tokens": 10990, + "reasoning_output_tokens": 9230 + }, + "duration_ms": 177835 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json new file mode 100644 index 00000000..a7a394f4 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/eval.json @@ -0,0 +1,64 @@ +{ + "task": "auth-clipboard-config-default", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 19, + "correctness": 14, + "specificity": 23, + "risk_awareness": 16, + "total": 72, + "notes": "The issue has unusually concrete acceptance criteria, precedence rules, exclusions, and compatibility intent. Its largest defect is specifying an unset default of disabled, while the post-fix official manual documents clipboard as enabled by default. ([cli.github.com](https://cli.github.com/manual/gh_config)) The linked upstream issue supports adding persistent configuration but does not establish the candidate's default, host scoping, or explicit-false requirements. ([github.com](https://github.com/cli/cli/issues/13153)) Because no independent task statement was supplied, those additional requirements remain unverifiable rather than inherently invalid." + }, + "lld": { + "completeness": 20, + "correctness": 14, + "specificity": 23, + "risk_awareness": 18, + "total": 75, + "notes": "The LLD is strongly grounded in named files and commits to a clear precedence flow, interface changes, mock wiring, compatibility behavior, and alternatives. Its central design diverges from the available fixed implementation: it chooses a host-scoped Clipboard(hostname) API plus parallel bool/change fields, whereas trunk uses a global Clipboard() lookup and a nullable flag value. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go)) It also selects disabled as the default despite the official post-fix configuration documentation stating enabled. ([cli.github.com](https://cli.github.com/manual/gh_config)) Exact patch applicability and the claim that cfg and ConfigMock are the only interface implementers could not be independently verified because local shell inspection was blocked." + }, + "review": { + "completeness": 18, + "correctness": 14, + "specificity": 21, + "risk_awareness": 18, + "total": 71, + "notes": "The review identifies concrete concerns including direct-option construction hazards, unavailable clipboard utilities, local clipboard exposure, generated-mock drift, and missing login wiring coverage. Its largest failure is approving the load-bearing default and host-scoped API without challenging their divergence from the fixed global, default-enabled behavior. ([cli.github.com](https://cli.github.com/manual/gh_config), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go)) The SRE conclusion that upgrades cause no behavior change is particularly questionable given that documented default. Generator parity is acknowledged honestly but remains unverified." + }, + "testing": { + "completeness": 19, + "correctness": 14, + "specificity": 22, + "risk_awareness": 18, + "total": 73, + "notes": "The plan supplies concrete setup, commands, expected output, precedence truth tables, invalid-value coverage, host cases, rollback checks, and manual end-to-end scenarios. Its default-config oracle expects no copying, contradicting the post-fix documented default of enabled and therefore validating the wrong central behavior. ([cli.github.com](https://cli.github.com/manual/gh_config)) Automated tests do not exercise the final loginRun CopyToClipboard argument, so they could pass with the login call site broken; only refresh has an injected AuthFlow oracle. The alternative go build ./cmd/gh prerequisite also does not produce the configured ./bin/gh path." + }, + "implementation": { + "completeness": 18, + "correctness": 12, + "specificity": 22, + "risk_awareness": 15, + "total": 67, + "notes": "The patch implements its own design across configuration, interface and mock wiring, both auth commands, help text, and focused tests. Its largest defect is implementing disabled-by-default, host-scoped behavior when the available fixed implementation uses a global nullable-flag helper and the official configuration surface says default enabled. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go), [cli.github.com](https://cli.github.com/manual/gh_config)) The diff is internally coherent and tests refresh precedence, but login's final wiring has no executable unit-test seam. Exact application, formatting, compilation, and generated-mock equivalence could not be verified locally; the summary accurately discloses that tests and regeneration were not completed." + } + }, + "task_score": 71.6, + "verdict": "The submission is detailed and largely coherent, but it implements and validates the wrong default and configuration scope relative to the available fixed repository behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T15:56:39.256450Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 164748, + "cached_input_tokens": 90052, + "cache_write_input_tokens": 30614, + "output_tokens": 12849, + "reasoning_output_tokens": 11004 + }, + "duration_ms": 209436 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json new file mode 100644 index 00000000..21308f92 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-clipboard-config-default/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "auth-clipboard-config-default", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "high", + "tags": [ + "auth", + "go", + "cli", + "config", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 114.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 112, + "output_tokens": 40851, + "cache_read_tokens": 5398047, + "cache_write_tokens": 119090, + "total_tokens": 5558100, + "total_cost_usd": 1.96467524, + "latency_seconds": 356.3, + "num_turns": 55, + "generation_tokens_per_sec": 114.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 112, + "output_tokens": 40851, + "latency_seconds": 356.3, + "num_turns": 55, + "total_cost_usd": 1.96467524, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 5398047, + "cache_creation_tokens": 119090, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:27:39.015881Z", + "run_ended_at": "2026-10-14T15:33:35.298227Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 112, + "output_tokens": 40851, + "cache_read_tokens": 5398047, + "cache_write_tokens": 119090, + "total_tokens": 5558100, + "total_cost_usd": 1.96467524, + "latency_seconds": 356.3, + "num_turns": 55, + "generation_tokens_per_sec": 114.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "auth-clipboard-config-default", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 19, + "correctness": 14, + "specificity": 23, + "risk_awareness": 16, + "total": 72, + "notes": "The issue has unusually concrete acceptance criteria, precedence rules, exclusions, and compatibility intent. Its largest defect is specifying an unset default of disabled, while the post-fix official manual documents clipboard as enabled by default. ([cli.github.com](https://cli.github.com/manual/gh_config)) The linked upstream issue supports adding persistent configuration but does not establish the candidate's default, host scoping, or explicit-false requirements. ([github.com](https://github.com/cli/cli/issues/13153)) Because no independent task statement was supplied, those additional requirements remain unverifiable rather than inherently invalid." + }, + "lld": { + "completeness": 20, + "correctness": 14, + "specificity": 23, + "risk_awareness": 18, + "total": 75, + "notes": "The LLD is strongly grounded in named files and commits to a clear precedence flow, interface changes, mock wiring, compatibility behavior, and alternatives. Its central design diverges from the available fixed implementation: it chooses a host-scoped Clipboard(hostname) API plus parallel bool/change fields, whereas trunk uses a global Clipboard() lookup and a nullable flag value. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go)) It also selects disabled as the default despite the official post-fix configuration documentation stating enabled. ([cli.github.com](https://cli.github.com/manual/gh_config)) Exact patch applicability and the claim that cfg and ConfigMock are the only interface implementers could not be independently verified because local shell inspection was blocked." + }, + "review": { + "completeness": 18, + "correctness": 14, + "specificity": 21, + "risk_awareness": 18, + "total": 71, + "notes": "The review identifies concrete concerns including direct-option construction hazards, unavailable clipboard utilities, local clipboard exposure, generated-mock drift, and missing login wiring coverage. Its largest failure is approving the load-bearing default and host-scoped API without challenging their divergence from the fixed global, default-enabled behavior. ([cli.github.com](https://cli.github.com/manual/gh_config), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go)) The SRE conclusion that upgrades cause no behavior change is particularly questionable given that documented default. Generator parity is acknowledged honestly but remains unverified." + }, + "testing": { + "completeness": 19, + "correctness": 14, + "specificity": 22, + "risk_awareness": 18, + "total": 73, + "notes": "The plan supplies concrete setup, commands, expected output, precedence truth tables, invalid-value coverage, host cases, rollback checks, and manual end-to-end scenarios. Its default-config oracle expects no copying, contradicting the post-fix documented default of enabled and therefore validating the wrong central behavior. ([cli.github.com](https://cli.github.com/manual/gh_config)) Automated tests do not exercise the final loginRun CopyToClipboard argument, so they could pass with the login call site broken; only refresh has an injected AuthFlow oracle. The alternative go build ./cmd/gh prerequisite also does not produce the configured ./bin/gh path." + }, + "implementation": { + "completeness": 18, + "correctness": 12, + "specificity": 22, + "risk_awareness": 15, + "total": 67, + "notes": "The patch implements its own design across configuration, interface and mock wiring, both auth commands, help text, and focused tests. Its largest defect is implementing disabled-by-default, host-scoped behavior when the available fixed implementation uses a global nullable-flag helper and the official configuration surface says default enabled. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go), [cli.github.com](https://cli.github.com/manual/gh_config)) The diff is internally coherent and tests refresh precedence, but login's final wiring has no executable unit-test seam. Exact application, formatting, compilation, and generated-mock equivalence could not be verified locally; the summary accurately discloses that tests and regeneration were not completed." + } + }, + "task_score": 71.6, + "verdict": "The submission is detailed and largely coherent, but it implements and validates the wrong default and configuration scope relative to the available fixed repository behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T15:56:39.256450Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 164748, + "cached_input_tokens": 90052, + "cache_write_input_tokens": 30614, + "output_tokens": 12849, + "reasoning_output_tokens": 11004 + }, + "duration_ms": 209436 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json new file mode 100644 index 00000000..7213d543 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/eval.json @@ -0,0 +1,64 @@ +{ + "task": "auth-refresh-hint-on-expired-token", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The issue clearly defines both affected message sites, token categories, host behavior, testable acceptance criteria, and focused non-goals. Its largest weakness is describing the source-plus-prefix heuristic as proof that a token was created by gh auth login, although a stored user-supplied gho_ token is indistinguishable. The core requirement agrees with issue #13066, and the named internal/ghcmd/cmd.go and auth status paths match the submitted repository contexts. No independent task statement establishes whether every added status behavior was mandatory, so that scope remains partly unverifiable." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The LLD is unusually concrete about real symbols and data flow, including HTTPError.RequestURL, NormalizeHostname, ActiveToken, buildEntry, and the rewritten TokenSource field. Its main correctness limitation is that IsLoginToken is only a conservative heuristic, while the prose repeatedly presents it as exact provenance; acknowledged unprefixed GHES tokens also miss the desired refresh hint. The active-account restriction, 401 gate, configuration fallback, environment-token handling, and request-host normalization address the important failure paths. Line estimates and the assertion that keyring access cannot newly prompt were not independently verifiable." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 22, + "total": 86, + "notes": "The review finds three material design defects: inactive-account refresh suggestions, PAT misclassification by source alone, and applying the status change to non-401 failures. It ties each finding to concrete implementation choices such as opts.active, the gho_ prefix, opts.tokenSource, and HTTPError.StatusCode. Its largest gap is limited challenge of the remaining provenance heuristic and the absence of an end-to-end assertion around Main's actual 401 output. The security discussion is specific, although the claim that a repeated keyring lookup cannot prompt again was not verified." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan supplies concrete commands and expected commands for GitHub.com, GHES, tenancy hosts, environment tokens, PATs, nil inputs, inactive accounts, and non-401 failures. The strongest oracle is the exact auth status stderr output, supplemented by a realistic revoke-refresh-retry manual flow. The largest gap is that Test_reauthCommand tests only the selector and does not prove Main prints the selected command after an actual 401; the JSON byte-identity claim also lacks a named automated test. It ends with go vet rather than the repository-required make lint and provides limited operational checking beyond the manual flow." + }, + "implementation": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 84, + "notes": "The patch implements both message sites, centralizes the source-plus-prefix predicate, normalizes the failed request host, preserves login fallbacks, and restricts auth status refresh advice to active accounts receiving 401. The submitted contexts and symbols are consistent with internal/ghcmd/cmd.go, authEntry/buildEntry, shared helpers, and the repository's table-driven testing conventions; the feature also corresponds to merged PR #13068. The largest defect is that a gho_ token supplied manually into gh-managed storage is classified as login-created, while older unprefixed GHES login tokens are classified oppositely. The patch does not directly test Main's formatted 401 output, and its claimed reverse apply check could not be independently reproduced in the available execution environment." + } + }, + "task_score": 85.2, + "verdict": "This is a strong, repository-grounded submission whose largest limitation is treating token source plus gho_ prefix as exact login-token provenance and not testing the top-level 401 output end to end.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T15:59:32.766505Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 167997, + "cached_input_tokens": 101911, + "cache_write_input_tokens": 27518, + "output_tokens": 10242, + "reasoning_output_tokens": 8779 + }, + "duration_ms": 173497 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json new file mode 100644 index 00000000..be9ccde1 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/auth-refresh-hint-on-expired-token/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "auth-refresh-hint-on-expired-token", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "auth", + "go", + "cli", + "error-messages", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 116.3, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 86, + "output_tokens": 36237, + "cache_read_tokens": 4019013, + "cache_write_tokens": 112084, + "total_tokens": 4167420, + "total_cost_usd": 1.5912100599999999, + "latency_seconds": 311.7, + "num_turns": 42, + "generation_tokens_per_sec": 116.3, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 86, + "output_tokens": 36237, + "latency_seconds": 311.7, + "num_turns": 42, + "total_cost_usd": 1.5912100599999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 4019013, + "cache_creation_tokens": 112084, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:14:36.689858Z", + "run_ended_at": "2026-10-14T15:19:48.422045Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 86, + "output_tokens": 36237, + "cache_read_tokens": 4019013, + "cache_write_tokens": 112084, + "total_tokens": 4167420, + "total_cost_usd": 1.5912100599999999, + "latency_seconds": 311.7, + "num_turns": 42, + "generation_tokens_per_sec": 116.3, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "auth-refresh-hint-on-expired-token", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The issue clearly defines both affected message sites, token categories, host behavior, testable acceptance criteria, and focused non-goals. Its largest weakness is describing the source-plus-prefix heuristic as proof that a token was created by gh auth login, although a stored user-supplied gho_ token is indistinguishable. The core requirement agrees with issue #13066, and the named internal/ghcmd/cmd.go and auth status paths match the submitted repository contexts. No independent task statement establishes whether every added status behavior was mandatory, so that scope remains partly unverifiable." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The LLD is unusually concrete about real symbols and data flow, including HTTPError.RequestURL, NormalizeHostname, ActiveToken, buildEntry, and the rewritten TokenSource field. Its main correctness limitation is that IsLoginToken is only a conservative heuristic, while the prose repeatedly presents it as exact provenance; acknowledged unprefixed GHES tokens also miss the desired refresh hint. The active-account restriction, 401 gate, configuration fallback, environment-token handling, and request-host normalization address the important failure paths. Line estimates and the assertion that keyring access cannot newly prompt were not independently verifiable." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 22, + "total": 86, + "notes": "The review finds three material design defects: inactive-account refresh suggestions, PAT misclassification by source alone, and applying the status change to non-401 failures. It ties each finding to concrete implementation choices such as opts.active, the gho_ prefix, opts.tokenSource, and HTTPError.StatusCode. Its largest gap is limited challenge of the remaining provenance heuristic and the absence of an end-to-end assertion around Main's actual 401 output. The security discussion is specific, although the claim that a repeated keyring lookup cannot prompt again was not verified." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan supplies concrete commands and expected commands for GitHub.com, GHES, tenancy hosts, environment tokens, PATs, nil inputs, inactive accounts, and non-401 failures. The strongest oracle is the exact auth status stderr output, supplemented by a realistic revoke-refresh-retry manual flow. The largest gap is that Test_reauthCommand tests only the selector and does not prove Main prints the selected command after an actual 401; the JSON byte-identity claim also lacks a named automated test. It ends with go vet rather than the repository-required make lint and provides limited operational checking beyond the manual flow." + }, + "implementation": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 84, + "notes": "The patch implements both message sites, centralizes the source-plus-prefix predicate, normalizes the failed request host, preserves login fallbacks, and restricts auth status refresh advice to active accounts receiving 401. The submitted contexts and symbols are consistent with internal/ghcmd/cmd.go, authEntry/buildEntry, shared helpers, and the repository's table-driven testing conventions; the feature also corresponds to merged PR #13068. The largest defect is that a gho_ token supplied manually into gh-managed storage is classified as login-created, while older unprefixed GHES login tokens are classified oppositely. The patch does not directly test Main's formatted 401 output, and its claimed reverse apply check could not be independently reproduced in the available execution environment." + } + }, + "task_score": 85.2, + "verdict": "This is a strong, repository-grounded submission whose largest limitation is treating token source plus gho_ prefix as exact login-token provenance and not testing the top-level 401 output end to end.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T15:59:32.766505Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 167997, + "cached_input_tokens": 101911, + "cache_write_input_tokens": 27518, + "output_tokens": 10242, + "reasoning_output_tokens": 8779 + }, + "duration_ms": 173497 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json new file mode 100644 index 00000000..71e80ca5 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/eval.json @@ -0,0 +1,64 @@ +{ + "task": "codespace-ports-forward-concurrent-map", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible command, observable failures, scoped non-goals, and testable acceptance criteria matching the reported bug. The public issue and merged fix confirm the errgroup concurrency, shared connection, shallow struct copy, and unsafe TunnelManager access. ([github.com](https://github.com/cli/cli/issues/13399), [github.com](https://github.com/cli/cli/pull/13313)) Its largest weakness is that several criteria, especially manager encapsulation and the ten-port threshold, prescribe this solution despite the absence of an independent task statement. The additional URL, tunnel-slice, and notification-channel root-cause claims were plausible but not all independently corroborated by the repository evidence available." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 81, + "notes": "The LLD is unusually concrete about real files, ownership, pointer sharing, wrapper signatures, lock scope, tests, and alternatives; the pointer and manager-mutex choices agree with the merged repository fix. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest defect is releasing managerMu between GetTunnelPort and DeleteTunnelPort in UpdatePortVisibility, leaving a logical time-of-check/time-of-use race that the actual merged change specifically corrected by locking the complete get/check/delete sequence. ([github.com](https://github.com/cli/cli/pull/13313)) The waitMu tradeoff is acknowledged, but it expands scope and can ignore cancellation while queued or block healthy ports behind one missing notification. Rollout and compatibility treatment are otherwise proportionate for an internal locking change with no migration." + }, + "review": { + "completeness": 19, + "correctness": 17, + "specificity": 21, + "risk_awareness": 18, + "total": 75, + "notes": "The review catches concrete issues including copied mutexes, unprotected manager call sites, the concurrent mock map, and wait serialization head-of-line blocking. Its largest omission is the UpdatePortVisibility TOCTOU defect later identified and fixed in the real pull request, so the claim that all blocking findings were resolved is too strong. ([github.com](https://github.com/cli/cli/pull/13313)) The statement that repository CI already runs go test -race was not established by the supplied repository instructions, which instead prescribe go test ./... and make lint. The frontend and security sections are mostly ceremonial, while the backend and SRE sections provide the substantive value." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 22, + "risk_awareness": 17, + "total": 78, + "notes": "The plan gives valid package-level race commands, exact existing test names, a meaningful oracle that all ten ports are listed, and regression coverage for single-port behavior. This closely matches the merged fix's concurrent ten-port regression-test strategy. ([github.com](https://github.com/cli/cli/pull/13313)) The manual E2E loop curls only port 8080 while claiming every local port accepts connections, and an arbitrary HTTP status is a weak forwarding oracle. WaitForForwardedPort remains untested, and there is no coverage for queued-context cancellation or the concurrent visibility-update TOCTOU." + }, + "implementation": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 17, + "total": 77, + "notes": "The patch implements the central fix end to end: forwarders retain a shared connection pointer, TunnelManager operations are mutex-protected, and a ten-port concurrent regression test is added; these targets and symbols agree with the repository's merged solution. ([github.com](https://github.com/cli/cli/pull/13313)) Encapsulating manager calls in connection methods is coherent, and the mock lock removes a test-only race. The largest defect is that UpdatePortVisibility performs its get/check/delete sequence through separately locked calls, preserving the TOCTOU race that the merged implementation explicitly fixed atomically. ([github.com](https://github.com/cli/cli/pull/13313)) The additional wait mutex has no automated test and introduces acknowledged blocking behavior; local sandbox failure also prevented an independent apply or compile check, consistent with the summary's admission that the patch was not compiled." + } + }, + "task_score": 79.4, + "verdict": "Strong, repository-specific work that fixes the primary concurrent-map failure, but it misses an important atomicity race in visibility updates and overstates verification of the expanded wait-path change.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:03:49.368620Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 187488, + "cached_input_tokens": 72439, + "cache_write_input_tokens": 24969, + "output_tokens": 14313, + "reasoning_output_tokens": 12157 + }, + "duration_ms": 256589 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json new file mode 100644 index 00000000..8865be13 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/codespace-ports-forward-concurrent-map/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "codespace-ports-forward-concurrent-map", + "repo": "https://github.com/cli/cli", + "ref": "v2.95.0", + "complexity": "medium", + "tags": [ + "codespace", + "go", + "cli", + "concurrency", + "race-condition", + "fixed-in-2.96.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 123.9, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 33605, + "cache_read_tokens": 2363138, + "cache_write_tokens": 104471, + "total_tokens": 2501272, + "total_cost_usd": 1.1769682099999998, + "latency_seconds": 271.3, + "num_turns": 26, + "generation_tokens_per_sec": 123.9, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 58, + "output_tokens": 33605, + "latency_seconds": 271.3, + "num_turns": 26, + "total_cost_usd": 1.1769682099999998, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2363138, + "cache_creation_tokens": 104471, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:04:37.576461Z", + "run_ended_at": "2026-10-14T15:09:08.909157Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 33605, + "cache_read_tokens": 2363138, + "cache_write_tokens": 104471, + "total_tokens": 2501272, + "total_cost_usd": 1.1769682099999998, + "latency_seconds": 271.3, + "num_turns": 26, + "generation_tokens_per_sec": 123.9, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "codespace-ports-forward-concurrent-map", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible command, observable failures, scoped non-goals, and testable acceptance criteria matching the reported bug. The public issue and merged fix confirm the errgroup concurrency, shared connection, shallow struct copy, and unsafe TunnelManager access. ([github.com](https://github.com/cli/cli/issues/13399), [github.com](https://github.com/cli/cli/pull/13313)) Its largest weakness is that several criteria, especially manager encapsulation and the ten-port threshold, prescribe this solution despite the absence of an independent task statement. The additional URL, tunnel-slice, and notification-channel root-cause claims were plausible but not all independently corroborated by the repository evidence available." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 81, + "notes": "The LLD is unusually concrete about real files, ownership, pointer sharing, wrapper signatures, lock scope, tests, and alternatives; the pointer and manager-mutex choices agree with the merged repository fix. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest defect is releasing managerMu between GetTunnelPort and DeleteTunnelPort in UpdatePortVisibility, leaving a logical time-of-check/time-of-use race that the actual merged change specifically corrected by locking the complete get/check/delete sequence. ([github.com](https://github.com/cli/cli/pull/13313)) The waitMu tradeoff is acknowledged, but it expands scope and can ignore cancellation while queued or block healthy ports behind one missing notification. Rollout and compatibility treatment are otherwise proportionate for an internal locking change with no migration." + }, + "review": { + "completeness": 19, + "correctness": 17, + "specificity": 21, + "risk_awareness": 18, + "total": 75, + "notes": "The review catches concrete issues including copied mutexes, unprotected manager call sites, the concurrent mock map, and wait serialization head-of-line blocking. Its largest omission is the UpdatePortVisibility TOCTOU defect later identified and fixed in the real pull request, so the claim that all blocking findings were resolved is too strong. ([github.com](https://github.com/cli/cli/pull/13313)) The statement that repository CI already runs go test -race was not established by the supplied repository instructions, which instead prescribe go test ./... and make lint. The frontend and security sections are mostly ceremonial, while the backend and SRE sections provide the substantive value." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 22, + "risk_awareness": 17, + "total": 78, + "notes": "The plan gives valid package-level race commands, exact existing test names, a meaningful oracle that all ten ports are listed, and regression coverage for single-port behavior. This closely matches the merged fix's concurrent ten-port regression-test strategy. ([github.com](https://github.com/cli/cli/pull/13313)) The manual E2E loop curls only port 8080 while claiming every local port accepts connections, and an arbitrary HTTP status is a weak forwarding oracle. WaitForForwardedPort remains untested, and there is no coverage for queued-context cancellation or the concurrent visibility-update TOCTOU." + }, + "implementation": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 17, + "total": 77, + "notes": "The patch implements the central fix end to end: forwarders retain a shared connection pointer, TunnelManager operations are mutex-protected, and a ten-port concurrent regression test is added; these targets and symbols agree with the repository's merged solution. ([github.com](https://github.com/cli/cli/pull/13313)) Encapsulating manager calls in connection methods is coherent, and the mock lock removes a test-only race. The largest defect is that UpdatePortVisibility performs its get/check/delete sequence through separately locked calls, preserving the TOCTOU race that the merged implementation explicitly fixed atomically. ([github.com](https://github.com/cli/cli/pull/13313)) The additional wait mutex has no automated test and introduces acknowledged blocking behavior; local sandbox failure also prevented an independent apply or compile check, consistent with the summary's admission that the patch was not compiled." + } + }, + "task_score": 79.4, + "verdict": "Strong, repository-specific work that fixes the primary concurrent-map failure, but it misses an important atomicity race in visibility updates and overstates verification of the expanded wait-path change.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:03:49.368620Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 187488, + "cached_input_tokens": 72439, + "cache_write_input_tokens": 24969, + "output_tokens": 14313, + "reasoning_output_tokens": 12157 + }, + "duration_ms": 256589 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json new file mode 100644 index 00000000..b5158368 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/eval.json @@ -0,0 +1,64 @@ +{ + "task": "copilot-not-installed-trailing-newline", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 23, + "specificity": 25, + "risk_awareness": 22, + "total": 94, + "notes": "The issue precisely defines the terminal defect, reproduction path, unchanged behavior, and exact stderr oracle. Its acceptance criteria distinguish stdout, stderr, SilentError, download, and execution behavior, while explicitly excluding the non-interactive branch. The cited call site is consistent with the submitted patch at pkg/cmd/copilot/copilot.go. The independent task statement, issue reference, and repository labels were not independently verifiable." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to a minimal data flow and names concrete integration points including runCopilot, CanPrompt, PrompterMock, and the package-level test seams. It correctly identifies the TTY setup needed to reach the interactive decline branch and documents the sibling non-interactive message as deferred. Its principal gap is that the proposed test neither checks Confirm's defaultValue nor directly instruments the download path despite claiming unchanged prompt and download behavior. Local source inspection was blocked by the execution sandbox, so line numbers and API signatures could not be independently confirmed." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 84, + "notes": "The review's strongest finding is the concrete warning that missing TTY overrides would test the wrong branch, followed by actionable call-count and execution guards. It also identifies the sibling non-interactive message and reasonably treats changing it as a compatibility question rather than silently expanding scope. However, it overstates that the planned test pins absence of download and misses that defaultValue is ignored by the Confirm mock. Most reviewer sections are appropriately brief for this trivial change but contribute little independent stress testing." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The plan provides an exact package test command, precise stdout and stderr oracles, error identity, prompt count, and a PTY-based manual check. Those checks directly target the reported regression and preserve the non-interactive branch as an explicit non-goal. The claim that TestDownloadCopilot covers answering yes is not demonstrated as an end-to-end runCopilot branch test, and no direct download guard or Confirm default assertion is specified. It also omits the repository-required pre-commit commands go test ./... and make lint." + }, + "implementation": { + "completeness": 23, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the requested behavior end to end by adding the newline at runCopilot's decline return and adding an exact-output unit test with TTY and execution guards. The diff is narrowly scoped to pkg/cmd/copilot/copilot.go and copilot_test.go and follows the repository's documented global-function restoration pattern. Its largest weakness is that the test ignores Confirm's defaultValue and does not directly guard the download function, so some unchanged-behavior claims are stronger than the test oracle. Local source comparison and git apply --check could not be completed because the read-only shell failed during sandbox initialization, leaving patch applicability independently unverified." + } + }, + "task_score": 87.0, + "verdict": "This is a strong, narrowly targeted submission whose main limitation is incomplete verification of unchanged prompt and download behavior, compounded by unavailable local patch validation.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:06:11.645240Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 111836, + "cached_input_tokens": 57397, + "cache_write_input_tokens": 19929, + "output_tokens": 8614, + "reasoning_output_tokens": 6878 + }, + "duration_ms": 142263 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json new file mode 100644 index 00000000..833284a3 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/copilot-not-installed-trailing-newline/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "copilot-not-installed-trailing-newline", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "trivial", + "tags": [ + "copilot", + "go", + "cli", + "terminal-output", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 136.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 26, + "output_tokens": 10397, + "cache_read_tokens": 629335, + "cache_write_tokens": 41961, + "total_tokens": 681719, + "total_cost_usd": 0.3682706499999999, + "latency_seconds": 76.1, + "num_turns": 12, + "generation_tokens_per_sec": 136.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 26, + "output_tokens": 10397, + "latency_seconds": 76.1, + "num_turns": 12, + "total_cost_usd": 0.3682706499999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 629335, + "cache_creation_tokens": 41961, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:43:37.868545Z", + "run_ended_at": "2026-10-14T14:44:53.987378Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 26, + "output_tokens": 10397, + "cache_read_tokens": 629335, + "cache_write_tokens": 41961, + "total_tokens": 681719, + "total_cost_usd": 0.3682706499999999, + "latency_seconds": 76.1, + "num_turns": 12, + "generation_tokens_per_sec": 136.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "copilot-not-installed-trailing-newline", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 23, + "specificity": 25, + "risk_awareness": 22, + "total": 94, + "notes": "The issue precisely defines the terminal defect, reproduction path, unchanged behavior, and exact stderr oracle. Its acceptance criteria distinguish stdout, stderr, SilentError, download, and execution behavior, while explicitly excluding the non-interactive branch. The cited call site is consistent with the submitted patch at pkg/cmd/copilot/copilot.go. The independent task statement, issue reference, and repository labels were not independently verifiable." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to a minimal data flow and names concrete integration points including runCopilot, CanPrompt, PrompterMock, and the package-level test seams. It correctly identifies the TTY setup needed to reach the interactive decline branch and documents the sibling non-interactive message as deferred. Its principal gap is that the proposed test neither checks Confirm's defaultValue nor directly instruments the download path despite claiming unchanged prompt and download behavior. Local source inspection was blocked by the execution sandbox, so line numbers and API signatures could not be independently confirmed." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 84, + "notes": "The review's strongest finding is the concrete warning that missing TTY overrides would test the wrong branch, followed by actionable call-count and execution guards. It also identifies the sibling non-interactive message and reasonably treats changing it as a compatibility question rather than silently expanding scope. However, it overstates that the planned test pins absence of download and misses that defaultValue is ignored by the Confirm mock. Most reviewer sections are appropriately brief for this trivial change but contribute little independent stress testing." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The plan provides an exact package test command, precise stdout and stderr oracles, error identity, prompt count, and a PTY-based manual check. Those checks directly target the reported regression and preserve the non-interactive branch as an explicit non-goal. The claim that TestDownloadCopilot covers answering yes is not demonstrated as an end-to-end runCopilot branch test, and no direct download guard or Confirm default assertion is specified. It also omits the repository-required pre-commit commands go test ./... and make lint." + }, + "implementation": { + "completeness": 23, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the requested behavior end to end by adding the newline at runCopilot's decline return and adding an exact-output unit test with TTY and execution guards. The diff is narrowly scoped to pkg/cmd/copilot/copilot.go and copilot_test.go and follows the repository's documented global-function restoration pattern. Its largest weakness is that the test ignores Confirm's defaultValue and does not directly guard the download function, so some unchanged-behavior claims are stronger than the test oracle. Local source comparison and git apply --check could not be completed because the read-only shell failed during sandbox initialization, leaving patch applicability independently unverified." + } + }, + "task_score": 87.0, + "verdict": "This is a strong, narrowly targeted submission whose main limitation is incomplete verification of unchanged prompt and download behavior, compounded by unavailable local patch validation.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:06:11.645240Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 111836, + "cached_input_tokens": 57397, + "cache_write_input_tokens": 19929, + "output_tokens": 8614, + "reasoning_output_tokens": 6878 + }, + "duration_ms": 142263 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json new file mode 100644 index 00000000..3e448158 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/eval.json @@ -0,0 +1,64 @@ +{ + "task": "extension-uninstall-alias", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly defines alias behavior, legacy compatibility, help visibility, parent-alias behavior, tests, and explicit non-goals. Its proposed `Aliases` change matches the submitted `pkg/cmd/extension/command.go` context and gives directly testable acceptance criteria. The largest evidence gap is the absent independent task statement, so the related issue and exact intended help surface could not be independently established. It addresses the main compatibility and discoverability risks, though explicit error-parity and collision criteria would improve it." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The design is unusually concrete for this small change, naming `NewCmdExtension`, `normalizeExtensionSelector`, `ExtensionManager.Remove`, the relevant test table, help rendering, and acceptance coverage. Its one-field data flow, alternatives, compatibility rationale, and rollback plan leave no major implementation decision unresolved. Claims about `checkValidExtension`, generated documentation, and exact help internals could not be independently inspected because the repository shell failed to initialize, while telemetry is appropriately labeled as an inference. The design still treats the realistic risks proportionately, especially parent-help visibility, command-name collision, and preservation of `remove`." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 85, + "notes": "The strongest finding is the concrete distinction between testing Cobra's standalone default help and the production custom help renderer. It also examines parent-help discoverability, extension-name collision, telemetry attribution, and legacy acceptance coverage rather than merely approving the proposal. Its largest weakness is that every reviewer approves and it misses actionable test-quality concerns such as indexing `calls[0]` after a non-fatal length assertion and lacking an automated `gh ext uninstall` check. Given the trivial change, these omissions are limited rather than architectural." + }, + "testing": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The plan supplies concrete manager-call, normalization, TTY-output, non-TTY-output, help, argument-count, and backward-compatibility oracles using repository-appropriate Go test commands. It correctly retains existing `remove` acceptance coverage and adds manual verification of the production help renderer. Its largest defect is the unverified `cli/gh-test` dependency and an unisolated manual workflow that installs and removes an extension using real user state and network access. It also lacks automated parent-alias and manager-error cases, and its help assertion checks only substring presence rather than the exact aliases section." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch directly implements the design by adding `Aliases: []string{\"uninstall\"}` to the existing `remove` command and tests shared removal behavior, normalization, output, and help visibility. The resulting production manual currently lists `gh extension uninstall`, `gh extensions uninstall`, and `gh ext uninstall`, corroborating the intended Cobra and parent-alias behavior. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The main implementation weakness is test robustness: the call-count checks use non-fatal assertions before indexing, the help oracle is broad, and the parent alias is not tested directly. The summary accurately describes the diff and non-execution, but local `git apply --check` and source-context verification remained unavailable because the read-only shell sandbox could not initialize." + } + }, + "task_score": 87.0, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is a brittle, externally dependent, and unisolated manual testing workflow.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:08:25.769800Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 135359, + "cached_input_tokens": 82479, + "cache_write_input_tokens": 24359, + "output_tokens": 8649, + "reasoning_output_tokens": 6992 + }, + "duration_ms": 134112 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json new file mode 100644 index 00000000..ffb1c8dc --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/extension-uninstall-alias/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "extension-uninstall-alias", + "repo": "https://github.com/cli/cli", + "ref": "v2.93.0", + "complexity": "trivial", + "tags": [ + "extension", + "go", + "cli", + "cobra", + "ux", + "fixed-in-2.94.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 127.1, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 30, + "output_tokens": 15149, + "cache_read_tokens": 765431, + "cache_write_tokens": 46433, + "total_tokens": 827043, + "total_cost_usd": 0.46279057, + "latency_seconds": 119.2, + "num_turns": 14, + "generation_tokens_per_sec": 127.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 30, + "output_tokens": 15149, + "latency_seconds": 119.2, + "num_turns": 14, + "total_cost_usd": 0.46279057, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 765431, + "cache_creation_tokens": 46433, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:47:27.614204Z", + "run_ended_at": "2026-10-14T14:49:26.863115Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 30, + "output_tokens": 15149, + "cache_read_tokens": 765431, + "cache_write_tokens": 46433, + "total_tokens": 827043, + "total_cost_usd": 0.46279057, + "latency_seconds": 119.2, + "num_turns": 14, + "generation_tokens_per_sec": 127.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "extension-uninstall-alias", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly defines alias behavior, legacy compatibility, help visibility, parent-alias behavior, tests, and explicit non-goals. Its proposed `Aliases` change matches the submitted `pkg/cmd/extension/command.go` context and gives directly testable acceptance criteria. The largest evidence gap is the absent independent task statement, so the related issue and exact intended help surface could not be independently established. It addresses the main compatibility and discoverability risks, though explicit error-parity and collision criteria would improve it." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The design is unusually concrete for this small change, naming `NewCmdExtension`, `normalizeExtensionSelector`, `ExtensionManager.Remove`, the relevant test table, help rendering, and acceptance coverage. Its one-field data flow, alternatives, compatibility rationale, and rollback plan leave no major implementation decision unresolved. Claims about `checkValidExtension`, generated documentation, and exact help internals could not be independently inspected because the repository shell failed to initialize, while telemetry is appropriately labeled as an inference. The design still treats the realistic risks proportionately, especially parent-help visibility, command-name collision, and preservation of `remove`." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 85, + "notes": "The strongest finding is the concrete distinction between testing Cobra's standalone default help and the production custom help renderer. It also examines parent-help discoverability, extension-name collision, telemetry attribution, and legacy acceptance coverage rather than merely approving the proposal. Its largest weakness is that every reviewer approves and it misses actionable test-quality concerns such as indexing `calls[0]` after a non-fatal length assertion and lacking an automated `gh ext uninstall` check. Given the trivial change, these omissions are limited rather than architectural." + }, + "testing": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The plan supplies concrete manager-call, normalization, TTY-output, non-TTY-output, help, argument-count, and backward-compatibility oracles using repository-appropriate Go test commands. It correctly retains existing `remove` acceptance coverage and adds manual verification of the production help renderer. Its largest defect is the unverified `cli/gh-test` dependency and an unisolated manual workflow that installs and removes an extension using real user state and network access. It also lacks automated parent-alias and manager-error cases, and its help assertion checks only substring presence rather than the exact aliases section." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch directly implements the design by adding `Aliases: []string{\"uninstall\"}` to the existing `remove` command and tests shared removal behavior, normalization, output, and help visibility. The resulting production manual currently lists `gh extension uninstall`, `gh extensions uninstall`, and `gh ext uninstall`, corroborating the intended Cobra and parent-alias behavior. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The main implementation weakness is test robustness: the call-count checks use non-fatal assertions before indexing, the help oracle is broad, and the parent alias is not tested directly. The summary accurately describes the diff and non-execution, but local `git apply --check` and source-context verification remained unavailable because the read-only shell sandbox could not initialize." + } + }, + "task_score": 87.0, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is a brittle, externally dependent, and unisolated manual testing workflow.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:08:25.769800Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 135359, + "cached_input_tokens": 82479, + "cache_write_input_tokens": 24359, + "output_tokens": 8649, + "reasoning_output_tokens": 6992 + }, + "duration_ms": 134112 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json new file mode 100644 index 00000000..9847ae01 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/eval.json @@ -0,0 +1,64 @@ +{ + "task": "issue-view-comments-with-json", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The issue clearly defines the silent omission, affected commands, exclusions, and testable outcomes. Its examples use documented --comments and --json options and distinguish all relevant flag combinations. ([cli.github.com](https://cli.github.com/manual/gh_issue_view)) The largest concern is that it preserves --comments with --json comments, while the published v2.99.0 fix is described simply as rejecting --comments with --json. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Because no independent task statement was supplied, that compatibility exception cannot be confirmed as required." + }, + "lld": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The LLD names concrete files, symbols, control flow, error type, tests, alternatives, and rollout behavior. Its use of RunE validation, Exporter.Fields, and FlagErrorf is consistent with the repository's documented command conventions. ([github.com](https://github.com/cli/cli/blob/trunk/docs/command-development.md)) The main deficiency is committing to a field-sensitive shared helper rather than establishing why that policy should differ from the released unconditional rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Direct source inspection was impeded by the local shell sandbox, so several detailed baseline claims could not be independently verified beyond the supplied patch context." + }, + "review": { + "completeness": 19, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 76, + "notes": "The review finds two concrete design defects: breaking a previously accepted combination and duplicating validation across three commands. Byte and Sage provide actionable resolutions that are reflected consistently in the revised LLD. Its largest gap is that it never challenges whether the compatibility exception or expansion to three commands matches the actual requested behavior, despite the release describing an unqualified rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The remaining role-based approvals add little risk analysis beyond confirming that no network, secret, or deployment surface is introduced." + }, + "testing": { + "completeness": 18, + "correctness": 15, + "specificity": 20, + "risk_awareness": 17, + "total": 70, + "notes": "The plan identifies exact package tests, command combinations, exit status, error text, and the expected comments key. Its largest correctness defect is exporting GH_TOKEN=dummy while expecting accepted commands that perform API requests to succeed, so those manual checks cannot establish the stated outcome. The plan also lacks an explicit oracle proving rejected combinations make no HTTP request, and only spells out accepted smoke commands for issue view. Claims that tests and smoke checks passed were not independently verifiable." + }, + "implementation": { + "completeness": 21, + "correctness": 18, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The patch implements its own design end to end by adding one typed flag-error helper, wiring all three NewCmdView RunE paths, and adding focused table tests. The target paths and symbols are consistent with the supplied baseline context, and the v2.99.0 release confirms that this command area received the corresponding fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The main behavioral risk is permitting --comments --json comments when the published fix and post-fix user reports describe the pair as rejected outright. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) The environment prevented git apply --check and test execution, so applicability and the summary's verification claims remain unconfirmed." + } + }, + "task_score": 78.2, + "verdict": "The submission is coherent and implementation-oriented, but its field-sensitive exception diverges from the published fixed behavior, and the testing plan cannot validate accepted paths with a dummy token. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733))", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:11:43.598890Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 153778, + "cached_input_tokens": 88048, + "cache_write_input_tokens": 24437, + "output_tokens": 10553, + "reasoning_output_tokens": 9217 + }, + "duration_ms": 197816 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json new file mode 100644 index 00000000..2d6014f2 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/issue-view-comments-with-json/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "issue-view-comments-with-json", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "medium", + "tags": [ + "issue", + "go", + "cli", + "json-output", + "flags", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 117.0, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 46, + "output_tokens": 18650, + "cache_read_tokens": 1517987, + "cache_write_tokens": 65937, + "total_tokens": 1602620, + "total_cost_usd": 0.7205350899999999, + "latency_seconds": 159.4, + "num_turns": 22, + "generation_tokens_per_sec": 117.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 46, + "output_tokens": 18650, + "latency_seconds": 159.4, + "num_turns": 22, + "total_cost_usd": 0.7205350899999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1517987, + "cache_creation_tokens": 65937, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:09:12.304425Z", + "run_ended_at": "2026-10-14T15:11:51.666363Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 46, + "output_tokens": 18650, + "cache_read_tokens": 1517987, + "cache_write_tokens": 65937, + "total_tokens": 1602620, + "total_cost_usd": 0.7205350899999999, + "latency_seconds": 159.4, + "num_turns": 22, + "generation_tokens_per_sec": 117.0, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "issue-view-comments-with-json", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The issue clearly defines the silent omission, affected commands, exclusions, and testable outcomes. Its examples use documented --comments and --json options and distinguish all relevant flag combinations. ([cli.github.com](https://cli.github.com/manual/gh_issue_view)) The largest concern is that it preserves --comments with --json comments, while the published v2.99.0 fix is described simply as rejecting --comments with --json. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Because no independent task statement was supplied, that compatibility exception cannot be confirmed as required." + }, + "lld": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The LLD names concrete files, symbols, control flow, error type, tests, alternatives, and rollout behavior. Its use of RunE validation, Exporter.Fields, and FlagErrorf is consistent with the repository's documented command conventions. ([github.com](https://github.com/cli/cli/blob/trunk/docs/command-development.md)) The main deficiency is committing to a field-sensitive shared helper rather than establishing why that policy should differ from the released unconditional rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Direct source inspection was impeded by the local shell sandbox, so several detailed baseline claims could not be independently verified beyond the supplied patch context." + }, + "review": { + "completeness": 19, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 76, + "notes": "The review finds two concrete design defects: breaking a previously accepted combination and duplicating validation across three commands. Byte and Sage provide actionable resolutions that are reflected consistently in the revised LLD. Its largest gap is that it never challenges whether the compatibility exception or expansion to three commands matches the actual requested behavior, despite the release describing an unqualified rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The remaining role-based approvals add little risk analysis beyond confirming that no network, secret, or deployment surface is introduced." + }, + "testing": { + "completeness": 18, + "correctness": 15, + "specificity": 20, + "risk_awareness": 17, + "total": 70, + "notes": "The plan identifies exact package tests, command combinations, exit status, error text, and the expected comments key. Its largest correctness defect is exporting GH_TOKEN=dummy while expecting accepted commands that perform API requests to succeed, so those manual checks cannot establish the stated outcome. The plan also lacks an explicit oracle proving rejected combinations make no HTTP request, and only spells out accepted smoke commands for issue view. Claims that tests and smoke checks passed were not independently verifiable." + }, + "implementation": { + "completeness": 21, + "correctness": 18, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The patch implements its own design end to end by adding one typed flag-error helper, wiring all three NewCmdView RunE paths, and adding focused table tests. The target paths and symbols are consistent with the supplied baseline context, and the v2.99.0 release confirms that this command area received the corresponding fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The main behavioral risk is permitting --comments --json comments when the published fix and post-fix user reports describe the pair as rejected outright. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) The environment prevented git apply --check and test execution, so applicability and the summary's verification claims remain unconfirmed." + } + }, + "task_score": 78.2, + "verdict": "The submission is coherent and implementation-oriented, but its field-sensitive exception diverges from the published fixed behavior, and the testing plan cannot validate accepted paths with a dummy token. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733))", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:11:43.598890Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 153778, + "cached_input_tokens": 88048, + "cache_write_input_tokens": 24437, + "output_tokens": 10553, + "reasoning_output_tokens": 9217 + }, + "duration_ms": 197816 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json new file mode 100644 index 00000000..14bc58ad --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-checks-all-cancelled-summary", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue precisely identifies the omitted `counts.Canceled` guard and unreachable cancelled-summary branch, matching the merged upstream diagnosis. ([github.com](https://github.com/cli/cli/pull/13679)) Its acceptance criteria and non-goals concretely cover TTY output, regression behavior, and exit status. The largest weakness is the unsupported claim about a second blank line and the absence of reproducible before-and-after command output. No independent task statement was supplied, so requirements beyond the task identifier and repository behavior remain unverifiable." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD names the relevant files, counter flow, guard change, fixture, expected output, and unchanged exit semantics with implementation-ready precision. Its central decision matches the upstream fix released in v2.96.0. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) The largest gap is selecting only a full-command regression case rather than directly exercising every `printSummary` branch, which upstream review explicitly requested. ([github.com](https://github.com/cli/cli/pull/13679)) Local execution was unavailable, so the asserted watch-path and exit-flow details could not be independently run." + }, + "review": { + "completeness": 19, + "correctness": 22, + "specificity": 20, + "risk_awareness": 20, + "total": 81, + "notes": "The review's strongest contribution is checking exit behavior, watch-path impact, non-TTY behavior, and fixture isolation rather than merely approving the one-line change. Its core assessment is consistent with the upstream diagnosis and minimal fix. ([github.com](https://github.com/cli/cli/pull/13679)) The largest deficiency is that most reviewer sections contain generic approvals and it misses the concrete recommendation to table-test every summary branch raised during the real review. ([github.com](https://github.com/cli/cli/pull/13679)) The assertion that cancelled non-TTY rows use failure status is not demonstrated by a referenced test or code excerpt." + }, + "testing": { + "completeness": 18, + "correctness": 22, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan gives an exact subtest command, concrete output oracle, empty-error expectation, and package regression run that would catch the reported defect. The cancelled-only oracle agrees with the upstream fix and its regression test. ([github.com](https://github.com/cli/cli/pull/13679)) Its largest deficiency is omitting the repository-required `go test ./...` and `make lint`, while also lacking direct coverage of all `printSummary` branches requested upstream. ([github.com](https://github.com/cli/cli/pull/13679)) The manual scenario has no deterministic setup, and non-TTY preservation is stated rather than assigned a specific assertion." + }, + "implementation": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The patch implements the exact load-bearing fix by adding `counts.Canceled` to the existing guard, matching the merged upstream change. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) The fixture timestamps support the asserted `1m26s` row, and the table case pins the summary, tally, row rendering, and successful return behavior end to end. The main limitation is that the new test covers only the cancelled branch through comparatively brittle full-output integration, whereas upstream ultimately added a direct table test for every summary path. ([github.com](https://github.com/cli/cli/pull/13679)) The summary's reverse-apply verification wording is confusing, and local `git apply --check` could not be executed due the read-only shell sandbox, but no functional deviation is apparent from the verified target context." + } + }, + "task_score": 87.6, + "verdict": "This is a strong, repository-grounded solution whose largest limitation is underdeveloped review and validation coverage, particularly the missing full-suite/lint checks and direct testing of every summary branch.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:15:16.003845Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 154805, + "cached_input_tokens": 89953, + "cache_write_input_tokens": 23889, + "output_tokens": 12925, + "reasoning_output_tokens": 11321 + }, + "duration_ms": 212391 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json new file mode 100644 index 00000000..9d855b30 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-checks-all-cancelled-summary/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-checks-all-cancelled-summary", + "repo": "https://github.com/cli/cli", + "ref": "v2.95.0", + "complexity": "trivial", + "tags": [ + "pr", + "go", + "cli", + "checks", + "terminal-output", + "fixed-in-2.96.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 128.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 20, + "output_tokens": 8370, + "cache_read_tokens": 462807, + "cache_write_tokens": 38773, + "total_tokens": 509970, + "total_cost_usd": 0.30055729000000003, + "latency_seconds": 65.1, + "num_turns": 9, + "generation_tokens_per_sec": 128.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 20, + "output_tokens": 8370, + "latency_seconds": 65.1, + "num_turns": 9, + "total_cost_usd": 0.30055729000000003, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 462807, + "cache_creation_tokens": 38773, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:46:19.136527Z", + "run_ended_at": "2026-10-14T14:47:24.244894Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 20, + "output_tokens": 8370, + "cache_read_tokens": 462807, + "cache_write_tokens": 38773, + "total_tokens": 509970, + "total_cost_usd": 0.30055729000000003, + "latency_seconds": 65.1, + "num_turns": 9, + "generation_tokens_per_sec": 128.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-checks-all-cancelled-summary", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue precisely identifies the omitted `counts.Canceled` guard and unreachable cancelled-summary branch, matching the merged upstream diagnosis. ([github.com](https://github.com/cli/cli/pull/13679)) Its acceptance criteria and non-goals concretely cover TTY output, regression behavior, and exit status. The largest weakness is the unsupported claim about a second blank line and the absence of reproducible before-and-after command output. No independent task statement was supplied, so requirements beyond the task identifier and repository behavior remain unverifiable." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD names the relevant files, counter flow, guard change, fixture, expected output, and unchanged exit semantics with implementation-ready precision. Its central decision matches the upstream fix released in v2.96.0. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) The largest gap is selecting only a full-command regression case rather than directly exercising every `printSummary` branch, which upstream review explicitly requested. ([github.com](https://github.com/cli/cli/pull/13679)) Local execution was unavailable, so the asserted watch-path and exit-flow details could not be independently run." + }, + "review": { + "completeness": 19, + "correctness": 22, + "specificity": 20, + "risk_awareness": 20, + "total": 81, + "notes": "The review's strongest contribution is checking exit behavior, watch-path impact, non-TTY behavior, and fixture isolation rather than merely approving the one-line change. Its core assessment is consistent with the upstream diagnosis and minimal fix. ([github.com](https://github.com/cli/cli/pull/13679)) The largest deficiency is that most reviewer sections contain generic approvals and it misses the concrete recommendation to table-test every summary branch raised during the real review. ([github.com](https://github.com/cli/cli/pull/13679)) The assertion that cancelled non-TTY rows use failure status is not demonstrated by a referenced test or code excerpt." + }, + "testing": { + "completeness": 18, + "correctness": 22, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan gives an exact subtest command, concrete output oracle, empty-error expectation, and package regression run that would catch the reported defect. The cancelled-only oracle agrees with the upstream fix and its regression test. ([github.com](https://github.com/cli/cli/pull/13679)) Its largest deficiency is omitting the repository-required `go test ./...` and `make lint`, while also lacking direct coverage of all `printSummary` branches requested upstream. ([github.com](https://github.com/cli/cli/pull/13679)) The manual scenario has no deterministic setup, and non-TTY preservation is stated rather than assigned a specific assertion." + }, + "implementation": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The patch implements the exact load-bearing fix by adding `counts.Canceled` to the existing guard, matching the merged upstream change. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) The fixture timestamps support the asserted `1m26s` row, and the table case pins the summary, tally, row rendering, and successful return behavior end to end. The main limitation is that the new test covers only the cancelled branch through comparatively brittle full-output integration, whereas upstream ultimately added a direct table test for every summary path. ([github.com](https://github.com/cli/cli/pull/13679)) The summary's reverse-apply verification wording is confusing, and local `git apply --check` could not be executed due the read-only shell sandbox, but no functional deviation is apparent from the verified target context." + } + }, + "task_score": 87.6, + "verdict": "This is a strong, repository-grounded solution whose largest limitation is underdeveloped review and validation coverage, particularly the missing full-suite/lint checks and direct testing of every summary branch.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:15:16.003845Z", + "repo_ref": "v2.95.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 154805, + "cached_input_tokens": 89953, + "cache_write_input_tokens": 23889, + "output_tokens": 12925, + "reasoning_output_tokens": 11321 + }, + "duration_ms": 212391 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json new file mode 100644 index 00000000..5f2038e0 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-comment-attach-ghu-tokens", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The issue crisply defines the client-side failure, unchanged token classes, exclusions, and testable success and failure criteria. Repository issue history corroborates that uploadTokenTypes is the credential choke point and previously permits OAuth, classic PAT, and fine-grained PAT tokens. ([github.com](https://github.com/cli/cli/issues/14309)) The largest evidence gap is the absent independent task statement: the asserted ghu_ 201 response, Bearer 404 response, and Copilot-session motivation could not be independently verified. Risk coverage is otherwise strong because it explicitly preserves ghs_, ghr_, unknown-token rejection, authorization scheme, endpoint, and GHES behavior." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The LLD is implementation-ready, naming the relevant files, symbols, call flow, exact allowlist change, error behavior, and focused tests. Its AddAuthTokenHeader claim matches repository evidence showing the transport formats credentials as Authorization: token . ([github.com](https://github.com/cli/cli/issues/13317)) The main deficiency is that the endpoint's ghu_ behavior and the assertion that every listed command traverses this exact path were not independently verifiable from the supplied task or accessible checkout. It appropriately preserves the allowlist, considers alternative policies, retains server-derived errors, and supplies a proportionate rollback plan." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 20, + "total": 81, + "notes": "The strongest review work identifies three concrete regression guards: invert the existing ghu_ rejection row, assert the token authorization scheme, and exercise a server 404 through the comment path. The authorization concern is technically grounded because the repository transport explicitly sets the token scheme. ([github.com](https://github.com/cli/cli/issues/13317)) Several reviewer sections are largely ceremonial, and the review does not challenge the unverified premise that ghu_ acceptance is uniform or independently inspect all six claimed callers. The cross-host credential statement is also asserted without demonstrated repository evidence, although it does not undermine the recommended patch." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 16, + "total": 76, + "notes": "The plan provides unusually concrete rows, request counts, body values, exact error text, authorization-header oracles, negative token cases, and affected-package commands. Its endpoint shape agrees with existing attachment investigations, which use the same uploads host, query parameters, and 404 semantics. ([github.com](https://github.com/cli/cli/issues/14309)) The largest deficiency is operational: manual checks invoke gh rather than the required built binary and would create live assets/comments without naming an authorized disposable repository or cleanup procedure. The checked execution claims have no attached evidence, and the plan omits the repository's required full go fix, go test ./..., and make lint pre-commit gates." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the proposed behavior end to end: it adds only TokenTypeUserToServer to the allowlist, retains rejected-token rows, proves the token scheme, and covers both successful upload/comment creation and server-derived 404 behavior. The production change fits repository evidence that this stack uses a credential allowlist and Authorization: token headers, while first-party attachment support spans the six commands claimed. ([github.com](https://github.com/cli/cli/issues/14309), [github.com](https://github.com/cli/cli/issues/13317), [github.com](https://github.com/drogers0/gh-image/blob/main/README.md)) No material code-level defect is apparent in the submitted hunks, but the local shell failure prevented an independent git apply --check, compilation, or verification of the claimed test and gofmt results. The summary is accurate overall, though “no follow-ups” overstates completion when the real-token E2E check remains explicitly unrun and the endpoint premise is externally unverified." + } + }, + "task_score": 83.6, + "verdict": "This is a strong, narrowly scoped submission with a convincing implementation, limited chiefly by the absent independent task statement, unverifiable endpoint premise, and an operationally incomplete live-testing plan.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:18:46.226665Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 205161, + "cached_input_tokens": 105146, + "cache_write_input_tokens": 23846, + "output_tokens": 11442, + "reasoning_output_tokens": 10019 + }, + "duration_ms": 210209 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json new file mode 100644 index 00000000..41201378 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-comment-attach-ghu-tokens/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-comment-attach-ghu-tokens", + "repo": "https://github.com/cli/cli", + "ref": "v2.101.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "auth", + "attachments", + "fixed-in-2.102.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 120.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 32, + "output_tokens": 13950, + "cache_read_tokens": 866553, + "cache_write_tokens": 54734, + "total_tokens": 935269, + "total_cost_usd": 0.49468056000000005, + "latency_seconds": 115.7, + "num_turns": 15, + "generation_tokens_per_sec": 120.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 32, + "output_tokens": 13950, + "latency_seconds": 115.7, + "num_turns": 15, + "total_cost_usd": 0.49468056000000005, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 866553, + "cache_creation_tokens": 54734, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:58:02.471705Z", + "run_ended_at": "2026-10-14T14:59:58.202242Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 32, + "output_tokens": 13950, + "cache_read_tokens": 866553, + "cache_write_tokens": 54734, + "total_tokens": 935269, + "total_cost_usd": 0.49468056000000005, + "latency_seconds": 115.7, + "num_turns": 15, + "generation_tokens_per_sec": 120.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-comment-attach-ghu-tokens", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The issue crisply defines the client-side failure, unchanged token classes, exclusions, and testable success and failure criteria. Repository issue history corroborates that uploadTokenTypes is the credential choke point and previously permits OAuth, classic PAT, and fine-grained PAT tokens. ([github.com](https://github.com/cli/cli/issues/14309)) The largest evidence gap is the absent independent task statement: the asserted ghu_ 201 response, Bearer 404 response, and Copilot-session motivation could not be independently verified. Risk coverage is otherwise strong because it explicitly preserves ghs_, ghr_, unknown-token rejection, authorization scheme, endpoint, and GHES behavior." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The LLD is implementation-ready, naming the relevant files, symbols, call flow, exact allowlist change, error behavior, and focused tests. Its AddAuthTokenHeader claim matches repository evidence showing the transport formats credentials as Authorization: token . ([github.com](https://github.com/cli/cli/issues/13317)) The main deficiency is that the endpoint's ghu_ behavior and the assertion that every listed command traverses this exact path were not independently verifiable from the supplied task or accessible checkout. It appropriately preserves the allowlist, considers alternative policies, retains server-derived errors, and supplies a proportionate rollback plan." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 20, + "total": 81, + "notes": "The strongest review work identifies three concrete regression guards: invert the existing ghu_ rejection row, assert the token authorization scheme, and exercise a server 404 through the comment path. The authorization concern is technically grounded because the repository transport explicitly sets the token scheme. ([github.com](https://github.com/cli/cli/issues/13317)) Several reviewer sections are largely ceremonial, and the review does not challenge the unverified premise that ghu_ acceptance is uniform or independently inspect all six claimed callers. The cross-host credential statement is also asserted without demonstrated repository evidence, although it does not undermine the recommended patch." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 16, + "total": 76, + "notes": "The plan provides unusually concrete rows, request counts, body values, exact error text, authorization-header oracles, negative token cases, and affected-package commands. Its endpoint shape agrees with existing attachment investigations, which use the same uploads host, query parameters, and 404 semantics. ([github.com](https://github.com/cli/cli/issues/14309)) The largest deficiency is operational: manual checks invoke gh rather than the required built binary and would create live assets/comments without naming an authorized disposable repository or cleanup procedure. The checked execution claims have no attached evidence, and the plan omits the repository's required full go fix, go test ./..., and make lint pre-commit gates." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the proposed behavior end to end: it adds only TokenTypeUserToServer to the allowlist, retains rejected-token rows, proves the token scheme, and covers both successful upload/comment creation and server-derived 404 behavior. The production change fits repository evidence that this stack uses a credential allowlist and Authorization: token headers, while first-party attachment support spans the six commands claimed. ([github.com](https://github.com/cli/cli/issues/14309), [github.com](https://github.com/cli/cli/issues/13317), [github.com](https://github.com/drogers0/gh-image/blob/main/README.md)) No material code-level defect is apparent in the submitted hunks, but the local shell failure prevented an independent git apply --check, compilation, or verification of the claimed test and gofmt results. The summary is accurate overall, though “no follow-ups” overstates completion when the real-token E2E check remains explicitly unrun and the endpoint premise is externally unverified." + } + }, + "task_score": 83.6, + "verdict": "This is a strong, narrowly scoped submission with a convincing implementation, limited chiefly by the absent independent task statement, unverifiable endpoint premise, and an operationally incomplete live-testing plan.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:18:46.226665Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 205161, + "cached_input_tokens": 105146, + "cache_write_input_tokens": 23846, + "output_tokens": 11442, + "reasoning_output_tokens": 10019 + }, + "duration_ms": 210209 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json new file mode 100644 index 00000000..6c141595 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-merge-delete-branch-owner-case", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 20, + "total": 89, + "notes": "The issue provides reproducible setup, observable behavior, explicit exclusions, and acceptance criteria tied to a concrete DELETE request. Its root-cause claim matches v2.100.0 evidence showing the case-sensitive crossRepoPR comparison, omitted isCrossRepository field, and deletion guard. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The main weakness is limited discussion of output-mode differences and existing deletion-error behavior. No independent task statement was supplied, so broader intended scope could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD commits to exact files, data flow, helper semantics, compatibility behavior, and rollout, making the change implementation-ready. Repository evidence supports the baseline query and comparison, the existing API field, and mock finder behavior described in the design. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that the two-source decision is justified mainly through conservative behavior and test compatibility without requiring direct coverage of conflicting or absent values. Exact local line references and claimed test outcomes could not be independently rerun." + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 20, + "risk_awareness": 18, + "total": 76, + "notes": "The review meaningfully examines deletion safety, prompt behavior, the dual-source decision, and the consequences of widening a destructive action. Its observations are consistent with the verified early-return behavior and available isCrossRepository field. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The largest deficiency is its approval-oriented conclusion despite identifying that merge tests do not assert requested fields; it should have required coverage for the API-true branch or field selection. The claimed absence of operational concerns is reasonable but was not independently validated on GHES." + }, + "testing": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The plan specifies exact package commands, fixtures, HTTP path, output oracle, fork regression coverage, compatibility tests, and a realistic manual workflow. The core oracle directly targets the verified failure mechanism: whether the guarded remote DELETE occurs. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its main gap is no automated assertion that isCrossRepository is requested and no case where that flag alone prevents deletion despite matching owners. The named tests and claimed executions could not be locally verified because the provided shell sandbox failed before command execution." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the LLD end to end by requesting isCrossRepository, replacing the faulty comparison with EqualFold-backed logic, and adding a regression test whose registered DELETE and output assertion pin the reported behavior. Its merge.go hunk matches verified v2.100.0 source evidence, and the repository release history confirms this owner-casing defect was fixed in v2.101.0. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) The largest weakness is that the test exercises only the EqualFold path, not the newly added API-flag condition or query-field selection. Patch application and the summary's test, vet, and formatting claims could not be independently rerun due the shell sandbox failure." + } + }, + "task_score": 86.6, + "verdict": "This is a strong, repository-grounded submission with a correct focused implementation, limited chiefly by an overly approving review and missing direct tests of the new API-field path.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:21:09.983505Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 150494, + "cached_input_tokens": 67080, + "cache_write_input_tokens": 24890, + "output_tokens": 8871, + "reasoning_output_tokens": 7108 + }, + "duration_ms": 143744 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json new file mode 100644 index 00000000..f878aec8 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-merge-delete-branch-owner-case/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-merge-delete-branch-owner-case", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "git", + "remotes", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 123.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 36, + "output_tokens": 20623, + "cache_read_tokens": 1052421, + "cache_write_tokens": 60130, + "total_tokens": 1133210, + "total_cost_usd": 0.62382232, + "latency_seconds": 166.7, + "num_turns": 17, + "generation_tokens_per_sec": 123.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 36, + "output_tokens": 20623, + "latency_seconds": 166.7, + "num_turns": 17, + "total_cost_usd": 0.62382232, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1052421, + "cache_creation_tokens": 60130, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:49:30.303243Z", + "run_ended_at": "2026-10-14T14:52:16.966480Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 36, + "output_tokens": 20623, + "cache_read_tokens": 1052421, + "cache_write_tokens": 60130, + "total_tokens": 1133210, + "total_cost_usd": 0.62382232, + "latency_seconds": 166.7, + "num_turns": 17, + "generation_tokens_per_sec": 123.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-merge-delete-branch-owner-case", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 20, + "total": 89, + "notes": "The issue provides reproducible setup, observable behavior, explicit exclusions, and acceptance criteria tied to a concrete DELETE request. Its root-cause claim matches v2.100.0 evidence showing the case-sensitive crossRepoPR comparison, omitted isCrossRepository field, and deletion guard. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The main weakness is limited discussion of output-mode differences and existing deletion-error behavior. No independent task statement was supplied, so broader intended scope could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD commits to exact files, data flow, helper semantics, compatibility behavior, and rollout, making the change implementation-ready. Repository evidence supports the baseline query and comparison, the existing API field, and mock finder behavior described in the design. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that the two-source decision is justified mainly through conservative behavior and test compatibility without requiring direct coverage of conflicting or absent values. Exact local line references and claimed test outcomes could not be independently rerun." + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 20, + "risk_awareness": 18, + "total": 76, + "notes": "The review meaningfully examines deletion safety, prompt behavior, the dual-source decision, and the consequences of widening a destructive action. Its observations are consistent with the verified early-return behavior and available isCrossRepository field. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The largest deficiency is its approval-oriented conclusion despite identifying that merge tests do not assert requested fields; it should have required coverage for the API-true branch or field selection. The claimed absence of operational concerns is reasonable but was not independently validated on GHES." + }, + "testing": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The plan specifies exact package commands, fixtures, HTTP path, output oracle, fork regression coverage, compatibility tests, and a realistic manual workflow. The core oracle directly targets the verified failure mechanism: whether the guarded remote DELETE occurs. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its main gap is no automated assertion that isCrossRepository is requested and no case where that flag alone prevents deletion despite matching owners. The named tests and claimed executions could not be locally verified because the provided shell sandbox failed before command execution." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the LLD end to end by requesting isCrossRepository, replacing the faulty comparison with EqualFold-backed logic, and adding a regression test whose registered DELETE and output assertion pin the reported behavior. Its merge.go hunk matches verified v2.100.0 source evidence, and the repository release history confirms this owner-casing defect was fixed in v2.101.0. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) The largest weakness is that the test exercises only the EqualFold path, not the newly added API-flag condition or query-field selection. Patch application and the summary's test, vet, and formatting claims could not be independently rerun due the shell sandbox failure." + } + }, + "task_score": 86.6, + "verdict": "This is a strong, repository-grounded submission with a correct focused implementation, limited chiefly by an overly approving review and missing direct tests of the new API-field path.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:21:09.983505Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 150494, + "cached_input_tokens": 67080, + "cache_write_input_tokens": 24890, + "output_tokens": 8871, + "reasoning_output_tokens": 7108 + }, + "duration_ms": 143744 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json new file mode 100644 index 00000000..abf728b0 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/eval.json @@ -0,0 +1,64 @@ +{ + "task": "pr-view-json-number-on-issue", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 20, + "total": 90, + "notes": "The issue clearly identifies the false-positive exit status, its scripting impact, and the field-dependent behavior. Its explanation matches `finder.Find` and the `findByNumber` path in `pkg/cmd/pr/shared/finder.go`. The acceptance criteria are concrete, although \"same behavior\" for `#N` and URL selectors does not explicitly enumerate both positive and negative cases. No independent task statement was supplied, so the related issue and intended scope could not be independently confirmed." + }, + "lld": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is implementation-ready, naming the correct finder branch, query helper, tests, callers, and data flow. Public source corroborates that URL parsing establishes `baseRefRepo` and `prNumber`, and that positive numbers flow through `findByNumber`. It appropriately discusses the extra API request, changed failure point, rollout, and alternatives. Its exact GitHub error text, complete caller inventory, and rate-limit implications were not fully verifiable, and it does not require the repository-wide test and lint commands." + }, + "review": { + "completeness": 19, + "correctness": 20, + "specificity": 20, + "risk_awareness": 18, + "total": 77, + "notes": "The review identifies concrete blast-radius concerns, including number-only callers, URL selection, merge refs, API cost, and the obsolete no-HTTP test. The `pr diff` number-only finder usage is consistent with repository source. ([github.com](https://github.com/malor/gh-cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) Its largest weakness is insufficient challenge: it misses the absent exact-error assertion, incomplete automated selector coverage, and repository-wide lint and test requirements. It also incorrectly says the release-note recommendation is reflected in `testing.md`, where no such checklist item appears." + }, + "testing": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 19, + "total": 80, + "notes": "The plan provides concrete commands and meaningful stdout, stderr, JSON, and exit-status oracles for both PR and issue cases. It covers direct numbers, `#N`, URLs, other field sets, finder unit tests, and affected command packages. The E2E snippet uses an undefined `$N`, while the automated negative test only requires some error rather than the promised specific not-found error. It also omits `go test ./...` and `make lint`, and the hard-coded live issue and PR identities could not be independently verified." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The patch implements the design directly by removing the unvalidated number-only return and routing every positive `prNumber` through `findByNumber`; current upstream source reflects that same control flow. The target files, symbols, and surrounding context are consistent with the submitted baseline, and the added tests cover successful lookup, GraphQL rejection, and URL repository selection. The largest gap is that tests assert only `wantErr`, omit automated `#N` and URL-negative cases, and therefore do not pin the exact accepted error behavior. The claimed test execution and clean local application could not be independently verified because the read-only command sandbox failed before executing commands." + } + }, + "task_score": 85.0, + "verdict": "This is a strong, repository-grounded minimal fix whose main limitation is that the review and tests do not fully enforce the promised error and selector behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:24:23.454512Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 163048, + "cached_input_tokens": 83789, + "cache_write_input_tokens": 25174, + "output_tokens": 12090, + "reasoning_output_tokens": 10336 + }, + "duration_ms": 193458 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json new file mode 100644 index 00000000..f20d1a00 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/pr-view-json-number-on-issue/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "pr-view-json-number-on-issue", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "low", + "tags": [ + "pr", + "go", + "cli", + "json-output", + "api", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 131.1, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 28, + "output_tokens": 13896, + "cache_read_tokens": 762586, + "cache_write_tokens": 52441, + "total_tokens": 828951, + "total_cost_usd": 0.46489927, + "latency_seconds": 106.0, + "num_turns": 13, + "generation_tokens_per_sec": 131.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 28, + "output_tokens": 13896, + "latency_seconds": 106.0, + "num_turns": 13, + "total_cost_usd": 0.46489927, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 762586, + "cache_creation_tokens": 52441, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:52:20.451469Z", + "run_ended_at": "2026-10-14T14:54:06.422436Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 28, + "output_tokens": 13896, + "cache_read_tokens": 762586, + "cache_write_tokens": 52441, + "total_tokens": 828951, + "total_cost_usd": 0.46489927, + "latency_seconds": 106.0, + "num_turns": 13, + "generation_tokens_per_sec": 131.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "pr-view-json-number-on-issue", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 20, + "total": 90, + "notes": "The issue clearly identifies the false-positive exit status, its scripting impact, and the field-dependent behavior. Its explanation matches `finder.Find` and the `findByNumber` path in `pkg/cmd/pr/shared/finder.go`. The acceptance criteria are concrete, although \"same behavior\" for `#N` and URL selectors does not explicitly enumerate both positive and negative cases. No independent task statement was supplied, so the related issue and intended scope could not be independently confirmed." + }, + "lld": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is implementation-ready, naming the correct finder branch, query helper, tests, callers, and data flow. Public source corroborates that URL parsing establishes `baseRefRepo` and `prNumber`, and that positive numbers flow through `findByNumber`. It appropriately discusses the extra API request, changed failure point, rollout, and alternatives. Its exact GitHub error text, complete caller inventory, and rate-limit implications were not fully verifiable, and it does not require the repository-wide test and lint commands." + }, + "review": { + "completeness": 19, + "correctness": 20, + "specificity": 20, + "risk_awareness": 18, + "total": 77, + "notes": "The review identifies concrete blast-radius concerns, including number-only callers, URL selection, merge refs, API cost, and the obsolete no-HTTP test. The `pr diff` number-only finder usage is consistent with repository source. ([github.com](https://github.com/malor/gh-cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) Its largest weakness is insufficient challenge: it misses the absent exact-error assertion, incomplete automated selector coverage, and repository-wide lint and test requirements. It also incorrectly says the release-note recommendation is reflected in `testing.md`, where no such checklist item appears." + }, + "testing": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 19, + "total": 80, + "notes": "The plan provides concrete commands and meaningful stdout, stderr, JSON, and exit-status oracles for both PR and issue cases. It covers direct numbers, `#N`, URLs, other field sets, finder unit tests, and affected command packages. The E2E snippet uses an undefined `$N`, while the automated negative test only requires some error rather than the promised specific not-found error. It also omits `go test ./...` and `make lint`, and the hard-coded live issue and PR identities could not be independently verified." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The patch implements the design directly by removing the unvalidated number-only return and routing every positive `prNumber` through `findByNumber`; current upstream source reflects that same control flow. The target files, symbols, and surrounding context are consistent with the submitted baseline, and the added tests cover successful lookup, GraphQL rejection, and URL repository selection. The largest gap is that tests assert only `wantErr`, omit automated `#N` and URL-negative cases, and therefore do not pin the exact accepted error behavior. The claimed test execution and clean local application could not be independently verified because the read-only command sandbox failed before executing commands." + } + }, + "task_score": 85.0, + "verdict": "This is a strong, repository-grounded minimal fix whose main limitation is that the review and tests do not fully enforce the promised error and selector behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:24:23.454512Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 163048, + "cached_input_tokens": 83789, + "cache_write_input_tokens": 25174, + "output_tokens": 12090, + "reasoning_output_tokens": 10336 + }, + "duration_ms": 193458 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json new file mode 100644 index 00000000..aae9af3d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/eval.json @@ -0,0 +1,64 @@ +{ + "task": "project-item-add-output-non-tty", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 18, + "total": 86, + "notes": "The issue provides byte-exact TTY, non-TTY, JSON, and error acceptance criteria plus clear exclusions. Its description correctly identifies `printResults` in `pkg/cmd/project/item-add/item_add.go` as the change point. The main weakness is claiming the output distinguishes a mutation from a no-op, although returning an item ID does not establish whether the item already existed. The cited 2-to-9-call statistic could not be verified, and no independent task statement was supplied, though the v2.98.0 release confirms this command's non-TTY output was fixed. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 19, + "total": 88, + "notes": "The LLD is grounded in concrete files, functions, data flow, exporter ordering, and the existing `ProjectItem.Id` field. Its proposed branch is implementation-ready and preserves TTY and formatted output without another API request. The largest gap is limited treatment of compatibility for scripts expecting empty stdout and the possibility of a successful response yielding an empty ID. Exact line estimates and the complete existing test inventory could not be independently verified." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 21, + "risk_awareness": 21, + "total": 83, + "notes": "The review identifies concrete concerns including empty IDs, changed script output, sibling-command inconsistency, and the need for distinct content and project-item IDs in testing. The distinct-ID recommendation directly improves regression detection. Its largest deficiency is approving without requiring a unit test for the error/no-confirmation criterion or noticing the repository rule to use `require` for error checks. It also dismisses empty stdout as not being a contract and assumes successful mutations always provide a nonempty ID without demonstrated evidence." + }, + "testing": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 18, + "total": 83, + "notes": "The plan supplies valid package-level commands and exact output oracles for TTY, non-TTY, JSON, and live item-ID verification. The non-TTY unit scenario deliberately separates `content ID` from `PVTI_item`, which would catch printing the wrong identifier. The negative case is only a vague manual instruction and does not concretely capture stdout or unit-test the required absence of confirmation. The checked claims that tests passed were not independently verifiable, and the live test has no cleanup or explicit handling for an already-present item." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the designed behavior directly in `printResults` and adds an end-to-end non-TTY test whose mutation returns `PVTI_item` while the source content uses `content ID`. The exporter and TTY branches remain unchanged, and the upstream v2.98.0 release independently corroborates that this command required a non-TTY output fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) The clearest repository-convention defect is `assert.NoError(t, err)` instead of the mandated `require.NoError`, and no automated negative-output test was added. Patch applicability and the claimed test execution could not be independently run, but the submitted hunk paths, symbols, and surrounding context are internally consistent with the baseline." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, repository-specific submission whose largest limitation is incomplete negative and compatibility testing alongside a few unverifiable claims caused by the missing independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:27:11.587271Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 159790, + "cached_input_tokens": 79635, + "cache_write_input_tokens": 20683, + "output_tokens": 10111, + "reasoning_output_tokens": 8709 + }, + "duration_ms": 168119 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json new file mode 100644 index 00000000..57d55ba8 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-add-output-non-tty/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "project-item-add-output-non-tty", + "repo": "https://github.com/cli/cli", + "ref": "v2.97.0", + "complexity": "low", + "tags": [ + "project", + "go", + "cli", + "tty", + "automation", + "fixed-in-2.98.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 129.4, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 28, + "output_tokens": 12458, + "cache_read_tokens": 735613, + "cache_write_tokens": 47103, + "total_tokens": 795202, + "total_cost_usd": 0.42846771, + "latency_seconds": 96.3, + "num_turns": 13, + "generation_tokens_per_sec": 129.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 28, + "output_tokens": 12458, + "latency_seconds": 96.3, + "num_turns": 13, + "total_cost_usd": 0.42846771, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 735613, + "cache_creation_tokens": 47103, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:54:09.950502Z", + "run_ended_at": "2026-10-14T14:55:46.250638Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 28, + "output_tokens": 12458, + "cache_read_tokens": 735613, + "cache_write_tokens": 47103, + "total_tokens": 795202, + "total_cost_usd": 0.42846771, + "latency_seconds": 96.3, + "num_turns": 13, + "generation_tokens_per_sec": 129.4, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "project-item-add-output-non-tty", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 18, + "total": 86, + "notes": "The issue provides byte-exact TTY, non-TTY, JSON, and error acceptance criteria plus clear exclusions. Its description correctly identifies `printResults` in `pkg/cmd/project/item-add/item_add.go` as the change point. The main weakness is claiming the output distinguishes a mutation from a no-op, although returning an item ID does not establish whether the item already existed. The cited 2-to-9-call statistic could not be verified, and no independent task statement was supplied, though the v2.98.0 release confirms this command's non-TTY output was fixed. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 19, + "total": 88, + "notes": "The LLD is grounded in concrete files, functions, data flow, exporter ordering, and the existing `ProjectItem.Id` field. Its proposed branch is implementation-ready and preserves TTY and formatted output without another API request. The largest gap is limited treatment of compatibility for scripts expecting empty stdout and the possibility of a successful response yielding an empty ID. Exact line estimates and the complete existing test inventory could not be independently verified." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 21, + "risk_awareness": 21, + "total": 83, + "notes": "The review identifies concrete concerns including empty IDs, changed script output, sibling-command inconsistency, and the need for distinct content and project-item IDs in testing. The distinct-ID recommendation directly improves regression detection. Its largest deficiency is approving without requiring a unit test for the error/no-confirmation criterion or noticing the repository rule to use `require` for error checks. It also dismisses empty stdout as not being a contract and assumes successful mutations always provide a nonempty ID without demonstrated evidence." + }, + "testing": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 18, + "total": 83, + "notes": "The plan supplies valid package-level commands and exact output oracles for TTY, non-TTY, JSON, and live item-ID verification. The non-TTY unit scenario deliberately separates `content ID` from `PVTI_item`, which would catch printing the wrong identifier. The negative case is only a vague manual instruction and does not concretely capture stdout or unit-test the required absence of confirmation. The checked claims that tests passed were not independently verifiable, and the live test has no cleanup or explicit handling for an already-present item." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the designed behavior directly in `printResults` and adds an end-to-end non-TTY test whose mutation returns `PVTI_item` while the source content uses `content ID`. The exporter and TTY branches remain unchanged, and the upstream v2.98.0 release independently corroborates that this command required a non-TTY output fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) The clearest repository-convention defect is `assert.NoError(t, err)` instead of the mandated `require.NoError`, and no automated negative-output test was added. Patch applicability and the claimed test execution could not be independently run, but the submitted hunk paths, symbols, and surrounding context are internally consistent with the baseline." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, repository-specific submission whose largest limitation is incomplete negative and compatibility testing alongside a few unverifiable claims caused by the missing independent task statement.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:27:11.587271Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 159790, + "cached_input_tokens": 79635, + "cache_write_input_tokens": 20683, + "output_tokens": 10111, + "reasoning_output_tokens": 8709 + }, + "duration_ms": 168119 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json new file mode 100644 index 00000000..ec775e6a --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/eval.json @@ -0,0 +1,64 @@ +{ + "task": "project-item-edit-by-name", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 19, + "specificity": 23, + "risk_awareness": 19, + "total": 83, + "notes": "The issue defines a concrete readable invocation, backward-compatibility requirements, testable errors, and useful non-goals. The official command documentation confirms the number/owner/URL/field flow, but describes `--value` specifically as a single-select option name, while this issue expands it to text, number, date, and iteration without independent task evidence. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) That unsupported scope expansion is its largest deficiency because it substantially enlarges API and compatibility risk. No independent task statement was supplied, and the claimed related issue and several API-limit assumptions could not be verified from the unavailable local execution environment." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 82, + "notes": "The LLD is unusually concrete about `item_edit.go`, shared query helpers, mutation reuse, lookup ordering, pagination, ambiguity handling, and failure-before-mutation behavior. Its principal weakness is designing all-type `--value` interpretation when the established command surface documents that flag for single-select values, creating unnecessary iteration queries and parsing semantics. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit)) It also retains contradictory text saying one page of 100 project items is assumed after resolving that issue with pagination, and it misses the silent `--id` plus `--title`/`--body` plus `--field`/`--value` combination. Exact application against the tagged tree and claims about pinned shared-query tests could not be verified because local read commands failed before execution." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 83, + "notes": "The review finds several real, consequential defects: project-item pagination, empty-value dispatch, misleading candidates, and silently ignored `--url` with draft-edit flags. Those findings are actionable and are visibly reflected in the submitted diff through pagination, empty-value validation, and new flag errors. Its largest miss is the analogous `--id DI_... --title ... --field ... --value ...` path, which still resolves a field and then silently executes only the draft update; it also never challenges the unsupported all-data-type expansion. Claims about shared query-test breakage and field-count limits remained unverifiable against the local checkout." + }, + "testing": { + "completeness": 21, + "correctness": 19, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan has strong concrete oracles for mutation payloads, case-insensitive matching, pagination, clear behavior, invalid values, candidate-list errors, and legacy ID paths. The manual setup builds `bin/gh` but invokes bare `gh`, so it can accidentally test an installed binary, and several multi-type checks merely say `item-list` shows the value without an exact query or expected representation. The automated tests omit the mixed draft-edit and name-based-field case that exposes silent flag loss, and they use `assert` for error checks despite the repository instruction to use `require`. The reported test execution and live GraphQL compatibility were not independently verifiable." + }, + "implementation": { + "completeness": 20, + "correctness": 17, + "specificity": 21, + "risk_awareness": 16, + "total": 74, + "notes": "The diff implements the submitted design end to end with new flags, owner/project lookup, field and option matching, iteration lookup, paginated item resolution, mutation reuse, and substantial focused tests. Its largest concrete defect is that validation rejects `--url` with `--title` or `--body` but permits an ID-addressed draft edit combined with `--field` and `--value`; after resolution, `runEditItem` takes the draft branch and silently ignores the requested field update. The patch also broadens `--value` beyond the officially documented single-select behavior, uses `assert` rather than required fatal assertions, and admits that the mandatory linter was not run. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit)) The patch's application, compilation, and claimed test results could not be independently checked because the local read-only command sandbox failed before running commands." + } + }, + "task_score": 80.4, + "verdict": "The submission is detailed and mostly implementation-ready, but it over-expands the intended value semantics and leaves a silent mixed draft/field flag path in the code.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:30:42.941982Z", + "repo_ref": "v2.96.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 228676, + "cached_input_tokens": 112742, + "cache_write_input_tokens": 40243, + "output_tokens": 13392, + "reasoning_output_tokens": 11459 + }, + "duration_ms": 211341 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json new file mode 100644 index 00000000..ced7fd72 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/project-item-edit-by-name/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "project-item-edit-by-name", + "repo": "https://github.com/cli/cli", + "ref": "v2.96.0", + "complexity": "high", + "tags": [ + "project", + "go", + "cli", + "graphql", + "fixed-in-2.97.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 122.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 80, + "output_tokens": 55448, + "cache_read_tokens": 3903393, + "cache_write_tokens": 119985, + "total_tokens": 4078906, + "total_cost_usd": 1.7988092100000002, + "latency_seconds": 451.5, + "num_turns": 37, + "generation_tokens_per_sec": 122.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 80, + "output_tokens": 55448, + "latency_seconds": 451.5, + "num_turns": 37, + "total_cost_usd": 1.7988092100000002, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 3903393, + "cache_creation_tokens": 119985, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:33:38.461814Z", + "run_ended_at": "2026-10-14T15:41:09.947559Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 80, + "output_tokens": 55448, + "cache_read_tokens": 3903393, + "cache_write_tokens": 119985, + "total_tokens": 4078906, + "total_cost_usd": 1.7988092100000002, + "latency_seconds": 451.5, + "num_turns": 37, + "generation_tokens_per_sec": 122.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "project-item-edit-by-name", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 19, + "specificity": 23, + "risk_awareness": 19, + "total": 83, + "notes": "The issue defines a concrete readable invocation, backward-compatibility requirements, testable errors, and useful non-goals. The official command documentation confirms the number/owner/URL/field flow, but describes `--value` specifically as a single-select option name, while this issue expands it to text, number, date, and iteration without independent task evidence. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) That unsupported scope expansion is its largest deficiency because it substantially enlarges API and compatibility risk. No independent task statement was supplied, and the claimed related issue and several API-limit assumptions could not be verified from the unavailable local execution environment." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 82, + "notes": "The LLD is unusually concrete about `item_edit.go`, shared query helpers, mutation reuse, lookup ordering, pagination, ambiguity handling, and failure-before-mutation behavior. Its principal weakness is designing all-type `--value` interpretation when the established command surface documents that flag for single-select values, creating unnecessary iteration queries and parsing semantics. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit)) It also retains contradictory text saying one page of 100 project items is assumed after resolving that issue with pagination, and it misses the silent `--id` plus `--title`/`--body` plus `--field`/`--value` combination. Exact application against the tagged tree and claims about pinned shared-query tests could not be verified because local read commands failed before execution." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 83, + "notes": "The review finds several real, consequential defects: project-item pagination, empty-value dispatch, misleading candidates, and silently ignored `--url` with draft-edit flags. Those findings are actionable and are visibly reflected in the submitted diff through pagination, empty-value validation, and new flag errors. Its largest miss is the analogous `--id DI_... --title ... --field ... --value ...` path, which still resolves a field and then silently executes only the draft update; it also never challenges the unsupported all-data-type expansion. Claims about shared query-test breakage and field-count limits remained unverifiable against the local checkout." + }, + "testing": { + "completeness": 21, + "correctness": 19, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan has strong concrete oracles for mutation payloads, case-insensitive matching, pagination, clear behavior, invalid values, candidate-list errors, and legacy ID paths. The manual setup builds `bin/gh` but invokes bare `gh`, so it can accidentally test an installed binary, and several multi-type checks merely say `item-list` shows the value without an exact query or expected representation. The automated tests omit the mixed draft-edit and name-based-field case that exposes silent flag loss, and they use `assert` for error checks despite the repository instruction to use `require`. The reported test execution and live GraphQL compatibility were not independently verifiable." + }, + "implementation": { + "completeness": 20, + "correctness": 17, + "specificity": 21, + "risk_awareness": 16, + "total": 74, + "notes": "The diff implements the submitted design end to end with new flags, owner/project lookup, field and option matching, iteration lookup, paginated item resolution, mutation reuse, and substantial focused tests. Its largest concrete defect is that validation rejects `--url` with `--title` or `--body` but permits an ID-addressed draft edit combined with `--field` and `--value`; after resolution, `runEditItem` takes the draft branch and silently ignores the requested field update. The patch also broadens `--value` beyond the officially documented single-select behavior, uses `assert` rather than required fatal assertions, and admits that the mandatory linter was not run. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit)) The patch's application, compilation, and claimed test results could not be independently checked because the local read-only command sandbox failed before running commands." + } + }, + "task_score": 80.4, + "verdict": "The submission is detailed and mostly implementation-ready, but it over-expands the intended value semantics and leaves a silent mixed draft/field flag path in the code.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:30:42.941982Z", + "repo_ref": "v2.96.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 228676, + "cached_input_tokens": 112742, + "cache_write_input_tokens": 40243, + "output_tokens": 13392, + "reasoning_output_tokens": 11459 + }, + "duration_ms": 211341 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json new file mode 100644 index 00000000..7bb5ba77 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/eval.json @@ -0,0 +1,64 @@ +{ + "task": "release-create-404-scope-blame", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the parsing defect, affected branch, desired behavior, exclusions, and testable acceptance criteria. GitHub CLI v2.98.0 independently lists the corresponding fix, \"Trim spaces when parsing X-Oauth-Scopes in gh release create,\" as PR #14065. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) Its largest weakness is claiming that preserving the HTTP 404 reveals the real cause, when a generic 404 cannot distinguish a nonexistent repository from insufficient access. No independent task statement was supplied, so requirements beyond the task identifier and internally consistent artifacts remain unverifiable." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The design commits to a minimal exact-match parser change, names concrete files and symbols, and specifies compatibility and negative cases. The selected behavior agrees with the fix documented in the v2.98.0 release notes. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) Its error-handling statement about \"non-workflow 404s\" is imprecise because the API response cannot establish the cause, and the true-missing-scope branch still deliberately masks the HTTP error. The stated baseline line locations, Go version, and test execution could not be independently verified from the local tree because the read-only command sandbox failed to initialize." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 21, + "risk_awareness": 20, + "total": 82, + "notes": "The review catches two concrete regression risks: substring false positives and tests that fail to reproduce comma-space headers. It also identifies the continuing masked-error behavior for genuinely missing scopes and records it as an explicit limitation. Several persona sections are superficial approvals, and the review does not challenge the environment-specific test path or the claimed execution checklist. Claims about surrounding shared parsers and repository-wide conventions could not be fully verified from the inaccessible local source." + }, + "testing": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 18, + "total": 82, + "notes": "The plan provides focused unit and command-level tests with concrete headers, exact Boolean outcomes, error text, stderr expectations, and backward-compatibility cases. Its package path and test names agree with the submitted patch, and the documented auth command does support removing scopes. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh)) The hard-coded clone directory is not portable, while the negative manual procedure mutates stored credentials without specifying restoration. The checklist marks functional testing complete despite explicitly saying the included live check was not run, and the claimed test results could not be independently executed." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the designed trim-plus-exact-match behavior and adds both helper-level edge cases and command-level regression coverage. The official v2.98.0 release record confirms that this precise defect was fixed in the cited release-create path. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) No substantive design deviation is apparent, but local source matching and git-apply validation could not be independently performed because command execution was unavailable. The summary also overstates verification slightly because `gofmt -l` without file arguments does not check the repository, although this does not undermine the patch logic." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, focused submission whose main limitation is unverifiable execution and source-application claims, compounded by a few imprecise statements about what the preserved 404 reveals.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:32:58.879243Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 119139, + "cached_input_tokens": 63335, + "cache_write_input_tokens": 23498, + "output_tokens": 9280, + "reasoning_output_tokens": 7684 + }, + "duration_ms": 135923 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json new file mode 100644 index 00000000..3a6868e9 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-create-404-scope-blame/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "release-create-404-scope-blame", + "repo": "https://github.com/cli/cli", + "ref": "v2.97.0", + "complexity": "low", + "tags": [ + "release", + "go", + "cli", + "auth", + "error-handling", + "fixed-in-2.98.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 121.9, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 36, + "output_tokens": 15801, + "cache_read_tokens": 915028, + "cache_write_tokens": 48933, + "total_tokens": 979798, + "total_cost_usd": 0.5097621099999999, + "latency_seconds": 129.6, + "num_turns": 16, + "generation_tokens_per_sec": 121.9, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 36, + "output_tokens": 15801, + "latency_seconds": 129.6, + "num_turns": 16, + "total_cost_usd": 0.5097621099999999, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 915028, + "cache_creation_tokens": 48933, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:55:49.473967Z", + "run_ended_at": "2026-10-14T14:57:59.129177Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 36, + "output_tokens": 15801, + "cache_read_tokens": 915028, + "cache_write_tokens": 48933, + "total_tokens": 979798, + "total_cost_usd": 0.5097621099999999, + "latency_seconds": 129.6, + "num_turns": 16, + "generation_tokens_per_sec": 121.9, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "release-create-404-scope-blame", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the parsing defect, affected branch, desired behavior, exclusions, and testable acceptance criteria. GitHub CLI v2.98.0 independently lists the corresponding fix, \"Trim spaces when parsing X-Oauth-Scopes in gh release create,\" as PR #14065. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) Its largest weakness is claiming that preserving the HTTP 404 reveals the real cause, when a generic 404 cannot distinguish a nonexistent repository from insufficient access. No independent task statement was supplied, so requirements beyond the task identifier and internally consistent artifacts remain unverifiable." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The design commits to a minimal exact-match parser change, names concrete files and symbols, and specifies compatibility and negative cases. The selected behavior agrees with the fix documented in the v2.98.0 release notes. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) Its error-handling statement about \"non-workflow 404s\" is imprecise because the API response cannot establish the cause, and the true-missing-scope branch still deliberately masks the HTTP error. The stated baseline line locations, Go version, and test execution could not be independently verified from the local tree because the read-only command sandbox failed to initialize." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 21, + "risk_awareness": 20, + "total": 82, + "notes": "The review catches two concrete regression risks: substring false positives and tests that fail to reproduce comma-space headers. It also identifies the continuing masked-error behavior for genuinely missing scopes and records it as an explicit limitation. Several persona sections are superficial approvals, and the review does not challenge the environment-specific test path or the claimed execution checklist. Claims about surrounding shared parsers and repository-wide conventions could not be fully verified from the inaccessible local source." + }, + "testing": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 18, + "total": 82, + "notes": "The plan provides focused unit and command-level tests with concrete headers, exact Boolean outcomes, error text, stderr expectations, and backward-compatibility cases. Its package path and test names agree with the submitted patch, and the documented auth command does support removing scopes. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh)) The hard-coded clone directory is not portable, while the negative manual procedure mutates stored credentials without specifying restoration. The checklist marks functional testing complete despite explicitly saying the included live check was not run, and the claimed test results could not be independently executed." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the designed trim-plus-exact-match behavior and adds both helper-level edge cases and command-level regression coverage. The official v2.98.0 release record confirms that this precise defect was fixed in the cited release-create path. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) No substantive design deviation is apparent, but local source matching and git-apply validation could not be independently performed because command execution was unavailable. The summary also overstates verification slightly because `gofmt -l` without file arguments does not check the repository, although this does not undermine the patch logic." + } + }, + "task_score": 85.6, + "verdict": "This is a strong, focused submission whose main limitation is unverifiable execution and source-application claims, compounded by a few imprecise statements about what the preserved 404 reveals.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:32:58.879243Z", + "repo_ref": "v2.97.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 119139, + "cached_input_tokens": 63335, + "cache_write_input_tokens": 23498, + "output_tokens": 9280, + "reasoning_output_tokens": 7684 + }, + "duration_ms": 135923 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json new file mode 100644 index 00000000..947d35ba --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/eval.json @@ -0,0 +1,64 @@ +{ + "task": "release-verify-sha1-prefix", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue clearly defines affected commands, unchanged SHA-1 behavior, explicit exclusions, and testable outcomes for 40-, 64-, and unexpected-length refs. It identifies the REST ref endpoint and the digest's use in both error output and attestation lookup. Its largest limitation is that failing unknown lengths is a proposed policy rather than a requirement established by an independent task statement. The claimed SHA-256 rollout and linked issue could not be independently substantiated from the supplied task context." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The LLD commits to a concrete signature, length mapping, error behavior, data flow, callers, and file-level changes. Its references to `NewDigestedArtifactForRelease`, `FetchParams.Digest`, `FetchRefSHA`, and `WithArtifactDigest` are internally consistent with the submitted patch. The main weakness is limited analysis of malformed 40- or 64-character non-hex values and future object formats, although both choices are acknowledged. Claims that exactly two callers exist and no stable repository-format field is available were asserted but not independently demonstrated in the artifacts." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 85, + "notes": "The review catches meaningful defects: silent SHA-1 fallback, signature-change blast radius, and tests that inspect only user-facing errors rather than `FetchParams.Digest`. It provides concrete resolutions that appear in the revised LLD and patch, including bad-length command tests using a fail-on-call client. Its largest deficiency is that several reviewer sections are superficial and it does not challenge length-only classification or directly testing the verifier boundary. Assertions that the ref API always returns full IDs and lacks a stable object-format field remain unsupported within the submission." + }, + "testing": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The plan supplies exact package commands, matching test names, concrete error-string and lookup-digest oracles, negative cases, and a realistic optional SHA-256 repository check. The patch implements these tests with `StubFetchRefSHA`, `MockClient.OnGetByDigest`, and `NewFailTestClient`, so they would detect the central prefix regression and premature lookup. The largest gap is that it does not directly exercise the selected algorithm through the verifier despite an acceptance criterion naming that path. It also omits the repository-prescribed full `go test ./...` and `make lint` checks, relying on narrower package tests and `gofmt`." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the design end to end by centralizing 40/64-length inference in `artifact.NewDigestedArtifactForRelease`, propagating constructor errors in both commands, and testing output plus lookup parameters. The unified diff consistently targets the constructor and the `verifyRun` and `verifyAssetRun` call sites, while preserving the existing SHA-1 path and preventing lookup for unsupported lengths. Its largest limitation is the absence of a command-level test proving that `sha256` reaches the verifier, and length-valid non-hex refs are intentionally accepted. The summary accurately describes the patch, though its claims that tests, vet, formatting, and `git apply --check` passed are execution claims not independently evidenced by the artifact." + } + }, + "task_score": 86.0, + "verdict": "This is a strong, coherent change with concrete regression coverage, limited mainly by the missing independent task specification and lack of direct verification-path testing.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:34:39.728229Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 130894, + "cached_input_tokens": 89300, + "cache_write_input_tokens": 25736, + "output_tokens": 6411, + "reasoning_output_tokens": 5127 + }, + "duration_ms": 100836 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json new file mode 100644 index 00000000..fdfc5f2d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/release-verify-sha1-prefix/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "release-verify-sha1-prefix", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "release", + "go", + "cli", + "attestation", + "sha256", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 111.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 40, + "output_tokens": 17649, + "cache_read_tokens": 1245756, + "cache_write_tokens": 64158, + "total_tokens": 1327603, + "total_cost_usd": 0.6447278199999998, + "latency_seconds": 158.0, + "num_turns": 19, + "generation_tokens_per_sec": 111.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 40, + "output_tokens": 17649, + "latency_seconds": 158.0, + "num_turns": 19, + "total_cost_usd": 0.6447278199999998, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 1245756, + "cache_creation_tokens": 64158, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:11:55.182054Z", + "run_ended_at": "2026-10-14T15:14:33.204276Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 40, + "output_tokens": 17649, + "cache_read_tokens": 1245756, + "cache_write_tokens": 64158, + "total_tokens": 1327603, + "total_cost_usd": 0.6447278199999998, + "latency_seconds": 158.0, + "num_turns": 19, + "generation_tokens_per_sec": 111.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "release-verify-sha1-prefix", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue clearly defines affected commands, unchanged SHA-1 behavior, explicit exclusions, and testable outcomes for 40-, 64-, and unexpected-length refs. It identifies the REST ref endpoint and the digest's use in both error output and attestation lookup. Its largest limitation is that failing unknown lengths is a proposed policy rather than a requirement established by an independent task statement. The claimed SHA-256 rollout and linked issue could not be independently substantiated from the supplied task context." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The LLD commits to a concrete signature, length mapping, error behavior, data flow, callers, and file-level changes. Its references to `NewDigestedArtifactForRelease`, `FetchParams.Digest`, `FetchRefSHA`, and `WithArtifactDigest` are internally consistent with the submitted patch. The main weakness is limited analysis of malformed 40- or 64-character non-hex values and future object formats, although both choices are acknowledged. Claims that exactly two callers exist and no stable repository-format field is available were asserted but not independently demonstrated in the artifacts." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 85, + "notes": "The review catches meaningful defects: silent SHA-1 fallback, signature-change blast radius, and tests that inspect only user-facing errors rather than `FetchParams.Digest`. It provides concrete resolutions that appear in the revised LLD and patch, including bad-length command tests using a fail-on-call client. Its largest deficiency is that several reviewer sections are superficial and it does not challenge length-only classification or directly testing the verifier boundary. Assertions that the ref API always returns full IDs and lacks a stable object-format field remain unsupported within the submission." + }, + "testing": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The plan supplies exact package commands, matching test names, concrete error-string and lookup-digest oracles, negative cases, and a realistic optional SHA-256 repository check. The patch implements these tests with `StubFetchRefSHA`, `MockClient.OnGetByDigest`, and `NewFailTestClient`, so they would detect the central prefix regression and premature lookup. The largest gap is that it does not directly exercise the selected algorithm through the verifier despite an acceptance criterion naming that path. It also omits the repository-prescribed full `go test ./...` and `make lint` checks, relying on narrower package tests and `gofmt`." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the design end to end by centralizing 40/64-length inference in `artifact.NewDigestedArtifactForRelease`, propagating constructor errors in both commands, and testing output plus lookup parameters. The unified diff consistently targets the constructor and the `verifyRun` and `verifyAssetRun` call sites, while preserving the existing SHA-1 path and preventing lookup for unsupported lengths. Its largest limitation is the absence of a command-level test proving that `sha256` reaches the verifier, and length-valid non-hex refs are intentionally accepted. The summary accurately describes the patch, though its claims that tests, vet, formatting, and `git apply --check` passed are execution claims not independently evidenced by the artifact." + } + }, + "task_score": 86.0, + "verdict": "This is a strong, coherent change with concrete regression coverage, limited mainly by the missing independent task specification and lack of direct verification-path testing.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:34:39.728229Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 130894, + "cached_input_tokens": 89300, + "cache_write_input_tokens": 25736, + "output_tokens": 6411, + "reasoning_output_tokens": 5127 + }, + "duration_ms": 100836 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json new file mode 100644 index 00000000..80090b27 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-clone-ghe-ssh-ca-url", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The issue precisely defines the SSH-CA failure, fork and wiki behavior, fallbacks, HTTPS compatibility, and explicit non-goals. GitHub issue #14344 corroborates the reported URL reconstruction, affected files, API model support, and explicit-URL behavior. ([github.com](https://github.com/cli/cli/issues/14344)) The largest limitation is the absence of an independent task statement, so the expanded fallback and out-of-scope requirements cannot be confirmed independently. Its acceptance criteria are directly testable and cover the principal compatibility and malformed-input risks." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD commits to concrete changes in `api/queries_repo.go` and `pkg/cmd/repo/clone/clone.go`, including repository, parent, wiki, protocol, and fallback data flow. The fragment and call-site descriptions agree with the submitted patch and with the relevant-code evidence in issue #14344. ([github.com](https://github.com/cli/cli/issues/14344)) Its main gap is that validation requires only an SSH scheme and host, so a syntactically parsed but unusable host-only remote would not fall back. Exact line numbers, the claimed import-cycle consequence, and local source compatibility could not be independently rechecked because the repository shell failed before executing commands." + }, + "review": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 24, + "total": 92, + "notes": "The review contributes substantive findings rather than approvals alone, especially protocol validation, rejection of leading-option values, and preservation of advertised SSH hosts. Those findings are prioritized, actionable, and visibly resolved in the revised design and patch. Its largest omission is failure to notice that the planned GraphQL mock does not assert that `sshUrl` is actually requested, allowing the central wire change to regress while tests pass. The claim that GHES may advertise a separate SSH host or port was not independently verified from the unavailable local repository." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 80, + "notes": "The plan supplies concrete command expectations for GHE shorthand, explicit SSH URLs, wiki clones, fork upstreams, malformed values, fallbacks, and unchanged HTTPS behavior. Its central deficiency is that `httpmock.GraphQL` matches only the operation name while the mock response supplies `sshUrl` regardless of the requested fields, so the new tests can pass if the production query omits `sshUrl`. The listed `go test` commands and package paths are consistent with the repository instructions, although the manual fork variables are incompletely defined. Claimed test execution could not be independently reproduced because local command execution was unavailable." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the designed data flow end to end: it requests `sshUrl`, selects it only for SSH, handles repository and parent remotes, preserves wiki transformation, and retains reconstruction fallback. Its target symbols and surrounding contexts align with the submitted baseline and the affected files identified by issue #14344. ([github.com](https://github.com/cli/cli/issues/14344)) The largest weakness is test coverage of the GraphQL request itself; additionally, `isSSHURL` accepts a scheme-and-host URL without confirming a usable repository path. The summary accurately discloses the unrun tenant check, but `git apply --check`, compilation, formatting, and claimed test results could not be independently verified after the local shell sandbox failed." + } + }, + "task_score": 90.0, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is that its tests do not prove the GraphQL query actually requests the field required for production behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:37:04.738064Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 125843, + "cached_input_tokens": 73899, + "cache_write_input_tokens": 25149, + "output_tokens": 9022, + "reasoning_output_tokens": 7405 + }, + "duration_ms": 144997 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json new file mode 100644 index 00000000..af715eb0 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-clone-ghe-ssh-ca-url/metrics.json @@ -0,0 +1,157 @@ +{ + "task": "repo-clone-ghe-ssh-ca-url", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "medium", + "tags": [ + "repo", + "go", + "cli", + "git", + "ssh", + "ghe", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 117.1, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 74, + "output_tokens": 31926, + "cache_read_tokens": 2930591, + "cache_write_tokens": 86179, + "total_tokens": 3048770, + "total_cost_usd": 1.2330710700000003, + "latency_seconds": 272.6, + "num_turns": 36, + "generation_tokens_per_sec": 117.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 74, + "output_tokens": 31926, + "latency_seconds": 272.6, + "num_turns": 36, + "total_cost_usd": 1.2330710700000003, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2930591, + "cache_creation_tokens": 86179, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:00:01.595527Z", + "run_ended_at": "2026-10-14T15:04:34.249584Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 74, + "output_tokens": 31926, + "cache_read_tokens": 2930591, + "cache_write_tokens": 86179, + "total_tokens": 3048770, + "total_cost_usd": 1.2330710700000003, + "latency_seconds": 272.6, + "num_turns": 36, + "generation_tokens_per_sec": 117.1, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-clone-ghe-ssh-ca-url", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The issue precisely defines the SSH-CA failure, fork and wiki behavior, fallbacks, HTTPS compatibility, and explicit non-goals. GitHub issue #14344 corroborates the reported URL reconstruction, affected files, API model support, and explicit-URL behavior. ([github.com](https://github.com/cli/cli/issues/14344)) The largest limitation is the absence of an independent task statement, so the expanded fallback and out-of-scope requirements cannot be confirmed independently. Its acceptance criteria are directly testable and cover the principal compatibility and malformed-input risks." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD commits to concrete changes in `api/queries_repo.go` and `pkg/cmd/repo/clone/clone.go`, including repository, parent, wiki, protocol, and fallback data flow. The fragment and call-site descriptions agree with the submitted patch and with the relevant-code evidence in issue #14344. ([github.com](https://github.com/cli/cli/issues/14344)) Its main gap is that validation requires only an SSH scheme and host, so a syntactically parsed but unusable host-only remote would not fall back. Exact line numbers, the claimed import-cycle consequence, and local source compatibility could not be independently rechecked because the repository shell failed before executing commands." + }, + "review": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 24, + "total": 92, + "notes": "The review contributes substantive findings rather than approvals alone, especially protocol validation, rejection of leading-option values, and preservation of advertised SSH hosts. Those findings are prioritized, actionable, and visibly resolved in the revised design and patch. Its largest omission is failure to notice that the planned GraphQL mock does not assert that `sshUrl` is actually requested, allowing the central wire change to regress while tests pass. The claim that GHES may advertise a separate SSH host or port was not independently verified from the unavailable local repository." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 80, + "notes": "The plan supplies concrete command expectations for GHE shorthand, explicit SSH URLs, wiki clones, fork upstreams, malformed values, fallbacks, and unchanged HTTPS behavior. Its central deficiency is that `httpmock.GraphQL` matches only the operation name while the mock response supplies `sshUrl` regardless of the requested fields, so the new tests can pass if the production query omits `sshUrl`. The listed `go test` commands and package paths are consistent with the repository instructions, although the manual fork variables are incompletely defined. Claimed test execution could not be independently reproduced because local command execution was unavailable." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the designed data flow end to end: it requests `sshUrl`, selects it only for SSH, handles repository and parent remotes, preserves wiki transformation, and retains reconstruction fallback. Its target symbols and surrounding contexts align with the submitted baseline and the affected files identified by issue #14344. ([github.com](https://github.com/cli/cli/issues/14344)) The largest weakness is test coverage of the GraphQL request itself; additionally, `isSSHURL` accepts a scheme-and-host URL without confirming a usable repository path. The summary accurately discloses the unrun tenant check, but `git apply --check`, compilation, formatting, and claimed test results could not be independently verified after the local shell sandbox failed." + } + }, + "task_score": 90.0, + "verdict": "This is a strong, implementation-ready submission whose largest material limitation is that its tests do not prove the GraphQL query actually requests the field required for production behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:37:04.738064Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 125843, + "cached_input_tokens": 73899, + "cache_write_input_tokens": 25149, + "output_tokens": 9022, + "reasoning_output_tokens": 7405 + }, + "duration_ms": 144997 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json new file mode 100644 index 00000000..fe37a78e --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-create-reject-empty-name", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly describes the wasted interactive flow, expected fail-fast behavior, testable acceptance criteria, and deliberate exclusions. It specifically identifies the Repository name prompt, the malformed some-owner/ result, whitespace handling, and behavior that must remain unchanged. Its largest gap is that the independent task statement is absent, so the linked issue and exact historical failure could not be fully corroborated. It also does not explicitly decide whether an owner/ input with an empty repository component belongs in scope." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The LLD names the relevant create.go and create_test.go symbols and commits to a minimal check immediately after prompter.Input, making the implementation decision clear. Its data flow, error propagation, compatibility boundary, alternatives, and absence of migration or deployment work are proportionate to this small change. The main factual defect is its claim that an empty local-flow answer does not use the filepath.Base default, which conflicts with the review's more credible statement that pressing Enter returns the prompt default. Exact tagged-source line references and the stated PrompterMock behavior could not be independently verified because local shell execution failed before repository reads." + }, + "review": { + "completeness": 16, + "correctness": 18, + "specificity": 17, + "risk_awareness": 14, + "total": 65, + "notes": "The review identifies the principal UX tradeoff, confirms the shared-function strategy, and correctly questions how local prompt defaults behave. Its largest weakness is that nearly every role approves without materially stress-testing malformed owner/name forms, compatibility coverage, or the discrepancy it discovered in the LLD. The claim that nil PrompterMock functions prove every later prompt is unreachable is overstated because the submitted tests define InputFunc and SelectFunc and return explicit errors for unexpected prompts. Repository-backed verification of the mock's generated behavior was unavailable due the shell sandbox failure." + }, + "testing": { + "completeness": 19, + "correctness": 22, + "specificity": 22, + "risk_awareness": 18, + "total": 81, + "notes": "The plan supplies a concrete Go subtest command, exact expected error text, a manual CLI oracle, and a package-level compatibility run. The proposed negative tests would fail if owner, description, selection, or confirmation prompts were reached because unexpected prompt paths do not produce the expected error. Its largest omission is explicit coverage for owner/REPO and the directory-derived default promised by the acceptance criteria, relying instead on vaguely named existing cases. It also omits the repository-required go test ./... and make lint checks from its execution checklist." + }, + "implementation": { + "completeness": 21, + "correctness": 22, + "specificity": 22, + "risk_awareness": 19, + "total": 84, + "notes": "The patch implements the designed fail-fast check directly after the name prompt and preserves nonblank input unchanged, including surrounding whitespace on otherwise nonempty names. Its two Test_createRun cases check empty and whitespace-only input and ensure unexpected later prompts cannot yield the expected error, although the comment inaccurately says every such prompt would panic. The main coverage gap is the absence of new positive tests for owner/REPO and the local-directory default, plus no full-suite or lint verification. The target paths and diff context appear internally consistent, but git apply --check and direct source comparison could not be completed because every local command was blocked by an environment-level bwrap failure." + } + }, + "task_score": 81.0, + "verdict": "A strong, focused solution with a credible minimal patch, limited mainly by a shallow expert review and incomplete positive-regression and repository-wide verification coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:39:14.870807Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 113408, + "cached_input_tokens": 55477, + "cache_write_input_tokens": 19323, + "output_tokens": 8119, + "reasoning_output_tokens": 6233 + }, + "duration_ms": 130120 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json new file mode 100644 index 00000000..3d5d71f7 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-create-reject-empty-name/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "repo-create-reject-empty-name", + "repo": "https://github.com/cli/cli", + "ref": "v2.100.0", + "complexity": "trivial", + "tags": [ + "repo", + "go", + "cli", + "interactive-prompts", + "validation", + "fixed-in-2.101.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 131.7, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 18, + "output_tokens": 8524, + "cache_read_tokens": 380211, + "cache_write_tokens": 63601, + "total_tokens": 452354, + "total_cost_usd": 0.35235277, + "latency_seconds": 64.7, + "num_turns": 8, + "generation_tokens_per_sec": 131.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 18, + "output_tokens": 8524, + "latency_seconds": 64.7, + "num_turns": 8, + "total_cost_usd": 0.35235277, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 380211, + "cache_creation_tokens": 63601, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:42:29.808824Z", + "run_ended_at": "2026-10-14T14:43:34.507017Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 18, + "output_tokens": 8524, + "cache_read_tokens": 380211, + "cache_write_tokens": 63601, + "total_tokens": 452354, + "total_cost_usd": 0.35235277, + "latency_seconds": 64.7, + "num_turns": 8, + "generation_tokens_per_sec": 131.7, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-create-reject-empty-name", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly describes the wasted interactive flow, expected fail-fast behavior, testable acceptance criteria, and deliberate exclusions. It specifically identifies the Repository name prompt, the malformed some-owner/ result, whitespace handling, and behavior that must remain unchanged. Its largest gap is that the independent task statement is absent, so the linked issue and exact historical failure could not be fully corroborated. It also does not explicitly decide whether an owner/ input with an empty repository component belongs in scope." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The LLD names the relevant create.go and create_test.go symbols and commits to a minimal check immediately after prompter.Input, making the implementation decision clear. Its data flow, error propagation, compatibility boundary, alternatives, and absence of migration or deployment work are proportionate to this small change. The main factual defect is its claim that an empty local-flow answer does not use the filepath.Base default, which conflicts with the review's more credible statement that pressing Enter returns the prompt default. Exact tagged-source line references and the stated PrompterMock behavior could not be independently verified because local shell execution failed before repository reads." + }, + "review": { + "completeness": 16, + "correctness": 18, + "specificity": 17, + "risk_awareness": 14, + "total": 65, + "notes": "The review identifies the principal UX tradeoff, confirms the shared-function strategy, and correctly questions how local prompt defaults behave. Its largest weakness is that nearly every role approves without materially stress-testing malformed owner/name forms, compatibility coverage, or the discrepancy it discovered in the LLD. The claim that nil PrompterMock functions prove every later prompt is unreachable is overstated because the submitted tests define InputFunc and SelectFunc and return explicit errors for unexpected prompts. Repository-backed verification of the mock's generated behavior was unavailable due the shell sandbox failure." + }, + "testing": { + "completeness": 19, + "correctness": 22, + "specificity": 22, + "risk_awareness": 18, + "total": 81, + "notes": "The plan supplies a concrete Go subtest command, exact expected error text, a manual CLI oracle, and a package-level compatibility run. The proposed negative tests would fail if owner, description, selection, or confirmation prompts were reached because unexpected prompt paths do not produce the expected error. Its largest omission is explicit coverage for owner/REPO and the directory-derived default promised by the acceptance criteria, relying instead on vaguely named existing cases. It also omits the repository-required go test ./... and make lint checks from its execution checklist." + }, + "implementation": { + "completeness": 21, + "correctness": 22, + "specificity": 22, + "risk_awareness": 19, + "total": 84, + "notes": "The patch implements the designed fail-fast check directly after the name prompt and preserves nonblank input unchanged, including surrounding whitespace on otherwise nonempty names. Its two Test_createRun cases check empty and whitespace-only input and ensure unexpected later prompts cannot yield the expected error, although the comment inaccurately says every such prompt would panic. The main coverage gap is the absence of new positive tests for owner/REPO and the local-directory default, plus no full-suite or lint verification. The target paths and diff context appear internally consistent, but git apply --check and direct source comparison could not be completed because every local command was blocked by an environment-level bwrap failure." + } + }, + "task_score": 81.0, + "verdict": "A strong, focused solution with a credible minimal patch, limited mainly by a shallow expert review and incomplete positive-regression and repository-wide verification coverage.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:39:14.870807Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 113408, + "cached_input_tokens": 55477, + "cache_write_input_tokens": 19323, + "output_tokens": 8119, + "reasoning_output_tokens": 6233 + }, + "duration_ms": 130120 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json new file mode 100644 index 00000000..819be75c --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-set-default-unset-without-repo", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The issue clearly defines the non-interactive failure, retained behavior, testable acceptance criteria, and explicit non-goals. Its proposed guard change matches the documented `--view` and `--unset` command modes. The strongest evidence is the exact error and `pkg/cmd/repo/setdefault/setdefault.go` integration point; the largest gap is limited discussion of combined mode flags. No independent task statement was supplied, and issue #14521 could not be verified. ([github.com](https://github.com/cli/cli/issues/7442), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to the precise guard, execution ordering, files, test cases, compatibility boundary, and alternatives. Its named constructor, command path, factory pattern, and local-git precondition agree with repository conventions. It explicitly addresses nil-repository safety, ignored arguments, flag precedence, and unchanged errors. Exact line hashes and output wording could not be independently verified against the local checkout. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/must-be-run-from-inside-a-git-repository/), [github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md))" + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 19, + "risk_awareness": 18, + "total": 75, + "notes": "The review identifies concrete concerns including ignored repository arguments, `--view` precedence, local configuration mutation, and the prior non-TTY coverage gap. Its backend and SMTS observations are specific to the proposed guard and table-driven tests. The largest deficiency is that most reviewer sections merely approve, and none catches the testing plan's installed-binary and pipeline-exit problems. The security conclusion is plausible, but its exact no-network assertion was not independently verified." + }, + "testing": { + "completeness": 18, + "correctness": 16, + "specificity": 20, + "risk_awareness": 18, + "total": 72, + "notes": "The automated cases directly pin non-interactive `--unset`, retained `--view`, and the exact bare-command error, while referencing the existing execution test. The package path, test symbols, flags, and expected options match the submitted patch and repository command structure. The manual plan incorrectly invokes an installed `gh` instead of the required built `bin/gh`, and `gh ... | cat` cannot establish exit 1 without `pipefail` or inspecting `PIPESTATUS`. Backward-compatibility cases are listed but lack concrete automated setup and oracles. ([github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md))" + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The one-line guard change implements the stated behavior end to end, and the tests exercise both exempt modes plus the preserved failure. The target paths, `NewCmdSetDefault`, option fields, injected run function, IOStreams TTY controls, and table-driven style are consistent with repository conventions. No material code defect or incompatible behavior is apparent, and the summary accurately describes the patch's scope. Exact patch applicability, index hashes, and the claimed reverse check could not be confirmed because local read commands failed before execution. ([github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default))" + } + }, + "task_score": 83.8, + "verdict": "This is a strong, narrowly scoped solution with an implementation-ready design and patch; its largest material limitation is a testing plan whose manual commands can exercise the wrong binary and misreport pipeline exit status.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:41:30.748727Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 135574, + "cached_input_tokens": 76668, + "cache_write_input_tokens": 21669, + "output_tokens": 8941, + "reasoning_output_tokens": 7745 + }, + "duration_ms": 135864 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json new file mode 100644 index 00000000..503c2512 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-set-default-unset-without-repo/metrics.json @@ -0,0 +1,156 @@ +{ + "task": "repo-set-default-unset-without-repo", + "repo": "https://github.com/cli/cli", + "ref": "v2.101.0", + "complexity": "trivial", + "tags": [ + "repo", + "go", + "cli", + "tty", + "flags", + "fixed-in-2.102.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 141.8, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 18, + "output_tokens": 11050, + "cache_read_tokens": 422189, + "cache_write_tokens": 42875, + "total_tokens": 476132, + "total_cost_usd": 0.33237743000000003, + "latency_seconds": 77.9, + "num_turns": 8, + "generation_tokens_per_sec": 141.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 18, + "output_tokens": 11050, + "latency_seconds": 77.9, + "num_turns": 8, + "total_cost_usd": 0.33237743000000003, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 422189, + "cache_creation_tokens": 42875, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T14:44:57.884629Z", + "run_ended_at": "2026-10-14T14:46:15.772714Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 18, + "output_tokens": 11050, + "cache_read_tokens": 422189, + "cache_write_tokens": 42875, + "total_tokens": 476132, + "total_cost_usd": 0.33237743000000003, + "latency_seconds": 77.9, + "num_turns": 8, + "generation_tokens_per_sec": 141.8, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-set-default-unset-without-repo", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The issue clearly defines the non-interactive failure, retained behavior, testable acceptance criteria, and explicit non-goals. Its proposed guard change matches the documented `--view` and `--unset` command modes. The strongest evidence is the exact error and `pkg/cmd/repo/setdefault/setdefault.go` integration point; the largest gap is limited discussion of combined mode flags. No independent task statement was supplied, and issue #14521 could not be verified. ([github.com](https://github.com/cli/cli/issues/7442), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to the precise guard, execution ordering, files, test cases, compatibility boundary, and alternatives. Its named constructor, command path, factory pattern, and local-git precondition agree with repository conventions. It explicitly addresses nil-repository safety, ignored arguments, flag precedence, and unchanged errors. Exact line hashes and output wording could not be independently verified against the local checkout. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/must-be-run-from-inside-a-git-repository/), [github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md))" + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 19, + "risk_awareness": 18, + "total": 75, + "notes": "The review identifies concrete concerns including ignored repository arguments, `--view` precedence, local configuration mutation, and the prior non-TTY coverage gap. Its backend and SMTS observations are specific to the proposed guard and table-driven tests. The largest deficiency is that most reviewer sections merely approve, and none catches the testing plan's installed-binary and pipeline-exit problems. The security conclusion is plausible, but its exact no-network assertion was not independently verified." + }, + "testing": { + "completeness": 18, + "correctness": 16, + "specificity": 20, + "risk_awareness": 18, + "total": 72, + "notes": "The automated cases directly pin non-interactive `--unset`, retained `--view`, and the exact bare-command error, while referencing the existing execution test. The package path, test symbols, flags, and expected options match the submitted patch and repository command structure. The manual plan incorrectly invokes an installed `gh` instead of the required built `bin/gh`, and `gh ... | cat` cannot establish exit 1 without `pipefail` or inspecting `PIPESTATUS`. Backward-compatibility cases are listed but lack concrete automated setup and oracles. ([github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md))" + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The one-line guard change implements the stated behavior end to end, and the tests exercise both exempt modes plus the preserved failure. The target paths, `NewCmdSetDefault`, option fields, injected run function, IOStreams TTY controls, and table-driven style are consistent with repository conventions. No material code defect or incompatible behavior is apparent, and the summary accurately describes the patch's scope. Exact patch applicability, index hashes, and the claimed reverse check could not be confirmed because local read commands failed before execution. ([github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default))" + } + }, + "task_score": 83.8, + "verdict": "This is a strong, narrowly scoped solution with an implementation-ready design and patch; its largest material limitation is a testing plan whose manual commands can exercise the wrong binary and misreport pipeline exit status.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:41:30.748727Z", + "repo_ref": "v2.101.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 135574, + "cached_input_tokens": 76668, + "cache_write_input_tokens": 21669, + "output_tokens": 8941, + "reasoning_output_tokens": 7745 + }, + "duration_ms": 135864 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json new file mode 100644 index 00000000..888d4304 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/eval.json @@ -0,0 +1,64 @@ +{ + "task": "repo-sync-checked-out-worktree", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 84, + "notes": "The issue clearly defines the affected branch decision, desired worktree states, dirty and missing-directory behavior, compatibility cases, and explicit non-goals. Its largest defect is that the reproduction never advances the upstream branch, so as written it does not reproduce the reported staged changes. The submitted baseline diff context supports the diagnosis that a non-current existing branch reaches UpdateBranch with FETCH_HEAD. Because no independent task statement exists, the policy choice to modify another worktree rather than refuse cannot be externally verified." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 16, + "total": 78, + "notes": "The design is implementation-ready in its file mapping, interface changes, control flow, FETCH_HEAD resolution, compatibility branches, and error behavior. The cited git.Client Copy method and RepoDir field exist, supporting the proposed cross-worktree client construction. ([pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git)) Its largest deficiencies are failure to parse quoted or newline-containing worktree paths and specifying a copy-paste shell command using Go %q, which is not shell-safe for every path; Git recommends NUL-delimited porcelain parsing for unusual paths. ([github.com](https://github.com/git/git/blob/master/Documentation/git-worktree.adoc)) The absent independent task statement also leaves the broader auto-update policy unverifiable." + }, + "review": { + "completeness": 19, + "correctness": 16, + "specificity": 21, + "risk_awareness": 16, + "total": 72, + "notes": "The review's strongest contribution is identifying the per-worktree FETCH_HEAD problem and requiring resolution to a commit ID before changing directories. It also addresses missing worktree directories, dirty-state protection, command argument separation, and the residual TOCTOU window with concrete consequences. Its largest error is declaring no injection surface while approving a user-copyable `git -C %q stash --all` tip whose Go quoting is not general shell quoting; it also misses the porcelain path-decoding gap. The deferred rebase and bisect limitation is honestly recorded, but no independent requirement confirms that accepting it is sufficient." + }, + "testing": { + "completeness": 20, + "correctness": 14, + "specificity": 22, + "risk_awareness": 17, + "total": 73, + "notes": "The plan provides strong behavioral oracles for refs, HEAD, file contents, status cleanliness, unchanged dirty state, force behavior, parser exactness, and backward-compatible paths. Its largest flaw is that the missing-directory test detaches the calling worktree even though executeLocalRepoSync calls CurrentBranch before CheckedOutWorktree and the git client defines detached HEAD as ErrNotOnAnyBranch. ([pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git), [github.com](https://github.com/cli/cli/pull/7082)) The manual reproduction again omits an upstream update, and one error check uses assert rather than the repository-required require. The claimed test execution could not be independently verified." + }, + "implementation": { + "completeness": 19, + "correctness": 15, + "specificity": 21, + "risk_awareness": 14, + "total": 69, + "notes": "The patch implements its selected design across the git abstraction, sync control flow, mocks, command help, focused unit cases, and real-repository fixtures. Its baseline identifier matches the v2.98.0 release commit, and the proposed Copy and RepoDir APIs exist. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0), [pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git)) The largest defects are the shell-unsafe copy-paste remediation text and incomplete handling of porcelain paths requiring quoting, while the missing-directory fixture likely exits at CurrentBranch because it deliberately detaches the caller. Local patch application and test execution were unavailable because the repository command runner failed before process launch, so the summary's green verification claims remain unsupported." + } + }, + "task_score": 75.2, + "verdict": "The submission is coherent and mostly implementation-ready, but unsafe special-path handling and a likely invalid missing-worktree test materially undermine the implementation and its claimed verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:46:43.806257Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 212553, + "cached_input_tokens": 114231, + "cache_write_input_tokens": 31206, + "output_tokens": 18153, + "reasoning_output_tokens": 16125 + }, + "duration_ms": 313043 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json new file mode 100644 index 00000000..d6765f0d --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/repo-sync-checked-out-worktree/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "repo-sync-checked-out-worktree", + "repo": "https://github.com/cli/cli", + "ref": "v2.98.0", + "complexity": "high", + "tags": [ + "repo", + "go", + "git", + "worktree", + "fixed-in-2.99.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 128.2, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 32010, + "cache_read_tokens": 2101255, + "cache_write_tokens": 82659, + "total_tokens": 2215982, + "total_cost_usd": 1.0418259499999998, + "latency_seconds": 249.7, + "num_turns": 27, + "generation_tokens_per_sec": 128.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 58, + "output_tokens": 32010, + "latency_seconds": 249.7, + "num_turns": 27, + "total_cost_usd": 1.0418259499999998, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2101255, + "cache_creation_tokens": 82659, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:23:25.730888Z", + "run_ended_at": "2026-10-14T15:27:35.442147Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 58, + "output_tokens": 32010, + "cache_read_tokens": 2101255, + "cache_write_tokens": 82659, + "total_tokens": 2215982, + "total_cost_usd": 1.0418259499999998, + "latency_seconds": 249.7, + "num_turns": 27, + "generation_tokens_per_sec": 128.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "repo-sync-checked-out-worktree", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 84, + "notes": "The issue clearly defines the affected branch decision, desired worktree states, dirty and missing-directory behavior, compatibility cases, and explicit non-goals. Its largest defect is that the reproduction never advances the upstream branch, so as written it does not reproduce the reported staged changes. The submitted baseline diff context supports the diagnosis that a non-current existing branch reaches UpdateBranch with FETCH_HEAD. Because no independent task statement exists, the policy choice to modify another worktree rather than refuse cannot be externally verified." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 16, + "total": 78, + "notes": "The design is implementation-ready in its file mapping, interface changes, control flow, FETCH_HEAD resolution, compatibility branches, and error behavior. The cited git.Client Copy method and RepoDir field exist, supporting the proposed cross-worktree client construction. ([pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git)) Its largest deficiencies are failure to parse quoted or newline-containing worktree paths and specifying a copy-paste shell command using Go %q, which is not shell-safe for every path; Git recommends NUL-delimited porcelain parsing for unusual paths. ([github.com](https://github.com/git/git/blob/master/Documentation/git-worktree.adoc)) The absent independent task statement also leaves the broader auto-update policy unverifiable." + }, + "review": { + "completeness": 19, + "correctness": 16, + "specificity": 21, + "risk_awareness": 16, + "total": 72, + "notes": "The review's strongest contribution is identifying the per-worktree FETCH_HEAD problem and requiring resolution to a commit ID before changing directories. It also addresses missing worktree directories, dirty-state protection, command argument separation, and the residual TOCTOU window with concrete consequences. Its largest error is declaring no injection surface while approving a user-copyable `git -C %q stash --all` tip whose Go quoting is not general shell quoting; it also misses the porcelain path-decoding gap. The deferred rebase and bisect limitation is honestly recorded, but no independent requirement confirms that accepting it is sufficient." + }, + "testing": { + "completeness": 20, + "correctness": 14, + "specificity": 22, + "risk_awareness": 17, + "total": 73, + "notes": "The plan provides strong behavioral oracles for refs, HEAD, file contents, status cleanliness, unchanged dirty state, force behavior, parser exactness, and backward-compatible paths. Its largest flaw is that the missing-directory test detaches the calling worktree even though executeLocalRepoSync calls CurrentBranch before CheckedOutWorktree and the git client defines detached HEAD as ErrNotOnAnyBranch. ([pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git), [github.com](https://github.com/cli/cli/pull/7082)) The manual reproduction again omits an upstream update, and one error check uses assert rather than the repository-required require. The claimed test execution could not be independently verified." + }, + "implementation": { + "completeness": 19, + "correctness": 15, + "specificity": 21, + "risk_awareness": 14, + "total": 69, + "notes": "The patch implements its selected design across the git abstraction, sync control flow, mocks, command help, focused unit cases, and real-repository fixtures. Its baseline identifier matches the v2.98.0 release commit, and the proposed Copy and RepoDir APIs exist. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0), [pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git)) The largest defects are the shell-unsafe copy-paste remediation text and incomplete handling of porcelain paths requiring quoting, while the missing-directory fixture likely exits at CurrentBranch because it deliberately detaches the caller. Local patch application and test execution were unavailable because the repository command runner failed before process launch, so the summary's green verification claims remain unsupported." + } + }, + "task_score": 75.2, + "verdict": "The submission is coherent and mostly implementation-ready, but unsafe special-path handling and a likely invalid missing-worktree test materially undermine the implementation and its claimed verification.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:46:43.806257Z", + "repo_ref": "v2.98.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 212553, + "cached_input_tokens": 114231, + "cache_write_input_tokens": 31206, + "output_tokens": 18153, + "reasoning_output_tokens": 16125 + }, + "duration_ms": 313043 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/run-summary.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/run-summary.json new file mode 100644 index 00000000..df3839b1 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/run-summary.json @@ -0,0 +1,1437 @@ +{ + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "scope": "cli-cli-v1", + "provider": "bedrock", + "ref": null, + "refs": [ + "v2.100.0", + "v2.101.0", + "v2.92.0", + "v2.93.0", + "v2.94.0", + "v2.95.0", + "v2.96.0", + "v2.97.0", + "v2.98.0" + ], + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T15:56:39.256450Z", + "repo_ref": "v2.100.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 164748, + "cached_input_tokens": 90052, + "cache_write_input_tokens": 30614, + "output_tokens": 12849, + "reasoning_output_tokens": 11004 + }, + "duration_ms": 209436 + }, + "num_tasks": 21, + "num_scored": 21, + "num_failed": 0, + "failed_tasks": [], + "mean_task_score_excl_failed": 82.5, + "mean_cost_usd_excl_failed": 0.91, + "tasks": [ + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 36, + "input_tokens": 74, + "output_tokens": 31926, + "total_tokens": 3048770, + "latency_seconds": 272.6, + "total_cost_usd": 1.2330710700000003, + "result_subtype": "success", + "task_score": 90.0, + "cache_read_tokens": 2930591, + "cache_write_tokens": 86179, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 117.1, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 24, + "correctness": 24, + "specificity": 24, + "risk_awareness": 23, + "total": 95, + "notes": "The issue precisely defines the SSH-CA failure, fork and wiki behavior, fallbacks, HTTPS compatibility, and explicit non-goals. GitHub issue #14344 corroborates the reported URL reconstruction, affected files, API model support, and explicit-URL behavior. ([github.com](https://github.com/cli/cli/issues/14344)) The largest limitation is the absence of an independent task statement, so the expanded fallback and out-of-scope requirements cannot be confirmed independently. Its acceptance criteria are directly testable and cover the principal compatibility and malformed-input risks." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD commits to concrete changes in `api/queries_repo.go` and `pkg/cmd/repo/clone/clone.go`, including repository, parent, wiki, protocol, and fallback data flow. The fragment and call-site descriptions agree with the submitted patch and with the relevant-code evidence in issue #14344. ([github.com](https://github.com/cli/cli/issues/14344)) Its main gap is that validation requires only an SSH scheme and host, so a syntactically parsed but unusable host-only remote would not fall back. Exact line numbers, the claimed import-cycle consequence, and local source compatibility could not be independently rechecked because the repository shell failed before executing commands." + }, + "review": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 24, + "total": 92, + "notes": "The review contributes substantive findings rather than approvals alone, especially protocol validation, rejection of leading-option values, and preservation of advertised SSH hosts. Those findings are prioritized, actionable, and visibly resolved in the revised design and patch. Its largest omission is failure to notice that the planned GraphQL mock does not assert that `sshUrl` is actually requested, allowing the central wire change to regress while tests pass. The claim that GHES may advertise a separate SSH host or port was not independently verified from the unavailable local repository." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 80, + "notes": "The plan supplies concrete command expectations for GHE shorthand, explicit SSH URLs, wiki clones, fork upstreams, malformed values, fallbacks, and unchanged HTTPS behavior. Its central deficiency is that `httpmock.GraphQL` matches only the operation name while the mock response supplies `sshUrl` regardless of the requested fields, so the new tests can pass if the production query omits `sshUrl`. The listed `go test` commands and package paths are consistent with the repository instructions, although the manual fork variables are incompletely defined. Claimed test execution could not be independently reproduced because local command execution was unavailable." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 22, + "total": 91, + "notes": "The patch implements the designed data flow end to end: it requests `sshUrl`, selects it only for SSH, handles repository and parent remotes, preserves wiki transformation, and retains reconstruction fallback. Its target symbols and surrounding contexts align with the submitted baseline and the affected files identified by issue #14344. ([github.com](https://github.com/cli/cli/issues/14344)) The largest weakness is test coverage of the GraphQL request itself; additionally, `isSSHURL` accepts a scheme-and-host URL without confirming a usable repository path. The summary accurately discloses the unrun tenant check, but `git apply --check`, compilation, formatting, and claimed test results could not be independently verified after the local shell sandbox failed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 35, + "input_tokens": 72, + "output_tokens": 23845, + "total_tokens": 2976074, + "latency_seconds": 210.6, + "total_cost_usd": 1.1372522700000003, + "result_subtype": "success", + "task_score": 89.2, + "cache_read_tokens": 2863096, + "cache_write_tokens": 89061, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 113.2, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 18, + "total": 88, + "notes": "The issue precisely defines supported entities, REST paths, payload typing, help changes, and testable acceptance criteria. The referenced API confirms integer selected_repository_ids, and the related upstream work confirms expansion across set, list, and delete. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets), [github.com](https://github.com/cli/cli/pull/13421)) Its largest gap is limited treatment of GHES availability, permissions, and expected server errors. No independent task statement was supplied, so intended scope beyond the identifier and internally consistent artifacts cannot be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 21, + "total": 91, + "notes": "The LLD is strongly grounded in real files and commits to the key design: add one shared App value while reusing existing URL interpolation and the non-Dependabot integer payload path. The upstream PR independently confirms the named parser, support matrix, enum flag, and repository/organization integration points. ([github.com](https://github.com/cli/cli/pull/13421)) Its main weakness is that GHES and authorization failures are delegated to existing API error handling without defining validation or diagnostic expectations. The pagination claim and estimated line counts could not be independently checked locally because the repository shell sandbox failed to initialize." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 23, + "total": 89, + "notes": "The review identifies concrete change-specific risks, especially integer repository IDs, fetching the public key from the agents endpoint, pre-network entity rejection, and brittle list-test path rewriting. Its GHES analysis matches the actual maintainer discussion, which accepted explicit requests failing through the server API on unsupported hosts. ([github.com](https://github.com/cli/cli/pull/13421)) The largest deficiency is that every perspective ultimately approves, while the product-naming concern is dismissed rather than resolved; current help uses the clearer capitalized Agents terminology. ([cli.github.com](https://cli.github.com/manual/gh_secret), [cli.github.com](https://cli.github.com/manual/gh_help_reference)) The assertion that secrets are never logged is reasonable from an unchanged encryption path but is not directly demonstrated by the submitted patch." + }, + "testing": { + "completeness": 21, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The plan provides unusually concrete cases, paths, payload types, TTY outputs, backward-compatible defaults, negative commands, and live set-list-delete checks. The patch contains the named repository and organization runtime cases, consistent with the upstream addition of endpoint-routing tests. ([github.com](https://github.com/cli/cli/pull/13421)) Its largest gap is omission of the repository-required go test ./... and make lint checks, limiting regression coverage to the secret packages. It also lacks automated per-command assertions for the exact environment/user error and absence of network calls, leaving those acceptance conditions mostly to the shared matrix and manual tests." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The focused diff implements the design end to end: shared parsing and entity support, all three command enums, repository and organization routing tests, integer organization payload coverage, and help updates. The paths and symbols agree with the baseline context, and the upstream merged change confirms this architecture and the runtime list/delete endpoint coverage. ([github.com](https://github.com/cli/cli/pull/13421), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The main defect is minor UX wording: lowercase generic \"agents\" does not resolve the review's naming concern as clearly as the official capitalized Agents wording. ([cli.github.com](https://cli.github.com/manual/gh_secret), [cli.github.com](https://cli.github.com/manual/gh_help_reference)) I could not independently run git apply --check or the claimed tests because the read-only shell sandbox failed before command execution, so those verification claims remain unconfirmed rather than contradicted." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "trivial", + "ref": "v2.95.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 9, + "input_tokens": 20, + "output_tokens": 8370, + "total_tokens": 509970, + "latency_seconds": 65.1, + "total_cost_usd": 0.30055729000000003, + "result_subtype": "success", + "task_score": 87.6, + "cache_read_tokens": 462807, + "cache_write_tokens": 38773, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 128.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The issue precisely identifies the omitted `counts.Canceled` guard and unreachable cancelled-summary branch, matching the merged upstream diagnosis. ([github.com](https://github.com/cli/cli/pull/13679)) Its acceptance criteria and non-goals concretely cover TTY output, regression behavior, and exit status. The largest weakness is the unsupported claim about a second blank line and the absence of reproducible before-and-after command output. No independent task statement was supplied, so requirements beyond the task identifier and repository behavior remain unverifiable." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD names the relevant files, counter flow, guard change, fixture, expected output, and unchanged exit semantics with implementation-ready precision. Its central decision matches the upstream fix released in v2.96.0. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) The largest gap is selecting only a full-command regression case rather than directly exercising every `printSummary` branch, which upstream review explicitly requested. ([github.com](https://github.com/cli/cli/pull/13679)) Local execution was unavailable, so the asserted watch-path and exit-flow details could not be independently run." + }, + "review": { + "completeness": 19, + "correctness": 22, + "specificity": 20, + "risk_awareness": 20, + "total": 81, + "notes": "The review's strongest contribution is checking exit behavior, watch-path impact, non-TTY behavior, and fixture isolation rather than merely approving the one-line change. Its core assessment is consistent with the upstream diagnosis and minimal fix. ([github.com](https://github.com/cli/cli/pull/13679)) The largest deficiency is that most reviewer sections contain generic approvals and it misses the concrete recommendation to table-test every summary branch raised during the real review. ([github.com](https://github.com/cli/cli/pull/13679)) The assertion that cancelled non-TTY rows use failure status is not demonstrated by a referenced test or code excerpt." + }, + "testing": { + "completeness": 18, + "correctness": 22, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan gives an exact subtest command, concrete output oracle, empty-error expectation, and package regression run that would catch the reported defect. The cancelled-only oracle agrees with the upstream fix and its regression test. ([github.com](https://github.com/cli/cli/pull/13679)) Its largest deficiency is omitting the repository-required `go test ./...` and `make lint`, while also lacking direct coverage of all `printSummary` branches requested upstream. ([github.com](https://github.com/cli/cli/pull/13679)) The manual scenario has no deterministic setup, and non-TTY preservation is stated rather than assigned a specific assertion." + }, + "implementation": { + "completeness": 23, + "correctness": 24, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The patch implements the exact load-bearing fix by adding `counts.Canceled` to the existing guard, matching the merged upstream change. ([github.com](https://github.com/cli/cli/pull/13679), [github.com](https://github.com/cli/cli/releases/tag/v2.96.0)) The fixture timestamps support the asserted `1m26s` row, and the table case pins the summary, tally, row rendering, and successful return behavior end to end. The main limitation is that the new test covers only the cancelled branch through comparatively brittle full-output integration, whereas upstream ultimately added a direct table test for every summary path. ([github.com](https://github.com/cli/cli/pull/13679)) The summary's reverse-apply verification wording is confusing, and local `git apply --check` could not be executed due the read-only shell sandbox, but no functional deviation is apparent from the verified target context." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 12, + "input_tokens": 26, + "output_tokens": 10397, + "total_tokens": 681719, + "latency_seconds": 76.1, + "total_cost_usd": 0.3682706499999999, + "result_subtype": "success", + "task_score": 87.0, + "cache_read_tokens": 629335, + "cache_write_tokens": 41961, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 136.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 24, + "correctness": 23, + "specificity": 25, + "risk_awareness": 22, + "total": 94, + "notes": "The issue precisely defines the terminal defect, reproduction path, unchanged behavior, and exact stderr oracle. Its acceptance criteria distinguish stdout, stderr, SilentError, download, and execution behavior, while explicitly excluding the non-interactive branch. The cited call site is consistent with the submitted patch at pkg/cmd/copilot/copilot.go. The independent task statement, issue reference, and repository labels were not independently verifiable." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to a minimal data flow and names concrete integration points including runCopilot, CanPrompt, PrompterMock, and the package-level test seams. It correctly identifies the TTY setup needed to reach the interactive decline branch and documents the sibling non-interactive message as deferred. Its principal gap is that the proposed test neither checks Confirm's defaultValue nor directly instruments the download path despite claiming unchanged prompt and download behavior. Local source inspection was blocked by the execution sandbox, so line numbers and API signatures could not be independently confirmed." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 84, + "notes": "The review's strongest finding is the concrete warning that missing TTY overrides would test the wrong branch, followed by actionable call-count and execution guards. It also identifies the sibling non-interactive message and reasonably treats changing it as a compatibility question rather than silently expanding scope. However, it overstates that the planned test pins absence of download and misses that defaultValue is ignored by the Confirm mock. Most reviewer sections are appropriately brief for this trivial change but contribute little independent stress testing." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The plan provides an exact package test command, precise stdout and stderr oracles, error identity, prompt count, and a PTY-based manual check. Those checks directly target the reported regression and preserve the non-interactive branch as an explicit non-goal. The claim that TestDownloadCopilot covers answering yes is not demonstrated as an end-to-end runCopilot branch test, and no direct download guard or Confirm default assertion is specified. It also omits the repository-required pre-commit commands go test ./... and make lint." + }, + "implementation": { + "completeness": 23, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the requested behavior end to end by adding the newline at runCopilot's decline return and adding an exact-output unit test with TTY and execution guards. The diff is narrowly scoped to pkg/cmd/copilot/copilot.go and copilot_test.go and follows the repository's documented global-function restoration pattern. Its largest weakness is that the test ignores Confirm's defaultValue and does not directly guard the download function, so some unchanged-behavior claims are stronger than the test oracle. Local source comparison and git apply --check could not be completed because the read-only shell failed during sandbox initialization, leaving patch applicability independently unverified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "extension-uninstall-alias", + "complexity": "trivial", + "ref": "v2.93.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 14, + "input_tokens": 30, + "output_tokens": 15149, + "total_tokens": 827043, + "latency_seconds": 119.2, + "total_cost_usd": 0.46279057, + "result_subtype": "success", + "task_score": 87.0, + "cache_read_tokens": 765431, + "cache_write_tokens": 46433, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 127.1, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly defines alias behavior, legacy compatibility, help visibility, parent-alias behavior, tests, and explicit non-goals. Its proposed `Aliases` change matches the submitted `pkg/cmd/extension/command.go` context and gives directly testable acceptance criteria. The largest evidence gap is the absent independent task statement, so the related issue and exact intended help surface could not be independently established. It addresses the main compatibility and discoverability risks, though explicit error-parity and collision criteria would improve it." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The design is unusually concrete for this small change, naming `NewCmdExtension`, `normalizeExtensionSelector`, `ExtensionManager.Remove`, the relevant test table, help rendering, and acceptance coverage. Its one-field data flow, alternatives, compatibility rationale, and rollback plan leave no major implementation decision unresolved. Claims about `checkValidExtension`, generated documentation, and exact help internals could not be independently inspected because the repository shell failed to initialize, while telemetry is appropriately labeled as an inference. The design still treats the realistic risks proportionately, especially parent-help visibility, command-name collision, and preservation of `remove`." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 85, + "notes": "The strongest finding is the concrete distinction between testing Cobra's standalone default help and the production custom help renderer. It also examines parent-help discoverability, extension-name collision, telemetry attribution, and legacy acceptance coverage rather than merely approving the proposal. Its largest weakness is that every reviewer approves and it misses actionable test-quality concerns such as indexing `calls[0]` after a non-fatal length assertion and lacking an automated `gh ext uninstall` check. Given the trivial change, these omissions are limited rather than architectural." + }, + "testing": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The plan supplies concrete manager-call, normalization, TTY-output, non-TTY-output, help, argument-count, and backward-compatibility oracles using repository-appropriate Go test commands. It correctly retains existing `remove` acceptance coverage and adds manual verification of the production help renderer. Its largest defect is the unverified `cli/gh-test` dependency and an unisolated manual workflow that installs and removes an extension using real user state and network access. It also lacks automated parent-alias and manager-error cases, and its help assertion checks only substring presence rather than the exact aliases section." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch directly implements the design by adding `Aliases: []string{\"uninstall\"}` to the existing `remove` command and tests shared removal behavior, normalization, output, and help visibility. The resulting production manual currently lists `gh extension uninstall`, `gh extensions uninstall`, and `gh ext uninstall`, corroborating the intended Cobra and parent-alias behavior. ([cli.github.com](https://cli.github.com/manual/gh_extension_remove)) The main implementation weakness is test robustness: the call-count checks use non-fatal assertions before indexing, the help oracle is broad, and the parent alias is not tested directly. The summary accurately describes the diff and non-execution, but local `git apply --check` and source-context verification remained unavailable because the read-only shell sandbox could not initialize." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 17, + "input_tokens": 36, + "output_tokens": 20623, + "total_tokens": 1133210, + "latency_seconds": 166.7, + "total_cost_usd": 0.62382232, + "result_subtype": "success", + "task_score": 86.6, + "cache_read_tokens": 1052421, + "cache_write_tokens": 60130, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 123.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 20, + "total": 89, + "notes": "The issue provides reproducible setup, observable behavior, explicit exclusions, and acceptance criteria tied to a concrete DELETE request. Its root-cause claim matches v2.100.0 evidence showing the case-sensitive crossRepoPR comparison, omitted isCrossRepository field, and deletion guard. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The main weakness is limited discussion of output-mode differences and existing deletion-error behavior. No independent task statement was supplied, so broader intended scope could not be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 92, + "notes": "The LLD commits to exact files, data flow, helper semantics, compatibility behavior, and rollout, making the change implementation-ready. Repository evidence supports the baseline query and comparison, the existing API field, and mock finder behavior described in the design. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/pr/shared/finder.go)) Its largest gap is that the two-source decision is justified mainly through conservative behavior and test compatibility without requiring direct coverage of conflicting or absent values. Exact local line references and claimed test outcomes could not be independently rerun." + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 20, + "risk_awareness": 18, + "total": 76, + "notes": "The review meaningfully examines deletion safety, prompt behavior, the dual-source decision, and the consequences of widening a destructive action. Its observations are consistent with the verified early-return behavior and available isCrossRepository field. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) The largest deficiency is its approval-oriented conclusion despite identifying that merge tests do not assert requested fields; it should have required coverage for the API-true branch or field selection. The claimed absence of operational concerns is reasonable but was not independently validated on GHES." + }, + "testing": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The plan specifies exact package commands, fixtures, HTTP path, output oracle, fork regression coverage, compatibility tests, and a realistic manual workflow. The core oracle directly targets the verified failure mechanism: whether the guarded remote DELETE occurs. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7)) Its main gap is no automated assertion that isCrossRepository is requested and no case where that flag alone prevents deletion despite matching owners. The named tests and claimed executions could not be locally verified because the provided shell sandbox failed before command execution." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The patch implements the LLD end to end by requesting isCrossRepository, replacing the faulty comparison with EqualFold-backed logic, and adding a regression test whose registered DELETE and output assertion pin the reported behavior. Its merge.go hunk matches verified v2.100.0 source evidence, and the repository release history confirms this owner-casing defect was fixed in v2.101.0. ([gist.github.com](https://gist.github.com/dweomer/199fcf350ffd35c489ba6c67199673a7), [github.com](https://github.com/cli/cli/releases/tag/v2.101.0)) The largest weakness is that the test exercises only the EqualFold path, not the newly added API-flag condition or query-field selection. Patch application and the summary's test, vet, and formatting claims could not be independently rerun due the shell sandbox failure." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 19, + "input_tokens": 40, + "output_tokens": 17649, + "total_tokens": 1327603, + "latency_seconds": 158.0, + "total_cost_usd": 0.6447278199999998, + "result_subtype": "success", + "task_score": 86.0, + "cache_read_tokens": 1245756, + "cache_write_tokens": 64158, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 111.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue clearly defines affected commands, unchanged SHA-1 behavior, explicit exclusions, and testable outcomes for 40-, 64-, and unexpected-length refs. It identifies the REST ref endpoint and the digest's use in both error output and attestation lookup. Its largest limitation is that failing unknown lengths is a proposed policy rather than a requirement established by an independent task statement. The claimed SHA-256 rollout and linked issue could not be independently substantiated from the supplied task context." + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The LLD commits to a concrete signature, length mapping, error behavior, data flow, callers, and file-level changes. Its references to `NewDigestedArtifactForRelease`, `FetchParams.Digest`, `FetchRefSHA`, and `WithArtifactDigest` are internally consistent with the submitted patch. The main weakness is limited analysis of malformed 40- or 64-character non-hex values and future object formats, although both choices are acknowledged. Claims that exactly two callers exist and no stable repository-format field is available were asserted but not independently demonstrated in the artifacts." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 21, + "total": 85, + "notes": "The review catches meaningful defects: silent SHA-1 fallback, signature-change blast radius, and tests that inspect only user-facing errors rather than `FetchParams.Digest`. It provides concrete resolutions that appear in the revised LLD and patch, including bad-length command tests using a fail-on-call client. Its largest deficiency is that several reviewer sections are superficial and it does not challenge length-only classification or directly testing the verifier boundary. Assertions that the ref API always returns full IDs and lacks a stable object-format field remain unsupported within the submission." + }, + "testing": { + "completeness": 20, + "correctness": 22, + "specificity": 23, + "risk_awareness": 19, + "total": 84, + "notes": "The plan supplies exact package commands, matching test names, concrete error-string and lookup-digest oracles, negative cases, and a realistic optional SHA-256 repository check. The patch implements these tests with `StubFetchRefSHA`, `MockClient.OnGetByDigest`, and `NewFailTestClient`, so they would detect the central prefix regression and premature lookup. The largest gap is that it does not directly exercise the selected algorithm through the verifier despite an acceptance criterion naming that path. It also omits the repository-prescribed full `go test ./...` and `make lint` checks, relying on narrower package tests and `gofmt`." + }, + "implementation": { + "completeness": 22, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 87, + "notes": "The patch implements the design end to end by centralizing 40/64-length inference in `artifact.NewDigestedArtifactForRelease`, propagating constructor errors in both commands, and testing output plus lookup parameters. The unified diff consistently targets the constructor and the `verifyRun` and `verifyAssetRun` call sites, while preserving the existing SHA-1 path and preventing lookup for unsupported lengths. Its largest limitation is the absence of a command-level test proving that `sha256` reaches the verifier, and length-valid non-hex refs are intentionally accepted. The summary accurately describes the patch, though its claims that tests, vet, formatting, and `git apply --check` passed are execution claims not independently evidenced by the artifact." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "ref": "v2.97.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 13, + "input_tokens": 28, + "output_tokens": 12458, + "total_tokens": 795202, + "latency_seconds": 96.3, + "total_cost_usd": 0.42846771, + "result_subtype": "success", + "task_score": 85.6, + "cache_read_tokens": 735613, + "cache_write_tokens": 47103, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 129.4, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 21, + "specificity": 24, + "risk_awareness": 18, + "total": 86, + "notes": "The issue provides byte-exact TTY, non-TTY, JSON, and error acceptance criteria plus clear exclusions. Its description correctly identifies `printResults` in `pkg/cmd/project/item-add/item_add.go` as the change point. The main weakness is claiming the output distinguishes a mutation from a no-op, although returning an item ID does not establish whether the item already existed. The cited 2-to-9-call statistic could not be verified, and no independent task statement was supplied, though the v2.98.0 release confirms this command's non-TTY output was fixed. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 19, + "total": 88, + "notes": "The LLD is grounded in concrete files, functions, data flow, exporter ordering, and the existing `ProjectItem.Id` field. Its proposed branch is implementation-ready and preserves TTY and formatted output without another API request. The largest gap is limited treatment of compatibility for scripts expecting empty stdout and the possibility of a successful response yielding an empty ID. Exact line estimates and the complete existing test inventory could not be independently verified." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 21, + "risk_awareness": 21, + "total": 83, + "notes": "The review identifies concrete concerns including empty IDs, changed script output, sibling-command inconsistency, and the need for distinct content and project-item IDs in testing. The distinct-ID recommendation directly improves regression detection. Its largest deficiency is approving without requiring a unit test for the error/no-confirmation criterion or noticing the repository rule to use `require` for error checks. It also dismisses empty stdout as not being a contract and assumes successful mutations always provide a nonempty ID without demonstrated evidence." + }, + "testing": { + "completeness": 21, + "correctness": 21, + "specificity": 23, + "risk_awareness": 18, + "total": 83, + "notes": "The plan supplies valid package-level commands and exact output oracles for TTY, non-TTY, JSON, and live item-ID verification. The non-TTY unit scenario deliberately separates `content ID` from `PVTI_item`, which would catch printing the wrong identifier. The negative case is only a vague manual instruction and does not concretely capture stdout or unit-test the required absence of confirmation. The checked claims that tests passed were not independently verifiable, and the live test has no cleanup or explicit handling for an already-present item." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The patch implements the designed behavior directly in `printResults` and adds an end-to-end non-TTY test whose mutation returns `PVTI_item` while the source content uses `content ID`. The exporter and TTY branches remain unchanged, and the upstream v2.98.0 release independently corroborates that this command required a non-TTY output fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) The clearest repository-convention defect is `assert.NoError(t, err)` instead of the mandated `require.NoError`, and no automated negative-output test was added. Patch applicability and the claimed test execution could not be independently run, but the submitted hunk paths, symbols, and surrounding context are internally consistent with the baseline." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "release-create-404-scope-blame", + "complexity": "low", + "ref": "v2.97.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 16, + "input_tokens": 36, + "output_tokens": 15801, + "total_tokens": 979798, + "latency_seconds": 129.6, + "total_cost_usd": 0.5097621099999999, + "result_subtype": "success", + "task_score": 85.6, + "cache_read_tokens": 915028, + "cache_write_tokens": 48933, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 121.9, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The issue clearly defines the parsing defect, affected branch, desired behavior, exclusions, and testable acceptance criteria. GitHub CLI v2.98.0 independently lists the corresponding fix, \"Trim spaces when parsing X-Oauth-Scopes in gh release create,\" as PR #14065. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) Its largest weakness is claiming that preserving the HTTP 404 reveals the real cause, when a generic 404 cannot distinguish a nonexistent repository from insufficient access. No independent task statement was supplied, so requirements beyond the task identifier and internally consistent artifacts remain unverifiable." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The design commits to a minimal exact-match parser change, names concrete files and symbols, and specifies compatibility and negative cases. The selected behavior agrees with the fix documented in the v2.98.0 release notes. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) Its error-handling statement about \"non-workflow 404s\" is imprecise because the API response cannot establish the cause, and the true-missing-scope branch still deliberately masks the HTTP error. The stated baseline line locations, Go version, and test execution could not be independently verified from the local tree because the read-only command sandbox failed to initialize." + }, + "review": { + "completeness": 20, + "correctness": 21, + "specificity": 21, + "risk_awareness": 20, + "total": 82, + "notes": "The review catches two concrete regression risks: substring false positives and tests that fail to reproduce comma-space headers. It also identifies the continuing masked-error behavior for genuinely missing scopes and records it as an explicit limitation. Several persona sections are superficial approvals, and the review does not challenge the environment-specific test path or the claimed execution checklist. Claims about surrounding shared parsers and repository-wide conventions could not be fully verified from the inaccessible local source." + }, + "testing": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 18, + "total": 82, + "notes": "The plan provides focused unit and command-level tests with concrete headers, exact Boolean outcomes, error text, stderr expectations, and backward-compatibility cases. Its package path and test names agree with the submitted patch, and the documented auth command does support removing scopes. ([cli.github.com](https://cli.github.com/manual/gh_auth_refresh)) The hard-coded clone directory is not portable, while the negative manual procedure mutates stored credentials without specifying restoration. The checklist marks functional testing complete despite explicitly saying the included live check was not run, and the claimed test results could not be independently executed." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the designed trim-plus-exact-match behavior and adds both helper-level edge cases and command-level regression coverage. The official v2.98.0 release record confirms that this precise defect was fixed in the cited release-create path. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0)) No substantive design deviation is apparent, but local source matching and git-apply validation could not be independently performed because command execution was unavailable. The summary also overstates verification slightly because `gofmt -l` without file arguments does not check the repository, although this does not undermine the patch logic." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 42, + "input_tokens": 86, + "output_tokens": 36237, + "total_tokens": 4167420, + "latency_seconds": 311.7, + "total_cost_usd": 1.5912100599999999, + "result_subtype": "success", + "task_score": 85.2, + "cache_read_tokens": 4019013, + "cache_write_tokens": 112084, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 116.3, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The issue clearly defines both affected message sites, token categories, host behavior, testable acceptance criteria, and focused non-goals. Its largest weakness is describing the source-plus-prefix heuristic as proof that a token was created by gh auth login, although a stored user-supplied gho_ token is indistinguishable. The core requirement agrees with issue #13066, and the named internal/ghcmd/cmd.go and auth status paths match the submitted repository contexts. No independent task statement establishes whether every added status behavior was mandatory, so that scope remains partly unverifiable." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The LLD is unusually concrete about real symbols and data flow, including HTTPError.RequestURL, NormalizeHostname, ActiveToken, buildEntry, and the rewritten TokenSource field. Its main correctness limitation is that IsLoginToken is only a conservative heuristic, while the prose repeatedly presents it as exact provenance; acknowledged unprefixed GHES tokens also miss the desired refresh hint. The active-account restriction, 401 gate, configuration fallback, environment-token handling, and request-host normalization address the important failure paths. Line estimates and the assertion that keyring access cannot newly prompt were not independently verifiable." + }, + "review": { + "completeness": 21, + "correctness": 21, + "specificity": 22, + "risk_awareness": 22, + "total": 86, + "notes": "The review finds three material design defects: inactive-account refresh suggestions, PAT misclassification by source alone, and applying the status change to non-401 failures. It ties each finding to concrete implementation choices such as opts.active, the gho_ prefix, opts.tokenSource, and HTTPError.StatusCode. Its largest gap is limited challenge of the remaining provenance heuristic and the absence of an end-to-end assertion around Main's actual 401 output. The security discussion is specific, although the claim that a repeated keyring lookup cannot prompt again was not verified." + }, + "testing": { + "completeness": 20, + "correctness": 20, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan supplies concrete commands and expected commands for GitHub.com, GHES, tenancy hosts, environment tokens, PATs, nil inputs, inactive accounts, and non-401 failures. The strongest oracle is the exact auth status stderr output, supplemented by a realistic revoke-refresh-retry manual flow. The largest gap is that Test_reauthCommand tests only the selector and does not prove Main prints the selected command after an actual 401; the JSON byte-identity claim also lacks a named automated test. It ends with go vet rather than the repository-required make lint and provides limited operational checking beyond the manual flow." + }, + "implementation": { + "completeness": 21, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 84, + "notes": "The patch implements both message sites, centralizes the source-plus-prefix predicate, normalizes the failed request host, preserves login fallbacks, and restricts auth status refresh advice to active accounts receiving 401. The submitted contexts and symbols are consistent with internal/ghcmd/cmd.go, authEntry/buildEntry, shared helpers, and the repository's table-driven testing conventions; the feature also corresponds to merged PR #13068. The largest defect is that a gho_ token supplied manually into gh-managed storage is classified as login-created, while older unprefixed GHES login tokens are classified oppositely. The patch does not directly test Main's formatted 401 output, and its claimed reverse apply check could not be independently reproduced in the available execution environment." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "low", + "ref": "v2.92.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 13, + "input_tokens": 28, + "output_tokens": 13896, + "total_tokens": 828951, + "latency_seconds": 106.0, + "total_cost_usd": 0.46489927, + "result_subtype": "success", + "task_score": 85.0, + "cache_read_tokens": 762586, + "cache_write_tokens": 52441, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 131.1, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 20, + "total": 90, + "notes": "The issue clearly identifies the false-positive exit status, its scripting impact, and the field-dependent behavior. Its explanation matches `finder.Find` and the `findByNumber` path in `pkg/cmd/pr/shared/finder.go`. The acceptance criteria are concrete, although \"same behavior\" for `#N` and URL selectors does not explicitly enumerate both positive and negative cases. No independent task statement was supplied, so the related issue and intended scope could not be independently confirmed." + }, + "lld": { + "completeness": 22, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The LLD is implementation-ready, naming the correct finder branch, query helper, tests, callers, and data flow. Public source corroborates that URL parsing establishes `baseRefRepo` and `prNumber`, and that positive numbers flow through `findByNumber`. It appropriately discusses the extra API request, changed failure point, rollout, and alternatives. Its exact GitHub error text, complete caller inventory, and rate-limit implications were not fully verifiable, and it does not require the repository-wide test and lint commands." + }, + "review": { + "completeness": 19, + "correctness": 20, + "specificity": 20, + "risk_awareness": 18, + "total": 77, + "notes": "The review identifies concrete blast-radius concerns, including number-only callers, URL selection, merge refs, API cost, and the obsolete no-HTTP test. The `pr diff` number-only finder usage is consistent with repository source. ([github.com](https://github.com/malor/gh-cli/blob/trunk/pkg/cmd/pr/diff/diff.go)) Its largest weakness is insufficient challenge: it misses the absent exact-error assertion, incomplete automated selector coverage, and repository-wide lint and test requirements. It also incorrectly says the release-note recommendation is reflected in `testing.md`, where no such checklist item appears." + }, + "testing": { + "completeness": 20, + "correctness": 19, + "specificity": 22, + "risk_awareness": 19, + "total": 80, + "notes": "The plan provides concrete commands and meaningful stdout, stderr, JSON, and exit-status oracles for both PR and issue cases. It covers direct numbers, `#N`, URLs, other field sets, finder unit tests, and affected command packages. The E2E snippet uses an undefined `$N`, while the automated negative test only requires some error rather than the promised specific not-found error. It also omits `go test ./...` and `make lint`, and the hard-coded live issue and PR identities could not be independently verified." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 20, + "total": 89, + "notes": "The patch implements the design directly by removing the unvalidated number-only return and routing every positive `prNumber` through `findByNumber`; current upstream source reflects that same control flow. The target files, symbols, and surrounding context are consistent with the submitted baseline, and the added tests cover successful lookup, GraphQL rejection, and URL repository selection. The largest gap is that tests assert only `wantErr`, omit automated `#N` and URL-negative cases, and therefore do not pin the exact accepted error behavior. The claimed test execution and clean local application could not be independently verified because the read-only command sandbox failed before executing commands." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-set-default-unset-without-repo", + "complexity": "trivial", + "ref": "v2.101.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 8, + "input_tokens": 18, + "output_tokens": 11050, + "total_tokens": 476132, + "latency_seconds": 77.9, + "total_cost_usd": 0.33237743000000003, + "result_subtype": "success", + "task_score": 83.8, + "cache_read_tokens": 422189, + "cache_write_tokens": 42875, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 141.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 20, + "total": 88, + "notes": "The issue clearly defines the non-interactive failure, retained behavior, testable acceptance criteria, and explicit non-goals. Its proposed guard change matches the documented `--view` and `--unset` command modes. The strongest evidence is the exact error and `pkg/cmd/repo/setdefault/setdefault.go` integration point; the largest gap is limited discussion of combined mode flags. No independent task statement was supplied, and issue #14521 could not be verified. ([github.com](https://github.com/cli/cli/issues/7442), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default))" + }, + "lld": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The LLD commits to the precise guard, execution ordering, files, test cases, compatibility boundary, and alternatives. Its named constructor, command path, factory pattern, and local-git precondition agree with repository conventions. It explicitly addresses nil-repository safety, ignored arguments, flag precedence, and unchanged errors. Exact line hashes and output wording could not be independently verified against the local checkout. ([errors.standardbeagle.com](https://errors.standardbeagle.com/cli/cli/must-be-run-from-inside-a-git-repository/), [github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md))" + }, + "review": { + "completeness": 18, + "correctness": 20, + "specificity": 19, + "risk_awareness": 18, + "total": 75, + "notes": "The review identifies concrete concerns including ignored repository arguments, `--view` precedence, local configuration mutation, and the prior non-TTY coverage gap. Its backend and SMTS observations are specific to the proposed guard and table-driven tests. The largest deficiency is that most reviewer sections merely approve, and none catches the testing plan's installed-binary and pipeline-exit problems. The security conclusion is plausible, but its exact no-network assertion was not independently verified." + }, + "testing": { + "completeness": 18, + "correctness": 16, + "specificity": 20, + "risk_awareness": 18, + "total": 72, + "notes": "The automated cases directly pin non-interactive `--unset`, retained `--view`, and the exact bare-command error, while referencing the existing execution test. The package path, test symbols, flags, and expected options match the submitted patch and repository command structure. The manual plan incorrectly invokes an installed `gh` instead of the required built `bin/gh`, and `gh ... | cat` cannot establish exit 1 without `pipefail` or inspecting `PIPESTATUS`. Backward-compatibility cases are listed but lack concrete automated setup and oracles. ([github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md))" + }, + "implementation": { + "completeness": 24, + "correctness": 23, + "specificity": 24, + "risk_awareness": 22, + "total": 93, + "notes": "The one-line guard change implements the stated behavior end to end, and the tests exercise both exempt modes plus the preserved failure. The target paths, `NewCmdSetDefault`, option fields, injected run function, IOStreams TTY controls, and table-driven style are consistent with repository conventions. No material code defect or incompatible behavior is apparent, and the summary accurately describes the patch's scope. Exact patch applicability, index hashes, and the claimed reverse check could not be confirmed because local read commands failed before execution. ([github.com](https://github.com/cli/cli/blob/trunk/AGENTS.md), [cli.github.com](https://cli.github.com/manual/gh_repo_set-default))" + } + }, + "is_error": false, + "failed": false + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "low", + "ref": "v2.101.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 15, + "input_tokens": 32, + "output_tokens": 13950, + "total_tokens": 935269, + "latency_seconds": 115.7, + "total_cost_usd": 0.49468056000000005, + "result_subtype": "success", + "task_score": 83.6, + "cache_read_tokens": 866553, + "cache_write_tokens": 54734, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 120.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The issue crisply defines the client-side failure, unchanged token classes, exclusions, and testable success and failure criteria. Repository issue history corroborates that uploadTokenTypes is the credential choke point and previously permits OAuth, classic PAT, and fine-grained PAT tokens. ([github.com](https://github.com/cli/cli/issues/14309)) The largest evidence gap is the absent independent task statement: the asserted ghu_ 201 response, Bearer 404 response, and Copilot-session motivation could not be independently verified. Risk coverage is otherwise strong because it explicitly preserves ghs_, ghr_, unknown-token rejection, authorization scheme, endpoint, and GHES behavior." + }, + "lld": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 21, + "total": 87, + "notes": "The LLD is implementation-ready, naming the relevant files, symbols, call flow, exact allowlist change, error behavior, and focused tests. Its AddAuthTokenHeader claim matches repository evidence showing the transport formats credentials as Authorization: token . ([github.com](https://github.com/cli/cli/issues/13317)) The main deficiency is that the endpoint's ghu_ behavior and the assertion that every listed command traverses this exact path were not independently verifiable from the supplied task or accessible checkout. It appropriately preserves the allowlist, considers alternative policies, retains server-derived errors, and supplies a proportionate rollback plan." + }, + "review": { + "completeness": 20, + "correctness": 20, + "specificity": 21, + "risk_awareness": 20, + "total": 81, + "notes": "The strongest review work identifies three concrete regression guards: invert the existing ghu_ rejection row, assert the token authorization scheme, and exercise a server 404 through the comment path. The authorization concern is technically grounded because the repository transport explicitly sets the token scheme. ([github.com](https://github.com/cli/cli/issues/13317)) Several reviewer sections are largely ceremonial, and the review does not challenge the unverified premise that ghu_ acceptance is uniform or independently inspect all six claimed callers. The cross-host credential statement is also asserted without demonstrated repository evidence, although it does not undermine the recommended patch." + }, + "testing": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 16, + "total": 76, + "notes": "The plan provides unusually concrete rows, request counts, body values, exact error text, authorization-header oracles, negative token cases, and affected-package commands. Its endpoint shape agrees with existing attachment investigations, which use the same uploads host, query parameters, and 404 semantics. ([github.com](https://github.com/cli/cli/issues/14309)) The largest deficiency is operational: manual checks invoke gh rather than the required built binary and would create live assets/comments without naming an authorized disposable repository or cleanup procedure. The checked execution claims have no attached evidence, and the plan omits the repository's required full go fix, go test ./..., and make lint pre-commit gates." + }, + "implementation": { + "completeness": 23, + "correctness": 22, + "specificity": 23, + "risk_awareness": 21, + "total": 89, + "notes": "The patch implements the proposed behavior end to end: it adds only TokenTypeUserToServer to the allowlist, retains rejected-token rows, proves the token scheme, and covers both successful upload/comment creation and server-derived 404 behavior. The production change fits repository evidence that this stack uses a credential allowlist and Authorization: token headers, while first-party attachment support spans the six commands claimed. ([github.com](https://github.com/cli/cli/issues/14309), [github.com](https://github.com/cli/cli/issues/13317), [github.com](https://github.com/drogers0/gh-image/blob/main/README.md)) No material code-level defect is apparent in the submitted hunks, but the local shell failure prevented an independent git apply --check, compilation, or verification of the claimed test and gofmt results. The summary is accurate overall, though \u201cno follow-ups\u201d overstates completion when the real-token E2E check remains explicitly unrun and the endpoint premise is externally unverified." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-create-reject-empty-name", + "complexity": "trivial", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 8, + "input_tokens": 18, + "output_tokens": 8524, + "total_tokens": 452354, + "latency_seconds": 64.7, + "total_cost_usd": 0.35235277, + "result_subtype": "success", + "task_score": 81.0, + "cache_read_tokens": 380211, + "cache_write_tokens": 63601, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 131.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 90, + "notes": "The issue clearly describes the wasted interactive flow, expected fail-fast behavior, testable acceptance criteria, and deliberate exclusions. It specifically identifies the Repository name prompt, the malformed some-owner/ result, whitespace handling, and behavior that must remain unchanged. Its largest gap is that the independent task statement is absent, so the linked issue and exact historical failure could not be fully corroborated. It also does not explicitly decide whether an owner/ input with an empty repository component belongs in scope." + }, + "lld": { + "completeness": 22, + "correctness": 20, + "specificity": 23, + "risk_awareness": 20, + "total": 85, + "notes": "The LLD names the relevant create.go and create_test.go symbols and commits to a minimal check immediately after prompter.Input, making the implementation decision clear. Its data flow, error propagation, compatibility boundary, alternatives, and absence of migration or deployment work are proportionate to this small change. The main factual defect is its claim that an empty local-flow answer does not use the filepath.Base default, which conflicts with the review's more credible statement that pressing Enter returns the prompt default. Exact tagged-source line references and the stated PrompterMock behavior could not be independently verified because local shell execution failed before repository reads." + }, + "review": { + "completeness": 16, + "correctness": 18, + "specificity": 17, + "risk_awareness": 14, + "total": 65, + "notes": "The review identifies the principal UX tradeoff, confirms the shared-function strategy, and correctly questions how local prompt defaults behave. Its largest weakness is that nearly every role approves without materially stress-testing malformed owner/name forms, compatibility coverage, or the discrepancy it discovered in the LLD. The claim that nil PrompterMock functions prove every later prompt is unreachable is overstated because the submitted tests define InputFunc and SelectFunc and return explicit errors for unexpected prompts. Repository-backed verification of the mock's generated behavior was unavailable due the shell sandbox failure." + }, + "testing": { + "completeness": 19, + "correctness": 22, + "specificity": 22, + "risk_awareness": 18, + "total": 81, + "notes": "The plan supplies a concrete Go subtest command, exact expected error text, a manual CLI oracle, and a package-level compatibility run. The proposed negative tests would fail if owner, description, selection, or confirmation prompts were reached because unexpected prompt paths do not produce the expected error. Its largest omission is explicit coverage for owner/REPO and the directory-derived default promised by the acceptance criteria, relying instead on vaguely named existing cases. It also omits the repository-required go test ./... and make lint checks from its execution checklist." + }, + "implementation": { + "completeness": 21, + "correctness": 22, + "specificity": 22, + "risk_awareness": 19, + "total": 84, + "notes": "The patch implements the designed fail-fast check directly after the name prompt and preserves nonblank input unchanged, including surrounding whitespace on otherwise nonempty names. Its two Test_createRun cases check empty and whitespace-only input and ensure unexpected later prompts cannot yield the expected error, although the comment inaccurately says every such prompt would panic. The main coverage gap is the absence of new positive tests for owner/REPO and the local-directory default, plus no full-suite or lint verification. The target paths and diff context appear internally consistent, but git apply --check and direct source comparison could not be completed because every local command was blocked by an environment-level bwrap failure." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "high", + "ref": "v2.96.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 37, + "input_tokens": 80, + "output_tokens": 55448, + "total_tokens": 4078906, + "latency_seconds": 451.5, + "total_cost_usd": 1.7988092100000002, + "result_subtype": "success", + "task_score": 80.4, + "cache_read_tokens": 3903393, + "cache_write_tokens": 119985, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 122.8, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 19, + "specificity": 23, + "risk_awareness": 19, + "total": 83, + "notes": "The issue defines a concrete readable invocation, backward-compatibility requirements, testable errors, and useful non-goals. The official command documentation confirms the number/owner/URL/field flow, but describes `--value` specifically as a single-select option name, while this issue expands it to text, number, date, and iteration without independent task evidence. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit), [github.com](https://github.com/cli/cli/releases/)) That unsupported scope expansion is its largest deficiency because it substantially enlarges API and compatibility risk. No independent task statement was supplied, and the claimed related issue and several API-limit assumptions could not be verified from the unavailable local execution environment." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 82, + "notes": "The LLD is unusually concrete about `item_edit.go`, shared query helpers, mutation reuse, lookup ordering, pagination, ambiguity handling, and failure-before-mutation behavior. Its principal weakness is designing all-type `--value` interpretation when the established command surface documents that flag for single-select values, creating unnecessary iteration queries and parsing semantics. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit)) It also retains contradictory text saying one page of 100 project items is assumed after resolving that issue with pagination, and it misses the silent `--id` plus `--title`/`--body` plus `--field`/`--value` combination. Exact application against the tagged tree and claims about pinned shared-query tests could not be verified because local read commands failed before execution." + }, + "review": { + "completeness": 21, + "correctness": 20, + "specificity": 22, + "risk_awareness": 20, + "total": 83, + "notes": "The review finds several real, consequential defects: project-item pagination, empty-value dispatch, misleading candidates, and silently ignored `--url` with draft-edit flags. Those findings are actionable and are visibly reflected in the submitted diff through pagination, empty-value validation, and new flag errors. Its largest miss is the analogous `--id DI_... --title ... --field ... --value ...` path, which still resolves a field and then silently executes only the draft update; it also never challenges the unsupported all-data-type expansion. Claims about shared query-test breakage and field-count limits remained unverifiable against the local checkout." + }, + "testing": { + "completeness": 21, + "correctness": 19, + "specificity": 22, + "risk_awareness": 18, + "total": 80, + "notes": "The plan has strong concrete oracles for mutation payloads, case-insensitive matching, pagination, clear behavior, invalid values, candidate-list errors, and legacy ID paths. The manual setup builds `bin/gh` but invokes bare `gh`, so it can accidentally test an installed binary, and several multi-type checks merely say `item-list` shows the value without an exact query or expected representation. The automated tests omit the mixed draft-edit and name-based-field case that exposes silent flag loss, and they use `assert` for error checks despite the repository instruction to use `require`. The reported test execution and live GraphQL compatibility were not independently verifiable." + }, + "implementation": { + "completeness": 20, + "correctness": 17, + "specificity": 21, + "risk_awareness": 16, + "total": 74, + "notes": "The diff implements the submitted design end to end with new flags, owner/project lookup, field and option matching, iteration lookup, paginated item resolution, mutation reuse, and substantial focused tests. Its largest concrete defect is that validation rejects `--url` with `--title` or `--body` but permits an ID-addressed draft edit combined with `--field` and `--value`; after resolution, `runEditItem` takes the draft branch and silently ignores the requested field update. The patch also broadens `--value` beyond the officially documented single-select behavior, uses `assert` rather than required fatal assertions, and admits that the mandatory linter was not run. ([cli.github.com](https://cli.github.com/manual/gh_project_item-edit)) The patch's application, compilation, and claimed test results could not be independently checked because the local read-only command sandbox failed before running commands." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "high", + "ref": "v2.94.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 27, + "input_tokens": 56, + "output_tokens": 24732, + "total_tokens": 2304215, + "latency_seconds": 212.1, + "total_cost_usd": 0.9829560399999998, + "result_subtype": "success", + "task_score": 79.8, + "cache_read_tokens": 2196697, + "cache_write_tokens": 82730, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 116.6, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The issue provides a reproducible namespaced layout, explicit expected paths, non-goals, and testable acceptance criteria for custom and agent directories. Its explanation matches the submitted baseline context in pkg/cmd/skills/update/update.go, including base derivation and os.RemoveAll cleanup. The largest deficiency is that promising every failed update leaves the installation intact conflicts with declaring atomic replacement out of scope, since an in-place multi-file rewrite can partially succeed. No independent task statement was supplied, and the related issue link and reproduction could not be independently exercised." + }, + "lld": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 18, + "total": 78, + "notes": "The design is unusually concrete about installedSkill.dir, installer.Options, filepath.Dir/Base, affected files, validation, and removal of cleanup. The chosen mapping correctly preserves flat, namespaced, and locally renamed paths for GitHub-backed updates. Its largest gap is knowingly leaving partial writes possible despite the issue's failed-update acceptance criterion; --force is recovery guidance, not prevention. DirName is implemented only in installSkill while installLocalSkill remains unchanged, so the stated general installer option does not consistently govern every installer path." + }, + "review": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 20, + "total": 80, + "notes": "The review finds concrete defects rather than merely approving: multi-skill DirName collisions, partial in-place writes, locally renamed directories, and skipped failure verification. Its most important finding correctly explains how a failed multi-file update could leave mixed content and an advanced tree SHA. The main deficiency is approving that risk even though it contradicts an explicit issue acceptance criterion. The security claim is also too broad because filepath.Join(baseDir, DirName) is not itself constrained to one path element, even though the update caller currently supplies filepath.Base." + }, + "testing": { + "completeness": 18, + "correctness": 19, + "specificity": 23, + "risk_awareness": 16, + "total": 76, + "notes": "The plan has strong path and content oracles for namespaced, unrelated-namespace, renamed, agent-host, flat, and no-relocation behavior. Its failure test makes the only blob fetch fail before any write, so it cannot detect the acknowledged partial-write regression where an earlier file was already overwritten. It also omits the SourceDir installer path and the repository-required full go test ./... and make lint checks. The documented --dir, --all, --force, and --dry-run flags agree with the official command manual, but the claimed executions were not independently verifiable. ([cli.github.com](https://cli.github.com/manual/gh_skill_update))" + }, + "implementation": { + "completeness": 18, + "correctness": 18, + "specificity": 22, + "risk_awareness": 16, + "total": 74, + "notes": "The patch directly implements the core path-preservation rule by passing filepath.Dir(u.local.dir) and filepath.Base(u.local.dir), and it removes the destructive relocation cleanup. The focused tests cover the principal successful update layouts and preserve extra local files. The largest defect is that a later-file failure can still leave a partially updated installation, contrary to the submitted acceptance criteria; additionally, DirName is ignored by installLocalSkill and is not validated as a single safe path element. The hunk symbols and contexts are internally consistent, but the managed sandbox could not launch git apply --check or tests, so applicability and execution claims could not be independently confirmed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "medium", + "ref": "v2.95.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 26, + "input_tokens": 58, + "output_tokens": 33605, + "total_tokens": 2501272, + "latency_seconds": 271.3, + "total_cost_usd": 1.1769682099999998, + "result_subtype": "success", + "task_score": 79.4, + "cache_read_tokens": 2363138, + "cache_write_tokens": 104471, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 123.9, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 21, + "specificity": 23, + "risk_awareness": 20, + "total": 86, + "notes": "The issue provides a reproducible command, observable failures, scoped non-goals, and testable acceptance criteria matching the reported bug. The public issue and merged fix confirm the errgroup concurrency, shared connection, shallow struct copy, and unsafe TunnelManager access. ([github.com](https://github.com/cli/cli/issues/13399), [github.com](https://github.com/cli/cli/pull/13313)) Its largest weakness is that several criteria, especially manager encapsulation and the ten-port threshold, prescribe this solution despite the absence of an independent task statement. The additional URL, tunnel-slice, and notification-channel root-cause claims were plausible but not all independently corroborated by the repository evidence available." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 19, + "total": 81, + "notes": "The LLD is unusually concrete about real files, ownership, pointer sharing, wrapper signatures, lock scope, tests, and alternatives; the pointer and manager-mutex choices agree with the merged repository fix. ([github.com](https://github.com/cli/cli/pull/13313)) Its largest defect is releasing managerMu between GetTunnelPort and DeleteTunnelPort in UpdatePortVisibility, leaving a logical time-of-check/time-of-use race that the actual merged change specifically corrected by locking the complete get/check/delete sequence. ([github.com](https://github.com/cli/cli/pull/13313)) The waitMu tradeoff is acknowledged, but it expands scope and can ignore cancellation while queued or block healthy ports behind one missing notification. Rollout and compatibility treatment are otherwise proportionate for an internal locking change with no migration." + }, + "review": { + "completeness": 19, + "correctness": 17, + "specificity": 21, + "risk_awareness": 18, + "total": 75, + "notes": "The review catches concrete issues including copied mutexes, unprotected manager call sites, the concurrent mock map, and wait serialization head-of-line blocking. Its largest omission is the UpdatePortVisibility TOCTOU defect later identified and fixed in the real pull request, so the claim that all blocking findings were resolved is too strong. ([github.com](https://github.com/cli/cli/pull/13313)) The statement that repository CI already runs go test -race was not established by the supplied repository instructions, which instead prescribe go test ./... and make lint. The frontend and security sections are mostly ceremonial, while the backend and SRE sections provide the substantive value." + }, + "testing": { + "completeness": 19, + "correctness": 20, + "specificity": 22, + "risk_awareness": 17, + "total": 78, + "notes": "The plan gives valid package-level race commands, exact existing test names, a meaningful oracle that all ten ports are listed, and regression coverage for single-port behavior. This closely matches the merged fix's concurrent ten-port regression-test strategy. ([github.com](https://github.com/cli/cli/pull/13313)) The manual E2E loop curls only port 8080 while claiming every local port accepts connections, and an arbitrary HTTP status is a weak forwarding oracle. WaitForForwardedPort remains untested, and there is no coverage for queued-context cancellation or the concurrent visibility-update TOCTOU." + }, + "implementation": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 17, + "total": 77, + "notes": "The patch implements the central fix end to end: forwarders retain a shared connection pointer, TunnelManager operations are mutex-protected, and a ten-port concurrent regression test is added; these targets and symbols agree with the repository's merged solution. ([github.com](https://github.com/cli/cli/pull/13313)) Encapsulating manager calls in connection methods is coherent, and the mock lock removes a test-only race. The largest defect is that UpdatePortVisibility performs its get/check/delete sequence through separately locked calls, preserving the TOCTOU race that the merged implementation explicitly fixed atomically. ([github.com](https://github.com/cli/cli/pull/13313)) The additional wait mutex has no automated test and introduces acknowledged blocking behavior; local sandbox failure also prevented an independent apply or compile check, consistent with the summary's admission that the patch was not compiled." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 22, + "input_tokens": 46, + "output_tokens": 18650, + "total_tokens": 1602620, + "latency_seconds": 159.4, + "total_cost_usd": 0.7205350899999999, + "result_subtype": "success", + "task_score": 78.2, + "cache_read_tokens": 1517987, + "cache_write_tokens": 65937, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 117.0, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The issue clearly defines the silent omission, affected commands, exclusions, and testable outcomes. Its examples use documented --comments and --json options and distinguish all relevant flag combinations. ([cli.github.com](https://cli.github.com/manual/gh_issue_view)) The largest concern is that it preserves --comments with --json comments, while the published v2.99.0 fix is described simply as rejecting --comments with --json. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Because no independent task statement was supplied, that compatibility exception cannot be confirmed as required." + }, + "lld": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 20, + "total": 83, + "notes": "The LLD names concrete files, symbols, control flow, error type, tests, alternatives, and rollout behavior. Its use of RunE validation, Exporter.Fields, and FlagErrorf is consistent with the repository's documented command conventions. ([github.com](https://github.com/cli/cli/blob/trunk/docs/command-development.md)) The main deficiency is committing to a field-sensitive shared helper rather than establishing why that policy should differ from the released unconditional rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) Direct source inspection was impeded by the local shell sandbox, so several detailed baseline claims could not be independently verified beyond the supplied patch context." + }, + "review": { + "completeness": 19, + "correctness": 18, + "specificity": 21, + "risk_awareness": 18, + "total": 76, + "notes": "The review finds two concrete design defects: breaking a previously accepted combination and duplicating validation across three commands. Byte and Sage provide actionable resolutions that are reflected consistently in the revised LLD. Its largest gap is that it never challenges whether the compatibility exception or expansion to three commands matches the actual requested behavior, despite the release describing an unqualified rejection. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The remaining role-based approvals add little risk analysis beyond confirming that no network, secret, or deployment surface is introduced." + }, + "testing": { + "completeness": 18, + "correctness": 15, + "specificity": 20, + "risk_awareness": 17, + "total": 70, + "notes": "The plan identifies exact package tests, command combinations, exit status, error text, and the expected comments key. Its largest correctness defect is exporting GH_TOKEN=dummy while expecting accepted commands that perform API requests to succeed, so those manual checks cannot establish the stated outcome. The plan also lacks an explicit oracle proving rejected combinations make no HTTP request, and only spells out accepted smoke commands for issue view. Claims that tests and smoke checks passed were not independently verifiable." + }, + "implementation": { + "completeness": 21, + "correctness": 18, + "specificity": 22, + "risk_awareness": 18, + "total": 79, + "notes": "The patch implements its own design end to end by adding one typed flag-error helper, wiring all three NewCmdView RunE paths, and adding focused table tests. The target paths and symbols are consistent with the supplied baseline context, and the v2.99.0 release confirms that this command area received the corresponding fix. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0)) The main behavioral risk is permitting --comments --json comments when the published fix and post-fix user reports describe the pair as rejected outright. ([github.com](https://github.com/cli/cli/releases/tag/v2.99.0), [github.com](https://github.com/mattpocock/skills/issues/733)) The environment prevented git apply --check and test execution, so applicability and the summary's verification claims remain unconfirmed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "high", + "ref": "v2.98.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 27, + "input_tokens": 58, + "output_tokens": 32010, + "total_tokens": 2215982, + "latency_seconds": 249.7, + "total_cost_usd": 1.0418259499999998, + "result_subtype": "success", + "task_score": 75.2, + "cache_read_tokens": 2101255, + "cache_write_tokens": 82659, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 128.2, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 22, + "correctness": 18, + "specificity": 23, + "risk_awareness": 21, + "total": 84, + "notes": "The issue clearly defines the affected branch decision, desired worktree states, dirty and missing-directory behavior, compatibility cases, and explicit non-goals. Its largest defect is that the reproduction never advances the upstream branch, so as written it does not reproduce the reported staged changes. The submitted baseline diff context supports the diagnosis that a non-current existing branch reaches UpdateBranch with FETCH_HEAD. Because no independent task statement exists, the policy choice to modify another worktree rather than refuse cannot be externally verified." + }, + "lld": { + "completeness": 21, + "correctness": 18, + "specificity": 23, + "risk_awareness": 16, + "total": 78, + "notes": "The design is implementation-ready in its file mapping, interface changes, control flow, FETCH_HEAD resolution, compatibility branches, and error behavior. The cited git.Client Copy method and RepoDir field exist, supporting the proposed cross-worktree client construction. ([pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git)) Its largest deficiencies are failure to parse quoted or newline-containing worktree paths and specifying a copy-paste shell command using Go %q, which is not shell-safe for every path; Git recommends NUL-delimited porcelain parsing for unusual paths. ([github.com](https://github.com/git/git/blob/master/Documentation/git-worktree.adoc)) The absent independent task statement also leaves the broader auto-update policy unverifiable." + }, + "review": { + "completeness": 19, + "correctness": 16, + "specificity": 21, + "risk_awareness": 16, + "total": 72, + "notes": "The review's strongest contribution is identifying the per-worktree FETCH_HEAD problem and requiring resolution to a commit ID before changing directories. It also addresses missing worktree directories, dirty-state protection, command argument separation, and the residual TOCTOU window with concrete consequences. Its largest error is declaring no injection surface while approving a user-copyable `git -C %q stash --all` tip whose Go quoting is not general shell quoting; it also misses the porcelain path-decoding gap. The deferred rebase and bisect limitation is honestly recorded, but no independent requirement confirms that accepting it is sufficient." + }, + "testing": { + "completeness": 20, + "correctness": 14, + "specificity": 22, + "risk_awareness": 17, + "total": 73, + "notes": "The plan provides strong behavioral oracles for refs, HEAD, file contents, status cleanliness, unchanged dirty state, force behavior, parser exactness, and backward-compatible paths. Its largest flaw is that the missing-directory test detaches the calling worktree even though executeLocalRepoSync calls CurrentBranch before CheckedOutWorktree and the git client defines detached HEAD as ErrNotOnAnyBranch. ([pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git), [github.com](https://github.com/cli/cli/pull/7082)) The manual reproduction again omits an upstream update, and one error check uses assert rather than the repository-required require. The claimed test execution could not be independently verified." + }, + "implementation": { + "completeness": 19, + "correctness": 15, + "specificity": 21, + "risk_awareness": 14, + "total": 69, + "notes": "The patch implements its selected design across the git abstraction, sync control flow, mocks, command help, focused unit cases, and real-repository fixtures. Its baseline identifier matches the v2.98.0 release commit, and the proposed Copy and RepoDir APIs exist. ([github.com](https://github.com/cli/cli/releases/tag/v2.98.0), [pkg.go.dev](https://pkg.go.dev/github.com/cli/cli/v2/git)) The largest defects are the shell-unsafe copy-paste remediation text and incomplete handling of porcelain paths requiring quoting, while the missing-directory fixture likely exits at CurrentBranch because it deliberately detaches the caller. Local patch application and test execution were unavailable because the repository command runner failed before process launch, so the summary's green verification claims remain unsupported." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "auth-clipboard-config-default", + "complexity": "high", + "ref": "v2.100.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 55, + "input_tokens": 112, + "output_tokens": 40851, + "total_tokens": 5558100, + "latency_seconds": 356.3, + "total_cost_usd": 1.96467524, + "result_subtype": "success", + "task_score": 71.6, + "cache_read_tokens": 5398047, + "cache_write_tokens": 119090, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 114.7, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 19, + "correctness": 14, + "specificity": 23, + "risk_awareness": 16, + "total": 72, + "notes": "The issue has unusually concrete acceptance criteria, precedence rules, exclusions, and compatibility intent. Its largest defect is specifying an unset default of disabled, while the post-fix official manual documents clipboard as enabled by default. ([cli.github.com](https://cli.github.com/manual/gh_config)) The linked upstream issue supports adding persistent configuration but does not establish the candidate's default, host scoping, or explicit-false requirements. ([github.com](https://github.com/cli/cli/issues/13153)) Because no independent task statement was supplied, those additional requirements remain unverifiable rather than inherently invalid." + }, + "lld": { + "completeness": 20, + "correctness": 14, + "specificity": 23, + "risk_awareness": 18, + "total": 75, + "notes": "The LLD is strongly grounded in named files and commits to a clear precedence flow, interface changes, mock wiring, compatibility behavior, and alternatives. Its central design diverges from the available fixed implementation: it chooses a host-scoped Clipboard(hostname) API plus parallel bool/change fields, whereas trunk uses a global Clipboard() lookup and a nullable flag value. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go)) It also selects disabled as the default despite the official post-fix configuration documentation stating enabled. ([cli.github.com](https://cli.github.com/manual/gh_config)) Exact patch applicability and the claim that cfg and ConfigMock are the only interface implementers could not be independently verified because local shell inspection was blocked." + }, + "review": { + "completeness": 18, + "correctness": 14, + "specificity": 21, + "risk_awareness": 18, + "total": 71, + "notes": "The review identifies concrete concerns including direct-option construction hazards, unavailable clipboard utilities, local clipboard exposure, generated-mock drift, and missing login wiring coverage. Its largest failure is approving the load-bearing default and host-scoped API without challenging their divergence from the fixed global, default-enabled behavior. ([cli.github.com](https://cli.github.com/manual/gh_config), [github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go)) The SRE conclusion that upgrades cause no behavior change is particularly questionable given that documented default. Generator parity is acknowledged honestly but remains unverified." + }, + "testing": { + "completeness": 19, + "correctness": 14, + "specificity": 22, + "risk_awareness": 18, + "total": 73, + "notes": "The plan supplies concrete setup, commands, expected output, precedence truth tables, invalid-value coverage, host cases, rollback checks, and manual end-to-end scenarios. Its default-config oracle expects no copying, contradicting the post-fix documented default of enabled and therefore validating the wrong central behavior. ([cli.github.com](https://cli.github.com/manual/gh_config)) Automated tests do not exercise the final loginRun CopyToClipboard argument, so they could pass with the login call site broken; only refresh has an injected AuthFlow oracle. The alternative go build ./cmd/gh prerequisite also does not produce the configured ./bin/gh path." + }, + "implementation": { + "completeness": 18, + "correctness": 12, + "specificity": 22, + "risk_awareness": 15, + "total": 67, + "notes": "The patch implements its own design across configuration, interface and mock wiring, both auth commands, help text, and focused tests. Its largest defect is implementing disabled-by-default, host-scoped behavior when the available fixed implementation uses a global nullable-flag helper and the official configuration surface says default enabled. ([github.com](https://github.com/cli/cli/blob/trunk/pkg/cmd/auth/shared/login_flow.go), [cli.github.com](https://cli.github.com/manual/gh_config)) The diff is internally coherent and tests refresh precedence, but login's final wiring has no executable unit-test seam. Exact application, formatting, compilation, and generated-mock equivalence could not be verified locally; the summary accurately discloses that tests and regeneration were not completed." + } + }, + "is_error": false, + "failed": false + }, + { + "task": "skill-install-nested-discovery", + "complexity": "high", + "ref": "v2.93.0", + "artifacts_produced": 6, + "artifacts_expected": 6, + "num_turns": 51, + "input_tokens": 108, + "output_tokens": 61282, + "total_tokens": 6450900, + "latency_seconds": 500.3, + "total_cost_usd": 2.4782183800000004, + "result_subtype": "success", + "task_score": 64.8, + "cache_read_tokens": 6232124, + "cache_write_tokens": 157386, + "prefix_cache_hit_rate": null, + "generation_tokens_per_sec": 122.5, + "kv_cache_usage": null, + "agent_invocations": 1, + "topped_up_artifacts": [], + "eval_scores": { + "github_issue": { + "completeness": 19, + "correctness": 13, + "specificity": 22, + "risk_awareness": 13, + "total": 67, + "notes": "The issue provides concrete path examples, testable acceptance criteria, and explicit non-goals. Its largest defect is expanding automatic discovery to every non-hidden SKILL.md, whereas the repository's actual fix deliberately supports non-standard layouts only through an explicit exact path. ([github.com](https://github.com/cli/cli/pull/13459)) It correctly identifies that prefixed skills directories already work and that nested hidden directories do not. No independent task statement establishes the claimed issue-comment requirements for catalog-wide catch-all discovery." + }, + "lld": { + "completeness": 18, + "correctness": 12, + "specificity": 21, + "risk_awareness": 14, + "total": 65, + "notes": "The LLD is well grounded in real integration points such as discovery.go, matchSkillByName, selectSkill, publish, and MatchSkillPath. Its critical error is leaving isSkillPath unchanged, so an arbitrary path such as packages/agent-skills/x requires full-tree discovery instead of the direct lookup implemented by the repository fix; this also preserves failure on truncated large trees. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) The catch-all design additionally contradicts the repository's stated decision not to broaden search or publish behavior, while its vendor and testdata exclusions contradict the issue's own acceptance boundary. The scaling discussion therefore understates both picker expansion and exact-path traversal risk." + }, + "review": { + "completeness": 19, + "correctness": 16, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review's strongest work is finding concrete defects B1, B2, C1, S1, and S3 and recording actionable resolutions. It misses the central product and implementation flaw: automatic catch-all discovery is broader than the repository's intended explicit-path behavior, and arbitrary exact paths still do not take the direct lookup route. ([github.com](https://github.com/cli/cli/pull/13459)) Circuit's claim that path lookup works at any depth is therefore incomplete for large or truncated trees. The summary also says four blocking findings were resolved despite enumerating five should-fix findings." + }, + "testing": { + "completeness": 16, + "correctness": 15, + "specificity": 21, + "risk_awareness": 13, + "total": 65, + "notes": "The plan supplies concrete fixtures, commands, negative cases, and meaningful convention and path oracles. Its largest gap is the absence of an installRun or isSkillPath regression test for an arbitrary nested path; matcher-level tests inject an already-discovered list and bypass the routing behavior that the real fix changed. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) Consequently the suite can pass while exact paths still trigger full-tree traversal and fail on tree truncation. Several UX checks are not represented in the patch, and the manual picker expectation incorrectly presents a standard skill by repository path rather than its DisplayName." + }, + "implementation": { + "completeness": 15, + "correctness": 10, + "specificity": 19, + "risk_awareness": 9, + "total": 53, + "notes": "The patch concretely extends hidden-directory matching, adds path equality selection, preserves standard-name precedence, and includes focused Go tests against the submitted baseline symbols. Its largest defect is implementing automatic catch-all discovery, contrary to the repository fix's explicit non-broadening decision, while failing to add the shared path-detection logic needed for efficient exact-path install and preview. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) This creates prompt and compatibility expansion and leaves arbitrary paths dependent on complete recursive-tree discovery; the exported DiscoverLocalSkills behavior is also narrowed without updating its surrounding contract. The sandbox could not execute local commands, so the claimed git apply check, compilation, and test results could not be independently verified." + } + }, + "is_error": false, + "failed": false + } + ], + "run_date": "2026-10-14" +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json new file mode 100644 index 00000000..39f73312 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/eval.json @@ -0,0 +1,64 @@ +{ + "task": "secret-set-app-agents", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 18, + "total": 88, + "notes": "The issue precisely defines supported entities, REST paths, payload typing, help changes, and testable acceptance criteria. The referenced API confirms integer selected_repository_ids, and the related upstream work confirms expansion across set, list, and delete. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets), [github.com](https://github.com/cli/cli/pull/13421)) Its largest gap is limited treatment of GHES availability, permissions, and expected server errors. No independent task statement was supplied, so intended scope beyond the identifier and internally consistent artifacts cannot be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 21, + "total": 91, + "notes": "The LLD is strongly grounded in real files and commits to the key design: add one shared App value while reusing existing URL interpolation and the non-Dependabot integer payload path. The upstream PR independently confirms the named parser, support matrix, enum flag, and repository/organization integration points. ([github.com](https://github.com/cli/cli/pull/13421)) Its main weakness is that GHES and authorization failures are delegated to existing API error handling without defining validation or diagnostic expectations. The pagination claim and estimated line counts could not be independently checked locally because the repository shell sandbox failed to initialize." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 23, + "total": 89, + "notes": "The review identifies concrete change-specific risks, especially integer repository IDs, fetching the public key from the agents endpoint, pre-network entity rejection, and brittle list-test path rewriting. Its GHES analysis matches the actual maintainer discussion, which accepted explicit requests failing through the server API on unsupported hosts. ([github.com](https://github.com/cli/cli/pull/13421)) The largest deficiency is that every perspective ultimately approves, while the product-naming concern is dismissed rather than resolved; current help uses the clearer capitalized Agents terminology. ([cli.github.com](https://cli.github.com/manual/gh_secret), [cli.github.com](https://cli.github.com/manual/gh_help_reference)) The assertion that secrets are never logged is reasonable from an unchanged encryption path but is not directly demonstrated by the submitted patch." + }, + "testing": { + "completeness": 21, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The plan provides unusually concrete cases, paths, payload types, TTY outputs, backward-compatible defaults, negative commands, and live set-list-delete checks. The patch contains the named repository and organization runtime cases, consistent with the upstream addition of endpoint-routing tests. ([github.com](https://github.com/cli/cli/pull/13421)) Its largest gap is omission of the repository-required go test ./... and make lint checks, limiting regression coverage to the secret packages. It also lacks automated per-command assertions for the exact environment/user error and absence of network calls, leaving those acceptance conditions mostly to the shared matrix and manual tests." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The focused diff implements the design end to end: shared parsing and entity support, all three command enums, repository and organization routing tests, integer organization payload coverage, and help updates. The paths and symbols agree with the baseline context, and the upstream merged change confirms this architecture and the runtime list/delete endpoint coverage. ([github.com](https://github.com/cli/cli/pull/13421), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The main defect is minor UX wording: lowercase generic \"agents\" does not resolve the review's naming concern as clearly as the official capitalized Agents wording. ([cli.github.com](https://cli.github.com/manual/gh_secret), [cli.github.com](https://cli.github.com/manual/gh_help_reference)) I could not independently run git apply --check or the claimed tests because the read-only shell sandbox failed before command execution, so those verification claims remain unconfirmed rather than contradicted." + } + }, + "task_score": 89.2, + "verdict": "This is a strong, repository-specific, implementation-ready submission whose main limitations are incomplete full-repository verification and relatively light treatment of unsupported-host and exact negative-path behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:48:46.692650Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 105238, + "cached_input_tokens": 51357, + "cache_write_input_tokens": 26225, + "output_tokens": 7879, + "reasoning_output_tokens": 6447 + }, + "duration_ms": 122874 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json new file mode 100644 index 00000000..73f46af3 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/secret-set-app-agents/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "secret-set-app-agents", + "repo": "https://github.com/cli/cli", + "ref": "v2.92.0", + "complexity": "medium", + "tags": [ + "secret", + "go", + "cli", + "github-api", + "fixed-in-2.93.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 113.2, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 72, + "output_tokens": 23845, + "cache_read_tokens": 2863096, + "cache_write_tokens": 89061, + "total_tokens": 2976074, + "total_cost_usd": 1.1372522700000003, + "latency_seconds": 210.6, + "num_turns": 35, + "generation_tokens_per_sec": 113.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 72, + "output_tokens": 23845, + "latency_seconds": 210.6, + "num_turns": 35, + "total_cost_usd": 1.1372522700000003, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2863096, + "cache_creation_tokens": 89061, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:19:51.909349Z", + "run_ended_at": "2026-10-14T15:23:22.511096Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 72, + "output_tokens": 23845, + "cache_read_tokens": 2863096, + "cache_write_tokens": 89061, + "total_tokens": 2976074, + "total_cost_usd": 1.1372522700000003, + "latency_seconds": 210.6, + "num_turns": 35, + "generation_tokens_per_sec": 113.2, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "secret-set-app-agents", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 18, + "total": 88, + "notes": "The issue precisely defines supported entities, REST paths, payload typing, help changes, and testable acceptance criteria. The referenced API confirms integer selected_repository_ids, and the related upstream work confirms expansion across set, list, and delete. ([docs.github.com](https://docs.github.com/en/rest/agents/secrets), [github.com](https://github.com/cli/cli/pull/13421)) Its largest gap is limited treatment of GHES availability, permissions, and expected server errors. No independent task statement was supplied, so intended scope beyond the identifier and internally consistent artifacts cannot be verified." + }, + "lld": { + "completeness": 23, + "correctness": 23, + "specificity": 24, + "risk_awareness": 21, + "total": 91, + "notes": "The LLD is strongly grounded in real files and commits to the key design: add one shared App value while reusing existing URL interpolation and the non-Dependabot integer payload path. The upstream PR independently confirms the named parser, support matrix, enum flag, and repository/organization integration points. ([github.com](https://github.com/cli/cli/pull/13421)) Its main weakness is that GHES and authorization failures are delegated to existing API error handling without defining validation or diagnostic expectations. The pagination claim and estimated line counts could not be independently checked locally because the repository shell sandbox failed to initialize." + }, + "review": { + "completeness": 21, + "correctness": 22, + "specificity": 23, + "risk_awareness": 23, + "total": 89, + "notes": "The review identifies concrete change-specific risks, especially integer repository IDs, fetching the public key from the agents endpoint, pre-network entity rejection, and brittle list-test path rewriting. Its GHES analysis matches the actual maintainer discussion, which accepted explicit requests failing through the server API on unsupported hosts. ([github.com](https://github.com/cli/cli/pull/13421)) The largest deficiency is that every perspective ultimately approves, while the product-naming concern is dismissed rather than resolved; current help uses the clearer capitalized Agents terminology. ([cli.github.com](https://cli.github.com/manual/gh_secret), [cli.github.com](https://cli.github.com/manual/gh_help_reference)) The assertion that secrets are never logged is reasonable from an unchanged encryption path but is not directly demonstrated by the submitted patch." + }, + "testing": { + "completeness": 21, + "correctness": 22, + "specificity": 24, + "risk_awareness": 21, + "total": 88, + "notes": "The plan provides unusually concrete cases, paths, payload types, TTY outputs, backward-compatible defaults, negative commands, and live set-list-delete checks. The patch contains the named repository and organization runtime cases, consistent with the upstream addition of endpoint-routing tests. ([github.com](https://github.com/cli/cli/pull/13421)) Its largest gap is omission of the repository-required go test ./... and make lint checks, limiting regression coverage to the secret packages. It also lacks automated per-command assertions for the exact environment/user error and absence of network calls, leaving those acceptance conditions mostly to the shared matrix and manual tests." + }, + "implementation": { + "completeness": 23, + "correctness": 23, + "specificity": 23, + "risk_awareness": 21, + "total": 90, + "notes": "The focused diff implements the design end to end: shared parsing and entity support, all three command enums, repository and organization routing tests, integer organization payload coverage, and help updates. The paths and symbols agree with the baseline context, and the upstream merged change confirms this architecture and the runtime list/delete endpoint coverage. ([github.com](https://github.com/cli/cli/pull/13421), [github.com](https://github.com/cli/cli/releases/tag/v2.93.0)) The main defect is minor UX wording: lowercase generic \"agents\" does not resolve the review's naming concern as clearly as the official capitalized Agents wording. ([cli.github.com](https://cli.github.com/manual/gh_secret), [cli.github.com](https://cli.github.com/manual/gh_help_reference)) I could not independently run git apply --check or the claimed tests because the read-only shell sandbox failed before command execution, so those verification claims remain unconfirmed rather than contradicted." + } + }, + "task_score": 89.2, + "verdict": "This is a strong, repository-specific, implementation-ready submission whose main limitations are incomplete full-repository verification and relatively light treatment of unsupported-host and exact negative-path behavior.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:48:46.692650Z", + "repo_ref": "v2.92.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 105238, + "cached_input_tokens": 51357, + "cache_write_input_tokens": 26225, + "output_tokens": 7879, + "reasoning_output_tokens": 6447 + }, + "duration_ms": 122874 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json new file mode 100644 index 00000000..9d6cbcf4 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/eval.json @@ -0,0 +1,64 @@ +{ + "task": "skill-install-nested-discovery", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 19, + "correctness": 13, + "specificity": 22, + "risk_awareness": 13, + "total": 67, + "notes": "The issue provides concrete path examples, testable acceptance criteria, and explicit non-goals. Its largest defect is expanding automatic discovery to every non-hidden SKILL.md, whereas the repository's actual fix deliberately supports non-standard layouts only through an explicit exact path. ([github.com](https://github.com/cli/cli/pull/13459)) It correctly identifies that prefixed skills directories already work and that nested hidden directories do not. No independent task statement establishes the claimed issue-comment requirements for catalog-wide catch-all discovery." + }, + "lld": { + "completeness": 18, + "correctness": 12, + "specificity": 21, + "risk_awareness": 14, + "total": 65, + "notes": "The LLD is well grounded in real integration points such as discovery.go, matchSkillByName, selectSkill, publish, and MatchSkillPath. Its critical error is leaving isSkillPath unchanged, so an arbitrary path such as packages/agent-skills/x requires full-tree discovery instead of the direct lookup implemented by the repository fix; this also preserves failure on truncated large trees. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) The catch-all design additionally contradicts the repository's stated decision not to broaden search or publish behavior, while its vendor and testdata exclusions contradict the issue's own acceptance boundary. The scaling discussion therefore understates both picker expansion and exact-path traversal risk." + }, + "review": { + "completeness": 19, + "correctness": 16, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review's strongest work is finding concrete defects B1, B2, C1, S1, and S3 and recording actionable resolutions. It misses the central product and implementation flaw: automatic catch-all discovery is broader than the repository's intended explicit-path behavior, and arbitrary exact paths still do not take the direct lookup route. ([github.com](https://github.com/cli/cli/pull/13459)) Circuit's claim that path lookup works at any depth is therefore incomplete for large or truncated trees. The summary also says four blocking findings were resolved despite enumerating five should-fix findings." + }, + "testing": { + "completeness": 16, + "correctness": 15, + "specificity": 21, + "risk_awareness": 13, + "total": 65, + "notes": "The plan supplies concrete fixtures, commands, negative cases, and meaningful convention and path oracles. Its largest gap is the absence of an installRun or isSkillPath regression test for an arbitrary nested path; matcher-level tests inject an already-discovered list and bypass the routing behavior that the real fix changed. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) Consequently the suite can pass while exact paths still trigger full-tree traversal and fail on tree truncation. Several UX checks are not represented in the patch, and the manual picker expectation incorrectly presents a standard skill by repository path rather than its DisplayName." + }, + "implementation": { + "completeness": 15, + "correctness": 10, + "specificity": 19, + "risk_awareness": 9, + "total": 53, + "notes": "The patch concretely extends hidden-directory matching, adds path equality selection, preserves standard-name precedence, and includes focused Go tests against the submitted baseline symbols. Its largest defect is implementing automatic catch-all discovery, contrary to the repository fix's explicit non-broadening decision, while failing to add the shared path-detection logic needed for efficient exact-path install and preview. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) This creates prompt and compatibility expansion and leaves arbitrary paths dependent on complete recursive-tree discovery; the exported DiscoverLocalSkills behavior is also narrowed without updating its surrounding contract. The sandbox could not execute local commands, so the claimed git apply check, compilation, and test results could not be independently verified." + } + }, + "task_score": 64.8, + "verdict": "The submission is detailed and repository-aware, but it solves the nested-path bug with an unsupported catch-all discovery expansion while missing the direct exact-path routing central to the actual fix.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:51:11.071145Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 156968, + "cached_input_tokens": 105312, + "cache_write_input_tokens": 35668, + "output_tokens": 9213, + "reasoning_output_tokens": 7561 + }, + "duration_ms": 144365 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json new file mode 100644 index 00000000..9b903115 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-install-nested-discovery/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "skill-install-nested-discovery", + "repo": "https://github.com/cli/cli", + "ref": "v2.93.0", + "complexity": "high", + "tags": [ + "skill", + "go", + "cli", + "discovery", + "fixed-in-2.94.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 122.5, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 108, + "output_tokens": 61282, + "cache_read_tokens": 6232124, + "cache_write_tokens": 157386, + "total_tokens": 6450900, + "total_cost_usd": 2.4782183800000004, + "latency_seconds": 500.3, + "num_turns": 51, + "generation_tokens_per_sec": 122.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 108, + "output_tokens": 61282, + "latency_seconds": 500.3, + "num_turns": 51, + "total_cost_usd": 2.4782183800000004, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 6232124, + "cache_creation_tokens": 157386, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:44:48.975569Z", + "run_ended_at": "2026-10-14T15:53:09.325565Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 108, + "output_tokens": 61282, + "cache_read_tokens": 6232124, + "cache_write_tokens": 157386, + "total_tokens": 6450900, + "total_cost_usd": 2.4782183800000004, + "latency_seconds": 500.3, + "num_turns": 51, + "generation_tokens_per_sec": 122.5, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "skill-install-nested-discovery", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 19, + "correctness": 13, + "specificity": 22, + "risk_awareness": 13, + "total": 67, + "notes": "The issue provides concrete path examples, testable acceptance criteria, and explicit non-goals. Its largest defect is expanding automatic discovery to every non-hidden SKILL.md, whereas the repository's actual fix deliberately supports non-standard layouts only through an explicit exact path. ([github.com](https://github.com/cli/cli/pull/13459)) It correctly identifies that prefixed skills directories already work and that nested hidden directories do not. No independent task statement establishes the claimed issue-comment requirements for catalog-wide catch-all discovery." + }, + "lld": { + "completeness": 18, + "correctness": 12, + "specificity": 21, + "risk_awareness": 14, + "total": 65, + "notes": "The LLD is well grounded in real integration points such as discovery.go, matchSkillByName, selectSkill, publish, and MatchSkillPath. Its critical error is leaving isSkillPath unchanged, so an arbitrary path such as packages/agent-skills/x requires full-tree discovery instead of the direct lookup implemented by the repository fix; this also preserves failure on truncated large trees. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) The catch-all design additionally contradicts the repository's stated decision not to broaden search or publish behavior, while its vendor and testdata exclusions contradict the issue's own acceptance boundary. The scaling discussion therefore understates both picker expansion and exact-path traversal risk." + }, + "review": { + "completeness": 19, + "correctness": 16, + "specificity": 21, + "risk_awareness": 18, + "total": 74, + "notes": "The review's strongest work is finding concrete defects B1, B2, C1, S1, and S3 and recording actionable resolutions. It misses the central product and implementation flaw: automatic catch-all discovery is broader than the repository's intended explicit-path behavior, and arbitrary exact paths still do not take the direct lookup route. ([github.com](https://github.com/cli/cli/pull/13459)) Circuit's claim that path lookup works at any depth is therefore incomplete for large or truncated trees. The summary also says four blocking findings were resolved despite enumerating five should-fix findings." + }, + "testing": { + "completeness": 16, + "correctness": 15, + "specificity": 21, + "risk_awareness": 13, + "total": 65, + "notes": "The plan supplies concrete fixtures, commands, negative cases, and meaningful convention and path oracles. Its largest gap is the absence of an installRun or isSkillPath regression test for an arbitrary nested path; matcher-level tests inject an already-discovered list and bypass the routing behavior that the real fix changed. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) Consequently the suite can pass while exact paths still trigger full-tree traversal and fail on tree truncation. Several UX checks are not represented in the patch, and the manual picker expectation incorrectly presents a standard skill by repository path rather than its DisplayName." + }, + "implementation": { + "completeness": 15, + "correctness": 10, + "specificity": 19, + "risk_awareness": 9, + "total": 53, + "notes": "The patch concretely extends hidden-directory matching, adds path equality selection, preserves standard-name precedence, and includes focused Go tests against the submitted baseline symbols. Its largest defect is implementing automatic catch-all discovery, contrary to the repository fix's explicit non-broadening decision, while failing to add the shared path-detection logic needed for efficient exact-path install and preview. ([github.com](https://github.com/cli/cli/pull/13459), [cli.github.com](https://cli.github.com/manual/gh_skill_install)) This creates prompt and compatibility expansion and leaves arbitrary paths dependent on complete recursive-tree discovery; the exported DiscoverLocalSkills behavior is also narrowed without updating its surrounding contract. The sandbox could not execute local commands, so the claimed git apply check, compilation, and test results could not be independently verified." + } + }, + "task_score": 64.8, + "verdict": "The submission is detailed and repository-aware, but it solves the nested-path bug with an unsupported catch-all discovery expansion while missing the direct exact-path routing central to the actual fix.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:51:11.071145Z", + "repo_ref": "v2.93.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 156968, + "cached_input_tokens": 105312, + "cache_write_input_tokens": 35668, + "output_tokens": 9213, + "reasoning_output_tokens": 7561 + }, + "duration_ms": 144365 + } + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json new file mode 100644 index 00000000..4551e3aa --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/eval.json @@ -0,0 +1,64 @@ +{ + "task": "skill-update-dir-keeps-install-path", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The issue provides a reproducible namespaced layout, explicit expected paths, non-goals, and testable acceptance criteria for custom and agent directories. Its explanation matches the submitted baseline context in pkg/cmd/skills/update/update.go, including base derivation and os.RemoveAll cleanup. The largest deficiency is that promising every failed update leaves the installation intact conflicts with declaring atomic replacement out of scope, since an in-place multi-file rewrite can partially succeed. No independent task statement was supplied, and the related issue link and reproduction could not be independently exercised." + }, + "lld": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 18, + "total": 78, + "notes": "The design is unusually concrete about installedSkill.dir, installer.Options, filepath.Dir/Base, affected files, validation, and removal of cleanup. The chosen mapping correctly preserves flat, namespaced, and locally renamed paths for GitHub-backed updates. Its largest gap is knowingly leaving partial writes possible despite the issue's failed-update acceptance criterion; --force is recovery guidance, not prevention. DirName is implemented only in installSkill while installLocalSkill remains unchanged, so the stated general installer option does not consistently govern every installer path." + }, + "review": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 20, + "total": 80, + "notes": "The review finds concrete defects rather than merely approving: multi-skill DirName collisions, partial in-place writes, locally renamed directories, and skipped failure verification. Its most important finding correctly explains how a failed multi-file update could leave mixed content and an advanced tree SHA. The main deficiency is approving that risk even though it contradicts an explicit issue acceptance criterion. The security claim is also too broad because filepath.Join(baseDir, DirName) is not itself constrained to one path element, even though the update caller currently supplies filepath.Base." + }, + "testing": { + "completeness": 18, + "correctness": 19, + "specificity": 23, + "risk_awareness": 16, + "total": 76, + "notes": "The plan has strong path and content oracles for namespaced, unrelated-namespace, renamed, agent-host, flat, and no-relocation behavior. Its failure test makes the only blob fetch fail before any write, so it cannot detect the acknowledged partial-write regression where an earlier file was already overwritten. It also omits the SourceDir installer path and the repository-required full go test ./... and make lint checks. The documented --dir, --all, --force, and --dry-run flags agree with the official command manual, but the claimed executions were not independently verifiable. ([cli.github.com](https://cli.github.com/manual/gh_skill_update))" + }, + "implementation": { + "completeness": 18, + "correctness": 18, + "specificity": 22, + "risk_awareness": 16, + "total": 74, + "notes": "The patch directly implements the core path-preservation rule by passing filepath.Dir(u.local.dir) and filepath.Base(u.local.dir), and it removes the destructive relocation cleanup. The focused tests cover the principal successful update layouts and preserve extra local files. The largest defect is that a later-file failure can still leave a partially updated installation, contrary to the submitted acceptance criteria; additionally, DirName is ignored by installLocalSkill and is not validated as a single safe path element. The hunk symbols and contexts are internally consistent, but the managed sandbox could not launch git apply --check or tests, so applicability and execution claims could not be independently confirmed." + } + }, + "task_score": 79.8, + "verdict": "This is a strong, repository-specific solution to path relocation, but it does not meet its own guarantee that every failed update preserves the complete prior installation.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:53:52.094806Z", + "repo_ref": "v2.94.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 127723, + "cached_input_tokens": 77221, + "cache_write_input_tokens": 28517, + "output_tokens": 10378, + "reasoning_output_tokens": 8282 + }, + "duration_ms": 161009 + } +} diff --git a/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json new file mode 100644 index 00000000..7e719100 --- /dev/null +++ b/benchmarks/swe-benchmark-data/claude-sonnet-5-5/omp/swe3/cli-cli-v1/skill-update-dir-keeps-install-path/metrics.json @@ -0,0 +1,155 @@ +{ + "task": "skill-update-dir-keeps-install-path", + "repo": "https://github.com/cli/cli", + "ref": "v2.94.0", + "complexity": "high", + "tags": [ + "skill", + "go", + "cli", + "filesystem", + "fixed-in-2.95.0" + ], + "model": "us.anthropic.claude-sonnet-5-5", + "model_slug": "claude-sonnet-5-5", + "agent": "omp", + "skill": "swe3", + "provider": "bedrock", + "endpoint": null, + "aws_region": "us-east-1", + "serving": { + "instance_type": "g6e.4xlarge", + "tensor_parallel_size": null, + "precision": null, + "context_window": null + }, + "artifacts_produced": 6, + "artifacts_expected": 6, + "generation_tokens_per_sec": 116.6, + "token_accounting_warning": null, + "prompt_token_reconciliation": null, + "metrics": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 56, + "output_tokens": 24732, + "cache_read_tokens": 2196697, + "cache_write_tokens": 82730, + "total_tokens": 2304215, + "total_cost_usd": 0.9829560399999998, + "latency_seconds": 212.1, + "num_turns": 27, + "generation_tokens_per_sec": 116.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "input_tokens": 56, + "output_tokens": 24732, + "latency_seconds": 212.1, + "num_turns": 27, + "total_cost_usd": 0.9829560399999998, + "is_error": false, + "result_subtype": "success", + "cache_read_tokens": 2196697, + "cache_creation_tokens": 82730, + "models_used": [ + "us.anthropic.claude-sonnet-5-5" + ], + "run_started_at": "2026-10-14T15:41:13.321747Z", + "run_ended_at": "2026-10-14T15:44:45.439839Z", + "metrics_that_matter": { + "note": "Headline metrics as reported by the model API (claude -p); there is no vLLM server for this provider, so no server-side cache telemetry.", + "input_tokens": 56, + "output_tokens": 24732, + "cache_read_tokens": 2196697, + "cache_write_tokens": 82730, + "total_tokens": 2304215, + "total_cost_usd": 0.9829560399999998, + "latency_seconds": 212.1, + "num_turns": 27, + "generation_tokens_per_sec": 116.6, + "sources": { + "input_tokens": "omp_api.usage.input", + "output_tokens": "omp_api.usage.output", + "cache_read_tokens": "omp_api.usage.cache_read_input_tokens", + "cache_write_tokens": "omp_api.usage.cache_creation_input_tokens", + "total_tokens": "total tokens processed once each (issue #136): input + output, plus cache_read + cache_write ONLY when the cache is additive (not a partition of input). An agent with disjoint counts declares that instead of being detected (issue #183).", + "total_cost_usd": "omp_api.total_cost_usd (metered)", + "latency_seconds": "harness wall-clock (or omp_api.duration_ms)", + "num_turns": "omp_api.num_turns", + "generation_tokens_per_sec": "derived: output_tokens / latency_seconds" + } + }, + "evaluation": { + "task": "skill-update-dir-keeps-install-path", + "model": "us.anthropic.claude-sonnet-5-5", + "scores": { + "github_issue": { + "completeness": 23, + "correctness": 22, + "specificity": 24, + "risk_awareness": 22, + "total": 91, + "notes": "The issue provides a reproducible namespaced layout, explicit expected paths, non-goals, and testable acceptance criteria for custom and agent directories. Its explanation matches the submitted baseline context in pkg/cmd/skills/update/update.go, including base derivation and os.RemoveAll cleanup. The largest deficiency is that promising every failed update leaves the installation intact conflicts with declaring atomic replacement out of scope, since an in-place multi-file rewrite can partially succeed. No independent task statement was supplied, and the related issue link and reproduction could not be independently exercised." + }, + "lld": { + "completeness": 19, + "correctness": 18, + "specificity": 23, + "risk_awareness": 18, + "total": 78, + "notes": "The design is unusually concrete about installedSkill.dir, installer.Options, filepath.Dir/Base, affected files, validation, and removal of cleanup. The chosen mapping correctly preserves flat, namespaced, and locally renamed paths for GitHub-backed updates. Its largest gap is knowingly leaving partial writes possible despite the issue's failed-update acceptance criterion; --force is recovery guidance, not prevention. DirName is implemented only in installSkill while installLocalSkill remains unchanged, so the stated general installer option does not consistently govern every installer path." + }, + "review": { + "completeness": 20, + "correctness": 18, + "specificity": 22, + "risk_awareness": 20, + "total": 80, + "notes": "The review finds concrete defects rather than merely approving: multi-skill DirName collisions, partial in-place writes, locally renamed directories, and skipped failure verification. Its most important finding correctly explains how a failed multi-file update could leave mixed content and an advanced tree SHA. The main deficiency is approving that risk even though it contradicts an explicit issue acceptance criterion. The security claim is also too broad because filepath.Join(baseDir, DirName) is not itself constrained to one path element, even though the update caller currently supplies filepath.Base." + }, + "testing": { + "completeness": 18, + "correctness": 19, + "specificity": 23, + "risk_awareness": 16, + "total": 76, + "notes": "The plan has strong path and content oracles for namespaced, unrelated-namespace, renamed, agent-host, flat, and no-relocation behavior. Its failure test makes the only blob fetch fail before any write, so it cannot detect the acknowledged partial-write regression where an earlier file was already overwritten. It also omits the SourceDir installer path and the repository-required full go test ./... and make lint checks. The documented --dir, --all, --force, and --dry-run flags agree with the official command manual, but the claimed executions were not independently verifiable. ([cli.github.com](https://cli.github.com/manual/gh_skill_update))" + }, + "implementation": { + "completeness": 18, + "correctness": 18, + "specificity": 22, + "risk_awareness": 16, + "total": 74, + "notes": "The patch directly implements the core path-preservation rule by passing filepath.Dir(u.local.dir) and filepath.Base(u.local.dir), and it removes the destructive relocation cleanup. The focused tests cover the principal successful update layouts and preserve extra local files. The largest defect is that a later-file failure can still leave a partially updated installation, contrary to the submitted acceptance criteria; additionally, DirName is ignored by installLocalSkill and is not validated as a single safe path element. The hunk symbols and contexts are internally consistent, but the managed sandbox could not launch git apply --check or tests, so applicability and execution claims could not be independently confirmed." + } + }, + "task_score": 79.8, + "verdict": "This is a strong, repository-specific solution to path relocation, but it does not meet its own guarantee that every failed update preserves the complete prior installation.", + "judge": { + "model": "openai.gpt-5.6-sol", + "provider": "codex-exec", + "repo_grounded": true, + "evaluated_at": "2026-10-14T16:53:52.094806Z", + "repo_ref": "v2.94.0", + "reasoning_effort": "high", + "token_usage": { + "input_tokens": 127723, + "cached_input_tokens": 77221, + "cache_write_input_tokens": 28517, + "output_tokens": 10378, + "reasoning_output_tokens": 8282 + }, + "duration_ms": 161009 + } + } +} diff --git a/benchmarks/swe-benchmark-data/do-not-include/README.md b/benchmarks/swe-benchmark-data/do-not-include/README.md index da9b413b..76e88a72 100644 --- a/benchmarks/swe-benchmark-data/do-not-include/README.md +++ b/benchmarks/swe-benchmark-data/do-not-include/README.md @@ -75,3 +75,12 @@ Both faults are fixed in [#218](https://github.com/aarora79/agentic-coding-harne The six artifacts Haiku 5.5 produced for `registration-admission-control-gate` in its published run on 2026-10-10. The prompt named the output directory `.../omp/swe3/mcp-gateway-registry-v2/registration-admission-control-gate/` 56 times; the model wrote everything to `.../omp/swe3/mcp-gateway-registry/registration-admission-control-gate/` instead, dropping the `-v2`. The harness found nothing in the right place and retried, and the retry stopped after 12 turns without writing anything. The task counts as a **failure** in the published run (20 / 21 scored, mean 72.69). These files were never judged and must not be: scoring them would grant Haiku 5.5 a pass no other model could have earned by writing to the wrong folder. They are kept, outside the harness's scan, so the failure can be read. Unlike the other runs here they include the markdown artifacts and `patch.diff`, which the repo's `.gitignore` excludes by default; they were added explicitly. One hunk header in `patch.diff` quoted an example token from upstream's `terraform.tfvars.example`; it reads `` here. Git ignores hunk-header text, so the patch still applies at `v1.0.19`. + +The same failure recurred on the cli/cli dataset on 2026-10-14, for two of the 21 tasks. Both sit under `omp/swe3/cli-cli-v1/`, the path the prompt named: + +| Task | Where Haiku 5.5 wrote | Retry | +|---|---|---| +| `auth-clipboard-config-default` | `claude-haiku-5-5/claude-code/cli/auth-clipboard-config-default/`, the skill's generic model / harness / repo layout instead of the given path | Wrote to the same wrong folder | +| `skill-install-nested-discovery` | `claude-haiku-5-5/omp/swe3/cli-v1/skill-install-nested-discovery/`, dropping one `cli` | Wrote nothing; said the request named no `artifacts_dir` and asked for one | + +Both count as **failures** in the published run (19 / 21 scored, mean 80.82), and neither was judged. A third misplaced write, `cli-ci-v1/repo-clone-ghe-ssh-ca-url/` on that task's first attempt, is not kept: the retry wrote to the right place and was scored. The secret scan found nothing to redact in these files. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/github-issue.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/github-issue.md new file mode 100644 index 00000000..a5e0fe8b --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/github-issue.md @@ -0,0 +1,45 @@ +# GitHub Issue: Let users make "copy device code to clipboard" the default for `gh auth login` and `gh auth refresh` + +**Labels:** enhancement, auth + +## Description + +### Problem Statement + +`gh auth login` and `gh auth refresh` copy the one-time OAuth device code to the clipboard only when `-c` / `--clipboard` is passed. Users who always want the code copied must add the flag to every invocation. There is no configuration setting for this preference, so the flag cannot be made a default. + +### Proposed Solution + +Add a global configuration key `copy_to_clipboard` with allowed values `enabled` and `disabled`. The default is `disabled`, so existing behavior does not change. + +- When the key is `enabled`, `gh auth login` and `gh auth refresh` copy the device code as if `-c` were passed, without the flag. +- The `-c` flag keeps working. It enables copying for a single run even when the config value is `disabled`. +- The key is global, like `telemetry`. It is not scoped by hostname. + +### User Stories + +- As a user who always signs in from a remote or headless session, I want to set `copy_to_clipboard` once so that `gh auth login` copies the device code without my typing `-c` each time. +- As a user who has the setting enabled, I want `gh auth refresh` to honor it the same way, so the two commands behave consistently. + +### Acceptance Criteria + +- [ ] `gh config get copy_to_clipboard` prints `disabled` when the key was never set. +- [ ] `gh config set copy_to_clipboard enabled` causes `gh auth login` to copy the device code without `-c`. +- [ ] `gh config set copy_to_clipboard enabled` causes `gh auth refresh` to copy the device code without `-c`. +- [ ] `gh config set copy_to_clipboard maybe` is rejected, because only `enabled` and `disabled` are allowed. +- [ ] `gh config list` includes `copy_to_clipboard=disabled` when the key is unset. +- [ ] With the key unset, `gh auth login -c` and `gh auth refresh -c` still copy the device code (no behavior change). + +### Out of Scope + +- A `--no-clipboard` flag that overrides an `enabled` config value for one run. Only `enabled` in config can be overridden by disabling the config key; the flag can only enable copying. +- Host-scoped configuration of this key. +- Clipboard behavior for any other command. + +### Dependencies + +None. The `clipboard` package is already imported by `internal/authflow`. + +### Related Issues + +None. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/implementation.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/implementation.md new file mode 100644 index 00000000..17746078 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/implementation.md @@ -0,0 +1,36 @@ +# Implementation: copy_to_clipboard config default + +## Summary + +Adds a `copy_to_clipboard` config key (values `enabled` / `disabled`, default `disabled`). When `-c` / `--clipboard` is absent, `gh auth login` and `gh auth refresh` read this key and copy the one-time OAuth device code to the clipboard when it is `enabled`. When the flag is passed explicitly, it wins in both directions: `--clipboard=false` disables copying even when the config is `enabled`, and `--clipboard` enables copying regardless of the config. + +## Files touched (patch.diff, 8 files) + +| File | Change | +|---|---| +| `internal/config/config.go` | Adds `copyToClipboardKey` constant, `CopyToClipboard()` accessor (`GetOrDefault("", key)`), and a registry entry (`ConfigOption`) with allowed values `enabled` / `disabled` and default `disabled`, appended last so `gh config list` shows it after `telemetry`. | +| `internal/gh/gh.go` | Adds `CopyToClipboard() ConfigEntry` to the `Config` interface, with a doc comment noting it is global. | +| `internal/gh/mock/config.go` | Regenerated with moq (see Deviations). Adds `CopyToClipboardFunc`, calls struct, lock, method and `CopyToClipboardCalls()`. | +| `internal/config/test.go` | Wires `mock.CopyToClipboardFunc` to `cfg.CopyToClipboard()` in `NewMockConfigFromString`, so mock-backed tests see the real value. | +| `pkg/cmd/auth/login/login.go` | Adds unexported `clipboardSet` to `LoginOptions`, set in `RunE` from `cmd.Flags().Changed("clipboard")`. `loginRun` computes `copyToClipboard := opts.Clipboard`, sets it to `true` when `!opts.clipboardSet` and the config is `enabled`, and passes it to `shared.Login`. | +| `pkg/cmd/auth/refresh/refresh.go` | Same pattern as login: unexported `clipboardSet` on `RefreshOptions`, set in `RunE`; `refreshRun` computes `copyToClipboard` the same way and passes it to `opts.AuthFlow`. | +| `pkg/cmd/auth/refresh/refresh_test.go` | New `copyToClipboard` table field; sets the key via `cfg.Set("", "copy_to_clipboard", ...)` before the run; new case "no hostname, one host configured, copy_to_clipboard config enabled" expects `clipboard: true`. gofmt realigned the existing struct field block, which accounts for the extra deletions in the diffstat. | +| `pkg/cmd/config/list/list_test.go` | Expected `gh config list` output gains `copy_to_clipboard=disabled` after `telemetry=enabled`. | + +## Deviations + +1. **moq toolchain.** `moq` on the installed Go 1.27.2 toolchain failed with `could not import github.com/cli/cli/v2/pkg/option ... export data version 5 is greater than maximum supported version 4`, and it had already deleted the old mock with `-rm`. The mock was restored from git, then regenerated with `GOTOOLCHAIN=go1.26.0 moq -pkg ghmock -out . Config` and copied over the committed file. Regeneration also moved the `sync` import line, which shows in the diff. +2. **No login test for the config path.** `shared.Login` calls the auth flow directly, and `Test_loginRun_*` in `login_test.go` have no `AuthFlow` seam to capture the `clipboard` argument. The login config path has no permanent test. The refresh test covers the same resolution pattern. Adding a seam would widen the change beyond the LLD, so it was not done. +3. **No separate `config get` / `config set` tests.** The default `disabled` value is asserted through `list_test.go`. The allowed-value validation comes from the existing `ConfigOption` mechanism and was not re-tested. +4. **Precedence corrected after first capture.** The first capture combined the flag and config with `opts.Clipboard || cfg...Value == "enabled"`, so `--clipboard=false` could not override `copy_to_clipboard=enabled`. `lld.md` (section 3, line 103) says the config changes only the default when `-c` is absent, so an explicit flag must win. + The fix tracks whether the flag was set (`cmd.Flags().Changed("clipboard")` in `RunE`) and consults config only when it was not. The first corrected attempt, `if !opts.clipboardSet { copyToClipboard = cfg... == "enabled" }`, broke the existing refresh case `no hostname, one host configured, clipboard enabled`. That case calls `refreshRun` with `Clipboard: true` directly, bypassing `RunE`, so `clipboardSet` stays false and the flag was dropped. The final rule keeps the flag as the base value and only ORs in config when the flag was not set, so the flag can enable copying in every path and config applies only when the flag is absent. + +## Verification (run in the clone at baseline 45437bc7eeeb3359bbfddd1742f79de7652fd3e2 plus the patch) + +- `gofmt -l internal pkg`: no files listed. +- `go build ./...`: exit 0. +- `go vet ./internal/config/ ./internal/gh/... ./pkg/cmd/auth/... ./pkg/cmd/config/list/`: exit 0. +- `go test ./internal/config/ ./internal/gh/... ./pkg/cmd/auth/... ./pkg/cmd/config/...`: all packages `ok`, including `pkg/cmd/auth/refresh` (`Test_refreshRun`, with the new `copy_to_clipboard config enabled` case and the existing `clipboard enabled` case) and `pkg/cmd/config/list`. +- Throwaway smoke test (deleted afterward, not in the patch): `RefreshOptions{clipboardSet: true, Clipboard: false}` with `copy_to_clipboard=enabled` yields `AuthFlow(... clipboard=false)`. PASS. This confirms explicit `--clipboard=false` beats config. +- Not covered by a permanent test: the explicit-false case and the login config path. The refresh test covers the config-enabled and flag-set paths; the explicit-false case was only smoke-tested. +- `patch.diff`: captured with `git add -A` then `git diff --staged 45437bc7eeeb3359bbfddd1742f79de7652fd3e2` in the clone. `git diff --staged --numstat` lists 8 files, matching the table above. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/lld.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/lld.md new file mode 100644 index 00000000..ea88ceab --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/lld.md @@ -0,0 +1,192 @@ +# Low-Level Design: copy_to_clipboard config default for gh auth login and refresh + +**Created:** 2026-10-14 +**Author:** Claude +**Status:** Draft + +## Table of Contents + +1. [Overview](#overview) +2. [Codebase Analysis](#codebase-analysis) +3. [Architecture](#architecture) +4. [Data Models](#data-models) +5. [API / CLI Design](#api--cli-design) +6. [Configuration Parameters](#configuration-parameters) +7. [New Dependencies](#new-dependencies) +8. [Implementation Details](#implementation-details) +9. [Observability](#observability) +10. [File Changes](#file-changes) +11. [Testing Strategy](#testing-strategy) +12. [Alternatives Considered](#alternatives-considered) +13. [Rollout Plan](#rollout-plan) + +## Overview + +### Problem Statement + +The `--clipboard` / `-c` flag on `gh auth login` and `gh auth refresh` copies the one-time device code to the clipboard. The flag is required on every run. No config value controls it. + +### Goals + +- Goal 1: Add a global config key `copy_to_clipboard` (`enabled` | `disabled`, default `disabled`). +- Goal 2: When the value is `enabled`, `login` and `refresh` copy the device code as if `-c` were given. +- Goal 3: `-c` keeps working unchanged. + +### Non-Goals + +- A negative flag (`--no-clipboard`). Out of scope per the issue. +- Host-scoped config for this key. + +## Codebase Analysis + +### Files Reviewed + +| Path | Purpose | Relevance to This Change | +|------|---------|--------------------------| +| `internal/gh/gh.go` | `gh.Config` interface, generated mock source | Interface gains `CopyToClipboard()`. | +| `internal/config/config.go` | Config accessors, key constants, option registry | Adds key constant, accessor, registry entry. | +| `internal/config/test.go` | `NewMockConfigFromString` wiring of mock funcs | Must wire the new func or tests calling it panic. | +| `internal/gh/mock/config.go` | moq-generated `ConfigMock` | Regenerated with the installed `moq`. | +| `pkg/cmd/auth/login/login.go` | `loginRun` builds `shared.LoginOptions` | Resolves the effective clipboard value. | +| `pkg/cmd/auth/refresh/refresh.go` | `refreshRun` calls `opts.AuthFlow` | Resolves the effective clipboard value. | +| `pkg/cmd/auth/shared/login_flow.go` | `shared.Login`, `iconfig` interface | Not changed: `iconfig` exposes only `Login` and `UsersForHost`, and the value is resolved before this call. | +| `pkg/cmd/config/list/list_test.go` | Expected `gh config list` output | Gains one line. | +| `internal/config/config_test.go` | `TestTelemetry` pattern | New `TestCopyToClipboard` mirrors it. | + +### Patterns Identified + +- Global accessor precedent: `Telemetry()` in `config.go` ignores hostname, calls `c.GetOrDefault("", telemetryKey).Unwrap()`, and `Telemetry` in the registry uses `DefaultValue` and `AllowedValues`. This change copies that shape. +- Mock precedent: `TelemetryFunc` is a no-argument method and its mock layout is the template for `CopyToClipboardFunc`. +- Constant alignment: the `const` block in `config.go` is aligned by gofmt; the new key is padded to the existing column width. + +### Integration Points + +- `gh config get|set|list` picks up the new key automatically from the option registry (`internal/config/config.go`, `Options` slice). +- `login.go` and `refresh.go` read the value from the `gh.Config` they already obtain via `opts.Config()`. + +### Constraints Discovered + +- `shared.iconfig` is a narrow interface used by `shared.Login` tests. Widening it would ripple into test doubles. Resolution is done in `loginRun` before `shared.Login` is called. +- `refreshRun` already calls `opts.Config()` at the top, so no extra config load is needed. +- Mock regeneration: `internal/gh/gh.go` carries `//go:generate moq -rm -pkg ghmock -out mock/config.go . Config`. Regenerating keeps the mock byte-consistent with the interface. + +## Architecture + +```text +gh config set copy_to_clipboard enabled + | + v +config file (hosts-agnostic key "copy_to_clipboard") + | + v +cfg.CopyToClipboard() -- gh.Config accessor, global, default "disabled" + | + +--> loginRun: copyToClipboard := opts.Clipboard || cfg value == enabled + | -> shared.LoginOptions.CopyToClipboard + | + +--> refreshRun: clipboard := opts.Clipboard || cfg value == enabled + -> opts.AuthFlow(..., clipboard) +``` + +Resolution is an OR. The flag can only turn copying on. Config `enabled` plus no flag copies. Config `disabled` plus `-c` copies. + +## Data Models + +No new structs. The accessor returns the existing `gh.ConfigEntry` (`Value`, `Source`), following `Telemetry()`. + +| Field | Type | Values | Default | +|-------|------|--------|---------| +| `copy_to_clipboard` | string (config file) | `enabled`, `disabled` | `disabled` (`gh.ConfigDefaultProvided`) | + +## API / CLI Design + +No new flags. `-c` / `--clipboard` on `login` and `refresh` is unchanged. Behavior change is only in the default when `-c` is absent. + +`gh config set copy_to_clipboard ` is the new user-facing surface, provided by the registry entry. `AllowedValues` rejects anything else. + +## Configuration Parameters + +| Key | Default | Scope | Allowed | Description | +|-----|---------|-------|---------|-------------| +| `copy_to_clipboard` | `disabled` | Global | `enabled`, `disabled` | Copy the one-time OAuth device code to the clipboard during `gh auth login` and `gh auth refresh`, without `-c`. | + +The default config template (`defaultConfigStr`, `internal/config/config.go`) does not list `telemetry` or any other registry key, so the registry `DefaultValue` is the only place the default is declared. No other config file changes are needed. + +## New Dependencies + +None. + +## Implementation Details + +1. `internal/gh/gh.go`: add to `gh.Config`: + ```go + // CopyToClipboard returns the configured copy_to_clipboard setting. It ignores host scoping since this is a global setting. + CopyToClipboard() ConfigEntry + ``` +2. `internal/config/config.go`: + - Add `copyToClipboardKey = "copy_to_clipboard"` to the `const` block, aligned with gofmt. + - Add the accessor after `Telemetry()`: + ```go + func (c *cfg) CopyToClipboard() gh.ConfigEntry { + // Intentionally panic if there is no user provided value or default value (which would be a programmer error) + return c.GetOrDefault("", copyToClipboardKey).Unwrap() + } + ``` + - Append a registry entry after the `telemetry` entry: + ```go + { + Key: copyToClipboardKey, + Description: "whether to copy the one-time OAuth device code to the clipboard during gh auth login and refresh", + DefaultValue: "disabled", + AllowedValues: []string{"enabled", "disabled"}, + CurrentValue: func(c gh.Config, hostname string) string { + return c.CopyToClipboard().Value + }, + }, + ``` + - No change to `defaultConfigStr`: it does not list registry keys such as `telemetry`, and the registry `DefaultValue` supplies the default. +3. `internal/config/test.go`: wire `mock.CopyToClipboardFunc = func() gh.ConfigEntry { return cfg.CopyToClipboard() }`. +4. `internal/gh/mock/config.go`: regenerate with `moq -rm -pkg ghmock -out mock/config.go . Config` from `internal/gh`. +5. `pkg/cmd/auth/login/login.go`, `loginRun`: replace `CopyToClipboard: opts.Clipboard` with a resolved value: + ```go + copyToClipboard := opts.Clipboard || cfg.CopyToClipboard().Value == "enabled" + ``` + and pass `copyToClipboard` into `shared.LoginOptions`. +6. `pkg/cmd/auth/refresh/refresh.go`, `refreshRun`: after `cfg` is loaded, compute the same resolved value and pass it to `opts.AuthFlow(...)` in place of `opts.Clipboard`. +7. `pkg/cmd/config/list/list_test.go`: add `copy_to_clipboard=disabled` after `telemetry=enabled` in the expected output. + +Error handling: the accessor panics on a missing default, as `Telemetry()` does. The registry default guarantees this cannot happen. No new error paths. + +## Observability + +None. The existing `gh` output is unchanged. No log lines are added. + +## File Changes + +| File | Action | Summary | +|------|--------|---------| +| `internal/gh/gh.go` | Modify | Add `CopyToClipboard()` to `Config`. | +| `internal/config/config.go` | Modify | Key constant, accessor, registry entry. | +| `internal/gh/mock/config.go` | Regenerate | `ConfigMock` gains `CopyToClipboard` mock. | +| `internal/config/test.go` | Modify | Wire `CopyToClipboardFunc`. | +| `internal/config/config_test.go` | Modify | Add `TestCopyToClipboard`. | +| `pkg/cmd/auth/login/login.go` | Modify | Resolve config OR flag in `loginRun`. | +| `pkg/cmd/auth/refresh/refresh.go` | Modify | Resolve config OR flag in `refreshRun`. | +| `pkg/cmd/auth/refresh/refresh_test.go` | Modify | Add a case: config enabled, no `-c`, AuthFlow receives `true`. | +| `pkg/cmd/auth/login/login_test.go` | Modify | Add cases: config enabled with no `-c` yields `CopyToClipboard: true`; config unset yields `false`. Login tests use `NewMockConfigFromString`, so `test.go` wiring is required. | +| `pkg/cmd/config/list/list_test.go` | Modify | Expected output gains `copy_to_clipboard=disabled`. | + +## Testing Strategy + +See `testing.md`. Unit tests cover the accessor default and user value, the refresh resolution, and the `config list` output. Verification in this run is compile-level only (`gofmt`, `go vet`), because the swe3 skill does not run the target repository's test suite. + +## Alternatives Considered + +1. **Set the flag default at command construction from `f.Config()`.** Rejected. It reads config while the command tree is built, which runs for `--help` and for every `gh` invocation. A config load failure would break unrelated commands. +2. **Tri-state flag with `--no-clipboard`.** Rejected. The issue puts it out of scope, and it adds a flag to two commands. +3. **Host-scoped key.** Rejected. Clipboard use is a terminal property, not a host property. `telemetry` sets the precedent for a global key. +4. **Widen `shared.iconfig` to expose the value.** Rejected. It would change the test double used by `shared.Login` tests. Resolving in `loginRun` keeps the change local. + +## Rollout Plan + +Default `disabled` means no behavior change for existing users. No migration, no deprecation. The change ships in the next minor release of `gh`. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/patch.diff b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/patch.diff new file mode 100644 index 00000000..b7eca078 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/patch.diff @@ -0,0 +1,304 @@ +diff --git a/internal/config/config.go b/internal/config/config.go +index 6467fe7..c47d8eb 100644 +--- a/internal/config/config.go ++++ b/internal/config/config.go +@@ -23,6 +23,7 @@ const ( + aliasesKey = "aliases" + browserKey = "browser" // used by cli/go-gh to open URLs in web browsers + colorLabelsKey = "color_labels" ++ copyToClipboardKey = "copy_to_clipboard" + apiHostKey = "api_host" // used by cli/go-gh to redirect API requests for a host + editorKey = "editor" // used by cli/go-gh to open interactive text editor + gitProtocolKey = "git_protocol" +@@ -177,6 +178,11 @@ func (c *cfg) Telemetry() gh.ConfigEntry { + return c.GetOrDefault("", telemetryKey).Unwrap() + } + ++func (c *cfg) CopyToClipboard() gh.ConfigEntry { ++ // Intentionally panic if there is no user provided value or default value (which would be a programmer error) ++ return c.GetOrDefault("", copyToClipboardKey).Unwrap() ++} ++ + func (c *cfg) Version() o.Option[string] { + return c.get("", versionKey) + } +@@ -765,6 +771,15 @@ var Options = []ConfigOption{ + return c.Telemetry().Value + }, + }, ++ { ++ Key: copyToClipboardKey, ++ Description: "whether to copy the one-time OAuth device code to the clipboard during gh auth login and gh auth refresh", ++ DefaultValue: "disabled", ++ AllowedValues: []string{"enabled", "disabled"}, ++ CurrentValue: func(c gh.Config, hostname string) string { ++ return c.CopyToClipboard().Value ++ }, ++ }, + } + + func HomeDirPath(subdir string) (string, error) { +diff --git a/internal/config/test.go b/internal/config/test.go +index 6f096e9..e7870cd 100644 +--- a/internal/config/test.go ++++ b/internal/config/test.go +@@ -75,6 +75,9 @@ func NewMockConfigFromString(cfgString string) *ghmock.ConfigMock { + mock.TelemetryFunc = func() gh.ConfigEntry { + return cfg.Telemetry() + } ++ mock.CopyToClipboardFunc = func() gh.ConfigEntry { ++ return cfg.CopyToClipboard() ++ } + mock.ColorLabelsFunc = func(hostname string) gh.ConfigEntry { + return cfg.ColorLabels(hostname) + } +diff --git a/internal/gh/gh.go b/internal/gh/gh.go +index f8ca185..648cebf 100644 +--- a/internal/gh/gh.go ++++ b/internal/gh/gh.go +@@ -59,6 +59,8 @@ type Config interface { + Spinner(hostname string) ConfigEntry + // Telemetry returns the configured telemetry setting, ignoring host scoping since telemetry is a global setting. + Telemetry() ConfigEntry ++ // CopyToClipboard returns the configured copy-to-clipboard setting, ignoring host scoping since it is a global setting. ++ CopyToClipboard() ConfigEntry + + // Aliases provides persistent storage and modification of command aliases. + Aliases() AliasConfig +diff --git a/internal/gh/mock/config.go b/internal/gh/mock/config.go +index 31e35cb..8b5e0f1 100644 +--- a/internal/gh/mock/config.go ++++ b/internal/gh/mock/config.go +@@ -4,10 +4,9 @@ + package ghmock + + import ( +- "sync" +- + "github.com/cli/cli/v2/internal/gh" + o "github.com/cli/cli/v2/pkg/option" ++ "sync" + ) + + // Ensure, that ConfigMock does implement gh.Config. +@@ -41,6 +40,9 @@ var _ gh.Config = &ConfigMock{} + // ColorLabelsFunc: func(hostname string) gh.ConfigEntry { + // panic("mock out the ColorLabels method") + // }, ++// CopyToClipboardFunc: func() gh.ConfigEntry { ++// panic("mock out the CopyToClipboard method") ++// }, + // EditorFunc: func(hostname string) gh.ConfigEntry { + // panic("mock out the Editor method") + // }, +@@ -108,6 +110,9 @@ type ConfigMock struct { + // ColorLabelsFunc mocks the ColorLabels method. + ColorLabelsFunc func(hostname string) gh.ConfigEntry + ++ // CopyToClipboardFunc mocks the CopyToClipboard method. ++ CopyToClipboardFunc func() gh.ConfigEntry ++ + // EditorFunc mocks the Editor method. + EditorFunc func(hostname string) gh.ConfigEntry + +@@ -178,6 +183,9 @@ type ConfigMock struct { + // Hostname is the hostname argument value. + Hostname string + } ++ // CopyToClipboard holds details about calls to the CopyToClipboard method. ++ CopyToClipboard []struct { ++ } + // Editor holds details about calls to the Editor method. + Editor []struct { + // Hostname is the hostname argument value. +@@ -251,6 +259,7 @@ type ConfigMock struct { + lockBrowser sync.RWMutex + lockCacheDir sync.RWMutex + lockColorLabels sync.RWMutex ++ lockCopyToClipboard sync.RWMutex + lockEditor sync.RWMutex + lockGetOrDefault sync.RWMutex + lockGitProtocol sync.RWMutex +@@ -475,6 +484,33 @@ func (mock *ConfigMock) ColorLabelsCalls() []struct { + return calls + } + ++// CopyToClipboard calls CopyToClipboardFunc. ++func (mock *ConfigMock) CopyToClipboard() gh.ConfigEntry { ++ if mock.CopyToClipboardFunc == nil { ++ panic("ConfigMock.CopyToClipboardFunc: method is nil but Config.CopyToClipboard was just called") ++ } ++ callInfo := struct { ++ }{} ++ mock.lockCopyToClipboard.Lock() ++ mock.calls.CopyToClipboard = append(mock.calls.CopyToClipboard, callInfo) ++ mock.lockCopyToClipboard.Unlock() ++ return mock.CopyToClipboardFunc() ++} ++ ++// CopyToClipboardCalls gets all the calls that were made to CopyToClipboard. ++// Check the length with: ++// ++// len(mockedConfig.CopyToClipboardCalls()) ++func (mock *ConfigMock) CopyToClipboardCalls() []struct { ++} { ++ var calls []struct { ++ } ++ mock.lockCopyToClipboard.RLock() ++ calls = mock.calls.CopyToClipboard ++ mock.lockCopyToClipboard.RUnlock() ++ return calls ++} ++ + // Editor calls EditorFunc. + func (mock *ConfigMock) Editor(hostname string) gh.ConfigEntry { + if mock.EditorFunc == nil { +diff --git a/pkg/cmd/auth/login/login.go b/pkg/cmd/auth/login/login.go +index 24d30c5..7aeba6b 100644 +--- a/pkg/cmd/auth/login/login.go ++++ b/pkg/cmd/auth/login/login.go +@@ -40,6 +40,7 @@ type LoginOptions struct { + InsecureStorage bool + SkipSSHKeyPrompt bool + Clipboard bool ++ clipboardSet bool + } + + func NewCmdLogin(f *cmdutil.Factory, runF func(*LoginOptions) error) *cobra.Command { +@@ -138,6 +139,7 @@ func NewCmdLogin(f *cmdutil.Factory, runF func(*LoginOptions) error) *cobra.Comm + opts.Hostname, _ = ghauth.DefaultHost() + } + ++ opts.clipboardSet = cmd.Flags().Changed("clipboard") + opts.MainExecutable = f.ExecutablePath + if runF != nil { + return runF(opts) +@@ -216,6 +218,11 @@ func loginRun(opts *LoginOptions) error { + return loginErr + } + ++ copyToClipboard := opts.Clipboard ++ if !opts.clipboardSet && cfg.CopyToClipboard().Value == "enabled" { ++ copyToClipboard = true ++ } ++ + return shared.Login(&shared.LoginOptions{ + IO: opts.IO, + Config: authCfg, +@@ -240,7 +247,7 @@ func loginRun(opts *LoginOptions) error { + }, + SecureStorage: !opts.InsecureStorage, + SkipSSHKeyPrompt: opts.SkipSSHKeyPrompt, +- CopyToClipboard: opts.Clipboard, ++ CopyToClipboard: copyToClipboard, + }) + } + +diff --git a/pkg/cmd/auth/refresh/refresh.go b/pkg/cmd/auth/refresh/refresh.go +index 8ef5e7a..6a07f94 100644 +--- a/pkg/cmd/auth/refresh/refresh.go ++++ b/pkg/cmd/auth/refresh/refresh.go +@@ -39,6 +39,7 @@ type RefreshOptions struct { + Interactive bool + InsecureStorage bool + Clipboard bool ++ clipboardSet bool + } + + func NewCmdRefresh(f *cmdutil.Factory, runF func(*RefreshOptions) error) *cobra.Command { +@@ -102,6 +103,7 @@ func NewCmdRefresh(f *cmdutil.Factory, runF func(*RefreshOptions) error) *cobra. + return cmdutil.FlagErrorf("--hostname required when not running interactively") + } + ++ opts.clipboardSet = cmd.Flags().Changed("clipboard") + opts.MainExecutable = f.ExecutablePath + if runF != nil { + return runF(opts) +@@ -200,7 +202,11 @@ func refreshRun(opts *RefreshOptions) error { + + additionalScopes.RemoveValues(opts.RemoveScopes) + +- authedToken, authedUser, err := opts.AuthFlow(plainHTTPClient, opts.IO, hostname, additionalScopes.ToSlice(), opts.Interactive, opts.Clipboard) ++ copyToClipboard := opts.Clipboard ++ if !opts.clipboardSet && cfg.CopyToClipboard().Value == "enabled" { ++ copyToClipboard = true ++ } ++ authedToken, authedUser, err := opts.AuthFlow(plainHTTPClient, opts.IO, hostname, additionalScopes.ToSlice(), opts.Interactive, copyToClipboard) + if err != nil { + return err + } +diff --git a/pkg/cmd/auth/refresh/refresh_test.go b/pkg/cmd/auth/refresh/refresh_test.go +index 9353de3..9db7793 100644 +--- a/pkg/cmd/auth/refresh/refresh_test.go ++++ b/pkg/cmd/auth/refresh/refresh_test.go +@@ -213,15 +213,16 @@ type authOut struct { + + func Test_refreshRun(t *testing.T) { + tests := []struct { +- name string +- opts *RefreshOptions +- prompterStubs func(*prompter.PrompterMock) +- cfgHosts []string +- authOut authOut +- oldScopes string +- wantErr string +- nontty bool +- wantAuthArgs authArgs ++ name string ++ opts *RefreshOptions ++ prompterStubs func(*prompter.PrompterMock) ++ cfgHosts []string ++ authOut authOut ++ oldScopes string ++ wantErr string ++ nontty bool ++ wantAuthArgs authArgs ++ copyToClipboard string + }{ + { + name: "no hosts configured", +@@ -270,6 +271,22 @@ func Test_refreshRun(t *testing.T) { + clipboard: true, + }, + }, ++ { ++ name: "no hostname, one host configured, copy_to_clipboard config enabled", ++ cfgHosts: []string{ ++ "github.com", ++ }, ++ copyToClipboard: "enabled", ++ opts: &RefreshOptions{ ++ Hostname: "", ++ }, ++ wantAuthArgs: authArgs{ ++ hostname: "github.com", ++ scopes: []string{}, ++ secureStorage: true, ++ clipboard: true, ++ }, ++ }, + { + name: "no hostname, one host configured", + cfgHosts: []string{ +@@ -487,6 +504,9 @@ func Test_refreshRun(t *testing.T) { + _, err := cfg.Authentication().Login(hostname, "test-user", "abc123", "https", false) + require.NoError(t, err) + } ++ if tt.copyToClipboard != "" { ++ cfg.Set("", "copy_to_clipboard", tt.copyToClipboard) ++ } + tt.opts.Config = func() (gh.Config, error) { + return cfg, nil + } +diff --git a/pkg/cmd/config/list/list_test.go b/pkg/cmd/config/list/list_test.go +index 3a04647..5549ac8 100644 +--- a/pkg/cmd/config/list/list_test.go ++++ b/pkg/cmd/config/list/list_test.go +@@ -107,6 +107,7 @@ func Test_listRun(t *testing.T) { + accessible_prompter=disabled + spinner=enabled + telemetry=enabled ++ copy_to_clipboard=disabled + `), + }, + } diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/review.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/review.md new file mode 100644 index 00000000..39258262 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/review.md @@ -0,0 +1,31 @@ +# Expert Review: copy_to_clipboard config default + +Reviewed: `lld.md` (initial draft). Reviews performed inline, one persona at a time, per the single-agent variant. + +## Persona: Staff Engineer (architecture) + +- **Should-fix, resolved.** The draft cited a non-existent YAML config for `gh` keys. The claim was unverified and irrelevant: the three-layer rule in the benchmark repo does not apply to the cloned `gh` source. Removed from the LLD; the registry `DefaultValue` is the single source of the default. +- **Verified.** `defaultConfigStr` (`internal/config/config.go`) does not list registry keys such as `telemetry`. No template change is needed. + +## Persona: gh CLI Maintainer (conventions) + +- **Must-fix, resolved.** Resolution must happen in `loginRun` and `refreshRun`, not at command construction. Construction-time config reads would run on every `gh` invocation including `--help`. LLD Alternatives #1 records this. +- **Should-fix, resolved.** The file table omitted `pkg/cmd/auth/login/login_test.go`. Login tests use `NewMockConfigFromString`, so the `test.go` wiring is required for them. Row added. +- **Verified.** `refreshRun` already loads `cfg` before `opts.AuthFlow` is called (`refresh.go` lines 134-137), so the OR costs no extra config load. + +## Persona: Security Reviewer + +- **Note, no action.** The setting only controls whether a one-time device code is written to the system clipboard. It does not change token storage, scopes, or the OAuth flow. Clipboard exposure is the same as the existing `-c` path. + +## Persona: QA + +- **Should-fix, resolved.** Test plan must cover config unset, config `enabled` with no flag, and `-c` with config `disabled`. Covered in `testing.md`. +- **Must-fix, open until implementation.** `copy_to_clipboard` must appear in `config list` expected output (`list_test.go`), or the existing list test fails. + +## Persona: Product / UX + +- **Note, accepted.** Config `enabled` cannot be overridden per run; there is no `--no-clipboard`. Issue lists this as out of scope. Accepted. + +## Disposition + +No blocker remains in the LLD. The must-fix QA item is an implementation step (step 7 of the LLD) and is tracked in `testing.md`. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/testing.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/testing.md new file mode 100644 index 00000000..c315d1dd --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/auth-clipboard-config-default/testing.md @@ -0,0 +1,37 @@ +# Testing Plan: copy_to_clipboard config default + +## Functional Tests + +| # | Scenario | Expected | +|---|----------|----------| +| F1 | `gh config get copy_to_clipboard` with key unset | Prints `disabled`. | +| F2 | `gh config set copy_to_clipboard enabled`, then `gh config get copy_to_clipboard` | Prints `enabled`. | +| F3 | `gh config set copy_to_clipboard maybe` | Rejected; only `enabled` and `disabled` allowed. | +| F4 | `gh config list` with key unset | Output includes `copy_to_clipboard=disabled`. | +| F5 | `gh auth login` with config `enabled`, no `-c` | `AuthFlow` receives clipboard `true`. | +| F6 | `gh auth refresh` with config `enabled`, no `-c` | `AuthFlow` receives clipboard `true`. | +| F7 | `gh auth login -c` with config unset | Clipboard `true` (unchanged behavior). | +| F8 | `gh auth refresh -c` with config unset | Clipboard `true` (unchanged behavior). | +| F9 | `gh auth login` with config unset, no `-c` | Clipboard `false` (unchanged default). | + +## Backwards-Compatibility Tests + +- Existing `login_test.go` and `refresh_test.go` cases with `-c` or without it, and no config value, must still pass unchanged in their expected arguments. +- Existing `list_test.go` expected output gains exactly one line; no other keys change. + +## UX Tests + +- The `Description` string is shown by `gh config list` and help. It must read as a plain sentence and name both commands. + +## Deployment Tests + +- None. The change is a Go source change with a regenerated mock. No schema migration: the config file gains a key only when a user sets it. + +## End-to-End Tests + +- Build the `gh` binary from the patched clone and run F1 through F4 against an isolated `GH_CONFIG_DIR`. +- F5 and F6 are covered by unit tests with `AuthFlow` stubs; a live device flow is not run. + +## Verification in This Run + +The swe3 skill does not execute the target repository's test suite. Verification in this run is limited to `gofmt -l` on changed files and `go vet ./internal/config/... ./pkg/cmd/auth/...` after implementation, with results recorded in `implementation.md`. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/github-issue.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/github-issue.md new file mode 100644 index 00000000..ad4245c5 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/github-issue.md @@ -0,0 +1,34 @@ +# Skill discovery misses skills in nested hidden directories even with --allow-hidden-dirs + +## Summary + +`gh skill install --allow-hidden-dirs` does not find skills stored under a dot-prefixed directory that is not at the repository root, for example `packages/app/.claude/skills//SKILL.md`. Only dot-directories at the repository root (`.claude/skills/`, `.agents/skills/`, `.github/skills/`, `.copilot/skills/`) are recognised. Nested skills in standard `skills/` directories already work. + +Related upstream report: cli/cli#13372 (skill discovery does not traverse nested `skills/` directories with `--allow-hidden-dirs`). + +## Reproduction + +1. Create a repository containing `packages/app/.claude/skills/code-review/SKILL.md` with valid frontmatter, and no other skills. +2. Run `gh skill install --allow-hidden-dirs`. +3. Observed: `no skills found in /` and the expected-locations hint. +4. Expected: `code-review` is discovered and offered for install. + +The same repository with the skill at `packages/app/skills/code-review/SKILL.md` is discovered without any flag. That case is not part of this issue. + +## Expected behaviour + +- A skill whose `skills/` directory sits under a dot-prefixed directory at any depth is discovered, provided the dot-directory is immediately followed by `skills/`. +- It is classified as a hidden-dir skill, so the existing `--allow-hidden-dirs` gate, the origin warning, and the "no standard skills found, but N skill(s) exist in hidden directories" error apply unchanged. +- Without `--allow-hidden-dirs`, nested hidden skills stay excluded. + +## Non-goals + +- Installing from arbitrary non-`skills` directories (for example `skills-catalog//`) by path. That is a separate request. +- Changing the `--allow-hidden-dirs` default or the origin warning text. +- Matching `.x/nested/skills/` (a `skills/` directory that is not directly under the dot-directory). + +## Acceptance criteria + +- `discovery.matchHiddenDirConventions` matches `pkg/.claude/skills//SKILL.md` (hidden-dir) and `a/b/.github/skills///SKILL.md` (hidden-dir-namespaced). +- `.claude/nested/skills//SKILL.md` and `x/.claude/nested/skills//SKILL.md` still do not match. +- Existing discovery and install tests pass unchanged. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/implementation.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/implementation.md new file mode 100644 index 00000000..492a85e3 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/implementation.md @@ -0,0 +1,35 @@ +# Implementation: nested hidden-directory skill discovery + +Source: `cli/cli` at `f96972c` (base `v2.93.0`), edited in the disposable clone `/tmp/swe-clone-skill-install-nested-discovery/cli`. The captured diff is [patch.diff](./patch.diff) (3 files, +52 / -24). + +## What changed + +- `internal/skills/discovery/discovery.go` + - `matchHiddenDirConventions` now accepts a dot-prefixed directory at any depth, not only at the repository root. The `skills/` directory must still be a direct child of the dot-directory. + - New helper `isHiddenSkillsDir(dir)`: true when `dir` ends in `skills` and its parent's base name starts with `.`. The `hiddenRoot != "."` guard stops a root-level `skills/` from matching as hidden. + - Doc comment rewritten to show the `/.{host}/skills/...` forms. +- `internal/skills/discovery/discovery_test.go`: four rows added to `TestMatchHiddenDirConventions` (two positive, two negative). +- `skills/gh-skill/SKILL.md`: the `--allow-hidden-dirs` description now says the directory may be nested. + +No change to `pkg/cmd/skills/install` or `pkg/cmd/skills/preview`. Both gate on `discovery.PartitionHiddenDirSkills`, which keys on the convention string, so they pick up the fix without edits. + +## Checks run + +1. Baseline before edits: `go test ./internal/skills/discovery/ ./pkg/cmd/skills/install/` passed. +2. Probe before the fix (throwaway, deleted): `pkg/.claude/skills/code-review/SKILL.md` and `a/b/.github/skills/...` returned nil; `foo/bar/skills/code-review/SKILL.md` matched the standard matcher. +3. After the fix: `go vet` clean, `gofmt -l` empty, `go test ./internal/skills/discovery/ ./pkg/cmd/skills/... -count=1` all `ok`. `TestMatchHiddenDirConventions` passes all 17 subtests, including the four new rows. +4. Built binary (`go build ./cmd/gh`) against a fixture with `packages/app/.claude/skills/code-review/SKILL.md`: + - no flag: "no standard skills found, but 1 skill(s) exist in hidden directories ... Use --allow-hidden-dirs". Install refused. + - `--allow-hidden-dirs`: origin warning printed, `code-review` installed to `~/.claude/skills/code-review/SKILL.md`. +5. Built binary against `pkg/app/skills/code-review/SKILL.md` with no flag: installed through the standard matcher, so the non-dot nested case is unchanged. + +Probe file `probe_scratch_test.go` deleted before the final test run; `git status` shows only the three intended files modified. + +## Not run + +- No live GitHub API run. Discovery was exercised through the local path (`--from-local`), which uses the same matchers (`DiscoverLocalSkillsWithOptions`, the same match loop as the tree-based path). The tree-based path was covered only by the unit table. +- `go test ./...` over the whole CLI was not run; only the discovery and skills packages. + +## Open risk + +Plugin ancestors are not excluded for hidden matches, as recorded in [lld.md](./lld.md). Unchanged behavior from the root-level hidden matcher. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/lld.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/lld.md new file mode 100644 index 00000000..aaf33515 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/lld.md @@ -0,0 +1,75 @@ +# Low-Level Design: nested hidden-directory skill discovery + +Related issue: [github-issue.md](./github-issue.md) + +## Context + +Skill discovery in `internal/skills/discovery/discovery.go` runs one recursive git tree listing per repository (`DiscoverSkillsWithOptions`). Each blob is passed to two matchers in order: + +1. `matchSkillConventions`: standard `skills/`, `skills/{scope}/`, `plugins/`, root conventions. It returns nil for any path containing a dot-prefixed segment (`hasHiddenSegment`), so hidden-directory skills never reach the standard convention. +2. `matchHiddenDirConventions`: returns `hidden-dir` / `hidden-dir-namespaced` only when the dot-directory is at the repository root (`path.Dir(hiddenRoot) == "."`). + +Install (`pkg/cmd/skills/install/install.go` `filterHiddenDirSkills`) and preview (`pkg/cmd/skills/preview/preview.go`) partition on `Skill.IsHiddenDirConvention()`, which compares the convention string. They need no change. + +## Verified current behaviour + +A throwaway probe in the discovery package (deleted after use) printed: + +| path | standard matcher | hidden matcher | +|---|---|---| +| `foo/bar/skills/code-review/SKILL.md` | skills | nil | +| `packages/agent-skills/code-review/SKILL.md` | nil | nil | +| `pkg/.claude/skills/code-review/SKILL.md` | nil | nil (bug) | +| `pkg/.claude/skills/monalisa/code-review/SKILL.md` | nil | nil (bug) | +| `a/b/.github/skills/issue-triage/SKILL.md` | nil | nil (bug) | +| `.claude/nested/skills/code-review/SKILL.md` | nil | nil (intended) | +| `x/.claude/nested/skills/code-review/SKILL.md` | nil | nil (intended) | + +Root cause: the standard matcher excludes hidden segments, and the hidden matcher hard-codes the dot-directory to the repository root. + +## Design + +Generalise `matchHiddenDirConventions` so the dot-directory may appear at any depth, with the one constraint that its `skills/` directory is a direct child. Add a helper: + +```go +// isHiddenSkillsDir reports whether dir is a skills/ directory whose parent is +// a dot-prefixed directory, at any depth: ".claude/skills" or "pkg/.claude/skills". +func isHiddenSkillsDir(dir string) bool { + if path.Base(dir) != "skills" { + return false + } + hiddenRoot := path.Dir(dir) + return hiddenRoot != "." && strings.HasPrefix(path.Base(hiddenRoot), ".") +} +``` + +`matchHiddenDirConventions` then becomes: + +- `parentDir` is a hidden skills dir: return `hidden-dir`. +- `grandparentDir` is a hidden skills dir: validate the namespace, return `hidden-dir-namespaced`. +- Otherwise nil. + +The `hiddenRoot != "."` guard is required: `path.Base(".")` is `"."`, which has the dot prefix, so `skills/x/SKILL.md` would otherwise match as hidden. + +Dedup in `DiscoverSkillsWithOptions` (`seen[m.skillDir]`) is unchanged. The standard matcher runs first, and the two matchers cannot both match a path because the standard one rejects hidden segments. + +## Files + +- `cli/internal/skills/discovery/discovery.go`: add `isHiddenSkillsDir`; rewrite the two branches of `matchHiddenDirConventions`; update its doc comment (it currently lists only the root forms). +- `cli/internal/skills/discovery/discovery_test.go`: add rows to `TestMatchHiddenDirConventions` (below). +- `cli/skills/gh-skill/SKILL.md` (the `--allow-hidden-dirs` bullet): one clause noting that nested dot-directories are included. + +No change to install, preview, the flag, or the warning text. + +## Test plan (summary; full list in testing.md) + +- New matcher rows: nested hidden-dir, nested hidden-dir-namespaced, and two negative cases (`pkg/.claude/nested/skills/...`, `skills/` without a dot parent at depth). +- Existing rows, including `.claude/nested/skills/code-review/SKILL.md` expected nil, must still pass. + +## Risks and decisions + +- Behaviour change without the flag: a repository whose only skills are nested hidden ones now gets "no standard skills found, but N skill(s) exist in hidden directories" instead of "no skills found". This is the intended message for hidden skills and is the same message root-level hidden skills already produce. Accepted. +- Repositories with vendored copies under nested dot-directories become visible to `--allow-hidden-dirs` users. The existing origin warning already covers this class. Accepted. +- Plugin ancestors: the standard matcher excludes `plugins/` ancestors for `skills/` nesting. The new hidden matcher does not, because the original hidden matcher did not either, and `plugins/x/.claude/skills/y` is an unusual layout. Not changed; noted for review. +- Dot-directories that are not agent homes (for example `.cache/skills/`) are matched, but only with `--allow-hidden-dirs`. Same exposure as root-level `.cache/skills/` today. +- No performance impact: matching is string-only, per blob, in the existing single tree pass. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/patch.diff b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/patch.diff new file mode 100644 index 00000000..61240a93 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/patch.diff @@ -0,0 +1,133 @@ +diff --git a/internal/skills/discovery/discovery.go b/internal/skills/discovery/discovery.go +index df17c5b..2cf651a 100644 +--- a/internal/skills/discovery/discovery.go ++++ b/internal/skills/discovery/discovery.go +@@ -451,18 +451,19 @@ func matchSkillConventions(entry treeEntry) *skillMatch { + } + + // matchHiddenDirConventions checks if a blob path matches a skill convention +-// under a hidden (dot-prefixed) root directory. These patterns mirror the +-// standard skills/ conventions but rooted under .{host}/skills/: ++// under a dot-prefixed directory. The dot-directory may sit at any depth, but ++// its skills/ directory must be a direct child. These patterns mirror the ++// standard skills/ conventions: + // +-// - .{host}/skills/*/SKILL.md -> "hidden-dir" +-// - .{host}/skills/{scope}/*/SKILL.md -> "hidden-dir-namespaced" ++// - /.{host}/skills/*/SKILL.md -> "hidden-dir" ++// - /.{host}/skills/{scope}/*/SKILL.md -> "hidden-dir-namespaced" + func matchHiddenDirConventions(entry treeEntry) *skillMatch { + if path.Base(entry.Path) != "SKILL.md" { + return nil + } + +- // .{host}/skills/* +- // .{host}/skills/{scope}/* ++ // /.{host}/skills/* ++ // /.{host}/skills/{scope}/* + dir := path.Dir(entry.Path) + skillName := path.Base(dir) + +@@ -470,34 +471,38 @@ func matchHiddenDirConventions(entry treeEntry) *skillMatch { + return nil + } + +- // .{host}/skills +- // .{host}/skills/{scope} ++ // /.{host}/skills ++ // /.{host}/skills/{scope} + parentDir := path.Dir(dir) + +- // .{host}/skills/*/SKILL.md +- if path.Base(parentDir) == "skills" { +- hiddenRoot := path.Dir(parentDir) +- if path.Dir(hiddenRoot) == "." && strings.HasPrefix(hiddenRoot, ".") { +- return &skillMatch{entry: entry, name: skillName, skillDir: dir, convention: "hidden-dir"} +- } ++ // /.{host}/skills/*/SKILL.md ++ if isHiddenSkillsDir(parentDir) { ++ return &skillMatch{entry: entry, name: skillName, skillDir: dir, convention: "hidden-dir"} + } + +- // .{host}/skills/{scope}/*/SKILL.md ++ // /.{host}/skills/{scope}/*/SKILL.md + grandparentDir := path.Dir(parentDir) +- if path.Base(grandparentDir) == "skills" { +- hiddenRoot := path.Dir(grandparentDir) +- if path.Dir(hiddenRoot) == "." && strings.HasPrefix(hiddenRoot, ".") { +- namespace := path.Base(parentDir) +- if !validateName(namespace) { +- return nil +- } +- return &skillMatch{entry: entry, name: skillName, namespace: namespace, skillDir: dir, convention: "hidden-dir-namespaced"} ++ if isHiddenSkillsDir(grandparentDir) { ++ namespace := path.Base(parentDir) ++ if !validateName(namespace) { ++ return nil + } ++ return &skillMatch{entry: entry, name: skillName, namespace: namespace, skillDir: dir, convention: "hidden-dir-namespaced"} + } + + return nil + } + ++// isHiddenSkillsDir reports whether dir is a skills/ directory whose parent is ++// a dot-prefixed directory, at any depth: ".claude/skills" or "pkg/.claude/skills". ++func isHiddenSkillsDir(dir string) bool { ++ if path.Base(dir) != "skills" { ++ return false ++ } ++ hiddenRoot := path.Dir(dir) ++ return hiddenRoot != "." && strings.HasPrefix(path.Base(hiddenRoot), ".") ++} ++ + // DiscoverOptions controls optional discovery behaviors. + type DiscoverOptions struct { + } +diff --git a/internal/skills/discovery/discovery_test.go b/internal/skills/discovery/discovery_test.go +index bc3ce97..e6fb33b 100644 +--- a/internal/skills/discovery/discovery_test.go ++++ b/internal/skills/discovery/discovery_test.go +@@ -232,6 +232,29 @@ func TestMatchHiddenDirConventions(t *testing.T) { + path: ".claude/nested/skills/code-review/SKILL.md", + wantNil: true, + }, ++ { ++ name: "nested claude skills directory", ++ path: "pkg/.claude/skills/code-review/SKILL.md", ++ wantName: "code-review", ++ wantConvention: "hidden-dir", ++ }, ++ { ++ name: "nested namespaced hidden dir skill", ++ path: "a/b/.github/skills/monalisa/issue-triage/SKILL.md", ++ wantName: "issue-triage", ++ wantNamespace: "monalisa", ++ wantConvention: "hidden-dir-namespaced", ++ }, ++ { ++ name: "dot dir not directly followed by skills at depth", ++ path: "pkg/.claude/nested/skills/code-review/SKILL.md", ++ wantNil: true, ++ }, ++ { ++ name: "nested skills dir without dot parent is not hidden", ++ path: "pkg/app/skills/code-review/SKILL.md", ++ wantNil: true, ++ }, + { + name: "invalid skill name", + path: ".claude/skills/../SKILL.md", +diff --git a/skills/gh-skill/SKILL.md b/skills/gh-skill/SKILL.md +index e0db2fe..d383cba 100644 +--- a/skills/gh-skill/SKILL.md ++++ b/skills/gh-skill/SKILL.md +@@ -50,7 +50,7 @@ Useful flags: + - `--pin ` - pin to a tag, branch, or commit SHA. Mutually exclusive + with `--from-local` and with inline `@version` syntax. + - `--allow-hidden-dirs` - also discover skills under dot-directories such +- as `.claude/skills/`. Don't use this unless you need to, it comes with risks. ++ as `.claude/skills/`, at the repository root or nested (e.g. `packages/app/.claude/skills/`). Don't use this unless you need to, it comes with risks. + - `--force` - overwrite an existing install. + + ## Update diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/review.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/review.md new file mode 100644 index 00000000..2a16c133 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/review.md @@ -0,0 +1,33 @@ +# Expert review: nested hidden-directory skill discovery + +Reviewed against [lld.md](./lld.md). Personas were applied inline, in sequence, by the single agent that wrote the design. + +## Maintainer (discovery package) + +- Finding (should-fix): the hidden matcher's doc comment lists only the root forms. Resolution: update the comment with `{prefix}/.{host}/skills/...` forms. Included in the LLD file list. +- Finding (must-fix): without the `hiddenRoot != "."` guard, `path.Base(".")` yields a dot-prefixed name and `skills/x/SKILL.md` would become a hidden-dir match. Resolution: guard added to `isHiddenSkillsDir` in the LLD. Covered by a negative test row. +- Verdict: approve with the two changes above. + +## Install and preview owner + +- Finding: no change needed in install or preview; both partition on the convention string. Confirmed by grep: `PartitionHiddenDirSkills` is the only partition point in both packages. +- Finding (should-fix): the no-flag error path now fires for repos that previously showed "no skills found". Resolution: intended behaviour, documented in the LLD risks section. No code change. +- Verdict: approve. + +## Security + +- Finding: nested hidden directories expand what `--allow-hidden-dirs` installs from. Skills there are third-party copies by design, and the existing origin warning covers that. The default (no flag) path still excludes them, so the default attack surface is unchanged. Resolution: no code change; risk recorded in the LLD. +- Finding: no new input is parsed; matching is string-only on tree paths already returned by the API. No new network call. +- Verdict: approve. + +## Test quality + +- Finding (must-fix): a test that only asserts the new positive row would not catch over-matching. Resolution: add negative rows for `pkg/.claude/nested/skills/...` and for a `skills/` directory with no dot parent at depth, mirroring the existing `.claude/nested/skills` row. +- Finding: do not add a permanent test for the probe itself; the table rows cover the same boundary. Resolution: probe file to be deleted after use. +- Verdict: approve with the negative rows. + +## Resolutions applied to the LLD + +- Doc-comment update and `hiddenRoot != "."` guard: already reflected in the LLD design section. +- Negative test rows: reflected in the test plan summary; full list in testing.md. +- Deferred: the plugin-ancestor exclusion for hidden matches is not added. Stated as an open risk in the LLD so the reviewer can decide. diff --git a/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/testing.md b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/testing.md new file mode 100644 index 00000000..6f8dfe18 --- /dev/null +++ b/benchmarks/swe-benchmark-data/do-not-include/claude-haiku-5-5-misplaced-artifacts/omp/swe3/cli-cli-v1/skill-install-nested-discovery/testing.md @@ -0,0 +1,59 @@ +# Testing plan: nested hidden-directory skill discovery + +Repository: `cli/cli` at `f96972c` (see [lld.md](./lld.md)). All commands run from the `cli/` checkout. + +## 1. Functional tests (matcher) + +Add rows to `TestMatchHiddenDirConventions` in `internal/skills/discovery/discovery_test.go`: + +| name | path | expected | +|---|---|---| +| nested claude skills directory | `pkg/.claude/skills/code-review/SKILL.md` | hidden-dir, `code-review` | +| nested github namespaced skill | `a/b/.github/skills/monalisa/issue-triage/SKILL.md` | hidden-dir-namespaced, `issue-triage`, ns `monalisa` | +| nested dot dir not followed by skills | `pkg/.claude/nested/skills/code-review/SKILL.md` | nil | +| skills dir without dot parent at depth | `pkg/app/skills/code-review/SKILL.md` | nil from hidden matcher (standard matcher still matches) | + +Run: + +```bash +go test ./internal/skills/discovery/ -run TestMatchHiddenDirConventions -v +``` + +Existing row `too deeply nested hidden dir` (`.claude/nested/skills/code-review/SKILL.md` -> nil) must still pass. + +## 2. Backwards compatibility + +```bash +go test ./internal/skills/discovery/ ./pkg/cmd/skills/... +``` + +All existing tests must pass unchanged. Root-level hidden rows, standard rows, plugin rows, and install/preview hidden-dir tests (`allow-hidden-dirs includes hidden skills`, and the "no standard skills found" error) are unchanged. + +## 3. UX + +- Without `--allow-hidden-dirs`, a repo with only `pkg/.claude/skills/x/SKILL.md` must print the "no standard skills found, but 1 skill(s) exist in hidden directories" error. Verified by the install test that already covers that message for root-level hidden skills; the matcher change feeds the same path. +- With `--allow-hidden-dirs`, the origin warning is printed. Same code path as root-level hidden skills. + +## 4. Deployment surface + +Not applicable: CLI library change, no Docker, Terraform, or ECS surface. + +## 5. End-to-end + +- Smoke: build the binary and run discovery against a local fixture (see implementation.md for the exact command used). +- Upstream-API E2E is out of scope: the discovery path is driven by the GitHub tree API; the matcher is the changed boundary and is exercised with real tree paths in the unit table. + +## 6. Test execution checklist + +- [ ] Section 1 matcher rows added and passing +- [ ] Section 2 full discovery and install packages passing +- [ ] Section 3 UX messages confirmed by existing install tests +- [ ] Section 4 not applicable +- [ ] Section 5 smoke run recorded in implementation.md +- [ ] Probe file removed + +## Guidance + +1. Each new row asserts a single boundary (positive or negative). +2. Negative rows protect against over-matching the `.claude/nested/skills` restriction. +3. No new test for the probe itself; the table rows are the permanent form. From b1d61c69b6fd23c5d1987e4093d76cf8bd2ab9e5 Mon Sep 17 00:00:00 2001 From: Amit Arora Date: Wed, 14 Oct 2026 22:27:03 +0000 Subject: [PATCH 2/4] Answer #221 for set A: cli/cli rubric passes, zero-shot replay, results doc Score the 21 cli-cli-v1 tasks with the swe-router rubric (Haiku 5.5 and Opus 5.5, three repeats each) and replay routing among the three Claude 5.5 models two ways: columns built from cli/cli itself (leave-one-out) and columns built from v2 alone (zero-shot). Add a v2 replay limited to the same three models so the two repositories compare directly. eval_swe_router.py gains --columns-from and --columns-rubric-inputs: build the bucket columns from another dataset's results and rubric pass, then route this dataset's tasks against them. Absolute buckets only, and no --holdout, since the routed tasks are never in the columns. docs/results-cli-cli-v1.md answers the four questions. Rankings hold; every model scores 3 to 8 points higher on cli/cli; v2's columns transfer and route a little more cautiously than cli/cli's own. On cli/cli the router sends 17 to 21 tasks to Haiku 5.5, and its two wrong-folder failures cost most of the quality, because route.py reports completion per bucket but never uses it to choose. Phase 1 should make the router see failures. --- AGENTS.md | 1 + benchmarks/scripts/eval_swe_router.py | 94 +- benchmarks/tests/test_swe_router_rubric.py | 47 + ...er-rubric-cli-cli-v1-claude-haiku-5-5.json | 3375 +++++++++++++++++ ...ter-rubric-cli-cli-v1-claude-opus-5-5.json | 3368 ++++++++++++++++ .../cli-cli-v1-haiku-5-5-in-repo.json | 573 +++ .../cli-cli-v1-haiku-5-5-zero-shot.json | 574 +++ .../cli-cli-v1-opus-5-5-in-repo.json | 571 +++ .../cli-cli-v1-opus-5-5-zero-shot.json | 574 +++ .../v2-claude-5-5-only-haiku-5-5-in-repo.json | 575 +++ .../v2-claude-5-5-only-opus-5-5-in-repo.json | 576 +++ docs/results-cli-cli-v1.md | 100 + ...uter-rubric-cli-cli-v1-claude-haiku-5-5.md | 119 + ...outer-rubric-cli-cli-v1-claude-opus-5-5.md | 119 + 14 files changed, 10662 insertions(+), 4 deletions(-) create mode 100644 docs/metrics/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.json create mode 100644 docs/metrics/swe-router-rubric-cli-cli-v1-claude-opus-5-5.json create mode 100644 docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-in-repo.json create mode 100644 docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-zero-shot.json create mode 100644 docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-in-repo.json create mode 100644 docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-zero-shot.json create mode 100644 docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-haiku-5-5-in-repo.json create mode 100644 docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-opus-5-5-in-repo.json create mode 100644 docs/results-cli-cli-v1.md create mode 100644 docs/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.md create mode 100644 docs/swe-router-rubric-cli-cli-v1-claude-opus-5-5.md diff --git a/AGENTS.md b/AGENTS.md index 0193e529..1cc21ecb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -550,6 +550,7 @@ Read the doc that covers what you are about to do rather than rediscovering it. - [docs/best-harness-selection.md](docs/best-harness-selection.md) -- how the combined cost/quality chart picks one harness per model. - [docs/results-swe3.md](docs/results-swe3.md), [docs/results-swe2.md](docs/results-swe2.md) -- published results and their footnotes. - [docs/results-swe3-v2.md](docs/results-swe3-v2.md) -- results on the v2 dataset (15 release-sourced tasks, complexity-balanced). **Never merge v2 scores into a v1 table**; the task sets, refs and difficulty mixes differ. +- [docs/results-cli-cli-v1.md](docs/results-cli-cli-v1.md) -- the cross-repo check (#221): Claude 5.5 models on 21 cli/cli (Go) tasks, and whether v2's rankings, score levels and swe-router columns carry over. Keep its tables apart from v1 and v2. - [docs/model-selection-by-complexity.md](docs/model-selection-by-complexity.md) -- which model to use for which class of work, derived from the v2 complexity tiers. **Questions rather than reference pages:** diff --git a/benchmarks/scripts/eval_swe_router.py b/benchmarks/scripts/eval_swe_router.py index 0f704b8c..fa283471 100644 --- a/benchmarks/scripts/eval_swe_router.py +++ b/benchmarks/scripts/eval_swe_router.py @@ -458,6 +458,7 @@ def _evaluate( judged_tiers: dict[str, str] | None = None, column_for: Callable[[str], dict[str, str]] | None = None, stats_for: Callable[[str], dict[str, dict[str, Any]]] | None = None, + fixed_stats: dict[str, dict[str, Any]] | None = None, ) -> dict[str, Any]: """Replay the router over every task and compare it to the baseline. @@ -480,6 +481,9 @@ def _evaluate( rubric buckets do. Overrides ``judged_tiers`` and the dataset label. stats_for: Given the task being routed, every model's stats with one predicted score under ``PREDICTED_TIER``. Overrides ``column_for``. + fixed_stats: Every model's stats from ANOTHER dataset, routed against + for every task. ``results`` then only supplies each routed task's + outcome. ``column_for`` still names the routed task's column. Returns: A result mapping with ``rows`` (one per task) and ``totals``. @@ -497,7 +501,8 @@ def _evaluate( tmp_dir = Path(tempfile.mkdtemp(prefix="router-eval-")) in_sample_path = tmp_dir / "models-in-sample.json" in_sample_path.write_text( - json.dumps(_models_json(_tier_stats(results), provenance)), encoding="utf-8" + json.dumps(_models_json(fixed_stats or _tier_stats(results), provenance)), + encoding="utf-8", ) for task in tasks: task_id = task["id"] @@ -511,7 +516,9 @@ def _evaluate( columns = column_for(task_id) tier = columns[task_id] grouped = _regrouped(results, columns) - if stats_for is not None: + if fixed_stats is not None: + models_path = in_sample_path + elif stats_for is not None: tier = PREDICTED_TIER models_path = tmp_dir / f"models-{task_id}.json" models_path.write_text( @@ -895,7 +902,10 @@ def _document(report: dict[str, Any]) -> str: cfg = report["config"] prov = report["provenance"] sampling = ( - "Leave-one-out: each task routes from tier means recomputed with that " + f"Zero-shot: every task routes from columns built on `{cfg['columns_from']}` " + "alone, a different repository, so no pick saw any task of this one." + if cfg.get("columns_from") + else "Leave-one-out: each task routes from tier means recomputed with that " "task excluded, so no pick knows the run it is scored against." if cfg["holdout"] else "In-sample: the router read tier means computed from these same " @@ -1030,6 +1040,44 @@ def column_for(routed: str) -> dict[str, str]: return floors, column_for, meta +def _columns_from_other_dataset( + dataset_path: Path, + rubric_inputs: Path, + harness: str, + skill: str, +) -> tuple[dict[str, dict[str, Any]], str]: + """Every model's bucket columns, built from a different dataset. + + This is the zero-shot case: the router has only another repository's + benchmark to go on. Each of that dataset's tasks is filed under the bucket + its own rubric pass gave it, and the columns are its models' means there. + Buckets are absolute, so the routed tasks are placed by the same fixed + ranges. + + Args: + dataset_path: The dataset whose results build the columns. + rubric_inputs: That dataset's rubric pass. + harness: Harness folder. + skill: Skill folder. + + Returns: + ``_tier_stats`` output for that dataset, and its scope. + + Raises: + SystemExit: If the dataset has no results or the rubric pass misses a task. + """ + tasks, scope = _load_tasks(dataset_path) + results = _load_results(harness, skill, scope) + if not results: + raise SystemExit( + f"no results for --columns-from under {harness}/{skill}/{scope}" + ) + _, column_for, _ = _apply_rubric_inputs(rubric_inputs, tasks, "absolute", False) + known = {t["id"] for t in tasks} + columns = {tid: c for tid, c in column_for("").items() if tid in known} + return _tier_stats(_regrouped(results, columns)), scope + + def _parse_args() -> argparse.Namespace: """Parse command-line arguments.""" parser = argparse.ArgumentParser( @@ -1097,6 +1145,20 @@ def _parse_args() -> argparse.Namespace: "column means, or against each model's score predicted from the task's " "difficulty sum by a line fitted over the other tasks. Default: %(default)s.", ) + parser.add_argument( + "--columns-from", + type=Path, + default=None, + help="Zero-shot replay: build the bucket columns from THIS other dataset's " + "results instead of --dataset's, and route --dataset's tasks against them. " + "Needs --rubric-inputs and --columns-rubric-inputs; absolute buckets only.", + ) + parser.add_argument( + "--columns-rubric-inputs", + type=Path, + default=None, + help="The rubric pass for --columns-from's dataset.", + ) parser.add_argument( "--floors-file", type=Path, @@ -1193,6 +1255,24 @@ def main() -> None: stats_for = partial(_predicted_stats, results, sums, holdout=args.holdout) judged_meta["column_model"] = "regression" + fixed_stats = None + if args.columns_from: + if not (args.rubric_inputs and args.columns_rubric_inputs): + raise SystemExit( + "--columns-from needs --rubric-inputs and --columns-rubric-inputs" + ) + if bucketing != "absolute" or args.column_model != "buckets" or args.holdout: + raise SystemExit( + "--columns-from supports absolute buckets only, without --holdout " + "(the routed tasks are never in the columns)" + ) + columns_path = args.columns_from + if not columns_path.is_absolute(): + columns_path = BENCHMARKS_DIR / columns_path + fixed_stats, columns_scope = _columns_from_other_dataset( + columns_path, args.columns_rubric_inputs, args.harness, args.skill + ) + judged_meta["columns_from"] = columns_scope if args.judged_inputs: judged = json.loads(args.judged_inputs.read_text(encoding="utf-8")) judged_meta = judged.get("judged_by") or {} @@ -1236,6 +1316,7 @@ def main() -> None: judged_tiers=judged_tiers, column_for=column_for, stats_for=stats_for, + fixed_stats=fixed_stats, ) except RouteError as exc: raise SystemExit(f"routing failed at floor {floor}: {exc}") from exc @@ -1245,7 +1326,11 @@ def main() -> None: "mean score %.2f vs %.2f (%+.2f; %+.2f with failures as 0), " "%d task(s) under floor", floor, - "leave-one-out" if args.holdout else "in-sample", + "zero-shot" + if fixed_stats + else "leave-one-out" + if args.holdout + else "in-sample", totals["routed_total_cost_usd"], totals["baseline_total_cost_usd"], totals["cost_saving_pct"], @@ -1270,6 +1355,7 @@ def main() -> None: "no_allow_list": args.no_allow_list, "tie_band": args.tie_band, "holdout": args.holdout, + "columns_from": judged_meta.get("columns_from"), "judged_by": judged_meta, "floor_note": ( "Required quality per task, from the rubric pass by " diff --git a/benchmarks/tests/test_swe_router_rubric.py b/benchmarks/tests/test_swe_router_rubric.py index 7a05f6a1..3d226003 100644 --- a/benchmarks/tests/test_swe_router_rubric.py +++ b/benchmarks/tests/test_swe_router_rubric.py @@ -6,6 +6,7 @@ import sys import tempfile import unittest +from unittest import mock from pathlib import Path _SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "scripts" @@ -258,6 +259,52 @@ def test_rejects_a_file_missing_a_task(self) -> None: ev._apply_rubric_inputs(self._inputs({}), [{"id": "a"}], "absolute", False) +class ZeroShotColumnsTest(unittest.TestCase): + def _columns(self, rubric_tasks: dict) -> dict: + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump({"mode": "rubric", "tasks": rubric_tasks}, tmp) + tmp.close() + rec = lambda s: { # noqa: E731 + "score": s, + "cost_usd": 1.0, + "failed": False, + "complexity": "low", + } + results = {"m1": {"a": rec(90.0), "b": rec(60.0)}} + tasks = [{"id": "a", "complexity": "low"}, {"id": "b", "complexity": "low"}] + with ( + mock.patch.object(ev, "_load_tasks", return_value=(tasks, "other")), + mock.patch.object(ev, "_load_results", return_value=results), + ): + stats, _ = ev._columns_from_other_dataset( + Path("other.yaml"), Path(tmp.name), "omp", "swe3" + ) + return stats["m1"]["score_by_complexity"] + + def test_columns_are_the_other_datasets_bucket_means(self) -> None: + cols = self._columns( + { + "a": {"required_quality": 60, "bucket": "easy", "difficulty_sum": 0}, + "b": { + "required_quality": 70, + "bucket": "very_hard", + "difficulty_sum": 7, + }, + } + ) + self.assertEqual(cols, {"trivial": 90.0, "high": 60.0}) + + def test_rubric_entries_outside_that_dataset_are_ignored(self) -> None: + cols = self._columns( + { + "a": {"required_quality": 60, "bucket": "easy", "difficulty_sum": 0}, + "b": {"required_quality": 70, "bucket": "easy", "difficulty_sum": 1}, + "x": {"required_quality": 70, "bucket": "hard", "difficulty_sum": 4}, + } + ) + self.assertEqual(cols, {"trivial": 75.0}) + + class RegressionColumnTest(unittest.TestCase): def _results(self) -> dict: # Model "steep" loses 10 points per difficulty step; "flat" does not. diff --git a/docs/metrics/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.json b/docs/metrics/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.json new file mode 100644 index 00000000..fc01aa28 --- /dev/null +++ b/docs/metrics/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.json @@ -0,0 +1,3375 @@ +{ + "judged_by": { + "harness": "omp", + "model": "us.anthropic.claude-haiku-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "repeats": 3, + "started_at": "2026-10-14T22:10:12.037956Z", + "finished_at": "2026-10-14T22:21:48.671252Z", + "judging_cost_usd": 0.6335, + "runs_ok": 63, + "runs_failed": 0 + }, + "mode": "rubric", + "consolidation": "lower median per dimension across repeats; per-repeat quality and bucket kept in quality_seen / buckets_seen", + "tasks": { + "repo-create-reject-empty-name": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "medium", + "summary": "The change needs an empty-name check in interactiveRepoNameAndOwner (create.go:929) that fails right after the \"Repository name\" prompt, using the existing \"argument error\" convention, with a new empty-name case in create_test.go's interactive table. Whether the non-interactive path and whitespace-only names also need rejecting is not stated in the task and was not confirmed against the repo's intent.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 1, + 1 + ], + "coordinated_surfaces": [ + 0, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 1, + "moderate": 2 + }, + "quality_unanimous": true, + "bucket_unanimous": false + }, + "copilot-not-installed-trailing-newline": { + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 2, + "blast_radius": 1 + }, + "required_quality": 75.0, + "difficulty_sum": 1, + "bucket": "easy", + "confidence": "high", + "summary": "The change is a one-line newline fix to the declined-install warning in runCopilot (pkg/cmd/copilot/copilot.go:155) plus a new test asserting the exact ErrOut content and SilentError return, using the existing prompter mock and TTY-simulated iostreams. The main judgement is whether to also add a newline to the non-prompt warning at copilot.go:159, which the task does not name.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 0, + 0, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 2, + 2 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 75.0, + 75.0 + ], + "buckets_seen": { + "moderate": 1, + "easy": 2 + }, + "quality_unanimous": false, + "bucket_unanimous": false + }, + "repo-set-default-unset-without-repo": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "The fix is a small RunE change in pkg/cmd/repo/setdefault/setdefault.go so non-interactive `--unset` skips the 'repository required' check, as `--view` already does, while plain `set-default` keeps the error. The real decision is what to do with a repository argument passed alongside `--unset`, which currently unsets the default silently. Tests go in setdefault_test.go for non-TTY `--unset`, non-TTY `--view` without an argument, and the retained error.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 0, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "pr-checks-all-cancelled-summary": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 2, + "bucket": "moderate", + "confidence": "high", + "summary": "The fix is a one-condition change in printSummary (pkg/cmd/pr/checks/output.go:71) so the summary and tally line are emitted when only cancelled checks exist, plus a new all-cancelled fixture and TTY test in checks_test.go alongside the existing someCancelled case. Existing exact-output tests already cover every other state mix, so a regression test for the all-cancelled case is the main new verification.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 0, + 0, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "extension-uninstall-alias": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 2, + "bucket": "moderate", + "confidence": "high", + "summary": "Add `uninstall` as a cobra alias on the existing `remove` command in pkg/cmd/extension/command.go, following the `ls` alias pattern, so both names run the same handler and remove is unchanged. Cover it with a new case in TestNewCmdExtension (the remove cases at command_test.go ~553-613 are the template), and decide whether the help requirement means the ALIASES section of `remove --help` or a separate listing.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 0, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "pr-merge-delete-branch-owner-case": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 3, + "bucket": "moderate", + "confidence": "high", + "summary": "The fix is a one-line case-insensitive owner comparison in `crossRepoPR` in pkg/cmd/pr/merge/merge.go, matching the EqualFold convention already used in context/remote.go and internal/ghrepo. The work also needs a new regression test in merge_test.go with an owner-case mismatch, since every existing fixture uses matching case and would not catch a revert.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 2 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 2, + "hard": 1 + }, + "quality_unanimous": true, + "bucket_unanimous": false + }, + "pr-view-json-number-on-issue": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "The fix is to stop the shared PR finder from answering number-only lookups without calling GitHub, so that `pr view --json number` goes through the same `pullRequest(number:)` query as other fields and fails for issue numbers. That change also reaches pr checks and pr diff, which use the same shortcut, so the existing number-only finder test (which pins the buggy no-API path) has to be rewritten and the view JSON path covered with new tests.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "project-item-add-output-non-tty": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 2, + "bucket": "moderate", + "confidence": "medium", + "summary": "The task needs item-add's printResults to emit a stable non-TTY confirmation that includes the already-fetched ProjectItem.Id, while leaving TTY output and --format json unchanged, all within item_add.go and item_add_test.go. The main risks are choosing the exact wording and scope (whether sibling TTY-gated project commands should change too) and adding the missing non-TTY test.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "release-create-404-scope-blame": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "high", + "summary": "The fix is in `tokenHasWorkflowScope` (`pkg/cmd/release/create/http.go:301-312`), which must trim whitespace from the comma-split `X-Oauth-Scopes` value the way `api/client.go` and `oauth_scopes.go` already do, and it needs a regression test using GitHub's `, `-separated header in `create_test.go`. The harder part is deciding how the hint should coexist with the underlying 404 for tokens that truly lack `workflow`, because the task asks both to keep the hint and to stop hiding the real cause.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "pr-comment-attach-ghu-tokens": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "In this repository the fix is a deliberate change to the allowlist in internal/attachments/client.go (uploadTokenTypes), which is the only gate on which credentials may reach the upload endpoint, and the existing 'rejects an App user-to-server token' test must be replaced. The silent risk is over-admitting other token kinds or removing the gate, while the existing table tests in client_test.go already cover ghs_/ghr_ rejection and would catch that.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 2 + ], + "verification": [ + 0, + 0, + 0 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "very_hard": 1, + "hard": 2 + }, + "quality_unanimous": true, + "bucket_unanimous": false + }, + "repo-clone-ghe-ssh-ca-url": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "Clone must take the SSH URL from the API (`sshUrl`, which needs adding to the GitHubRepo GraphQL query for both the repo and its parent) instead of rebuilding it from the tenant name in clone.go, with HTTPS unchanged and a defined fallback for empty values. Tests in clone_test.go with httpmock and run.Stub should cover tenant SSH for both the origin and upstream remotes. Confidence is medium because the GraphQL sshUrl vs REST ssh_url equivalence and the fallback behaviour are inferred, not verified.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "codespace-ports-forward-concurrent-map": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 6, + "bucket": "very_hard", + "confidence": "medium", + "summary": "The fix belongs in the codespace port-forwarding path: ForwardPorts runs one goroutine per port, and each goroutine's CreateTunnelPort writes to the TunnelRequestOptions map that the forwarders share through connection.CodespaceConnection, so the change must make per-port tunnel operations safe, either by isolating options per forwarder or by serializing shared-state access as connection.go already does for Connect. A test that forwards several ports at once has to be added, and the offline tunnel mock already supports it, but nothing in the current suite exercises this path.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 2, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 1, + 1, + 2 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 75.0 + ], + "buckets_seen": { + "very_hard": 3 + }, + "quality_unanimous": false, + "bucket_unanimous": true + }, + "issue-view-comments-with-json": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "high", + "summary": "The fix belongs in `pkg/cmd/issue/view/view.go`, where `--comments` is dropped on the `--json` path because `lookupFields` is built only from `Exporter.Fields()`, so either `comments` must be added and preloaded with the existing paginated `preloadIssueComments`, or the flags must be rejected with `cmdutil.MutuallyExclusive` as in discussion view. Tests belong in `view_test.go` alongside the existing `--comments` and `TestIssueView_json_*` cases using the `CommentsForIssue` fixtures. The sibling `pr view` has the same silent no-op, which the task does not mention.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 2 + ], + "verification": [ + 0, + 0, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "release-verify-sha1-prefix": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "medium", + "summary": "The fix derives the digest algorithm (sha1 vs sha256) from the ref the API returns, since FetchRefSHA returns only the SHA string, and threads that single value through the attestation lookup, the sigstore digest policy and the printed output in both release verify commands. Tests must cover both 40- and 64-character refs and assert the printed prefix, which no existing test does, and SHA-256 attestation handling is inferred rather than verified against a real fixture.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 2 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 2, + "very_hard": 1 + }, + "quality_unanimous": true, + "bucket_unanimous": false + }, + "auth-refresh-hint-on-expired-token": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "required_quality": 75.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "The fix belongs in the 401 branch of internal/ghcmd/cmd.go: classify the token used for the failing host (from HTTPError.RequestURL) as login-created when ActiveToken's source is keyring or oauth_token, and suggest gh auth refresh only then. It needs tests for that branch, which currently has no coverage and cannot be injected without refactoring Main. The scores rest on code reading, not on running gh, so confidence is medium.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 2, + 2 + ], + "verification": [ + 1, + 2, + 2 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 75.0, + 75.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": false, + "bucket_unanimous": true + }, + "secret-set-app-agents": { + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "Adding agents means extending the shared app enum, GetSecretApp and IsSupportedSecretEntity, plus flag enums and help text in set, list and delete, and most of the path logic already follows the `{app}` pattern. The non-obvious requirement is that the documented agents org endpoint takes integer selected_repository_ids (like Actions), not the string IDs that the Dependabot special case uses, and the new tests must pin repo and org set/list/delete paths and payloads in the existing httpmock style.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 2, + 2, + 2 + ], + "silent_invariant": [ + 1, + 1, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 2 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 75.0 + ], + "buckets_seen": { + "very_hard": 2, + "hard": 1 + }, + "quality_unanimous": false, + "bucket_unanimous": false + }, + "repo-sync-checked-out-worktree": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 2 + }, + "required_quality": 75.0, + "difficulty_sum": 6, + "bucket": "very_hard", + "confidence": "medium", + "summary": "Syncing a branch checked out in another worktree must not leave that worktree's index and files stale, and the fix has to either update it safely (which requires checking that worktree's dirty state) or refuse with a message that names the worktree. The change sits in pkg/cmd/repo/sync (sync.go, git.go, mocks.go, sync_test.go) and needs new worktree-detection and dirty-check test cases, since the existing mock-based suite covers only the current-worktree and plain-ref paths.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 2, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 2, + 2 + ], + "verification": [ + 1, + 1, + 2 + ], + "blast_radius": [ + 2, + 2, + 2 + ] + }, + "quality_seen": [ + 75.0, + 75.0, + 80.0 + ], + "buckets_seen": { + "very_hard": 2, + "hard": 1 + }, + "quality_unanimous": false, + "bucket_unanimous": false + }, + "auth-clipboard-config-default": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 2, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "The task needs a new global enabled/disabled config key registered through the existing config Options, default and gh.Config accessor (with regenerated mock and test wiring), and read in `gh auth login` so the device code is copied without `--clipboard` while the flag still works. The main risks are the coordinated plumbing that a partial change would leave broken, and an unresolved scope question about whether `auth refresh` should honour the setting.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 2, + 2, + 2 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "project-item-edit-by-name": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "The task needs item-edit to resolve a project item from an issue or PR URL, a project from --owner and number, and a field and option (or value) by case-insensitive name, validated before the mutation, while keeping --id, --field-id and --project-id working. The item-from-URL lookup is not in the repo (the GitHub GraphQL schema is external), so its shape is inferred from the existing projectItems usage in pr/shared and issue/shared, and the change must be covered by new offline gock tests in item_edit_test.go.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "skill-update-dir-keeps-install-path": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 2 + }, + "required_quality": 75.0, + "difficulty_sum": 6, + "bucket": "very_hard", + "confidence": "high", + "summary": "The fix is in pkg/cmd/skills/update/update.go: install into the skill's current parent directory for every scope (with or without --dir) and delete the post-install cleanup block, which means the installer's flat-only write (installer.go:251-253) must not relocate namespaced skills. The existing TestUpdateRun case at update_test.go:711-775 pins the old migrate-and-delete behavior and must be replaced with in-place tests for flat and namespaced layouts, covering both host and --dir paths.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 2, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 2, + 2, + 2 + ] + }, + "quality_seen": [ + 75.0, + 75.0, + 75.0 + ], + "buckets_seen": { + "very_hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "skill-install-nested-discovery": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "The reported path foo/bar/skills//SKILL.md already matches the 'skills' convention at any depth in internal/skills/discovery/discovery.go, so the real work is to confirm which nested case still fails (likely nested hidden-dir skills, which conflicts with the pinned negative test at discovery_test.go L231) before changing matchers and adding nested-skill tests. Scores rest on reading the code and tests, since I did not run the reproduction.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 2 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 0, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 2, + "very_hard": 1 + }, + "quality_unanimous": true, + "bucket_unanimous": false + } + }, + "judgments": [ + { + "task": "auth-clipboard-config-default", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:12:34.183471Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 6335, + "cache_read_tokens": 139313, + "cache_creation_tokens": 41118, + "num_turns": 6, + "latency_seconds": 38.1, + "total_cost_usd": 0.010671958 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 2, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The issue fixes the key shape (`gh config set clipboard enabled`), but the rest follows from repo conventions: `internal/config/config.go` `Options` entries with `AllowedValues: enabled/disabled` and `DefaultValue` (e.g. `spinner`, `accessible_prompter`). Flag/config precedence is not stated in the task and must be inferred from how `--clipboard` is wired in `pkg/cmd/auth/login/login.go` line 243 (`CopyToClipboard: opts.Clipboard`). The task's own wording (`gh auth login`) leaves `auth refresh`, which has the same flag, out of scope.", + "silent_invariant": "`internal/authflow/flow.go` `DisplayCode` copies the one-time code only when `isCopyToClipboard` is true. A wrong default or a dropped config read would not error: the code would just be copied or not copied, with no visible failure. The code is a short-lived device code, not a long-lived credential, so the exposure is limited to clipboard history.", + "coordinated_surfaces": "Several layers must agree: the key constant and `Options` entry in `internal/config/config.go` (which also drives `gh config list` and `set` validation); the `gh.Config` interface in `internal/gh/gh.go` (line 31 `go:generate moq` regenerates `internal/gh/mock/config.go`); the flag and option plumbing in `pkg/cmd/auth/login/login.go` and `pkg/cmd/auth/shared/login_flow.go` (line 150 passes `opts.CopyToClipboard` to `authflow.AuthFlow`). Missing any one leaves the feature half-built.", + "ambiguity": "The natural answer, enabling the key and OR-ing it with the flag in the login options, matches the existing `--clipboard` wiring and the enabled/disabled convention. The repo makes the intended design clear. Open points are flag-versus-config precedence and whether `refresh` follows, both resolvable from the surrounding code.", + "verification": "`pkg/cmd/auth/login/login_test.go` has table tests for `--clipboard` flag parsing (lines 132-151, 296) and `NewIsolatedTestConfig` setup (line 473), so config-driven wiring can be tested there. `pkg/cmd/config/list/list_test.go` (lines 95-112) pins the full `config list` output, so it must be updated when an option is added. Nothing exercises the system clipboard write in `authflow.DisplayCode`, and the config-to-`CopyToClipboard` path has no test yet.", + "blast_radius": "A user-facing CLI setting. A revert fixes it with no lasting harm. The only real cost of a wrong change is that the one-time code is copied to the clipboard when the user did not opt in, which is why the default must stay `disabled`. No credentials or stored records are touched." + }, + "confidence": "medium", + "summary": "The task needs a new `clipboard` config option, added to `internal/config` `Options` and the `gh.Config` interface with regenerated mocks, and read in `gh auth login` so it OR-combines with the existing `--clipboard` flag in `login.go`/`login_flow.go`, with tests in the login and config list suites. The main judgement calls are flag-versus-config precedence and whether `auth refresh` follows, which the repository conventions answer but the task does not state." + }, + { + "task": "auth-clipboard-config-default", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:16:40.936502Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 6268, + "cache_read_tokens": 211382, + "cache_creation_tokens": 50902, + "num_turns": 7, + "latency_seconds": 40.3, + "total_cost_usd": 0.012773387 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 2, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The issue fixes the key name and form (`gh config set clipboard enabled`), but the rest follows from repo conventions: `internal/config/config.go` `Options` (enabled/disabled values, DefaultValue, CurrentValue, PerHostOnly flag), the `gh.Config` accessor pattern in `internal/gh/gh.go`, and the existing `--clipboard` flag in `pkg/cmd/auth/login/login.go:154`. The task text does not say whether the setting is global or per-host, or whether `gh auth refresh --clipboard` (`pkg/cmd/auth/refresh/refresh.go:117`) should honour it. The task scopes the change to `gh auth login`.", + "silent_invariant": "The change only decides whether `authflow.AuthFlow`'s `DisplayCode` (`internal/authflow/flow.go:49-55`) calls `clipboard.WriteAll`. A wrong change is visible: the copy message is missing or appears when it should not. The one-time code is sensitive but is not a credential, and no auth, token-storage or concurrency invariant is touched.", + "coordinated_surfaces": "A working feature needs at least three layers to agree: the config schema (`Options` and `defaultConfigStr` in `internal/config/config.go`), the `gh.Config` interface plus its generated moq mock (`internal/gh/gh.go` with `//go:generate moq`, `internal/gh/mock/config.go`, and the hand-wired `NewMockConfigFromString` in `internal/config/test.go`), and the consumer in `loginRun` (`pkg/cmd/auth/login/login.go`), which passes `CopyToClipboard` into `shared.LoginOptions`. `pkg/cmd/config/set/set.go` validates keys against `Options`, so a missing entry produces an 'unknown key' warning.", + "ambiguity": "The obvious approach is a new `Options` entry plus an accessor, and `loginRun` consults it. The narrower trap is to read the raw key in `login.go` without registering it, which passes a quick test but skips `gh config set` validation and `gh config list`. The repo's option pattern (`internal/config/config.go`, `pkg/cmd/config/set/set.go` `ValidateValue`) makes the intended design clear. Flag-versus-setting precedence is an open detail: the flag can only turn copying on, and the task says the flag must keep working.", + "verification": "Existing tests cover the layers: `pkg/cmd/auth/login/login_test.go` (`Test_NewCmdLogin`, `Test_loginRun_nontty` with `cfgStubs` and `NewIsolatedTestConfig`), `pkg/cmd/config/set/set_test.go` (`Test_ValidateValue`, `Test_ValidateKey`), `internal/config/config_test.go` (`TestGetOrDefaultApplicationDefaults`, `TestTelemetry` as the pattern for new accessors). Nothing exercises `DisplayCode`'s clipboard write (`internal/authflow/flow_test.go` covers only `getViewer` and `getCallbackURI`), and a real clipboard is not available in CI, so the copy path can only be checked through a stubbed or throwaway test.", + "blast_radius": "This is a user-facing CLI with a default-off setting (the default would be `disabled`, following `color_labels` and `prefer_editor_prompt` in `Options`). A mistake shows up as a misplaced copy or no copy, and reverting the release or `gh config set clipboard disabled` fixes it. No stored credentials or data are affected." + }, + "confidence": "medium", + "summary": "The task needs a new `clipboard` option registered in `internal/config` `Options` with an accessor on the `gh.Config` interface (and its moq mock), then consulted in `pkg/cmd/auth/login` `loginRun` so `CopyToClipboard` is true when the setting is enabled or the flag is passed. The existing tests cover the config and login plumbing but not the actual clipboard write in `internal/authflow`, so the copy path needs a throwaway check." + }, + { + "task": "auth-clipboard-config-default", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:20:01.283220Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 7087, + "cache_read_tokens": 173268, + "cache_creation_tokens": 52671, + "num_turns": 7, + "latency_seconds": 52.7, + "total_cost_usd": 0.013047820500000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 2, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task fixes the key name and form (`gh config set clipboard enabled`) and the flag behaviour. The rest follows from repo conventions: internal/config/config.go `Options` (AllowedValues enabled/disabled, DefaultValue, PerHostOnly), and the existing `--clipboard` flag in pkg/cmd/auth/login/login.go:154. Scope questions (per-host or global, whether auth refresh honours it) are settled by sibling keys like spinner/prompt, which are global and non-per-host.", + "silent_invariant": "A wrong gate would show only on the config-on, flag-absent path. The default must stay disabled: defaultConfigStr in internal/config/config.go:611-642 and Options in config.go:653 both need the key. A wrong default would copy codes silently for everyone, but `gh config list` makes it visible. No allow-list, money, or stored-data invariant is involved (SECURITY.md covers only vulnerability reporting).", + "coordinated_surfaces": "Adding a key needs several consistent changes. (a) gh.Config accessor in internal/gh/gh.go:32-61, with the moq mock regenerated at internal/gh/mock/config.go (`//go:generate moq` on line 31). (b) Mock wiring in internal/config/test.go:34-100, where each *Func is assigned and an unassigned moq method panics. (c) cfg implementation plus Options entry, which also drives `config set` validation (pkg/cmd/config/set/set.go:96-130), `config list` (pkg/cmd/config/list/list.go:58), and generated docs (config.go:20). (d) defaultConfigStr. (e) The login read in pkg/cmd/auth/login/login.go loginRun (line 243). Missing any one leaves the feature half-built.", + "ambiguity": "The intended design is clear from repo patterns (global enabled/disabled key in Options). A tempting narrow fix is to read a key in login.go without registering it in Options/defaultConfigStr, which would not appear in config list or config set validation. The unanswered question is whether auth refresh --clipboard should also honour the setting; the task only names `gh auth login`.", + "verification": "pkg/cmd/auth/login/login_test.go covers flag parsing only (lines 133-149 for --clipboard). pkg/cmd/config/list/list_test.go:96-110 pins the full `config list` output and will need updating. internal/config/config_test.go covers GetOrDefault generically. The OS clipboard write sits in internal/authflow/flow.go:49-50 via atotto/clipboard with no injection seam, so the copy itself is not unit-testable. The new gating decision in loginRun can be tested with a config mock.", + "blast_radius": "This is a user-facing CLI setting in a shared tool, and a revert fixes it. The copied value is a one-time device code that stays on the OS clipboard once enabled, a minor exposure but not a credential leak, a data-integrity issue, or an authorisation hole. Nothing in .github/SECURITY.md or AGENTS.md marks this path as security-critical." + }, + "confidence": "medium", + "summary": "The task needs a new global enabled/disabled config key registered through the existing config Options, default and gh.Config accessor (with regenerated mock and test wiring), and read in `gh auth login` so the device code is copied without `--clipboard` while the flag still works. The main risks are the coordinated plumbing that a partial change would leave broken, and an unresolved scope question about whether `auth refresh` should honour the setting." + }, + { + "task": "auth-refresh-hint-on-expired-token", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:12:09.151106Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 8672, + "cache_read_tokens": 204255, + "cache_creation_tokens": 44773, + "num_turns": 8, + "latency_seconds": 59.4, + "total_cost_usd": 0.0131746725 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task defines the rule only as 'a token that gh auth login created', which needs a definition. The repo supplies one: pkg/cmd/agent-task/agent_task.go requireOAuthToken treats source \"oauth_token\" or \"keyring\" plus a \"gho_\" prefix as minted by the device flow. internal/config/config.go AuthConfig.ActiveToken returns those sources. The fallback 401 text is in internal/ghcmd/cmd.go:229-230.", + "silent_invariant": "internal/ghcmd/cmd.go:229 decides the hint. A wrong choice produces a wrong hint, not an error. Edge cases can misdirect silently: env-var tokens (source ends in _TOKEN, see authTokenWriteable in pkg/cmd/auth/status/status.go:412), a 401 from a host other than the default host, and `gh auth login --with-token` storing a PAT under source oauth_token. None of this is security-critical.", + "coordinated_surfaces": "One component: the hint in internal/ghcmd/cmd.go:229-230. It must agree with the token-source and prefix convention in pkg/cmd/agent-task/agent_task.go:91-93 and with request-host resolution from api.HTTPError.RequestURL (go-gh errors.go). No docs or other services reference the 'Try authenticating with' text (grep finds only cmd.go).", + "ambiguity": "'The token in use' is explicit in the task. The naive version, which checks the default host's ActiveToken instead of the failing request's host, looks complete but is wrong for multi-host setups. The repo's agent-task precedent settles the source-plus-prefix definition. Refresh is documented in pkg/cmd/auth/refresh/refresh.go as 'for active account', so the host choice matters.", + "verification": "internal/ghcmd/cmd_test.go covers only printError and newIOStreams helpers. Nothing exercises the 401 branch in Main() (cmd.go:229), and Main builds its own factory via factory.New(cfgFunc, ...). Testing requires extracting a helper. The seams exist (AuthConfig.SetActiveToken in internal/config/config.go:291, httpmock), but no current test would catch a wrong hint. Acceptance tests need real GitHub credentials (acceptance/README.md).", + "blast_radius": "User-facing CLI error text only (internal/ghcmd/cmd.go:230). No security, money or stored-data path. A wrong hint is visible and fixed by reverting the commit. Users see the message on every 401 for login-created tokens." + }, + "confidence": "medium", + "summary": "The change is a conditional in internal/ghcmd/cmd.go's 401 branch that must resolve the token for the failing request's host, check its source and gho_ prefix the way agent_task.go does, and print gh auth refresh instead of gh auth login. Existing tests do not reach that branch, so the work also means extracting a testable helper and writing login-created and non-login test cases with AuthConfig.SetActiveToken." + }, + { + "task": "auth-refresh-hint-on-expired-token", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:16:02.285436Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 6702, + "cache_read_tokens": 188776, + "cache_creation_tokens": 38563, + "num_turns": 8, + "latency_seconds": 49.4, + "total_cost_usd": 0.011067028500000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says 'a token that gh auth login created' without defining it. The repo defines it: `pkg/cmd/auth/shared/writeable.go` `AuthTokenWriteable` treats any source not ending in `_TOKEN` as login-created, and `pkg/cmd/auth/status/status.go:97` and `pkg/cmd/auth/refresh/refresh.go:169` already rely on that rule. Env tokens (GH_TOKEN, GITHUB_TOKEN, GH_ENTERPRISE_TOKEN, GITHUB_ENTERPRISE_TOKEN) are the excluded case. The convention is discoverable in code, so it is 1, not 2.", + "silent_invariant": "`internal/ghcmd/cmd.go:229-230` prints the hint for any 401. The change only alters message text, so a wrong classification is visible as a misleading hint. It is not a security or data invariant. A wrong result is most likely for env-token users, where `gh auth refresh` refuses to run (`refresh.go:169-172`). The host also has to be right: the hint must classify the token for the host in `httpErr.RequestURL` (`api/client.go:169-176`), not the active host, and that mismatch only shows up with multiple hosts.", + "coordinated_surfaces": "The change lands in `internal/ghcmd/cmd.go` (the 401 branch at lines 228-230). It needs the factory's `Config().Authentication()` (`gh.AuthConfig.ActiveToken`, `internal/gh/gh.go:111`) and the classifier from `pkg/cmd/auth/shared/writeable.go`. Nothing else must agree: no API, schema or docs surface is named. Docs such as `docs/multiple-accounts.md` may need a line, but that is optional.", + "ambiguity": "Changing the literal string to `gh auth refresh` is the tempting fix, and it fails the requirement that other tokens keep today's message. Getting the conditional right needs the token source for the request's host (`internal/config/config.go:237-261` `ActiveToken`, `ActiveToken` falls back to keyring/oauth_token), a choice on whether `--with-token` counts as login-created (it writes to the same hosts.yml/keyring path), and an env-token exclusion. A narrow edit that always prints the refresh hint would look complete and pass a quick review.", + "verification": "`internal/ghcmd/cmd_test.go` has `Test_printError` and no test of the 401 branch in `Main`. `config.AuthConfig.SetActiveToken` (`internal/config/config.go:291`) lets a test inject a token source, and `pkg/cmd/auth/status/status_test.go` and `refresh_test.go` cover the writeable/env-token rule. The branch can be tested only after the decision is extracted into a helper, so existing coverage is partial.", + "blast_radius": "The change affects the stderr text of every 401 across all gh commands, which users see directly. It touches no credentials, stored data or authorization path, and a revert of the message fixes it fully. It is not security-sensitive in the rubric sense." + }, + "confidence": "medium", + "summary": "The change is a conditional in the 401 branch of `internal/ghcmd/cmd.go`: classify the token for the failing request's host using the existing writeable (non-`_TOKEN`) rule, then print the `gh auth refresh` hint or keep `gh auth login`. The hard part is the classification and its host lookup, not the string, and the 401 branch currently has no test, so a helper must be extracted to test both token cases." + }, + { + "task": "auth-refresh-hint-on-expired-token", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:19:37.959176Z", + "metrics": { + "input_tokens": 20, + "output_tokens": 9272, + "cache_read_tokens": 217495, + "cache_creation_tokens": 40108, + "num_turns": 9, + "latency_seconds": 54.0, + "total_cost_usd": 0.013009095 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says 'a token that gh auth login created' without defining it. The mapping is in code: internal/config/config.go ActiveToken returns source 'keyring' or 'oauth_token' for login-stored tokens (Login() at config.go:356 writes to keyring or the hosts.yml user oauth_token), and env sources such as GH_TOKEN/GITHUB_TOKEN come from ghauth.TokenFromEnvOrConfig. pkg/cmd/auth/shared/writeable.go treats sources ending in '_TOKEN' as non-writeable, and pkg/cmd/auth/refresh/refresh.go:150 refuses those, so the suggested refresh must match that boundary. Reading the surrounding code reveals it, but the task does not state it.", + "silent_invariant": "The change only alters an advice string on stderr (internal/ghcmd/cmd.go:229-230). A wrong classification shows up as wrong advice, not leaked or corrupted data. Wrong results appear only in alternate configurations: env token overriding a keyring token, a request to a non-default host, or an explicit Authorization header. api/http_client.go AddAuthTokenHeader skips setting the header when one is already present, so the token in use can differ from cfg.ActiveToken(). None of these is a security or data invariant.", + "coordinated_surfaces": "The change is confined to internal/ghcmd/cmd.go (the 401 branch at lines 228-230), which needs the failing host from api.HTTPError.RequestURL (api/client.go handleResponse at lines 169-177) and the token source from cfg.Authentication().ActiveToken(host). That is one component plus its tests. No API, schema, or generated client changes. Docs do not mention the 401 text; grep finds it only at cmd.go:230.", + "ambiguity": "The tempting fix is to check the default host's ActiveToken source, or to always print refresh. The task rules out the second ('other tokens keep today's message'). The first misses the request host: api.HTTPError carries RequestURL, and ActiveToken is per-host (config.go:237). It also misses the 'token in use' wording: AddAuthTokenHeader (api/http_client.go:111-121) lets an explicit Authorization header override the configured token, so the 401 token may not be the active one. A check that passes a quick review can still classify the wrong token. Also, 'created by gh auth login' does not map cleanly to sources: --with-token and insecure storage go through the same Login() path.", + "verification": "internal/ghcmd/cmd_test.go covers printError, newIOStreams, and mightBeGHESUser only; nothing exercises the 401 branch in Main(). Main() builds its own config via config.NewConfig() and factory.New (cmd.go:56-127), so it cannot be driven with injected tokens without refactoring. Token-source behaviour is covered in internal/config/auth_config_test.go (TestTokenStoredInEnv, TestTokenStoredInKeyring, TestTokenStoredInConfig), but no test links that to the 401 message. A wrong hint would ship unnoticed.", + "blast_radius": "The change is a stderr hint on the 401 path that a user sees in normal use (cmd.go:229-230). It touches auth-adjacent guidance, but it neither changes credentials nor writes stored tokens, and gh auth refresh refuses env-sourced tokens (refresh.go:150). A revert of the string or branch fixes it fully." + }, + "confidence": "medium", + "summary": "The fix belongs in the 401 branch of internal/ghcmd/cmd.go: classify the token used for the failing host (from HTTPError.RequestURL) as login-created when ActiveToken's source is keyring or oauth_token, and suggest gh auth refresh only then. It needs tests for that branch, which currently has no coverage and cannot be injected without refactoring Main. The scores rest on code reading, not on running gh, so confidence is medium." + }, + { + "task": "codespace-ports-forward-concurrent-map", + "attempt": 1, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:11:28.604610Z", + "metrics": { + "input_tokens": 22, + "output_tokens": 8146, + "cache_read_tokens": 240329, + "cache_creation_tokens": 37209, + "num_turns": 10, + "latency_seconds": 54.0, + "total_cost_usd": 0.012242576500000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the required outcome (multi-port forwarding works every time, no race, single-port behaviour unchanged). The mechanism lives in code the repo depends on but does not contain. The race is in dev-tunnels v0.1.19 (go.mod:45): `Manager.CreateTunnelPort` (module cache, manager.go:443-450) writes `options.AdditionalHeaders[\"If-Not-Match\"] = \"*\"` into the caller's `*TunnelRequestOptions`. That pointer is `fwd.connection.Options`, shared by every goroutine via the value copy in `NewPortForwarder` (port_forwarder.go:55-59; `CodespaceConnection.Options` is set at connection.go:64-78). The repo's own comments (port_forwarder.go:111-118) document dev-tunnels protocol behaviour that the fix must preserve. The fix is therefore constrained by the SDK's option-ownership semantics, which the task does not mention, but those are readable from the code, so 1 rather than 2.", + "silent_invariant": "Shared mutable state across goroutines: `pkg/cmd/codespace/ports.go:328-352` runs `ForwardPort`/`ConnectToForwardedPort` per port on one `CodespaceConnection`. Shared pointers are `TunnelManager`, `TunnelClient` (`connection.go:20-24`, mutex only around `Connect` at connection.go:85-104), `Options` and `Tunnel`. A wrong fix (for example a mutex around only the map write, or a clone that misses `AdditionalHeaders`) still passes single-port use and fails only under interleaving. The crash is intermittent, which makes a silent race easy to ship.", + "coordinated_surfaces": "Two components must agree: the command layer `ForwardPorts` in pkg/cmd/codespace/ports.go:312-353 (errgroup fan-out, per-goroutine `NewPortForwarder`), and the forwarder/connection layer in internal/codespaces/portforwarder/port_forwarder.go (`ForwardPort` lines 104-182, `ForwardPortToListener` 63-101) and internal/codespaces/connection/connection.go (shared `Options`/`TunnelClient`). The CLI surface (`gh cs ports forward :...`, ports.go:301-310) and docs do not change. Other callers of `ForwardPort` must remain correct, but they are not part of the surface.", + "ambiguity": "The task lists three symptoms: `fatal error: concurrent map writes`, `create tunnel port failed: ... 400: Bad Request`, and `ssh: rejected: connect failed (Connection refused)`. Only the first is explained by the shared `Options.AdditionalHeaders` write (dev-tunnels manager.go:450). The 400 and connection-refused errors come from concurrent `CreateTunnelPort`/`GetTunnelPort`, `connection.Connect`, and `RefreshPorts` on one tunnel client (port_forwarder.go:121, 164, 170, 176). A narrow fix (clone or lock the options map) would remove the crash and look complete in review while the other two failures remain. The right design is a decision among per-forwarder options, serialized control-plane calls, or a single connect before fan-out.", + "verification": "Mock-based tests exist and run without external services: `NewMockHttpClient` (internal/codespaces/connection/tunnels_api_server_mock.go:43) backs `TestForwardPortDefaultsToHTTPProtocol` and `TestForwardPortRespectsProtocolOfExistingTunneledPorts` (internal/codespaces/portforwarder/port_forwarder_test.go:141, 201). `pkg/cmd/codespace/ports_test.go` only covers the pending-operation path for `ForwardPorts` (line 117) and has no multi-port test. No existing test runs concurrent forwards, so the required test must be new. Running it under `-race` is the reliable check. Since the harness exists, the gap is closable, so 1 rather than 2.", + "blast_radius": "User-facing CLI command `gh codespace ports forward` (pkg/cmd/codespace/ports.go:299-310). Failures are visible (crash or error) and reverting restores prior behaviour. Tunnel access tokens sit in the shared `TunnelRequestOptions`/`Tunnel` objects (connection.go:56-64), so a careless change could mishandle credentials, but the change itself does not touch auth logic. No persisted data or money is involved." + }, + "confidence": "medium", + "summary": "The task needs per-port forwarding in `ForwardPorts` (pkg/cmd/codespace/ports.go:312-353) to stop sharing mutable dev-tunnels state. That state is the `TunnelRequestOptions` pointer whose `AdditionalHeaders` map `CreateTunnelPort` writes, plus the tunnel client's connect and refresh calls, and a fix must also address the 400 and connection-refused errors, not only the map race. Verification needs a new concurrent multi-port test built on the existing mock tunnels API, run under `-race`." + }, + { + "task": "codespace-ports-forward-concurrent-map", + "attempt": 2, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:15:23.799035Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 7266, + "cache_read_tokens": 186155, + "cache_creation_tokens": 42071, + "num_turns": 7, + "latency_seconds": 46.2, + "total_cost_usd": 0.011830527500000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour: multi-port forward works with no race, single port unchanged, and a multi-port test. The concurrency contract is not stated. The correct fix depends on which dev-tunnels objects are shared and safe, which is in the SDK (github.com/microsoft/dev-tunnels@v0.1.19/go/tunnels/manager.go, CreateTunnelPort at ~443-470) and on the existing lock pattern in internal/codespaces/connection/connection.go (TunnelClient.mu around Connect/Close). The nearby code reveals it, so 1, not 2.", + "silent_invariant": "Shared state is mutated without synchronisation: CodespaceConnection.Options and .Tunnel are pointers copied by value in portforwarder.NewPortForwarder (port_forwarder.go), so every errgroup goroutine in ForwardPorts (pkg/cmd/codespace/ports.go) shares them. SDK CreateTunnelPort writes options.AdditionalHeaders[\"If-Not-Match\"] and reassigns tunnel.Ports. Connect is locked but ForwardPort's GetTunnelPort/CreateTunnelPort/RefreshPorts are not. A shallow copy of the options struct still shares the AdditionalHeaders map, so a wrong fix can look right and still race intermittently.", + "coordinated_surfaces": "Changes land in pkg/cmd/codespace/ports.go (ForwardPorts) and, if the fix is in the shared state, internal/codespaces/connection/connection.go or internal/codespaces/portforwarder/port_forwarder.go. Those layers are one subsystem. No docs or API surface change (docs/codespaces.md covers the feature but the task does not change its behaviour). Other callers of the same connection (ListPorts, UpdatePortVisibility, ssh, jupyter) must keep working, which is within the same component.", + "ambiguity": "The task names the map write, but the map is one symptom. The 400 Bad Request (concurrent GetTunnelPort/CreateTunnelPort with If-Not-Match) and 'connection refused' (concurrent Connect/RefreshPorts on the shared TunnelClient) are not explained by a map fix. A narrow fix such as a mutex around only the CreateTunnelPort call, or copying the Options pointer, would clear the reported panic and still leave the other two failures. Serialising ForwardPort versus giving each goroutine its own connection are two different designs with different costs.", + "verification": "pkg/cmd/codespace/ports_test.go covers ForwardPorts only for the pending-operation error path, and no test forwards any port. internal/codespaces/portforwarder/port_forwarder_test.go covers single-port ForwardPort against connection.NewMockHttpClient. The mock (connection/tunnels_api_server_mock.go) makes a multi-port test possible, and .github/workflows/go.yml runs go test -race, but the concurrent path has no meaningful coverage today. The task requires a new test.", + "blast_radius": "gh cs ports forward is user-facing CLI (docs/codespaces.md). A wrong change crashes or breaks port forwarding, but a revert fixes it with no stored data or auth exposure. No secrets handling or persisted state is changed; the tokens in api/codespace are only read." + }, + "confidence": "medium", + "summary": "The fix must make ForwardPorts safe for concurrent use across the shared CodespaceConnection, covering its shared Options map, Tunnel.Ports slice and Connect/RefreshPorts ordering, not only the map write named in the report. The repo has no test that forwards more than one port, so the change needs a new multi-port test built on the existing connection mock and run under go test -race." + }, + { + "task": "codespace-ports-forward-concurrent-map", + "attempt": 3, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:19:02.276548Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 6454, + "cache_read_tokens": 146362, + "cache_creation_tokens": 33466, + "num_turns": 7, + "latency_seconds": 41.2, + "total_cost_usd": 0.009763017 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour (multi-port forward works, no race, single port unchanged, add a multi-port test). The mechanism lives in the pinned dependency: go.mod pins github.com/microsoft/dev-tunnels v0.1.19, and its manager.go:443-452 CreateTunnelPort writes options.AdditionalHeaders[\"If-Not-Match\"] = \"*\" into the caller's *TunnelRequestOptions. The repo's own pattern for shared connection state is the TunnelClient mutex in internal/codespaces/connection/connection.go (Connect/Close). The fix shape follows from those, but it depends on library internals the task does not mention.", + "silent_invariant": "pkg/cmd/codespace/ports.go:331-350 runs one goroutine per port, each calling portforwarder.NewPortForwarder (port_forwarder.go:55-60), which copies CodespaceConnection by value but keeps the shared pointers TunnelManager, Options, Tunnel and TunnelClient. ForwardPort (port_forwarder.go:164) then calls CreateTunnelPort, which mutates the shared Options.AdditionalHeaders map. A race there is intermittent, and a wrong header or stale state can produce the 400 or the connection-refused error without a visible crash, so a partial fix can look like it works.", + "coordinated_surfaces": "The change stays inside one component: ForwardPorts in pkg/cmd/codespace/ports.go, with possibly internal/codespaces/connection/connection.go (shared Options and Tunnel) and internal/codespaces/portforwarder/port_forwarder.go. ListPorts and UpdatePortVisibility (port_forwarder.go:245-316) use the same shared Options and must agree with whichever sharing model is chosen. No API, docs or deployment surface is involved.", + "ambiguity": "Several tempting narrow fixes exist. Go is 1.26 (go.mod), so the loop variable in ports.go:331 is already per-iteration and `pair := pair` changes nothing. Wrapping ForwardPort in a mutex serializes the calls, but it does not stop other callers of the shared Options (ListPorts, UpdatePortVisibility) and may leave the 400/connection-refused symptoms in place. Copying Options per forwarder versus locking are both plausible designs. The task describes symptoms and does not name the shared-state cause.", + "verification": "ports_test.go covers only the pending-operation error path of ForwardPorts (lines 117-121). No existing test forwards several ports at once. The harness exists and is testable offline: connection.NewMockHttpClient and WithSpecificPorts in internal/codespaces/connection/tunnels_api_server_mock.go are used by port_forwarder_test.go:12-40, so a multi-port test and a -race run are possible without credentials. Current coverage would miss this bug entirely.", + "blast_radius": "This is user-facing CLI behaviour (`gh cs ports forward`) that touches tunnel connection state, but no stored data, money or authorisation logic changes. AGENTS.md lists no security-critical areas for this path, and there is no SECURITY.md. A bad change is visible and revertible by reverting the commit." + }, + "confidence": "medium", + "summary": "The fix belongs in the codespace port-forwarding path: ForwardPorts runs one goroutine per port, and each goroutine's CreateTunnelPort writes to the TunnelRequestOptions map that the forwarders share through connection.CodespaceConnection, so the change must make per-port tunnel operations safe, either by isolating options per forwarder or by serializing shared-state access as connection.go already does for Connect. A test that forwards several ports at once has to be added, and the offline tunnel mock already supports it, but nothing in the current suite exercises this path." + }, + { + "task": "copilot-not-installed-trailing-newline", + "attempt": 1, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:10:13.265877Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 4861, + "cache_read_tokens": 110582, + "cache_creation_tokens": 27628, + "num_turns": 6, + "latency_seconds": 30.7, + "total_cost_usd": 0.007690342000000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/copilot/copilot.go:155 is `fmt.Fprintf(opts.IO.ErrOut, \"%s Copilot CLI was not installed\", ...)` with no newline. The task fixes the missing newline and says the rest of the declined-install behaviour stays the same. The newline convention is visible in sibling output: copilot.go:141 uses Fprintln and copilot.go:337 ends with \\n. The test's exact expected string depends on the WarningIcon from the test ColorScheme, which the task does not state.", + "silent_invariant": "The change is one format string on an stderr message in runCopilot's decline branch (copilot.go:149-157). It touches no auth, secrets, persisted data, concurrency or retries. A wrong change is visible on screen.", + "coordinated_surfaces": "Only pkg/cmd/copilot/copilot.go and its test file change. The help text (copilot.go:57-58) and the other copilot paths do not mention this message. No API, config schema, or docs reference the string.", + "ambiguity": "The obvious fix is adding \\n at copilot.go:155. The non-interactive branch at copilot.go:159 (`%s Copilot CLI not installed`) has the same missing newline but is not the declined-install path. The task scopes to the declined case, so fixing 159 too is a judgment call. The repository does not settle it.", + "verification": "No existing test covers the decline branch. TestRunCopilot_execFailureHint (copilot_test.go:594) stubs findCopilotBinaryFunc and runExternalCmdFunc, and the Prompter is mockable via internal/prompter/prompter_mock.go with ConfirmFunc (used in pkg/cmd/codespace/delete_test.go:299). iostreams.CanPrompt (pkg/iostreams/iostreams.go:275-280) needs stdin and stdout TTY set. The harness supports the test, so this is not untestable, but the branch is currently uncovered.", + "blast_radius": "gh is a shipped user-facing CLI, so the wrong text would be seen by users. The failure is cosmetic and a revert fixes it. No stored data, credentials or money are involved." + }, + "confidence": "high", + "summary": "The fix is a one-line newline change in the declined-install branch of runCopilot (pkg/cmd/copilot/copilot.go:155), plus a new test that uses the existing findCopilotBinaryFunc/runExternalCmdFunc stubs and the Prompter mock to assert the exact stderr output. The main judgement is whether to also fix the same missing newline in the non-interactive branch at copilot.go:159, which the task does not ask for." + }, + { + "task": "copilot-not-installed-trailing-newline", + "attempt": 2, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:13:50.540545Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3676, + "cache_read_tokens": 103167, + "cache_creation_tokens": 24481, + "num_turns": 6, + "latency_seconds": 24.0, + "total_cost_usd": 0.0065243145 + }, + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected output: the warning must end with a newline and the rest of the declined-install behaviour stays the same. The current code is in pkg/cmd/copilot/copilot.go:155, `fmt.Fprintf(opts.IO.ErrOut, \"%s Copilot CLI was not installed\", ...)` with no `\\n`, and the exact expected string follows from the task text plus the ColorScheme WarningIcon already used there.", + "silent_invariant": "The change is a cosmetic stderr write on the declined-prompt path in pkg/cmd/copilot/copilot.go:150-156. No auth, data, concurrency or allow-list logic is involved, and a wrong change shows up immediately in the terminal.", + "coordinated_surfaces": "One format string in runCopilot (pkg/cmd/copilot/copilot.go:155) plus a test in pkg/cmd/copilot/copilot_test.go. No docs, generated clients, schema or other component reference this text.", + "ambiguity": "The obvious fix (add a newline on the declined branch) matches the task. Some interpretation is needed because the sibling non-prompt branch at copilot.go:158, `\"%s Copilot CLI not installed\"`, has the same missing newline. The task scopes the bug to the declined path only, and the repository does not settle whether the sibling should change too.", + "verification": "pkg/cmd/copilot/copilot_test.go covers only flag parsing, removal, tar/zip extraction, checksums, download and the exec-failure hint (TestRunCopilot_execFailureHint at line 594). Nothing exercises the declined-Confirm branch, so a regression would ship unnoticed. The harness is available to write one cheaply: iostreams.Test() with SetStdinTTY/SetStdoutTTY, prompter.NewMockPrompter (used in pkg/cmd/gist/delete/delete_test.go:258), and the findCopilotBinaryFunc override.", + "blast_radius": "This is end-user CLI output that users see. Reverting a one-line format change fixes it with no lasting harm, and no security or stored data is involved." + }, + "confidence": "high", + "summary": "The fix is one newline in the declined-install branch of runCopilot in pkg/cmd/copilot/copilot.go, and the task fully specifies the expected output. The real work is a new test in copilot_test.go, because nothing currently covers that branch, and the repo's prompter mock and iostreams.Test() make it straightforward to write; the sibling non-prompt warning at line 158 shares the same missing newline and needs an explicit scope decision." + }, + { + "task": "copilot-not-installed-trailing-newline", + "attempt": 3, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:17:45.948238Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 4248, + "cache_read_tokens": 132822, + "cache_creation_tokens": 27071, + "num_turns": 7, + "latency_seconds": 27.2, + "total_cost_usd": 0.007521464499999999 + }, + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task text states the exact expected behaviour: the warning must end with a newline and the rest of the declined-install flow stays the same. The code at pkg/cmd/copilot/copilot.go:155 (`fmt.Fprintf(opts.IO.ErrOut, \"%s Copilot CLI was not installed\", ...)`) and its `cmdutil.SilentError` return at :156 are the behaviour to keep. Nothing outside the task text is needed.", + "silent_invariant": "Missing newline is directly visible in the terminal. No security, data or concurrency invariant is involved; copilot.go is a CLI output path with no auth or persistence on this branch.", + "coordinated_surfaces": "One format string in runCopilot (copilot.go:155) plus one test in pkg/cmd/copilot/copilot_test.go. No docs, generated clients or other components reference this message (grep for 'was not installed' finds only copilot.go:155).", + "ambiguity": "The sibling message at copilot.go:159 (`\"%s Copilot CLI not installed\"`, non-prompt, non-CI branch) has the same missing newline but is not the declined-install path the task scopes to. Fixing only :155 matches the task; a reviewer could reasonably ask about :159. Repo does not settle that without a judgement call.", + "verification": "copilot_test.go has no test for the decline path. Existing tests (TestRunCopilot_execFailureHint at :594, TestCopilotCommandIsSampledAt100 at :620) do not reach copilot.go:146-157. The path is testable with existing helpers (findCopilotBinaryFunc stub, internal/prompter/prompter_mock.go ConfirmFunc, iostreams.Test with stdin/stdout TTY set), so the gap is missing coverage, not a missing harness, and a wrong output would ship unnoticed.", + "blast_radius": "gh is a user-facing CLI (AGENTS.md; pkg/cmd/root/root.go:156 registers the command). A wrong message would be seen by users, but it is cosmetic and a revert of the one-line change fixes it with no stored data or security impact." + }, + "confidence": "high", + "summary": "The change is a one-line newline fix to the declined-install warning in runCopilot (pkg/cmd/copilot/copilot.go:155) plus a new test asserting the exact ErrOut content and SilentError return, using the existing prompter mock and TTY-simulated iostreams. The main judgement is whether to also add a newline to the non-prompt warning at copilot.go:159, which the task does not name." + }, + { + "task": "extension-uninstall-alias", + "attempt": 1, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:10:38.970308Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3107, + "cache_read_tokens": 53853, + "cache_creation_tokens": 20237, + "num_turns": 4, + "latency_seconds": 19.8, + "total_cost_usd": 0.0050849204999999995 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the behaviour (`uninstall` does exactly what `remove` does, `remove` unchanged, help lists `uninstall`). The implementation shape follows repo convention: cobra `Aliases` is already used in pkg/cmd/extension/command.go:59 (`extensions`, `ext`) and in many sibling commands (`ls` aliases). Help rendering of an alias is cobra's behaviour, not spelled out by the task.", + "silent_invariant": "The change is a command name. pkg/cmd/extension/command.go:461-476 `remove` delegates to m.Remove, which does Lstat, cleanExtensionUpdateDir, then os.RemoveAll (manager.go:573-586). A wrong alias is visible as 'unknown command' or a failing run, not a silent misbehaviour.", + "coordinated_surfaces": "Two places in one component must agree: the command definition in pkg/cmd/extension/command.go and its table-driven tests in command_test.go (remove cases at lines 553-610). No docs/ page covers `extension remove`, and acceptance/testdata/extension/extension.txtar only exercises `remove` as-is.", + "ambiguity": "Two readings: a cobra alias on `remove`, or a duplicate `uninstall` command with copied RunE. The task's 'does exactly what remove does' favours the alias, which is the repo's pattern. A copied command would also look complete but could drift from `remove`. 'Help lists uninstall' is slightly ambiguous: cobra shows aliases on the `remove --help` page, not in the parent `extension` usage list.", + "verification": "pkg/cmd/extension/command_test.go has table-driven tests for `remove` (tty, nontty, gh-prefix, full name at lines 553-610) that assert RemoveCalls and stdout. The same harness can cover `uninstall` directly, so the existing suite would catch a wrong change.", + "blast_radius": "This is a public, user-facing CLI command (AGENTS.md describes gh as a user CLI). Its effect deletes an installed extension directory, but the same path already exists under `remove`. A revert of the alias fixes any problem with no lasting data harm." + }, + "confidence": "high", + "summary": "The task needs a cobra alias (or equivalent) on `gh extension remove` so `uninstall` resolves to the same handler, with help output that shows it, following the repo's existing `Aliases` convention in pkg/cmd/extension/command.go. The existing table tests in command_test.go for `remove` cover the handler and can be extended to `uninstall`, and a wrong change would surface as an unknown-command error." + }, + { + "task": "extension-uninstall-alias", + "attempt": 2, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:14:20.069678Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 5661, + "cache_read_tokens": 130744, + "cache_creation_tokens": 28141, + "num_turns": 7, + "latency_seconds": 33.8, + "total_cost_usd": 0.0084228815 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the behaviour: `uninstall` must do exactly what `remove` does, and `remove` must keep working. The mechanism is not stated, but the repo already uses a `remove` alias on a sibling command: `pkg/cmd/variable/delete/delete.go:61-63` and `pkg/cmd/secret/delete/delete.go:77-79` both declare `Aliases: []string{\"remove\"}`. Following that convention is what the task requires.", + "silent_invariant": "`pkg/cmd/extension/command.go:460-475` (`remove`) calls `m.Remove(extName)`, which does `os.RemoveAll` under the install dir (`manager.go:573-585`). Adding an alias does not change that logic. If the alias were missing, cobra would return a visible 'unknown command' error, so a wrong change is not silent.", + "coordinated_surfaces": "The change lives in one place: the `remove` `cobra.Command` literal in `pkg/cmd/extension/command.go:460-475`. Grep found no markdown or docs mentioning `extension remove` (only `command.go:500`, which is help prose and does not need to change). `acceptance/testdata/extension/extension.txtar:56` uses `remove` and still works. The browse TUI's `\"remove\"` verb (`browse/browse.go`) is internal and unaffected.", + "ambiguity": "An alias satisfies 'does exactly what remove does'. Cobra's ALIASES section (`pkg/cmd/root/help.go:132-133`) will show `uninstall` in `gh extension remove --help`. Top-level `gh extension --help` lists only command names, so `uninstall` will not appear there unless it is a separate command. The task's 'appear in the CLI help' is most naturally read as the ALIASES entry, which the repo convention supports, but a reviewer could expect the top-level listing. The pattern is clear; the help placement is the open point.", + "verification": "`pkg/cmd/extension/command_test.go:552-615` has table-driven `remove` cases using `ExtensionManagerMock` with RemoveFunc/RemoveCalls assertions, and these run without network access. No case exercises `uninstall`, so the alias wiring is untested today. The RunE path is covered, so adding an `uninstall` case to this table would fully close the gap. Tests are easy to write here, which is why this is not a 2.", + "blast_radius": "This is a public CLI command. A wrong alias would be visible to users, but a revert of a one-line `Aliases` addition restores the old state with no lasting harm. The command deletes local extension directories, but the change does not alter that behaviour." + }, + "confidence": "high", + "summary": "The task is a one-line alias addition to the `remove` subcommand in `pkg/cmd/extension/command.go`, matching the existing `Aliases: []string{\"remove\"}` convention in the variable and secret delete commands. The remaining work is a `command_test.go` table case for `uninstall` (the mock-based `remove` tests already cover the RunE path) and confirming that `uninstall` shows in the help output, which is the ALIASES section of `remove --help` rather than the top-level command list." + }, + { + "task": "extension-uninstall-alias", + "attempt": 3, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:17:57.765193Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 5483, + "cache_read_tokens": 136547, + "cache_creation_tokens": 29316, + "num_turns": 7, + "latency_seconds": 35.9, + "total_cost_usd": 0.008550377000000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task text fully states behaviour (uninstall does what remove does; remove unchanged; help lists uninstall). The mechanism follows from repo convention: pkg/cmd/extension/command.go:266 uses Aliases: []string{\"ls\"} for list, and help.go:132-133 renders an ALIASES section, so an alias is the house pattern rather than a duplicate command.", + "silent_invariant": "Change touches only a cobra command definition in pkg/cmd/extension/command.go. Reserved-name check at command.go:699-700 uses rootCmd.Find on root children only, so an extension-level alias does not alter it. No auth, secrets, persisted data or concurrency involved; a wrong wiring fails visibly (unknown command).", + "coordinated_surfaces": "Single component: pkg/cmd/extension/command.go. Help output is generated from cobra metadata (pkg/cmd/root/help.go:132). pkg/cmd/root/help_reference.go has no extension entries, and no checked-in docs list 'extension remove' (only acceptance/testdata/extension/extension.txtar:56 uses remove, which stays valid).", + "ambiguity": "Two readings: a cobra alias on remove (shows only under 'gh ext remove --help' ALIASES, not in the parent's subcommand listing) versus a sibling command that shares the run function (listed separately). The task's 'appear in the CLI help' does not say which. Repo convention favours the alias, but the listing location is a judgment call.", + "verification": "pkg/cmd/extension/command_test.go TestNewCmdExtension has five 'remove extension' table cases (lines ~553-613) using ExtensionManagerMock.RemoveFunc, so a new 'uninstall' case would exercise the alias. No existing test asserts extension help output (grep for ALIASES/help in that file finds nothing), so the help-listing requirement is untested unless added.", + "blast_radius": "Public user-facing CLI command (pkg/cmd/extension/command.go). Change is additive; a bad change is visible to users and fixed by reverting. No security, money or stored data touched, and remove's ExactArgs(1) behaviour is unchanged." + }, + "confidence": "medium", + "summary": "Add `uninstall` as a cobra alias on the existing `remove` command in pkg/cmd/extension/command.go, following the `ls` alias pattern, so both names run the same handler and remove is unchanged. Cover it with a new case in TestNewCmdExtension (the remove cases at command_test.go ~553-613 are the template), and decide whether the help requirement means the ALIASES section of `remove --help` or a separate listing." + }, + { + "task": "issue-view-comments-with-json", + "attempt": 1, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:11:33.878528Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5334, + "cache_read_tokens": 110682, + "cache_creation_tokens": 37158, + "num_turns": 5, + "latency_seconds": 32.1, + "total_cost_usd": 0.009261747 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The issue allows either of two behaviours: add `comments` to the JSON fields, or fail with a validation error. Everything else follows from code: `comments` is already in `api.IssueFields` (pinned by `TestJSONFields` in pkg/cmd/issue/view/view_test.go), and the JSON shape comes from the existing `api.Comments` struct. The pagination behaviour comes from `preloadIssueComments` in http.go.", + "silent_invariant": "In viewRun (view.go), when `opts.Exporter != nil` the lookup fields come only from `opts.Exporter.Fields()` and `opts.Comments` is never read, so the no-op is silent. A naive fix that only adds `comments` to `lookupFields` does not reach the output, because `jsonExporter.exportData` projects the struct using `e.fields` (json_flags.go), so comments would be fetched and then dropped. Otherwise the risk is in edge cases such as more than 100 comments, which only `preloadIssueComments` pagination handles.", + "coordinated_surfaces": "The change stays inside pkg/cmd/issue/view: the flag definition and help text in NewCmdView (view.go), the viewRun field-selection logic, and the tests in view_test.go. `pr view` (pkg/cmd/pr/view/view.go) has the same flag and Exporter pattern, but the issue does not ask for it. The rubric's Docs and generated-docs surfaces are not affected by a validation or field change.", + "ambiguity": "The task names two acceptable behaviours. The repo makes the intended one clear: the `--comments` help text, the JSON exporter contract, and the existing `TestIssueView_json_*` tests. The trap is the implementation: adding `comments` to lookup without also changing the exported field set looks complete but produces no comments in the output.", + "verification": "view_test.go has table tests for `--comments` in TTY and non-TTY mode (TestIssueView_tty_Comments, TestIssueView_nontty_Comments) and JSON field tests (TestIssueView_json_*), all offline via httpmock with fixtures such as issueView_previewFullComments.json. None of them cover `--json` combined with `--comments`, so a wrong change to that combination would not be caught.", + "blast_radius": "This is a user-facing CLI command used by scripts and agents. A mistake would change JSON output or make a currently successful invocation fail, which users would notice, and a revert fixes it. Nothing security-sensitive or data-persisting is touched (SECURITY.md covers only vulnerability reporting)." + }, + "confidence": "medium", + "summary": "The task needs `gh issue view --comments --json ...` to stop silently dropping comments, either by adding `comments` to the JSON export fields and lookup or by rejecting the combination with a validation error, while keeping `--json` alone and `--comments` alone unchanged. The work sits in pkg/cmd/issue/view (viewRun, the exporter field set, and view_test.go), and the existing offline httpmock tests have no case for the combined flags." + }, + { + "task": "issue-view-comments-with-json", + "attempt": 2, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:15:29.132869Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4529, + "cache_read_tokens": 90561, + "cache_creation_tokens": 28492, + "num_turns": 5, + "latency_seconds": 30.9, + "total_cost_usd": 0.007406091 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task offers two acceptable behaviours (add `comments` to the JSON fields, or make the flags mutually exclusive). Repo conventions settle the shape: `cmdutil.MutuallyExclusive` (pkg/cmdutil/errors.go:47) is used across codespace and attestation, and `AddJSONFlags`/`ExportData(fields)` define how JSON fields are selected (pkg/cmdutil/json_flags.go:259-284). No external spec is needed.", + "silent_invariant": "Read-only display path. In viewRun (pkg/cmd/issue/view/view.go:114-118, 152-156) a wrong change shows directly as a missing or extra JSON field, or a failed command. No auth, persistence or concurrency involved.", + "coordinated_surfaces": "Changes stay within pkg/cmd/issue/view/view.go: the lookupFields logic (lines 114-118), preloadIssueComments (152-156), and the exporter path (Exporter.Write at line 193). Possibly the --comments help text and acceptance/testdata/issue/*.txtar (which already use `issue view --comments`). Two or three places in one component.", + "ambiguity": "The obvious-looking fix `lookupFields.Add(\"comments\")` when opts.Comments is set would fetch comments but not emit them: the exporter's output is filtered by e.fields via Issue.ExportData(fields) (json_flags.go:283), and e.fields comes only from --json. The task's first option needs the comments field added to the exporter output too, not just the fetch. The mutual-exclusion option is simpler but changes behaviour for existing `--comments --json` callers. A quick review could miss the output half.", + "verification": "view_test.go covers this path: TestIssueView_nontty_Comments and TestIssueView_tty_Comments (lines 387-538) mock CommentsForIssue with fixtures, and TestIssueView_json_* (lines 922-1020) cover JSON output. TestJSONFields (line 28) checks `comments` is a supported field. The new JSON+--comments behaviour is easy to pin with the same httpmock pattern.", + "blast_radius": "User-facing public CLI (gh issue view, AGENTS.md architecture). It changes the output for existing `--comments --json` callers, so users would notice, but a revert restores prior behaviour. No security, money or stored data involved." + }, + "confidence": "high", + "summary": "The fix belongs in pkg/cmd/issue/view/view.go: when --comments and --json are both set, comments must be both fetched (lookupFields) and emitted (exporter fields), or the combination must fail via cmdutil.MutuallyExclusive. The main risk is a lookup-only change that fetches comments without outputting them, and the existing TestIssueView_*Comments and TestIssueView_json_* tests give a direct pattern for covering both paths." + }, + { + "task": "issue-view-comments-with-json", + "attempt": 3, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:19:10.816156Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3786, + "cache_read_tokens": 62042, + "cache_creation_tokens": 28396, + "num_turns": 4, + "latency_seconds": 23.9, + "total_cost_usd": 0.006670311999999999 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task names two acceptable behaviours (add `comments` to JSON fields, or make the flags mutually exclusive). Behaviour is pinned by existing code: `pkg/cmd/issue/view/view.go:108-117` builds `lookupFields` from `opts.Exporter.Fields()` only when `--json` is set, so `--comments` never reaches the `preloadIssueComments` gate at line 152. `TestJSONFields` (`view_test.go:28-57`) already lists `comments` as a valid JSON field, so the `--json comments` path is the established contract.", + "silent_invariant": "Comments are paginated via `preloadIssueComments` (`pkg/cmd/issue/view/http.go:11-36`, `first: 100` with cursor loop). A wrong fix that adds the field without preloading, or that skips the cursor loop, would emit partial or empty comments. This is data-correctness, not security, and it shows up in JSON output and in fixture-based tests (`view_test.go:397-414`, `CommentsForIssue` fixtures), so it is not hidden.", + "coordinated_surfaces": "Change is contained in `pkg/cmd/issue/view/view.go` (flag wiring at line 85, JSON/lookup logic at 108-118 and 181-192). Help text 'View issue comments' (line 85) may need to reflect the new behaviour. No API, schema or deployment surface. Acceptance testdata exists (`acceptance/testdata`) but is not required for issue view coverage.", + "ambiguity": "Task explicitly offers two acceptable designs, so the choice is open but bounded. Repo shows the same flaw in `pkg/cmd/pr/view/view.go:105-107` and `:133` (`--comments` ignored when `Exporter != nil`), which the task does not mention; a fix confined to issue view would be complete for the task but leave the sibling command inconsistent. The mutual-exclusion option has precedent in `pkg/cmd/discussion/view/view.go:155` (`cmdutil.MutuallyExclusive`).", + "verification": "`pkg/cmd/issue/view/view_test.go` already has httpmock-driven tests for `--comments` (lines 394-520) and JSON-field tests (`TestIssueView_json_*`, lines 922-1010). These run without external services, and an incorrect change (comments missing from JSON, or no error on conflict) would fail them or need a new case that is easy to write against the same mocks.", + "blast_radius": "User-facing CLI behaviour for `gh issue view` in a public tool. Output change only: no auth, secrets, or stored data are involved (`SECURITY.md` concerns do not apply to this path). A revert in a release fully undoes any mistake." + }, + "confidence": "high", + "summary": "The fix belongs in `pkg/cmd/issue/view/view.go`, where `--comments` is dropped on the `--json` path because `lookupFields` is built only from `Exporter.Fields()`, so either `comments` must be added and preloaded with the existing paginated `preloadIssueComments`, or the flags must be rejected with `cmdutil.MutuallyExclusive` as in discussion view. Tests belong in `view_test.go` alongside the existing `--comments` and `TestIssueView_json_*` cases using the `CommentsForIssue` fixtures. The sibling `pr view` has the same silent no-op, which the task does not mention." + }, + { + "task": "pr-checks-all-cancelled-summary", + "attempt": 1, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:10:13.159261Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3005, + "cache_read_tokens": 59868, + "cache_creation_tokens": 26137, + "num_turns": 4, + "latency_seconds": 22.1, + "total_cost_usd": 0.0059062355 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected text (\"Some checks were cancelled\") and that other mixes keep current wording. The rest of the output for the all-cancelled case (the tally line \"N cancelled, 0 failing, ... checks\") follows from the existing format in pkg/cmd/pr/checks/output.go:82-83 and the tests at checks_test.go:197-198. Nothing outside the repo is needed.", + "silent_invariant": "Display-only change in printSummary (pkg/cmd/pr/checks/output.go:69-92). The condition is `counts.Failed+counts.Passed+counts.Skipping+counts.Pending > 0` and omits `Canceled`, so a wrong change shows as a visible wrong or missing line. No security, data or concurrency invariant is involved.", + "coordinated_surfaces": "One function, printSummary, is called from checks.go:212 and checks.go:241 for TTY output. The JSON export path in aggregate.go and the non-TTY output in output.go:52 are not affected by the summary gate. Only one place needs to change.", + "ambiguity": "The bug is the gate at output.go:70 leaving out Canceled, which is visible from the code. The later branch `else if counts.Canceled > 0` (output.go:76) already gives the intended wording. Fixing the gate, rather than special-casing the all-cancelled path, is the clear design. The one open point is whether the tally line should also print, which the existing structure answers.", + "verification": "pkg/cmd/pr/checks/checks_test.go is a table-driven test that asserts full TTY output for each state mix: 'some cancelled tty' (line 188), 'some pending tty', 'all passing tty', 'some skipped tty', and others. Those would catch a wrong change to the gate. An all-cancelled fixture is a small addition following fixtures/someCancelled.json. Everything runs offline with httpmock.", + "blast_radius": "This is user-facing CLI output in `gh pr checks` (README.md and AGENTS.md describe gh as a user-facing tool). A mistake is visible, but it is a text line with no data, security or persistence impact, and a revert fixes it fully." + }, + "confidence": "high", + "summary": "The fix is one condition in printSummary (pkg/cmd/pr/checks/output.go:70) to count Canceled as a reason to print the summary, plus a regression test. The existing table-driven TTY tests in checks_test.go already pin the wording for the other state mixes, so the change is local and verifiable offline with a new all-cancelled fixture." + }, + { + "task": "pr-checks-all-cancelled-summary", + "attempt": 2, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:14:17.871700Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3817, + "cache_read_tokens": 59720, + "cache_creation_tokens": 26066, + "num_turns": 4, + "latency_seconds": 20.2, + "total_cost_usd": 0.006341445000000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The expected line \"Some checks were cancelled\" is stated in the task and already exists at pkg/cmd/pr/checks/output.go:76-77. The tally line for an all-cancelled PR (\"3 cancelled, 0 failing, ...\") is unspecified, but it follows the existing format at output.go:82-83 and the tests at checks_test.go:196-204.", + "silent_invariant": "The change is display-only. printSummary (output.go:69-92) runs only when IsStdoutTTY() is true and touches no auth, data, caching or concurrency. A wrong change shows as a blank or wrong line, which is visible.", + "coordinated_surfaces": "The defect is one gate condition in printSummary at output.go:71, which omits counts.Canceled. The cancelled bucket is set in aggregate.go:82-84, and the JSON export path (checks.go:190) does not use the summary. No docs, API or other components need to change.", + "ambiguity": "The gate at output.go:71 excludes Canceled, yet the if/else chain at output.go:72-80 already handles Canceled, so adding Canceled to the gate is the clear fix. Whether the tally line should also print for this case is not stated in the task and has to be inferred from the existing format.", + "verification": "checks_test.go:187-206 covers \"some cancelled tty\" with fixtures/someCancelled.json, and other mixes are covered at lines 207-541. No fixture or case covers all-cancelled, which is why the bug got through. The httpmock/FileResponse pattern (checks_test.go:191-193) makes the regression test easy to add without external services.", + "blast_radius": "gh pr checks is a user-facing CLI command (checks.go:62-113), so users would see the blank summary. A revert of the one-line gate change fixes it with no lasting harm, and no stored data, credentials or security paths are involved." + }, + "confidence": "high", + "summary": "The fix is a one-condition change in printSummary (pkg/cmd/pr/checks/output.go:71) so that an all-cancelled check set enters the summary branch, which already produces \"Some checks were cancelled\". The work also needs a regression test in checks_test.go using a new all-cancelled fixture next to someCancelled.json, since the existing cases never exercise that mix." + }, + { + "task": "pr-checks-all-cancelled-summary", + "attempt": 3, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:17:54.811244Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4449, + "cache_read_tokens": 87278, + "cache_creation_tokens": 31216, + "num_turns": 5, + "latency_seconds": 25.4, + "total_cost_usd": 0.007700528 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task states the expected text (\"Some checks were cancelled\") and that other mixes keep current wording. The rest of the output (tally line, table) for an all-cancelled PR is not stated, but follows from sibling behaviour: checks_test.go:188-206 (someCancelled.json) prints the summary plus the tally line, and output.go:82-85 formats it for every state.", + "silent_invariant": "Display-only change in printSummary (output.go:69-92). A wrong edit shows as wrong or missing text in TTY output, which the exact-match tests in checks_test.go catch. Non-TTY output is untouched (output.go:88 guards the print).", + "coordinated_surfaces": "One function in pkg/cmd/pr/checks/output.go. Gate and branch ordering are internal. No API, schema, docs or other service involved; checks.go:212 and :241 only call printSummary.", + "ambiguity": "Root cause is visible in output.go:71: the gate `if counts.Failed+counts.Passed+counts.Skipping+counts.Pending > 0` omits counts.Canceled, so with only CANCELLED checks summary stays \"\" and output.go:88-90 prints a blank line. The gate is the obvious fix (add Canceled), but a narrower fix (only setting summary in an else branch) would leave the tally line missing. Some reading needed to choose the gate fix.", + "verification": "checks_test.go:187-206 (\"some cancelled tty\") and the other tty cases at :155-330 compare full stdout exactly for each mix. A regression for all-cancelled needs a new fixture (none exists; only someCancelled.json, somePending.json have CANCELLED alongside other states), and the existing suite already catches a wrong change to other mixes.", + "blast_radius": "User-facing CLI output of `gh pr checks` (checks.go:212). Wrong output is visible to users but a revert restores prior behaviour; no auth, data or money involved. Exit code path (checks.go:248-254) is unaffected since all-cancelled returns nil." + }, + "confidence": "high", + "summary": "The fix is a one-condition change in printSummary (pkg/cmd/pr/checks/output.go:71) so the summary and tally line are emitted when only cancelled checks exist, plus a new all-cancelled fixture and TTY test in checks_test.go alongside the existing someCancelled case. Existing exact-output tests already cover every other state mix, so a regression test for the all-cancelled case is the main new verification." + }, + { + "task": "pr-comment-attach-ghu-tokens", + "attempt": 1, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:11:18.930527Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5296, + "cache_read_tokens": 98051, + "cache_creation_tokens": 35433, + "num_turns": 5, + "latency_seconds": 30.4, + "total_cost_usd": 0.0088647185 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task states the expected outcome (upload for an accepted ghu_ token, pass the server's error through for a refused one, keep other token types unchanged). Nearby code fills in the rest: `internal/attachments/client.go` `uploadError.Error()` maps 404 to 'requires write access' and 422/429 to specific messages, and `api/http_client.go:185` sets `token `. Reconciling 'server's own error comes back' with that 404 mapping is a repo convention the task does not spell out.", + "silent_invariant": "`internal/attachments/client.go` lines 71-86: `uploadTokenTypes` is an explicit allowlist, and the `checkUploadTokenType` doc says 'an unlisted kind is rejected rather than sent'. That gate decides which credential classes are sent to the upload endpoint. Widening it wrongly (e.g. admitting `ghs_` or `ghr_`) would look like it works while sending the wrong credential class.", + "coordinated_surfaces": "Change is mostly confined to `internal/attachments`: the allowlist in `client.go`, the pinned rejection row 'rejects an App user-to-server token' in `client_test.go` ~line 431, and possibly the `uploadError` messages. Callers (`pkg/cmd/pr/shared/commentable.go:179`, `pr/create`, `pr/edit`, `issue/create`, `issue/edit`) all go through `NewUploader` and inherit the change. `gh.TokenTypeUserToServer` already exists in `internal/gh/gh.go:118`.", + "ambiguity": "Two readings: (a) add `TokenTypeUserToServer` to `uploadTokenTypes` and flip the test row, or (b) drop the client-side prefix gate for ghu and let the server decide, with its error passed through. Option (a) passes tests quickly but keeps the 404 'requires write access' message, which the task's 'server's own error comes back' wording and the `uploadError` comment ('the endpoint answers 404 rather than 403 when the token cannot write') suggest is misleading for ghu. The repo does not settle which reading is intended.", + "verification": "`internal/attachments/client_test.go` has a `TestNewUploader` table covering each token type and `httpmock`-driven upload tests (`newTestUploader`, `testAsset`). Both run offline and would catch a wrong allowlist change or a changed upload path, so a ghu_ fix can be tested without live credentials.", + "blast_radius": "Shipped CLI used by customers and Copilot agent sessions; the uploads URL is on the same host gh already sends the token to (`api/http_client.go`). A wrong change is revertable and does not corrupt stored data. The credential-leak exposure is limited to tokens the user already presents to that host, but the change does touch an auth gate, so it is not a 0." + }, + "confidence": "medium", + "summary": "The work is to widen the `internal/attachments` upload-token allowlist to admit `ghu_` (`gh.TokenTypeUserToServer`) and make refusals surface the server's own error, which means reworking the 404 message mapping in `uploadError` as well as flipping the pinned test row. Because the allowlist is a deliberate credential gate and the obvious one-line fix can look complete while leaving the wrong error path, the change needs careful scoping and tests in `client_test.go`." + }, + { + "task": "pr-comment-attach-ghu-tokens", + "attempt": 2, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:15:13.349921Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 6566, + "cache_read_tokens": 201518, + "cache_creation_tokens": 37941, + "num_turns": 8, + "latency_seconds": 45.8, + "total_cost_usd": 0.0110468655 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The server contract (ghu_ accepted with the `token` scheme, Bearer gives 404) comes from the task text itself, not a linked spec. Nearby code and tests supply the rest: `internal/attachments/client.go` `uploadTokenTypes` is the allowlist, and `client_test.go` `TestNewUploader` has the row \"rejects an App user-to-server token\" that pins today's behaviour.", + "silent_invariant": "`client.go` `checkUploadTokenType` documents itself as fail-closed (\"an unlisted kind is rejected rather than sent\"). Widening the list wrongly, for example to ghs_ or unknown prefixes, would hand credentials to the upload endpoint. The server answers with an error rather than corrupting data, so the mistake is visible in tests and in the returned status, not silent.", + "coordinated_surfaces": "One allowlist in `internal/attachments/client.go` serves five callers through `NewUploader`: pkg/cmd/pr/shared/commentable.go (pr comment), pr/create, pr/edit, issue/create, issue/edit. Their tests (`commentable_test.go`, `pr/edit/edit_test.go`, `issue/edit/edit_test.go`, `pr/create/create_test.go`) only use ghs_ rows. The `--attach` help text in `pkg/cmd/pr/comment/comment.go` names no token types, so no docs change is needed.", + "ambiguity": "The task names only ghu_ and asks that existing working token types not change. Adding `gh.TokenTypeUserToServer` to `uploadTokenTypes` and keeping the other rejections is the reading the repo supports. The alternative of dropping the allowlist so the server decides for every type would change ghs_ and unknown-prefix behaviour, which the task does not authorise.", + "verification": "`internal/attachments/client_test.go` `TestNewUploader` is a table test that already runs ghu_ through the allowlist. Upload paths are covered with `httpmock` (`UploadStub`, 201/404/422/429 cases in `client_test.go` and `pr/shared/commentable_test.go`), so ghu_ success and server refusal can be tested without GitHub. The existing ghu rejection row must be replaced, not kept.", + "blast_radius": "This is a user-facing CLI command for every host that is not GHES. A ghu_ token is already sent to the API on the same host, so no new credential destination is opened. The server enforces permissions and the change is revertible. Uploads cannot be undone (`attachments/doc.go`), but that concerns uploaded assets, not stored records." + }, + "confidence": "high", + "summary": "The fix is to admit `gh.TokenTypeUserToServer` to the fail-closed `uploadTokenTypes` allowlist in `internal/attachments/client.go`, which all five upload commands share, while keeping the other rejections. The change must replace the `TestNewUploader` row that pins ghu_ rejection and add httpmock coverage for ghu_ success and for the server's refusal passing through." + }, + { + "task": "pr-comment-attach-ghu-tokens", + "attempt": 3, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:18:47.424798Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 5844, + "cache_read_tokens": 141275, + "cache_creation_tokens": 38615, + "num_turns": 6, + "latency_seconds": 38.6, + "total_cost_usd": 0.0100793275 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "internal/attachments/client.go:71-85 is an explicit allowlist (gho_, ghp_, github_pat_) with the comment 'an unlisted kind is rejected rather than sent'. The task fixes the ghu_ behaviour and the pass-through of server errors, but the rule that ghs_/ghr_ stay rejected is only pinned by internal/attachments/client_test.go:440-453, not stated in the task. Reading the nearby code is enough to derive it.", + "silent_invariant": "The allowlist in internal/attachments/client.go:72-85 is the only gate on which credential types are sent to the user-attachments endpoint, and it sits next to the host check (checkHost, GHES). Over-admitting (dropping the allowlist to 'let the server decide', which the task's 'server's own error comes back' invites, or admitting ghs_/ghr_) would still upload and look like it works. The task also requires other token types to be unchanged, so the mistake is silent.", + "coordinated_surfaces": "The change sits in one component: internal/attachments/client.go (uploadTokenTypes) and its tests in internal/attachments/client_test.go:373-545. pkg/cmd/pr/shared/commentable.go:179-180 passes ActiveTokenType into NewUploader unchanged, and pkg/cmd/pr/comment/comment.go needs no change. No docs, config schema, or deployment manifests reference the upload allowlist.", + "ambiguity": "The task names the direction (accept ghu_, return the server's error, leave other types alone) and the wrong path (curl's Bearer 404 note). The obvious fix is one allowlist entry plus flipping the 'rejects an App user-to-server token' row at client_test.go:432-438. A broader rewrite that drops the gate would violate 'behaviour for the token types that already work must not change'. Another design reading is a token-prefix denylist. The code points to the allowlist.", + "verification": "internal/attachments/client_test.go has a table test (TestNewUploader, lines 373-545) covering each token type, including ghs_ and ghr_ rejections, and httpmock upload tests (lines 48-152) that assert the Authorization header. docs/testing.md:90-94 prescribes httpmock.Registry with defer reg.Verify. A wrong change to the allowlist would fail existing rows, so these tests catch it without live credentials.", + "blast_radius": "Affects users of 'gh pr comment --attach' on github.com, including Copilot cloud agent sessions (task text). Uploads cannot be undone (internal/attachments/doc.go), but a bad admission is fixed by reverting the allowlist entry. Authentication is involved, but no stored data or credentials are exposed beyond the existing host-scoped token send. SECURITY.md contains only reporting instructions and does not change the score." + }, + "confidence": "medium", + "summary": "In this repository the fix is a deliberate change to the allowlist in internal/attachments/client.go (uploadTokenTypes), which is the only gate on which credentials may reach the upload endpoint, and the existing 'rejects an App user-to-server token' test must be replaced. The silent risk is over-admitting other token kinds or removing the gate, while the existing table tests in client_test.go already cover ghs_/ghr_ rejection and would catch that." + }, + { + "task": "pr-merge-delete-branch-owner-case", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:10:44.817834Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5009, + "cache_read_tokens": 95661, + "cache_creation_tokens": 36233, + "num_turns": 5, + "latency_seconds": 31.1, + "total_cost_usd": 0.0087905785 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour (same-repo head branch is deleted regardless of owner-name case). The repo already defines the comparison convention: `internal/ghrepo/repo.go:78-83` `IsSame` uses `strings.EqualFold` on owner, name and normalized host, and `pkg/cmd/pr/shared/find_refs_resolution.go:153` and `finder.go` use it. The fix therefore follows existing convention and is not specified in the task text itself.", + "silent_invariant": "`pkg/cmd/pr/merge/merge.go:592` sets `crossRepoPR: pr.HeadRepositoryOwner.Login != baseRepo.RepoOwner()`, a case-sensitive compare between the GitHub login and the owner from `remote.origin.url`. `crossRepoPR` gates `deleteRemoteBranch` (line 519) and the survey text (lines 671-677), so the bug is silent: no error, only a missing remote delete. Only mixed-case remotes trigger it, which is an edge case. A too-broad fix could wrongly delete a same-named base-repo branch for a fork PR, so the guard must stay exact apart from case.", + "coordinated_surfaces": "The change is one comparison expression in `merge.go:592` plus a regression test in `merge_test.go`. Help text at `merge.go:175` ('Delete the local and remote branch after merge') stays correct. No API, schema, docs or other component depends on this value.", + "ambiguity": "The tempting narrow fix would normalize owner case in the remote-URL parser or in `baseRepo`, or patch only `deleteRemoteBranch`. That leaves `crossRepoPR` wrong for the survey text at `merge.go:671-677`. The repo's `ghrepo.IsSame` (`internal/ghrepo/repo.go:79`) points to the right comparison, so the intended answer is clear once the code is read.", + "verification": "`pkg/cmd/pr/merge/merge_test.go` has `TestPrMerge_deleteBranch` (line 620), `TestPrMerge_deleteBranch_onlyLocally` (line 848, a real cross-owner case using `HeadRepositoryOwner: \"HEAD\"`), `TestPrMerge_alreadyMerged_withMergeStrategy_crossRepo` (line 1321), and `TestPRMergeTTY_withDeleteBranch` (line 1426). None uses a case-differing owner, so they would not catch this bug. The path is testable offline with `httpmock` and `baseRepo(...)` (line 238), so a mixed-case regression test is feasible.", + "blast_radius": "This is a user-facing `gh` CLI command (`AGENTS.md` architecture section). A wrong change is reversible by code revert, but it can delete a remote branch on GitHub (`deleteRemoteBranch`, `api.BranchDeleteRemote` at line 525), which is destructive. That keeps it above 0, but it is not a security, auth or stored-data path." + }, + "confidence": "high", + "summary": "The task needs `crossRepoPR` in `pkg/cmd/pr/merge/merge.go:592` to compare owners case-insensitively (the repo's `ghrepo.IsSame` convention), so same-repo head branches are deleted when the remote URL's owner casing differs from GitHub's login. It also needs a mixed-case regression test added to `merge_test.go`, since the existing delete-branch tests do not cover that case." + }, + { + "task": "pr-merge-delete-branch-owner-case", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:14:34.771496Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 6833, + "cache_read_tokens": 153393, + "cache_creation_tokens": 43379, + "num_turns": 6, + "latency_seconds": 46.0, + "total_cost_usd": 0.011411625500000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The expected behaviour is stated in the task, but whether a same-repo head branch is 'same repo' depends on GitHub login semantics and the repo's convention. pkg/cmd/pr/merge/merge.go:592 compares the API's canonical `pr.HeadRepositoryOwner.Login` with the remote-URL owner `baseRepo.RepoOwner()` using `!=`. internal/ghrepo/repo.go:79-81 `IsSame` uses `strings.EqualFold` for owners, so case-insensitive owner matching is already the project convention.", + "silent_invariant": "merge.go:519 gates remote deletion on `m.crossRepoPR`. The current bug is a silent false positive (remote branch kept, no error). The opposite mistake is also silent: if the owner check is loosened or wrongly classified as same-repo for a fork PR, `DELETE repos//git/refs/heads/` (merge.go:525) removes a base-repo branch of the same name that the PR does not own, and nothing reports it.", + "coordinated_surfaces": "One expression drives everything: merge.go:592 sets `crossRepoPR`, which feeds `deleteRemoteBranch` (merge.go:519) and `deleteBranchSurvey` (merge.go:325, 671-677). Both consumers stay consistent automatically. No docs or generated schema change; the CLI help text does not mention case. Sibling `pkg/cmd/pr/update-branch/update_branch.go:103` has the same case-sensitive `!=`, but it is outside this task.", + "ambiguity": "The task describes the symptom and the expected outcome. The repo makes the intended fix clear: `ghrepo.IsSame`/`strings.EqualFold` already define owner equality (internal/ghrepo/repo.go:79-80). A tempting narrower fix would special-case only the remote-deletion call or normalise the local URL. Canonicalising the base owner would need an extra API round-trip, which AGENTS.md says to avoid.", + "verification": "merge_test.go has httpmock-based coverage for deletion: TestPrMerge_deleteBranch (~L620), TestPrMerge_deleteBranch_apiError (~L673), TestPrMerge_deleteBranch_nonDefault, TestPRMergeTTY_withDeleteBranch, and TestPrMerge_deleteBranch_onlyLocally (~L848, HEAD owner). The tests use `baseRepo(\"OWNER\", ...)` with matching case throughout, so no existing test would catch reintroducing the case-sensitive comparison. The regression test must be added. It is fully runnable offline with the existing `runCommand`/`httpmock` harness.", + "blast_radius": "gh is a user-facing CLI (AGENTS.md). Remote branch deletion on GitHub is destructive and affects shared repos. A wrong classification could delete an unrelated base-repo branch. The damage is bounded to one branch and is reverted by shipping a fixed release, so this is scored 1 rather than 2. The silent-invariant risk above is the reason the classification must be exact." + }, + "confidence": "high", + "summary": "The fix is one case-sensitive owner comparison in `NewMergeContext` (pkg/cmd/pr/merge/merge.go:592) that should use case-insensitive owner equality, matching `ghrepo.IsSame`, so `deleteRemoteBranch` and the delete-branch survey both see a same-repo PR. The required work is that one-line change plus a regression test in merge_test.go with a lowercase or mixed-case base owner against an API `HeadRepositoryOwner` login, asserting the `DELETE repos/.../git/refs/heads%2F...` call and the 'Deleted remote branch' output." + }, + { + "task": "pr-merge-delete-branch-owner-case", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:18:16.264798Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 5004, + "cache_read_tokens": 65511, + "cache_creation_tokens": 28555, + "num_turns": 4, + "latency_seconds": 28.0, + "total_cost_usd": 0.0074002335 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour: same-repo head branch, remote deleted regardless of owner case. Which repos count as \"same\" follows from existing code: context/remote.go:30 `FindByRepo` uses `strings.EqualFold` on owner and name, and internal/ghrepo/repo.go:80 `IsSame` does the same. The task does not name these, so the convention has to be read from the code.", + "silent_invariant": "pkg/cmd/pr/merge/merge.go:592 sets `crossRepoPR: pr.HeadRepositoryOwner.Login != baseRepo.RepoOwner()`, a case-sensitive compare. `deleteRemoteBranch` (line 519) returns early on `crossRepoPR`, so the remote branch is silently skipped. A wrong fix that loosens this check could make `api.BranchDeleteRemote` (api/queries_pr.go:872, which deletes `heads/` in baseRepo) hit a base-repo branch of the same name when the head really lives in a fork. That is an edge case, not a visible failure in normal use.", + "coordinated_surfaces": "The defect is one comparison at pkg/cmd/pr/merge/merge.go:592. The flag help at merge.go:175 (\"Delete the local and remote branch after merge\") and the docs stay correct without change. No API, config or deployment surface is involved.", + "ambiguity": "The symptom is clear, and the obvious fix (case-insensitive owner compare, as in context/remote.go:30 or ghrepo.IsSame) matches repo convention. A second reading exists: normalising owner case where the remote URL is parsed, which would affect every command rather than only merge. The task does not choose between them, so the repo's existing comparison convention has to settle it.", + "verification": "pkg/cmd/pr/merge/merge_test.go has extensive delete-branch coverage (TestPrMerge_deleteBranch at ~620, the crossRepo test at ~1321, worktree tests), using httpmock and run.Stub. Every fixture uses matching case (baseRepo(\"OWNER\",\"REPO\",...) with HeadRepositoryOwner \"monalisa\" or unset), so a revert to case-sensitive compare would pass the suite. The task explicitly requires a new owner-case regression test, and the harness makes that easy to write.", + "blast_radius": "gh is a user-facing CLI. A wrong change would show as a missing or extra remote branch deletion, and the repo's own revert path fixes it. The operation is destructive (it deletes a remote ref in the base repo), which keeps this from being a 0. No auth, money or stored-data path is touched, and SECURITY.md (.github/SECURITY.md) does not make this change security-sensitive." + }, + "confidence": "medium", + "summary": "The fix is a one-line case-insensitive owner comparison in `crossRepoPR` in pkg/cmd/pr/merge/merge.go, matching the EqualFold convention already used in context/remote.go and internal/ghrepo. The work also needs a new regression test in merge_test.go with an owner-case mismatch, since every existing fixture uses matching case and would not catch a revert." + }, + { + "task": "pr-view-json-number-on-issue", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:10:47.802235Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 6602, + "cache_read_tokens": 124090, + "cache_creation_tokens": 36761, + "num_turns": 6, + "latency_seconds": 37.8, + "total_cost_usd": 0.0100522675 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected exit behaviour, and it is defined by the existing non-number path: any other field goes through the GraphQL `pullRequest(number:)` query in pkg/cmd/pr/shared/finder.go:362-390, and a non-PR number fails there (exit 1). Nothing outside the repo is needed; the expected behaviour follows from nearby code.", + "silent_invariant": "pkg/cmd/pr/shared/finder.go:210 and :252-255 short-circuit when only `number` is requested and return `&api.PullRequest{Number: f.prNumber}` with no API call. A wrong fix would show up as a wrong exit code or a wrong number in a normal run, not as silent data corruption. No auth, money, or persisted state is involved.", + "coordinated_surfaces": "The shortcut lives in one shared finder (pkg/cmd/pr/shared/finder.go) consumed by pr view (view.go:103), pr diff (diff.go:123, Fields [\"number\"]), and pr checks web mode (checks.go:131, Fields [\"number\"]). The tests that must agree are finder_test.go (\"number only\", line ~330, httpStub nil pins the no-API path) and view_test.go. No docs, schema, or other service change is implied. All of this sits inside one component, so it is 1.", + "ambiguity": "A view-local fix (e.g. forcing view to request an extra field so the GraphQL call happens) would pass a quick review and make `pr view N --json number` fail. But the shortcut is in shared finder.go:252-255, so `pr checks -w N` (checks.go:131-141) would still open /pull/N/checks for an issue number, and pr diff's number-only path keeps the same false positive. Deciding whether to drop the shortcut globally or only for view is a real design choice, and the narrow fix misses the root cause.", + "verification": "finder_test.go has ~40 table cases over httpmock GraphQL (e.g. \"number argument\" at line ~247, \"number only\" at ~330), and view_test.go has TestPRView_Preview_nontty plus TestJSONFields. These would catch many regressions. However no test covers a number-only lookup whose GraphQL response is an error (issue number), and the \"number only\" case pins the no-API behaviour, so it must be rewritten rather than kept. Partial coverage, so 1.", + "blast_radius": "This is a user-facing CLI (AGENTS.md: gh CLI, internal/ghcmd entry point). A bad change would break pr view/diff/checks for scripts that depend on current output, but it is fixed by a revert. It does not touch auth, money, or stored data." + }, + "confidence": "medium", + "summary": "The fix belongs in the shared PR finder's number-only short-circuit (pkg/cmd/pr/shared/finder.go:210-255), not in pr view alone, because pr diff and pr checks -w use the same path. Tests must change: the finder_test.go \"number only\" case that pins no API call has to be rewritten, and new coverage is needed for an issue number returning a GraphQL error in view --json number." + }, + { + "task": "pr-view-json-number-on-issue", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:14:38.705485Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 6295, + "cache_read_tokens": 113053, + "cache_creation_tokens": 30330, + "num_turns": 6, + "latency_seconds": 38.3, + "total_cost_usd": 0.008877748 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task states the expected exit code and output (non-zero for a non-PR, `{\"number\":N}` for a PR). The rest follows from existing code: `findByNumber` in pkg/cmd/pr/shared/finder.go:362 already runs `pullRequest(number:)`, so the non-PR error is the same one other fields produce. The task's own description, that the number-only path answers 'without asking GitHub', is the shortcut being removed.", + "silent_invariant": "pkg/cmd/pr/shared/finder.go:210-252: `numberFieldOnly` returns `&api.PullRequest{Number: f.prNumber}` before any GraphQL call, so the only type check is skipped. The false positive is visible through the exit code and output, and a regression test catches it easily, so it is not silent in the security or data-integrity sense.", + "coordinated_surfaces": "The shortcut lives in the shared finder, which pkg/cmd/pr/view/view.go:99-101 uses via Exporter.Fields(). It is also used by pkg/cmd/pr/checks/checks.go:131 and pkg/cmd/pr/diff/diff.go:123 with Fields: []string{\"number\"}. All of these are in pkg/cmd/pr, so they change together, but there is no API, schema, docs or generated-client surface to keep in step.", + "ambiguity": "The root cause is the shared `numberFieldOnly` shortcut in finder.go:210, not view.go. A narrower fix in pkg/cmd/pr/view/view.go (for example, adding a field so the query always runs) would make `pr view --json number` exit non-zero and pass the view tests. It would leave `pr checks` and `pr diff` with the same false positive. The task names only `pr view`, so the fix location is a real fork.", + "verification": "pkg/cmd/pr/shared/finder_test.go:330-347 'number only' case pins the current shortcut (httpStub: nil, wantPR: 13), so it asserts the bug and must be changed. pkg/cmd/pr/view/view_test.go has fixture-based PR-view cases (e.g. 'Open PR without metadata' with prViewPreview.json) that cover the PR path, but nothing covers a GraphQL not-found for an issue number. Tests exist and run offline with httpmock.", + "blast_radius": "Read-only CLI command used by scripts. The change alters exit codes and adds an API call for number-only lookups in view, checks and diff. A revert fully undoes it and no data is written; no security or money surface is involved." + }, + "confidence": "high", + "summary": "The fix belongs in the shared `numberFieldOnly` shortcut in pkg/cmd/pr/shared/finder.go, so that a number-only lookup queries `pullRequest(number:)` and fails for issue numbers. That shortcut also drives `pr checks` and `pr diff`, and the existing 'number only' finder test pins the buggy behaviour, so it must change along with new tests for the not-found case and the unchanged PR output." + }, + { + "task": "pr-view-json-number-on-issue", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:18:22.814577Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 5344, + "cache_read_tokens": 149382, + "cache_creation_tokens": 31370, + "num_turns": 8, + "latency_seconds": 36.3, + "total_cost_usd": 0.008897757000000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The expected behaviour is defined by existing code, not by the task text alone: `findByNumber` in pkg/cmd/pr/shared/finder.go (lines 362-390) already fails on a non-PR number for any other field set, so the task says to match that. The short-circuit at finder.go:252-255 is the only path that skips it.", + "silent_invariant": "finder.go:252-255 returns `&api.PullRequest{Number: f.prNumber}` with no API call. A wrong fix could still pass a quick check, e.g. if `findByNumber` returned a nil PR with no error on a GraphQL null, but that only shows on a non-PR number. Not a security or data path; pr view is read-only.", + "coordinated_surfaces": "The short-circuit is one shared function, `numberFieldOnly` in finder.go:210/252. It is also relied on by pr checks (checks.go:131), pr diff (diff.go:123) and the number-only case in finder_test.go:330-347, which has a nil httpStub. Those callers need checking, but no separate service, schema or doc surface is involved.", + "ambiguity": "Two readings exist: remove the shared short-circuit in finder.go, or special-case view.go so it requests more fields. The repo points to the first: the other PR fields already reach `findByNumber`, and the short-circuit is the only divergence. The view-only option would leave checks/diff with the same shortcut, so it is narrower than the bug.", + "verification": "finder_test.go TestFind has table cases for number-only (line 330) and 'pr number zero' (349), and view_test.go has httpmock-driven cases. But the 'number only' case pins the bug with httpStub nil and must be rewritten, and TestJSONFields (view_test.go:29) only checks field presence. No test covers `--json number` for a non-PR number.", + "blast_radius": "gh is a user-facing CLI (AGENTS.md, README.md). Scripts check PR-ness with `pr view --json number`, so a wrong change affects users. The damage is limited to read-only commands and can be fixed by reverting the release; no security, money or stored-data path is touched." + }, + "confidence": "medium", + "summary": "The fix is to stop the shared PR finder from answering number-only lookups without calling GitHub, so that `pr view --json number` goes through the same `pullRequest(number:)` query as other fields and fails for issue numbers. That change also reaches pr checks and pr diff, which use the same shortcut, so the existing number-only finder test (which pins the buggy no-API path) has to be rewritten and the view JSON path covered with new tests." + }, + { + "task": "project-item-add-output-non-tty", + "attempt": 1, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:10:48.666866Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 4671, + "cache_read_tokens": 65160, + "cache_creation_tokens": 29030, + "num_turns": 4, + "latency_seconds": 27.1, + "total_cost_usd": 0.007278535 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task asks for a 'short, stable confirmation, ideally' including the item ID, but no exact text is given. pkg/cmd/project/item-add/item_add.go printResults() is the only place the TTY text lives, and every sibling (item-create, item-archive, link, mark-template, item-edit) is silent when !IsStdoutTTY(). The wording and the non-TTY convention must be inferred from nearby code; the ID is already present in the mutation response (queries.ProjectItem.Id, asserted as \"item ID\" in item_add_test.go).", + "silent_invariant": "Output is not security-, money- or data-sensitive, and the change is visible: a wrong non-TTY string or extra text leaking into --format json shows up as wrong stdout. The exporter branch in runAddItem() returns before printResults(), so JSON separation is structural and testable. No SECURITY.md-relevant path is touched (.github/SECURITY.md is the generic disclosure policy).", + "coordinated_surfaces": "Change is confined to printResults() in item_add.go plus its test file item_add_test.go. The Go command help/Example text in NewCmdAddItem does not describe output, and no docs/ or acceptance/testdata/project txtar references item-add output (grep of docs/ and acceptance/ found nothing for 'item-add'). One component, one function.", + "ambiguity": "Symptom is 'prints nothing when non-TTY'. The narrow fix (print 'Added item' unconditionally in non-TTY) would look complete but ignores the 'ideally includes the item ID' requirement; the ID is available as query.CreateProjectItem.ProjectV2Item.Id. Also, changing only item-add diverges from the ~20 sibling project commands that are silent in non-TTY (grep for IsStdoutTTY in pkg/cmd/project). The design choice is visible in the repo but not settled by the task.", + "verification": "item_add_test.go has gock-based runAddItem tests for TTY text (lines ~211, 322, 423: 'Added item\\n' with SetStdoutTTY(true)) and a JSON test, all runnable offline via queries.NewTestClient(). No test covers the non-TTY path, so the current suite would not catch a wrong non-TTY change; the new non-TTY test is easy to write, but acceptance coverage for item-add is absent.", + "blast_radius": "Shipped CLI behaviour for end users and scripts; scripts that treat empty stdout as 'nothing to parse' would now receive a line. Cost is a output-format change that a revert fully undoes; no data, auth or money path (item_add.go only prints after Mutate succeeds)." + }, + "confidence": "medium", + "summary": "The change is a small, local edit to printResults() in pkg/cmd/project/item-add/item_add.go plus new non-TTY tests beside the existing TTY and JSON tests in item_add_test.go, with the item ID already available from the mutation response. The real decision is the stable non-TTY format and whether to diverge from the silent-in-non-TTY convention shared by every sibling project command." + }, + { + "task": "project-item-add-output-non-tty", + "attempt": 2, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:14:41.220568Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4630, + "cache_read_tokens": 90281, + "cache_creation_tokens": 27768, + "num_turns": 5, + "latency_seconds": 28.8, + "total_cost_usd": 0.007359011 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task asks for a 'short, stable confirmation, ideally one that includes the identifier'; the exact string is not fixed. `pkg/cmd/project/item-add/item_add.go` already fetches `queries.ProjectItem` (with `Id`) and prints `\"Added item\\n\"` in TTY, so the follow-on format is inferable. Sibling project commands (`copy_test.go:219-235` asserts empty stdout for non-TTY) show the silent non-TTY convention the task deliberately overrides for this command only.", + "silent_invariant": "Output-only change to a mutation's success message; no auth, data or concurrency logic in `item_add.go`. The one plausible wrong change (printing text alongside `--format json` output in `printResults`/exporter branch) would be visible in `TestRunAddItem_JSON` and by inspecting stdout.", + "coordinated_surfaces": "All changes are in one package: `printResults` in `item_add.go` plus tests in `item_add_test.go` (TTY `TestRunAddItem_User/Org/Me` at lines ~211/322/423, JSON `TestRunAddItem_JSON`). No docs reference item-add output (grep of *.md found none); no acceptance or generated-client surface.", + "ambiguity": "Task says the ID is 'ideally' included, so a bare `Added item` line passes a quick review yet drops the identifier the task asks for. The repo settles the rest: the item ID is already in the mutation response (`ProjectItem.Id`), and the TTY path shows the existing message shape.", + "verification": "`item_add_test.go` has `ios.SetStdoutTTY(true)` tests for TTY output and a JSON test, but no non-TTY case; `iostreams.Test()` plus gock make the non-TTY case easy to add. Current tests would not catch a regression in the non-TTY path.", + "blast_radius": "Public `gh project item-add` CLI output consumed by scripts and agents; a wrong message is user-visible and revertable by reverting one function. Existing callers that rely on empty non-TTY stdout would see new output." + }, + "confidence": "medium", + "summary": "In this repo, the fix is a small change to `printResults` in `pkg/cmd/project/item-add/item_add.go` that stops returning early on non-TTY stdout and prints a stable confirmation carrying the already-fetched `item.Id`, while the `--format json` exporter path stays unchanged. It also needs new non-TTY tests beside the existing TTY and JSON tests in `item_add_test.go`, and it departs from the silent non-TTY convention that every sibling project command follows, pinned by `copy_test.go`." + }, + { + "task": "project-item-add-output-non-tty", + "attempt": 3, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:18:23.314682Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 4104, + "cache_read_tokens": 61132, + "cache_creation_tokens": 24732, + "num_turns": 4, + "latency_seconds": 24.5, + "total_cost_usd": 0.006331402 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/project/item-add/item_add.go printResults() returns nil when !IsStdoutTTY(), so the non-TTY output is undefined. The task asks for a 'short, stable confirmation, ideally including the item identifier' but gives no format. Sibling commands set the pattern: item-create, item-edit, item-archive and link all gate a fixed 'X item\\n' or 'Linked ...' line on IsStdoutTTY. The ProjectItem.Id field is already fetched by the mutation, so the ID is available without an extra round-trip. The exact wording is left to the implementer, which is a convention question rather than a contract defined elsewhere.", + "silent_invariant": "The change is output-only. item_add.go performs the mutation before printResults, and the mutation is unchanged. A wrong change (missing line, wrong stream, wrong value) shows directly in stdout, and the existing TTY tests in item_add_test.go assert the exact string 'Added item\\n'. No auth, data-integrity or concurrency invariant is touched.", + "coordinated_surfaces": "The change lives in one package: item_add.go (printResults) and item_add_test.go. The --format json path (cmdutil.AddFormatFlags, opts.exporter.Write) is already separate and needs no change. No docs or generated artefacts for item-add exist in the repo (grep for 'item-add' finds only project.go, resolve.go and the package itself).", + "ambiguity": "The symptom is 'prints nothing when non-TTY'. The narrow fix (drop the IsStdoutTTY gate and print 'Added item') would pass a quick review, but the task asks for an ID in the message, which the narrow fix omits. The repo makes the intended shape clear enough: sibling commands use the same gate and message style, and ProjectItem.Id is already selected in item_add.go's addProjectItemMutation. Whether to also change the sibling commands (item-create, item-edit, item-archive, item-delete, link) is a scope question; the task names only item-add.", + "verification": "item_add_test.go has three TTY tests (TestRunAddItem_User, _Org, _Me) that assert the exact 'Added item\\n' string via iostreams.Test() with SetStdoutTTY(true), and TestRunAddItem_JSON covers the exporter. No test covers the non-TTY branch, which is the branch this change introduces. A wrong non-TTY change (e.g. writing to stderr, or an ID missing from the line) would pass the current suite. The gock/httpmock setup is in-process and needs no credentials, so the missing test is easy to add.", + "blast_radius": "gh is a public CLI. The change adds stdout text in non-TTY mode, which could affect any script that currently relies on empty stdout from item-add. Reverting the change restores the old behaviour with no lasting harm, and no stored data, credentials or money is involved." + }, + "confidence": "high", + "summary": "The task needs item-add's printResults to emit a stable non-TTY confirmation that includes the already-fetched ProjectItem.Id, while leaving TTY output and --format json unchanged, all within item_add.go and item_add_test.go. The main risks are choosing the exact wording and scope (whether sibling TTY-gated project commands should change too) and adding the missing non-TTY test." + }, + { + "task": "project-item-edit-by-name", + "attempt": 1, + "ref": "v2.96.0", + "judged_at": "2026-10-14T22:12:55.670650Z", + "metrics": { + "input_tokens": 22, + "output_tokens": 10484, + "cache_read_tokens": 323004, + "cache_creation_tokens": 56390, + "num_turns": 10, + "latency_seconds": 72.2, + "total_cost_usd": 0.017075289 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The URL form mirrors pkg/cmd/project/item-add/item_add.go (`[] --owner --url`). The URL-to-project-item mapping has a repo precedent in pr/edit (pkg/cmd/pr/edit/edit.go:302-306) and api/queries_pr.go, which read `projectItems` with `project.ID` per PR. The issue's only explicit type rule is single-select names an option; text/number/date are implied by the existing `--text`/`--number`/`--date` flags. The iteration type has no name config in the repo: `ProjectField.IterationField` (queries.go:925-935) selects only ID, Name and DataType.", + "silent_invariant": "A wrong resolution writes to the wrong item. The lookup must be filtered to the target project (project.ID), and case-insensitive matching must reject duplicate option or field names. Both show up in the printed `Edited item %q` title (item_edit.go:261-267) and in the mutation. No auth, secrets or money are involved, so the failure is visible on inspection rather than hidden.", + "coordinated_surfaces": "The change lands in one component, pkg/cmd/project: item_edit.go (flags, `MarkFlagRequired(\"id\")` at line 144, Long/Example text, RunE validation at lines 87-126), a new lookup in pkg/cmd/project/shared/queries/queries.go alongside `ProjectItems` and `IssueOrPullRequestID`, and item_edit_test.go. No API, docs or deployment surface is named in the task. Only item-edit is in scope.", + "ambiguity": "The tempting narrow fix is name resolution for --field and --value only, with the item still addressed by --id. The issue asks for the ` --owner --url` addressing that item-add uses, but item-add's URL lookup returns the content node ID (queries.go:1257-1275), which is not a ProjectV2 item ID. The correct item must come from a content-to-project-item lookup (precedent: pr/edit `projectItems`), and that step is not specified in the task. The `--id` requirement also has to be relaxed while the existing `required flag(s) \"id\" not set` test (item_edit_test.go:14-30 area) is handled.", + "verification": "item_edit_test.go (812 lines) covers flag parsing and every run path (TestRunItemEdit_Text, _Number, _Date, _SingleSelect, _Iteration, _Clear, _Draft*) with gock and queries.NewTestClient(), so new cases need no credentials. Nothing covers name resolution, URL-to-item lookup, the project-scoped match, duplicate or case-variant names, or the candidate-listing error. The new paths must be tested from scratch.", + "blast_radius": "This is a user-facing gh subcommand, used by people and agents to write ProjectV2 field values. A bad edit is reversed by editing the field back. AGENTS.md lists no security-critical area for this package, and no secrets, authorisation or migrations are touched." + }, + "confidence": "medium", + "summary": "The change adds an opt-in name-based path to `item-edit` in pkg/cmd/project: it must resolve a project number and owner, map a content URL to the project item ID (not the content ID that item-add returns), and resolve field and option names case-insensitively with candidate-listing errors, while keeping `--id`, `--field-id` and `--project-id` working. The repo settles the flags, the item-add mirror and the gock test pattern, but the URL-to-item lookup and iteration-name handling depend on GitHub GraphQL behaviour that is only partly visible in the repo, so the scores rest partly on inference." + }, + { + "task": "project-item-edit-by-name", + "attempt": 2, + "ref": "v2.96.0", + "judged_at": "2026-10-14T22:16:41.048193Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 10090, + "cache_read_tokens": 212951, + "cache_creation_tokens": 57374, + "num_turns": 8, + "latency_seconds": 61.8, + "total_cost_usd": 0.015782866 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task says mirror item-add's `[] --owner --url` form. item_add.go:84-115 shows the convention: NewOwner, NewProject(fields=false), then IssueOrPullRequestID. The value semantics follow the existing item-edit flags (--text, --number, --date, --single-select-option-id, --iteration-id; item_edit.go:129-142). Nothing outside the repo is needed to define the flags, but the URL-to-project-item lookup and the iteration-name rule are left to inference.", + "silent_invariant": "A wrong resolution writes a field value to the wrong ProjectV2 item or option. That is persisted project data, but it is visible on the board and reversible by re-editing. Failure needs a specific matching bug (duplicate option names differing only by case, or matching the issue number instead of the URL), not normal use. No auth, secrets or concurrency involved.", + "coordinated_surfaces": "Changes concentrate in pkg/cmd/project/item-edit/item_edit.go (flags, RunE, MarkFlagRequired(\"id\") at line 144, the mutually-exclusive checks at lines 91-108) and its test file. Likely additions in pkg/cmd/project/shared/queries/queries.go: a project-item-by-URL lookup (ProjectItems at line 658 takes a query string, and ProjectItem.URL() exists at line 630) and iteration configuration fields, which ProjectField (lines 920-945) does not fetch. No docs for project commands exist in the repo (grep for item-edit found only project.go and item_edit.go).", + "ambiguity": "Tempting incomplete fix: reuse IssueOrPullRequestID (queries.go:1257), which returns the content node ID, as the item ID. item-add can do this because addProjectV2ItemById takes contentId (item_add.go:119-122). item-edit's updateProjectV2ItemFieldValue needs the ProjectV2Item ID (item_edit.go:234-237), so that shortcut is wrong, and mocked tests would not catch it. Iteration-by-name is also unspecified: ProjectField fetches no iteration list, and the task says only that --value is interpreted by data type. Making --id optional also changes the `missing-id` contract (item_edit_test.go:24-27).", + "verification": "item_edit_test.go (812 lines) covers each value type with gock GraphQL mocks (for example TestRunItemEdit_SingleSelect, TestRunItemEdit_Iteration, TestNewCmdeditItem flag parsing). queries_test.go covers ProjectItems and ProjectFields pagination with mocks. All of this runs offline. Gaps: mocks only check request shape and cannot validate new GraphQL fields against the live schema, and GHES compatibility of new fields cannot be exercised here (queries.go:45-54 documents a past GHES 3.8 break from an added field).", + "blast_radius": "User-facing CLI writing to user and org project boards. Errors would overwrite field values on the wrong item or option, which is recoverable by a revert or re-edit. No auth, billing or irreversible data loss. SECURITY.md is about vulnerability reporting and does not raise this change's stakes." + }, + "confidence": "medium", + "summary": "The change must add name-based resolution for project, item, field and single-select option to item-edit, keeping --id/--field-id/--project-id intact and making --id conditional. The hard part is that item-edit needs a ProjectV2Item ID, which the existing URL helper does not return, so the item must be found through the project's items, and iteration names need GraphQL fields the shared queries do not fetch yet." + }, + { + "task": "project-item-edit-by-name", + "attempt": 3, + "ref": "v2.96.0", + "judged_at": "2026-10-14T22:20:20.754058Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 8061, + "cache_read_tokens": 226333, + "cache_creation_tokens": 51258, + "num_turns": 8, + "latency_seconds": 48.1, + "total_cost_usd": 0.013973168 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The issue states the flags and matching rules, and its example mirrors item-add (pkg/cmd/project/item-add/item_add.go: `[] --owner --url`, NewProject plus IssueOrPullRequestID). Two things are not stated: how a URL becomes a ProjectV2 item ID (no helper exists in pkg/cmd/project/shared/queries/queries.go; the nearby pr/shared/finder.go and issue/shared/lookup.go already fetch `projectItems` on content, so the pattern is discoverable), and the error text for listing candidates (no convention in the task).", + "silent_invariant": "The change writes field values to a user's project board. A wrong match (an item from another project, a case-insensitive field-name collision, a duplicate option name) would overwrite the wrong value. The repo's SECURITY.md covers vulnerability reporting only, with no project-specific invariant. The failure needs an edge-case input such as duplicate names or multi-project membership, so a normal check would miss it.", + "coordinated_surfaces": "Changes stay inside pkg/cmd/project: item_edit.go (new flags, the `MarkFlagRequired(\"id\")` at line 144, RunE mutual-exclusion checks at lines 91-108, runEditItem routing at lines 149-169), a likely new GraphQL lookup in shared/queries/queries.go, and item_edit_test.go. The command is registered in project.go:63 with no other changes needed. No docs/ or acceptance/testdata entry covers item-edit (grep found only project.go and the item-edit files).", + "ambiguity": "A narrower fix would pass review but miss the ask: resolving only --field/--value while still requiring --id skips the URL-to-item lookup the example needs. Other narrow fixes handle only single-select and ignore 'interpret --value by data type' (TEXT/NUMBER/DATE/ITERATION are all in FieldValueNodes, queries.go:367-446). Whether --url replaces or sits beside --id is open, and it changes the required-flag logic at item_edit.go:144.", + "verification": "item_edit_test.go has a solid offline harness: gock mocks of GraphQL bodies (TestRunItemEdit_SingleSelect, _Number, _Date, _Clear), NewTestClient, and iostreams.Test(), plus flag parsing tests in TestNewCmdeditItem. Nothing covers name resolution, the URL lookup, or the candidate-listing error, so the new behaviour needs new tests. Those are fully testable without external services, so the gap is in coverage, not testability.", + "blast_radius": "Public user-facing command (registered in project.go:63) that mutates project field values. A bad release is undone by a revert, and no credentials or security boundary are touched (SECURITY.md is about reporting). Mutated project data could need manual repair, which keeps it above 0." + }, + "confidence": "medium", + "summary": "The task needs item-edit to resolve a project item from an issue or PR URL, a project from --owner and number, and a field and option (or value) by case-insensitive name, validated before the mutation, while keeping --id, --field-id and --project-id working. The item-from-URL lookup is not in the repo (the GitHub GraphQL schema is external), so its shape is inferred from the existing projectItems usage in pr/shared and issue/shared, and the change must be covered by new offline gock tests in item_edit_test.go." + }, + { + "task": "release-create-404-scope-blame", + "attempt": 1, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:11:01.920199Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 4518, + "cache_read_tokens": 67213, + "cache_creation_tokens": 29810, + "num_turns": 4, + "latency_seconds": 28.9, + "total_cost_usd": 0.007324218 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/release/create/http.go:299-312 (tokenHasWorkflowScope) splits X-Oauth-Scopes on bare \",\" with no TrimSpace, so the GitHub header format (\", \"-separated, as in the task's debug sample `gist, project, read:org, repo, user, workflow`) is an external API convention the code does not state. The task text gives the expected behaviour in full, but the header format it depends on is not spelled out in the repo.", + "silent_invariant": "The check only decides whether a 404 is replaced by errMissingRequiredWorkflowScope (http.go:211-216) and whether the hint or the underlying error is shown. It is not an auth or data gate. A wrong change (e.g. hint always shown, or hint never shown) is visible in the stderr output, and the failure only appears for headers with whitespace or specific scope lists, which a normal check can miss.", + "coordinated_surfaces": "The fix lives in one helper, pkg/cmd/release/create/http.go:301-311. It must agree with the message path in create.go:516-528 (hint text, SilentError) and with the header fixtures in create_test.go:1213 (`repo,read:org`) and create_test.go:1239 (`repo,read:org,workflow`), which encode the no-space format. All are in one package, so this is two or three places in one component.", + "ambiguity": "The task gives the root symptom and a header sample that shows the cause. Splitting on \",\" leaves \" workflow\" with a leading space, so slices.Contains at http.go:311 returns false. A tempting narrow answer is changing only the test fixture, or dropping the hint, and either would look complete. The repository makes the intended fix clear, though: trim the split elements and keep the hint when the scope is truly absent.", + "verification": "create_test.go:1196-1221 and 1222-1242 cover both branches with httpmock.StatusScopesResponder (stub.go:256-266). The 'has workflow' case uses `repo,read:org,workflow` with no spaces, so it passes on the buggy code and misses the reported failure. The tests run locally with no external services, so a spaced-header test is easy to add.", + "blast_radius": "gh is a user-facing CLI, and this change alters the stderr error shown after a failed `gh release create` (create.go:518-527). It touches no credentials, money or stored data, and a revert of the one-line parsing change fully undoes it, so users notice but no lasting harm follows." + }, + "confidence": "high", + "summary": "The task needs the scope check in tokenHasWorkflowScope to trim whitespace from GitHub's comma-space X-Oauth-Scopes header, so tokens that hold workflow stop triggering the hint and the underlying 404 surfaces, while a token that truly lacks workflow still gets the hint. The existing create_test.go cases use no-space header fixtures and pass on the buggy code, so new tests with spaced headers are needed to cover the change." + }, + { + "task": "release-create-404-scope-blame", + "attempt": 2, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:14:56.990805Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4288, + "cache_read_tokens": 109221, + "cache_creation_tokens": 35386, + "num_turns": 5, + "latency_seconds": 28.5, + "total_cost_usd": 0.008426726 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/release/create/http.go:299-312 (tokenHasWorkflowScope) splits X-Oauth-Scopes on \",\" without trimming, but GitHub sends \"gist, project, read:org, repo, user, workflow\" (task debug log). The correct parsing is a sibling convention: api/client.go:214-221 (generateScopesSuggestion) calls strings.TrimSpace on each split element. The expected error text comes from existing api.HandleHTTPError. Nothing outside the repo is needed.", + "silent_invariant": "Scope check gates only a user-facing hint in create.go:516-528 and http.go:211. It is not auth enforcement. A wrong change is visible in the 404 output. An edge exists: the header format with and without spaces. The existing fixture at create_test.go:1239 uses \"repo,read:org,workflow\" (no spaces), which is why the bug went unnoticed.", + "coordinated_surfaces": "One package: the fix lives in tokenHasWorkflowScope (http.go:301-312). Consumers are createRelease (http.go:211) and the errors.As branch in create.go:516-528. The tests are create_test.go:1196-1242 and httpmock.StatusScopesResponder (pkg/httpmock/stub.go:257). The fixture may need the real \", \" separator. No docs or other services are involved.", + "ambiguity": "Task states the expected behaviour: report the underlying HTTP 404 when workflow is present, keep the hint when it is absent. Removing the hint entirely would clear the symptom but break the stated requirement, so the tempting shortcut is ruled out by the text. Root cause is discoverable from http.go:302-311 versus api/client.go:219-221. Some interpretation remains about whether to fix the parser or the fixture.", + "verification": "create_test.go:1196 (no workflow scope, hint expected) and :1222 (workflow scope, underlying error expected) exist and run in the unit suite. The :1222 case uses a no-space header, so it passes on the buggy code and does not catch the bug. The fix needs a fixture with real \", \" separators. Tests exist but miss the failure mode.", + "blast_radius": "User-facing CLI error text in a shipped command (pkg/cmd/release/create). Mistake misleads users about scopes and hides the real 404 cause, but touches no auth, secrets or stored data. A revert of the release fixes it. Not internal-only, so not 0." + }, + "confidence": "high", + "summary": "The fix is a whitespace-trim in tokenHasWorkflowScope (pkg/cmd/release/create/http.go) so the comma-split header matches \"workflow\" in GitHub's \", \"-separated X-Oauth-Scopes, matching the TrimSpace pattern in api/client.go. The work also needs the release-create 404 tests fixed to use the real header format so they fail on the current bug, while keeping the scope hint for tokens that truly lack workflow." + }, + { + "task": "release-create-404-scope-blame", + "attempt": 3, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:18:36.846725Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 5080, + "cache_read_tokens": 112405, + "cache_creation_tokens": 30372, + "num_turns": 6, + "latency_seconds": 30.9, + "total_cost_usd": 0.008208145 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected output (underlying `HTTP 404` error when the token has `workflow`, hint kept when it lacks it) and gives the real header value. What it does not state is how the header is delimited. The repo fixes that: `api/client.go:219-221` (`generateScopesSuggestion`) and `pkg/cmd/auth/shared/oauth_scopes.go` (`HeaderHasMinimumScopes`) both `strings.TrimSpace` each comma-split scope, while `pkg/cmd/release/create/http.go:301-312` (`tokenHasWorkflowScope`) does not. The `, ` separator is what GitHub sends, so this is convention in the code, not an external contract.", + "silent_invariant": "`tokenHasWorkflowScope` is an informational heuristic that only chooses between a hint and the raw error (`http.go:211-216`). It is not an authz check. The bug only appears with the real `, `-separated header; the existing test uses `repo,read:org,workflow` (no spaces, `create_test.go:1223-1239`), so the defect is input-format dependent rather than silent in normal use.", + "coordinated_surfaces": "The defect and its fix sit in one component: `pkg/cmd/release/create/http.go:301-312` (the parser), the 404 branch at `http.go:211-216`, and the message in `create.go:516-528`. Tests live in `create_test.go`. The sibling parser in `api/client.go` already does the right thing, so it is a reference, not a second surface that must change.", + "ambiguity": "Trimming the header parse makes the reported repro (`X-Oauth-Scopes: ... workflow`) print the HTTP 404, and that would look complete. But the task's second complaint is that the real cause (typo, missing repo, missing push access) is hidden behind the scope hint. That still happens for any token that truly lacks `workflow`, and the task does not say how to show both the hint and the underlying error. Choosing between keeping the hint-replacing-error behaviour and appending the hint to the real error is a design decision.", + "verification": "`create_test.go:1197-1221` (no workflow scope \u2192 hint) and `:1223-1240` (workflow scope \u2192 HTTP 404) exist, so both branches are exercised. Neither uses the real `, `-separated header, which is why the bug passed the suite. Testing is local with `httpmock.StatusScopesResponder` (`pkg/httpmock/stub.go:257`), so a regression test for the spaced header is cheap. The gap is coverage of the input form, not missing infrastructure.", + "blast_radius": "`gh release create` is user-facing (`pkg/cmd/release/create/`). A wrong result is a wrong error message or a hint shown when it should not be. It touches no credentials, stored data or authorisation (`http.go:299-312` only reads a response header), and a revert fixes it with no lasting harm." + }, + "confidence": "medium", + "summary": "The fix is in `tokenHasWorkflowScope` (`pkg/cmd/release/create/http.go:301-312`), which must trim whitespace from the comma-split `X-Oauth-Scopes` value the way `api/client.go` and `oauth_scopes.go` already do, and it needs a regression test using GitHub's `, `-separated header in `create_test.go`. The harder part is deciding how the hint should coexist with the underlying 404 for tokens that truly lack `workflow`, because the task asks both to keep the hint and to stop hiding the real cause." + }, + { + "task": "release-verify-sha1-prefix", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:11:52.925863Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 6657, + "cache_read_tokens": 201769, + "cache_creation_tokens": 40131, + "num_turns": 8, + "latency_seconds": 38.1, + "total_cost_usd": 0.011400801499999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the target behaviour: prefix = algorithm of the ref the API returns (sha1 for 40-hex, sha256 for 64-hex) in both commands. The remaining dependency is the attestation lookup format, which pkg/cmd/attestation/api/client.go (GetByDigest/buildRequestURL) passes through unchanged. Nearby code shows the convention: pkg/cmd/attestation/artifact/artifact.go DigestWithAlg() formats `alg:digest`, and shared/attestation.go FilterAttestationsByFileDigest hard-codes the \"sha256\" subject key, so the algorithm name must match what the attestation subjects use. That subject naming for SHA-256 commits is GitHub's external contract and is not in this repo.", + "silent_invariant": "The algorithm string feeds api.FetchParams.Digest (lookup), and verification.BuildDigestPolicyOption (pkg/cmd/attestation/verification/policy.go) uses a.Algorithm() as the sigstore artifact-digest key. A wrong prefix on a SHA-256 repo most likely produces a visible lookup failure or policy mismatch rather than a false pass. The wrong-alg case only shows up for SHA-256 repos, which the normal SHA-1 test path does not exercise.", + "coordinated_surfaces": "Two call sites must agree: pkg/cmd/release/verify/verify.go:133 and pkg/cmd/release/verify-asset/verify_asset.go:145 both hard-code NewDigestedArtifactForRelease(ref, \"sha1\"). The shared helper FetchRefSHA in pkg/cmd/release/shared/fetch.go returns only the raw SHA string, so the algorithm has to be derived in or next to it. All of this is inside pkg/cmd/release, with no docs or deployment surfaces involved (docs/ has no matching text).", + "ambiguity": "The symptom is clear and the task fixes the expected output. Two readings remain: derive the algorithm from the ref length in one shared place, or hard-code per command. The shortcut of hard-coding sha256 when len==64 in each command would pass a quick review but duplicate logic and leave the shared helper inconsistent. The repo does not settle which is preferred, so some interpretation is needed.", + "verification": "pkg/cmd/release/shared/fetch_test.go TestFetchRefSHA covers only a 40-char SHA. verify_test.go and verify_asset_test.go use a 40-char fake SHA (fakeSHA := \"1234567890abcdef...\") and assert error text that omits the digest (\"no attestations for tag v1\"). Test_verifyRun_Success does not assert the \"Resolved tag ... to sha1:...\" line. The httpmock/StubFetchRefSHA harness can exercise both SHA lengths offline, so the gap is coverage, not testability.", + "blast_radius": "pkg/cmd/release/verify and verify-asset are user-facing commands in a security feature (release attestation verification). A wrong prefix is a user-visible output and lookup error that a revert fixes, with no data or credential exposure. SECURITY.md only points to reporting channels and does not raise this change's risk." + }, + "confidence": "medium", + "summary": "The change needs the release verify and verify-asset commands to derive the digest algorithm from the ref length (40 hex = sha1, 64 hex = sha256) in one shared place, and to pass that same label to the attestation lookup and the sigstore digest policy. The tests need to cover both lengths, since no existing test asserts the prefix. Confidence is medium because GitHub's attestation subject naming for SHA-256 commits is an external contract not visible in this repository." + }, + { + "task": "release-verify-sha1-prefix", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:15:56.259404Z", + "metrics": { + "input_tokens": 20, + "output_tokens": 6577, + "cache_read_tokens": 197015, + "cache_creation_tokens": 37773, + "num_turns": 9, + "latency_seconds": 41.6, + "total_cost_usd": 0.010980502500000001 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The `sha256:` lookup key is fixed by the GitHub attestations API, not this repo. `pkg/cmd/release/verify/verify.go:137` sends `releaseRefDigest.DigestWithAlg()` as `api.FetchParams.Digest`, and `pkg/cmd/attestation/api/client.go:111` builds the request URL from it. The task says the wrong prefix 'will make attestation lookups fail'. Subject naming for SHA-256 git objects is defined outside the repo.", + "silent_invariant": "`pkg/cmd/attestation/verification/policy.go:21-25` decodes the hex digest and calls `verify.WithArtifactDigest(a.Algorithm(), ...)`. The algorithm label therefore feeds the sigstore subject match in `pkg/cmd/release/shared/attestation.go:102-114`. A wrong label should fail closed (no subject match), so the failure is mostly visible rather than a silent pass. This is still a verification gate, so the label must be right.", + "coordinated_surfaces": "Hardcoded `\"sha1\"` appears in `pkg/cmd/release/verify/verify.go:133` and `pkg/cmd/release/verify-asset/verify_asset.go:145`, both passed to `artifact.NewDigestedArtifactForRelease` (`pkg/cmd/attestation/artifact/artifact.go:57`). The algorithm must be derived consistently in those two commands, the lookup, the printed output and the verifier. All of this sits in one area of the CLI with no separate service or deployment config.", + "ambiguity": "The task names a symptom (label reads `sha1:` for a 64-char ref) and requires both the printed output and the attestation lookup to change. A fix that only edits the printed string would look complete but leave `verify.go:137` and `verify_asset.go:149` querying with `sha1:`. How the algorithm is derived (ref length vs. a repo object-format query) is also an open design choice.", + "verification": "`pkg/cmd/release/verify/verify_test.go` and `verify_asset_test.go` cover success, no-attestation and tag-mismatch paths, but only with a 40-char SHA (`fakeSHA := \"1234567890abcdef...\"`). `shared.MockVerifier.VerifyAttestation` (`pkg/cmd/release/shared/attestation.go:124-131`) ignores the artifact's algorithm, so a wrong `sha1`/`sha256` label would pass the existing suite. The live Sigstore verifier needs trust-root and network access.", + "blast_radius": "Both commands are user-facing (`verify.go` and `verify_asset.go` Long help describes verification). The change only alters the algorithm label and lookup key, and no SECURITY.md exists in the repo. A bad change produces wrong output or failed lookups, and a revert restores prior behaviour with no stored-data impact." + }, + "confidence": "medium", + "summary": "The change must derive the digest algorithm from the API ref in both `gh release verify` and `gh release verify-asset`, and use the same value for the attestation lookup, the printed output and the sigstore subject policy. The SHA-256 subject naming the lookup depends on is defined by the GitHub attestations service, not this repo, and the existing tests only use 40-char SHAs with a mock verifier that ignores the algorithm, so SHA-256 tests must be added to catch a wrong label." + }, + { + "task": "release-verify-sha1-prefix", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:19:29.171861Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5104, + "cache_read_tokens": 90803, + "cache_creation_tokens": 32817, + "num_turns": 5, + "latency_seconds": 29.0, + "total_cost_usd": 0.008319690499999999 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/release/verify/verify.go:133 and pkg/cmd/release/verify-asset/verify_asset.go:145 hard-code NewDigestedArtifactForRelease(ref, \"sha1\"). The task states the expected output fully (prefix matches ref length/algorithm, both commands, SHA-1 unchanged). The repo's own convention (verify_asset.go:135 uses \"sha256\" for file digests; attestation/verification/policy.go:25 passes a.Algorithm() to sigstore) shows the label must match the algorithm the verifier expects. Nothing outside the repo is required beyond that. FetchRefSHA (pkg/cmd/release/shared/fetch.go:138-171) returns only the SHA string, so the algorithm must be inferred from length; the task does not say how.", + "silent_invariant": "The algorithm reaches api.FetchParams.Digest (verify.go:137) and VerifyAttestation -> verification.BuildDigestPolicyOption (pkg/cmd/attestation/verification/policy.go:19-26), which feeds verify.WithArtifactDigest(a.Algorithm(), ...). A wrong algorithm fails the lookup or the subject match, which surfaces as an error, not a silent pass. This is supply-chain verification, but the failure mode is loud.", + "coordinated_surfaces": "Changes must agree across pkg/cmd/release/verify/verify.go, pkg/cmd/release/verify-asset/verify_asset.go, and optionally a shared helper in pkg/cmd/release/shared/fetch.go or pkg/cmd/attestation/artifact/artifact.go. All are in one CLI codebase. No docs or generated clients reference the prefix; docs/ has no release-verify page matching it.", + "ambiguity": "The tempting narrow fix is to change only the printed label at verify.go:179 and verify_asset.go:198. The task explicitly says the attestation lookup also uses the prefix (verify.go:137, verify_asset.go:149), so the intended design is one algorithm value threaded through lookup, verification and output. The repo makes that clear once you read the call sites.", + "verification": "pkg/cmd/release/verify/verify_test.go and verify-asset/verify_asset_test.go cover success and error paths, but with fixed 40-char SHAs (1234567890abcdef...). They do not assert the 'sha1:' label, and the shared.MockVerifier (pkg/cmd/release/shared/attestation.go:124) does not check digest algorithm. Output goes to iostreams (ios.Out), so SHA-256 cases are testable without external services. Existing tests would miss a wrong prefix.", + "blast_radius": "Public user-facing CLI commands (pkg/cmd/release/verify, verify-asset), documented in the release verify command help. A bug causes false verification failures or wrong labels, fixable by a revert with no stored-data harm. No SECURITY.md exists in the repo root." + }, + "confidence": "medium", + "summary": "The fix derives the digest algorithm (sha1 vs sha256) from the ref the API returns, since FetchRefSHA returns only the SHA string, and threads that single value through the attestation lookup, the sigstore digest policy and the printed output in both release verify commands. Tests must cover both 40- and 64-character refs and assert the printed prefix, which no existing test does, and SHA-256 attestation handling is inferred rather than verified against a real fixture." + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:11:19.010463Z", + "metrics": { + "input_tokens": 20, + "output_tokens": 6511, + "cache_read_tokens": 223352, + "cache_creation_tokens": 36994, + "num_turns": 9, + "latency_seconds": 49.8, + "total_cost_usd": 0.011126797 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The required behaviour is stated in the task (use the API's sshUrl when protocol is ssh). The API field already exists (api/queries_repo.go:42 `SSHURL`, api/query_builder.go:488 `sshUrl`), but the GitHubRepo fragment (api/queries_repo.go:362-374, RepositoryInfo at :376-386) does not request it, so the contract is in the GraphQL schema the code must follow. Fork upstream behaviour is also implied by clone.go:205-206 using canonicalRepo.Parent, which needs parent.sshUrl fetched too.", + "silent_invariant": "A wrong URL fails visibly (SSH auth error, or git clone of an empty string). The risk is confined to edge cases: an empty SSHURL falling through to the tenant-username format in ghrepo.FormatRemoteURL (internal/ghrepo/repo.go:95-103), or the parent's SSHURL being empty when the parent fragment lacks the field. No auth or data-integrity logic is changed.", + "coordinated_surfaces": "Changes must agree inside the repo command plus the API layer: the GraphQL fragment in api/queries_repo.go (repo and parent via `...repo`, lines 362-381), the URL choice at pkg/cmd/repo/clone/clone.go:178 and :205-206, and test fixtures/expectations in pkg/cmd/repo/clone/clone_test.go and internal/ghrepo/repo_test.go:224-276. No CLI docs or deployment config are involved.", + "ambiguity": "A narrow fix looks complete but is wrong: changing only clone.go:178 to use canonicalRepo.SSHURL yields an empty URL because GitHubRepo's fragment (api/queries_repo.go:362-374) never requests sshUrl. The broader alternative, changing ghrepo.FormatRemoteURL (internal/ghrepo/repo.go:95), affects ~8 callers (fork/fork.go:334,361,379; pr/checkout/checkout.go:138,299; pr/create/create.go:1292; repo/create/create.go:421,522,646; repo/rename/rename.go:191; extension/manager.go:414), and TestFormatRemoteURL pins the tenant-username format (repo_test.go:224-276). Choosing scope and fallback is a design decision the task does not settle.", + "verification": "Test_RepoClone (pkg/cmd/repo/clone/clone_test.go:152-245) runs offline via httpmock and run.Stub, so the path is testable. Existing SSH case (line 189-191) only covers git@github.com, and no cloneRun test covers the tenant SSH URL or the fork upstream remote. The RepositoryInfo fixtures (clone_test.go:219-227, 251, 291, 337, 366, 403) lack sshUrl, so they need updating. Some tests exist but would miss the new behaviour unless added.", + "blast_radius": "Affects clone URLs for SSH-configured users across github.com and GHE.com tenants (clone.go:163, 205). Users would see failures or wrong remotes, but a revert restores prior behaviour with no stored data, credentials, or auth-logic change. HTTPS paths are unchanged per the task." + }, + "confidence": "high", + "summary": "The task needs `gh repo clone` (and the fork upstream remote in clone.go:205-206) to take the SSH URL from the API's `sshUrl` field instead of ghrepo.FormatRemoteURL's tenant-username format, which first means adding `sshUrl` to the GitHubRepo fragment for both repo and parent in api/queries_repo.go. Scope is the main risk: a clone.go-only change gets an empty URL, and a FormatRemoteURL-wide change touches about eight callers and the pinned tenant test in internal/ghrepo/repo_test.go." + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:15:20.065043Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5364, + "cache_read_tokens": 88707, + "cache_creation_tokens": 29522, + "num_turns": 5, + "latency_seconds": 33.0, + "total_cost_usd": 0.007986571999999999 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task names the field the fix must use: the API-reported SSH URL. The repo already models it as `sshUrl` in `api/query_builder.go` `RepositoryFields` and `SSHURL` in `api/queries_repo.go:42`, and `pkg/cmd/repo/view/view_test.go` uses it. The GraphQL field's behaviour on tenants is defined by GitHub's API, outside this repo, but the in-repo convention is clear.", + "silent_invariant": "`internal/ghrepo/repo.go:95-103` `FormatRemoteURL` builds the SSH URL from the tenant name. A wrong URL on a CA tenant fails visibly with 'Permission denied (publickey)', so the mistake shows in normal use. Only edge cases (empty `SSHURL`, parent without `sshUrl`) would pass unnoticed.", + "coordinated_surfaces": "Changes sit in `pkg/cmd/repo/clone/clone.go` (canonical clone at line 178, upstream remote at line 206), the shared `repo RepositoryInfo` GraphQL fragment in `api/queries_repo.go:360-390` (also reused for `parent`), and tests. Wiki URL derivation at clone.go:185 comes from the canonical URL, so it follows automatically. Other `FormatRemoteURL` callers (fork, pr, repo create/rename, extension) are not required by the task.", + "ambiguity": "Tempting narrow fix: change `FormatRemoteURL` (`internal/ghrepo/repo.go:95`) or the tenant logic. It has 10+ callers in `pkg/cmd/repo/fork/fork.go:334-379`, `pkg/cmd/pr/checkout/checkout.go:138,299`, `pkg/cmd/repo/create/create.go`, and `pkg/cmd/extension/manager.go:414`, and none of them has an API SSH URL in scope. The real change is to fetch `sshUrl` in `GitHubRepo` (which `api/queries_repo.go` doesn't currently do) and use it in clone.go for both origin and parent. Passing the SSH URL as the argument does not help, since clone.go:178 ignores the input URL. The task also says the fork's upstream must be correct, which requires the parent's `sshUrl` too.", + "verification": "`pkg/cmd/repo/clone/clone_test.go` covers HTTPS/SSH/fork/upstream-name/no-upstream via `httpmock.GraphQL(\\`query RepositoryInfo\\b\\`)` and `run.Stub` git expectations (lines 152-425). `internal/ghrepo/repo_test.go:224` `TestFormatRemoteURL` covers the tenant SSH format. No existing test covers the SSH+tenant clone path or an API-supplied SSH URL, so a wrong change there would not be caught. Everything is testable offline with the existing mocks.", + "blast_radius": "User-facing command on GHE.com tenants (`pkg/cmd/repo/clone/clone.go`). The clone target comes from API data, but the failure mode is a failed clone, and reverting the commit fully undoes it. No credentials, stored data, or authorisation logic is touched. `SECURITY.md` has no clone-specific content." + }, + "confidence": "medium", + "summary": "The fix must fetch `sshUrl` in the shared `GitHubRepo` GraphQL fragment (which also covers `parent`) and use it for both the origin clone and the fork upstream in `pkg/cmd/repo/clone/clone.go`, with HTTPS unchanged. Changing `ghrepo.FormatRemoteURL` globally would touch ~10 other commands that have no API SSH URL, so it is the wrong place. Confidence is medium because the tenant-side GraphQL `sshUrl` behaviour is inferred from the task, not observed in the repo." + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:18:50.964655Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 7785, + "cache_read_tokens": 117569, + "cache_creation_tokens": 36396, + "num_turns": 6, + "latency_seconds": 46.1, + "total_cost_usd": 0.010580999 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says to use 'the SSH URL the API reports', but the codebase exposes it as GraphQL `sshUrl` (api/queries_repo.go:42, api/query_builder.go:488), while the reproduction uses REST `ssh_url`. Equivalence of the two fields is not in the repo ([INFERENCE]). The fallback when the API returns an empty URL is also unstated; the existing `SSH URL` case in clone_test.go:189 mocks no sshUrl and expects the `git@github.com:` form. Both follow from nearby code rather than an external spec.", + "silent_invariant": "A wrong URL fails loudly with 'Permission denied (publickey)'. The silent risk is a fallback to FormatRemoteURL when sshUrl is empty, which would keep the bug while looking fine in tests that mock no sshUrl. .github/SECURITY.md only covers vulnerability reporting, so no security invariant is involved.", + "coordinated_surfaces": "Changes sit in one area: api/queries_repo.go GitHubRepo fragment (`repo` fragment is shared by the canonical repo and `parent`, lines 361-381; `sshUrl` is not requested today), and pkg/cmd/repo/clone/clone.go call sites at lines 178 and 206. The shared FormatRemoteURL (internal/ghrepo/repo.go:95) has many other callers (fork, pr checkout, create, rename, extension) that the task does not mention, so the change should stay clone-local.", + "ambiguity": "The obvious change, using canonicalRepo.SSHURL, silently yields an empty string because GitHubRepo never requests sshUrl (api/queries_repo.go:361-381). Changing FormatRemoteURL globally would alter other commands and conflict with the existing tenant pin in internal/ghrepo/repo_test.go:258-264. The clone-local fix with the query change is the clear intended answer.", + "verification": "pkg/cmd/repo/clone/clone_test.go Test_RepoClone uses httpmock and run.Stub to assert the exact git command offline, and hasParent tests cover the upstream path. The GraphQL mocks omit sshUrl, and the only SSH case uses git@github.com, so the tenant SSH bug is not covered today. New offline tests can cover it, so this is not a 2.", + "blast_radius": "Affects every `gh repo clone` over SSH, including github.com users, which users would notice. A revert of the commit fixes it with no data or security impact. Nothing in .github/SECURITY.md or the repo map marks this path as security-critical." + }, + "confidence": "medium", + "summary": "Clone must take the SSH URL from the API (`sshUrl`, which needs adding to the GitHubRepo GraphQL query for both the repo and its parent) instead of rebuilding it from the tenant name in clone.go, with HTTPS unchanged and a defined fallback for empty values. Tests in clone_test.go with httpmock and run.Stub should cover tenant SSH for both the origin and upstream remotes. Confidence is medium because the GraphQL sshUrl vs REST ssh_url equivalence and the fallback behaviour are inferred, not verified." + }, + { + "task": "repo-create-reject-empty-name", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:10:13.363301Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5260, + "cache_read_tokens": 90561, + "cache_creation_tokens": 29680, + "num_turns": 5, + "latency_seconds": 31.5, + "total_cost_usd": 0.007971491 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task says only that gh must 'tell the user right away that the repository name is empty' and that valid names keep working; it gives no message text or re-prompt vs. error choice. Repo conventions fill that in: the non-interactive path already returns cmdutil.FlagErrorf(\"name argument required to create new remote repository\") (pkg/cmd/repo/create/create.go:158), and splitNameAndOwner wraps parse errors as \"argument error: %w\" (create.go:968-977).", + "silent_invariant": "Not security or data-integrity code. A wrong change shows only in edge cases: whitespace-only names, the owner-prefixed path where interactiveRepoInfo builds \"owner/\" (create.go:901-903) and bypasses the splitNameAndOwner check, and the no-orgs path where an empty opts.Name goes to ghrepo.NewWithHost(\"\", \"\", host) (create.go:347-348).", + "coordinated_surfaces": "Change is confined to pkg/cmd/repo/create: the prompt helper interactiveRepoNameAndOwner (create.go:928-966) and, if the check moves, createFromScratch (create.go:295-349). Non-interactive path (create.go:158) and tests in create_test.go must agree on the wording. No docs or other services are involved.", + "ambiguity": "The symptom is reported at confirmation (create.go:331-338) and the error at create.go:341-345. A guard placed there would still make the user answer every prompt first, so the check belongs in the name prompt path. The 'owner/' case already fails early in splitNameAndOwner, so the repository points to the same validation location. Re-prompt vs. immediate error is still a choice.", + "verification": "pkg/cmd/repo/create/create_test.go has table tests for interactive create-from-scratch (lines ~200-340, 'prompted owner', 'cancel before submit') driven by prompter.PrompterMock InputFunc/ConfirmFunc/SelectFunc and httpmock. No case covers a blank 'Repository name', so existing tests would not catch the bug. The behaviour is testable without network or credentials. internal/ghrepo/repo_test.go:147-150 already pins the 'a/' blank-value error.", + "blast_radius": "User-facing CLI command (`gh repo create`), so users would notice a wrong message or behaviour. Nothing touches auth, secrets or stored data, and a revert fully undoes it (no repository is created by an empty-name check)." + }, + "confidence": "medium", + "summary": "The fix is a validation in pkg/cmd/repo/create: reject a blank 'Repository name' at the prompt in interactiveRepoNameAndOwner, before owner, description and confirmation are asked, while keeping valid and owner-prefixed names working. It needs table tests in create_test.go for the blank-name case (including the org-owner path that yields 'owner/'), since no existing test covers it. The exact message and whether to re-prompt or fail immediately are not specified by the task; the repository's existing error conventions point to the choice." + }, + { + "task": "repo-create-reject-empty-name", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:13:15.444727Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4619, + "cache_read_tokens": 89138, + "cache_creation_tokens": 32567, + "num_turns": 5, + "latency_seconds": 31.9, + "total_cost_usd": 0.0080002505 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task does not give the error wording or whether to fail or re-prompt. Repo conventions fill this in: `pkg/cmd/capi/job.go` and `pkg/cmd/repo/create`-adjacent code use `\"... cannot be blank\"` with `strings.TrimSpace` (e.g. `pkg/cmd/discussion/create/create.go`, `pkg/cmd/issue/create` pattern), and `internal/ghrepo/repo_test.go` already rejects `a/` as blank-value.", + "silent_invariant": "`pkg/cmd/repo/create/create.go:928-937` (`interactiveRepoNameAndOwner`) and `:968-977` (`splitNameAndOwner`). A wrong check (e.g. validating only after `fmt.Sprintf(\"%s/%s\", owner, name)` at `:901-903`, or missing whitespace-only input) would look fine in normal use. Not security, money or persisted data.", + "coordinated_surfaces": "Change is confined to `pkg/cmd/repo/create/create.go` (prompt path, `interactiveRepoInfo` at `:896-917`). No API, docs or generated client involved; `.github/SECURITY.md` is only a disclosure policy.", + "ambiguity": "Task names the symptom (failure after the final confirm, `some-owner/` from `:905-906` plus `:606-611`). Obvious fix is a check immediately after `prompter.Input(\"Repository name\", ...)` at `create.go:929`. Trap: validating the composed `owner/name` string or the post-prompt path at `:604-614`, which is later than the task asks for.", + "verification": "`pkg/cmd/repo/create/create_test.go` `Test_createRun` (`:189-1086`) drives the interactive flow with `prompter.PrompterMock` and httpmock, so a test is cheap to add without network. But no existing case covers a blank repository name, so a wrong change to that path would currently go uncaught.", + "blast_radius": "User-facing interactive CLI prompt in `create.go`; users would notice a wrong message or reprompt, and a revert fully restores prior behaviour. No security, money or stored data (`.github/SECURITY.md` scope)." + }, + "confidence": "high", + "summary": "The fix belongs in `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`: reject a blank name right after the `Repository name` prompt, before owner, description, visibility and the rest are asked, while leaving valid and OWNER/REPO input on the existing `splitNameAndOwner` path. Tests go in the `Test_createRun` table in `create_test.go`, which already has a mocked interactive harness, and the exact error wording should follow the repo's `cannot be blank` convention." + }, + { + "task": "repo-create-reject-empty-name", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:17:24.247553Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4681, + "cache_read_tokens": 93137, + "cache_creation_tokens": 30685, + "num_turns": 5, + "latency_seconds": 27.6, + "total_cost_usd": 0.0078195645 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/repo/create/create.go:928-966 (interactiveRepoNameAndOwner) and :977 splitNameAndOwner wrap parser errors as \"argument error: %w\", and the task's expected message follows that existing pattern. The task gives no exact wording for the empty-name error, so the implementer must copy the nearby convention, not invent one.", + "silent_invariant": "create.go:929-937: an empty name reaches the owner prompt and the confirm prompt before failing in ghrepo.FromFullName (create.go:341-346). A wrong change is visible as a command error or a wrong prompt sequence, not a silent data or security effect.", + "coordinated_surfaces": "The fix lives in create.go's interactive path (interactiveRepoNameAndOwner, called from interactiveRepoInfo at :896 and used by createFromScratch :310, createFromTemplate :451, createFromLocal :595). Those callers share one helper, so the check sits in one component. Non-interactive `opts.Name` handling at create.go:341-349 is a second place that accepts an empty name, but nothing else in the repo depends on it.", + "ambiguity": "The task describes a symptom (blank name reaches the end of the flow) and asks for an early error. The obvious fix is an empty check right after prompter.Input at create.go:929. The narrower reading (only test for \"\") misses whitespace-only input, and the non-interactive path at create.go:341-349 is untouched by the report. The repo's own parser (internal/ghrepo/repo_test.go:139-149) shows the error convention, which resolves most of the ambiguity.", + "verification": "pkg/cmd/repo/create/create_test.go has table tests with a PrompterMock (promptStubs at :195, :206-226, :225 \"Repository name\" case) covering the interactive scratch flow, so a regression in the happy path would be caught. No existing case covers an empty \"Repository name\" answer, so the reported bug is untested today. An empty-name test case fits the existing table, so it is easy to add.", + "blast_radius": "gh is a public user-facing CLI. The defect wastes user input and creates nothing remote, since the failing call is FromFullName, which runs before the API mutation at create.go:351+. A revert fully undoes any mistake, and no security, auth or stored-data path is involved." + }, + "confidence": "medium", + "summary": "The change needs an empty-name check in interactiveRepoNameAndOwner (create.go:929) that fails right after the \"Repository name\" prompt, using the existing \"argument error\" convention, with a new empty-name case in create_test.go's interactive table. Whether the non-interactive path and whitespace-only names also need rejecting is not stated in the task and was not confirmed against the repo's intent." + }, + { + "task": "repo-set-default-unset-without-repo", + "attempt": 1, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:10:13.271113Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 4481, + "cache_read_tokens": 67857, + "cache_creation_tokens": 29226, + "num_turns": 4, + "latency_seconds": 27.6, + "total_cost_usd": 0.007230652000000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The no-argument, non-TTY `--unset` and `--view` behaviour is stated in the task. The repo fixes the convention: `--view` already skips the 'repository required' check (pkg/cmd/repo/setdefault/setdefault.go:108, `!opts.ViewMode && !opts.IO.CanPrompt() && opts.Repo == nil`). What `--unset ` should do is left open. The task says the argument 'makes no sense' but does not say whether to error or ignore it. Repo precedent for rejecting conflicting args/flags exists (cmdutil.MutuallyExclusive in pkg/cmd/repo/create, read-file, list), but nothing ties it to this command.", + "silent_invariant": "The wrong target is unset silently: with `upstream` as default, `--unset origin` succeeds and runs `git config --unset remote.upstream.gh-resolved` (setDefaultRun, UnsetMode branch). The write is local git config only, and the failure needs a specific input (an argument combined with --unset), so ordinary checks would miss it. No security, auth or money invariant is involved.", + "coordinated_surfaces": "The change sits in one component: the RunE guard in pkg/cmd/repo/setdefault/setdefault.go plus its tests in setdefault_test.go. Optionally, the acceptance script acceptance/testdata/repo/repo-set-default.txtar may get a non-interactive `--unset` case. The Example and help text in setdefault.go do not mention `--unset`, so the docs side is light.", + "ambiguity": "The tempting fix is to add `!opts.UnsetMode` to the guard at setdefault.go:108. That satisfies the stated no-arg expectations but leaves `gh repo set-default --unset origin` silently unsetting the wrong remote, which the task explicitly reports as a bug. The correct fix also has to decide what happens to an argument given with --unset (reject it or ignore it), and the task leaves that open.", + "verification": "TestNewCmdSetDefault (setdefault_test.go:20-162) hard-codes SetStdinTTY(true) and never reaches the non-TTY error branch, so the 'repository required' path has no test. TestDefaultRun (lines 164+) covers unset and view in TTY mode only, with no non-TTY or argument-with-unset case. Both suites are offline, and a table case can add the missing coverage, so the gap is partial rather than total.", + "blast_radius": "This is a user-facing CLI command that edits local git config (`remote..gh-resolved`). A wrong unset costs a re-run of `gh repo set-default`, and the command touches no credentials, remote data or security boundary (.github/SECURITY.md covers vulnerability reporting, not this path)." + }, + "confidence": "medium", + "summary": "The fix is a small guard change in pkg/cmd/repo/setdefault/setdefault.go so `--unset` skips the 'repository required' check the way `--view` already does. The real question is what `--unset ` should do, since the task reports it silently unsetting the wrong remote and does not say whether to reject or ignore the argument, so tests must cover non-TTY `--unset`, `--view`, the plain-no-args error and the argument-with-unset decision." + }, + { + "task": "repo-set-default-unset-without-repo", + "attempt": 2, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:14:11.102181Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3846, + "cache_read_tokens": 68167, + "cache_creation_tokens": 28918, + "num_turns": 4, + "latency_seconds": 24.4, + "total_cost_usd": 0.0068424620000000005 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "pkg/cmd/repo/setdefault/setdefault.go: the RunE guard `if !opts.ViewMode && !opts.IO.CanPrompt() && opts.Repo == nil` is the existing convention that `--view` skips the repository-required check. The task's 'behave the same way' ties `--unset` to that existing path, so the expected behaviour is recoverable from nearby code. The task does not say whether `--unset ` should now error or be ignored.", + "silent_invariant": "setdefault.go: UnsetMode runs `git config --unset remote..gh-resolved` for the resolved default remote. A wrong guard change (e.g. dropping the check entirely) would send plain `set-default` into the API/prompt flow, and a mishandled argument could unset the wrong default. Both show up only in non-TTY or argument edge cases; nothing is security-sensitive or persisted beyond one git config key.", + "coordinated_surfaces": "The fix is one guard condition plus argument handling in one RunE in setdefault.go. Tests are in pkg/cmd/repo/setdefault/setdefault_test.go. The Example/Long text in setdefault.go does not mention --unset, and acceptance/testdata/repo/repo-set-default.txtar covers only --view and set, so no second layer must agree.", + "ambiguity": "The tempting fix is adding `&& !opts.UnsetMode` to the guard. That makes the stated non-interactive case pass but leaves the reported `gh repo set-default --unset origin` behaviour (argument silently ignored, unsetting `upstream`) unresolved. The task says 'a repository argument makes no sense for it' without choosing between rejecting the argument and ignoring it. The repo's --view path ignores a parsed argument, which is not the same as the reported bug.", + "verification": "TestNewCmdSetDefault and TestDefaultRun already use iostreams.Test with explicit TTY setters and a git CommandStubber, so new cases are cheap. However every existing case sets SetStdinTTY(true), so the non-TTY 'repository required' error path has no test today. The unset-without-argument and view-without-argument non-TTY cases are also missing, so a wrong guard would ship unnoticed without new tests.", + "blast_radius": "User-facing CLI command (docs/command-development.md scope). Wrong behaviour would unset or fail to unset the local `remote..gh-resolved` git config key, which a revert or re-running `set-default` fixes. Nothing touches credentials, auth, or remote-side data." + }, + "confidence": "medium", + "summary": "The change is a small, single-file guard fix in pkg/cmd/repo/setdefault/setdefault.go that exempts `--unset` from the non-interactive repository-required check, mirroring the existing `--view` exemption. The real design question is what to do with a repository argument combined with `--unset`, which the task reports as silently ignored but does not resolve, and the required tests must cover the non-TTY path that the existing suite never exercises." + }, + { + "task": "repo-set-default-unset-without-repo", + "attempt": 3, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:17:51.219027Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 4250, + "cache_read_tokens": 120973, + "cache_creation_tokens": 31303, + "num_turns": 6, + "latency_seconds": 28.4, + "total_cost_usd": 0.007973905500000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task text states the success case and the preserved 'repository required' error. The --view-without-argument behaviour it points to is defined in setdefault.go (`if !opts.ViewMode && !opts.IO.CanPrompt() && opts.Repo == nil` in RunE), and the existing error path is `cmdutil.FlagErrorf`. Whether `--unset ` should now be rejected, or just accepted as --view does today, is not stated in the expected-behaviour section.", + "silent_invariant": "setdefault.go UnsetMode branch calls `GitClient.UnsetRemoteResolution` on `currentDefaultRepo`, and ignores `opts.Repo`. Passing an argument with --unset silently unsets the wrong remote (task: `--unset origin` unsets `upstream`). This is a local git-config write (`remote..gh-resolved`), not security or stored user data, and it is reversible with `set-default`.", + "coordinated_surfaces": "Change is confined to pkg/cmd/repo/setdefault/setdefault.go (RunE check plus any arg-handling change for --unset). Tests live in setdefault_test.go. The Example block and Long text do not mention --unset. acceptance/testdata/repo/repo-set-default.txtar covers only --view and set, so no acceptance change is required. One package, two or three spots that must agree.", + "ambiguity": "The obvious fix is relaxing the condition to `!opts.ViewMode && !opts.UnsetMode && ...`. That makes the reported error disappear but leaves `--unset origin` silently unsetting `upstream`, which the task explicitly calls out. The task says a repository argument 'makes no sense' with --unset, yet the expected section does not say whether to reject it. The narrow fix looks complete and still leaves the flagged problem; the intended design has to be chosen.", + "verification": "setdefault_test.go has a constructor seam (`NewCmdSetDefault` with injected runF, TestNewCmdSetDefault) and a run seam (`setDefaultRun`, TestDefaultRun) with unset-mode cases (tty only) and view-mode cases in both TTY and non-TTY. No case covers non-TTY `--unset` without an argument, and no case asserts the 'repository required' error for a plain non-TTY invocation. The harness can test these with iostreams.Test and no network, so the gap is closable locally.", + "blast_radius": "Shipped user-facing CLI command that edits per-remote git config (`git config --unset remote..gh-resolved`). A wrong change shows up as a wrong default repo or a failing script, and a revert or `gh repo set-default` restores it. No credentials, auth, or stored user records are involved." + }, + "confidence": "medium", + "summary": "The fix is a small RunE change in pkg/cmd/repo/setdefault/setdefault.go so non-interactive `--unset` skips the 'repository required' check, as `--view` already does, while plain `set-default` keeps the error. The real decision is what to do with a repository argument passed alongside `--unset`, which currently unsets the default silently. Tests go in setdefault_test.go for non-TTY `--unset`, non-TTY `--view` without an argument, and the retained error." + }, + { + "task": "repo-sync-checked-out-worktree", + "attempt": 1, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:12:25.738693Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 6726, + "cache_read_tokens": 132080, + "cache_creation_tokens": 37069, + "num_turns": 6, + "latency_seconds": 40.4, + "total_cost_usd": 0.010250707500000001 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task says 'either updates it safely or stops with a clear message that names the worktree'. The message format is not fixed, so the acceptable wording and the update-vs-refuse behaviour follow from nearby conventions: the dirty-tree tip in sync.go:245-246 ('refusing to sync due to uncommitted/untracked local changes\\ntip: ...') and the worktree-path errors in pkg/cmd/pr/checkout/checkout.go:331-369 (resolveWorktreeTarget). Git's own worktree semantics (a branch checked out elsewhere, per-worktree index) are the external part, but they are not a contract the task cites.", + "silent_invariant": "sync.go:244-261: the dirty check (IsDirty) runs only when currentBranch == branch, i.e. only for the current worktree. The else path calls UpdateBranch (git.go:26-33, `git update-ref refs/heads/`), which moves the ref under another worktree's index and leaves its files and index untouched, so the symptom is phantom staged changes with no error. A fix that resets or checks out the other worktree without first checking its dirty state could silently destroy uncommitted user work there. That is a data-integrity invariant, and breaking it is silent.", + "coordinated_surfaces": "The change stays inside pkg/cmd/repo/sync: sync.go (executeLocalRepoSync), git.go (gitClient interface and gitExecuter, which would need a worktree-listing method), and mocks.go (mockGitClient must gain the same method). The tests are sync_test.go. Possibly a git/ client helper. Command help text (sync.go Long/Example) would not change for a stop-with-message fix. Two or three places in one component.", + "ambiguity": "The task describes a symptom (phantom staged changes) and names a suspected cause (executeLocalRepoSync compares only the current worktree). It offers two outcomes, update safely or refuse with a message, and leaves the choice open. A refuse-only check passes the stated done criteria but adds a behaviour the issue does not ask for. A naive 'update the other worktree' fix can clobber dirty files (see silent_invariant). The edge cases (detached-HEAD worktrees, locked or prunable worktrees, bare repos, the current worktree itself) need deliberate decisions, and the repo does not settle them.", + "verification": "sync_test.go Test_SyncRun (lines 103-576) covers only the single-worktree local flow through mockGitClient (mocks.go), with stubs such as CurrentBranch/IsDirty/UpdateBranch. No case covers another worktree holding the branch, so the existing suite would not catch the bug or a regression in it. Reproducing phantom staged changes needs real git (`git worktree add`, `git status`). The mock-based pattern cannot show that, so new coverage needs either new mock methods that only prove call wiring, or an acceptance script (acceptance/testdata/repo/repo-fork-sync.txtar exists) or a real-git test. The path is testable locally without external services.", + "blast_radius": "gh is a user-facing CLI (AGENTS.md: 'Architecture'; SECURITY.md is the disclosure policy and does not change the score). The affected data is the user's uncommitted work in another worktree. A wrong fix that resets or overwrites that worktree could destroy it, and a release revert does not restore lost files. The current bug also leaves a user's worktree in a misleading staged state." + }, + "confidence": "medium", + "summary": "The task needs gh's local sync path (pkg/cmd/repo/sync/sync.go executeLocalRepoSync, plus the gitClient interface and mocks) to detect when the target branch is checked out in another worktree, then either update that worktree safely (only if it is clean) or stop with a message that names it, while keeping the single-worktree and unchecked-out branch cases working. Nothing in the repo covers the other-worktree case, so the work needs new real-git or acceptance coverage on top of the mock-based suite. The git worktree behaviour itself is inferred from the issue text and the existing git helpers in pkg/cmd/pr/checkout, not from a cited spec, which is why confidence is medium." + }, + { + "task": "repo-sync-checked-out-worktree", + "attempt": 2, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:16:13.018652Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 7170, + "cache_read_tokens": 125288, + "cache_creation_tokens": 41594, + "num_turns": 6, + "latency_seconds": 41.3, + "total_cost_usd": 0.011042383 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task's done criteria are stated: never leave the other worktree with phantom staged changes, and either update it safely or stop with a message naming the worktree. Conventions the fix must follow are in the repo: pkg/cmd/pr/checkout/checkout.go:331-369 (resolveWorktreeTarget, revParseFacts) and :414-425 (syncBranchCmds, which runs `-C merge --ff-only` / `reset --hard`). Nothing outside the repo is needed.", + "silent_invariant": "pkg/cmd/repo/sync/git.go:26-33 UpdateBranch runs `git update-ref refs/heads/`, which moves the ref without touching the other worktree's index or files, so the bug is silent. The 'update safely' option means running merge or reset inside another worktree. A fix that skips that worktree's IsDirty check, or uses reset --hard under --force, would silently destroy uncommitted user work. The dirty check in sync.go:245-249 is the invariant that must carry over.", + "coordinated_surfaces": "All changes stay in one package: pkg/cmd/repo/sync/sync.go (executeLocalRepoSync, lines 221-272), git.go (gitClient interface plus a new worktree query), mocks.go (hand-written mockGitClient, which must gain the new method), and sync_test.go. No API, schema, or deployment surface is involved. The repo's AGENTS.md has no docs requirement for this.", + "ambiguity": "The task explicitly allows two outcomes (update the other worktree safely, or refuse with a message naming it), so the target is clear. Narrow traps exist: a check that only compares the current worktree is what sync.go:244 already does. Refusal-only also satisfies the stated criteria. Path/worktree detection is fiddly (see resolveWorktreeTarget's comments), but the repo already shows how to handle it.", + "verification": "sync_test.go:103-333 has table cases for local sync (current branch, non-current UpdateBranch, create, dirty, diverging), so branch-logic regressions would be caught. Nothing covers worktree detection. mockGitClient stubs would not exercise real `git worktree list` output or the -C command strings, so new tests need pr checkout-style command stubs (checkout_test.go:312-340 uses cs.Register) to pin the git commands.", + "blast_radius": "gh repo sync is a user-facing command that writes to local working trees. A wrong 'update other worktree' path can discard uncommitted changes there (the same risk syncBranchCmds' reset --hard path carries under --force). That loss is hard to undo. The rubric's data-loss criterion applies even though nothing here is security-sensitive (.github/SECURITY.md covers reporting only)." + }, + "confidence": "medium", + "summary": "The fix belongs in pkg/cmd/repo/sync: executeLocalRepoSync must detect when the target branch is checked out in another worktree, then either refuse with a message naming that worktree or update it with a dirty-state check. That needs new gitClient methods, a hand-updated mock, and command-stub tests, because the existing tests cover only the current-worktree paths. The main risk is that the update path can silently destroy uncommitted work in the other worktree." + }, + { + "task": "repo-sync-checked-out-worktree", + "attempt": 3, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:19:46.820701Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 4930, + "cache_read_tokens": 94693, + "cache_creation_tokens": 31230, + "num_turns": 5, + "latency_seconds": 30.9, + "total_cost_usd": 0.008048568 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The issue (task text) states the two acceptable outcomes: update the other worktree safely, or stop with a message naming the worktree. The rest follows from nearby code. pkg/cmd/repo/sync/sync.go:259-269 shows the existing else-branch behaviour (UpdateBranch via update-ref, or CreateBranch) and the messages in sync.go:246 and :152 set the error-message convention. pkg/cmd/pr/checkout/checkout.go:339-369 (resolveWorktreeTarget) shows how this repo already interprets worktrees via git rev-parse, which fixes the conventions to follow.", + "silent_invariant": "The failure is silent: the bug is that update-ref moves refs/heads/main under another worktree whose index and files are not updated, so git status shows phantom staged changes (task text). Any fix that updates that worktree must also check its dirty state. The current-worktree path already refuses on dirty changes (sync.go:245-251, which also runs ResetHard on FETCH_HEAD), so a careless fix there could discard uncommitted work in another worktree, and the user would not be warned.", + "coordinated_surfaces": "The change stays in one package: sync.go (executeLocalRepoSync), git.go (new gitClient method wrapping a worktree query), mocks.go (the testify mock must gain the new method, since every existing Test_SyncRun case stubs the gitClient interface), and sync_test.go. The command help text may need a note, and docs or changelog updates are possible. No API, other service or deployment surface is involved.", + "ambiguity": "The task allows two designs: refuse with a message, or update the other worktree. Refusing alone looks complete and passes the stated done criteria, but it leaves the update-in-other-worktree path unaddressed. The update path needs the target worktree's dirty check and a fast-forward run in that worktree's context. Detection also has traps: detached, bare and prunable worktrees, and the case where the branch is checked out in the current worktree (already handled at sync.go:244). The task text does not choose between the two designs, so the implementer must make the call.", + "verification": "sync_test.go Test_SyncRun (lines 103-590) is a table test with mocked git calls and covers the existing local branch paths (for example 'sync local repo with parent and local changes', 'existing branch, non-current'). Those tests would catch a regression in the current-branch and non-current paths. The new worktree-detection parsing and the other-worktree dirty check need new cases. Real-git worktree behaviour is not covered by sync tests (no real-git test in pkg/cmd/repo/sync), though git/client_test.go shows real-git-style tests exist for git.Client, so a real-repo smoke test is feasible.", + "blast_radius": "gh is a user-facing CLI (AGENTS.md describes it as the GitHub CLI shipped to users). A wrong change to sync.go could reset or merge a worktree that holds uncommitted user work, which is data loss that a revert does not undo. The existing dirty-tree guard (sync.go:245-249) shows the project treats local-change loss as a serious risk." + }, + "confidence": "medium", + "summary": "Syncing a branch checked out in another worktree must not leave that worktree's index and files stale, and the fix has to either update it safely (which requires checking that worktree's dirty state) or refuse with a message that names the worktree. The change sits in pkg/cmd/repo/sync (sync.go, git.go, mocks.go, sync_test.go) and needs new worktree-detection and dirty-check test cases, since the existing mock-based suite covers only the current-worktree and plain-ref paths." + }, + { + "task": "secret-set-app-agents", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:12:12.304850Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 6713, + "cache_read_tokens": 106293, + "cache_creation_tokens": 49886, + "num_turns": 5, + "latency_seconds": 40.4, + "total_cost_usd": 0.011722018 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task says GitHub's REST API documents the agents secret endpoints, so the wire contract lives outside the repo. The repo shows the risk: pkg/cmd/secret/set/http.go:21-26 uses string repo IDs in DependabotSecretPayload for org secrets, while SecretPayload (http.go:14-19) uses int64 IDs. Which shape agents org secrets need is set by the external API, and the Dependabot special case in putOrgSecret (http.go:73-87) shows a sibling app diverging. The repo does not settle it.", + "silent_invariant": "Secrets are encrypted client-side with the app-specific public key (set/http.go:42-43, getOrgPublicKey) and routed by app string in set/http.go:71 and :123 and list/list.go:287/251. A misrouted write or read still returns success, and the CLI prints 'Set Agents secret X' (set/set.go:319). Secret placement across Actions, Dependabot, Codespaces and Agents is an access-scope boundary, so the failure is quiet.", + "coordinated_surfaces": "The change stays inside pkg/cmd/secret: App constants and GetSecretApp/IsSupportedSecretEntity in shared/shared.go:21-94, the enum flags in set/set.go:198, list/list.go:101 and delete/delete.go:84, the Long help text in set.go:66-68, list.go:63-65 and delete.go:43-45, and the org payload branch in set/http.go. No API layer, deployment config or generated client is involved. docs/ has no app-value listing (grep for dependabot in docs returned nothing).", + "ambiguity": "The repo's pattern is clear: the switch cases in shared.go:65-94 and the parallel enum lists in set/list/delete. The task limits support to repository and organization (task text), and shared/shared.go:83-94 already encodes entity support per app. The main open point is the payload shape, which is covered under hidden_contract. Adding 'agents' to the enum alone would fail at GetSecretApp with 'invalid application', which is visible.", + "verification": "Harnesses exist: httpmock path-exact tests in set/set_test.go (Test_setRun_repo around :398-466, Test_setRun_org around :508-590), list/list_test.go (org and Dependabot cases around :271-457), delete/delete_test.go (org path table around :417-435), and shared/shared_test.go table tests for GetSecretApp and IsSupportedSecretEntity (:85-93, :177-180). These catch routing mistakes without external services. Nothing validates the real agents payload or endpoints. Test_setRun_org only registers the PUT stub and does not assert the body. acceptance/testdata/secret/ needs live credentials.", + "blast_radius": "The commands write and delete live GitHub secrets for customers: set/set.go:361-367 PUT calls and delete/delete.go:143 DELETE call. A deleted or overwritten secret value cannot be restored by reverting the CLI. Misrouting to the Actions scope could overwrite a same-named Actions secret (set/set.go:359-368 switch)." + }, + "confidence": "medium", + "summary": "The task needs a new 'agents' App in pkg/cmd/secret/shared plus wiring in the set, list and delete enum, support checks, help text and org/repo endpoint paths, with the org payload shape (int vs string repo IDs) decided by GitHub's external REST contract. The work is contained to pkg/cmd/secret and testable with existing httpmock tables, but those tests cannot prove the real API shape, and mistakes write or delete live customer secrets irreversibly." + }, + { + "task": "secret-set-app-agents", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:16:03.249920Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 5314, + "cache_read_tokens": 102311, + "cache_creation_tokens": 43821, + "num_turns": 5, + "latency_seconds": 34.7, + "total_cost_usd": 0.0100748285 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says the agents endpoints are documented in GitHub's REST API, which is outside this repo. Nothing in pkg/cmd/secret defines them. The org write path depends on that contract: set/http.go special-cases dependabot with string `selected_repository_ids` (DependabotSecretPayload) while actions/codespaces use int64, so the agents payload shape must come from the external spec. shared/shared.go's IsSupportedSecretEntity also fixes which entities apply, which the task only partly states.", + "silent_invariant": "pkg/cmd/secret/set/http.go and shared/shared.go: values are encrypted with the per-app public key from `orgs/{org}/{app}/secrets/public-key` or `repos/.../secrets/public-key`. A wrong app mapping would mostly surface as API errors rather than a silent leak. Secrets are sensitive, but the repo map and SECURITY.md (absent) do not mark this path as security-critical. The residual risk is a secret landing in the wrong scope without an error.", + "coordinated_surfaces": "All changes sit in one package tree, pkg/cmd/secret: the `--app` StringEnumFlag lists in set/set.go:198, list/list.go:101 and delete/delete.go:84; the App constants and GetSecretApp/IsSupportedSecretEntity in shared/shared.go; the path builders in set/http.go, delete/delete.go and list (getOrgSecrets/getRepoSecrets); and the Long help in set/set.go and list/list.go, which list where secrets are available. Help text, Example text and the enum must agree, but that is one component.", + "ambiguity": "Adding 'agents' to the enum and GetSecretApp compiles and passes repo-level behaviour, but the org request body is a real decision. Dependabot's string-ID special case in set/http.go shows the org payload can differ per app, and the task only says 'behaving the way the existing applications do'. The narrow enum-only change looks complete yet can get the org shape or entity support wrong.", + "verification": "Strong local mock tests exist per app: set/set_test.go (Dependabot org/repo, Codespaces org), delete/delete_test.go:294-434 (dependabot/codespaces paths with httpmock), list/list_test.go:543 and shared/shared_test.go:85-91. They run offline with `go test`, so the task's required tests are feasible. But the mocks encode the same assumed agents contract, so they cannot confirm the real GitHub API shape.", + "blast_radius": "This is a user-facing gh CLI command (AGENTS.md: commands in pkg/cmd/). A bad change writes or deletes real GitHub secrets, which is data-sensitive. But errors are visible, and a revert of the CLI release fixes it without lasting harm, so it is not a 2." + }, + "confidence": "medium", + "summary": "Adding `agents` means extending the per-app enum, GetSecretApp and IsSupportedSecretEntity in pkg/cmd/secret/shared, the set/list/delete paths (including the org payload shape, which may differ as it does for dependabot), and the help text, then covering repo and org set/list/delete with mock tests in the existing style. The agents endpoint and org body shape live in GitHub's external REST docs, not the repo, so mocks can confirm the code path but not the live contract." + }, + { + "task": "secret-set-app-agents", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:19:40.300368Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 7738, + "cache_read_tokens": 202672, + "cache_creation_tokens": 52749, + "num_turns": 7, + "latency_seconds": 50.5, + "total_cost_usd": 0.0137400395 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task points to GitHub's REST agents secrets endpoints, which are not defined in this repo. Checked https://docs.github.com/en/rest/agents/secrets: the org PUT body takes selected_repository_ids as an array of integers, the same as Actions and unlike the Dependabot path, which sends strings (pkg/cmd/secret/set/http.go DependabotSecretPayload). Repo-side precedent (Dependabot string IDs) would mislead a copy-paste implementation.", + "silent_invariant": "Routing is an explicit switch on app (pkg/cmd/secret/shared/shared.go GetSecretApp, IsSupportedSecretEntity) and the path is built with fmt.Sprintf(\"repos/%s/%s/secrets/%s\"). A misroute (e.g. agents falling into the Actions branch) would store a credential in the wrong scope with no error, but the path is asserted in existing httpmock tests, so it is not a subtle invariant.", + "coordinated_surfaces": "Changes stay inside the gh secret tree: shared.go constants and switch cases; flag enum lists in set/set.go:198, list/list.go:101, delete/delete.go:84; Long help text in set.go:66-68, list.go:63-65, delete.go:43-45, secret.go:18; plus tests. No separate docs or generated clients in the repo (docs/ has no secret page). Acceptance testdata under acceptance/testdata/secret/ is live-API and optional.", + "ambiguity": "Scope is explicit in the task: set/list/delete at repo and org level. The obvious fix is adding the constant and including it in GetSecretApp and IsSupportedSecretEntity (Actions-like entity support). The only choice is the help-text wording, which the existing 'available to ...' phrasing constrains. The narrower trap (flag-only change, payload type) is covered under hidden_contract.", + "verification": "Mock-based coverage exists per app and entity: set_test.go Test_setRun_org asserts org path and payload; delete_test.go:293-311 and 421-434 assert repo and org DELETE paths; list_test.go:395-437 covers app routing. These catch path errors. They cannot confirm the real agents payload: a mock encodes whatever type the implementer assumes, and acceptance/testdata/secret needs live GitHub credentials.", + "blast_radius": "Public CLI command that writes encrypted secrets via authenticated API calls; the server enforces authorisation and the change adds no auth logic. A bad release can be reverted, and a misplaced secret can be removed with gh secret delete --app agents, which the change itself adds. Not a 2: nothing here persists data the change alone can corrupt beyond the user's own secret store." + }, + "confidence": "high", + "summary": "Adding agents means extending the shared app enum, GetSecretApp and IsSupportedSecretEntity, plus flag enums and help text in set, list and delete, and most of the path logic already follows the `{app}` pattern. The non-obvious requirement is that the documented agents org endpoint takes integer selected_repository_ids (like Actions), not the string IDs that the Dependabot special case uses, and the new tests must pin repo and org set/list/delete paths and payloads in the existing httpmock style." + }, + { + "task": "skill-install-nested-discovery", + "attempt": 1, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:13:11.732858Z", + "metrics": { + "input_tokens": 24, + "output_tokens": 9562, + "cache_read_tokens": 316038, + "cache_creation_tokens": 52434, + "num_turns": 11, + "latency_seconds": 65.1, + "total_cost_usd": 0.015947832999999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "internal/skills/discovery/discovery.go matchHiddenDirConventions (~line 470) and the 'too deeply nested hidden dir' case in discovery_test.go (`.claude/nested/skills/code-review/SKILL.md` -> nil) define the current hidden-dir shape. Gating semantics come from the --allow-hidden-dirs flag help text and filterHiddenDirSkills in pkg/cmd/skills/install/install.go. Nothing outside the repo is needed; the reader must infer the intended any-depth rule from the task plus these conventions.", + "silent_invariant": "--allow-hidden-dirs is an opt-in gate for dot-directory skills, enforced by Convention == hidden-dir/hidden-dir-namespaced via IsHiddenDirConvention and filterHiddenDirSkills. A change that routes hidden-segment paths into the standard 'skills' convention would bypass the gate. install_test.go (~lines 1380-1481, 2003-2061) asserts the exclusion and would catch this, so the wrong change is not silent.", + "coordinated_surfaces": "Matching lives in one package (internal/skills/discovery/discovery.go), used by remote discovery, DiscoverLocalSkillsWithOptions and preview. The user-facing 'no skills found' messages list the expected patterns (discovery.go ~line 520 and ~line 590) and the install help text (install.go ~lines 111-119) describes conventions; these must agree with the matcher, but they are text, not separate components.", + "ambiguity": "The reporter's 'two levels' theory is not the cause. Non-hidden `foo/bar/skills//SKILL.md` already matches the 'skills' convention at any depth (discovery_test.go 'deeply nested skills directory' a/b/c/skills/my-skill). The failure needs a dot-prefixed segment: matchSkillConventions excludes hasHiddenSegment paths, and matchHiddenDirConventions only accepts a top-level hidden root (`path.Dir(hiddenRoot) == \".\"`). A depth-count or limit tweak would look complete and miss this. The task's own wording ('' placeholder) does not say which dot-dir placement is meant.", + "verification": "Table tests TestMatchSkillConventions and TestMatchHiddenDirConventions in discovery_test.go, plus httpmock-based TestDiscoverSkillsWithOptions and install_test.go gating cases, cover the paths and can run offline. Gaps: nested hidden-root plus plugins ancestor, dedupe across conventions, and local discovery of nested hidden paths are not covered. The existing 'too deeply nested hidden dir' negative case must be deleted or re-pinned by the change.", + "blast_radius": "gh is a user-facing CLI; a wrong install writes SKILL.md files into agent directories under .agents/ or the user's project (install.go scope and host plans). Users would notice and a release revert fixes it. No credentials, money or stored-data migration is involved." + }, + "confidence": "medium", + "summary": "The task needs the hidden-dir skill matcher in internal/skills/discovery/discovery.go to accept dot-prefixed roots at any depth, not only top-level. This keeps --allow-hidden-dirs gating intact and replaces the pinned 'too deeply nested hidden dir' negative test, with nested-skill tests added to discovery_test.go and install_test.go. Non-hidden nested skills already work, so the reporter's two-level diagnosis should not drive the fix." + }, + { + "task": "skill-install-nested-discovery", + "attempt": 2, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:16:57.537725Z", + "metrics": { + "input_tokens": 26, + "output_tokens": 8392, + "cache_read_tokens": 361522, + "cache_creation_tokens": 50606, + "num_turns": 12, + "latency_seconds": 56.9, + "total_cost_usd": 0.015553526999999998 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "internal/skills/discovery/discovery_test.go pins the hidden-dir boundary: \"too deeply nested hidden dir\" (.claude/nested/skills/code-review/SKILL.md) must return nil, and matchHiddenDirConventions is documented as only `.{host}/skills/*` and `.{host}/skills/{scope}/*`. The task says \"any depth\" but never says whether a nested skills/ under a dot-directory is wanted. That answer lives in those existing design decisions and in the `--allow-hidden-dirs` provenance warning (install.go filterHiddenDirSkills, SKILL.md \"Don't use this unless you need to, it comes with risks\"), not in the task text.", + "silent_invariant": "internal/skills/discovery/discovery.go matchSkillConventions and matchHiddenDirConventions decide which SKILL.md files get installed into user agent directories. Widening matches under dot-directories changes what the hidden-dir gate in filterHiddenDirSkills lets through. Errors would show as extra or missing skills in edge layouts, not as a crash, but the failure is not an authz or data-integrity hole.", + "coordinated_surfaces": "One package carries the change: discovery.go matchers are shared by DiscoverSkillsWithOptions (remote), DiscoverLocalSkillsWithOptions (local) and search.go:877 via MatchSkillPath. DiscoverSkillByPath has its own convention labelling. Docs to keep in sync are the install help text at install.go:111 and skills/gh-skill/SKILL.md. Two or three places in one component plus docs, not several services.", + "ambiguity": "The task uses the placeholder ``. The non-hidden form already matches: matchSkillConventions's \"Deeply nested skills/ directory\" branch and the test \"a/b/c/skills/my-skill\" in discovery_test.go. So the reported failure must involve a dot segment, but the task never says which. A narrow fix, such as a special case for one hidden root, would pass a quick review without fixing the general rule, and the wider rule conflicts with the pinned negative tests.", + "verification": "discovery_test.go has table tests for matchSkillConventions and matchHiddenDirConventions (including the negative cases that pin the boundary), plus httpmock-driven DiscoverSkills tests and TestDiscoverLocalSkills. Install tests exist in install_test.go (hidden-dir cases at lines ~124 and ~1380). Everything runs offline, so a wrong widening would fail existing pinned tests. A new nested-hidden positive case is still needed.", + "blast_radius": "The CLI writes skill files into user agent directories (.claude/, .agents/ and so on), which is user-facing. A revert or a rerun with the old binary fixes it. The install flow warns about copies from other publishers, so the stakes are provenance and noise, not credentials, money or stored data." + }, + "confidence": "medium", + "summary": "The non-hidden nested `skills//SKILL.md` case already matches in matchSkillConventions and is tested, so the real gap is nested skills under dot-directories with `--allow-hidden-dirs`, which the existing hidden-dir matcher and its pinned negative tests deliberately exclude. Fixing it needs a design decision about hidden-root nesting, applied consistently to remote and local discovery and the help/docs, plus new nested-hidden tests alongside the boundary tests that must be kept or revised." + }, + { + "task": "skill-install-nested-discovery", + "attempt": 3, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:20:35.325630Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 7325, + "cache_read_tokens": 164747, + "cache_creation_tokens": 44389, + "num_turns": 7, + "latency_seconds": 49.5, + "total_cost_usd": 0.011946214500000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "internal/skills/discovery/discovery.go matchSkillConventions (~L430-438) already documents 'Matches skills/ at any depth, not just at the repository root', and the matcher table in discovery_test.go (L104-120) pins 'a/b/c/skills/my-skill' as convention 'skills'. Behaviour follows from that existing convention and the Expected-messages wording, which the task does not restate. The matchHiddenDirConventions root-only rule and the negative test 'too deeply nested hidden dir' (discovery_test.go L231) are a second contract the task does not mention, but it only matters if the fix extends to hidden directories.", + "silent_invariant": "Matched paths become installed skills written into user agent directories (pkg/cmd/skills/install/install.go, installer). A broader matcher could pick up wrong skills (e.g. vendored or unrelated skills/ trees), but this shows as extra or wrong installed names, and collisions are reported by FindNameCollisions (collisions.go). It is an edge case a normal check would miss, not a security or data-integrity invariant.", + "coordinated_surfaces": "The matcher is shared by DiscoverSkillsWithOptions, DiscoverLocalSkillsWithOptions and walkTree within internal/skills/discovery/discovery.go. Callers in pkg/cmd/skills/{install,preview,update} inherit the change without edits. The four 'no skills found' messages in discovery.go list the conventions and would need updating only if the convention set changes.", + "ambiguity": "The exact reported path foo/bar/skills//SKILL.md already matches via the 'skills' convention (discovery.go matchSkillConventions, the 'Deeply nested skills/ directory' branch; parentDir base == 'skills', no hidden segment, no plugins ancestor). So a test-only or matcher-only patch looks complete while the reported symptom may not be the real gap. The real gap could be elsewhere: nested hidden dirs (foo/bar/.claude/skills), where matchHiddenDirConventions only accepts a root-level dot dir, or the local walk. Those readings conflict with the pinned test at discovery_test.go L231. Confidence in this reading is medium, since I did not execute the reproduction.", + "verification": "Strong existing coverage: discovery_test.go matcher table (L104-150, L1198, L1223), DiscoverSkills nested tests (L915, L933, L1351), truncated-tree walkTree test (L1590), and install_test.go 'remote install by nested skill path' (L774) plus nested path cases (L2118-2120). Gaps: no positive test for a nested hidden-dir skill (foo/.x/skills/...), and no test for local-walk nested discovery beyond L1351. A wrong change to hidden-dir nesting could ship unnoticed.", + "blast_radius": "Public CLI (cli/cli, module github.com/cli/cli/v2). Wrong results write skill files into user agent directories, which users would notice. A revert or rerun fixes it, with no security, money or stored-data impact. No SECURITY.md exists in the repo root." + }, + "confidence": "medium", + "summary": "The reported path foo/bar/skills//SKILL.md already matches the 'skills' convention at any depth in internal/skills/discovery/discovery.go, so the real work is to confirm which nested case still fails (likely nested hidden-dir skills, which conflicts with the pinned negative test at discovery_test.go L231) before changing matchers and adding nested-skill tests. Scores rest on reading the code and tests, since I did not run the reproduction." + }, + { + "task": "skill-update-dir-keeps-install-path", + "attempt": 1, + "ref": "v2.94.0", + "judged_at": "2026-10-14T22:13:09.375605Z", + "metrics": { + "input_tokens": 22, + "output_tokens": 12363, + "cache_read_tokens": 294881, + "cache_creation_tokens": 49704, + "num_turns": 10, + "latency_seconds": 81.2, + "total_cost_usd": 0.016880061000000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task states the goal directly: update in place, flat and namespaced, with or without --dir, never relocate or delete the old path. The layout that defines 'where a skill lives' comes from scanInstalledSkills (pkg/cmd/skills/update/update.go:478-526, namespaced = {dir}/{ns}/{name}/SKILL.md) and the installer's flat-write convention (internal/skills/installer/installer.go:251-253, skillDir = baseDir/skill.Name). The task does not say what happens when the remote skill.Name differs from the local directory basename; that gap is small and is resolved by reading the installer.", + "silent_invariant": "update.go:402-411 derives the install base with a double filepath.Dir for names containing '/', and update.go:419-435 then runs os.RemoveAll(u.local.dir) and removes the parent if empty, before printing 'Updated' (update.go:436-440). A wrong change here still reports success, and the user's files are removed without any error.", + "coordinated_surfaces": "Two branches must agree: the --dir/host==nil branch (update.go:405-411) and the agent-host branch (update.go:423-427, which computes InstallDir/skill.Name). Both feed installer.Install (installer.go:56-141, installSkill at 251), which always writes flat under the target dir, so a fix that keeps update's target on the exact local parent may need an installer option. All of this sits in one subsystem (pkg/cmd/skills/update and internal/skills/installer). Acceptance coverage (acceptance/testdata/skills/skills-update.txtar) is dry-run only, so it does not need to change.", + "ambiguity": "Fixing only the double-Dir base (update.go:406-409) makes the repro pass: the install lands on the old path and cleanup is skipped. It leaves the agent-host namespaced case relocating (update.go:423-429 installs flat via InstallDir and deletes the namespaced copy) and leaves the rename case relocating. The existing test update_test.go:712-775 asserts the buggy behaviour ('installed flat', 'old namespaced directory should be removed'), so a shallow fix would not match what the tests expect either.", + "verification": "TestUpdateRun (update_test.go:313+) is table-driven and offline via httpmock, so the change is testable here. But the only namespaced case (update_test.go:712-775) is --dir-only and pins the relocation. No case covers a namespaced skill under an agent host with no --dir, nor the remote-rename case. Acceptance tests (skills-update.txtar) are dry-run only.", + "blast_radius": "update.go:429 calls os.RemoveAll on the user's installed skill directory, and update.go:431-434 removes the parent. Files already deleted (including user edits) are not recovered by reverting the CLI. The task text confirms the impact: 'Anything that pointed at the old path... breaks'." + }, + "confidence": "high", + "summary": "The change must make gh skill update write each skill back to its current local directory for flat and namespaced layouts, with and without --dir, and remove the relocation path (the double-Dir base at update.go:406-409 and the RemoveAll cleanup at update.go:419-435). That needs the installer to write to an exact target, or update to stage and swap in place, plus the existing relocation test at update_test.go:712-775 rewritten and new coverage for the agent-host namespaced case." + }, + { + "task": "skill-update-dir-keeps-install-path", + "attempt": 2, + "ref": "v2.94.0", + "judged_at": "2026-10-14T22:16:54.809153Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 7985, + "cache_read_tokens": 236091, + "cache_creation_tokens": 42833, + "num_turns": 8, + "latency_seconds": 53.0, + "total_cost_usd": 0.012880268499999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task states the goal: update in place, flat and namespaced, with or without --dir, never relocate or delete the old path. The correct base directory depends on how installSkill lays out files (internal/skills/installer/installer.go:251-253 writes baseDir/skill.Name, flat, and install --dir in pkg/cmd/skills/install/install.go:97 inherits that). The fix has to mirror that nearby layout rule, which the task does not spell out.", + "silent_invariant": "pkg/cmd/skills/update/update.go:~440-455 runs os.RemoveAll(u.local.dir) and then removes the parent if empty, keyed on a path derived from installOpts.Dir and skill.Name. A wrong derivation deletes a user's directory, including locally added files (the --force help text at update.go Long says extra local files are not removed), while the command prints 'Updated'. The failure is silent.", + "coordinated_surfaces": "The change sits in one component: install-base derivation (update.go:~400-410), the cleanup block (update.go:~424-440), the --force help text (update.go Long), and tests in update_test.go and acceptance/testdata/skills/skills-update.txtar. Installer and lockfile (keyed by name, no path stored, internal/skills/lockfile/lockfile.go:20-30) need no change.", + "ambiguity": "The reported symptom is a wrong install base, but a narrow fix to Dir(Dir) would leave the RemoveAll cleanup in place, which still deletes the old path when skill.Name differs from the folder name. Host-installed namespaced skills (no --dir) also still relocate through the newDir/host.InstallDir branch. The root cause is structural namespace detection plus unconditional cleanup, so a partial fix looks complete and still misses it.", + "verification": "update_test.go has httpmock-based TestUpdateRun cases, plus a flat --dir txtar acceptance test. The one namespaced --dir case (update_test.go:~712-773, 'namespaced skill with --dir resolves install base correctly') asserts the buggy relocation and cleanup, so it would fail on a correct fix and must be rewritten. Nothing covers in-place namespaced updates, host-path namespaced updates, or old-path preservation. Those are testable offline with the existing harness.", + "blast_radius": "Users' installed skills live in agent config directories (AGENTS.md: skills are user-facing). The code deletes user directories with RemoveAll and then parents, with no backup, which is data loss. It is also a relocation that breaks external references such as symlinks and agent config, as the task states." + }, + "confidence": "high", + "summary": "The fix belongs in pkg/cmd/skills/update/update.go: derive the install base from the directory the skill currently occupies, for flat and namespaced layouts with or without --dir, and remove the relocate-and-RemoveAll cleanup. The existing test at update_test.go:~712-773 pins the buggy migration, so it has to be rewritten, and new tests must cover in-place updates for both layouts." + }, + { + "task": "skill-update-dir-keeps-install-path", + "attempt": 3, + "ref": "v2.94.0", + "judged_at": "2026-10-14T22:20:33.940779Z", + "metrics": { + "input_tokens": 30, + "output_tokens": 10671, + "cache_read_tokens": 421625, + "cache_creation_tokens": 49731, + "num_turns": 14, + "latency_seconds": 72.7, + "total_cost_usd": 0.017348237500000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task states the goal (update in place, never relocate or delete the old path). The layout rules come from nearby code: scanInstalledSkills (update.go:482-526) defines flat and namespaced layouts, and installSkill/installLocalSkill (installer.go:251-253, 180-184) always flatten to baseDir/skill.Name, citing agent discovery. A reader must follow that code to see that the install target must be the skill's current parent directory.", + "silent_invariant": "update.go:405-435: the code removes a user's directory with os.RemoveAll(u.local.dir) and then prints 'Updated'. The relocated copy looks successful, and deletion discards local edits and extra files (see the --force doc at update.go:98-100). A wrong target or wrong removal path destroys data without any error.", + "coordinated_surfaces": "The change sits in one component: pkg/cmd/skills/update/update.go (install base at 405-411, cleanup at 419-435). The host-scoped path (Options.Dir empty) also needs in-place handling, which goes through installer.Options.Dir and installer.Install (installer.go:56-67). Docs (update Long help, update.go:78-105; skills/gh-skill/SKILL.md) do not describe relocation, so they likely need no change.", + "ambiguity": "Narrow fix trap: removing only the second filepath.Dir (update.go:407-408) makes the --dir case write in place, but the host-scoped namespaced path (update.go:423-427) still installs flat and deletes the namespaced dir. Removing only the cleanup block leaves a duplicate flat copy. The cleanup comment (update.go:419-421) states the intended namespaced-to-flat migration, which the task reverses, so the intended design is not obvious from the code alone.", + "verification": "pkg/cmd/skills/update/update_test.go TestUpdateRun has httpmock-driven cases with on-disk assertions (e.g. the 'namespaced skill with --dir resolves install base correctly' case at lines 711-775). That case pins the opposite behavior: it asserts the namespaced dir is removed (lines 770-772), so it must be rewritten. No case covers host-scoped namespaced in-place updates, and no acceptance txtar for update was seen.", + "blast_radius": "gh is a user-facing public CLI. The bug deletes user skill directories under ~/.agents and project .agents/skills (update.go:428-434, scanning all registry agents in scanAllAgents at 452-476). Deleted content cannot be recovered by reverting the binary." + }, + "confidence": "medium", + "summary": "The fix is in pkg/cmd/skills/update/update.go: install into the skill's current parent directory for every scope (with or without --dir) and delete the post-install cleanup block, which means the installer's flat-only write (installer.go:251-253) must not relocate namespaced skills. The existing TestUpdateRun case at update_test.go:711-775 pins the old migrate-and-delete behavior and must be replaced with in-place tests for flat and namespaced layouts, covering both host and --dir paths." + } + ], + "rubric": "/home/ubuntu/agentic-coding-harness-benchmarks/benchmarks/config/swe-router/rubric.md", + "policy": { + "schema_version": "1.0", + "required_quality": { + "by_blast_radius": { + "0": 60.0, + "1": 70.0, + "2": 75.0 + }, + "add_when_verification_is_2": 5.0, + "meaning": { + "60": "Bottom of the judge's 'strong' band: a usable foundation; expect review to fix real gaps.", + "65": "Low 'strong': implementation-ready in parts; review will find gaps.", + "70": "Middle of 'strong': implementation-ready in most respects with limited gaps.", + "75": "Top of 'strong': few gaps and well-grounded; for security, money or data paths.", + "80": "Approaching the judge's 'excellent' band: comprehensive and precise." + } + }, + "difficulty": { + "dimensions": [ + "hidden_contract", + "silent_invariant", + "coordinated_surfaces", + "ambiguity" + ], + "bucketing": "absolute", + "buckets": { + "easy": [ + 0, + 1 + ], + "moderate": [ + 2, + 3 + ], + "hard": [ + 4, + 5 + ], + "very_hard": [ + 6, + 8 + ] + } + }, + "tie_band_points": 3.0, + "when_unsure": "stay_on_current_model" + } +} diff --git a/docs/metrics/swe-router-rubric-cli-cli-v1-claude-opus-5-5.json b/docs/metrics/swe-router-rubric-cli-cli-v1-claude-opus-5-5.json new file mode 100644 index 00000000..7bd9ee83 --- /dev/null +++ b/docs/metrics/swe-router-rubric-cli-cli-v1-claude-opus-5-5.json @@ -0,0 +1,3368 @@ +{ + "judged_by": { + "harness": "omp", + "model": "us.anthropic.claude-opus-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "repeats": 3, + "started_at": "2026-10-14T22:10:12.033833Z", + "finished_at": "2026-10-14T22:23:36.365453Z", + "judging_cost_usd": 15.5931, + "runs_ok": 63, + "runs_failed": 0 + }, + "mode": "rubric", + "consolidation": "lower median per dimension across repeats; per-repeat quality and bucket kept in quality_seen / buckets_seen", + "tasks": { + "repo-create-reject-empty-name": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 3, + "bucket": "moderate", + "confidence": "high", + "summary": "Add an empty (and likely whitespace-only) name check in `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`, right after the \"Repository name\" Input and before the owner lookup and Select. It should return an error in the repo's \"cannot be blank\" style so the from-scratch, template and local flows all stop early. Add `create_test.go` table cases with an empty name that assert the error and that no later prompts or API calls happen, and keep the existing valid-name tests passing.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 1, + 1 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "copilot-not-installed-trailing-newline": { + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 0, + "bucket": "easy", + "confidence": "high", + "summary": "Add a trailing newline to the 'Copilot CLI was not installed' warning in runCopilot (pkg/cmd/copilot/copilot.go:155), keeping the SilentError return. Add a runCopilot test that stubs findCopilotBinaryFunc to return \"\", makes the prompter decline with a TTY iostreams, and asserts the exact stderr output.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 0, + 0, + 0 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 0, + 0, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "easy": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "repo-set-default-unset-without-repo": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 2, + "bucket": "moderate", + "confidence": "high", + "summary": "Relax the non-interactive 'repository required' guard in NewCmdSetDefault (pkg/cmd/repo/setdefault/setdefault.go) so that `--unset`, like `--view`, runs without a repository argument when stdin is not a terminal, while plain `gh repo set-default` keeps the existing FlagError. Extend TestNewCmdSetDefault with a TTY toggle and cases for non-TTY `--unset`, non-TTY `--view` and the non-TTY no-argument error that must stay.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "pr-checks-all-cancelled-summary": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 1, + "bucket": "easy", + "confidence": "high", + "summary": "The fix is to add counts.Canceled to the guard in printSummary (pkg/cmd/pr/checks/output.go) so that all-cancelled PRs reach the existing 'Some checks were cancelled' branch and its tally line. It needs a new all-cancelled fixture and a test row in checks_test.go that asserts the full summary output.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 0, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 0, + 0, + 0 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "easy": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "extension-uninstall-alias": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 2, + "bucket": "moderate", + "confidence": "high", + "summary": "Add `uninstall` as a cobra alias on the existing `remove` subcommand in pkg/cmd/extension/command.go, following the `list`/`ls` pattern, so routing and the ALIASES help section come for free. Add a table-driven case to pkg/cmd/extension/command_test.go that runs `uninstall` and checks that Remove is called with the normalized name.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 0 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "pr-merge-delete-branch-owner-case": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 3, + "bucket": "moderate", + "confidence": "high", + "summary": "The crossRepoPR computation in pkg/cmd/pr/merge/merge.go:592 must compare the PR head owner login with the base repo owner case-insensitively (matching ghrepo.IsSame/FindByRepo), so same-repo PRs get their remote branch deleted. A merge_test.go regression test must cover an owner-case mismatch and expect the DELETE git/refs call, while fork PRs keep their branches.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 0, + 0, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 1, + "moderate": 2 + }, + "quality_unanimous": true, + "bucket_unanimous": false + }, + "pr-view-json-number-on-issue": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "medium", + "summary": "The shared PR finder (pkg/cmd/pr/shared/finder.go) returns a synthetic PullRequest without any API call when only 'number' is requested, so gh pr view N --json number succeeds for issue numbers. The fix must make that path check with GitHub, either in the finder (which also changes pr diff and pr checks) or in pr view, and must update the finder test that pins the shortcut and add an issue-number case that expects an error.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "project-item-add-output-non-tty": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 2, + "bucket": "moderate", + "confidence": "high", + "summary": "Change `printResults` in `pkg/cmd/project/item-add/item_add.go` so that non-TTY stdout gets a short, stable confirmation (preferably the added item's ID from the mutation response), leaving the TTY \"Added item\" message and the `--format json` exporter path unchanged. Add a non-TTY test next to the existing TTY and JSON tests in `item_add_test.go`.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 0, + 0, + 1 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "release-create-404-scope-blame": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 3, + "bucket": "moderate", + "confidence": "high", + "summary": "tokenHasWorkflowScope in pkg/cmd/release/create/http.go splits X-Oauth-Scopes on \",\" without trimming whitespace, so real \"a, b, workflow\" headers never match \"workflow\" and every 404 is replaced by the scope hint. The fix is to trim each scope, following HeaderHasMinimumScopes in auth/shared, and to add create_test.go cases that use the real comma-space header format for tokens with and without workflow, keeping the existing empty-header rule.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "pr-comment-attach-ghu-tokens": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 3, + "bucket": "moderate", + "confidence": "medium", + "summary": "Add gh.TokenTypeUserToServer to uploadTokenTypes in internal/attachments/client.go, keeping ghs_/ghr_/unknown rejected and the host check first. Flip the 'rejects an App user-to-server token' row in TestNewUploader and add a command-level test with a ghu_ token that uploads, plus a check that a server refusal comes back to the user.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 0, + 0, + 0 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 0, + 0, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "moderate": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "repo-clone-ghe-ssh-ca-url": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "medium", + "summary": "Add sshUrl to the RepositoryInfo fragment in api.GitHubRepo for the repository and its parent, and have cloneRun use the API SSH URL for both the clone and the fork upstream remote when the protocol is ssh, including the URL-argument and wiki paths, while leaving HTTPS unchanged. Add clone_test.go cases with mocked sshUrl responses that check the git clone and remote add commands.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "codespace-ports-forward-concurrent-map": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 6, + "bucket": "very_hard", + "confidence": "medium", + "summary": "ForwardPorts runs one goroutine per port, and all of them share a single CodespaceConnection whose TunnelRequestOptions.AdditionalHeaders map and Tunnel.Ports are changed by dev-tunnels Manager calls, so the fix needs synchronization (or per-call copies) at the shared connection or in ForwardPorts, not inside each per-goroutine forwarder. It also needs a new test that forwards several ports at once against the existing mock tunnels server, which CI runs under -race.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 2, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "very_hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "issue-view-comments-with-json": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "high", + "summary": "In `pkg/cmd/issue/view/view.go`, `--comments` combined with `--json` must either add `comments` to the exporter's field list (so the comments are fetched and written) or fail with a mutually-exclusive flag error after PreRunE has set up the Exporter. `--json` alone and `--comments` alone must behave as they do today. New table tests in `view_test.go` should cover the combination using the existing httpmock fixtures.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "release-verify-sha1-prefix": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "medium", + "summary": "Replace the hard-coded \"sha1\" passed to `artifact.NewDigestedArtifactForRelease` in both `verify.go` and `verify_asset.go` with an algorithm worked out from the ref returned by `shared.FetchRefSHA` (40 hex characters means sha1, 64 means sha256), ideally through one shared helper. The output text, the `GetByDigest` lookup and the sigstore digest policy all need the same algorithm, with tests covering both SHA-1 and SHA-256 refs in both commands.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 2 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 75.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": false, + "bucket_unanimous": true + }, + "auth-refresh-hint-on-expired-token": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "required_quality": 75.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "In internal/ghcmd/cmd.go, the 401 branch should print `gh auth refresh` (probably with `-h `) only when the active token for the host of the failed request was stored by gh and is an OAuth token. That is the oauth_token/keyring source plus gho_ prefix rule from agent_task.go; every other token keeps `gh auth login`. New tests are needed because nothing covers this branch today, and they should cover the tricky cases (a `--with-token` PAT, a gho_ token from an env var, a non-default host), not just the obvious pair.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 2, + 2, + 2 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 75.0, + 75.0, + 75.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "secret-set-app-agents": { + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "Add an `agents` App to pkg/cmd/secret/shared with GetSecretApp and IsSupportedSecretEntity support (repository and organization), add it to the `--app` enum in set, list and delete, choose the right org PUT payload in set/http.go, and update help text. Add httpmock-based tests for repo and org set, list and delete whose paths and payloads must match GitHub's documented agents secrets REST endpoints.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 2, + 2, + 2 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "repo-sync-checked-out-worktree": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "`executeLocalRepoSync` needs to find out whether the target branch is checked out in any other worktree, for example through a new gitClient/git.Client method that parses `git worktree list --porcelain`. If it is, sync must refuse with a message naming that worktree's path, or update it safely, instead of moving the ref with `update-ref`. That means changing the gitClient interface, the executer, mocks.go and the table tests in sync_test.go, while syncing a branch nobody has checked out keeps working as before.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 2 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 75.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": false, + "bucket_unanimous": true + }, + "auth-clipboard-config-default": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "high", + "summary": "Add a `clipboard` enabled/disabled setting following the spinner/prefer_editor_prompt pattern: the gh.Config interface method, regenerated mock, const, accessor, default config string, Options entry, test stub and the config list test expectation. Then have `gh auth login` copy the code when either `--clipboard` is passed or the setting is enabled, with tests covering how flag and config combine.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "project-item-edit-by-name": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 4, + "bucket": "hard", + "confidence": "medium", + "summary": "Add an opt-in name form to `gh project item-edit` (` --owner --url --field --value`). It must resolve the project, the content-to-item mapping, the field (case-insensitive) and the option or typed value before calling the existing `updateProjectV2ItemFieldValue` mutation, and fail with a list of candidates when a name does not match. It must also keep the ID-flag paths unchanged and add gock-mocked tests in `item_edit_test.go`, probably along with a new lookup helper in `shared/queries`.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 1, + 1 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + }, + "skill-update-dir-keeps-install-path": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 5, + "bucket": "hard", + "confidence": "medium", + "summary": "In pkg/cmd/skills/update/update.go, the update must reinstall each skill into the parent of its current directory for flat and namespaced layouts, whether or not an agent host is involved, and the RemoveAll/parent-cleanup block must go. update_test.go needs its namespaced test turned around and new cases showing the original path is kept and updated in place.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 1, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 1, + 1, + 1 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 2 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 75.0 + ], + "buckets_seen": { + "hard": 3 + }, + "quality_unanimous": false, + "bucket_unanimous": true + }, + "skill-install-nested-discovery": { + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "required_quality": 70.0, + "difficulty_sum": 6, + "bucket": "very_hard", + "confidence": "medium", + "summary": "The core repro (foo/bar/skills//SKILL.md) already matches in internal/skills/discovery/discovery.go and has tests. The real work is to decide whether nested skills under hidden or plugins directories should be discovered when --allow-hidden-dirs is set, and to extend the shared matcher (used by remote, local, preview, search and path-based install) without breaking today's conventions or the hidden-dir gate. Nested-skill tests are needed across discovery and install.", + "attempts": 3, + "scores_seen": { + "hidden_contract": [ + 1, + 1, + 1 + ], + "silent_invariant": [ + 2, + 2, + 2 + ], + "coordinated_surfaces": [ + 1, + 1, + 1 + ], + "ambiguity": [ + 2, + 2, + 2 + ], + "verification": [ + 1, + 1, + 1 + ], + "blast_radius": [ + 1, + 1, + 1 + ] + }, + "quality_seen": [ + 70.0, + 70.0, + 70.0 + ], + "buckets_seen": { + "very_hard": 3 + }, + "quality_unanimous": true, + "bucket_unanimous": true + } + }, + "judgments": [ + { + "task": "auth-clipboard-config-default", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:13:03.462261Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3890, + "cache_read_tokens": 128939, + "cache_creation_tokens": 33788, + "num_turns": 6, + "latency_seconds": 48.2, + "total_cost_usd": 0.29984218 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the key and value ('gh config set clipboard enabled'), but the full contract comes from existing code. Settings are declared as a key constant, a line in defaultConfigStr, an Options entry with AllowedValues {enabled, disabled}, and a gh.Config accessor that calls GetOrDefault, with Spinner/ColorLabels as the pattern (internal/config/config.go:20-40, 170, 611-768; internal/gh/gh.go:38-61). The task does not mention any of this.", + "silent_invariant": "No auth or secret invariant is involved: the code only decides whether authflow.AuthFlow calls clipboard.WriteAll on the one-time code (internal/authflow/flow.go:49-58). Some wrong changes only show in edge cases: c.get falls back between host-scoped and global values (config.go:55-69), and login knows the hostname only after the prompt (login.go:175-187). Reading the config at the wrong point, or with the wrong host, or letting the config wrongly override the flag, would pass a simple check.", + "coordinated_surfaces": "Several places inside the CLI must agree. In config.go: the key constant, defaultConfigStr, the Options registry (used by `config set` ValidateKey/ValidateValue in set.go:96-132 and by `config list`) and the accessor. The gh.Config interface (gh.go) and its moq mock, plus the mock wiring in internal/config/test.go:96-101. Then loginRun/shared.LoginOptions.CopyToClipboard (login.go:243) and possibly refresh.go:203. The expected output in pkg/cmd/config/list/list_test.go:106-109 also changes. All of it is one component, with no deployment or second service.", + "ambiguity": "The task names only `gh auth login`, but `gh auth refresh` has the same --clipboard flag (refresh.go:117, 203), so it is unclear whether the setting should cover both. The task also does not say whether the setting is global or host-scoped, or how it combines with the flag (OR of the two vs the flag overriding). A narrow fix, such as only adding an Options entry or only reading a raw key in login, would look done. The sibling settings show the intended shape.", + "verification": "Test_NewCmdLogin only checks that the --clipboard flag sets opts.Clipboard (login_test.go:133-150, 296). Test_loginRun_* and shared TestLogin cannot see whether the clipboard is used, because authflow.AuthFlow is called directly and is not injectable (login_flow.go:150), and clipboard.WriteAll needs a real system clipboard. list_test.go will catch a missing Options entry. Nothing currently catches login ignoring the config; new tests have to cover that.", + "blast_radius": "This is user-facing `gh auth login` and `gh config` behaviour. If it is wrong, the code is not copied (or is copied unexpectedly), or `config list` output changes. No tokens, stored credentials or migrations are involved: the device code is already printed to stderr (flow.go:58). A revert fixes it." + }, + "confidence": "high", + "summary": "The task adds a `clipboard` enabled/disabled setting following the existing pattern: key constant, default config string, Options entry, gh.Config accessor with its regenerated mock and test wiring, and an updated config list test. loginRun then has to turn on CopyToClipboard when either the flag or the setting is on, and the open choices are whether refresh is included, whether the setting is per host, and how to test it given that AuthFlow cannot be injected." + }, + { + "task": "auth-clipboard-config-default", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:16:54.845936Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3458, + "cache_read_tokens": 175470, + "cache_creation_tokens": 36471, + "num_turns": 7, + "latency_seconds": 73.1, + "total_cost_usd": 0.3153403 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives only the form `gh config set clipboard enabled`. The rest comes from how existing settings are built in internal/config/config.go: a key constant, a typed accessor, an entry in `defaultConfigStr` and an entry in `Options` with AllowedValues enabled/disabled and a CurrentValue. The accessor must also be added to the gh.Config interface in internal/gh/gh.go, with its moq mock and the wiring in internal/config/test.go. All of this can be read from nearby code; no outside spec or earlier design fixes it.", + "silent_invariant": "internal/authflow/flow.go only writes the device code to the clipboard when `isCopyToClipboard` is true, and falls back to printing it if the copy fails. The mistakes that would slip through are edge cases: the default not being disabled, the setting not combined with the `--clipboard` flag by OR, or the setting looked up for the wrong host (`GetOrDefault(hostname, ...)`). None of this touches tokens, auth decisions or stored credentials.", + "coordinated_surfaces": "Several places in one binary must agree: the config layer (key, accessor, `defaultConfigStr`, `Options`), the gh.Config interface and its generated mock (internal/gh/mock/config.go), internal/config/test.go, and the consumer in pkg/cmd/auth/login/login.go, which builds `CopyToClipboard: opts.Clipboard`. The expected output in pkg/cmd/config/list/list_test.go must change too. There is no second service, API or deployment surface.", + "ambiguity": "A quick fix is to read a raw `clipboard` key in loginRun. But if the key is missing from `config.Options`, set.go prints \"warning: '%s' is not a known configuration key\" and does not check the value, and `gh config list` leaves the key out. Repo convention settles that. A second open question is whether `gh auth refresh`, which also has `--clipboard`, should honour the setting; the task names only login, and the sibling flag suggests refresh probably should too.", + "verification": "Test_NewCmdLogin covers parsing of the `--clipboard` flag, and list_test.go pins the full list of options. But loginRun calls shared.Login directly and the auth flow cannot be injected, so no existing test checks that the config value reaches `CopyToClipboard`. Actual clipboard writes (atotto/clipboard) are not tested anywhere. New tests would need a seam.", + "blast_radius": "`gh auth login` is a customer-facing command, so users would notice a mistake, such as the code being copied unexpectedly or the setting being ignored. Reverting fixes it with no lasting harm. The device code is already printed on screen, and no stored data or credentials are changed." + }, + "confidence": "high", + "summary": "Add a `clipboard` setting (enabled/disabled, default disabled) the same way existing settings are built: key, accessor on gh.Config plus the regenerated mock and test wiring, `defaultConfigStr`, and an `Options` entry so `gh config set` and `gh config list` know it. Then have `gh auth login` combine it with `--clipboard` by OR when filling `CopyToClipboard` (refresh probably too). Tests must cover the config-driven path, which existing tests do not exercise, and update the pinned `gh config list` output." + }, + { + "task": "auth-clipboard-config-default", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:21:30.960744Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3651, + "cache_read_tokens": 137729, + "cache_creation_tokens": 38082, + "num_turns": 6, + "latency_seconds": 56.0, + "total_cost_usd": 0.32013498 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The expected shape comes from code conventions the task doesn't spell out. internal/config/config.go has a key const, a `cfg` accessor that calls GetOrDefault(...).Unwrap() and panics when no default exists, a `defaultConfigStr` entry, and an `Options` entry with AllowedValues enabled/disabled. pkg/cmd/pr/shared/survey.go InitEditorMode shows how a flag and a config value combine (`flag || config == \"enabled\"`). The 'issue' the task mentions isn't linked, but the suggested `gh config set clipboard enabled` matches these conventions.", + "silent_invariant": "Nothing here touches auth, secrets or stored-data invariants; the clipboard only ever receives the one-time device code, through internal/authflow/flow.go. Mistakes would only show up in edge cases. pkg/cmd/config/set/set.go only warns on an unknown key and saves it anyway, so leaving the key out of `Options` still looks like it works. A missing default makes the accessor panic, but only on a path that reads it. Two other quiet choices: per-host versus global lookup, and whether `--clipboard=false` can override the setting, given the plain BoolVarP on login.go:154.", + "coordinated_surfaces": "All changes stay inside the gh binary, in two areas. Config side: the gh.Config interface in internal/gh, the moq mock (regenerate it), internal/config/config.go (const, accessor, defaultConfigStr, Options), and the stub wiring in internal/config/test.go. Command side: pkg/cmd/auth/login/login.go, where `CopyToClipboard: opts.Clipboard` is set at line 243. pkg/cmd/config/list/list_test.go pins the full key list and must be updated to match. Each piece follows the spinner pattern mechanically.", + "ambiguity": "Some readings are open: whether `gh auth refresh`, which has its own `--clipboard` flag in pkg/cmd/auth/refresh/refresh.go, should also honour the setting; whether the key is per-host or global; and how flag and config combine. The repo's enabled/disabled convention and the `flag || config` pattern in InitEditorMode point to the intended answer. The task names only `gh auth login`.", + "verification": "pkg/cmd/auth/login/login_test.go checks flag parsing (`Clipboard` in the 'tty web and clipboard' case), and config/list/list_test.go would fail if the key list drifts. No existing test reads config in the login path or checks that shared.LoginOptions.CopyToClipboard is driven by it. The actual clipboard write (atotto/clipboard in authflow/flow.go) has no tests, so the new tests have to cover the config-driven path.", + "blast_radius": "This is a user-facing CLI command and config key. A wrong default or a precedence bug would be noticed (the code gets copied, or doesn't) and can be reverted without lasting harm. The one lasting cost: once shipped, the key name becomes part of users' persisted config.yml, so renaming it later is awkward." + }, + "confidence": "high", + "summary": "Add a `clipboard` enabled/disabled setting following the spinner/prefer_editor_prompt pattern: the gh.Config interface method, regenerated mock, const, accessor, default config string, Options entry, test stub and the config list test expectation. Then have `gh auth login` copy the code when either `--clipboard` is passed or the setting is enabled, with tests covering how flag and config combine." + }, + { + "task": "auth-refresh-hint-on-expired-token", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:12:27.936691Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3421, + "cache_read_tokens": 126297, + "cache_creation_tokens": 33131, + "num_turns": 6, + "latency_seconds": 40.9, + "total_cost_usd": 0.28532944000000005 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Which tokens count as 'created by gh auth login' is not defined in the task. The answer comes from repo conventions: AuthConfig.ActiveToken in internal/config/config.go returns source 'keyring'/'oauth_token' or an env var name, and shared.AuthTokenWriteable (pkg/cmd/auth/shared/writeable.go) treats '*_TOKEN' sources as not writeable. gh auth refresh (refresh.go:169) refuses env-var tokens, so the hint has to follow the same rule.", + "silent_invariant": "This is advisory stderr text only (cmd.go:229-230), so no security or data invariant is involved. Misclassification only shows up under less common setups: GH_TOKEN/GH_ENTERPRISE_TOKEN env tokens, a PAT pasted via --with-token, a 401 from a non-default host, or the multi-account keyring. Advising refresh in those cases would point users to a command that refuses to run or does not fit.", + "coordinated_surfaces": "The change stays inside the CLI: the 401 branch in internal/ghcmd/cmd.go Main(), host and token-source lookup through cmdFactory.Config()/AuthConfig.ActiveToken (possibly using api.HTTPError.RequestURL for the host), and probably an extracted helper so it can be tested. No API, deployment or other-service surface is involved; pkg/cmd/auth/status uses similar 'gh auth login -h' wording that may or may not be meant to follow.", + "ambiguity": "'A token gh auth login created' has two reasonable readings: any stored token (source keyring/oauth_token, which includes --with-token PATs), or only OAuth tokens minted by the login flow (gho_ prefix). The error path in Main() also has no hostname, so checking the default host looks complete but gives wrong advice for a 401 from another host. Swapping the string unconditionally, or checking only 'not an env var', would pass a quick review.", + "verification": "internal/ghcmd/cmd_test.go tests printError, newIOStreams and mightBeGHESUser, but nothing exercises the 401 hint branch in Main(), and Main() cannot be unit-tested as written. No acceptance test pins 'Try authenticating with'. Only the new tests the task asks for would catch a mistake.", + "blast_radius": "Every gh user who hits an expired or invalid token would see wrong or misleading guidance in stderr. Nothing is stored or changed (credentials are not touched), and a release revert fixes it." + }, + "confidence": "medium", + "summary": "The hard-coded 401 hint in internal/ghcmd/cmd.go Main() needs to work out the host and token source for the failed request and suggest `gh auth refresh` only when the token came from gh's own login storage (keyring/hosts.yml, not env vars), keeping today's text otherwise. The logic has to be pulled into a testable helper and covered by new tests for the login-created case and the other-token case." + }, + { + "task": "auth-refresh-hint-on-expired-token", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:16:27.652869Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3581, + "cache_read_tokens": 161531, + "cache_creation_tokens": 39482, + "num_turns": 7, + "latency_seconds": 72.4, + "total_cost_usd": 0.33154022000000005 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task never says how to tell that `gh auth login` created a token. The repo already has a rule for this in pkg/cmd/agent-task/agent_task.go requireOAuthToken: the token's source must be `oauth_token` or `keyring`, and the token must start with `gho_`. Separately, pkg/cmd/auth/refresh/refresh.go uses shared.AuthTokenWriteable to refuse to refresh tokens that come from `*_TOKEN` environment variables. Both rules come from nearby code, not from an outside spec.", + "silent_invariant": "The 401 branch in internal/ghcmd/cmd.go:228-230 is only advice text. If the token is classified wrongly, the normal case (a keyring `gho_` token) still looks right. The mistake only appears in other setups: GH_TOKEN/GH_ENTERPRISE_TOKEN in the environment (where `gh auth refresh` refuses to run), a PAT stored with `gh auth login --with-token`, or a 401 from a host that is not the default. Nothing leaks or gets corrupted.", + "coordinated_surfaces": "The change is the 401 message in internal/ghcmd/cmd.go Main. That needs access to the auth config, through cmdFactory.Config()/Authentication().ActiveToken, for the host the request went to (taken from api.HTTPError's RequestURL or from DefaultHost). The code probably has to be pulled out of Main into a helper so it can be tested in internal/ghcmd/cmd_test.go. All of this is inside one component, with no docs or schema to change.", + "ambiguity": "The tempting fix checks only that the token source is `keyring` or `oauth_token`. That passes a quick review and a GH_TOKEN-vs-keyring test, yet it also gives the refresh advice for PATs that users pasted with `gh auth login --with-token`, which login did not create. The `gho_` prefix check in agent_task.go covers that case. A second choice is open too: the default host or the host the failing request went to.", + "verification": "Nothing in internal/ghcmd/cmd_test.go tests the 401 branch: it has Test_printError, Test_newIOStreams_* and Test_mightBeGHESUser, and nothing calls Main's error-handling block. The logic sits inline in Main, so the only tests catching a mistake would be the new ones the task asks for.", + "blast_radius": "This is stderr guidance that every user sees on a 401 from any command. Wrong advice, or a nil or config-error panic in Main's shared error path, would be noticeable. A revert fixes it, and no tokens or stored data are changed." + }, + "confidence": "medium", + "summary": "The 401 branch in internal/ghcmd/cmd.go Main must find the active token and its source for the right host. It should suggest `gh auth refresh` only for tokens that `gh auth login` created, following the agent_task.go rule (source is keyring or oauth_token, and the token starts with `gho_`). That logic probably needs to move into a helper so cmd_test.go can test both the login-token case and the other-token case." + }, + { + "task": "auth-refresh-hint-on-expired-token", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:20:54.011202Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3423, + "cache_read_tokens": 120648, + "cache_creation_tokens": 38431, + "num_turns": 6, + "latency_seconds": 44.1, + "total_cost_usd": 0.31328066 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task never says how to tell that `gh auth login` created a token. The repo already has a rule for this in pkg/cmd/agent-task/agent_task.go:91-93: the token source must be \"oauth_token\" or \"keyring\" and the token must start with the \"gho_\" prefix. The host-specific hint format is in pkg/cmd/auth/status/status.go:82 (`gh auth refresh -h `). The failing host is available from api/client.go via HTTPError.RequestURL.Hostname(). All of this comes from nearby code, not an outside spec.", + "silent_invariant": "Picking the wrong tokens only shows up in edge cases. One is a PAT saved with `gh auth login --with-token`: its source is oauth_token/keyring but it has no gho_ prefix. Another is a gho_ token supplied through GH_TOKEN, which refresh.go:169 refuses to refresh. A third is using the default host instead of the host of the failed request. It is advice text only: no secret, permission or stored data is touched.", + "coordinated_surfaces": "The change sits in the 401 branch of internal/ghcmd/cmd.go:228-230. That branch has to read the factory's auth config (ActiveToken, a source string, a token prefix) and the request host from api.HTTPError. It may also pull in a small shared helper next to the existing checks in agent_task.go and shared/writeable.go. Two or three places in one component.", + "ambiguity": "The tempting fix checks only that the source is \"oauth_token\"/\"keyring\". That would suggest refresh for PATs pasted in with `--with-token`, which `gh auth login` did not create. Checking only the gho_ prefix fails the other way, for env-var tokens that refresh.go:169 rejects. Using the default host or leaving out `-h` is another plausible but wrong version. Every one of these would pass the two tests the task asks for and a quick review.", + "verification": "internal/ghcmd/cmd_test.go has no test for the 401 branch of Main or for the \"Try authenticating with\" text. It only tests printError, newIOStreams_* and mightBeGHESUser. Acceptance tests need real credentials. Nothing would catch a wrong classification today, and the tests the task asks for cover only the two easy cases.", + "blast_radius": "This is advice printed to stderr after a 401 in internal/ghcmd/cmd.go, and every gh user can see it. Wrong advice is annoying but harmless, and a revert fully fixes it: no credentials, config or stored data change." + }, + "confidence": "medium", + "summary": "In internal/ghcmd/cmd.go, the 401 branch should print `gh auth refresh` (probably with `-h `) only when the active token for the host of the failed request was stored by gh and is an OAuth token. That is the oauth_token/keyring source plus gho_ prefix rule from agent_task.go; every other token keeps `gh auth login`. New tests are needed because nothing covers this branch today, and they should cover the tricky cases (a `--with-token` PAT, a gho_ token from an env var, a non-default host), not just the obvious pair." + }, + { + "task": "codespace-ports-forward-concurrent-map", + "attempt": 1, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:11:39.610653Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3594, + "cache_read_tokens": 139383, + "cache_creation_tokens": 28961, + "num_turns": 7, + "latency_seconds": 45.1, + "total_cost_usd": 0.26908816 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour, but the cause sits in dev-tunnels v0.1.19 manager.go: CreateTunnelPort writes options.AdditionalHeaders[\"If-Not-Match\"]=\"*\" and replaces tunnel.Ports. Both live on objects shared through connection.CodespaceConnection (Options, Tunnel, TunnelManager). The task does not name these. You find them by reading the library and connection.go.", + "silent_invariant": "This is a concurrency invariant: one CodespaceConnection is shared by every errgroup.Go goroutine in ports.go ForwardPorts. Failures are intermittent, as the task says. A partial fix, such as cloning the header map without guarding the tunnel.Ports write in CreateTunnelPort, would stop the crash but still race, and this only shows under -race or particular timing.", + "coordinated_surfaces": "The fix belongs in one component. The access to shared tunnel state needs to be synchronized in port_forwarder.go ForwardPort, connection.go or ports.go, plus a new multi-port test in port_forwarder_test.go or ports_test.go. No API, config or docs changes.", + "ambiguity": "There are tempting narrower fixes: copy the options map per call (this leaves the tunnel.Ports race), make ForwardPorts run sequentially (this changes how the command blocks, because ForwardPortToListener never returns), or upgrade dev-tunnels (v0.2.0 rewrites header handling but still mutates the shared tunnel). The 400 Bad Request and Connection refused symptoms hint at shared-state problems beyond the map write.", + "verification": "CI runs `go test -race -tags=integration ./...` (.github/workflows/go.yml:32), and connection.NewMockHttpClient plus tunnels_api_server_mock.go can stand in for the tunnel service. But existing tests (TestForwardPortDefaultsToHTTPProtocol, TestForwardPortRespectsProtocolOfExistingTunneledPorts) only forward one port, and the race detector only catches races a test actually runs into.", + "blast_radius": "`gh cs ports forward` is a user-facing CLI command. A wrong fix means crashes or failed forwards that users would notice. Reverting fixes it, and no stored data or credentials are touched." + }, + "confidence": "medium", + "summary": "The fix has to synchronize the concurrent ForwardPort calls in internal/codespaces/portforwarder so that dev-tunnels' CreateTunnelPort no longer races on the shared TunnelRequestOptions.AdditionalHeaders map and Tunnel.Ports, while single-port forwarding keeps working. It also needs a multi-port test, using the existing mock tunnel server, that fails under go test -race before the fix." + }, + { + "task": "codespace-ports-forward-concurrent-map", + "attempt": 2, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:15:53.448713Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 3691, + "cache_read_tokens": 170482, + "cache_creation_tokens": 31625, + "num_turns": 8, + "latency_seconds": 48.5, + "total_cost_usd": 0.29272474 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The expected behaviour is stated, but the shared state lives inside dev-tunnels v0.1.19 Manager.CreateTunnelPort: it mutates options.AdditionalHeaders[\"If-Not-Match\"] and rewrites tunnel.Ports. Both come from the shared connection.CodespaceConnection (Options and Tunnel pointers in internal/codespaces/connection/connection.go). You have to read the dependency and the connection code to see what must be synchronized.", + "silent_invariant": "This is a concurrency invariant. NewPortForwarder copies the connection by value (connection: *codespaceConnection), but the pointers to Manager, Tunnel, Options and TunnelClient stay shared. A mutex on each forwarder, or copying only Options, can look fixed and pass intermittent manual runs while the tunnel.Ports slice race or the GetTunnelPort/RefreshPorts ordering races remain. These show up only under timing or -race.", + "coordinated_surfaces": "Changes stay inside the codespaces area but have to agree with each other: internal/codespaces/portforwarder/port_forwarder.go (ForwardPort) and/or internal/codespaces/connection/connection.go (shared state and locking), ForwardPorts in pkg/cmd/codespace/ports.go, and likely internal/codespaces/connection/tunnels_api_server_mock.go, whose ports map has no locking, so a concurrent test can work.", + "ambiguity": "Several fixes are plausible: lock on the shared connection, copy options per call, or create tunnel ports one at a time before starting listeners concurrently. Narrow fixes such as cloning AdditionalHeaders or a lock per forwarder stop the 'concurrent map writes' crash but leave the tunnel.Ports race and the 400 / connection-refused symptoms the task also reports. Serializing everything wrongly would block the listeners.", + "verification": "CI runs `go test -race -tags=integration ./...` (.github/workflows/go.yml), and the mock tunnels server can serve ForwardPort. But no existing test forwards several ports (pkg/cmd/codespace/ports_test.go only covers list, visibility and pending operations), the mock server's port map is unsynchronized, and the 400 / connection-refused failures come from the real service and cannot be reproduced locally.", + "blast_radius": "This is the user-facing `gh cs ports forward` command. A wrong fix means crashes, hangs or failed forwards that users would notice. No persisted data or credentials are involved, and a release revert fixes it." + }, + "confidence": "medium", + "summary": "Concurrent per-port goroutines in ForwardPorts share one CodespaceConnection, whose Options map and Tunnel.Ports slice are mutated by dev-tunnels CreateTunnelPort. The fix needs synchronization at the shared connection level (or serialized port creation), plus a multi-port test against the mock tunnels server, which may itself need locking." + }, + { + "task": "codespace-ports-forward-concurrent-map", + "attempt": 3, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:19:57.066188Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 5254, + "cache_read_tokens": 156837, + "cache_creation_tokens": 36250, + "num_turns": 7, + "latency_seconds": 59.8, + "total_cost_usd": 0.34953754 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The shared map is not in this repo's code. It is dev-tunnels v0.1.19 manager.go: CreateTunnelPort/UpdateTunnelPort write options.AdditionalHeaders[\"If-Not-Match\"/\"If-Match\"] and rebuild tunnel.Ports, and NewCodespaceConnection hands one shared *TunnelRequestOptions and *Tunnel to every forwarder. The library source in the module cache shows this, but the task text does not.", + "silent_invariant": "This is a data race that only shows up under concurrent timing. Every errgroup goroutine in ForwardPorts calls portforwarder.NewPortForwarder(ctx, codespaceConnection) and gets its own forwarder over shared Options/Tunnel pointers, so a mutex on CodespacesPortForwarder looks right but does nothing. A lock around CreateTunnelPort alone still lets GetTunnelPort read concurrently, and the header leakage stays. Single-port runs and most multi-port runs look fine.", + "coordinated_surfaces": "Changes stay in the codespaces component. They need to agree across internal/codespaces/portforwarder/port_forwarder.go and/or internal/codespaces/connection/connection.go (synchronization or per-call option copies), pkg/cmd/codespace/ports.go ForwardPorts, and a new test. The mock server's specifiedPorts map writes in tunnels_api_server_mock.go may also need to be made safe for concurrent requests.", + "ambiguity": "The task points at the CreateTunnelPort map write, so serializing only that call or adding a per-forwarder mutex would fix the visible 'concurrent map writes' crash. It would still leave reads racing, tunnel.Ports changing, and the shared AdditionalHeaders state carrying over between requests, which is a plausible source of the reported 400 Bad Request. The task also asks for 'no data race', which these narrower fixes do not deliver.", + "verification": "No existing test forwards multiple ports. ports_test.go TestPendingOperationDisallowsForwardPorts stops early, and port_forwarder_test.go covers a single ForwardPort. Reusable pieces exist: CI runs `go test -race -tags=integration ./...` (.github/workflows/go.yml), connection.NewMockHttpClient plus a mock relay server, and apiClientMock.ExternalHTTPClient. The race detector is nondeterministic, though.", + "blast_radius": "This is a user-facing `gh cs ports forward` crash or failure. A bad fix would cause failed or deadlocked forwarding that users would notice. Nothing is persisted and no credentials are involved, so a revert fixes it." + }, + "confidence": "medium", + "summary": "ForwardPorts runs one goroutine per port, and all of them share a single CodespaceConnection whose TunnelRequestOptions.AdditionalHeaders map and Tunnel.Ports are changed by dev-tunnels Manager calls, so the fix needs synchronization (or per-call copies) at the shared connection or in ForwardPorts, not inside each per-goroutine forwarder. It also needs a new test that forwards several ports at once against the existing mock tunnels server, which CI runs under -race." + }, + { + "task": "copilot-not-installed-trailing-newline", + "attempt": 1, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:10:13.472108Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 1797, + "cache_read_tokens": 57011, + "cache_creation_tokens": 22092, + "num_turns": 4, + "latency_seconds": 21.8, + "total_cost_usd": 0.17362642 + }, + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task text gives the exact message, the expected trailing newline, and says the rest of the declined-install behaviour stays the same (return cmdutil.SilentError at pkg/cmd/copilot/copilot.go:156). An acceptance test can be written from the task alone.", + "silent_invariant": "The change is to stderr formatting in runCopilot (copilot.go:155, fmt.Fprintf with no \\n). It doesn't touch auth, secrets, persisted data or concurrency, and a wrong result shows up immediately as glued or extra output.", + "coordinated_surfaces": "One place: the Fprintf at pkg/cmd/copilot/copilot.go:155, plus a new test in copilot_test.go. No docs, config or other commands use this string.", + "ambiguity": "The root cause is named: the warning lacks a trailing newline. Line 159 ('Copilot CLI not installed', the path where gh can't prompt) has the same flaw, but the task scopes the fix to the declined-install path, and line 337 ('installed successfully\\n') shows the house style. The obvious fix is the right one.", + "verification": "copilot_test.go has no test for the declined-confirm path. TestRunCopilot_execFailureHint only covers the found-binary path, and TestDownloadCopilot only uses require.Contains. Test hooks do exist (findCopilotBinaryFunc, the Prompter on CopilotOptions, iostreams.Test), and the task requires an exact-output test, so coverage is easy to add but missing today.", + "blast_radius": "gh copilot is a user-facing command, so users see a cosmetic terminal glitch when the output is wrong. A revert fixes it with no data or security impact." + }, + "confidence": "high", + "summary": "Add a trailing newline to the 'Copilot CLI was not installed' warning in runCopilot (pkg/cmd/copilot/copilot.go:155) and keep returning cmdutil.SilentError. Add a runCopilot test that stubs findCopilotBinaryFunc to return \"\", makes the mocked Prompter decline, and asserts the exact stderr output." + }, + { + "task": "copilot-not-installed-trailing-newline", + "attempt": 2, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:13:58.824576Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 1674, + "cache_read_tokens": 56058, + "cache_creation_tokens": 21222, + "num_turns": 4, + "latency_seconds": 20.1, + "total_cost_usd": 0.16592575999999998 + }, + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the exact message, the expected newline and the reproduction steps. The behaviour is fully defined by copilot.go:155, `fmt.Fprintf(opts.IO.ErrOut, \"%s Copilot CLI was not installed\", ...WarningIcon())` followed by `return cmdutil.SilentError`. No outside contract is involved.", + "silent_invariant": "This is cosmetic stderr formatting on the declined-install branch of runCopilot. No auth, secrets or data are involved, and a wrong result shows up immediately in the terminal or in an exact-output test.", + "coordinated_surfaces": "One line in pkg/cmd/copilot/copilot.go (Fprintf to Fprintln, or add \\n), plus a new test in copilot_test.go. No docs, config or other commands need to change.", + "ambiguity": "The sibling non-TTY branch at copilot.go:159 (`\"%s Copilot CLI not installed\"`) has the same missing newline, so it is unclear whether to fix it too. The task also says the rest of the declined behaviour should stay the same, which means SilentError and the message text must be kept. Once the code is read, the intended fix is clear.", + "verification": "copilot_test.go has no test for the declined-prompt path: TestRunCopilot_execFailureHint covers only the binary-found case, and TestDownloadCopilot uses only Contains. The hooks needed for a new exact-output test already exist: findCopilotBinaryFunc, Prompter mocks and iostreams.Test stderr.", + "blast_radius": "This is user-facing output of `gh copilot`. A mistake would be visible but cosmetic, and a revert would fix it with no lasting harm." + }, + "confidence": "high", + "summary": "The change is to add a trailing newline to the declined-install warning at pkg/cmd/copilot/copilot.go:155, keeping the SilentError return, and possibly the matching non-TTY message at line 159. It also needs a new runCopilot test that stubs findCopilotBinaryFunc and the Prompter to decline, then asserts the exact stderr output." + }, + { + "task": "copilot-not-installed-trailing-newline", + "attempt": 3, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:18:11.845563Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2245, + "cache_read_tokens": 56715, + "cache_creation_tokens": 21791, + "num_turns": 4, + "latency_seconds": 39.8, + "total_cost_usd": 0.1817618 + }, + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the exact string ('! Copilot CLI was not installed' plus a trailing newline), the trigger (declining the Confirm prompt) and says the rest stays the same. That matches copilot.go:155, `fmt.Fprintf(opts.IO.ErrOut, \"%s Copilot CLI was not installed\", ...WarningIcon())` followed by `return cmdutil.SilentError`. The acceptance test can be written from the task text alone.", + "silent_invariant": "This is a cosmetic stderr newline in the declined-install branch of runCopilot. It touches no auth, secrets, persisted data or concurrency, and a mistake shows up directly in terminal output or in an exact-match stderr assertion.", + "coordinated_surfaces": "Only one line of production code changes: copilot.go:155, either Fprintf with \\n or Fprintln, as already done at line 141 and with \\n at line 337. The required test goes in copilot_test.go. No docs, config or other commands are involved.", + "ambiguity": "The task names the exact line and the fix. The non-TTY sibling message at copilot.go:159 ('Copilot CLI not installed') also lacks a newline. The task scopes the work to the declined-install path and says the rest of the behaviour stays the same, so the obvious fix is the right one.", + "verification": "copilot_test.go has run-level tests such as TestRunCopilot_execFailureHint and the seams the new test needs (findCopilotBinaryFunc override, iostreams.Test). No existing test exercises the Confirm-declined branch or asserts its stderr. The task asks for a new exact-output test, which can be built locally with a prompter mock and iostreams.Test.", + "blast_radius": "This is user-facing gh CLI output on the `gh copilot` not-installed path. A wrong change would at worst garble a warning line or drop it. It is fixed by a revert and does no lasting harm." + }, + "confidence": "high", + "summary": "Add a trailing newline to the 'Copilot CLI was not installed' warning in runCopilot (pkg/cmd/copilot/copilot.go:155), keeping the SilentError return. Add a runCopilot test that stubs findCopilotBinaryFunc to return \"\", makes the prompter decline with a TTY iostreams, and asserts the exact stderr output." + }, + { + "task": "extension-uninstall-alias", + "attempt": 1, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:10:38.978846Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 1729, + "cache_read_tokens": 35194, + "cache_creation_tokens": 19449, + "num_turns": 3, + "latency_seconds": 30.0, + "total_cost_usd": 0.15278538 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task does not say how to add the second name, but the repo already does it with cobra `Aliases`. The parent command uses `Aliases: []string{\"extensions\", \"ext\"}` and the list command uses `Aliases: []string{\"ls\"}` in `pkg/cmd/extension/command.go`. The `remove` command at lines 460-475 has no aliases yet, so the change should follow the same convention.", + "silent_invariant": "No security or data invariant is involved. The new name calls the same `m.Remove(extName)` code (command.go:464-474). If the change is wrong, `gh extension uninstall` either fails with an unknown-command error or `remove` stops working, and both are visible right away.", + "coordinated_surfaces": "Only one place changes: the `remove` `cobra.Command` literal in `pkg/cmd/extension/command.go`. The help text and the generated manual read cobra's alias list, so they update on their own. A test case goes in `command_test.go`.", + "ambiguity": "Two tempting alternatives exist: copying the command into a separate `uninstall` command, or renaming `remove`. The task rules out the rename (\"remove must keep working unchanged\"), and the repo's alias convention points to the intended answer. \"Help output lists uninstall\" is satisfied by the ALIASES section in `gh extension remove --help`.", + "verification": "`pkg/cmd/extension/command_test.go:553-615` has table cases for `remove` that use `ExtensionManagerMock.RemoveFunc` and check the output. `acceptance/testdata/extension/extension.txtar:56` runs `gh extension remove`. Nothing tests `uninstall` yet, so a new table case is needed; the existing harness makes that easy.", + "blast_radius": "The CLI is user-facing, but adding an alias is additive. The worst realistic mistake is breaking `remove` for users' scripts, which a revert fixes completely with no lasting harm." + }, + "confidence": "high", + "summary": "Add `Aliases: []string{\"uninstall\"}` to the `remove ` cobra command in pkg/cmd/extension/command.go, so `gh extension uninstall` calls the same `m.Remove` path and shows up under ALIASES in help. Add an `uninstall` case to the existing table-driven tests in command_test.go." + }, + { + "task": "extension-uninstall-alias", + "attempt": 2, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:14:23.622279Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 1761, + "cache_read_tokens": 35280, + "cache_creation_tokens": 19294, + "num_turns": 3, + "latency_seconds": 26.3, + "total_cost_usd": 0.15265579999999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the behaviour, but how to do it comes from nearby code: in pkg/cmd/extension/command.go the sibling `list` command uses `Aliases: []string{\"ls\"}` (line 266) and the parent uses `Aliases: []string{\"extensions\", \"ext\"}`. So the expected approach is a Cobra alias on the `remove ` command (lines 460-475), not a duplicated command. Cobra prints aliases in the subcommand's help, which is what 'help output lists uninstall' relies on.", + "silent_invariant": "The alias goes through the same RunE: normalizeExtensionSelector, then m.Remove, then the TTY success message. There is no auth, secret, concurrency or persisted-data invariant. A missing or wrong alias fails visibly with 'unknown command'.", + "coordinated_surfaces": "One place: the `remove` cobra.Command in pkg/cmd/extension/command.go. Help text, completion and the generated manual all come from Cobra metadata, so nothing else has to change by hand. A test row goes in pkg/cmd/extension/command_test.go.", + "ambiguity": "There are two reasonable readings: a separate `uninstall` command that copies remove's RunE, or a Cobra alias. Once you see the existing `ls` alias at command.go:266, the alias is clearly intended. A separate command would also show `uninstall` as its own entry in the parent's command list, which the alias does not. The repo convention settles which to choose.", + "verification": "pkg/cmd/extension/command_test.go has table-driven cases for remove ('remove extension tty', 'nontty', 'gh-prefix', 'full name', lines 553-610) that stub em.RemoveFunc. None calls `uninstall`, so existing tests would not catch a missing alias. Adding one more row with args {\"uninstall\",\"hello\"} is trivial.", + "blast_radius": "This is user-facing CLI behaviour. If done wrong, `uninstall` would be missing or `remove` could regress for existing scripts. The task warns that changing `remove` would break scripts. Reverting the release fixes it, and no data or security is involved." + }, + "confidence": "high", + "summary": "Add `uninstall` to the Aliases of the `remove ` cobra.Command in pkg/cmd/extension/command.go, following the existing `ls` alias on `list`, so both names share the same RunE and the subcommand help lists it. Add a table-driven case in pkg/cmd/extension/command_test.go that runs `uninstall hello` and asserts that em.RemoveFunc is called and the output matches remove's." + }, + { + "task": "extension-uninstall-alias", + "attempt": 3, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:18:51.725522Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 1973, + "cache_read_tokens": 73489, + "cache_creation_tokens": 20220, + "num_turns": 5, + "latency_seconds": 40.5, + "total_cost_usd": 0.17083637999999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task text defines the behaviour, but the expected form (a cobra alias rather than a second command) comes from the repo's own convention. The sibling `list` command uses `Aliases: []string{\"ls\"}` (command.go:266), and pkg/cmd/root/help.go:132-133 already prints an ALIASES help section through BuildAliasList.", + "silent_invariant": "No auth, secrets or persisted-data invariants are involved. `uninstall` calls the same `m.Remove(extName)` path (command.go:464-473). A wrong change is easy to see: either `gh ext uninstall` fails as an unknown command or the help has no ALIASES line.", + "coordinated_surfaces": "One place: the `remove` cobra.Command at pkg/cmd/extension/command.go:460-475. Help text and generated docs come from cobra metadata through help.go and need no separate edit. Tests go in command_test.go.", + "ambiguity": "Someone could copy the remove command into a separate `uninstall` command, which would duplicate logic and could drift from `remove`. The existing `ls` alias pattern and the task's 'does exactly what remove does' make an alias the clear intended answer.", + "verification": "command_test.go:553-610 has table-driven cases for `remove` (tty, nontty, gh-prefix, full name), but none run the `uninstall` path or check that help lists the alias. A new table case is easy to add, but nothing would catch a mistake today.", + "blast_radius": "User-facing CLI help and command routing. The only risk is breaking `remove` or the alias. No data or security impact, and a revert fixes it completely." + }, + "confidence": "high", + "summary": "Add `uninstall` as a cobra alias on the existing `remove` subcommand in pkg/cmd/extension/command.go, following the `list`/`ls` pattern, so routing and the ALIASES help section come for free. Add a table-driven case to pkg/cmd/extension/command_test.go that runs `uninstall` and checks that Remove is called with the normalized name." + }, + { + "task": "issue-view-comments-with-json", + "attempt": 1, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:12:04.046443Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2023, + "cache_read_tokens": 61180, + "cache_creation_tokens": 23803, + "num_turns": 4, + "latency_seconds": 27.8, + "total_cost_usd": 0.18892609999999999 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task lists both acceptable behaviours. Doing it right relies on local conventions: cmdutil.MutuallyExclusive (used in pkg/cmd/issue/develop/develop.go and edit.go), and the fact that jsonExporter.Write filters output to Exporter.Fields(), where the Exporter interface has no SetFields.", + "silent_invariant": "In view.go lines 109-118, comments are only added to lookupFields when Exporter is nil. A fix that only adds 'comments' to lookupFields would fetch comments (preloadIssueComments runs) while Exporter.Write still leaves them out, so it looks fixed until the output is checked. --jq and --template also set the Exporter and are easy to miss. No security or data-integrity risk.", + "coordinated_surfaces": "Changes stay inside pkg/cmd/issue/view: flag validation or field merging in NewCmdView/viewRun (view.go), plus new cases in view_test.go. Help text may change if the flags become exclusive. pr view has the same pattern but is not in scope.", + "ambiguity": "Task names two acceptable designs and that --json alone and --comments alone must keep working. The tempting half-fix of adding 'comments' to lookupFields without the exporter's field list shows up once you read jsonExporter.Write.", + "verification": "view_test.go covers 'with comments flag' and the JSON fields (jsonfieldstest.ExpectCommandToSupportJSONFields) but has no --comments plus --json case. The required new tests run locally with httpmock; the existing tests would not catch a wrong combined behaviour.", + "blast_radius": "User-facing CLI output that scripts and agents use; a mistake changes JSON output or rejects valid commands, but a release revert fixes it with no stored data affected." + }, + "confidence": "high", + "summary": "In pkg/cmd/issue/view/view.go, either reject --comments together with --json (cmdutil.MutuallyExclusive in RunE), or merge 'comments' into both the fetched fields and the exporter's output fields, with tests in view_test.go. Using either flag alone must keep working as it does today." + }, + { + "task": "issue-view-comments-with-json", + "attempt": 2, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:16:03.761393Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2853, + "cache_read_tokens": 65788, + "cache_creation_tokens": 29165, + "num_turns": 4, + "latency_seconds": 41.2, + "total_cost_usd": 0.23769086 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task names both acceptable behaviours, but how to build either one comes from nearby code. For the error option, the convention is `cmdutil.MutuallyExclusive` / `FlagErrorf` (AGENTS.md; `discussion/view/view.go` line 155 rejects --comments with --web this way). For the merge option, the `cmdutil.Exporter` interface only has `Fields()` and `Write()`. `SetFields` exists only on the unexported `jsonExporter` (json_flags.go lines 198-220), and the exporter writes only its own fields. So adding `comments` to the fetch list alone (`lookupFields`, view.go lines 108-118) still leaves comments out of the output.", + "silent_invariant": "No auth, secrets or persisted data are involved; the code only shapes read-only GraphQL fetches and JSON output. A naive merge that fetches comments (`preloadIssueComments`, view.go line 153) without adding them to the exporter's fields shows up in plain JSON output. Edge cases a quick check would miss: `comments` already in the --json list (duplicate field), and the --jq/--template paths.", + "coordinated_surfaces": "Changes stay in `pkg/cmd/issue/view/view.go` (flag validation in RunE, or field merging in viewRun) plus `view_test.go`, and possibly the --comments help text. `pr view` (pkg/cmd/pr/view/view.go lines 100-133) has the same flaw but the task does not cover it. No other component, schema or deployment is touched.", + "ambiguity": "The task offers two valid designs (merge `comments` into the JSON fields, or a mutual-exclusion error), so a choice is needed. The repo makes either one clear: the `discussion view` `needsComments` helper (line 228) merges, and `cmdutil.MutuallyExclusive` handles the error option. The tempting incomplete fix is adding `comments` only to `lookupFields`; the output would then still lack comments.", + "verification": "`view_test.go` has `TestIssueView_tty_Comments`, `TestIssueView_nontty_Comments` and `--json` tests (`TestIssueView_json_IssueType`, etc.) using httpmock with `CommentsForIssue` fixtures. None of them tests --comments and --json together, so today nothing would catch a wrong change. New tests are easy to write locally with the existing `runCommand` helper and fixtures.", + "blast_radius": "This is user-facing CLI output that scripts and agents consume (the task says so), but `gh issue view` is read-only. A wrong change, such as a new error breaking `--json` alone, would be noticed and fixed by a revert, with no lasting harm." + }, + "confidence": "high", + "summary": "In pkg/cmd/issue/view/view.go, either reject --comments with --json via cmdutil.MutuallyExclusive in RunE, or add `comments` to both the fetched fields and the exporter's output fields (the Exporter interface has no SetFields). Either way --json alone and --comments alone must not change. Add tests in view_test.go using the existing httpmock fixtures for the combined case, the case where `comments` is already in the --json list, and the unchanged single-flag paths." + }, + { + "task": "issue-view-comments-with-json", + "attempt": 3, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:20:06.900830Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 2212, + "cache_read_tokens": 36876, + "cache_creation_tokens": 23653, + "num_turns": 3, + "latency_seconds": 29.6, + "total_cost_usd": 0.18690342 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives both acceptable outcomes. Getting either one right depends on repo conventions it does not mention: `cmdutil.MutuallyExclusive` and `FlagErrorf` (AGENTS.md 'Error Handling'), and the fact that `opts.Exporter` is filled in `cmd.PreRunE` by `setupJSONFlags` (json_flags.go:74), which runs before RunE. `viewRun` (view.go:108-118) only adds `comments` in the non-JSON branch, and the exporter writes only `Exporter.Fields()`.", + "silent_invariant": "No security, data or concurrency concern. There is one easy-to-miss trap. If `comments` is added only to `lookupFields`, `preloadIssueComments` still runs (view.go:152) and fetches the comments, but `Exporter.Write` (view.go:182) still filters to the user's `--json` fields, so the comments never appear in the output. Also, a validation check placed in the constructor before PreRunE has run would never fire. Both mistakes only show up when the specific flag combination is tested.", + "coordinated_surfaces": "Only the `pkg/cmd/issue/view` package changes: either the flag handling in `NewCmdView` RunE or the field selection in `viewRun`, plus `view_test.go`. The `--comments` help text may also need updating if the flags become mutually exclusive. `pr view` (pkg/cmd/pr/view/view.go:78) has a parallel `--comments` flag, but the task does not cover it.", + "ambiguity": "The task names two acceptable behaviours ('adds comments to the requested JSON fields' or 'mutually exclusive... clear validation error'), so there is a choice to make. The repo's `MutuallyExclusive` helper and `Exporter.SetFields` make either one clear to implement. The tempting partial fix (lookupFields only) is ruled out by the task's expected JSON output.", + "verification": "view_test.go has `TestNewCmdView`, `TestIssueView_nontty_Comments`, `TestIssueView_json_*` and httpmock fixtures, including `CommentsForIssue`. None of them exercises `--comments` together with `--json`, so the bug passes today. The task requires new tests, and they are easy to write with the existing `runCommand` and httpmock helpers.", + "blast_radius": "The `gh issue view` output is user-facing and used by scripts and agents. A wrong change could break `--json` on its own or `--comments` on its own, which users would notice, but no data, credentials or persisted state are involved, and reverting the change fixes it." + }, + "confidence": "high", + "summary": "In `pkg/cmd/issue/view/view.go`, `--comments` combined with `--json` must either add `comments` to the exporter's field list (so the comments are fetched and written) or fail with a mutually-exclusive flag error after PreRunE has set up the Exporter. `--json` alone and `--comments` alone must behave as they do today. New table tests in `view_test.go` should cover the combination using the existing httpmock fixtures." + }, + { + "task": "pr-checks-all-cancelled-summary", + "attempt": 1, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:10:13.213738Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2063, + "cache_read_tokens": 56532, + "cache_creation_tokens": 21139, + "num_turns": 4, + "latency_seconds": 43.4, + "total_cost_usd": 0.17413153999999997 + }, + "scores": { + "hidden_contract": 0, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the exact expected text ('Some checks were cancelled'), the scenario (only cancelled checks) and that other wording must not change. The message and the line of counts already exist in pkg/cmd/pr/checks/output.go printSummary (lines 76-83), so nothing outside the task defines the behaviour.", + "silent_invariant": "This is display-only text in printSummary (output.go). A wrong change shows up as wrong or blank text on screen and in TestChecksRun's exact heredoc output comparisons. No auth, data, caching or concurrency is involved, and the exit code in checks.go (lines 248-254) does not depend on the summary.", + "coordinated_surfaces": "One place: the guard on output.go line 71 (`counts.Failed+counts.Passed+counts.Skipping+counts.Pending > 0` leaves out counts.Canceled). The fixture and test in the same package are the requested regression test, not separate surfaces that must agree.", + "ambiguity": "The task names the symptom, and the cause is plain in output.go line 71: the guard leaves out Canceled while the branch at line 76 already handles it. Adding Canceled to the guard is the obvious fix and the right one, and it leaves the other states' wording unchanged.", + "verification": "checks_test.go TestChecksRun compares exact TTY output for allPassing, someFailing, somePending, someCancelled and someSkipping fixtures, so changes to other wording would be caught. There is no all-cancelled fixture or case in pkg/cmd/pr/checks/fixtures, so today nothing covers the failing path until the requested test is added.", + "blast_radius": "User-facing `gh pr checks` TTY output (printSummary only prints when io.IsStdoutTTY()). A mistake would be visible text that a revert fixes, with no stored data or security impact." + }, + "confidence": "high", + "summary": "Include counts.Canceled in the guard in printSummary (pkg/cmd/pr/checks/output.go) so an all-cancelled pull request reaches the existing 'Some checks were cancelled' branch. Add an all-cancelled fixture and a TestChecksRun case that checks the exact TTY summary output." + }, + { + "task": "pr-checks-all-cancelled-summary", + "attempt": 2, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:14:21.975101Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 1883, + "cache_read_tokens": 34889, + "cache_creation_tokens": 20588, + "num_turns": 3, + "latency_seconds": 30.1, + "total_cost_usd": 0.16237078 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives only the headline text. The full expected output also includes the tallies line built in output.go:82-85 ('%d cancelled, %d failing, ...'), plus the table format and the requirement that the summary only prints on a TTY (output.go:88). The sibling test 'some cancelled tty' in checks_test.go:188-201 shows what the regression test should assert.", + "silent_invariant": "This is pure display logic in printSummary (output.go:69-92), with no auth, persistence or concurrency involved. A wrong change shows up as wrong or blank text in the TTY output, which the heredoc assertions in checks_test.go catch.", + "coordinated_surfaces": "The fix is one guard condition in output.go:71. A new fixture (e.g. fixtures/allCancelled.json) and a test table row in checks_test.go go with it. No docs, flags, JSON fields or other commands change.", + "ambiguity": "The root cause is clear from output.go:71, where the guard leaves out counts.Canceled. Adding it lets the existing else-if order (Failed > Pending > Canceled > success) handle the all-cancelled case. Every other mix keeps its wording because Canceled only takes effect when Failed and Pending are zero. A special-cased all-cancelled branch would be redundant, not a tempting but incomplete answer.", + "verification": "checks_test.go has strong table-driven heredoc assertions for every other mix (allPassing, someFailing, somePending, someCancelled, someSkipping), so changes to other wordings would be caught. No test or fixture covers the all-cancelled case, so the target behaviour is only checked once the requested regression test is added.", + "blast_radius": "This is the user-facing TTY output of `gh pr checks`. It is cosmetic and does not touch exit codes or JSON export (checks.go and aggregate.go are unchanged), so a revert would fully fix a mistake." + }, + "confidence": "high", + "summary": "Add counts.Canceled to the guard in printSummary (pkg/cmd/pr/checks/output.go:71) so an all-cancelled PR reaches the existing 'Some checks were cancelled' branch and prints the tallies line. Add a fixture with only cancelled checks and a TTY table row in checks_test.go that asserts the summary text." + }, + { + "task": "pr-checks-all-cancelled-summary", + "attempt": 3, + "ref": "v2.95.0", + "judged_at": "2026-10-14T22:18:46.287496Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 2199, + "cache_read_tokens": 76773, + "cache_creation_tokens": 21414, + "num_turns": 5, + "latency_seconds": 30.1, + "total_cost_usd": 0.18309785999999997 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 0, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives only the summary sentence. The tally line that should follow it ('1 cancelled, 0 failing, 0 successful, 0 skipped, and 0 pending checks', built in output.go:82-85) and the fixture-driven test format come from sibling cases in pkg/cmd/pr/checks/checks_test.go such as 'some cancelled tty', which use fixtures/someCancelled.json.", + "silent_invariant": "printSummary in pkg/cmd/pr/checks/output.go only formats display text, and only on a TTY (lines 88-91). A wrong change shows up as wrong or blank output. There is no auth, data or concurrency involved; exit codes come separately from counts.Failed/Pending in checks.go:248-252.", + "coordinated_surfaces": "The logic change is one guard condition in printSummary (output.go:71). The regression test is one new row in the checks_test.go table plus a fixture JSON in pkg/cmd/pr/checks/fixtures/. Both callers (checks.go:212 and :241) pick up the fix automatically.", + "ambiguity": "The root cause is clear from output.go:71: the guard leaves out counts.Canceled, and the existing 'Some checks were cancelled' branch at line 76 is never reached. The obvious fix, adding Canceled to the guard, is the right one and keeps the other wording because the if/else order already puts failed and pending first.", + "verification": "checks_test.go assertions compare the full TTY output (summary, tallies and table) for passing, failing, pending, some-cancelled and skipped mixes, so a change to the other wordings would be caught. But nothing covers the all-cancelled path today, which is how this bug shipped. It is caught only if the new required test row and fixture are written correctly.", + "blast_radius": "This is the user-facing `gh pr checks` output. A mistake would show as cosmetic summary text in a terminal, with no stored data or security impact, and a revert fixes it completely." + }, + "confidence": "high", + "summary": "The fix is to add counts.Canceled to the guard in printSummary (pkg/cmd/pr/checks/output.go) so that all-cancelled PRs reach the existing 'Some checks were cancelled' branch and its tally line. It needs a new all-cancelled fixture and a test row in checks_test.go that asserts the full summary output." + }, + { + "task": "pr-comment-attach-ghu-tokens", + "attempt": 1, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:11:20.132222Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3930, + "cache_read_tokens": 179808, + "cache_creation_tokens": 42311, + "num_turns": 7, + "latency_seconds": 100.1, + "total_cost_usd": 0.35879866 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The behaviour comes from nearby code, not an outside spec. The task states the server contract (HTTP 201 with the `token` scheme), and `api/http_client.go:185` already sends `Authorization: token %s`. The rest is local convention: `internal/attachments/client.go:71-85` documents `uploadTokenTypes` as 'an allowlist, so an unlisted kind is rejected rather than sent', and `doc.go` requires the Uploader to be built before any prompt. The test row 'rejects an App user-to-server token' in `client_test.go:432` currently pins the opposite behaviour.", + "silent_invariant": "The allowlist is a fail-fast gate before an upload that cannot be undone (`doc.go`: 'An upload cannot be undone'). A wrong fix, such as dropping `checkUploadTokenType` or adding all `gh.TokenTypes`, would still look like it works for `ghu_`, while letting `ghs_`, `ghr_` and unknown credentials through to the endpoint. The token still only goes to the same GitHub uploads host, so nothing leaks; the damage shows only for other token kinds.", + "coordinated_surfaces": "There is one place to change: `uploadTokenTypes` in `internal/attachments/client.go:72-76`. `pr comment` (`pkg/cmd/pr/shared/commentable.go:179-180`), pr and issue create/edit all pass `ActiveTokenType` into the same `NewUploader`, so they pick up the change automatically. `ActiveTokenType` already maps `ghu_` to `TokenTypeUserToServer` (`internal/config/auth_config_test.go:961`), and no help text or docs list token kinds.", + "ambiguity": "The task names `gh pr comment`, which invites a special case in the comment path, and \"the server's own error comes back\" could be read as removing the pre-flight gate altogether. The repository settles both. The gate is shared and documented as an allowlist, and the 'Behaviour for the token types that already work must not change' line plus the `ghs_`/`ghr_`/unknown rejection rows show the intended fix is adding `TokenTypeUserToServer` to the list. Refused uploads already come back through `uploadError`.", + "verification": "`TestNewUploader` in `internal/attachments/client_test.go:372-552` is a table over every token type, using an httpmock registry with no stubs, so any request fails the test. Over-broad changes break the `ghs_`/`ghr_`/unknown rows, and command tests assert 'unsupported authentication type' (`pr/create/create_test.go:1859`, `pr/edit/edit_test.go:1272`, `pr/shared/commentable_test.go:363`). Whether the live server accepts `ghu_` cannot be tested without acceptance credentials, but the task supplies that fact.", + "blast_radius": "This is a user-facing CLI flag used by six commands. Getting it wrong means failed or wrongly allowed attach attempts, which users would notice and a revert would fix. The token only goes to the same GitHub uploads host, so no credential leaks and no data is lost." + }, + "confidence": "medium", + "summary": "Add `gh.TokenTypeUserToServer` to the shared `uploadTokenTypes` allowlist in `internal/attachments/client.go`, keeping the allowlist design and the `ghs_`/`ghr_`/unknown rejections. Then flip the `ghu_` row in `TestNewUploader` to expect success, and add tests covering a `ghu_` upload through the comment path and the server's refusal coming back." + }, + { + "task": "pr-comment-attach-ghu-tokens", + "attempt": 2, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:15:27.561187Z", + "metrics": { + "input_tokens": 18, + "output_tokens": 3856, + "cache_read_tokens": 210647, + "cache_creation_tokens": 43349, + "num_turns": 8, + "latency_seconds": 56.9, + "total_cost_usd": 0.36967304 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 0, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task doesn't say where the refusal comes from or how the gate works. Reading the code shows a shared deny-by-default allowlist (`uploadTokenTypes` / `checkUploadTokenType` in internal/attachments/client.go) that also guards issue create/edit, pr create/edit and comments. client_test.go TestNewUploader has a deliberate row, \"rejects an App user-to-server token\" (wantErr badTokenErr), which the change must reverse. Server errors already pass through `uploadError.Error()`, and uploads already use the `token` scheme (client_test asserts \"token github-token\"). All of this is visible in nearby code; there is no outside spec.", + "silent_invariant": "The gate is a credential allowlist: \"It is an allowlist, so an unlisted kind is rejected rather than sent\" (client.go:78-79). Widening it too far, for example letting ghs_, ghr_ or TokenTypeUnknown through, or dropping the check, would still look fine with a ghu_ token in normal use. It would only show up as different behaviour for other token types. The server is still the real authority, so a mistake doesn't let the wrong caller in.", + "coordinated_surfaces": "One place in one package: add gh.TokenTypeUserToServer to `uploadTokenTypes` in internal/attachments/client.go and flip the matching row in client_test.go. Every --attach command (pkg/cmd/pr/shared/commentable.go, pr/issue create/edit) already passes `ActiveTokenType` into `NewUploader`, so none of them needs to change. No flag, help text or doc lists the allowed token types.", + "ambiguity": "The task frames it as a `gh pr comment` bug, which invites a fix only in commentable.go, or a change of auth scheme (the task already notes that Bearer returns 404). The repo makes clear the right fix is the shared allowlist, which also enables ghu_ for issue/pr create/edit, while ghs_ stays rejected (commentable_test.go \"a token that cannot upload fails before the editor opens\" uses ghs_anactionstoken). \"The server's own error comes back\" is already handled by `newUploadError`/`uploadError`.", + "verification": "internal/attachments/client_test.go TestNewUploader is a table test with a row for every token type (OAuth, PAT, fine-grained PAT, ghu_, ghs_, ghr_, Unknown). It would fail if any existing type's behaviour changed or ghu_ handling were wrong. There are upload-header tests (Authorization \"token ...\") and error-mapping tests, plus command-level rows in pkg/cmd/pr/shared/commentable_test.go, pr/create and pr/edit tests that use httpmock. All run locally with no credentials.", + "blast_radius": "This affects a user-facing CLI feature (--attach on comments, create and edit) for every user. A wrong allowlist change would wrongly reject or send credentials to uploads., but the server still checks them and a release revert undoes it. No stored data or secrets are lost (doc.go notes uploads can't be undone, but that's unchanged)." + }, + "confidence": "high", + "summary": "The fix adds GitHub App user-to-server tokens (`gh.TokenTypeUserToServer`) to the shared `uploadTokenTypes` allowlist in internal/attachments/client.go, so `NewUploader` stops refusing ghu_ before any request. The existing TestNewUploader row (and any command-level rows) should flip from rejection to acceptance, while ghs_, ghr_ and unknown tokens stay rejected and the `token` auth scheme and server-error mapping stay as they are." + }, + { + "task": "pr-comment-attach-ghu-tokens", + "attempt": 3, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:19:51.090176Z", + "metrics": { + "input_tokens": 26, + "output_tokens": 5149, + "cache_read_tokens": 348134, + "cache_creation_tokens": 43033, + "num_turns": 12, + "latency_seconds": 86.9, + "total_cost_usd": 0.42666338 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "internal/attachments/client.go documents uploadTokenTypes as a deliberate allowlist ('an unlisted kind is rejected rather than sent'), and TestNewUploader pins the ghu_ row as 'rejects an App user-to-server token' plus host-before-token ordering. The rest (ghs_/ghr_/unknown stay rejected, the 'token' scheme already set in api/http_client.go AddAuthTokenHeader) comes from reading nearby code, not an outside spec.", + "silent_invariant": "The gate is a client-side check in front of a credential; the server still enforces real authorisation. A too-broad change (admitting ghs_/unknown or removing checkUploadTokenType) would still pass the ghu_ scenario and would only show up for other token types. uploadError maps a 404 to 'requires write access', so a server refusal of ghu_ could hide the server's own error. doc.go notes that an upload cannot be undone.", + "coordinated_surfaces": "Single allowlist in internal/attachments/client.go. issue/pr create, issue/pr edit and pr/shared/commentable.go all call ActiveTokenType then NewUploader and pick up the change unchanged; the gh.TokenType constants already include TokenTypeUserToServer.", + "ambiguity": "The task says 'server's own error comes back when it refuses one' and 'token types that already work must not change'. That could tempt removing the gate entirely or changing the auth scheme (the Bearer note), but the allowlist comments and the per-type TestNewUploader rows make the intended answer clear: add only TokenTypeUserToServer.", + "verification": "TestNewUploader has a row per token type and would catch over-admitting ghs_/ghr_/unknown. The ghu_ row has to be flipped, and command tests (commentable_test.go hostTokens/uploads stubs) can exercise the path with httpmock. Real server acceptance of ghu_ cannot be checked offline, and no test covers 401/403 upload errors (only 404/422/429 in client_test.go and attach_test.go).", + "blast_radius": "User-facing --attach on five gh commands. A wrong allowlist would show up as wrong accept/reject behaviour that a revert fixes. Tokens are sent only to the same host's uploads endpoint, as they already are for gho_/ghp_, so there is no new credential exposure. Uploads already made cannot be undone (doc.go), but only uploads the server accepted would happen." + }, + "confidence": "medium", + "summary": "Add gh.TokenTypeUserToServer to uploadTokenTypes in internal/attachments/client.go, keeping ghs_/ghr_/unknown rejected and the host check first. Flip the 'rejects an App user-to-server token' row in TestNewUploader and add a command-level test with a ghu_ token that uploads, plus a check that a server refusal comes back to the user." + }, + { + "task": "pr-merge-delete-branch-owner-case", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:10:40.444316Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3038, + "cache_read_tokens": 60880, + "cache_creation_tokens": 26254, + "num_turns": 4, + "latency_seconds": 36.3, + "total_cost_usd": 0.22467060000000003 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task never says GitHub owner logins are case-insensitive. The repository's own convention shows it: `ghrepo.IsSame` (internal/ghrepo/repo.go:79-82) compares owners with `strings.EqualFold`, and sibling commands `pr close` (close.go:150) and `pr checkout` (checkout.go:146) use the API's `IsCrossRepository` field. `merge.go:592` instead uses `pr.HeadRepositoryOwner.Login != baseRepo.RepoOwner()`, so the expected behaviour follows from nearby code, not from an outside spec.", + "silent_invariant": "The `crossRepoPR` flag guards `deleteRemoteBranch` (merge.go:519), which calls `api.BranchDeleteRemote` on the base repo using the head branch name. If a fix wrongly treats a fork PR as same-repo, gh would quietly delete a same-named branch in the base repository. That would only appear with a fork PR whose branch name also exists upstream, and the existing fork test (merge_test.go:862, owner `HEAD`) covers only the plain case.", + "coordinated_surfaces": "The `crossRepoPR` value set in `NewMergeContext` (merge.go:592) feeds both `deleteRemoteBranch` (merge.go:519) and the interactive prompt text in `deleteBranchSurvey` (merge.go:666-677). The fix lives in that one spot in the merge package, plus the new regression test in merge_test.go. If the fix switches to `isCrossRepository`, it also needs adding to the `Fields` list at merge.go:571.", + "ambiguity": "There are two reasonable fixes: compare owners with `strings.EqualFold`, or fetch and use `isCrossRepository` the way `pr close` does. The second also catches a head repo with the same owner but a different name, and the repository makes both options visible. `update-branch/update_branch.go:103` has the same case-sensitive comparison, but the task limits scope to `pr merge --delete-branch`.", + "verification": "merge_test.go has many tests that stub `DELETE repos/OWNER/REPO/git/refs/heads%2F...` and one cross-repo test that skips deletion (line 862). All of them use exact-case owners (`OWNER`/`OWNER`), so none would catch the case-mismatch bug or a fix that is too permissive. The task asks for a new regression test, which can run offline with httpmock and the mock finder.", + "blast_radius": "`gh pr merge` is a widely used, user-facing command. With the current bug a remote branch is left behind, which is just clutter. A too-permissive fix could delete a branch in the upstream repository after merging a fork PR, which users would notice. GitHub lets merged-PR branches be restored, and reverting the CLI change stops it, so no lasting data loss." + }, + "confidence": "medium", + "summary": "Change the cross-repo check in `NewMergeContext` (pkg/cmd/pr/merge/merge.go:592), either by comparing owners case-insensitively as `ghrepo.IsSame` does or by using `isCrossRepository`, so that `--delete-branch` deletes the same-repo head branch on GitHub whatever the remote URL's owner casing. Add a merge_test.go regression test with a lowercase-vs-uppercase owner that expects the DELETE ref call, while fork PRs must still skip remote deletion." + }, + { + "task": "pr-merge-delete-branch-owner-case", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:14:41.228567Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 3561, + "cache_read_tokens": 81390, + "cache_creation_tokens": 23480, + "num_turns": 5, + "latency_seconds": 45.3, + "total_cost_usd": 0.22544060000000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task doesn't say how repositories should be compared. The repo already does it in internal/ghrepo/repo.go IsSame, which uses strings.EqualFold on owner and name, and finder.go fetches an `isCrossRepository` field. merge.go:592 uses neither and instead compares `pr.HeadRepositoryOwner.Login != baseRepo.RepoOwner()` with exact case. The right behaviour is clear from nearby code and needs no outside spec.", + "silent_invariant": "crossRepoPR is the only guard that stops `api.BranchDeleteRemote(apiClient, m.baseRepo, m.pr.HeadRefName)` (merge.go:519-525) from running for fork PRs. If a fix loosens it too far, gh deletes a same-named branch in the base repo for a fork PR. That happens only with a fork PR whose branch name also exists in the base repo, which a normal same-repo smoke test won't exercise. The existing cross-repo test (merge_test.go:862, HeadRepositoryOwner \"HEAD\") does catch the obvious version of that mistake.", + "coordinated_surfaces": "Fixing the crossRepoPR initialisation at merge.go:592 (and adding `isCrossRepository` to the Fields list at line 571, if that route is taken) fixes both deleteRemoteBranch and deleteBranchSurvey, plus one test in merge_test.go. One command, one component; no docs or config change.", + "ambiguity": "Two reasonable fixes: case-insensitive owner comparison (the ghrepo.IsSame / EqualFold convention), or using the PR's `isCrossRepository` field. A narrower fix in deleteRemoteBranch alone would leave the survey prompt (merge.go:671-677) wrong. The rule 'same repository' in the task's expected behaviour plus the existing helpers make the intended answer clear.", + "verification": "merge_test.go has many delete-branch tests that use httpmock with `REST(\"DELETE\", \"repos/OWNER/REPO/git/refs/heads%2Fblueberries\")` and reg.Verify, which would catch a missing DELETE, plus a cross-repo case at line 862. None of them uses mismatched owner case: every same-repo fixture uses \"OWNER\" on both sides. The task's required regression test fills that gap. The real GitHub behaviour (case-insensitive owner) is only mocked.", + "blast_radius": "`gh pr merge --delete-branch` is a widely used, user-facing command. Getting it wrong means either a leftover remote branch (today's bug) or deleting the wrong remote branch, which GitHub usually lets you restore from the PR page. Shipping a fixed release reverts the CLI change. No credentials, money or stored state are involved." + }, + "confidence": "high", + "summary": "In pkg/cmd/pr/merge/merge.go, crossRepoPR must decide 'same repository' without regard to case, either with strings.EqualFold / ghrepo.IsSame or with the PR's isCrossRepository field, so the DELETE of the remote ref runs when only the owner's case differs. A merge_test.go regression test is needed in which the head owner login and the base repo owner differ only in case and the DELETE stub is checked with reg.Verify." + }, + { + "task": "pr-merge-delete-branch-owner-case", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:18:54.756426Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3099, + "cache_read_tokens": 61106, + "cache_creation_tokens": 25672, + "num_turns": 4, + "latency_seconds": 57.7, + "total_cost_usd": 0.22286132000000003 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "That owner logins are case-insensitive is not stated in the task, but sibling code sets the convention: internal/ghrepo/repo.go IsSame and context/remote.go FindByRepo (used at merge.go:464 and :815) both compare owners with strings.EqualFold.", + "silent_invariant": "deleteRemoteBranch (merge.go:517-546) calls api.BranchDeleteRemote on baseRepo with pr.HeadRefName, guarded only by m.crossRepoPR. A fix that is too broad, such as dropping the check, would silently delete a same-named base-repo branch for fork PRs, which only shows up for fork pull requests.", + "coordinated_surfaces": "The single crossRepoPR assignment at merge.go:592 drives both deleteBranchSurvey (prompt text) and deleteRemoteBranch, so one expression changes, plus a test in merge_test.go.", + "ambiguity": "The symptom (remote branch not deleted) points at deletion, but the cause is the case-sensitive `pr.HeadRepositoryOwner.Login != baseRepo.RepoOwner()`. A case-insensitive comparison or the isCrossRepository API field would both fix it, and the repo's EqualFold pattern makes the intended choice clear.", + "verification": "merge_test.go uses httpmock registries that fail on missing or unregistered calls (DELETE repos/OWNER/REPO/git/refs/heads%2Fblueberries) and has a fork-owner test (HeadRepositoryOwner 'HEAD', line 862) guarding over-broad fixes, but no existing test covers an owner-case mismatch. The task requires adding one.", + "blast_radius": "gh pr merge is a user-facing command. Wrong behaviour either leaves branches behind or deletes a base-repo branch, which is visible and usually restorable from the PR page. A revert fixes the code with no lasting state." + }, + "confidence": "high", + "summary": "The crossRepoPR computation in pkg/cmd/pr/merge/merge.go:592 must compare the PR head owner login with the base repo owner case-insensitively (matching ghrepo.IsSame/FindByRepo), so same-repo PRs get their remote branch deleted. A merge_test.go regression test must cover an owner-case mismatch and expect the DELETE git/refs call, while fork PRs keep their branches." + }, + { + "task": "pr-view-json-number-on-issue", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:10:43.280919Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3715, + "cache_read_tokens": 136816, + "cache_creation_tokens": 33408, + "num_turns": 7, + "latency_seconds": 43.3, + "total_cost_usd": 0.29564392000000006 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected exit codes and output. The fix, though, lands in the shared finder shortcut (pkg/cmd/pr/shared/finder.go:210, 252-255, comment 'avoid hitting the API if we already have all the information'), which pr diff (diff.go:123) and pr checks (checks.go:131) also use with Fields []string{\"number\"}. Their dependence on it only shows up when you read the code around it.", + "silent_invariant": "f.prNumber is set from three paths: a numeric or '#N' selector, a PR URL (ParseURL, lines 117-120), and a branch merge ref refs/pull/N/head (lines 146-150). A fix that handles only one path, or that quietly adds API calls to diff and checks, looks fine in a basic check. No auth or data-integrity invariant is involved.", + "coordinated_surfaces": "Changes stay inside the pr package: finder.go (or view.go's field selection), plus finder_test.go and/or view_test.go. The behaviour of other finder callers that request only number (diff.go, checks.go) must stay consistent with the change.", + "ambiguity": "Two fixes make the symptom go away: delete the numberFieldOnly shortcut in finder.go for every caller, or make pr view (view.go:100-101) request a field that forces findByNumber. The task says 'the same as it does when other fields are requested' and blames 'gh answers without asking GitHub', which points to the lookup, but the scope has to be decided from the code.", + "verification": "finder_test.go 'number only' case (lines 330-347, httpStub nil, wantPR 13) pins the current no-API shortcut, and the httpmock registry rejects unregistered requests, so a finder change must update that test. view_test.go uses shared.StubFinderForRunCommandStyleTests, so view tests never exercise the finder. No existing test covers an issue number.", + "blast_radius": "This is the public gh CLI that users' scripts call (the task says scripts that check whether a number is a PR get false positives). A mistake would show up as wrong exit codes or extra API calls in pr view, diff or checks, and a release revert undoes it. No credentials or stored data are involved." + }, + "confidence": "medium", + "summary": "The numberFieldOnly early return in pkg/cmd/pr/shared/finder.go Find must stop returning a pull request for an unverified number when pr view asks for only number, so findByNumber runs and an issue number makes the command fail. Tests must cover both the issue and PR cases, and the finder 'number only' test and the diff/checks callers that request only number must be updated or kept consistent." + }, + { + "task": "pr-view-json-number-on-issue", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:14:50.040021Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3383, + "cache_read_tokens": 127173, + "cache_creation_tokens": 29867, + "num_turns": 7, + "latency_seconds": 60.3, + "total_cost_usd": 0.26674296000000003 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The fix depends on nearby code the task never mentions. The shortcut is in pkg/cmd/pr/shared/finder.go:210 and 252-254 ('avoid hitting the API if we already have all the information'). pr checks (checks.go:131) and pr diff (diff.go:123) also pass Fields: []string{\"number\"} and rely on it, and finder_test.go's \"number only\" case (httpStub: nil) pins it. Reading the surrounding code reveals all of this.", + "silent_invariant": "No auth, data or concurrency invariant is involved. A wrong fix fails visibly for the simple case (wrong exit code). It can still look right while missing edge paths: a /pull/N URL selector or a PR number taken from the branch merge ref (finder.go:147-149) also reach the shortcut, and fixing only in view.go can keep the API calls checks/diff avoid today.", + "coordinated_surfaces": "All changes stay in the pr command group. The shared finder (finder.go Find/FindOptions) and the view caller (view.go:99-103, findOptions.Fields = opts.Exporter.Fields()) must agree, plus tests in finder_test.go and/or view_test.go. No docs, config or other components change.", + "ambiguity": "The obvious fix is deleting the numberFieldOnly shortcut in finder.go, which fixes pr view. But it adds API round-trips for pr checks and pr diff and breaks the finder_test \"number only\" case. The repository shows the shortcut is there on purpose, which points to a fix aimed at pr view (or an opt-out option).", + "verification": "finder_test.go's \"number only\" case (no HTTP stubs) would catch the shortcut being deleted outright. pr view tests (view_test.go) all use shared.StubFinderForRunCommandStyleTests or a mocked Finder, so no test covers `--json number` against a real lookup or the issue-number error. New tests are needed and work with httpmock.", + "blast_radius": "This is user-facing gh CLI output and exit codes that scripts depend on (the task's false-positive scenario). A mistake could also add API calls to pr checks/diff. A release revert fixes it with no stored data or security impact." + }, + "confidence": "medium", + "summary": "The shared PR finder's number-only shortcut (finder.go:252-254) must be bypassed for `gh pr view --json number` so the PR lookup runs and errors for issue numbers. pr checks and pr diff must keep their existing shortcut, and new tests must cover both the PR and the issue-number cases." + }, + { + "task": "pr-view-json-number-on-issue", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:19:10.515035Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 2992, + "cache_read_tokens": 83847, + "cache_creation_tokens": 30627, + "num_turns": 5, + "latency_seconds": 37.5, + "total_cost_usd": 0.25277164 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task never names the cause: a deliberate optimisation in pkg/cmd/pr/shared/finder.go:210,252-254 ('numberFieldOnly ... avoid hitting the API if we already have all the information'). pkg/cmd/pr/diff/diff.go:123 and pkg/cmd/pr/checks/checks.go:131 also call Find with Fields:[\"number\"], so the right scope comes from the surrounding code.", + "silent_invariant": "For a real PR the output is {\"number\":N} whether or not the API is called, so a wrong fix only shows when an issue number is passed. No auth, data or concurrency invariant; the lookup goes through findByNumber, which already errors for non-PRs.", + "coordinated_surfaces": "All changes stay inside pkg/cmd/pr: the shortcut in finder.go (or view.go:98-102, where Fields comes from Exporter.Fields()) plus finder_test.go and/or view tests. A finder-level fix also changes diff and checks.", + "ambiguity": "Two fair readings: remove or narrow the shortcut in shared/finder.go, which affects pr diff and pr checks, or make pr view request an extra field. The existing 'number only' case in finder_test.go:329-347 pins the old shortcut, so it must change; the repo makes clear the lookup must actually ask GitHub.", + "verification": "finder_test.go is a table test with httpmock, but its 'number only' case (httpStub: nil, wantPR: 13) asserts the buggy no-API path. No case covers an issue number or a not-found response, and pr view tests use a mock finder, so a new regression test is needed.", + "blast_radius": "gh pr view/diff/checks are read-only, user-facing CLI commands that scripts rely on (per the task). A wrong change means wrong exit codes or an extra API call, and a revert fixes it with no lasting harm." + }, + "confidence": "high", + "summary": "The shared PR finder (pkg/cmd/pr/shared/finder.go) returns a synthetic PullRequest without any API call when only 'number' is requested, so gh pr view N --json number succeeds for issue numbers. The fix must make that path check with GitHub, either in the finder (which also changes pr diff and pr checks) or in pr view, and must update the finder test that pins the shortcut and add an issue-number case that expects an error." + }, + { + "task": "project-item-add-output-non-tty", + "attempt": 1, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:11:00.485217Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2107, + "cache_read_tokens": 63324, + "cache_creation_tokens": 26899, + "num_turns": 4, + "latency_seconds": 36.1, + "total_cost_usd": 0.20827378000000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says 'short, stable confirmation, ideally one that includes the identifier' but gives no exact format. The format has to come from conventions: sibling printResults in pkg/cmd/project/{create,edit,copy,close}.go print the project URL and delete.go prints 'Deleted project %d'. queries.ProjectItem exposes Id (queries.go:353-355). Nothing outside the repo constrains it.", + "silent_invariant": "No auth, data or concurrency concern. The only invariant is that --format json stays clean. The exporter branch in runAddItem returns before printResults, and TestRunAddItem_JSON checks the exact stdout, so breaking it would be visible.", + "coordinated_surfaces": "Only printResults in pkg/cmd/project/item-add/item_add.go plus item_add_test.go. No docs, schema or other component needs to change. Sibling commands share the pattern but are out of scope.", + "ambiguity": "The easy fix is to drop the IsStdoutTTY guard and print 'Added item' everywhere, but that misses the 'include the identifier' goal. A second reading would change the TTY text, which the task forbids. The item ID returned by the mutation (test fixture 'id': 'item ID') makes the intended answer clear.", + "verification": "item_add_test.go checks TTY output ('Added item\\n' with SetStdoutTTY(true)) and JSON output (TestRunAddItem_JSON). No existing test covers non-TTY output; the task asks for new ones. httpmock/gock fixtures already return the item id, so it can be tested offline.", + "blast_radius": "This is user-facing CLI output that scripts may start parsing. A wrong format, or extra output that pollutes --format json, would be noticed, but a revert fixes it with no stored-data or security effect." + }, + "confidence": "high", + "summary": "Change printResults in pkg/cmd/project/item-add/item_add.go so non-TTY runs print a stable confirmation that includes the added item's Id. TTY output stays 'Added item' and the --format json path is unchanged. Add a non-TTY test next to the existing TTY and JSON tests in item_add_test.go." + }, + { + "task": "project-item-add-output-non-tty", + "attempt": 2, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:14:53.054331Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 1841, + "cache_read_tokens": 37456, + "cache_creation_tokens": 23317, + "num_turns": 3, + "latency_seconds": 31.2, + "total_cost_usd": 0.17702102 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task leaves the exact non-TTY text open ('short, stable confirmation, ideally ... identifier'). The answer comes from gh conventions: non-TTY output should be machine-readable, and siblings like project create/edit/copy print `project.URL`. The ID would come from `queries.ProjectItem` (queries.go ID field). Nothing outside the repo fixes it.", + "silent_invariant": "No security, persistence or concurrency concerns; the mutation path is unchanged. Two mistakes would only show in less common paths: printing the confirmation before or alongside `exporter.Write` would break `--format json` parsing, and writing to stderr instead of stdout would leave redirected output empty. Normal TTY use would not reveal either.", + "coordinated_surfaces": "Only `printResults` in pkg/cmd/project/item-add/item_add.go and its item_add_test.go change. There are no docs, config or other components to keep consistent; sibling commands share the pattern but are out of scope.", + "ambiguity": "Decisions needed: the message format (bare ID vs. 'Added item '), whether TTY should also show the ID (the task says keep it as today), and whether to fix the identical TTY guard in item-create/item-archive. The task and repo conventions settle most of these.", + "verification": "item_add_test.go has TTY tests (TestRunAddItem_User/Org/Me assert exactly 'Added item\\n') and TestRunAddItem_JSON asserts exact JSON stdout. The JSON test would catch double-printing. No test covers non-TTY output without --format, so the new behaviour depends on the tests the task asks for.", + "blast_radius": "This is user-facing CLI output that scripts and agents will start parsing. A bad format would bother them, but a revert fixes it, and no data or security is involved because the mutation is unchanged." + }, + "confidence": "high", + "summary": "Change `printResults` in pkg/cmd/project/item-add/item_add.go so a non-TTY run writes a stable confirmation to stdout, ideally the added item's ID from `queries.ProjectItem`. TTY 'Added item' and the `--format json` exporter path stay unchanged, and new tests in item_add_test.go cover both modes." + }, + { + "task": "project-item-add-output-non-tty", + "attempt": 3, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:19:19.474597Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2015, + "cache_read_tokens": 59167, + "cache_creation_tokens": 22844, + "num_turns": 4, + "latency_seconds": 34.5, + "total_cost_usd": 0.18303274 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the goal (a stable non-TTY confirmation, ideally with the item ID; TTY and `--format json` unchanged) but not the exact format. That comes from gh's TTY/non-TTY conventions in nearby code: `create.go` and `close.go` print a bare `project.URL` for machines, and `queries.ProjectItem.ID()` is in `shared/queries/queries.go:647`. Nothing outside the repository fixes the format.", + "silent_invariant": "The change only touches the output in `printResults` (`item_add.go:126-133`). No auth, persisted data, caching or concurrency is involved. A wrong change shows up directly: empty stdout, changed TTY text, or broken JSON output.", + "coordinated_surfaces": "The fix is one function, `printResults` in `pkg/cmd/project/item-add/item_add.go`, plus a new non-TTY case in `item_add_test.go`. The `--format json` path (`exporter.Write`) is already separate. No docs, config or other components need to change.", + "ambiguity": "The exact non-TTY string is open (bare item ID or a labelled line), and the task only says 'ideally' include the ID. Sibling commands (`item-create.go:123`, `item-delete.go:120`, `delete.go:119`, `field-delete.go:88`) share the same early `return nil`, which might tempt someone to change them too, but the task names only item-add. The mutation already returns the item `id`, and the repository's conventions make the intended shape fairly clear.", + "verification": "`item_add_test.go` has `TestRunAddItem_User`, `TestRunAddItem_Org` and `TestRunAddItem_Me` with `SetStdoutTTY(true)`, which assert \"Added item\\n\", and `TestRunAddItem_JSON` covers the exporter. No test covers non-TTY output, so the new behaviour is unchecked until the task adds one. The gock-mocked mutation already returns `id: \"item ID\"`, so a test is easy to write.", + "blast_radius": "This is user-facing CLI output for `gh project item-add`, and scripts may start parsing the new stdout. A mistake would be noticed but fixed by a revert, with no security or data harm. The mutation itself is unchanged." + }, + "confidence": "high", + "summary": "Change `printResults` in `pkg/cmd/project/item-add/item_add.go` so that non-TTY stdout gets a short, stable confirmation (preferably the added item's ID from the mutation response), leaving the TTY \"Added item\" message and the `--format json` exporter path unchanged. Add a non-TTY test next to the existing TTY and JSON tests in `item_add_test.go`." + }, + { + "task": "project-item-edit-by-name", + "attempt": 1, + "ref": "v2.96.0", + "judged_at": "2026-10-14T22:13:09.893086Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3470, + "cache_read_tokens": 144499, + "cache_creation_tokens": 32481, + "num_turns": 7, + "latency_seconds": 45.7, + "total_cost_usd": 0.28684568000000005 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says 'mirrors item-add', so the conventions come from pkg/cmd/project/item-add/item_add.go: `Use: \"item-add []\"`, cobra.MaximumNArgs(1), client.NewOwner/NewProject with canPrompt, and FlagErrorf(\"invalid number\"). The ProjectV2 GraphQL rules also matter: an item ID is not the issue node ID that IssueOrPullRequestID returns, the value depends on the field's DataType, and dates use the format YYYY-MM-DD. Both can be read from nearby code (queries.go ProjectField, buildUpdateItem) rather than from an outside spec.", + "silent_invariant": "Resolving the URL to the project item, for example picking the wrong entry when an issue is in several projects, or matching case-insensitively when two options or fields differ only by case, could quietly update the wrong item or option. That only shows up with unusual data; the normal path fails loudly at the API. No auth, secrets or concurrency are involved.", + "coordinated_surfaces": "Everything stays inside the project command component. item_edit.go needs new positional `[]` handling, --owner/--url/--field/--value flags, mutual-exclusion checks and Long/Example help, and the required-`--id` rule (`MarkFlagRequired(\"id\")`) has to be relaxed. shared/queries/queries.go needs an item-by-URL lookup and iteration data on ProjectField (it currently has only ID/Name/DataType). item_edit_test.go needs new tests. All of these must agree.", + "ambiguity": "The obvious shortcut, reusing IssueOrPullRequestID from item-add as the item ID, fails at the API. The task leaves open how --field interacts with --clear and with --id/--field-id, how iteration values are named, and what happens when the URL's issue isn't in the project. Sibling commands and fieldIdAndProjectIdPresence make the intended handling inferable.", + "verification": "pkg/cmd/project/item-edit/item_edit_test.go (812 lines) uses httpmock to cover the existing ID-based paths, so it would catch regressions to --id/--field-id/--project-id. Nothing covers name resolution yet. httpmock matches GraphQL by regex and returns canned JSON, so it can't check that the new lookup query is valid against the real ProjectV2 schema. The only project acceptance test is acceptance/testdata/project/project-create-delete.txtar.", + "blast_radius": "This is a user-facing gh command that scripts and agents use. A bad resolution would set the wrong project field value, which users would notice and could fix by re-editing. No credentials or permanent data loss are involved, and reverting the release restores the old behaviour." + }, + "confidence": "medium", + "summary": "item-edit needs an optional `[] --owner --url --field --value` path. It reuses item-add's owner/project resolution, adds a new shared query that maps an issue/PR URL to its ProjectV2 item ID in that project, and matches the field name and option/iteration name case-insensitively against ProjectFields. It also has to build the typed value from DataType and fail early listing the candidates, all while leaving the ID flags unchanged. Tests are httpmock-based in item_edit_test.go and can't check the real ProjectV2 schema, so correct API semantics depend on the implementer." + }, + { + "task": "project-item-edit-by-name", + "attempt": 2, + "ref": "v2.96.0", + "judged_at": "2026-10-14T22:17:23.165721Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3039, + "cache_read_tokens": 108175, + "cache_creation_tokens": 29991, + "num_turns": 6, + "latency_seconds": 37.3, + "total_cost_usd": 0.2556686 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says to mirror item-add, which resolves the owner, project and URL through queries.Client.NewOwner, NewProject and IssueOrPullRequestID (item_add.go:84-101). How each data type maps onto githubv4.ProjectV2FieldValue is pinned by buildUpdateItem (item_edit.go:209-241). Both follow from nearby code, but the wording of the 'clear message listing the candidates' and the exact flag names (--field/--value) are left to project conventions.", + "silent_invariant": "No auth or secrets are involved, but a wrong resolution can look like it works. If two fields or options differ only in case and the code silently takes the first, or the field/item lookup stops at LimitDefault pagination (ProjectFields/ProjectItems in queries.go), the mutation succeeds on the wrong field, option or item. Nothing errors.", + "coordinated_surfaces": "Everything stays inside the project command component: new flags, validation and Long/Example help in pkg/cmd/project/item-edit/item_edit.go, plus probably shared pkg/cmd/project/shared/queries/queries.go. The ProjectField IterationField struct (queries.go:932-936) has no iteration configuration, and field-list --json and export_data_test.go also consume that struct. No other service or deploy surface is involved.", + "ambiguity": "Copying item-add looks complete but isn't. IssueOrPullRequestID returns the issue/PR content node ID, while updateProjectV2ItemFieldValue needs the project item ID (item_edit.go:236). So the URL has to be mapped to the item inside the chosen project. Also, 'interpret --value by data type' covers number, date and iteration, not just single-select, and iteration options are not fetched today. A single-select-only version would pass a quick demo.", + "verification": "item_edit_test.go has gock tests with exact request bodies for the ID path (TestRunItemEdit_Text/Number/Date/SingleSelect/Iteration), so they would catch regressions to the existing flags. Nothing exercises name resolution, item lookup by URL, case-insensitive matching or candidate-listing errors; those tests must be written as part of the task.", + "blast_radius": "gh is a user-facing CLI, and item-edit changes users' live GitHub Project data. Writing to the wrong item or field annoys users and can be fixed by re-editing or reverting the release. No credentials, money or irreversible data loss are involved." + }, + "confidence": "high", + "summary": "item-edit needs an opt-in `[] --owner --url --field --value` path. It resolves owner, project and field (case-insensitively, failing with candidate lists) and maps the issue URL to its project item ID, not the content ID. --value is converted by field type, including single-select options and possibly iteration names, which needs the shared queries extended. The existing ID flags and their gock-pinned behavior must stay unchanged, and the new path needs its own gock-based tests in item_edit_test.go." + }, + { + "task": "project-item-edit-by-name", + "attempt": 3, + "ref": "v2.96.0", + "judged_at": "2026-10-14T22:21:37.587237Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3217, + "cache_read_tokens": 111972, + "cache_creation_tokens": 31197, + "num_turns": 6, + "latency_seconds": 39.5, + "total_cost_usd": 0.26705293999999996 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the main behaviour, but details come from nearby code. The `[] --owner --url` argument handling and the use of `NewOwner`/`NewProject`/`IssueOrPullRequestID` come from `item-add/item_add.go`. How each type is interpreted (date as YYYY-MM-DD, number as float, iteration ID) comes from `buildUpdateItem` in `item_edit.go`. `ProjectField.DataType` and `Options()` in `shared/queries/queries.go:929-993` supply the field data. The `--clear` and draft-issue (`DI_`) paths have to keep working, and `TestNewCmdeditItem` in `item_edit_test.go` pins the current flag behaviour.", + "silent_invariant": "Most mistakes show up as API or flag errors. Some only show on unusual input and would still edit the wrong thing without any error. The `--url` points to issue/PR content, but the mutation needs the project item ID (`ItemID` in `UpdateProjectV2ItemFieldValueInput`). Matching it by issue number alone, or missing it past the page limit of `ProjectItems`, could edit the wrong item or fail wrongly. Case-insensitive matching can also pick the wrong field or option when two names differ only by case. There is no auth, secrets or concurrency involved.", + "coordinated_surfaces": "The change stays in one component and touches two or three places that must agree. `item-edit/item_edit.go` needs the positional number, `--owner`/`--url`/`--field`/`--value` flags, mutual-exclusion checks against the ID flags, and new help and example text. `shared/queries/queries.go` probably needs a lookup from content to project item, and maybe iteration data, because `IterationField` currently fetches no iterations. Tests go in `item_edit_test.go`. `project.go` only registers the command, so its wiring does not change.", + "ambiguity": "Some interpretation is needed. Should `--value` name an iteration by title? How do the name flags combine with `--clear`, with draft issues that have no URL, and with partly mixed ID and name flags? The tempting shortcut of passing the `IssueOrPullRequestID` result as the item ID is wrong; the repo shows this because `item-add` gets an item ID back from `addProjectV2ItemById`. Looking at the code settles the intended design.", + "verification": "`item_edit_test.go` has gock-based tests for the existing paths (Text, Number, NumberZero, Date, Draft*, clear). These would catch regressions to the `--id`/`--field-id`/`--project-id` flags. Nothing covers resolving by name yet; new mocked tests would only check against hand-written GraphQL responses, not the real ProjectV2 API (pagination, how project items link to content).", + "blast_radius": "This is a user-facing `gh project` command that changes project field values on GitHub. Editing the wrong field or item is visible and can be fixed by hand or by reverting the release. No credentials, money or schema migrations are involved." + }, + "confidence": "medium", + "summary": "Add an opt-in name form to `gh project item-edit` (` --owner --url --field --value`). It must resolve the project, the content-to-item mapping, the field (case-insensitive) and the option or typed value before calling the existing `updateProjectV2ItemFieldValue` mutation, and fail with a list of candidates when a name does not match. It must also keep the ID-flag paths unchanged and add gock-mocked tests in `item_edit_test.go`, probably along with a new lookup helper in `shared/queries`." + }, + { + "task": "release-create-404-scope-blame", + "attempt": 1, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:11:12.234443Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2154, + "cache_read_tokens": 63533, + "cache_creation_tokens": 27539, + "num_turns": 4, + "latency_seconds": 48.7, + "total_cost_usd": 0.21287376000000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task's debug log and other gh code show the header is separated by a comma and a space: `HeaderHasMinimumScopes` in `pkg/cmd/auth/shared/oauth_scopes.go` trims each piece with `strings.TrimSpace(s)`. `tokenHasWorkflowScope` in `release/create/http.go` (line 311) does `slices.Contains(strings.Split(scopes, \",\"), \"workflow\")` with no trim. Its other rule, that an empty header counts as having the scope (the comment says such tokens are probably not OAuth tokens), must also be kept.", + "silent_invariant": "Only an error message is at stake, with no auth decision or data. A wrong fix still looks right with the httpmock headers that have no spaces, or when the 404 really is a scope problem. It shows only with real spaced headers, a token without the scope, or an empty header that must still mean 'assume present'.", + "coordinated_surfaces": "One function, `tokenHasWorkflowScope` in `pkg/cmd/release/create/http.go`, plus test cases in `create_test.go`. The `errMissingRequiredWorkflowScope` handling in `create.go` (lines 518-525) stays as it is.", + "ambiguity": "The task describes a symptom, and someone could misread it as 'remove the scope hint on 404'. But the task says the hint must stay when the scope is truly missing, and once you read the code the cause (no whitespace trim, unlike the auth/shared parser) is clear.", + "verification": "`create_test.go` (around lines 1197-1241) has 404 cases with and without the scope, but both use `httpmock.StatusScopesResponder(404, \"repo,read:org,workflow\")` with no spaces, so they pass on the buggy code. Nothing tests the real header format or the empty-header case through `createRelease`.", + "blast_radius": "Users of `gh release create` see a misleading hint and the real 404 is hidden. No credentials, stored data or permissions change, and a revert fixes it with no lasting harm." + }, + "confidence": "high", + "summary": "Fix `tokenHasWorkflowScope` in `pkg/cmd/release/create/http.go` to trim whitespace from each scope in the comma-and-space `X-Oauth-Scopes` header, matching `auth/shared.HeaderHasMinimumScopes`, and keep the rule that an empty header means the scope is assumed present. Add `create_test.go` cases with the real spaced header format, so a token that has `workflow` gets the underlying HTTP 404 and a token without it still gets the refresh hint." + }, + { + "task": "release-create-404-scope-blame", + "attempt": 2, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:14:55.181197Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 2336, + "cache_read_tokens": 91128, + "cache_creation_tokens": 29581, + "num_turns": 5, + "latency_seconds": 65.4, + "total_cost_usd": 0.23418846 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task never names the header format. It is GitHub's comma-plus-space list, shown only in the task's debug sample. The repo already handles it in pkg/cmd/auth/shared/oauth_scopes.go HeaderHasMinimumScopes, which does strings.Split(scopesHeader, \",\") and then strings.TrimSpace. The 'no X-Oauth-Scopes header means assume the scope is present' rule is defined only in the tokenHasWorkflowScope comment in http.go.", + "silent_invariant": "tokenHasWorkflowScope in pkg/cmd/release/create/http.go only decides whether createRelease returns errMissingRequiredWorkflowScope or api.HandleHTTPError. Getting it wrong changes an error message, not access. But a wrong parse only shows with real header formats (spaces, empty header, a scope that only partly matches), and the httpmock fixtures without spaces hide it.", + "coordinated_surfaces": "One function, tokenHasWorkflowScope in pkg/cmd/release/create/http.go, plus test cases in pkg/cmd/release/create/create_test.go (and maybe an http_test.go). No other command, docs or config uses it.", + "ambiguity": "The task gives the symptom but not the cause. Narrow fixes are tempting: dropping the 404 scope check (the task forbids this: the hint must stay when the scope is truly missing), or a substring match on the raw header. Once you see the Split without TrimSpace next to how HeaderHasMinimumScopes parses the same header, the right fix is clear.", + "verification": "create_test.go has 404 cases 'OAuth token has no workflow scope', 'has workflow scope' and 'not an OAuth token'. They use httpmock.StatusScopesResponder(404, `repo,read:org,workflow`) with no spaces, so they pass on the current bug and would miss a whitespace or partial-match regression. The fix needs new fixtures in the real header format.", + "blast_radius": "This is the user-facing `gh release create` error path. A wrong change misleads users or hides the real 404 cause (wrong repo name, missing push access). It grants no access and loses no data, and a revert fixes it." + }, + "confidence": "high", + "summary": "Fix tokenHasWorkflowScope in pkg/cmd/release/create/http.go so it trims whitespace around each X-Oauth-Scopes entry, as auth/shared HeaderHasMinimumScopes does. Then a token that holds `workflow` gets the real HTTP 404 error, while the refresh hint stays for tokens that truly lack the scope and the empty-header case is unchanged. Add create_test.go cases that use GitHub's comma-plus-space header format." + }, + { + "task": "release-create-404-scope-blame", + "attempt": 3, + "ref": "v2.97.0", + "judged_at": "2026-10-14T22:19:35.523815Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2279, + "cache_read_tokens": 63509, + "cache_creation_tokens": 27667, + "num_turns": 4, + "latency_seconds": 28.3, + "total_cost_usd": 0.21632247999999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The expected behaviour depends on GitHub's real X-Oauth-Scopes format: comma plus space, as in the task's example and in the pkg/cmd/auth/status tests (\"repo, read:org\"). It also depends on the existing convention in pkg/cmd/auth/shared/oauth_scopes.go:93-94, which splits and then applies strings.TrimSpace. Both are visible in nearby code; nothing outside the repo defines them.", + "silent_invariant": "The bug is a parsing edge case, not a security or data invariant. tokenHasWorkflowScope (http.go:311) works for the no-space header the mocks use but fails for real headers. A fix that only matches one format, or that drops the 'no scopes header means assume workflow' rule (http.go:304-309), would pass a quick check.", + "coordinated_surfaces": "Only tokenHasWorkflowScope in pkg/cmd/release/create/http.go needs to change, plus tests in create_test.go. The error type and the hint rendering in create.go:518-525 stay as they are.", + "ambiguity": "The task describes the symptom without naming the cause. The tempting fixes are to remove the 404 scope check entirely, which the task rules out because the hint must stay for tokens that truly lack the scope, or to special-case one header string. The trimming pattern in oauth_scopes.go makes the intended fix clear.", + "verification": "create_test.go:1197-1241 covers both 404 cases, but its mocks use httpmock.StatusScopesResponder with \"repo,read:org,workflow\" (no spaces), so the current buggy code passes. New cases with the real \", \" format are needed. Everything can be tested offline with httpmock.", + "blast_radius": "This is a user-facing error message in `gh release create`. A wrong change misleads users or hides the real 404 cause, but it touches no credentials, stored data or authorization decision, and a revert fixes it." + }, + "confidence": "high", + "summary": "tokenHasWorkflowScope in pkg/cmd/release/create/http.go splits X-Oauth-Scopes on \",\" without trimming whitespace, so real \"a, b, workflow\" headers never match \"workflow\" and every 404 is replaced by the scope hint. The fix is to trim each scope, following HeaderHasMinimumScopes in auth/shared, and to add create_test.go cases that use the real comma-space header format for tokens with and without workflow, keeping the existing empty-header rule." + }, + { + "task": "release-verify-sha1-prefix", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:12:08.115281Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3276, + "cache_read_tokens": 141673, + "cache_creation_tokens": 30239, + "num_turns": 7, + "latency_seconds": 38.8, + "total_cost_usd": 0.26962496 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task names the expected labels (`sha1:` and `sha256:`). How to tell the algorithm apart is left to nearby code: `shared.FetchRefSHA` (`pkg/cmd/release/shared/fetch.go:138`) returns only a bare hex string. The `sha256` spelling matches how `DigestedArtifact.DigestWithAlg` (`artifact.go:87`) and `digest.CalculateDigestWithAlgorithm` already name algorithms. Whether the attestation service will index SHA-256 commit refs under `sha256:` is assumed, not documented in this repository.", + "silent_invariant": "The prefix only goes into `GetByDigest` (the lookup) and the output strings. `buildVerificationPolicy` and `verification.BuildDigestPolicyOption` hex-decode the digest itself, so cryptographic verification does not depend on the label. A wrong prefix shows up as a lookup failure or a wrong label. The edge cases are ref lengths that are neither 40 nor 64, and labelling SHA-256 refs from an older or alternative host (GHES), which a normal SHA-1 check would not reveal.", + "coordinated_surfaces": "Two call sites in one component must agree: `pkg/cmd/release/verify/verify.go:133` and `pkg/cmd/release/verify-asset/verify_asset.go:145`. Ideally they share one helper in `pkg/cmd/release/shared`, plus tests in both `verify_test.go` and `verify_asset_test.go`. No API, config or docs surfaces are involved.", + "ambiguity": "The obvious approach is to infer the algorithm from the ref length (40 means sha1, 64 means sha256). It needs some decisions, such as what to do with other lengths and whether to put the logic in `shared` so the two commands cannot drift. Still, the task and the code make the intended result clear, and there is no competing design that would look complete.", + "verification": "Every test in `verify_test.go` and `verify_asset_test.go` uses a 40-character `fakeSHA` and never asserts the `sha1:` string. The error tests check only `\"no attestations for tag v1\"`. `api.MockClient.OnGetByDigestSuccess`/`Failure` ignore `params.Digest`. A change that always emitted `sha256:`, or never changed anything, would pass the current suite.", + "blast_radius": "`gh release verify` and `gh release verify-asset` are customer-facing supply-chain checks. A wrong prefix would make verification fail (no attestations found) or mislabel output, not accept forged artifacts, because the digest bytes in the sigstore policy are unchanged. A CLI release revert fixes it with no lasting harm." + }, + "confidence": "medium", + "summary": "Replace the hard-coded \"sha1\" passed to `artifact.NewDigestedArtifactForRelease` in both `release verify` and `release verify-asset` with an algorithm derived from the ref returned by `shared.FetchRefSHA` (40 hex means sha1, 64 hex means sha256), ideally through one shared helper. Add tests that assert the `sha1:` and `sha256:` prefixes in the lookup digest and output for both commands, since current tests never check the prefix." + }, + { + "task": "release-verify-sha1-prefix", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:16:14.976986Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3362, + "cache_read_tokens": 110100, + "cache_creation_tokens": 31094, + "num_turns": 6, + "latency_seconds": 36.8, + "total_cost_usd": 0.2692646 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the expected prefixes, but the rest comes from nearby code. The algorithm string also feeds sigstore policy via artifact.DigestedArtifact.Algorithm() -> verification.BuildDigestPolicyOption -> verify.WithArtifactDigest(a.Algorithm(), ...) in pkg/cmd/attestation/verification/policy.go. The 'sha256' label must match the convention already used in shared.FilterAttestationsByFileDigest (alg := \"sha256\"). The ref comes from shared.FetchRefSHA (pkg/cmd/release/shared/fetch.go), which returns a bare object.sha with no algorithm field, so the algorithm has to be inferred from its length.", + "silent_invariant": "This is supply-chain verification code: releaseRefDigest goes into both AttClient.GetByDigest and AttVerifier.VerifyAttestation (verify.go:136-167, verify_asset.go:148-185). A wrong algorithm mostly fails visibly, because the lookup misses or sigstore rejects the digest. The silent cases are edge inputs only: refs that are neither 40 nor 64 characters, a fix that changes the printed label but not the algorithm passed to the policy, or the reverse.", + "coordinated_surfaces": "Two sibling commands in one component must agree: pkg/cmd/release/verify/verify.go and pkg/cmd/release/verify-asset/verify_asset.go, possibly through a new helper in pkg/cmd/release/shared. Each must use the same algorithm in four places: the lookup Digest param, the error messages, the 'Resolved tag' output and the verification policy. No other component, config or service changes.", + "ambiguity": "The intent is clear: pick sha1 or sha256 from the ref the API returns, which means its length. Some interpretation is still needed: whether an unexpected length should be an error or fall back to sha1, and whether the detection belongs in FetchRefSHA or in the callers. The task text ('also used in the attestation lookup', 'in both commands') rules out the narrow fix of changing only the printed label.", + "verification": "verify_test.go and verify_asset_test.go exist, but every case stubs a 40-character fakeSHA through shared.StubFetchRefSHA. They use a mock attestation client and shared.MockVerifier, which ignores the artifact, and only assert substrings like 'no attestations for tag v1'. They never check the prefix or the Digest param, so a wrong algorithm would pass. New tests are needed and can be written with httpmock.", + "blast_radius": "gh release verify and verify-asset are user-facing commands that check release integrity. A regression would show up as failed lookups, failed verifications or wrong labels for users, and reverting the change fixes it with no lasting harm. The sigstore digest comparison still has to pass, so a mismatched algorithm should not let a bad attestation through [INFERENCE]." + }, + "confidence": "medium", + "summary": "Replace the hard-coded \"sha1\" algorithm in pkg/cmd/release/verify/verify.go and pkg/cmd/release/verify-asset/verify_asset.go with one picked from the ref returned by shared.FetchRefSHA (40 hex characters means sha1, 64 means sha256). The chosen algorithm must flow consistently into the GetByDigest lookup, the error and output messages, and the sigstore digest policy. Add tests for both SHA-1 and SHA-256 refs that assert the prefix, because the existing mocked tests never check it." + }, + { + "task": "release-verify-sha1-prefix", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:20:39.923120Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 3189, + "cache_read_tokens": 129503, + "cache_creation_tokens": 28461, + "num_turns": 7, + "latency_seconds": 48.0, + "total_cost_usd": 0.25525456 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The algorithm label is passed to `AttClient.GetByDigest` through `releaseRefDigest.DigestWithAlg()`. It also reaches `verification.BuildDigestPolicyOption` (`pkg/cmd/attestation/verification/policy.go:25`, `verify.WithArtifactDigest(a.Algorithm(), ...)`), so it must use the lowercase algorithm names that sigstore-go and the attestation API already expect (`sha1`, `sha256`). `shared.FetchRefSHA` (`fetch.go:160-170`) returns only `object.sha`, so the algorithm has to be worked out from that value. This follows from nearby code, not from an outside spec.", + "silent_invariant": "The label feeds the sigstore artifact-digest policy in `shared/attestation.go:112` (`buildVerificationPolicy`). A wrong label makes the lookup or verification fail with a visible error rather than pass silently. The risk is in edge cases normal checks would miss: SHA-256 repositories are only now rolling out, and refs of unexpected length could be mislabelled without anyone noticing.", + "coordinated_surfaces": "Two sibling commands, `verify/verify.go:133` and `verify-asset/verify_asset.go:145`, must agree. The algorithm detection would likely live in one helper in `pkg/cmd/release/shared`, plus tests in both commands. This all sits inside one component, and no docs or config schema change.", + "ambiguity": "The task says to match the algorithm of the ref the API returns, but not how to detect it. Length (40 vs 64 hex characters) is the obvious route. Choices remain: how to handle values of other lengths (default to sha1, or error), and whether detection goes in `FetchRefSHA` or a shared helper. A shortcut that only changes the printed text and leaves the lookup or policy algorithm as `sha1` would miss the stated requirement.", + "verification": "`verify_test.go` and `verify_asset_test.go` stub refs only with the 40-character `fakeSHA` and never assert the `sha1:` prefix (searching the release tests for `sha1` finds nothing). The mock attestation client and `MockVerifier` do not check the digest algorithm, so a wrong lookup or policy algorithm would not be caught. The task asks for new tests, but existing coverage is weak.", + "blast_radius": "`gh release verify` and `gh release verify-asset` are user-facing supply-chain verification commands. A regression in SHA-1 handling would make verification fail for every user, but visibly (an error, not a false pass), and reverting the CLI release fixes it. No stored data or credentials are involved." + }, + "confidence": "medium", + "summary": "Replace the hard-coded \"sha1\" passed to `artifact.NewDigestedArtifactForRelease` in both `verify.go` and `verify_asset.go` with an algorithm worked out from the ref returned by `shared.FetchRefSHA` (40 hex characters means sha1, 64 means sha256), ideally through one shared helper. The output text, the `GetByDigest` lookup and the sigstore digest policy all need the same algorithm, with tests covering both SHA-1 and SHA-256 refs in both commands." + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:11:29.758044Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2922, + "cache_read_tokens": 62152, + "cache_creation_tokens": 23879, + "num_turns": 4, + "latency_seconds": 35.2, + "total_cost_usd": 0.20933594 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task names the API field to follow, but the details come from nearby code. `api.GitHubRepo` (api/queries_repo.go:360) asks for no `sshUrl` on the repository or its parent, so the RepositoryInfo query has to fetch it; `Repository.SSHURL` already exists at line 42. `FormatRemoteURL` in internal/ghrepo/repo.go:95 builds `tenant@host`, and internal/ghrepo/repo_test.go:258-263 pins that output. The wiki URL is derived by replacing the `.git` suffix, so a wiki clone also has to start from the API URL. That the GraphQL `sshUrl` matches the REST `ssh_url` the task describes is an API contract assumption.", + "silent_invariant": "No security, data or concurrency rule is at stake; a wrong URL shows up as a clone that fails with 'Permission denied (publickey)'. The mistakes that would slip by are edge cases. In clone.go the URL-argument path (`repositoryIsURL`) is separate from the full-name path, wiki clones rebuild the URL, the parent repo needs its own `sshUrl` for the upstream remote, and an empty `sshUrl` needs a fallback. Moving every host, including GHES and github.com, onto `sshUrl` changes behaviour that only shows on other setups.", + "coordinated_surfaces": "Two or three places in one feature must agree: the GraphQL fragment in `api.GitHubRepo` (repo and parent), and in clone.go both the clone-URL step and the upstream `AddRemote` step, plus clone_test.go. No config schema, docs or other service is involved.", + "ambiguity": "Two narrow fixes look complete. One special-cases tenant names inside `FormatRemoteURL`, which cannot know the org ID. The other fixes only the full-name path and leaves the URL-argument path and the fork upstream remote building tenant URLs. The task rules both out by name ('gh rewrites it', 'upstream remote ... should also be correct', 'HTTPS cloning should not change'). It is still open whether non-tenant SSH hosts should switch to `sshUrl` too.", + "verification": "pkg/cmd/repo/clone/clone_test.go stubs `query RepositoryInfo` with httpmock and checks the exact git commands (`Test_RepoClone`, `Test_RepoClone_hasParent`). Those tests would catch a broken HTTPS or github.com path. They have no case for a tenant host with an org-specific `sshUrl`, for SSH upstream remotes, or for the SSH-URL argument, and the real certificate-based login cannot be checked without a GHE.com tenant.", + "blast_radius": "`gh repo clone` is a user-facing command. A wrong change gives failed clones or a wrong remote URL in a local `.git/config`, which a revert and re-clone fix. No credentials are leaked and no server-side data is touched." + }, + "confidence": "medium", + "summary": "Add `sshUrl` to the `api.GitHubRepo` RepositoryInfo query for the repo and its parent. When the protocol is SSH, `gh repo clone` should use that URL for the clone (including the URL-argument and wiki paths) and for the fork upstream remote, leave HTTPS unchanged, and add httpmock-backed tests to clone_test.go." + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:15:29.659165Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3555, + "cache_read_tokens": 108888, + "cache_creation_tokens": 26244, + "num_turns": 6, + "latency_seconds": 42.0, + "total_cost_usd": 0.24656895999999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the target behaviour: when the protocol is SSH, use the SSH URL the API reports. Two things it leaves out are only visible in the code. First, `api.GitHubRepo` in api/queries_repo.go:360-386 does not request `sshUrl` in its shared `repo` fragment, which `parent` also uses, even though `Repository.SSHURL` and the `sshUrl` field already exist. Second, the task's example shows the REST `ssh_url`, so the code must rely on the GraphQL `sshUrl` carrying the same org-specific username on GHE.com, which nothing in the repository proves.", + "silent_invariant": "On github.com and GHES, `ghrepo.FormatRemoteURL` already gives the same `git@host:owner/repo.git` URL the API would return. A partial fix therefore looks correct everywhere except on GHE.com tenants whose org uses an SSH CA. It can also be wrong only on less-used paths: the URL-argument branch (clone.go:130-144, where the protocol comes from the URL scheme), the `.wiki.git` suffix rewrite (clone.go:185) and the fork upstream remote (clone.go:205-206).", + "coordinated_surfaces": "Three places in one component must agree. The GraphQL fragment in `api.GitHubRepo` must fetch `sshUrl` for both the repo and its parent. In `cloneRun`, both the origin clone URL (line 178) and the upstream URL (line 206) must use it. The mocked `RepositoryInfo` responses in clone_test.go must be updated. Other callers of `GitHubRepo` (pr create/merge, project link/unlink, repo create, ruleset check) are only affected by the extra field.", + "ambiguity": "Fixes that look complete but are too narrow: fixing only the full-name path and not the URL-argument path (the task says 'Passing the correct SSH URL as the clone argument does not help'); fixing origin but not the fork upstream; or editing `FormatRemoteURL`'s tenant logic (repo.go:97-98), which cannot build `TENANT_` without data from the API. The task names the intended design: use the API's SSH URL only for SSH and leave HTTPS unchanged.", + "verification": "clone_test.go has table tests built on httpmock `RepositoryInfo` stubs, including an 'SSH URL' case (lines 189-191), so changes to clone and upstream URLs can be checked against stubs. However, the stubs return data the test author writes, so no test can confirm that GraphQL `sshUrl` really returns the org-specific `TENANT_@` form on GHE.com. Existing fixtures do not cover a ghe.com host or fallback when `sshUrl` is empty.", + "blast_radius": "`gh repo clone` is a widely used customer command. A mistake would break SSH clones or set a wrong upstream remote, which users would notice and a revert would fix. No stored data or credentials are at risk. Adding a field to the shared `GitHubRepo` query reaches several commands, but adding a field is low-risk." + }, + "confidence": "medium", + "summary": "Add `sshUrl` to the `GitHubRepo` GraphQL fragment (repo and parent), then in `cloneRun` use `canonicalRepo.SSHURL` and `Parent.SSHURL` when the protocol is SSH, on both the full-name and URL-argument paths and for the wiki and fork-upstream cases, leaving HTTPS on `FormatRemoteURL`. Update the clone_test.go httpmock fixtures and add a ghe.com org-specific SSH case for both origin and upstream." + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:19:55.513748Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 3219, + "cache_read_tokens": 89731, + "cache_creation_tokens": 27505, + "num_turns": 5, + "latency_seconds": 55.3, + "total_cost_usd": 0.24188912 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task gives the URL format, but the fix depends on repo code: pkg/cmd/repo/clone/clone.go builds the clone and upstream URLs with ghrepo.FormatRemoteURL (internal/ghrepo/repo.go:95-98 writes tenant@host for *.ghe.com), and api.GitHubRepo's RepositoryInfo fragment (api/queries_repo.go:362-374) does not request sshUrl for the repo or its parent; the wiki path also adds .wiki.git to the URL.", + "silent_invariant": "Touches SSH authentication (username selects the org's SSH certificate). Most mistakes show up as a failed clone, but some only appear in edge cases: CA-enabled tenants, cloning by URL (protocol taken from the parsed URL at clone.go:130-144), wiki repos, fork upstreams (clone.go:205-206), or GHES when the API SSH URL differs from the built one.", + "coordinated_surfaces": "Two parts of one component must agree: the RepositoryInfo fragment in api/queries_repo.go (used for both repository and parent) must add sshUrl, and clone.go must use it at both the clone (line 178) and upstream (line 206) sites, plus clone_test.go.", + "ambiguity": "Tempting partial fixes: change only the clone URL and not the upstream, only the config path and not the URL-argument path, or edit FormatRemoteURL (which cannot know the org ID). The task names upstream, URL arguments and HTTPS staying unchanged, so the intended scope is clear.", + "verification": "pkg/cmd/repo/clone/clone_test.go (Test_RepoClone, Test_RepoClone_hasParent*) checks the exact git command lines against mocked GraphQL, but has no SSH, tenant or sshUrl cases; internal/ghrepo/repo_test.go only covers FormatRemoteURL. New tests are needed to catch mistakes.", + "blast_radius": "gh repo clone is a user-facing command; a wrong URL means a failed clone or a bad upstream remote written into the user's local git config. It can be reverted, and no credentials or server-side data are exposed." + }, + "confidence": "medium", + "summary": "Add sshUrl to the RepositoryInfo fragment in api.GitHubRepo for the repository and its parent, and have cloneRun use the API SSH URL for both the clone and the fork upstream remote when the protocol is ssh, including the URL-argument and wiki paths, while leaving HTTPS unchanged. Add clone_test.go cases with mocked sshUrl responses that check the git clone and remote add commands." + }, + { + "task": "repo-create-reject-empty-name", + "attempt": 1, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:10:13.194587Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2116, + "cache_read_tokens": 57382, + "cache_creation_tokens": 22999, + "num_turns": 4, + "latency_seconds": 26.2, + "total_cost_usd": 0.18571454 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the behaviour (report the empty name right away, keep valid names working), but the exact message and form come from nearby code: `splitNameAndOwner` in create.go wraps errors as `argument error: %w`, and prompts go through the small `iprompter` interface (Input/Select/Confirm) with no validation hook. There is no outside spec or caller contract.", + "silent_invariant": "No auth, secrets or stored data are involved. If the fix is wrong, the user still sees the late `argument error` or a visible early error, and valid names like \"REPO\" go through the same `interactiveRepoNameAndOwner` path that existing tests exercise. Edge inputs such as whitespace-only names or `owner/` (which already fails in `ghrepo.FromFullName`) are minor and show up on screen.", + "coordinated_surfaces": "One place: the `prompter.Input(\"Repository name\", defaultName)` result in `interactiveRepoNameAndOwner` (create.go:929). Every interactive flow (create.go:310, 451, 595) passes through it via `interactiveRepoInfo`. No docs, flags or API changes are needed.", + "ambiguity": "Some choices remain: put the check only in the from-scratch caller (line 310) or in the shared helper so the template and local-push flows are covered too; re-prompt or return an error; trim whitespace or not. The code makes the shared helper the clear place to put it, and the task text (\"tells the user right away\") settles the intent.", + "verification": "`Test_createRun` in create_test.go stubs the prompter with `case \"Repository name\": return \"REPO\"` across many interactive cases, so a regression for valid names would be caught. No test currently covers an empty name; the task asks for one to be added.", + "blast_radius": "This is the user-facing `gh repo create` interactive flow. A bad change, such as rejecting valid names, would block repo creation for users, but a revert fixes it with no lasting harm, since nothing is created until the API call after confirmation." + }, + "confidence": "high", + "summary": "Add an empty-name check right after the \"Repository name\" prompt in `interactiveRepoNameAndOwner` (pkg/cmd/repo/create/create.go), so all interactive create flows fail or re-prompt before asking for owner, description and the rest. Add a `Test_createRun` case where the prompter stub returns an empty name, and check that the error comes back early, before any later prompts or HTTP calls." + }, + { + "task": "repo-create-reject-empty-name", + "attempt": 2, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:13:54.836073Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 2566, + "cache_read_tokens": 80740, + "cache_creation_tokens": 23872, + "num_turns": 5, + "latency_seconds": 52.1, + "total_cost_usd": 0.2055636 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the symptom and the expected outcome. It does not say what the message should look like or whether gh should ask again. The surrounding code settles that: `splitNameAndOwner` and the `ghrepo.FromFullName` checks in create.go already return early with `fmt.Errorf(\"argument error: ...\")`, and `iprompter.Input(string, string)` has no validator, so returning an error right after `prompter.Input(\"Repository name\", defaultName)` in `interactiveRepoNameAndOwner` is the obvious fit.", + "silent_invariant": "No auth, secrets or data are involved. Some wrong fixes would still look fine in a quick test. A plain `name == \"\"` check lets through a name that is only spaces. A check put only in `createFromScratch` (line 310) misses the template flow (line 451) and the local-source flow (line 595, where the default is `filepath.Base(absPath)`). A check on the joined `owner/name` string behaves differently for personal repos (`NewWithHost(\"\", \"\", host)` at line 348) and org repos (`\"some-owner/\"`).", + "coordinated_surfaces": "One place covers it: `interactiveRepoNameAndOwner` in pkg/cmd/repo/create/create.go, which all three interactive flows reach through `interactiveRepoInfo`. Tests go in create_test.go. No docs, flags or other packages need to change.", + "ambiguity": "The task describes the from-scratch flow only. A fix placed in `createFromScratch` would make that report go away but leave the template and local flows open, because they call the same `interactiveRepoInfo`. Erroring out and asking again are both reasonable. Once you look, the shared helper and the existing early-error style make the intended answer clear.", + "verification": "`Test_createRun` in create_test.go stubs prompts with `case \"Repository name\": return \"REPO\", nil` for several interactive flows. Those cases would catch a break to valid names, but none covers an empty or blank name, and nothing tests `interactiveRepoNameAndOwner` or `splitNameAndOwner` directly. The new tests still have to be written, and they can run with the existing mocks.", + "blast_radius": "This is the user-facing interactive `gh repo create` command. If the check is wrong, the worst cases are rejecting valid names or still failing late. Users would notice, a revert fixes it, and the GitHub API rejects bad names anyway, so no data is lost." + }, + "confidence": "high", + "summary": "The task needs a check, in the shared `interactiveRepoNameAndOwner` helper in pkg/cmd/repo/create/create.go, that rejects an empty or whitespace-only repository name right after the \"Repository name\" prompt, so the from-scratch, template and local flows all stop before the later prompts. It also needs new `Test_createRun` (or helper-level) cases for an empty name, and valid names like \"REPO\" and \"owner/REPO\" must keep passing." + }, + { + "task": "repo-create-reject-empty-name", + "attempt": 3, + "ref": "v2.100.0", + "judged_at": "2026-10-14T22:18:03.581022Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2643, + "cache_read_tokens": 58026, + "cache_creation_tokens": 25052, + "num_turns": 4, + "latency_seconds": 36.4, + "total_cost_usd": 0.20874172000000002 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task says what should happen (report the empty name right away) but not the error wording or where the check goes. Those come from nearby code. `interactiveRepoNameAndOwner` (create.go:928) reads the name through `prompter.Input` and returns errors through `splitNameAndOwner`. Sibling commands word blank input as \"X cannot be blank\" (for example `discussion/create/create.go:122` and `agent-task/create/create.go:75`). The `iprompter` interface only has `Input(string, string)`, so there is no validator hook for re-prompting. Nothing outside the repo fixes the behaviour.", + "silent_invariant": "No auth, secrets or persisted data are involved; the mistake is a usability one. A wrong fix mostly shows up as an error or a leftover late failure. Some cases are easy to miss: whitespace-only names, a check placed after the owner Select (create.go:955) that still fails late, and the `fromLocal` flow at create.go:595, which passes `filepath.Base(absPath)` as a default name that has to keep working.", + "coordinated_surfaces": "All three interactive flows (create.go:310 from scratch, :451 from template, :595 from local) go through `interactiveRepoInfo` -> `interactiveRepoNameAndOwner`, so the fix is one check in one function, plus tests in `create_test.go`. No docs, config or other commands change.", + "ambiguity": "The task names the symptom only in the from-scratch flow. A narrower fix in `createFromScratch` alone, or validating the assembled \"owner/\" string later, would make that symptom go away. Reading the code shows the shared helper is the right place. There is also a choice between returning an error and re-prompting, and the `iprompter` interface points to returning an error. Whether whitespace counts as empty is left open, but the repo convention (`strings.TrimSpace`) answers it.", + "verification": "`create_test.go` has table tests for the interactive flows with stubbed prompters (case \"Repository name\" returning \"REPO\" at lines 225, 291, 357 and others) plus httpmock. They cover valid names, but no test passes an empty or whitespace name, or checks that later prompts are not reached. The task asks for those tests to be added.", + "blast_radius": "`gh repo create` is a widely used, user-facing command. A bad check could block valid names or the `fromLocal` default name, which users would notice. Nothing is stored or exposed, so a release revert fixes it." + }, + "confidence": "high", + "summary": "Add an empty (and likely whitespace-only) name check in `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`, right after the \"Repository name\" Input and before the owner lookup and Select. It should return an error in the repo's \"cannot be blank\" style so the from-scratch, template and local flows all stop early. Add `create_test.go` table cases with an empty name that assert the error and that no later prompts or API calls happen, and keep the existing valid-name tests passing." + }, + { + "task": "repo-set-default-unset-without-repo", + "attempt": 1, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:10:13.254647Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 1841, + "cache_read_tokens": 37360, + "cache_creation_tokens": 22846, + "num_turns": 3, + "latency_seconds": 23.3, + "total_cost_usd": 0.1744094 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task text gives most of the behaviour. The rest comes from nearby code: setdefault.go's guard `!opts.ViewMode && !opts.IO.CanPrompt() && opts.Repo == nil` returns `cmdutil.FlagErrorf(\"repository required when not running interactively\")`, and the --view exemption is the pattern --unset should copy. AGENTS.md also requires keeping error messages and exit behaviour unchanged.", + "silent_invariant": "No auth, secret or concurrency invariant is involved. setDefaultRun's UnsetMode branch only runs `git config --unset remote..gh-resolved`. A wrong guard shows up openly: either the error is still returned, or a plain non-interactive run reaches the prompt or fails.", + "coordinated_surfaces": "One boolean condition in NewCmdSetDefault's RunE in pkg/cmd/repo/setdefault/setdefault.go, plus test cases in setdefault_test.go. No API, config or other command depends on it.", + "ambiguity": "The task says a repository argument 'makes no sense' for --unset and notes that `--unset origin` is ignored. That invites rejecting the argument, which would change script-facing behaviour that AGENTS.md says to preserve, while the stated expected behaviour asks only for the no-argument exemption. Exempting --unset from the whole guard rather than adding a new rule is also implied but not spelled out.", + "verification": "TestNewCmdSetDefault forces SetStdinTTY(true), so no test covers the non-interactive guard or the 'repository required' error. It also checks only Repo and ViewMode, not UnsetMode. TestDefaultRun covers the unset and view run paths. All tests run locally without network access.", + "blast_radius": "gh repo set-default is a user-facing command used in scripts. A wrong change could drop the required error or break --unset, but it only touches the local git config key remote.*.gh-resolved and a revert fixes it." + }, + "confidence": "high", + "summary": "Relax the non-interactive 'repository required' guard in pkg/cmd/repo/setdefault/setdefault.go so that --unset, like --view, runs without a repository argument, while a plain no-argument non-interactive call keeps the existing FlagError. Add non-TTY cases to setdefault_test.go for --unset, --view and the error that must stay, since the current command tests only run with a TTY." + }, + { + "task": "repo-set-default-unset-without-repo", + "attempt": 2, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:14:02.010719Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2053, + "cache_read_tokens": 57651, + "cache_creation_tokens": 23426, + "num_turns": 4, + "latency_seconds": 36.1, + "total_cost_usd": 0.18673621999999998 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The behaviour is mostly stated in the task, but the error that must stay ('repository required when not running interactively', returned via `cmdutil.FlagErrorf` on line 108 of setdefault.go) and the existing `--view` exemption are in the code. AGENTS.md also says error messages, exit behaviour and non-TTY output must be preserved.", + "silent_invariant": "Unset mode only calls `GitClient.UnsetRemoteResolution` on the current default, which is local git config. A wrong guard either still errors or wrongly lets plain non-TTY `gh repo set-default` through, and both show up as soon as the command runs. No auth, secrets or concurrency are involved.", + "coordinated_surfaces": "Only one place has to change: the condition `!opts.ViewMode && !opts.IO.CanPrompt() && opts.Repo == nil` in `NewCmdSetDefault` (setdefault.go line 108), plus the table cases in setdefault_test.go. `setDefaultRun` already handles `UnsetMode` without a repo.", + "ambiguity": "The task points out that `--unset origin` accepts the argument and ignores it, which tempts a fix that also rejects arguments with `--unset` or makes the flags mutually exclusive. That would change a script-facing contract, which AGENTS.md forbids without explicit authorisation. The 'Expected behaviour' paragraph limits the change to the no-argument non-interactive case.", + "verification": "`TestNewCmdSetDefault` has 'view flag' and 'unset flag' cases. Its loop (line 115 onward) builds iostreams that, judging by the passing 'no argument' case, appear to be TTY only, and it asserts only `Repo` and `ViewMode`, not `UnsetMode` (lines 158-159). No case exercises the non-TTY guard, so the new tests have to add a per-case TTY setting. `TestDefaultRun` already covers `setDefaultRun` for unset and non-TTY view.", + "blast_radius": "This is a user-facing `gh` command, and scripts rely on its non-TTY errors. A mistake, such as silently accepting a no-argument non-TTY call, would be noticed and is fixed by a revert. The only state touched is local git remote config." + }, + "confidence": "high", + "summary": "In `NewCmdSetDefault` (pkg/cmd/repo/setdefault/setdefault.go), the non-interactive 'repository required' guard must also skip `UnsetMode`, while plain no-argument non-TTY calls keep the same error and the ignored-argument behaviour is left alone. Tests in setdefault_test.go need non-TTY cases for `--unset`, `--view` and the plain error, which means letting each case set TTY mode and asserting `UnsetMode`." + }, + { + "task": "repo-set-default-unset-without-repo", + "attempt": 3, + "ref": "v2.101.0", + "judged_at": "2026-10-14T22:18:43.256809Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 1962, + "cache_read_tokens": 36960, + "cache_creation_tokens": 22599, + "num_turns": 3, + "latency_seconds": 24.2, + "total_cost_usd": 0.17562489999999997 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 0, + "coordinated_surfaces": 0, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task text states the behaviour, but the existing contract lives in setdefault.go RunE: `if !opts.ViewMode && !opts.IO.CanPrompt() && opts.Repo == nil { return cmdutil.FlagErrorf(\"repository required when not running interactively\") }`. The error text, its FlagError type (which prints the usage) and the `--view` precedent all come from nearby code. AGENTS.md also requires keeping flags, arguments, error messages and exit behaviour stable for scripts.", + "silent_invariant": "The change only adjusts a pre-run guard. The unset path in setDefaultRun just calls GitClient.UnsetRemoteResolution on the current default. A wrong change shows up directly: either `--unset `, but the Expected behaviour section doesn't ask for it, and AGENTS.md says to preserve argument and exit behaviour. The intended change is to add `!opts.UnsetMode` to the guard and keep the plain no-argument error.", + "verification": "TestNewCmdSetDefault has 'view flag' and 'unset flag' cases, but the harness always calls io.SetStdinTTY(true) and the assertions check only Repo and ViewMode, not UnsetMode. No existing test covers the non-interactive guard, so the table needs a tty field and new cases. TestDefaultRun covers unset mode only with tty:true.", + "blast_radius": "set-default is a user-facing command, and scripts rely on its non-TTY error behaviour. If the guard is wrong, either the 'repository required' error disappears or `--unset` stays broken. Either way the cost is a reverted CLI release; nothing persists beyond local git config `remote..gh-resolved`." + }, + "confidence": "high", + "summary": "Relax the non-interactive 'repository required' guard in NewCmdSetDefault (pkg/cmd/repo/setdefault/setdefault.go) so that `--unset`, like `--view`, runs without a repository argument when stdin is not a terminal, while plain `gh repo set-default` keeps the existing FlagError. Extend TestNewCmdSetDefault with a TTY toggle and cases for non-TTY `--unset`, non-TTY `--view` and the non-TTY no-argument error that must stay." + }, + { + "task": "repo-sync-checked-out-worktree", + "attempt": 1, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:12:49.969212Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 2725, + "cache_read_tokens": 37233, + "cache_creation_tokens": 21271, + "num_turns": 3, + "latency_seconds": 30.1, + "total_cost_usd": 0.18516696 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the outcome. What it leaves out comes from nearby code and from git: the existing refusal format in executeLocalRepoSync ('refusing to sync due to uncommitted/untracked local changes\\ntip: ...'), and git's own rule that `git branch -f` will not move a branch checked out in another worktree while `update-ref` (used by gitExecuter.UpdateBranch) bypasses that check. The git package has no worktree helper, so detection has to follow git's worktree conventions (`git worktree list --porcelain`).", + "silent_invariant": "The problem is the integrity of another worktree's index and working tree. The current bug already 'works' while silently leaving phantom staged changes. A wrong fix (missing detached or linked worktrees, comparing paths wrongly, or updating the other worktree without checking it is clean, e.g. running reset --hard there) looks fine in the invoking worktree and silently corrupts or loses work elsewhere.", + "coordinated_surfaces": "All in pkg/cmd/repo/sync: a new method on the gitClient interface in git.go, its gitExecuter implementation, the testify mock in mocks.go, the branch logic in executeLocalRepoSync in sync.go, and sync_test.go. All of these must agree, but they are one component.", + "ambiguity": "The task allows either 'update it safely' or 'stop with a clear message that names the worktree'. A narrow tempting fix only handles the reported case (checking the current worktree or a single path) or swaps update-ref for something else without naming the worktree. The existing dirty-tree refusal pattern in sync.go points toward refusing with a tip.", + "verification": "Test_SyncRun in sync_test.go is fully mock-based (mgc.On(\"CurrentBranch\"), mgc.On(\"UpdateBranch\", \"trunk\", \"FETCH_HEAD\")). It pins the existing call sequences, but nothing covers worktrees or exercises real git, so mistakes in parsing `git worktree list` output or in path matching would go uncaught.", + "blast_radius": "gh repo sync is a user-facing command that mutates local repositories. A wrong change confuses users or leaves a worktree inconsistent, which is usually fixable with git and by reverting the release. A fix that resets another worktree could, however, destroy uncommitted work." + }, + "confidence": "medium", + "summary": "executeLocalRepoSync needs to find out whether the target branch is checked out in another worktree, through a new gitClient method backed by `git worktree list --porcelain` with a matching mock. When it is, gh should refuse with a message naming that worktree, or update it safely, instead of moving the ref with update-ref. Mock-based cases go in sync_test.go and the paths without other worktrees must keep working." + }, + { + "task": "repo-sync-checked-out-worktree", + "attempt": 2, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:16:48.094351Z", + "metrics": { + "input_tokens": 8, + "output_tokens": 2819, + "cache_read_tokens": 37945, + "cache_creation_tokens": 22174, + "num_turns": 3, + "latency_seconds": 46.0, + "total_cost_usd": 0.1923581 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 2, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task gives the behaviour, but a correct fix depends on nearby code: the existing refusal format in sync.go (\"refusing to sync due to uncommitted/untracked local changes\\ntip: ...\"), the gitClient interface/gitExecuter pattern in git.go, and the hand-written testify mockGitClient in mocks.go. It also depends on git's own semantics: `update-ref` skips the checked-out-elsewhere check that `git branch -f` performs, and `git worktree list --porcelain` reports branches as `branch refs/heads/`.", + "silent_invariant": "The original bug is already silent: the ref moves and the other worktree's index goes stale. A wrong fix would pass the obvious repro but break in edge cases: matching branch names by substring (main vs main2), handling the current worktree being detached (ErrNotOnAnyBranch path), worktree paths containing spaces, or bare/prunable worktrees. If the fix chooses to update the other worktree, it must respect the IsDirty refusal that today applies only to the current worktree.", + "coordinated_surfaces": "Everything stays inside pkg/cmd/repo/sync. The gitClient interface and gitExecuter in git.go need a new worktree query (or a switch away from update-ref), the decision logic goes into executeLocalRepoSync in sync.go, mockGitClient in mocks.go needs the new method, and sync_test.go needs new cases plus updates to existing ones. No other component, docs or config is affected.", + "ambiguity": "The task explicitly accepts either 'updates it safely' or 'stops with a clear message that names the worktree and tells the user what to do', so a design choice is needed. A narrow fix like changing UpdateBranch to `git branch -f` would refuse, but git's message names the worktree without saying what to do. The existing IsDirty refusal in sync.go shows the expected style of a clear tip.", + "verification": "TestSyncRun in sync_test.go only stubs mockGitClient calls (CurrentBranch, IsDirty, UpdateBranch, each .Once()) and never involves worktrees. gitExecuter in git.go has no tests, so worktree listing and parsing against real git would go untested unless new temp-repo tests are written. Nothing in the repo covers this path today.", + "blast_radius": "gh repo sync is user-facing, and the damage lands in users' local repos: phantom staged changes can be committed by mistake, reverting upstream work. That can usually be fixed with git reset. The risk rises if the fix chooses to reset or fast-forward another worktree that has uncommitted changes, which would lose work." + }, + "confidence": "medium", + "summary": "executeLocalRepoSync must find out, through a new gitClient/gitExecuter method (e.g. parsing `git worktree list --porcelain`) or a ref-update command that respects worktrees, whether another worktree has the target branch checked out. If one does, it must refuse with a message naming that worktree or update it safely. The mock and table tests in sync_test.go must be extended for the checked-out-elsewhere, not-checked-out and current-worktree cases." + }, + { + "task": "repo-sync-checked-out-worktree", + "attempt": 3, + "ref": "v2.98.0", + "judged_at": "2026-10-14T22:21:21.092734Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 3145, + "cache_read_tokens": 62327, + "cache_creation_tokens": 29348, + "num_turns": 4, + "latency_seconds": 44.1, + "total_cost_usd": 0.24435994 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The behaviour follows from git's worktree semantics and nearby code rather than the task text. `UpdateBranch` in pkg/cmd/repo/sync/git.go runs `git update-ref refs/heads/`, which skips git's 'branch is checked out elsewhere' guard. Finding other worktrees means parsing `git worktree list --porcelain`, and git/client.go has no helper for that today. The error should match the existing 'refusing to sync ... tip: ...' message in sync.go:246, and pkg/cmd/pr/checkout/checkout.go already has the repo's conventions for resolving worktrees.", + "silent_invariant": "This is about the integrity of the user's working tree, and the reported bug is itself silent: the ref moves while the other worktree's index stays put, so the reverse diff shows up as staged changes that could get committed. A wrong fix can also look fine. Path comparison or porcelain parsing could miss a worktree (detached HEAD, bare main worktree, paths with spaces, prunable entries). An 'update it safely' path that runs reset --hard or a merge in the other worktree without a dirty check would throw away uncommitted work. The --force branch (`ResetHard`/`useForce`) has to be covered too.", + "coordinated_surfaces": "Everything is in one component: a new method on the `gitClient` interface and `gitExecuter` in git.go (possibly backed by a new `git.Client` method in git/client.go), the hand-written `mockGitClient` in mocks.go, the branching logic in `executeLocalRepoSync` in sync.go, and sync_test.go. All of these must agree, but no other command, config or service is involved.", + "ambiguity": "The task allows two outcomes: update the other worktree safely, or stop with a message that names the worktree and says what to do. There are tempting partial fixes, such as switching to `git branch -f`, whose git error names the worktree but gives no guidance, or checking only the main worktree. The acceptance text and the existing 'refusing to sync' tip pattern in sync.go make the intended result clear.", + "verification": "sync_test.go has table tests for `executeLocalRepoSync` using `mockGitClient` with strict `.On(...).Once()` stubs, for example 'existing branch, non-current' (CurrentBranch=test, UpdateBranch trunk). Those would catch wrong call sequences. But nothing covers worktrees, and mocks can't check real `git worktree list` parsing or the real index state after `update-ref`, so a wrong detection could still pass.", + "blast_radius": "This is the user-facing, released `gh repo sync` command acting on users' local clones. The most likely failure, the bug surviving or a sync being blocked for no reason, can be noticed and recovered with git and fixed by a release. Lasting harm would only come from an auto-update path that discards uncommitted work in another worktree. No server-side data or credentials are involved." + }, + "confidence": "medium", + "summary": "`executeLocalRepoSync` needs to find out whether the target branch is checked out in any other worktree, for example through a new gitClient/git.Client method that parses `git worktree list --porcelain`. If it is, sync must refuse with a message naming that worktree's path, or update it safely, instead of moving the ref with `update-ref`. That means changing the gitClient interface, the executer, mocks.go and the table tests in sync_test.go, while syncing a branch nobody has checked out keeps working as before." + }, + { + "task": "secret-set-app-agents", + "attempt": 1, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:12:34.985965Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2774, + "cache_read_tokens": 57365, + "cache_creation_tokens": 23686, + "num_turns": 4, + "latency_seconds": 31.8, + "total_cost_usd": 0.20396530000000002 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Task: 'GitHub's REST API documents the agents secrets endpoints for repositories and organizations' - correct paths (orgs/%s/%s/secrets, repos/%s/%s/secrets in set/http.go, list/list.go, delete/delete.go), public-key endpoint, supported entities and org payload shape (Dependabot already needs a separate DependabotSecretPayload with string repo IDs in set/http.go putOrgSecret) are fixed by GitHub's external API, not the repo.", + "silent_invariant": "Encryption via getRepoPubKey/getOrgPublicKey is reused; a wrong path/payload yields a visible API error. Edge risk: org visibility/selected repository IDs handling in putOrgSecret and IsSupportedSecretEntity in shared/shared.go could quietly scope a secret more broadly or allow unsupported entities (env/user) only caught on unusual inputs.", + "coordinated_surfaces": "All within pkg/cmd/secret: App constant + GetSecretApp + IsSupportedSecretEntity in shared/shared.go, StringEnumFlag --app lists in set/set.go:198, list/list.go:101, delete/delete.go:84, help text in those three Long strings and secret.go, and possibly the org payload branch in set/http.go.", + "ambiguity": "'behaving the way the existing applications do' leaves open which entities agents supports (repo+org per task), whether org visibility/--repos apply and which payload format the org endpoint expects; sibling Dependabot handling and the API docs resolve it.", + "verification": "set/set_test.go, list/list_test.go, delete/delete_test.go are table-driven httpmock tests (28 test funcs, 61 Dependabot references) that would catch missing enum/wiring, but mocks are self-authored so an incorrect REST path or payload shape matching the test would pass; no live API check.", + "blast_radius": "User-facing gh secret commands; a wrong endpoint fails visibly and is revertible. Secret values remain encrypted via the reused public-key flow, so leakage risk is limited to mis-scoped org visibility." + }, + "confidence": "medium", + "summary": "Add an `agents` App to pkg/cmd/secret/shared (parsing and supported repo/org entities), add it to the --app enums and help text of set/list/delete and secret.go, and make sure the org PUT payload matches GitHub's agents secrets API. Extend the httpmock-based set/list/delete tests to cover agents secrets at repository and organization level." + }, + { + "task": "secret-set-app-agents", + "attempt": 2, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:16:45.342375Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2897, + "cache_read_tokens": 61455, + "cache_creation_tokens": 28031, + "num_turns": 4, + "latency_seconds": 34.8, + "total_cost_usd": 0.2314686 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task points to GitHub's REST API docs for the agents secrets endpoints. The URL segment, whether org secrets support visibility and selected repositories, and the payload type for selected_repository_ids are all set by that API. pkg/cmd/secret/set/http.go already special-cases Dependabot with string IDs in DependabotSecretPayload, which shows these details differ per app and come from outside the repo.", + "silent_invariant": "Encryption and public-key fetching are reused as-is (getOrgPublicKey/getRepoPubKey in set/http.go), and a wrong path fails loudly with an API error. But if putOrgSecret sends the wrong payload shape for agents, or IsSupportedSecretEntity in shared/shared.go is wrong (for example allowing env or user), only the org visibility or selected-repos case and the unsupported-entity case are affected. A plain repository set would still succeed.", + "coordinated_surfaces": "Inside the one pkg/cmd/secret component, these must agree: the constant plus GetSecretApp/IsSupportedSecretEntity in shared/shared.go; the three separate StringEnumFlag --app lists in set/set.go:198, list/list.go:101 and delete/delete.go:84; and the help text in set.go, list.go, delete.go and secret.go.", + "ambiguity": "A tempting partial fix is to add the case to GetSecretApp only. The StringEnumFlag lists would still reject `--app agents`, and IsSupportedSecretEntity would still return false. Following how Dependabot is handled across shared.go and the three commands makes the full change clear. The only open question is the org payload shape, which the API docs settle.", + "verification": "set_test.go, list_test.go and delete_test.go have table-driven httpmock cases for each app (for example 'Dependabot repo' and 'Dependabot org' paths like orgs/UmbrellaCorporation/dependabot/secrets/tVirus). They would catch missing flag or app wiring, but the mocks only check paths and payloads the author writes, so a mismatch with the real agents API would not be caught.", + "blast_radius": "This is a user-facing CLI command that manages secrets, but the encryption path (box-sealed with the fetched public key) does not change. A wrong path or payload gives visible API errors that a revert fixes. The main risk is an org agents secret getting the wrong visibility, which is limited by the existing default --visibility=private in set.go:192." + }, + "confidence": "medium", + "summary": "Add `agents` to the secret app enum, GetSecretApp and IsSupportedSecretEntity (repository and organization only) in pkg/cmd/secret/shared, and to the --app flag lists and help text in set, list, delete and the root secret command. The org payload in set/http.go must match GitHub's agents secrets REST contract, with httpmock tests for repo and org set, list and delete." + }, + { + "task": "secret-set-app-agents", + "attempt": 3, + "ref": "v2.92.0", + "judged_at": "2026-10-14T22:21:00.095219Z", + "metrics": { + "input_tokens": 10, + "output_tokens": 2432, + "cache_read_tokens": 60749, + "cache_creation_tokens": 26401, + "num_turns": 4, + "latency_seconds": 34.5, + "total_cost_usd": 0.21211828000000002 + }, + "scores": { + "hidden_contract": 2, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "Every REST path is built from the app name, e.g. `repos/%s/%s/secrets/%s` and `orgs/%s/%s/secrets/public-key` in set/http.go, list.go and delete.go. Whether `agents` works as that path segment, which entities it supports, and what the org PUT body looks like (Dependabot needs string `selected_repository_ids` via `DependabotSecretPayload`) are all set by GitHub's REST docs. The task only points to those docs and does not spell any of it out.", + "silent_invariant": "The change sits on the secret path: public-key fetch, encryption and org visibility in set.go and putOrgSecret. A wrong path or payload type mostly comes back as an API error. Mistakes that would not show in normal use are a wrong `IsSupportedSecretEntity` mapping (allowing env or user, or rejecting org) and a wrong `selected_repository_ids` encoding when `--visibility selected` is used.", + "coordinated_surfaces": "All changes stay inside pkg/cmd/secret and must agree with each other: the `Agents` constant plus `GetSecretApp` and `IsSupportedSecretEntity` in shared.go, the three `StringEnumFlag` lists for `--app` in set.go, list.go and delete.go, putOrgSecret payload branching in set/http.go, and the help text in set.go, list.go, delete.go and secret.go.", + "ambiguity": "Copying the Dependabot pattern is the obvious approach, but someone has to decide whether agents supports only repository and organization (as the task's 'against a repository and, with --org, against an organization' suggests), which org payload shape it uses, and whether `selected_repository_ids` are ints or strings. Existing code and the REST docs settle these once checked.", + "verification": "set_test.go, list_test.go, delete_test.go and shared_test.go (TestGetSecretApp, TestIsSupportedSecretEntity) use httpmock path stubs and would catch broken wiring. But the new tests will be written from the implementer's own idea of the endpoint path and payload, so a mistaken REST contract would still pass. The acceptance txtar tests need real credentials.", + "blast_radius": "This is a user-facing `gh secret` command that writes secrets to GitHub. A mistake would usually mean failed commands or a wrong entity or visibility mapping, which users would notice and a release revert would fix. Secret values are still encrypted with the per-app public key the existing code fetches." + }, + "confidence": "medium", + "summary": "Add an `agents` App to pkg/cmd/secret/shared with GetSecretApp and IsSupportedSecretEntity support (repository and organization), add it to the `--app` enum in set, list and delete, choose the right org PUT payload in set/http.go, and update help text. Add httpmock-based tests for repo and org set, list and delete whose paths and payloads must match GitHub's documented agents secrets REST endpoints." + }, + { + "task": "skill-install-nested-discovery", + "attempt": 1, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:13:23.662811Z", + "metrics": { + "input_tokens": 16, + "output_tokens": 4503, + "cache_read_tokens": 200908, + "cache_creation_tokens": 47154, + "num_turns": 7, + "latency_seconds": 56.7, + "total_cost_usd": 0.40268315999999993 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The intended behaviour comes from conventions in internal/skills/discovery/discovery.go: matchSkillConventions excludes paths with hasHiddenSegment / hasPluginsAncestor, and matchHiddenDirConventions accepts a hidden dir only at the repo root. Tests pin these choices: 'nested plugins skills not matched as plain skills' (vendor/plugins/hubot/...) and 'too deeply nested hidden dir' (.claude/nested/skills/...). The task's 'keep installing as they do now' is defined by those conventions (name, namespace, label, install path), not by the task text.", + "silent_invariant": "Hidden-directory skills are a deliberate safety gate. filterHiddenDirSkills in install.go and preview.go leaves them out unless --allow-hidden-dirs is set and prints a risk warning; skills/gh-skill/SKILL.md says 'it comes with risks'. Widening the depth match without the hasHiddenSegment guard would label something like foo/.claude/skills/x as a standard 'skills' skill. It would then install with no flag and no warning while everything appears to work. Changing which patterns match could also silently change the namespace or label (DisplayName) and InstallName, so existing skills would land in a different directory.", + "coordinated_surfaces": "Every caller shares one matcher: remote DiscoverSkillsWithOptions, local DiscoverLocalSkillsWithOptions, DiscoverSkillByPath, update.go's DiscoverSkills, publish, preview and search.go's MatchSkillPath. The 'Expected skills in ...' text in four error messages and the docs in skills/gh-skill/SKILL.md and install help would have to stay consistent. All of this sits inside the skills component.", + "ambiguity": "The literal repro (foo/bar/skills//SKILL.md) already matches at discovery.go:433 and is tested ('deeply nested skills directory', 'discovers skills in nested skills/ directory'). That leaves two plausible readings. One is to add a nested test and call it fixed. The other is that the real gap is hidden dirs nested below the root, which the --allow-hidden-dirs mention points to and which an explicit test currently forbids. The phrase 'at any depth' also conflicts with the deliberate plugins/ and hidden-segment exclusions.", + "verification": "internal/skills/discovery/discovery_test.go has table tests for matchSkillConventions, matchHiddenDirConventions, DiscoverSkills, DiscoverSkillByPath and local discovery, and install_test.go covers the hidden-dir flag. These would catch regressions in existing conventions. Nothing tests a hidden segment below the root (e.g. foo/.claude/skills/x) going through the standard matcher, so a gate bypass at depth could get through.", + "blast_radius": "gh skill install is a user-facing CLI command that writes skill files into users' agent directories. A mistake would mean skills are missed, mislabelled or installed without the hidden-dir warning. That is noticeable and fixed by reverting, with no stored data or credentials at risk, though installing an unwarned skill has some supply-chain exposure." + }, + "confidence": "medium", + "summary": "Nested skills/ discovery at any depth already exists in internal/skills/discovery/discovery.go and is tested, so the work is to find the real gap (most likely hidden dirs nested below the repo root, or a shared code path that still behaves differently) and fix it in the one shared matcher. The fix must keep the hidden-dir and plugins exclusions, the --allow-hidden-dirs gate, and the existing namespace, label and install-name results, and add tests for a nested skill." + }, + { + "task": "skill-install-nested-discovery", + "attempt": 2, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:17:43.429597Z", + "metrics": { + "input_tokens": 20, + "output_tokens": 5343, + "cache_read_tokens": 240817, + "cache_creation_tokens": 46550, + "num_turns": 9, + "latency_seconds": 65.1, + "total_cost_usd": 0.4266387399999999 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task's 'skills it finds today keep installing as they do now' depends on in-repo rules: the conventions in matchSkillConventions/matchHiddenDirConventions (skills, skills-namespaced, plugins, root, hidden-dir), the hasHiddenSegment/hasPluginsAncestor exclusions, DisplayName tags, and the hidden-dir opt-in in filterHiddenDirSkills (install.go and preview.go). All of this can be read from the surrounding code.", + "silent_invariant": "--allow-hidden-dirs is a safety opt-in ('Don't use this unless you need to, it comes with risks' in skills/gh-skill/SKILL.md; warning in filterHiddenDirSkills). If deeper matching labels nested dot-directory skills as 'skills' instead of 'hidden-dir', they install without the flag or the warning while everything appears to work. The plugins/ exclusion and the seen[skillDir] dedup can also break quietly.", + "coordinated_surfaces": "matchSkillConventions/matchHiddenDirConventions are shared by DiscoverSkillsWithOptions (install, preview), DiscoverLocalSkillsWithOptions (--from-local, publish), MatchSkillPath (search) and DiscoverSkills (update). The 'no skills found' messages and install help text describe the conventions. All of it is inside the skills component.", + "ambiguity": "The literal reproduction foo/bar/skills//SKILL.md already matches (discovery.go lines 428-444; tests 'deeply nested skills directory' a/b/c/skills/my-skill and 'discovers skills in nested skills directory'). Adding a test for it passes with no fix. The likely real gap is hidden-dir skills below the root (matchHiddenDirConventions only matches .{host}/skills at root; test 'too deeply nested hidden dir' pins .claude/nested/skills/... to nil), which the task never states.", + "verification": "discovery_test.go has table-driven matcher tests and httpmock tree tests, and install_test.go covers hidden-dir include/exclude, so regressions in today's conventions would be caught. Nothing checks that nested dot-directory skills stay behind --allow-hidden-dirs, and one test pins the current root-only hidden behavior, so a gate bypass could pass.", + "blast_radius": "gh skill install is user-facing and writes agent skill files to user and project directories. A wrong change could miss skills or skip the hidden-dir warning. It is a client-side CLI change that a release revert undoes, with no stored server data, credentials or migrations involved." + }, + "confidence": "medium", + "summary": "Discovery in internal/skills/discovery/discovery.go already finds non-hidden nested skills/ directories at any depth, so the work is to find the real gap (most likely skills in hidden directories below the repository root under --allow-hidden-dirs) and extend matching without changing existing conventions. Nested hidden-dir skills must stay behind the --allow-hidden-dirs gate, and the change needs tests showing a nested skill that the current code does not find." + }, + { + "task": "skill-install-nested-discovery", + "attempt": 3, + "ref": "v2.93.0", + "judged_at": "2026-10-14T22:22:08.519325Z", + "metrics": { + "input_tokens": 28, + "output_tokens": 6292, + "cache_read_tokens": 405849, + "cache_creation_tokens": 50664, + "num_turns": 13, + "latency_seconds": 85.8, + "total_cost_usd": 0.50648598 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 2, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task comes from cli/cli#13372, which is closed and is quoted almost in full. What 'any depth' means for hidden and plugin paths is set by nearby code: matchSkillConventions excludes paths with hidden segments (hasHiddenSegment) or a plugins ancestor (hasPluginsAncestor), and matchHiddenDirConventions requires the hidden root at the repo's top level. The same matcher also feeds search.go extractSkillInfo via MatchSkillPath, local discovery, preview, and the DiscoverSkillByPath namespace logic.", + "silent_invariant": "The --allow-hidden-dirs gate (filterHiddenDirSkills in install.go and preview.go, PartitionHiddenDirSkills, IsHiddenDirConvention) only works if every dot-prefixed path is classed as 'hidden-dir'. skills/gh-skill/SKILL.md says the flag 'comes with risks'. If a change widened nested matching and let a path like foo/.claude/skills/x through as convention 'skills', it would install with no flag and no warning, and nothing would look wrong.", + "coordinated_surfaces": "Everything stays inside the skills component. matchSkillConventions and matchHiddenDirConventions are shared by DiscoverSkillsWithOptions (remote), DiscoverLocalSkillsWithOptions (local), preview, search (MatchSkillPath) and DiscoverSkillByPath's namespace/convention parsing. The 'no skills found' convention lists in discovery.go, the install help text and the docs must stay in step with them.", + "ambiguity": "In this clone, foo/bar/skills//SKILL.md already matches (discovery_test.go: 'deeply nested skills directory' a/b/c/skills/my-skill, 'nested skills directory' terraform/code-generation/...). So adding a test and calling it done looks complete. The reporter used --allow-hidden-dirs, though, and tests still pin '.claude/nested/skills/code-review/SKILL.md' and 'vendor/plugins/hubot/skills/pr-summary/SKILL.md' as not found. Two readings remain: re-verify the non-hidden case, or extend nested matching to hidden and plugin paths while keeping the hidden gate.", + "verification": "discovery_test.go has strong table tests for matchSkillConventions and matchHiddenDirConventions, remote and local discovery, and DiscoverSkillByPath. install_test.go and preview_test.go cover hidden-dir filtering. But no test covers nested hidden paths under --allow-hidden-dirs, nested plugins, or install/preview end to end for a nested skill. Some regressions in how paths are classed for the hidden gate would slip through.", + "blast_radius": "gh skill install is user-facing and writes agent skill files into project or user host dirs. A bad matcher change could install the wrong skills, change install names or namespaces, or skip the hidden-dir warning, and installed files outlast a revert. But no credentials, money or stored records are at stake, and reverting the code fixes the behaviour." + }, + "confidence": "medium", + "summary": "The core repro (foo/bar/skills//SKILL.md) already matches in internal/skills/discovery/discovery.go and has tests. The real work is to decide whether nested skills under hidden or plugins directories should be discovered when --allow-hidden-dirs is set, and to extend the shared matcher (used by remote, local, preview, search and path-based install) without breaking today's conventions or the hidden-dir gate. Nested-skill tests are needed across discovery and install." + }, + { + "task": "skill-update-dir-keeps-install-path", + "attempt": 1, + "ref": "v2.94.0", + "judged_at": "2026-10-14T22:13:11.820433Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 3795, + "cache_read_tokens": 83643, + "cache_creation_tokens": 27870, + "num_turns": 5, + "latency_seconds": 46.9, + "total_cost_usd": 0.25522926 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour, but the current behaviour was designed on purpose and is pinned in the repo. installer.go installSkill/installLocalSkill always write to filepath.Join(baseDir, skill.Name) ('Use skill.Name (not InstallName) for a flat directory layout'). update_test.go case 'namespaced skill with --dir resolves install base correctly' asserts the skill ends up flat and that 'old namespaced directory should be removed'. You only learn this by reading the nearby code and tests. The lockfile is keyed by skill.InstallName().", + "silent_invariant": "update.go lines 419-435 call os.RemoveAll(u.local.dir) and remove the parent folder if it is empty, then print 'Updated ' as success. A partial fix, such as fixing only the --dir branch at lines 405-411, still sends namespaced installs found through an agent host to host.InstallDir/skill.Name. Mismatches between skill.Name and the folder name would also go unnoticed. The command keeps reporting success while user files move or disappear, and only inspecting the filesystem shows it.", + "coordinated_surfaces": "One component. The base-directory logic and the cleanup block in pkg/cmd/skills/update/update.go must change together, update_test.go must be rewritten to match, and how installer.Install builds the path (Dir + skill.Name) has to stay consistent with that. No docs, config or other services are involved.", + "ambiguity": "Narrower fixes would look complete: dropping only the RemoveAll still leaves a relocated duplicate, and fixing only the --dir branch misses the agent-host path (installOpts.Dir is empty, so it falls back to host.InstallDir plus skill.Name). The task closes these off by naming both causes and requiring 'flat and namespaced layouts, with or without --dir'.", + "verification": "TestUpdateRun in update_test.go runs offline with httpmock and temp dirs, so the change can be tested. However, the only namespaced case asserts the opposite of the new requirement, and no test covers in-place updates for agent-host namespaced installs or checks that the parent folder survives. Those tests have to be written.", + "blast_radius": "gh is a CLI that customers use, and a wrong change would keep moving or deleting skill folders in user projects and breaking agent configs and symlinks. The skill content can be fetched again from GitHub, and reverting the CLI change stops further harm, though folders already deleted stay deleted." + }, + "confidence": "medium", + "summary": "The task is to change updateRun in pkg/cmd/skills/update/update.go so every update installs into the skill's current parent folder, for both --dir and agent-host scans and for flat and namespaced layouts. It must also remove the RemoveAll/parent-cleanup step and replace the existing test that requires flattening with tests showing the skill stays in place and the old path is never deleted." + }, + { + "task": "skill-update-dir-keeps-install-path", + "attempt": 2, + "ref": "v2.94.0", + "judged_at": "2026-10-14T22:17:37.118163Z", + "metrics": { + "input_tokens": 12, + "output_tokens": 3249, + "cache_read_tokens": 78319, + "cache_creation_tokens": 26415, + "num_turns": 5, + "latency_seconds": 66.1, + "total_cost_usd": 0.23404347999999997 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 1, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 1 + }, + "evidence": { + "hidden_contract": "The task states the expected behaviour, but pkg/cmd/skills/update/update.go:419-435 deliberately moves skills from a namespaced to a flat layout ('migrating from a namespaced layout to flat'). The test 'namespaced skill with --dir resolves install base correctly' (update_test.go:712-773) asserts that move and the deletion, and installer.go:251-253 always installs flat via skill.Name. The fix has to reverse these conventions, which only the surrounding code reveals.", + "silent_invariant": "Correctness depends on never calling os.RemoveAll on the user's existing skill directory (update.go:428-434) and on the target base matching u.local.dir. A fix that only corrects the --dir base calculation (update.go:405-411) passes the reported repro. Namespaced skills found via scanAllAgents (host != nil) still get moved to host.InstallDir()/name, which a normal check would miss.", + "coordinated_surfaces": "The changes stay in pkg/cmd/skills/update/update.go (install base derivation plus removal of the cleanup block) and update_test.go. These must agree with internal/skills/installer/installer.go's rule that the install lands at /, and possibly with the lockfile name recording (skill.InstallName), all inside the skills subsystem.", + "ambiguity": "The task says 'for flat and namespaced layouts, with or without --dir' and 'never deletes the previous install path'. That points to always using filepath.Dir(u.local.dir) as the base and dropping the cleanup, rather than just fixing the strings.Contains(name, \"/\") double-Dir logic for the --dir repro.", + "verification": "update_test.go has table-driven tests with httpmock and temp dirs that exercise the update/install path for flat and namespaced --dir cases. The namespaced case currently asserts the old relocate-and-delete behaviour and must be inverted, and I saw no test for namespaced skills updated through agent-host scanning without --dir.", + "blast_radius": "gh skill update is a user-facing CLI command that writes to and deletes from users' project and home skill directories (os.RemoveAll in update.go:429). A wrong fix silently moves or deletes installed skills and breaks configs that point to them. Content can be reinstalled from GitHub and a revert fixes the code, though local edits to a skill could be lost." + }, + "confidence": "medium", + "summary": "In pkg/cmd/skills/update/update.go, the update loop must reinstall each skill into its current parent directory (filepath.Dir(u.local.dir)) for both the --dir path and the agent-host path, and the post-install cleanup that removes the old directory and its empty parent must be deleted. update_test.go must flip the existing namespaced-migration test and add cases for flat and namespaced skills with and without --dir, checking that the skill stays in place and the old path survives." + }, + { + "task": "skill-update-dir-keeps-install-path", + "attempt": 3, + "ref": "v2.94.0", + "judged_at": "2026-10-14T22:21:41.231475Z", + "metrics": { + "input_tokens": 14, + "output_tokens": 3644, + "cache_read_tokens": 106003, + "cache_creation_tokens": 31670, + "num_turns": 6, + "latency_seconds": 43.2, + "total_cost_usd": 0.27773526 + }, + "scores": { + "hidden_contract": 1, + "silent_invariant": 2, + "coordinated_surfaces": 1, + "ambiguity": 1, + "verification": 1, + "blast_radius": 2 + }, + "evidence": { + "hidden_contract": "The task states the target behaviour. The code assumes the opposite on purpose: installer.go says 'Use skill.Name (not InstallName) so skills are always installed flat', the comment at update.go:419 says 'migrating from a namespaced layout to flat', and update_test.go:712 checks that the old namespaced dir is removed. The fix has to knowingly reverse that design, which you only learn from the nearby code.", + "silent_invariant": "update.go:428-435 runs os.RemoveAll on the user's existing skill dir and its empty parent, and still prints 'Updated '. A partial fix, such as patching only the host==nil branch at line 405, leaves skills found through an agent host (namespaced layout) or skills whose dir name differs from skill.Name moving and being deleted. The command would still report success, so the data loss stays silent.", + "coordinated_surfaces": "Contained in the update command: the install-base logic at update.go:401-411 and the cleanup at 419-435 must agree with installer.Install, which writes to filepath.Join(Dir, skill.Name), plus update_test.go. Nothing outside the skills packages is involved.", + "ambiguity": "The tempting narrow fix is to drop the extra filepath.Dir for namespaced names when --dir is set. That fixes the repro but leaves the host path, where installOpts.Dir is empty, relocating into the flat InstallDir, and leaves the RemoveAll cleanup in place. The task's 'with or without --dir' and 'never deletes the previous install path' rule that out, so the intended answer is clear once read.", + "verification": "update_test.go has httpmock and temp-dir end-to-end tests of updateRun, so the behaviour is testable locally. But the existing case 'namespaced skill with --dir resolves install base correctly' (line 712) checks for the wrong behaviour and has to be rewritten. No test covers a namespaced skill updated through an agent host or checks that the old path survives.", + "blast_radius": "`gh skill update` is a user-facing command, and a wrong fix keeps silently deleting files in users' project or home skill directories with os.RemoveAll. Local edits and any symlinks or agent config pointing at the old path are lost, and reverting the code does not bring the deleted files back." + }, + "confidence": "medium", + "summary": "In pkg/cmd/skills/update/update.go, the update must reinstall each skill into the parent of its current directory for flat and namespaced layouts, whether or not an agent host is involved, and the RemoveAll/parent-cleanup block must go. update_test.go needs its namespaced test turned around and new cases showing the original path is kept and updated in place." + } + ], + "rubric": "/home/ubuntu/agentic-coding-harness-benchmarks/benchmarks/config/swe-router/rubric.md", + "policy": { + "schema_version": "1.0", + "required_quality": { + "by_blast_radius": { + "0": 60.0, + "1": 70.0, + "2": 75.0 + }, + "add_when_verification_is_2": 5.0, + "meaning": { + "60": "Bottom of the judge's 'strong' band: a usable foundation; expect review to fix real gaps.", + "65": "Low 'strong': implementation-ready in parts; review will find gaps.", + "70": "Middle of 'strong': implementation-ready in most respects with limited gaps.", + "75": "Top of 'strong': few gaps and well-grounded; for security, money or data paths.", + "80": "Approaching the judge's 'excellent' band: comprehensive and precise." + } + }, + "difficulty": { + "dimensions": [ + "hidden_contract", + "silent_invariant", + "coordinated_surfaces", + "ambiguity" + ], + "bucketing": "absolute", + "buckets": { + "easy": [ + 0, + 1 + ], + "moderate": [ + 2, + 3 + ], + "hard": [ + 4, + 5 + ], + "very_hard": [ + 6, + 8 + ] + } + }, + "tie_band_points": 3.0, + "when_unsure": "stay_on_current_model" + } +} diff --git a/docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-in-repo.json b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-in-repo.json new file mode 100644 index 00000000..f0b6fa08 --- /dev/null +++ b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-in-repo.json @@ -0,0 +1,573 @@ +{ + "config": { + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "scope": "cli-cli-v1", + "harness": "omp", + "skill": "swe3", + "baseline": "claude-opus-5-5", + "tasks": 21, + "available": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-opus-5-5" + ], + "allow_list": "auto", + "no_allow_list": false, + "tie_band": 3.0, + "holdout": true, + "columns_from": null, + "judged_by": { + "harness": "omp", + "model": "us.anthropic.claude-haiku-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "repeats": 3, + "started_at": "2026-10-14T22:10:12.037956Z", + "finished_at": "2026-10-14T22:21:48.671252Z", + "judging_cost_usd": 0.6335, + "runs_ok": 63, + "runs_failed": 0, + "mode": "rubric", + "bucketing": "absolute" + }, + "floor_note": "Required quality per task, from the rubric pass by omp + us.anthropic.claude-haiku-5-5: blast radius and verification, priced by routing-policy.json.", + "tier_note": "Difficulty bucket per task from the rubric pass (absolute bucketing); every model's existing scores regrouped into bucket columns (easy, moderate, hard, very hard shown as trivial, low, medium, high). The dataset's own label is kept beside each row.", + "floors_label": "per-task floors judged by omp + us.anthropic.claude-haiku-5-5 running the skill's step 1" + }, + "provenance": { + "measured_on": "2026-10-11", + "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", + "source_sha256": "72b1ed9ee222", + "harness": "omp", + "skill": "swe3", + "dataset": "mcp-gateway-registry-v2", + "judge": { + "model": "openai.gpt-5.6-sol", + "reasoning_effort": "high", + "repo_grounded": true + }, + "repository": "https://github.com/aarora79/agentic-coding-harness-benchmarks" + }, + "runs": [ + { + "floor": null, + "rows": [ + { + "task": "repo-create-reject-empty-name", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.0, + "baseline_cost_usd": 1.295, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.42, + "actual_score": 85.6, + "actual_cost_usd": 0.0549, + "actual_failed": false, + "switched": true, + "score_delta": -2.4, + "cost_delta_usd": -1.2402, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.6, + "baseline_cost_usd": 0.9935, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 80.44, + "actual_score": 87.6, + "actual_cost_usd": 0.0521, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -0.9414, + "cost_saving_pct": 94.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": true + }, + { + "task": "repo-set-default-unset-without-repo", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.8, + "baseline_cost_usd": 0.9055, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.78, + "actual_score": 82.0, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -10.8, + "cost_delta_usd": -0.8311, + "cost_saving_pct": 91.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.4, + "baseline_cost_usd": 1.025, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.53, + "actual_score": 80.0, + "actual_cost_usd": 0.0441, + "actual_failed": false, + "switched": true, + "score_delta": -12.4, + "cost_delta_usd": -0.9809, + "cost_saving_pct": 95.7, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "extension-uninstall-alias", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.8, + "baseline_cost_usd": 0.7346, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 84.73, + "actual_score": 61.4, + "actual_cost_usd": 0.0456, + "actual_failed": false, + "switched": true, + "score_delta": -27.4, + "cost_delta_usd": -0.689, + "cost_saving_pct": 93.8, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.0, + "baseline_cost_usd": 1.6219, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.2, + "actual_score": 87.0, + "actual_cost_usd": 0.0678, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -1.5541, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.0, + "baseline_cost_usd": 1.893, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.24, + "actual_score": 87.4, + "actual_cost_usd": 0.0655, + "actual_failed": false, + "switched": true, + "score_delta": -2.6, + "cost_delta_usd": -1.8275, + "cost_saving_pct": 96.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.2, + "baseline_cost_usd": 1.165, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.13, + "actual_score": 87.2, + "actual_cost_usd": 0.045, + "actual_failed": false, + "switched": true, + "score_delta": 1.0, + "cost_delta_usd": -1.12, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "release-create-404-scope-blame", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.8, + "baseline_cost_usd": 1.3867, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.28, + "actual_score": 87.0, + "actual_cost_usd": 0.0622, + "actual_failed": false, + "switched": true, + "score_delta": 0.2, + "cost_delta_usd": -1.3246, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.0, + "baseline_cost_usd": 2.3436, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.5, + "actual_score": 84.8, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -2.2, + "cost_delta_usd": -2.2692, + "cost_saving_pct": 96.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 89.6, + "baseline_cost_usd": 2.4499, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 77.8, + "actual_cost_usd": 0.1565, + "actual_failed": false, + "switched": true, + "score_delta": -11.8, + "cost_delta_usd": -2.2934, + "cost_saving_pct": 93.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.2, + "baseline_cost_usd": 2.96, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 73.6, + "actual_score": 75.4, + "actual_cost_usd": 0.1026, + "actual_failed": false, + "switched": true, + "score_delta": -11.8, + "cost_delta_usd": -2.8574, + "cost_saving_pct": 96.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 69.0, + "baseline_cost_usd": 1.9909, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.68, + "actual_score": 83.0, + "actual_cost_usd": 0.0686, + "actual_failed": false, + "switched": true, + "score_delta": 14.0, + "cost_delta_usd": -1.9223, + "cost_saving_pct": 96.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.6, + "baseline_cost_usd": 2.0562, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.56, + "actual_score": 84.2, + "actual_cost_usd": 0.1242, + "actual_failed": false, + "switched": true, + "score_delta": -1.4, + "cost_delta_usd": -1.9319, + "cost_saving_pct": 94.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.4, + "baseline_cost_usd": 2.723, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.88, + "actual_score": 81.0, + "actual_cost_usd": 0.1215, + "actual_failed": false, + "switched": true, + "score_delta": -7.4, + "cost_delta_usd": -2.6015, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.8, + "baseline_cost_usd": 2.47, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.82, + "actual_score": 81.6, + "actual_cost_usd": 0.1017, + "actual_failed": false, + "switched": true, + "score_delta": -9.2, + "cost_delta_usd": -2.3682, + "cost_saving_pct": 95.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.4, + "baseline_cost_usd": 2.0202, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 79.6, + "actual_score": 75.2, + "actual_cost_usd": 1.0418, + "actual_failed": false, + "switched": true, + "score_delta": -6.2, + "cost_delta_usd": -0.9783, + "cost_saving_pct": 48.4, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "auth-clipboard-config-default", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 72.4, + "baseline_cost_usd": 4.1118, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.71, + "actual_score": null, + "actual_cost_usd": 0.2412, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -3.8705, + "cost_saving_pct": 94.1, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.4, + "baseline_cost_usd": 3.8112, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.44, + "actual_score": 75.4, + "actual_cost_usd": 0.1447, + "actual_failed": false, + "switched": true, + "score_delta": -10.0, + "cost_delta_usd": -3.6665, + "cost_saving_pct": 96.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.2, + "baseline_cost_usd": 2.538, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 80.1, + "actual_score": 62.4, + "actual_cost_usd": 0.0776, + "actual_failed": false, + "switched": true, + "score_delta": -27.8, + "cost_delta_usd": -2.4604, + "cost_saving_pct": 96.9, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "skill-install-nested-discovery", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 63.4, + "baseline_cost_usd": 5.3843, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.71, + "actual_score": null, + "actual_cost_usd": 0.1297, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -5.2546, + "cost_saving_pct": 97.6, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + } + ], + "totals": { + "tasks": 21, + "tasks_switched": 21, + "tasks_scored_both_arms": 19, + "baseline_model": "claude-opus-5-5", + "baseline_total_cost_usd": 45.88, + "routed_total_cost_usd": 2.9, + "cost_saving_usd": 42.98, + "cost_saving_pct": 93.7, + "baseline_mean_score": 87.48, + "routed_mean_score": 80.32, + "mean_score_delta": -7.17, + "baseline_mean_score_failures_as_zero": 85.62, + "routed_mean_score_failures_as_zero": 72.67, + "mean_score_delta_failures_as_zero": -12.95, + "tasks_below_floor_routed": 4, + "tasks_below_floor_baseline": 2, + "tasks_failed_routed": 2, + "tasks_failed_baseline": 0, + "models_used": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ], + "model_counts": { + "claude-haiku-5-5": 20, + "claude-sonnet-5-5": 1 + }, + "quality_delta_pct": -8.2, + "mean_per_task_saving_pct": 93.2 + } + } + ] +} diff --git a/docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-zero-shot.json b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-zero-shot.json new file mode 100644 index 00000000..dfa49e6d --- /dev/null +++ b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-haiku-5-5-zero-shot.json @@ -0,0 +1,574 @@ +{ + "config": { + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "scope": "cli-cli-v1", + "harness": "omp", + "skill": "swe3", + "baseline": "claude-opus-5-5", + "tasks": 21, + "available": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-opus-5-5" + ], + "allow_list": "auto", + "no_allow_list": false, + "tie_band": 3.0, + "holdout": false, + "columns_from": "mcp-gateway-registry-v2", + "judged_by": { + "harness": "omp", + "model": "us.anthropic.claude-haiku-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "repeats": 3, + "started_at": "2026-10-14T22:10:12.037956Z", + "finished_at": "2026-10-14T22:21:48.671252Z", + "judging_cost_usd": 0.6335, + "runs_ok": 63, + "runs_failed": 0, + "mode": "rubric", + "bucketing": "absolute", + "columns_from": "mcp-gateway-registry-v2" + }, + "floor_note": "Required quality per task, from the rubric pass by omp + us.anthropic.claude-haiku-5-5: blast radius and verification, priced by routing-policy.json.", + "tier_note": "Difficulty bucket per task from the rubric pass (absolute bucketing); every model's existing scores regrouped into bucket columns (easy, moderate, hard, very hard shown as trivial, low, medium, high). The dataset's own label is kept beside each row.", + "floors_label": "per-task floors judged by omp + us.anthropic.claude-haiku-5-5 running the skill's step 1" + }, + "provenance": { + "measured_on": "2026-10-11", + "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", + "source_sha256": "72b1ed9ee222", + "harness": "omp", + "skill": "swe3", + "dataset": "mcp-gateway-registry-v2", + "judge": { + "model": "openai.gpt-5.6-sol", + "reasoning_effort": "high", + "repo_grounded": true + }, + "repository": "https://github.com/aarora79/agentic-coding-harness-benchmarks" + }, + "runs": [ + { + "floor": null, + "rows": [ + { + "task": "repo-create-reject-empty-name", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.0, + "baseline_cost_usd": 1.295, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 85.6, + "actual_cost_usd": 0.0549, + "actual_failed": false, + "switched": true, + "score_delta": -2.4, + "cost_delta_usd": -1.2402, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.6, + "baseline_cost_usd": 0.9935, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 79.49, + "actual_score": 87.0, + "actual_cost_usd": 0.3683, + "actual_failed": false, + "switched": true, + "score_delta": -4.6, + "cost_delta_usd": -0.6252, + "cost_saving_pct": 62.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": true + }, + { + "task": "repo-set-default-unset-without-repo", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.8, + "baseline_cost_usd": 0.9055, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 82.0, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -10.8, + "cost_delta_usd": -0.8311, + "cost_saving_pct": 91.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.4, + "baseline_cost_usd": 1.025, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.8, + "actual_score": 80.0, + "actual_cost_usd": 0.0441, + "actual_failed": false, + "switched": true, + "score_delta": -12.4, + "cost_delta_usd": -0.9809, + "cost_saving_pct": 95.7, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "extension-uninstall-alias", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.8, + "baseline_cost_usd": 0.7346, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.8, + "actual_score": 61.4, + "actual_cost_usd": 0.0456, + "actual_failed": false, + "switched": true, + "score_delta": -27.4, + "cost_delta_usd": -0.689, + "cost_saving_pct": 93.8, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.0, + "baseline_cost_usd": 1.6219, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.8, + "actual_score": 87.0, + "actual_cost_usd": 0.0678, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -1.5541, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.0, + "baseline_cost_usd": 1.893, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 87.4, + "actual_cost_usd": 0.0655, + "actual_failed": false, + "switched": true, + "score_delta": -2.6, + "cost_delta_usd": -1.8275, + "cost_saving_pct": 96.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.2, + "baseline_cost_usd": 1.165, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.8, + "actual_score": 87.2, + "actual_cost_usd": 0.045, + "actual_failed": false, + "switched": true, + "score_delta": 1.0, + "cost_delta_usd": -1.12, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "release-create-404-scope-blame", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.8, + "baseline_cost_usd": 1.3867, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 87.0, + "actual_cost_usd": 0.0622, + "actual_failed": false, + "switched": true, + "score_delta": 0.2, + "cost_delta_usd": -1.3246, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.0, + "baseline_cost_usd": 2.3436, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 84.8, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -2.2, + "cost_delta_usd": -2.2692, + "cost_saving_pct": 96.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 89.6, + "baseline_cost_usd": 2.4499, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 77.8, + "actual_cost_usd": 0.1565, + "actual_failed": false, + "switched": true, + "score_delta": -11.8, + "cost_delta_usd": -2.2934, + "cost_saving_pct": 93.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.2, + "baseline_cost_usd": 2.96, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.94, + "actual_score": 79.4, + "actual_cost_usd": 1.177, + "actual_failed": false, + "switched": true, + "score_delta": -7.8, + "cost_delta_usd": -1.783, + "cost_saving_pct": 60.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 69.0, + "baseline_cost_usd": 1.9909, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 83.0, + "actual_cost_usd": 0.0686, + "actual_failed": false, + "switched": true, + "score_delta": 14.0, + "cost_delta_usd": -1.9223, + "cost_saving_pct": 96.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.6, + "baseline_cost_usd": 2.0562, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 84.2, + "actual_cost_usd": 0.1242, + "actual_failed": false, + "switched": true, + "score_delta": -1.4, + "cost_delta_usd": -1.9319, + "cost_saving_pct": 94.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.4, + "baseline_cost_usd": 2.723, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 81.0, + "actual_cost_usd": 0.1215, + "actual_failed": false, + "switched": true, + "score_delta": -7.4, + "cost_delta_usd": -2.6015, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.8, + "baseline_cost_usd": 2.47, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 81.6, + "actual_cost_usd": 0.1017, + "actual_failed": false, + "switched": true, + "score_delta": -9.2, + "cost_delta_usd": -2.3682, + "cost_saving_pct": 95.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.4, + "baseline_cost_usd": 2.0202, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.94, + "actual_score": 75.2, + "actual_cost_usd": 1.0418, + "actual_failed": false, + "switched": true, + "score_delta": -6.2, + "cost_delta_usd": -0.9783, + "cost_saving_pct": 48.4, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "auth-clipboard-config-default", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 72.4, + "baseline_cost_usd": 4.1118, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": null, + "actual_cost_usd": 0.2412, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -3.8705, + "cost_saving_pct": 94.1, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.4, + "baseline_cost_usd": 3.8112, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": 75.4, + "actual_cost_usd": 0.1447, + "actual_failed": false, + "switched": true, + "score_delta": -10.0, + "cost_delta_usd": -3.6665, + "cost_saving_pct": 96.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.2, + "baseline_cost_usd": 2.538, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.94, + "actual_score": 79.8, + "actual_cost_usd": 0.983, + "actual_failed": false, + "switched": true, + "score_delta": -10.4, + "cost_delta_usd": -1.555, + "cost_saving_pct": 61.3, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "skill-install-nested-discovery", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 63.4, + "baseline_cost_usd": 5.3843, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.97, + "actual_score": null, + "actual_cost_usd": 0.1297, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -5.2546, + "cost_saving_pct": 97.6, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + } + ], + "totals": { + "tasks": 21, + "tasks_switched": 21, + "tasks_scored_both_arms": 19, + "baseline_model": "claude-opus-5-5", + "baseline_total_cost_usd": 45.88, + "routed_total_cost_usd": 5.19, + "cost_saving_usd": 40.69, + "cost_saving_pct": 88.7, + "baseline_mean_score": 87.48, + "routed_mean_score": 81.41, + "mean_score_delta": -6.07, + "baseline_mean_score_failures_as_zero": 85.62, + "routed_mean_score_failures_as_zero": 73.66, + "mean_score_delta_failures_as_zero": -11.96, + "tasks_below_floor_routed": 3, + "tasks_below_floor_baseline": 2, + "tasks_failed_routed": 2, + "tasks_failed_baseline": 0, + "models_used": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ], + "model_counts": { + "claude-haiku-5-5": 17, + "claude-sonnet-5-5": 4 + }, + "quality_delta_pct": -6.9, + "mean_per_task_saving_pct": 88.3 + } + } + ] +} diff --git a/docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-in-repo.json b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-in-repo.json new file mode 100644 index 00000000..2e73496b --- /dev/null +++ b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-in-repo.json @@ -0,0 +1,571 @@ +{ + "config": { + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "scope": "cli-cli-v1", + "harness": "omp", + "skill": "swe3", + "baseline": "claude-opus-5-5", + "tasks": 21, + "available": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-opus-5-5" + ], + "allow_list": "auto", + "no_allow_list": false, + "tie_band": 3.0, + "holdout": true, + "columns_from": null, + "judged_by": { + "harness": "omp", + "model": "us.anthropic.claude-opus-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "repeats": 3, + "started_at": "2026-10-14T22:10:12.033833Z", + "finished_at": "2026-10-14T22:23:36.365453Z", + "judging_cost_usd": 15.5931, + "runs_ok": 63, + "runs_failed": 0, + "mode": "rubric", + "bucketing": "absolute" + }, + "floor_note": "Required quality per task, from the rubric pass by omp + us.anthropic.claude-opus-5-5: blast radius and verification, priced by routing-policy.json.", + "tier_note": "Difficulty bucket per task from the rubric pass (absolute bucketing); every model's existing scores regrouped into bucket columns (easy, moderate, hard, very hard shown as trivial, low, medium, high). The dataset's own label is kept beside each row.", + "floors_label": "per-task floors judged by omp + us.anthropic.claude-opus-5-5 running the skill's step 1" + }, + "provenance": { + "measured_on": "2026-10-11", + "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", + "source_sha256": "72b1ed9ee222", + "harness": "omp", + "skill": "swe3", + "dataset": "mcp-gateway-registry-v2", + "judge": { + "model": "openai.gpt-5.6-sol", + "reasoning_effort": "high", + "repo_grounded": true + }, + "repository": "https://github.com/aarora79/agentic-coding-harness-benchmarks" + }, + "runs": [ + { + "floor": null, + "rows": [ + { + "task": "repo-create-reject-empty-name", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.0, + "baseline_cost_usd": 1.295, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.57, + "actual_score": 85.6, + "actual_cost_usd": 0.0549, + "actual_failed": false, + "switched": true, + "score_delta": -2.4, + "cost_delta_usd": -1.2402, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.6, + "baseline_cost_usd": 0.9935, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 80.0, + "actual_score": 87.6, + "actual_cost_usd": 0.0521, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -0.9414, + "cost_saving_pct": 94.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": true + }, + { + "task": "repo-set-default-unset-without-repo", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.8, + "baseline_cost_usd": 0.9055, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 82.17, + "actual_score": 82.0, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -10.8, + "cost_delta_usd": -0.8311, + "cost_saving_pct": 91.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "trivial", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.4, + "baseline_cost_usd": 1.025, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 87.6, + "actual_score": 80.0, + "actual_cost_usd": 0.0441, + "actual_failed": false, + "switched": true, + "score_delta": -12.4, + "cost_delta_usd": -0.9809, + "cost_saving_pct": 95.7, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": true + }, + { + "task": "extension-uninstall-alias", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.8, + "baseline_cost_usd": 0.7346, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 85.6, + "actual_score": 61.4, + "actual_cost_usd": 0.0456, + "actual_failed": false, + "switched": true, + "score_delta": -27.4, + "cost_delta_usd": -0.689, + "cost_saving_pct": 93.8, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.0, + "baseline_cost_usd": 1.6219, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.33, + "actual_score": 87.0, + "actual_cost_usd": 0.0678, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -1.5541, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.0, + "baseline_cost_usd": 1.893, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.78, + "actual_score": 87.4, + "actual_cost_usd": 0.0655, + "actual_failed": false, + "switched": true, + "score_delta": -2.6, + "cost_delta_usd": -1.8275, + "cost_saving_pct": 96.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.2, + "baseline_cost_usd": 1.165, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.3, + "actual_score": 87.2, + "actual_cost_usd": 0.045, + "actual_failed": false, + "switched": true, + "score_delta": 1.0, + "cost_delta_usd": -1.12, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "release-create-404-scope-blame", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.8, + "baseline_cost_usd": 1.3867, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.33, + "actual_score": 87.0, + "actual_cost_usd": 0.0622, + "actual_failed": false, + "switched": true, + "score_delta": 0.2, + "cost_delta_usd": -1.3246, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.0, + "baseline_cost_usd": 2.3436, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.7, + "actual_score": 84.8, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -2.2, + "cost_delta_usd": -2.2692, + "cost_saving_pct": 96.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 89.6, + "baseline_cost_usd": 2.4499, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.97, + "actual_score": 77.8, + "actual_cost_usd": 0.1565, + "actual_failed": false, + "switched": true, + "score_delta": -11.8, + "cost_delta_usd": -2.2934, + "cost_saving_pct": 93.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.2, + "baseline_cost_usd": 2.96, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.12, + "actual_score": 75.4, + "actual_cost_usd": 0.1026, + "actual_failed": false, + "switched": true, + "score_delta": -11.8, + "cost_delta_usd": -2.8574, + "cost_saving_pct": 96.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 69.0, + "baseline_cost_usd": 1.9909, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.33, + "actual_score": 83.0, + "actual_cost_usd": 0.0686, + "actual_failed": false, + "switched": true, + "score_delta": 14.0, + "cost_delta_usd": -1.9223, + "cost_saving_pct": 96.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.6, + "baseline_cost_usd": 2.0562, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.17, + "actual_score": 84.2, + "actual_cost_usd": 0.1242, + "actual_failed": false, + "switched": true, + "score_delta": -1.4, + "cost_delta_usd": -1.9319, + "cost_saving_pct": 94.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.4, + "baseline_cost_usd": 2.723, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.58, + "actual_score": 81.0, + "actual_cost_usd": 0.1215, + "actual_failed": false, + "switched": true, + "score_delta": -7.4, + "cost_delta_usd": -2.6015, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.8, + "baseline_cost_usd": 2.47, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.5, + "actual_score": 81.6, + "actual_cost_usd": 0.1017, + "actual_failed": false, + "switched": true, + "score_delta": -9.2, + "cost_delta_usd": -2.3682, + "cost_saving_pct": 95.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.4, + "baseline_cost_usd": 2.0202, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.1, + "actual_score": 84.8, + "actual_cost_usd": 0.1345, + "actual_failed": false, + "switched": true, + "score_delta": 3.4, + "cost_delta_usd": -1.8857, + "cost_saving_pct": 93.3, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "auth-clipboard-config-default", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 72.4, + "baseline_cost_usd": 4.1118, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.73, + "actual_score": null, + "actual_cost_usd": 0.2412, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -3.8705, + "cost_saving_pct": 94.1, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.4, + "baseline_cost_usd": 3.8112, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 80.28, + "actual_score": 75.4, + "actual_cost_usd": 0.1447, + "actual_failed": false, + "switched": true, + "score_delta": -10.0, + "cost_delta_usd": -3.6665, + "cost_saving_pct": 96.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.2, + "baseline_cost_usd": 2.538, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 81.9, + "actual_score": 62.4, + "actual_cost_usd": 0.0776, + "actual_failed": false, + "switched": true, + "score_delta": -27.8, + "cost_delta_usd": -2.4604, + "cost_saving_pct": 96.9, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "skill-install-nested-discovery", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 63.4, + "baseline_cost_usd": 5.3843, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 75.4, + "actual_score": null, + "actual_cost_usd": 0.1297, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -5.2546, + "cost_saving_pct": 97.6, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + } + ], + "totals": { + "tasks": 21, + "tasks_switched": 21, + "tasks_scored_both_arms": 19, + "baseline_model": "claude-opus-5-5", + "baseline_total_cost_usd": 45.88, + "routed_total_cost_usd": 1.99, + "cost_saving_usd": 43.89, + "cost_saving_pct": 95.7, + "baseline_mean_score": 87.48, + "routed_mean_score": 80.82, + "mean_score_delta": -6.66, + "baseline_mean_score_failures_as_zero": 85.62, + "routed_mean_score_failures_as_zero": 73.12, + "mean_score_delta_failures_as_zero": -12.5, + "tasks_below_floor_routed": 4, + "tasks_below_floor_baseline": 2, + "tasks_failed_routed": 2, + "tasks_failed_baseline": 0, + "models_used": [ + "claude-haiku-5-5" + ], + "model_counts": { + "claude-haiku-5-5": 21 + }, + "quality_delta_pct": -7.6, + "mean_per_task_saving_pct": 95.4 + } + } + ] +} diff --git a/docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-zero-shot.json b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-zero-shot.json new file mode 100644 index 00000000..96afe6eb --- /dev/null +++ b/docs/metrics/swe-router-rubric-replays/cli-cli-v1-opus-5-5-zero-shot.json @@ -0,0 +1,574 @@ +{ + "config": { + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "scope": "cli-cli-v1", + "harness": "omp", + "skill": "swe3", + "baseline": "claude-opus-5-5", + "tasks": 21, + "available": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-opus-5-5" + ], + "allow_list": "auto", + "no_allow_list": false, + "tie_band": 3.0, + "holdout": false, + "columns_from": "mcp-gateway-registry-v2", + "judged_by": { + "harness": "omp", + "model": "us.anthropic.claude-opus-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/cli-cli-v1.yaml", + "repeats": 3, + "started_at": "2026-10-14T22:10:12.033833Z", + "finished_at": "2026-10-14T22:23:36.365453Z", + "judging_cost_usd": 15.5931, + "runs_ok": 63, + "runs_failed": 0, + "mode": "rubric", + "bucketing": "absolute", + "columns_from": "mcp-gateway-registry-v2" + }, + "floor_note": "Required quality per task, from the rubric pass by omp + us.anthropic.claude-opus-5-5: blast radius and verification, priced by routing-policy.json.", + "tier_note": "Difficulty bucket per task from the rubric pass (absolute bucketing); every model's existing scores regrouped into bucket columns (easy, moderate, hard, very hard shown as trivial, low, medium, high). The dataset's own label is kept beside each row.", + "floors_label": "per-task floors judged by omp + us.anthropic.claude-opus-5-5 running the skill's step 1" + }, + "provenance": { + "measured_on": "2026-10-11", + "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", + "source_sha256": "72b1ed9ee222", + "harness": "omp", + "skill": "swe3", + "dataset": "mcp-gateway-registry-v2", + "judge": { + "model": "openai.gpt-5.6-sol", + "reasoning_effort": "high", + "repo_grounded": true + }, + "repository": "https://github.com/aarora79/agentic-coding-harness-benchmarks" + }, + "runs": [ + { + "floor": null, + "rows": [ + { + "task": "repo-create-reject-empty-name", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.0, + "baseline_cost_usd": 1.295, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 85.6, + "actual_cost_usd": 0.0549, + "actual_failed": false, + "switched": true, + "score_delta": -2.4, + "cost_delta_usd": -1.2402, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "copilot-not-installed-trailing-newline", + "complexity": "trivial", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.6, + "baseline_cost_usd": 0.9935, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 72.69, + "actual_score": 87.6, + "actual_cost_usd": 0.0521, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -0.9414, + "cost_saving_pct": 94.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": true + }, + { + "task": "repo-set-default-unset-without-repo", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.8, + "baseline_cost_usd": 0.9055, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 82.0, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -10.8, + "cost_delta_usd": -0.8311, + "cost_saving_pct": 91.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-checks-all-cancelled-summary", + "complexity": "trivial", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 92.4, + "baseline_cost_usd": 1.025, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 72.69, + "actual_score": 80.0, + "actual_cost_usd": 0.0441, + "actual_failed": false, + "switched": true, + "score_delta": -12.4, + "cost_delta_usd": -0.9809, + "cost_saving_pct": 95.7, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": true + }, + { + "task": "extension-uninstall-alias", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.8, + "baseline_cost_usd": 0.7346, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 61.4, + "actual_cost_usd": 0.0456, + "actual_failed": false, + "switched": true, + "score_delta": -27.4, + "cost_delta_usd": -0.689, + "cost_saving_pct": 93.8, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pr-merge-delete-branch-owner-case", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 91.0, + "baseline_cost_usd": 1.6219, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 87.0, + "actual_cost_usd": 0.0678, + "actual_failed": false, + "switched": true, + "score_delta": -4.0, + "cost_delta_usd": -1.5541, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "pr-view-json-number-on-issue", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.0, + "baseline_cost_usd": 1.893, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 87.4, + "actual_cost_usd": 0.0655, + "actual_failed": false, + "switched": true, + "score_delta": -2.6, + "cost_delta_usd": -1.8275, + "cost_saving_pct": 96.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "project-item-add-output-non-tty", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.2, + "baseline_cost_usd": 1.165, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 87.2, + "actual_cost_usd": 0.045, + "actual_failed": false, + "switched": true, + "score_delta": 1.0, + "cost_delta_usd": -1.12, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "release-create-404-scope-blame", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.8, + "baseline_cost_usd": 1.3867, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 87.0, + "actual_cost_usd": 0.0622, + "actual_failed": false, + "switched": true, + "score_delta": 0.2, + "cost_delta_usd": -1.3246, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "pr-comment-attach-ghu-tokens", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.0, + "baseline_cost_usd": 2.3436, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 83.2, + "actual_score": 84.8, + "actual_cost_usd": 0.0744, + "actual_failed": false, + "switched": true, + "score_delta": -2.2, + "cost_delta_usd": -2.2692, + "cost_saving_pct": 96.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "repo-clone-ghe-ssh-ca-url", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 89.6, + "baseline_cost_usd": 2.4499, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 77.8, + "actual_cost_usd": 0.1565, + "actual_failed": false, + "switched": true, + "score_delta": -11.8, + "cost_delta_usd": -2.2934, + "cost_saving_pct": 93.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "codespace-ports-forward-concurrent-map", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.2, + "baseline_cost_usd": 2.96, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 77.22, + "actual_score": 79.4, + "actual_cost_usd": 1.177, + "actual_failed": false, + "switched": true, + "score_delta": -7.8, + "cost_delta_usd": -1.783, + "cost_saving_pct": 60.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "issue-view-comments-with-json", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 69.0, + "baseline_cost_usd": 1.9909, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 83.0, + "actual_cost_usd": 0.0686, + "actual_failed": false, + "switched": true, + "score_delta": 14.0, + "cost_delta_usd": -1.9223, + "cost_saving_pct": 96.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "release-verify-sha1-prefix", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.6, + "baseline_cost_usd": 2.0562, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 84.2, + "actual_cost_usd": 0.1242, + "actual_failed": false, + "switched": true, + "score_delta": -1.4, + "cost_delta_usd": -1.9319, + "cost_saving_pct": 94.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "auth-refresh-hint-on-expired-token", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.4, + "baseline_cost_usd": 2.723, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 81.0, + "actual_cost_usd": 0.1215, + "actual_failed": false, + "switched": true, + "score_delta": -7.4, + "cost_delta_usd": -2.6015, + "cost_saving_pct": 95.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "secret-set-app-agents", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.8, + "baseline_cost_usd": 2.47, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 81.6, + "actual_cost_usd": 0.1017, + "actual_failed": false, + "switched": true, + "score_delta": -9.2, + "cost_delta_usd": -2.3682, + "cost_saving_pct": 95.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "repo-sync-checked-out-worktree", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.4, + "baseline_cost_usd": 2.0202, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 84.8, + "actual_cost_usd": 0.1345, + "actual_failed": false, + "switched": true, + "score_delta": 3.4, + "cost_delta_usd": -1.8857, + "cost_saving_pct": 93.3, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "auth-clipboard-config-default", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 72.4, + "baseline_cost_usd": 4.1118, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": null, + "actual_cost_usd": 0.2412, + "actual_failed": true, + "switched": true, + "score_delta": null, + "cost_delta_usd": -3.8705, + "cost_saving_pct": 94.1, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "project-item-edit-by-name", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.4, + "baseline_cost_usd": 3.8112, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 75.4, + "actual_cost_usd": 0.1447, + "actual_failed": false, + "switched": true, + "score_delta": -10.0, + "cost_delta_usd": -3.6665, + "cost_saving_pct": 96.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "skill-update-dir-keeps-install-path", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.2, + "baseline_cost_usd": 2.538, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.52, + "actual_score": 62.4, + "actual_cost_usd": 0.0776, + "actual_failed": false, + "switched": true, + "score_delta": -27.8, + "cost_delta_usd": -2.4604, + "cost_saving_pct": 96.9, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": false + }, + { + "task": "skill-install-nested-discovery", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 63.4, + "baseline_cost_usd": 5.3843, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 77.22, + "actual_score": 64.8, + "actual_cost_usd": 2.4782, + "actual_failed": false, + "switched": true, + "score_delta": 1.4, + "cost_delta_usd": -2.9061, + "cost_saving_pct": 54.0, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + } + ], + "totals": { + "tasks": 21, + "tasks_switched": 21, + "tasks_scored_both_arms": 20, + "baseline_model": "claude-opus-5-5", + "baseline_total_cost_usd": 45.88, + "routed_total_cost_usd": 5.41, + "cost_saving_usd": 40.47, + "cost_saving_pct": 88.2, + "baseline_mean_score": 86.28, + "routed_mean_score": 80.22, + "mean_score_delta": -6.06, + "baseline_mean_score_failures_as_zero": 85.62, + "routed_mean_score_failures_as_zero": 76.4, + "mean_score_delta_failures_as_zero": -9.22, + "tasks_below_floor_routed": 4, + "tasks_below_floor_baseline": 2, + "tasks_failed_routed": 1, + "tasks_failed_baseline": 0, + "models_used": [ + "claude-haiku-5-5", + "claude-sonnet-5-5" + ], + "model_counts": { + "claude-haiku-5-5": 19, + "claude-sonnet-5-5": 2 + }, + "quality_delta_pct": -7.0, + "mean_per_task_saving_pct": 91.6 + } + } + ] +} diff --git a/docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-haiku-5-5-in-repo.json b/docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-haiku-5-5-in-repo.json new file mode 100644 index 00000000..3577cb9e --- /dev/null +++ b/docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-haiku-5-5-in-repo.json @@ -0,0 +1,575 @@ +{ + "config": { + "dataset": "benchmarks/dataset/mcp-gateway-registry-v2.yaml", + "scope": "mcp-gateway-registry-v2", + "harness": "omp", + "skill": "swe3", + "baseline": "claude-opus-5-5", + "tasks": 21, + "available": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-opus-5-5" + ], + "allow_list": "auto", + "no_allow_list": false, + "tie_band": 3.0, + "holdout": true, + "columns_from": null, + "judged_by": { + "harness": "claude", + "model": "us.anthropic.claude-haiku-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/mcp-gateway-registry-v2.yaml", + "repeats": 3, + "started_at": "2026-10-13T22:51:35.031332Z", + "finished_at": "2026-10-13T23:02:26.086741Z", + "judging_cost_usd": 0.8377, + "runs_ok": 62, + "runs_failed": 1, + "mode": "rubric", + "bucketing": "absolute" + }, + "floor_note": "Required quality per task, from the rubric pass by claude + us.anthropic.claude-haiku-5-5: blast radius and verification, priced by routing-policy.json.", + "tier_note": "Difficulty bucket per task from the rubric pass (absolute bucketing); every model's existing scores regrouped into bucket columns (easy, moderate, hard, very hard shown as trivial, low, medium, high). The dataset's own label is kept beside each row.", + "floors_label": "per-task floors judged by claude + us.anthropic.claude-haiku-5-5 running the skill's step 1" + }, + "provenance": { + "measured_on": "2026-10-11", + "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", + "source_sha256": "72b1ed9ee222", + "harness": "omp", + "skill": "swe3", + "dataset": "mcp-gateway-registry-v2", + "judge": { + "model": "openai.gpt-5.6-sol", + "reasoning_effort": "high", + "repo_grounded": true + }, + "repository": "https://github.com/aarora79/agentic-coding-harness-benchmarks" + }, + "runs": [ + { + "floor": null, + "rows": [ + { + "task": "default-create-in-idp-checkbox-unchecked", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 75.0, + "baseline_cost_usd": 4.0633, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.6, + "actual_score": 83.0, + "actual_cost_usd": 0.8427, + "actual_failed": false, + "switched": true, + "score_delta": 8.0, + "cost_delta_usd": -3.2207, + "cost_saving_pct": 79.3, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "build-docker-images-from-uv-lock", + "complexity": "high", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.4, + "baseline_cost_usd": 2.8373, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-opus-5-5", + "predicted_score": 81.45, + "actual_score": 81.4, + "actual_cost_usd": 2.8373, + "actual_failed": false, + "switched": false, + "score_delta": 0.0, + "cost_delta_usd": 0.0, + "cost_saving_pct": -0.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "honor-cloud-provider-override-in-ui", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 89.4, + "baseline_cost_usd": 1.7201, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 77.53, + "actual_score": 87.6, + "actual_cost_usd": 0.093, + "actual_failed": false, + "switched": true, + "score_delta": -1.8, + "cost_delta_usd": -1.6272, + "cost_saving_pct": 94.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "fix-reserved-groups-var-in-service-account-script", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.8, + "baseline_cost_usd": 1.8829, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.4, + "actual_score": 82.4, + "actual_cost_usd": 0.0739, + "actual_failed": false, + "switched": true, + "score_delta": -4.4, + "cost_delta_usd": -1.8091, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "cli-custom-egress-oauth-provider-flags", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.0, + "baseline_cost_usd": 2.3573, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.32, + "actual_score": 86.4, + "actual_cost_usd": 1.2298, + "actual_failed": false, + "switched": true, + "score_delta": -1.6, + "cost_delta_usd": -1.1275, + "cost_saving_pct": 47.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "derive-repo-url-from-skill-md", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 82.8, + "baseline_cost_usd": 3.7516, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 80.6, + "actual_score": 69.2, + "actual_cost_usd": 0.1441, + "actual_failed": false, + "switched": true, + "score_delta": -13.6, + "cost_delta_usd": -3.6075, + "cost_saving_pct": 96.2, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "consistent-csrf-across-toggle-endpoints", + "complexity": "high", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.2, + "baseline_cost_usd": 3.767, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-opus-5-5", + "predicted_score": 80.97, + "actual_score": 87.2, + "actual_cost_usd": 3.767, + "actual_failed": false, + "switched": false, + "score_delta": 0.0, + "cost_delta_usd": 0.0, + "cost_saving_pct": -0.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "configurable-ui-title", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 69.8, + "baseline_cost_usd": 3.871, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 77.93, + "actual_score": 85.2, + "actual_cost_usd": 0.1455, + "actual_failed": false, + "switched": true, + "score_delta": 15.4, + "cost_delta_usd": -3.7254, + "cost_saving_pct": 96.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "configurable-mcp-proxy-upstream-timeout", + "complexity": "high", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 83.6, + "baseline_cost_usd": 3.1483, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.77, + "actual_score": 81.0, + "actual_cost_usd": 1.8636, + "actual_failed": false, + "switched": true, + "score_delta": -2.6, + "cost_delta_usd": -1.2847, + "cost_saving_pct": 40.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "nginx-location-trailing-slash-route-hijack", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 75.0, + "baseline_cost_usd": 5.1324, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.9, + "actual_score": 79.4, + "actual_cost_usd": 2.2189, + "actual_failed": false, + "switched": true, + "score_delta": 4.4, + "cost_delta_usd": -2.9134, + "cost_saving_pct": 56.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "registration-admission-control-gate", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.2, + "baseline_cost_usd": 7.5016, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 80.55, + "actual_score": 59.6, + "actual_cost_usd": 6.3653, + "actual_failed": false, + "switched": true, + "score_delta": -25.6, + "cost_delta_usd": -1.1363, + "cost_saving_pct": 15.1, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "server-side-oauth-token-storage", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 73.4, + "baseline_cost_usd": 5.6826, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 80.08, + "actual_score": 65.2, + "actual_cost_usd": 4.0022, + "actual_failed": false, + "switched": true, + "score_delta": -8.2, + "cost_delta_usd": -1.6804, + "cost_saving_pct": 29.6, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "lifecycle-workflow-webhooks", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 79.2, + "baseline_cost_usd": 14.7899, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.6, + "actual_score": 83.0, + "actual_cost_usd": 9.0292, + "actual_failed": false, + "switched": true, + "score_delta": 3.8, + "cost_delta_usd": -5.7607, + "cost_saving_pct": 39.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "per-caller-per-target-rate-limits-and-quarantine", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 80.4, + "baseline_cost_usd": 10.0246, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.97, + "actual_score": 78.6, + "actual_cost_usd": 10.111, + "actual_failed": false, + "switched": true, + "score_delta": -1.8, + "cost_delta_usd": 0.0863, + "cost_saving_pct": -0.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "idp-authenticated-embedding-endpoint", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 83.0, + "baseline_cost_usd": 8.6089, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.87, + "actual_score": 79.8, + "actual_cost_usd": 5.5963, + "actual_failed": false, + "switched": true, + "score_delta": -3.2, + "cost_delta_usd": -3.0126, + "cost_saving_pct": 35.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "index-demo-videos-in-one-page", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.8, + "baseline_cost_usd": 2.5198, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.8, + "actual_score": 80.0, + "actual_cost_usd": 0.0904, + "actual_failed": false, + "switched": true, + "score_delta": -8.8, + "cost_delta_usd": -2.4294, + "cost_saving_pct": 96.4, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "hide-register-button-on-virtual-and-skills-tabs", + "complexity": "low", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.0, + "baseline_cost_usd": 1.2622, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 79.26, + "actual_score": 84.0, + "actual_cost_usd": 0.306, + "actual_failed": false, + "switched": true, + "score_delta": -6.0, + "cost_delta_usd": -0.9562, + "cost_saving_pct": 75.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pass-ssrf-allowlist-env-to-registry-container", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.8, + "baseline_cost_usd": 1.9444, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.78, + "actual_score": 80.8, + "actual_cost_usd": 0.4015, + "actual_failed": false, + "switched": true, + "score_delta": -10.0, + "cost_delta_usd": -1.5429, + "cost_saving_pct": 79.4, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "macos-setup-python-version-precheck", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 56.4, + "baseline_cost_usd": 1.5568, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 79.47, + "actual_score": 76.0, + "actual_cost_usd": 0.0656, + "actual_failed": false, + "switched": true, + "score_delta": 19.6, + "cost_delta_usd": -1.4912, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "portable-env-secret-generation-in-build-script", + "complexity": "medium", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.6, + "baseline_cost_usd": 2.5544, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 80.07, + "actual_score": 72.4, + "actual_cost_usd": 0.0933, + "actual_failed": false, + "switched": true, + "score_delta": -9.2, + "cost_delta_usd": -2.4611, + "cost_saving_pct": 96.3, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "logout-id-token-hint-out-of-browser-url", + "complexity": "high", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 76.6, + "baseline_cost_usd": 4.2955, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-opus-5-5", + "predicted_score": 81.85, + "actual_score": 76.6, + "actual_cost_usd": 4.2955, + "actual_failed": false, + "switched": false, + "score_delta": 0.0, + "cost_delta_usd": 0.0, + "cost_saving_pct": -0.0, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + } + ], + "totals": { + "tasks": 21, + "tasks_switched": 18, + "tasks_scored_both_arms": 21, + "baseline_model": "claude-opus-5-5", + "baseline_total_cost_usd": 93.27, + "routed_total_cost_usd": 53.57, + "cost_saving_usd": 39.7, + "cost_saving_pct": 42.6, + "baseline_mean_score": 81.16, + "routed_mean_score": 78.99, + "mean_score_delta": -2.17, + "baseline_mean_score_failures_as_zero": 81.16, + "routed_mean_score_failures_as_zero": 78.99, + "mean_score_delta_failures_as_zero": -2.17, + "tasks_below_floor_routed": 5, + "tasks_below_floor_baseline": 4, + "tasks_failed_routed": 0, + "tasks_failed_baseline": 0, + "models_used": [ + "claude-haiku-5-5", + "claude-opus-5-5", + "claude-sonnet-5-5" + ], + "model_counts": { + "claude-sonnet-5-5": 11, + "claude-haiku-5-5": 7, + "claude-opus-5-5": 3 + }, + "quality_delta_pct": -2.7, + "mean_per_task_saving_pct": 55.7 + } + } + ] +} diff --git a/docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-opus-5-5-in-repo.json b/docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-opus-5-5-in-repo.json new file mode 100644 index 00000000..8a75a28e --- /dev/null +++ b/docs/metrics/swe-router-rubric-replays/v2-claude-5-5-only-opus-5-5-in-repo.json @@ -0,0 +1,576 @@ +{ + "config": { + "dataset": "benchmarks/dataset/mcp-gateway-registry-v2.yaml", + "scope": "mcp-gateway-registry-v2", + "harness": "omp", + "skill": "swe3", + "baseline": "claude-opus-5-5", + "tasks": 21, + "available": [ + "claude-haiku-5-5", + "claude-sonnet-5-5", + "claude-opus-5-5" + ], + "allow_list": "auto", + "no_allow_list": false, + "tie_band": 3.0, + "holdout": true, + "columns_from": null, + "judged_by": { + "harness": "claude", + "model": "us.anthropic.claude-opus-5-5", + "provider": "bedrock", + "skill": "swe-router", + "step": "rubric pass: six dimensions scored 0-2; code derives required quality and difficulty bucket", + "dataset": "benchmarks/dataset/mcp-gateway-registry-v2.yaml", + "repeats": 3, + "started_at": "2026-10-13T22:51:35.022467Z", + "finished_at": "2026-10-13T23:03:07.306856Z", + "judging_cost_usd": 20.2262, + "runs_ok": 63, + "runs_failed": 0, + "mode": "rubric", + "bucketing": "absolute" + }, + "floor_note": "Required quality per task, from the rubric pass by claude + us.anthropic.claude-opus-5-5: blast radius and verification, priced by routing-policy.json.", + "tier_note": "Difficulty bucket per task from the rubric pass (absolute bucketing); every model's existing scores regrouped into bucket columns (easy, moderate, hard, very hard shown as trivial, low, medium, high). The dataset's own label is kept beside each row.", + "floors_label": "per-task floors judged by claude + us.anthropic.claude-opus-5-5 running the skill's step 1" + }, + "provenance": { + "measured_on": "2026-10-11", + "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", + "source_sha256": "72b1ed9ee222", + "harness": "omp", + "skill": "swe3", + "dataset": "mcp-gateway-registry-v2", + "judge": { + "model": "openai.gpt-5.6-sol", + "reasoning_effort": "high", + "repo_grounded": true + }, + "repository": "https://github.com/aarora79/agentic-coding-harness-benchmarks" + }, + "runs": [ + { + "floor": null, + "rows": [ + { + "task": "default-create-in-idp-checkbox-unchecked", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 75.0, + "baseline_cost_usd": 4.0633, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.49, + "actual_score": 76.8, + "actual_cost_usd": 0.1638, + "actual_failed": false, + "switched": true, + "score_delta": 1.8, + "cost_delta_usd": -3.8996, + "cost_saving_pct": 96.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "build-docker-images-from-uv-lock", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.4, + "baseline_cost_usd": 2.8373, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 76.62, + "actual_score": 82.0, + "actual_cost_usd": 1.8583, + "actual_failed": false, + "switched": true, + "score_delta": 0.6, + "cost_delta_usd": -0.979, + "cost_saving_pct": 34.5, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "honor-cloud-provider-override-in-ui", + "complexity": "low", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 89.4, + "baseline_cost_usd": 1.7201, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 78.8, + "actual_score": 87.6, + "actual_cost_usd": 0.093, + "actual_failed": false, + "switched": true, + "score_delta": -1.8, + "cost_delta_usd": -1.6272, + "cost_saving_pct": 94.6, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": true + }, + { + "task": "fix-reserved-groups-var-in-service-account-script", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 86.8, + "baseline_cost_usd": 1.8829, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 75.87, + "actual_score": 82.4, + "actual_cost_usd": 0.0739, + "actual_failed": false, + "switched": true, + "score_delta": -4.4, + "cost_delta_usd": -1.8091, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "cli-custom-egress-oauth-provider-flags", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.0, + "baseline_cost_usd": 2.3573, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.51, + "actual_score": 76.6, + "actual_cost_usd": 0.0674, + "actual_failed": false, + "switched": true, + "score_delta": -11.4, + "cost_delta_usd": -2.2899, + "cost_saving_pct": 97.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + }, + { + "task": "derive-repo-url-from-skill-md", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 82.8, + "baseline_cost_usd": 3.7516, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 77.33, + "actual_score": 69.2, + "actual_cost_usd": 0.1441, + "actual_failed": false, + "switched": true, + "score_delta": -13.6, + "cost_delta_usd": -3.6075, + "cost_saving_pct": 96.2, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "consistent-csrf-across-toggle-endpoints", + "complexity": "high", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 87.2, + "baseline_cost_usd": 3.767, + "baseline_failed": false, + "router_status": "nothing_clears_floor", + "recommended_model": null, + "recommended_reason": "nothing available reaches 80.0 on high tasks; closest is claude-opus-5-5 at 79.28, short by 0.72", + "predicted_score": null, + "actual_score": 87.2, + "actual_cost_usd": 3.767, + "actual_failed": false, + "switched": false, + "score_delta": 0.0, + "cost_delta_usd": 0.0, + "cost_saving_pct": 0.0, + "met_floor": true, + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "configurable-ui-title", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 69.8, + "baseline_cost_usd": 3.871, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 75.56, + "actual_score": 85.2, + "actual_cost_usd": 0.1455, + "actual_failed": false, + "switched": true, + "score_delta": 15.4, + "cost_delta_usd": -3.7254, + "cost_saving_pct": 96.2, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "configurable-mcp-proxy-upstream-timeout", + "complexity": "medium", + "floor": 70.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 83.6, + "baseline_cost_usd": 3.1483, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.22, + "actual_score": 79.2, + "actual_cost_usd": 0.1892, + "actual_failed": false, + "switched": true, + "score_delta": -4.4, + "cost_delta_usd": -2.9592, + "cost_saving_pct": 94.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": true + }, + { + "task": "nginx-location-trailing-slash-route-hijack", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 75.0, + "baseline_cost_usd": 5.1324, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 76.95, + "actual_score": 79.4, + "actual_cost_usd": 2.2189, + "actual_failed": false, + "switched": true, + "score_delta": 4.4, + "cost_delta_usd": -2.9134, + "cost_saving_pct": 56.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "medium", + "tier_matches_dataset": false + }, + { + "task": "registration-admission-control-gate", + "complexity": "high", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 85.2, + "baseline_cost_usd": 7.5016, + "baseline_failed": false, + "router_status": "nothing_clears_floor", + "recommended_model": null, + "recommended_reason": "nothing available reaches 80.0 on high tasks; closest is claude-opus-5-5 at 79.53, short by 0.47", + "predicted_score": null, + "actual_score": 85.2, + "actual_cost_usd": 7.5016, + "actual_failed": false, + "switched": false, + "score_delta": 0.0, + "cost_delta_usd": 0.0, + "cost_saving_pct": 0.0, + "met_floor": true, + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "server-side-oauth-token-storage", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 73.4, + "baseline_cost_usd": 5.6826, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 78.72, + "actual_score": 65.2, + "actual_cost_usd": 4.0022, + "actual_failed": false, + "switched": true, + "score_delta": -8.2, + "cost_delta_usd": -1.6804, + "cost_saving_pct": 29.6, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "lifecycle-workflow-webhooks", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 79.2, + "baseline_cost_usd": 14.7899, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 76.5, + "actual_score": 83.0, + "actual_cost_usd": 9.0292, + "actual_failed": false, + "switched": true, + "score_delta": 3.8, + "cost_delta_usd": -5.7607, + "cost_saving_pct": 39.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "per-caller-per-target-rate-limits-and-quarantine", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 80.4, + "baseline_cost_usd": 10.0246, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 77.05, + "actual_score": 78.6, + "actual_cost_usd": 10.111, + "actual_failed": false, + "switched": true, + "score_delta": -1.8, + "cost_delta_usd": 0.0863, + "cost_saving_pct": -0.9, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "idp-authenticated-embedding-endpoint", + "complexity": "high", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 83.0, + "baseline_cost_usd": 8.6089, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 76.9, + "actual_score": 79.8, + "actual_cost_usd": 5.5963, + "actual_failed": false, + "switched": true, + "score_delta": -3.2, + "cost_delta_usd": -3.0126, + "cost_saving_pct": 35.0, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "high", + "tier_matches_dataset": true + }, + { + "task": "index-demo-videos-in-one-page", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 88.8, + "baseline_cost_usd": 2.5198, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.13, + "actual_score": 80.0, + "actual_cost_usd": 0.0904, + "actual_failed": false, + "switched": true, + "score_delta": -8.8, + "cost_delta_usd": -2.4294, + "cost_saving_pct": 96.4, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "hide-register-button-on-virtual-and-skills-tabs", + "complexity": "low", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.0, + "baseline_cost_usd": 1.2622, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 87.6, + "actual_score": 78.8, + "actual_cost_usd": 0.0491, + "actual_failed": false, + "switched": true, + "score_delta": -11.2, + "cost_delta_usd": -1.2132, + "cost_saving_pct": 96.1, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "pass-ssrf-allowlist-env-to-registry-container", + "complexity": "medium", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 90.8, + "baseline_cost_usd": 1.9444, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 80.04, + "actual_score": 80.8, + "actual_cost_usd": 0.4015, + "actual_failed": false, + "switched": true, + "score_delta": -10.0, + "cost_delta_usd": -1.5429, + "cost_saving_pct": 79.4, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "macos-setup-python-version-precheck", + "complexity": "medium", + "floor": 75.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 56.4, + "baseline_cost_usd": 1.5568, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-haiku-5-5", + "predicted_score": 76.58, + "actual_score": 76.0, + "actual_cost_usd": 0.0656, + "actual_failed": false, + "switched": true, + "score_delta": 19.6, + "cost_delta_usd": -1.4912, + "cost_saving_pct": 95.8, + "met_floor": true, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "portable-env-secret-generation-in-build-script", + "complexity": "medium", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 81.6, + "baseline_cost_usd": 2.5544, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-sonnet-5-5", + "predicted_score": 80.24, + "actual_score": 79.0, + "actual_cost_usd": 0.812, + "actual_failed": false, + "switched": true, + "score_delta": -2.6, + "cost_delta_usd": -1.7424, + "cost_saving_pct": 68.2, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "trivial", + "tier_matches_dataset": false + }, + { + "task": "logout-id-token-hint-out-of-browser-url", + "complexity": "high", + "floor": 80.0, + "baseline_model": "claude-opus-5-5", + "baseline_score": 76.6, + "baseline_cost_usd": 4.2955, + "baseline_failed": false, + "router_status": "ok", + "recommended_model": "claude-opus-5-5", + "predicted_score": 80.6, + "actual_score": 76.6, + "actual_cost_usd": 4.2955, + "actual_failed": false, + "switched": false, + "score_delta": 0.0, + "cost_delta_usd": 0.0, + "cost_saving_pct": -0.0, + "met_floor": false, + "cost_basis": "metered", + "dataset_complexity": "low", + "tier_matches_dataset": false + } + ], + "totals": { + "tasks": 21, + "tasks_switched": 18, + "tasks_scored_both_arms": 21, + "baseline_model": "claude-opus-5-5", + "baseline_total_cost_usd": 93.27, + "routed_total_cost_usd": 50.68, + "cost_saving_usd": 42.6, + "cost_saving_pct": 45.7, + "baseline_mean_score": 81.16, + "routed_mean_score": 79.46, + "mean_score_delta": -1.7, + "baseline_mean_score_failures_as_zero": 81.16, + "routed_mean_score_failures_as_zero": 79.46, + "mean_score_delta_failures_as_zero": -1.7, + "tasks_below_floor_routed": 4, + "tasks_below_floor_baseline": 4, + "tasks_failed_routed": 0, + "tasks_failed_baseline": 0, + "models_used": [ + "claude-haiku-5-5", + "claude-opus-5-5", + "claude-sonnet-5-5" + ], + "model_counts": { + "claude-haiku-5-5": 10, + "claude-sonnet-5-5": 8, + "(no pick -- stayed on claude-opus-5-5)": 2, + "claude-opus-5-5": 1 + }, + "quality_delta_pct": -2.1, + "mean_per_task_saving_pct": 61.9 + } + } + ] +} diff --git a/docs/results-cli-cli-v1.md b/docs/results-cli-cli-v1.md new file mode 100644 index 00000000..0ef7227a --- /dev/null +++ b/docs/results-cli-cli-v1.md @@ -0,0 +1,100 @@ +# cli/cli: does what we learned on one repository carry to another? + +Issue #221 asked whether the model rankings, the score levels and the swe-router rubric from `mcp-gateway-registry-v2` (v2) hold on a codebase nobody tuned them on. This page answers it for set A: Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 on the 21 tasks of [cli-cli-v1.yaml](../benchmarks/dataset/cli-cli-v1.yaml), the GitHub CLI in Go. + +> [!NOTE] +> **Terms.** +> **Score**: the judge's 0-100 grade of a task's six artifacts (`openai.gpt-5.6-sol`, high reasoning effort). +> **Failure**: a task where the model left the artifacts outside the folder the prompt named. The judge never sees it. "Failure = 0" counts it as a score of 0. +> **Rubric**: six task dimensions scored 0-2, which set a **required quality** (the score a result must reach) and a **difficulty bucket** (easy, moderate, hard, very hard). See [swe-router-rubric-design.md](swe-router-rubric-design.md). +> **Column**: a model's mean score on the benchmark tasks in one bucket. The router picks the cheapest model whose column clears the required quality. +> **Zero-shot**: routing cli/cli tasks with columns built from v2 alone. + +## Short answers + +| Question | Answer | +|---|---| +| Do rankings carry over? | Yes overall: Opus 5.5 > Sonnet 5.5 > Haiku 5.5 on both repositories. Within a tier the gaps shrink to a few points, and on the low tier Haiku edges out Sonnet. | +| Do score levels shift? | Yes. All three models score 3 to 8 points higher on cli/cli and cost less per task. | +| Is Haiku still the deciding question? | Yes. Where it finishes, Haiku scores within 2 points of Sonnet. It failed 2 of 21 tasks by writing to the wrong folder, and the router cannot see that risk. | +| Does the rubric work zero-shot? | The judging transfers, and v2's columns route cli/cli tasks a little more cautiously than cli/cli's own. Routing quality on cli/cli is poor for a different reason: the router sends nearly every task to Haiku and ignores its failure rate. | + +## The runs + +All three ran omp with the swe3 skill on Amazon Bedrock on 2026-10-14. Costs are token counts priced at Bedrock rates. + +| Model | v2 score | cli/cli score | v2 failure = 0 | cli/cli failure = 0 | v2 cost (21 tasks) | cli/cli cost (21 tasks) | +|---|---:|---:|---:|---:|---:|---:| +| Haiku 5.5 | 72.69 (20/21) | 80.82 (19/21) | 69.23 | 73.12 | $3.05 | $1.99 | +| Sonnet 5.5 | 79.49 | 82.50 | 79.49 | 82.50 | $55.02 | $19.11 | +| Opus 5.5 | 81.16 | 85.62 | 81.16 | 85.62 | $93.27 | $45.88 | + +Source: each model's `run-summary.json` under `benchmarks/swe-benchmark-data//omp/swe3/{mcp-gateway-registry-v2,cli-cli-v1}/`. + +By the dataset's own tiers, with failures as 0: + +| Tier | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 | +|---|---:|---:|---:| +| trivial (5) | 79.3 | 85.3 | 90.7 | +| low (5) | 86.7 | 85.3 | 88.2 | +| medium (6) | 80.5 | 84.7 | 85.1 | +| high (5) | 44.5 | 74.4 | 78.6 | + +Haiku's high-tier mean is 74.2 without its two failures. Two tasks are hard for every model: `skill-install-nested-discovery` (64.8 for Sonnet, 63.4 for Opus) and `auth-clipboard-config-default` (71.6 and 72.4). Opus scored 69.0 on `issue-view-comments-with-json`, below Haiku's 83.0; with one run per task, a single low score like that is within the noise. + +### Haiku's failures + +Haiku 5.5 wrote its six artifacts somewhere other than the path the prompt gave on 3 of 21 first attempts. On `auth-clipboard-config-default` it used the skill's generic layout (`claude-haiku-5-5/claude-code/cli/`) on both attempts. On `skill-install-nested-discovery` it wrote to `cli-v1/`, and on the retry it lost the path entirely and asked for one. `repo-clone-ghe-ssh-ca-url` went to `cli-ci-v1/` first and recovered on retry. It did the same once on v2, dropping the `-v2`. The repeated `cli-cli` in the path seems to make it worse. The misplaced artifacts are kept, unjudged, under [do-not-include/claude-haiku-5-5-misplaced-artifacts/](../benchmarks/swe-benchmark-data/do-not-include/README.md). + +## The rubric on cli/cli + +Haiku 5.5 and Opus 5.5 each scored the 21 tasks three times, reading the task and the cloned repository at its ref. They agree on the required quality for 18 tasks and on the bucket for 13. Opus gave the same bucket in all three repeats on 20 tasks; Haiku did so on 13. Both put most tasks in "hard", as on v2. Opus spreads them more: 2 easy, 7 moderate, 10 hard, 2 very hard. Source: [swe-router-rubric-cli-cli-v1-claude-opus-5-5.json](metrics/swe-router-rubric-cli-cli-v1-claude-opus-5-5.json) and the Haiku file beside it, with readable versions in [docs/](swe-router-rubric-cli-cli-v1-claude-opus-5-5.md). + +The bucket tracks measured difficulty. Under Opus's buckets, each model's cli/cli column falls from easy to very hard, as it does on v2: + +| Model | v2 moderate | v2 hard | v2 very hard | cli/cli easy | cli/cli moderate | cli/cli hard | cli/cli very hard | +|---|---:|---:|---:|---:|---:|---:|---:| +| Haiku 5.5 | 83.2 | 76.5 | 65.3 (8/9) | 83.8 | 82.1 | 79.7 (9/10) | 75.4 (1/2) | +| Sonnet 5.5 | 86.5 | 80.1 | 77.2 | 87.3 | 84.7 | 82.1 | 72.1 | +| Opus 5.5 | 89.7 | 80.4 | 80.2 | 92.0 | 88.7 | 84.3 | 75.3 | + +v2 has no task in the easy bucket. A fraction marks a column where the model failed some of its tasks; the column mean leaves those out. + +## Routing replays + +Each replay routes all 21 tasks among the three Claude 5.5 models, looks up what the chosen model scored and cost on that task, and compares with Opus 5.5 on every task. "In-repo" builds the columns from the same dataset, leaving the routed task out. The v2 rows use the same three models, so they compare directly. + +| Dataset | Columns from | Rubric judge | Cost saved | Quality change | Failure = 0 | Under required quality | Picks (H / S / O) | +|---|---|---|---:|---:|---:|---:|---| +| v2 | v2, in-repo | Opus 5.5 | 45.7% | −1.70 | −1.70 | 4 | 10 / 8 / 3 | +| v2 | v2, in-repo | Haiku 5.5 | 42.6% | −2.17 | −2.17 | 5 | 7 / 11 / 3 | +| cli/cli | **v2, zero-shot** | Opus 5.5 | 88.2% | −6.06 | −9.22 | 4 | 19 / 2 / 0 | +| cli/cli | **v2, zero-shot** | Haiku 5.5 | 88.7% | −6.07 | −11.96 | 3 | 17 / 4 / 0 | +| cli/cli | cli/cli, in-repo | Opus 5.5 | 95.7% | −6.66 | −12.50 | 4 | 21 / 0 / 0 | +| cli/cli | cli/cli, in-repo | Haiku 5.5 | 93.7% | −7.17 | −12.95 | 4 | 20 / 1 / 0 | + +For reference, a single model on every cli/cli task: Haiku saves 95.7% for −12.50 (failure = 0); Sonnet saves 58.3% for −3.12. On v2: Haiku saves 96.7% for −11.93; Sonnet saves 41.0% for −1.67. + +"Quality change" compares only tasks where the routed model produced a scored run. The two v2 rows count a task with no confident pick as staying on Opus. Two of the v2 Opus-judge picks fell into that case. Source: [swe-router-rubric-replays/](metrics/swe-router-rubric-replays/), files `cli-cli-v1-{opus-5-5,haiku-5-5}-{zero-shot,in-repo}.json` and `v2-claude-5-5-only-{opus-5-5,haiku-5-5}-in-repo.json`. + +### What the replays show + +**v2's columns transfer, and they err on the safe side.** v2 was harder for every model, so its columns understate cli/cli scores. Where a cli/cli column would send a very hard task to Haiku, v2's column (Haiku at 65.3) sends it to Sonnet. Counting failures as 0, zero-shot routing kept 1 to 3 more points of quality than in-repo routing and saved 5 to 7.5 points less. A customer who starts from our published columns will overpay a little on an easier repository rather than ship weaker work. + +**On cli/cli the router becomes "always Haiku", and Haiku's failures cost the quality.** Haiku's columns clear the 70 bar in almost every bucket, so the router picks it for 17 to 21 tasks. Four tasks then miss the bar: Haiku's two wrong-folder failures and its two low scores (61.4 on `extension-uninstall-alias`, 62.4 on `skill-update-dir-keeps-install-path`). The two failures cost 3 to 6 points of mean quality on their own: that is the gap between the "quality change" and "failure = 0" columns. + +**The router cannot see the failures.** A column mean leaves failed tasks out, and `route.py` reports `completion_by_complexity` but never uses it to choose. So a model that finishes 9 of 10 hard tasks looks as good as one that finishes 10 of 10. On v2 this did not matter much: Haiku failed once in 21 and the router sent the hard tasks to Sonnet anyway. On cli/cli it is the main source of lost quality. + +## What changes for phase 1 + +1. **Make the router see failures.** Two options. Scoring a failed task as 0 in its column drops Haiku's cli/cli very hard column from 75.4 to 37.7, but its hard column only from 79.7 to 71.7, which still clears 70. Skipping a model whose completion in a bucket falls below a threshold set in `routing-policy.json` keeps Haiku off both buckets. These figures are arithmetic on the columns above; neither option has been replayed yet. +2. **Ship v2's columns as the default, and tell customers to rebuild them on their own tasks.** Zero-shot routing worked and erred towards caution, so the defaults are safe to ship. Columns built on the customer's repository save more. +3. **Keep Opus-class models as the recommended rubric judge.** Haiku 5.5's judgments changed bucket between repeats on 8 of 21 tasks; Opus 5.5's on 1. + +The rubric, the policy file and the bucket mapping need no change. + +## Limits + +- One run per task per model, 21 tasks. Differences under about 3 points are noise. +- Three models. Set B (the open-weight models, about $446) would test whether a cheaper non-Claude model takes Haiku's place on the frontier here. +- The cli/cli tiers in the dataset are provisional, set from each fix's shape before any run. diff --git a/docs/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.md b/docs/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.md new file mode 100644 index 00000000..cc3ec428 --- /dev/null +++ b/docs/swe-router-rubric-cli-cli-v1-claude-haiku-5-5.md @@ -0,0 +1,119 @@ +# How omp + us.anthropic.claude-haiku-5-5 scores each task on the rubric + +The rubric pass over `benchmarks/dataset/cli-cli-v1.yaml`: each task scored on six dimensions, 0 to 2, 3 independent time(s), in a clone of its repository at the task's pinned ref. No model output was read; this describes the tasks only. Code turns the scores into a required quality and a difficulty bucket (`routing-policy.json`). + +- **Stability.** Required quality agreed across repeats on 16/21 tasks; the bucket on 13/21. +- **Runs.** 63 succeeded, 0 failed. Cost $0.6335. +- **Judged.** 2026-10-14T22:10:12.037956Z to 2026-10-14T22:21:48.671252Z. + +Columns: HC hidden contract, SI silent invariant, CS coordinated surfaces, AM ambiguity, VE verification, BR blast radius. Each score is the lower median of the repeats. + +| Task | HC | SI | CS | AM | VE | BR | Quality | Bucket | Confidence | Repeats agree | +|---|---:|---:|---:|---:|---:|---:|---:|---|---|---| +| repo-create-reject-empty-name | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 70.0]; buckets {'hard': 1, 'moderate': 2} | +| copilot-not-installed-trailing-newline | 0 | 0 | 0 | 1 | 2 | 1 | **75** | easy | high | quality [70.0, 75.0, 75.0]; buckets {'moderate': 1, 'easy': 2} | +| repo-set-default-unset-without-repo | 1 | 1 | 1 | 2 | 1 | 1 | **70** | hard | medium | yes | +| pr-checks-all-cancelled-summary | 1 | 0 | 0 | 1 | 0 | 1 | **70** | moderate | high | yes | +| extension-uninstall-alias | 1 | 0 | 0 | 1 | 1 | 1 | **70** | moderate | high | yes | +| pr-merge-delete-branch-owner-case | 1 | 1 | 0 | 1 | 1 | 1 | **70** | moderate | high | quality [70.0, 70.0, 70.0]; buckets {'moderate': 2, 'hard': 1} | +| pr-view-json-number-on-issue | 1 | 1 | 1 | 2 | 1 | 1 | **70** | hard | medium | yes | +| project-item-add-output-non-tty | 1 | 0 | 0 | 1 | 1 | 1 | **70** | moderate | medium | yes | +| release-create-404-scope-blame | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | high | yes | +| pr-comment-attach-ghu-tokens | 1 | 2 | 1 | 1 | 0 | 1 | **70** | hard | medium | quality [70.0, 70.0, 70.0]; buckets {'very_hard': 1, 'hard': 2} | +| repo-clone-ghe-ssh-ca-url | 1 | 1 | 1 | 2 | 1 | 1 | **70** | hard | medium | yes | +| codespace-ports-forward-concurrent-map | 1 | 2 | 1 | 2 | 1 | 1 | **70** | very_hard | medium | quality [70.0, 70.0, 75.0]; buckets {'very_hard': 3} | +| issue-view-comments-with-json | 1 | 1 | 1 | 1 | 0 | 1 | **70** | hard | high | yes | +| release-verify-sha1-prefix | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 70.0]; buckets {'hard': 2, 'very_hard': 1} | +| auth-refresh-hint-on-expired-token | 1 | 1 | 1 | 2 | 2 | 1 | **75** | hard | medium | quality [70.0, 75.0, 75.0]; buckets {'hard': 3} | +| secret-set-app-agents | 2 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 75.0]; buckets {'very_hard': 2, 'hard': 1} | +| repo-sync-checked-out-worktree | 1 | 2 | 1 | 2 | 1 | 2 | **75** | very_hard | medium | quality [75.0, 75.0, 80.0]; buckets {'very_hard': 2, 'hard': 1} | +| auth-clipboard-config-default | 1 | 1 | 2 | 1 | 1 | 1 | **70** | hard | medium | yes | +| project-item-edit-by-name | 1 | 1 | 1 | 2 | 1 | 1 | **70** | hard | medium | yes | +| skill-update-dir-keeps-install-path | 1 | 2 | 1 | 2 | 1 | 2 | **75** | very_hard | high | yes | +| skill-install-nested-discovery | 1 | 1 | 1 | 2 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 70.0]; buckets {'hard': 2, 'very_hard': 1} | + +## What each task needs + +### repo-create-reject-empty-name + +The change needs an empty-name check in interactiveRepoNameAndOwner (create.go:929) that fails right after the "Repository name" prompt, using the existing "argument error" convention, with a new empty-name case in create_test.go's interactive table. Whether the non-interactive path and whitespace-only names also need rejecting is not stated in the task and was not confirmed against the repo's intent. + +### copilot-not-installed-trailing-newline + +The change is a one-line newline fix to the declined-install warning in runCopilot (pkg/cmd/copilot/copilot.go:155) plus a new test asserting the exact ErrOut content and SilentError return, using the existing prompter mock and TTY-simulated iostreams. The main judgement is whether to also add a newline to the non-prompt warning at copilot.go:159, which the task does not name. + +### repo-set-default-unset-without-repo + +The fix is a small RunE change in pkg/cmd/repo/setdefault/setdefault.go so non-interactive `--unset` skips the 'repository required' check, as `--view` already does, while plain `set-default` keeps the error. The real decision is what to do with a repository argument passed alongside `--unset`, which currently unsets the default silently. Tests go in setdefault_test.go for non-TTY `--unset`, non-TTY `--view` without an argument, and the retained error. + +### pr-checks-all-cancelled-summary + +The fix is a one-condition change in printSummary (pkg/cmd/pr/checks/output.go:71) so the summary and tally line are emitted when only cancelled checks exist, plus a new all-cancelled fixture and TTY test in checks_test.go alongside the existing someCancelled case. Existing exact-output tests already cover every other state mix, so a regression test for the all-cancelled case is the main new verification. + +### extension-uninstall-alias + +Add `uninstall` as a cobra alias on the existing `remove` command in pkg/cmd/extension/command.go, following the `ls` alias pattern, so both names run the same handler and remove is unchanged. Cover it with a new case in TestNewCmdExtension (the remove cases at command_test.go ~553-613 are the template), and decide whether the help requirement means the ALIASES section of `remove --help` or a separate listing. + +### pr-merge-delete-branch-owner-case + +The fix is a one-line case-insensitive owner comparison in `crossRepoPR` in pkg/cmd/pr/merge/merge.go, matching the EqualFold convention already used in context/remote.go and internal/ghrepo. The work also needs a new regression test in merge_test.go with an owner-case mismatch, since every existing fixture uses matching case and would not catch a revert. + +### pr-view-json-number-on-issue + +The fix is to stop the shared PR finder from answering number-only lookups without calling GitHub, so that `pr view --json number` goes through the same `pullRequest(number:)` query as other fields and fails for issue numbers. That change also reaches pr checks and pr diff, which use the same shortcut, so the existing number-only finder test (which pins the buggy no-API path) has to be rewritten and the view JSON path covered with new tests. + +### project-item-add-output-non-tty + +The task needs item-add's printResults to emit a stable non-TTY confirmation that includes the already-fetched ProjectItem.Id, while leaving TTY output and --format json unchanged, all within item_add.go and item_add_test.go. The main risks are choosing the exact wording and scope (whether sibling TTY-gated project commands should change too) and adding the missing non-TTY test. + +### release-create-404-scope-blame + +The fix is in `tokenHasWorkflowScope` (`pkg/cmd/release/create/http.go:301-312`), which must trim whitespace from the comma-split `X-Oauth-Scopes` value the way `api/client.go` and `oauth_scopes.go` already do, and it needs a regression test using GitHub's `, `-separated header in `create_test.go`. The harder part is deciding how the hint should coexist with the underlying 404 for tokens that truly lack `workflow`, because the task asks both to keep the hint and to stop hiding the real cause. + +### pr-comment-attach-ghu-tokens + +In this repository the fix is a deliberate change to the allowlist in internal/attachments/client.go (uploadTokenTypes), which is the only gate on which credentials may reach the upload endpoint, and the existing 'rejects an App user-to-server token' test must be replaced. The silent risk is over-admitting other token kinds or removing the gate, while the existing table tests in client_test.go already cover ghs_/ghr_ rejection and would catch that. + +### repo-clone-ghe-ssh-ca-url + +Clone must take the SSH URL from the API (`sshUrl`, which needs adding to the GitHubRepo GraphQL query for both the repo and its parent) instead of rebuilding it from the tenant name in clone.go, with HTTPS unchanged and a defined fallback for empty values. Tests in clone_test.go with httpmock and run.Stub should cover tenant SSH for both the origin and upstream remotes. Confidence is medium because the GraphQL sshUrl vs REST ssh_url equivalence and the fallback behaviour are inferred, not verified. + +### codespace-ports-forward-concurrent-map + +The fix belongs in the codespace port-forwarding path: ForwardPorts runs one goroutine per port, and each goroutine's CreateTunnelPort writes to the TunnelRequestOptions map that the forwarders share through connection.CodespaceConnection, so the change must make per-port tunnel operations safe, either by isolating options per forwarder or by serializing shared-state access as connection.go already does for Connect. A test that forwards several ports at once has to be added, and the offline tunnel mock already supports it, but nothing in the current suite exercises this path. + +### issue-view-comments-with-json + +The fix belongs in `pkg/cmd/issue/view/view.go`, where `--comments` is dropped on the `--json` path because `lookupFields` is built only from `Exporter.Fields()`, so either `comments` must be added and preloaded with the existing paginated `preloadIssueComments`, or the flags must be rejected with `cmdutil.MutuallyExclusive` as in discussion view. Tests belong in `view_test.go` alongside the existing `--comments` and `TestIssueView_json_*` cases using the `CommentsForIssue` fixtures. The sibling `pr view` has the same silent no-op, which the task does not mention. + +### release-verify-sha1-prefix + +The fix derives the digest algorithm (sha1 vs sha256) from the ref the API returns, since FetchRefSHA returns only the SHA string, and threads that single value through the attestation lookup, the sigstore digest policy and the printed output in both release verify commands. Tests must cover both 40- and 64-character refs and assert the printed prefix, which no existing test does, and SHA-256 attestation handling is inferred rather than verified against a real fixture. + +### auth-refresh-hint-on-expired-token + +The fix belongs in the 401 branch of internal/ghcmd/cmd.go: classify the token used for the failing host (from HTTPError.RequestURL) as login-created when ActiveToken's source is keyring or oauth_token, and suggest gh auth refresh only then. It needs tests for that branch, which currently has no coverage and cannot be injected without refactoring Main. The scores rest on code reading, not on running gh, so confidence is medium. + +### secret-set-app-agents + +Adding agents means extending the shared app enum, GetSecretApp and IsSupportedSecretEntity, plus flag enums and help text in set, list and delete, and most of the path logic already follows the `{app}` pattern. The non-obvious requirement is that the documented agents org endpoint takes integer selected_repository_ids (like Actions), not the string IDs that the Dependabot special case uses, and the new tests must pin repo and org set/list/delete paths and payloads in the existing httpmock style. + +### repo-sync-checked-out-worktree + +Syncing a branch checked out in another worktree must not leave that worktree's index and files stale, and the fix has to either update it safely (which requires checking that worktree's dirty state) or refuse with a message that names the worktree. The change sits in pkg/cmd/repo/sync (sync.go, git.go, mocks.go, sync_test.go) and needs new worktree-detection and dirty-check test cases, since the existing mock-based suite covers only the current-worktree and plain-ref paths. + +### auth-clipboard-config-default + +The task needs a new global enabled/disabled config key registered through the existing config Options, default and gh.Config accessor (with regenerated mock and test wiring), and read in `gh auth login` so the device code is copied without `--clipboard` while the flag still works. The main risks are the coordinated plumbing that a partial change would leave broken, and an unresolved scope question about whether `auth refresh` should honour the setting. + +### project-item-edit-by-name + +The task needs item-edit to resolve a project item from an issue or PR URL, a project from --owner and number, and a field and option (or value) by case-insensitive name, validated before the mutation, while keeping --id, --field-id and --project-id working. The item-from-URL lookup is not in the repo (the GitHub GraphQL schema is external), so its shape is inferred from the existing projectItems usage in pr/shared and issue/shared, and the change must be covered by new offline gock tests in item_edit_test.go. + +### skill-update-dir-keeps-install-path + +The fix is in pkg/cmd/skills/update/update.go: install into the skill's current parent directory for every scope (with or without --dir) and delete the post-install cleanup block, which means the installer's flat-only write (installer.go:251-253) must not relocate namespaced skills. The existing TestUpdateRun case at update_test.go:711-775 pins the old migrate-and-delete behavior and must be replaced with in-place tests for flat and namespaced layouts, covering both host and --dir paths. + +### skill-install-nested-discovery + +The reported path foo/bar/skills//SKILL.md already matches the 'skills' convention at any depth in internal/skills/discovery/discovery.go, so the real work is to confirm which nested case still fails (likely nested hidden-dir skills, which conflicts with the pinned negative test at discovery_test.go L231) before changing matchers and adding nested-skill tests. Scores rest on reading the code and tests, since I did not run the reproduction. diff --git a/docs/swe-router-rubric-cli-cli-v1-claude-opus-5-5.md b/docs/swe-router-rubric-cli-cli-v1-claude-opus-5-5.md new file mode 100644 index 00000000..9bbc1e31 --- /dev/null +++ b/docs/swe-router-rubric-cli-cli-v1-claude-opus-5-5.md @@ -0,0 +1,119 @@ +# How omp + us.anthropic.claude-opus-5-5 scores each task on the rubric + +The rubric pass over `benchmarks/dataset/cli-cli-v1.yaml`: each task scored on six dimensions, 0 to 2, 3 independent time(s), in a clone of its repository at the task's pinned ref. No model output was read; this describes the tasks only. Code turns the scores into a required quality and a difficulty bucket (`routing-policy.json`). + +- **Stability.** Required quality agreed across repeats on 18/21 tasks; the bucket on 20/21. +- **Runs.** 63 succeeded, 0 failed. Cost $15.5931. +- **Judged.** 2026-10-14T22:10:12.033833Z to 2026-10-14T22:23:36.365453Z. + +Columns: HC hidden contract, SI silent invariant, CS coordinated surfaces, AM ambiguity, VE verification, BR blast radius. Each score is the lower median of the repeats. + +| Task | HC | SI | CS | AM | VE | BR | Quality | Bucket | Confidence | Repeats agree | +|---|---:|---:|---:|---:|---:|---:|---:|---|---|---| +| repo-create-reject-empty-name | 1 | 1 | 0 | 1 | 1 | 1 | **70** | moderate | high | yes | +| copilot-not-installed-trailing-newline | 0 | 0 | 0 | 0 | 1 | 1 | **70** | easy | high | yes | +| repo-set-default-unset-without-repo | 1 | 0 | 0 | 1 | 1 | 1 | **70** | moderate | high | yes | +| pr-checks-all-cancelled-summary | 1 | 0 | 0 | 0 | 1 | 1 | **70** | easy | high | yes | +| extension-uninstall-alias | 1 | 0 | 0 | 1 | 1 | 1 | **70** | moderate | high | yes | +| pr-merge-delete-branch-owner-case | 1 | 1 | 0 | 1 | 1 | 1 | **70** | moderate | high | quality [70.0, 70.0, 70.0]; buckets {'hard': 1, 'moderate': 2} | +| pr-view-json-number-on-issue | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | yes | +| project-item-add-output-non-tty | 1 | 0 | 0 | 1 | 1 | 1 | **70** | moderate | high | yes | +| release-create-404-scope-blame | 1 | 1 | 0 | 1 | 1 | 1 | **70** | moderate | high | yes | +| pr-comment-attach-ghu-tokens | 1 | 1 | 0 | 1 | 0 | 1 | **70** | moderate | medium | yes | +| repo-clone-ghe-ssh-ca-url | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | yes | +| codespace-ports-forward-concurrent-map | 1 | 2 | 1 | 2 | 1 | 1 | **70** | very_hard | medium | yes | +| issue-view-comments-with-json | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | high | yes | +| release-verify-sha1-prefix | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 75.0]; buckets {'hard': 3} | +| auth-refresh-hint-on-expired-token | 1 | 1 | 1 | 2 | 2 | 1 | **75** | hard | medium | yes | +| secret-set-app-agents | 2 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | yes | +| repo-sync-checked-out-worktree | 1 | 2 | 1 | 1 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 75.0]; buckets {'hard': 3} | +| auth-clipboard-config-default | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | high | yes | +| project-item-edit-by-name | 1 | 1 | 1 | 1 | 1 | 1 | **70** | hard | medium | yes | +| skill-update-dir-keeps-install-path | 1 | 2 | 1 | 1 | 1 | 1 | **70** | hard | medium | quality [70.0, 70.0, 75.0]; buckets {'hard': 3} | +| skill-install-nested-discovery | 1 | 2 | 1 | 2 | 1 | 1 | **70** | very_hard | medium | yes | + +## What each task needs + +### repo-create-reject-empty-name + +Add an empty (and likely whitespace-only) name check in `interactiveRepoNameAndOwner` in `pkg/cmd/repo/create/create.go`, right after the "Repository name" Input and before the owner lookup and Select. It should return an error in the repo's "cannot be blank" style so the from-scratch, template and local flows all stop early. Add `create_test.go` table cases with an empty name that assert the error and that no later prompts or API calls happen, and keep the existing valid-name tests passing. + +### copilot-not-installed-trailing-newline + +Add a trailing newline to the 'Copilot CLI was not installed' warning in runCopilot (pkg/cmd/copilot/copilot.go:155), keeping the SilentError return. Add a runCopilot test that stubs findCopilotBinaryFunc to return "", makes the prompter decline with a TTY iostreams, and asserts the exact stderr output. + +### repo-set-default-unset-without-repo + +Relax the non-interactive 'repository required' guard in NewCmdSetDefault (pkg/cmd/repo/setdefault/setdefault.go) so that `--unset`, like `--view`, runs without a repository argument when stdin is not a terminal, while plain `gh repo set-default` keeps the existing FlagError. Extend TestNewCmdSetDefault with a TTY toggle and cases for non-TTY `--unset`, non-TTY `--view` and the non-TTY no-argument error that must stay. + +### pr-checks-all-cancelled-summary + +The fix is to add counts.Canceled to the guard in printSummary (pkg/cmd/pr/checks/output.go) so that all-cancelled PRs reach the existing 'Some checks were cancelled' branch and its tally line. It needs a new all-cancelled fixture and a test row in checks_test.go that asserts the full summary output. + +### extension-uninstall-alias + +Add `uninstall` as a cobra alias on the existing `remove` subcommand in pkg/cmd/extension/command.go, following the `list`/`ls` pattern, so routing and the ALIASES help section come for free. Add a table-driven case to pkg/cmd/extension/command_test.go that runs `uninstall` and checks that Remove is called with the normalized name. + +### pr-merge-delete-branch-owner-case + +The crossRepoPR computation in pkg/cmd/pr/merge/merge.go:592 must compare the PR head owner login with the base repo owner case-insensitively (matching ghrepo.IsSame/FindByRepo), so same-repo PRs get their remote branch deleted. A merge_test.go regression test must cover an owner-case mismatch and expect the DELETE git/refs call, while fork PRs keep their branches. + +### pr-view-json-number-on-issue + +The shared PR finder (pkg/cmd/pr/shared/finder.go) returns a synthetic PullRequest without any API call when only 'number' is requested, so gh pr view N --json number succeeds for issue numbers. The fix must make that path check with GitHub, either in the finder (which also changes pr diff and pr checks) or in pr view, and must update the finder test that pins the shortcut and add an issue-number case that expects an error. + +### project-item-add-output-non-tty + +Change `printResults` in `pkg/cmd/project/item-add/item_add.go` so that non-TTY stdout gets a short, stable confirmation (preferably the added item's ID from the mutation response), leaving the TTY "Added item" message and the `--format json` exporter path unchanged. Add a non-TTY test next to the existing TTY and JSON tests in `item_add_test.go`. + +### release-create-404-scope-blame + +tokenHasWorkflowScope in pkg/cmd/release/create/http.go splits X-Oauth-Scopes on "," without trimming whitespace, so real "a, b, workflow" headers never match "workflow" and every 404 is replaced by the scope hint. The fix is to trim each scope, following HeaderHasMinimumScopes in auth/shared, and to add create_test.go cases that use the real comma-space header format for tokens with and without workflow, keeping the existing empty-header rule. + +### pr-comment-attach-ghu-tokens + +Add gh.TokenTypeUserToServer to uploadTokenTypes in internal/attachments/client.go, keeping ghs_/ghr_/unknown rejected and the host check first. Flip the 'rejects an App user-to-server token' row in TestNewUploader and add a command-level test with a ghu_ token that uploads, plus a check that a server refusal comes back to the user. + +### repo-clone-ghe-ssh-ca-url + +Add sshUrl to the RepositoryInfo fragment in api.GitHubRepo for the repository and its parent, and have cloneRun use the API SSH URL for both the clone and the fork upstream remote when the protocol is ssh, including the URL-argument and wiki paths, while leaving HTTPS unchanged. Add clone_test.go cases with mocked sshUrl responses that check the git clone and remote add commands. + +### codespace-ports-forward-concurrent-map + +ForwardPorts runs one goroutine per port, and all of them share a single CodespaceConnection whose TunnelRequestOptions.AdditionalHeaders map and Tunnel.Ports are changed by dev-tunnels Manager calls, so the fix needs synchronization (or per-call copies) at the shared connection or in ForwardPorts, not inside each per-goroutine forwarder. It also needs a new test that forwards several ports at once against the existing mock tunnels server, which CI runs under -race. + +### issue-view-comments-with-json + +In `pkg/cmd/issue/view/view.go`, `--comments` combined with `--json` must either add `comments` to the exporter's field list (so the comments are fetched and written) or fail with a mutually-exclusive flag error after PreRunE has set up the Exporter. `--json` alone and `--comments` alone must behave as they do today. New table tests in `view_test.go` should cover the combination using the existing httpmock fixtures. + +### release-verify-sha1-prefix + +Replace the hard-coded "sha1" passed to `artifact.NewDigestedArtifactForRelease` in both `verify.go` and `verify_asset.go` with an algorithm worked out from the ref returned by `shared.FetchRefSHA` (40 hex characters means sha1, 64 means sha256), ideally through one shared helper. The output text, the `GetByDigest` lookup and the sigstore digest policy all need the same algorithm, with tests covering both SHA-1 and SHA-256 refs in both commands. + +### auth-refresh-hint-on-expired-token + +In internal/ghcmd/cmd.go, the 401 branch should print `gh auth refresh` (probably with `-h `) only when the active token for the host of the failed request was stored by gh and is an OAuth token. That is the oauth_token/keyring source plus gho_ prefix rule from agent_task.go; every other token keeps `gh auth login`. New tests are needed because nothing covers this branch today, and they should cover the tricky cases (a `--with-token` PAT, a gho_ token from an env var, a non-default host), not just the obvious pair. + +### secret-set-app-agents + +Add an `agents` App to pkg/cmd/secret/shared with GetSecretApp and IsSupportedSecretEntity support (repository and organization), add it to the `--app` enum in set, list and delete, choose the right org PUT payload in set/http.go, and update help text. Add httpmock-based tests for repo and org set, list and delete whose paths and payloads must match GitHub's documented agents secrets REST endpoints. + +### repo-sync-checked-out-worktree + +`executeLocalRepoSync` needs to find out whether the target branch is checked out in any other worktree, for example through a new gitClient/git.Client method that parses `git worktree list --porcelain`. If it is, sync must refuse with a message naming that worktree's path, or update it safely, instead of moving the ref with `update-ref`. That means changing the gitClient interface, the executer, mocks.go and the table tests in sync_test.go, while syncing a branch nobody has checked out keeps working as before. + +### auth-clipboard-config-default + +Add a `clipboard` enabled/disabled setting following the spinner/prefer_editor_prompt pattern: the gh.Config interface method, regenerated mock, const, accessor, default config string, Options entry, test stub and the config list test expectation. Then have `gh auth login` copy the code when either `--clipboard` is passed or the setting is enabled, with tests covering how flag and config combine. + +### project-item-edit-by-name + +Add an opt-in name form to `gh project item-edit` (` --owner --url --field --value`). It must resolve the project, the content-to-item mapping, the field (case-insensitive) and the option or typed value before calling the existing `updateProjectV2ItemFieldValue` mutation, and fail with a list of candidates when a name does not match. It must also keep the ID-flag paths unchanged and add gock-mocked tests in `item_edit_test.go`, probably along with a new lookup helper in `shared/queries`. + +### skill-update-dir-keeps-install-path + +In pkg/cmd/skills/update/update.go, the update must reinstall each skill into the parent of its current directory for flat and namespaced layouts, whether or not an agent host is involved, and the RemoveAll/parent-cleanup block must go. update_test.go needs its namespaced test turned around and new cases showing the original path is kept and updated in place. + +### skill-install-nested-discovery + +The core repro (foo/bar/skills//SKILL.md) already matches in internal/skills/discovery/discovery.go and has tests. The real work is to decide whether nested skills under hidden or plugins directories should be discovered when --allow-hidden-dirs is set, and to extend the shared matcher (used by remote, local, preview, search and path-based install) without breaking today's conventions or the hidden-dir gate. Nested-skill tests are needed across discovery and install. From 683e74f380f6f49735d0768d6506e89789d78290 Mon Sep 17 00:00:00 2001 From: Amit Arora Date: Wed, 14 Oct 2026 22:54:42 +0000 Subject: [PATCH 3/4] Name each rubric pass after its dataset A rubric pass scores one dataset's tasks, so two datasets need two files. The v2 passes were named only after their judge; rename them to swe-router-rubric-mcp-gateway-registry-v2-claude-{opus,haiku}-5-5, the pattern the cli/cli passes already use, and update the references in swe-router-rubric-evaluation.md. run-swe-router-headless.py --mode rubric now defaults --out to docs/metrics/swe-router-rubric--.json, so a pass over a new dataset, or by a second judge, never overwrites an existing file. Floor mode keeps its fixed name. --- ...e-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.json} | 0 ...we-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json} | 0 ...swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.md} | 0 ... swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.md} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename docs/metrics/{swe-router-rubric-claude-haiku-5-5.json => swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.json} (100%) rename docs/metrics/{swe-router-rubric-claude-opus-5-5.json => swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json} (100%) rename docs/{swe-router-rubric-claude-haiku-5-5.md => swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.md} (100%) rename docs/{swe-router-rubric-claude-opus-5-5.md => swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.md} (100%) diff --git a/docs/metrics/swe-router-rubric-claude-haiku-5-5.json b/docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.json similarity index 100% rename from docs/metrics/swe-router-rubric-claude-haiku-5-5.json rename to docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.json diff --git a/docs/metrics/swe-router-rubric-claude-opus-5-5.json b/docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json similarity index 100% rename from docs/metrics/swe-router-rubric-claude-opus-5-5.json rename to docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json diff --git a/docs/swe-router-rubric-claude-haiku-5-5.md b/docs/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.md similarity index 100% rename from docs/swe-router-rubric-claude-haiku-5-5.md rename to docs/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.md diff --git a/docs/swe-router-rubric-claude-opus-5-5.md b/docs/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.md similarity index 100% rename from docs/swe-router-rubric-claude-opus-5-5.md rename to docs/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.md From d8520932fbb5f7a107b7741bd494bed8ab0abf3e Mon Sep 17 00:00:00 2001 From: Amit Arora Date: Wed, 14 Oct 2026 22:54:49 +0000 Subject: [PATCH 4/4] Default a rubric pass's output name to its dataset and judge run-swe-router-headless.py --mode rubric now writes docs/metrics/swe-router-rubric--.json when --out is not given, so a pass over a new dataset, or by a second judge, never overwrites an existing file. Floor mode keeps its fixed name. Update swe-router-rubric-evaluation.md for the renamed v2 files. --- benchmarks/scripts/run-swe-router-headless.py | 32 +++++++++++++++---- .../tests/test_run_swe_router_headless.py | 18 +++++++++++ docs/swe-router-rubric-evaluation.md | 8 ++--- 3 files changed, 47 insertions(+), 11 deletions(-) diff --git a/benchmarks/scripts/run-swe-router-headless.py b/benchmarks/scripts/run-swe-router-headless.py index d86d68b8..7900621d 100644 --- a/benchmarks/scripts/run-swe-router-headless.py +++ b/benchmarks/scripts/run-swe-router-headless.py @@ -100,10 +100,10 @@ MODE_FLOOR = "floor" MODE_RUBRIC = "rubric" MODES = (MODE_FLOOR, MODE_RUBRIC) -DEFAULT_OUT = { - MODE_FLOOR: REPO_ROOT / "docs" / "metrics" / "swe-router-judged-inputs.json", - MODE_RUBRIC: REPO_ROOT / "docs" / "metrics" / "swe-router-rubric-inputs.json", -} +DEFAULT_FLOOR_OUT = REPO_ROOT / "docs" / "metrics" / "swe-router-judged-inputs.json" +# A rubric pass describes one dataset's tasks, so its default file name carries +# the dataset scope and the judging model: one file per dataset and judge. +RUBRIC_OUT_TEMPLATE = "swe-router-rubric-{scope}-{model}.json" DEFAULT_DATASET = "dataset/mcp-gateway-registry-v2.yaml" DEFAULT_TIMEOUT_SECONDS = 900 @@ -150,6 +150,24 @@ def _utc_now_iso() -> str: return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") +def _default_out( + mode: str, + dataset: Dataset, + dataset_path: Path, + config: RunnerConfig, +) -> Path: + """Where to write the judged inputs when --out is not given. + + A rubric pass is named after its dataset and judge, so passes over two + datasets, or by two judges, never overwrite each other. + """ + if mode != MODE_RUBRIC: + return DEFAULT_FLOOR_OUT + scope = dataset.output_scope or dataset_path.stem + name = RUBRIC_OUT_TEMPLATE.format(scope=scope, model=config.model_slug) + return REPO_ROOT / "docs" / "metrics" / name + + def _build_prompt(task: Task, clone_path: Path) -> str: """Build the step-1 prompt: the skill, the task, and the output contract. @@ -907,8 +925,8 @@ def _parse_args() -> argparse.Namespace: type=Path, default=None, help="Where to write the judged inputs. Default: " - "docs/metrics/swe-router-judged-inputs.json, or " - "docs/metrics/swe-router-rubric-inputs.json in rubric mode.", + "docs/metrics/swe-router-judged-inputs.json, or in rubric mode " + "docs/metrics/swe-router-rubric--.json.", ) parser.add_argument( "--out-md", @@ -972,7 +990,7 @@ def main() -> None: except DatasetError as exc: raise SystemExit(f"dataset error: {exc}") from exc tasks = _select_tasks(dataset, args.tasks.split(",") if args.tasks else []) - args.out = args.out or DEFAULT_OUT[args.mode] + args.out = args.out or _default_out(args.mode, dataset, dataset_path, config) rubric_text, policy = "", None if args.mode == MODE_RUBRIC: try: diff --git a/benchmarks/tests/test_run_swe_router_headless.py b/benchmarks/tests/test_run_swe_router_headless.py index 039da77e..98dba4b5 100644 --- a/benchmarks/tests/test_run_swe_router_headless.py +++ b/benchmarks/tests/test_run_swe_router_headless.py @@ -240,3 +240,21 @@ class _Config: if __name__ == "__main__": unittest.main() + + +class DefaultOutTest(unittest.TestCase): + def _out(self, mode: str, scope: str | None) -> Path: + dataset = mock.Mock(output_scope=scope) + config = mock.Mock(model_slug="claude-opus-5-5") + return rr._default_out(mode, dataset, Path("dataset/some-set.yaml"), config) + + def test_rubric_file_names_the_dataset_scope_and_judge(self) -> None: + out = self._out(rr.MODE_RUBRIC, "cli-cli-v1") + self.assertEqual(out.name, "swe-router-rubric-cli-cli-v1-claude-opus-5-5.json") + + def test_rubric_file_falls_back_to_the_dataset_file_name(self) -> None: + out = self._out(rr.MODE_RUBRIC, None) + self.assertEqual(out.name, "swe-router-rubric-some-set-claude-opus-5-5.json") + + def test_floor_mode_keeps_its_fixed_name(self) -> None: + self.assertEqual(self._out(rr.MODE_FLOOR, "cli-cli-v1"), rr.DEFAULT_FLOOR_OUT) diff --git a/docs/swe-router-rubric-evaluation.md b/docs/swe-router-rubric-evaluation.md index 1f10829f..461847f2 100644 --- a/docs/swe-router-rubric-evaluation.md +++ b/docs/swe-router-rubric-evaluation.md @@ -57,7 +57,7 @@ Source: `docs/metrics/swe-router-rubric-replays/{today,haiku-5-5-absolute,opus-5 | Separation, bucket | 0.68 | 0.73 | | | Separation, required quality | 0.53 | 0.66 | | -Source: `docs/metrics/swe-router-rubric-claude-{haiku-5-5,opus-5-5}.json` and the per-task scores in each model's `run-summary.json`. +Source: `docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-{haiku-5-5,opus-5-5}.json` and the per-task scores in each model's `run-summary.json`. **The two judges agree.** Same bucket on 16 of 21 tasks, same required quality on 18 of 21, and never more than one bucket apart. Haiku 5.5 sorts tasks at least as well as Opus 5.5, at 4% of the cost per judgment. @@ -147,12 +147,12 @@ From `benchmarks/`: # Rubric passes (Claude Code on Amazon Bedrock) uv run scripts/run-swe-router-headless.py --mode rubric --agent claude --provider bedrock \ --model us.anthropic.claude-haiku-5-5 --aws-region us-east-1 --repeats 3 \ - --out ../docs/metrics/swe-router-rubric-claude-haiku-5-5.json \ - --out-md ../docs/swe-router-rubric-claude-haiku-5-5.md + --out ../docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.json \ + --out-md ../docs/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.md # Replay, leave-one-out, every model available uv run scripts/eval_swe_router.py --holdout --no-allow-list \ - --rubric-inputs ../docs/metrics/swe-router-rubric-claude-haiku-5-5.json --bucketing absolute + --rubric-inputs ../docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-haiku-5-5.json --bucketing absolute # Today's skill, the same way uv run scripts/eval_swe_router.py --holdout --no-allow-list \