Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions offer/SCORECARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ Source: issue #442.
| 2 | defense-in-depth | security | 2026-10-06 (offer init + sibling effort registry + AGENTS wiring committed to main; pilot files only) | **First-audit day-one catches**: stale CLI refs (hints-emit nonexistent, --export-lessons→--export-rules, --name→--metric, "11 commands"→10), federation doc v0.6.0 vs repo 1.0.0, 180-day-old audit report presented as current, 3+ broken quickstart links, relative-depth link rot — 43 files, 291K tokens, LOW tier auto-classified via the security-profile registry | M1 ✓ (2/5 onboarded) · M2 ✓ (real catches day-one, [CODE]-cited) — wave 2 live |
| 3 | aegis | infra | 2026-10-06 (sibling effort registry + AGENTS wiring committed; lane/trust scaffolding pre-existing) | **First-audit day-one**: 19 files (docs/adrs/ 14 ADRs + 5 root docs) — 4 orphaned/stale root scratch artifacts, legacy repo name (`web-login-solo`) citations, ADR status/version drift; 246K tokens, LOW auto-classified via the infra-profile registry | M1 ✓ (3/5 — falsification target hit day 1 of the window) · M2 ✓ — wave 3 live, in the weekly patrol |
| 3 | homelab-proxmox | infra | 2026-10-14 (commit 982f0a1: `g8s offer init --profile infra` + `.g8s/effort-classes.yml` + `CONTRIBUTING.md` wiring) | **First-audit day-one** (task `dd889ebc`, 121.8s, 103.6K tokens, `LOW` auto-classified via sibling `.g8s/effort-classes.yml`): G3 `ci_link_integrity.sh` caught dead relative link `README.md:230 -> 'authelia/'`; 4 live service dirs (`caddy/`, `kutt/`, `semaphore/`, `teleport/`) omitted from `README.md` inventory/structure tables; stale `CT105/authelia` & `CT118/gitea` rows; broken `.agents/AGENTS.md` script path `MCP-Server/tools/bootstrap_orchestrator.sh` | M1 ✓ (4/5 onboarded) · M2 ✓ (gate + verifier `[CODE]` catches day-one) — wave 3 live |
| 5 | hash-checker | utility | 2026-10-18 (PR #80: g8s offer init --profile utility + .g8s/verifier-classes.yml + CI gates) | **First-audit M2 catches**: #63 conflicting --algorithm vs hash length silent override fixed with HashError::ConflictingAlgorithms + CLI DAST tests, #53 scripts/report-deps.sh, #52/#54 signing & roadmap sync | M1 ✓ (5/5 onboarded) · M2 ✓ — wave 5 live |
| 4 | tuneflow | utility | 2026-10-14 (commit 7129e2d: `g8s offer init --profile utility` + `.g8s/effort-classes.yml` + `AGENTS.md` wiring) | **First-audit day-one** (task `d0b3a325`, 171.9s, 141.5K tokens, `LOW` auto-classified via sibling `.g8s/effort-classes.yml`): 5 `[CODE]`-cited catches — (1) `AGENTS.md:46` claims `8/8 tests pass` vs 11 unit test blocks in `tests/engine.test.js`, (2) `AGENTS.md:54-71` cites removed `src/engine.js` (now 8 modules in `src/engine/`) and omits `src/db/`, `src/auth/`, `src/middleware/`, `src/desktop/`, `src/security/`, (3) `package.json` `tests/*.test.js` vs `test:unit` mandate drift, (4) `AGENTS.md:27` strict `pnpm` law vs `npm run` table & dual `package-lock.json` + `pnpm-lock.yaml`, (5) `README.md:93` `compose.yaml` vs `docker-compose.yml` | M1 ✓ (4/4 existing repos onboarded across all 4 profiles) · M2 ✓ — wave 4 live |
| 5 | hash-checker | utility | 2026-10-18 (PR #80: `g8s offer init --profile utility` + `.g8s/verifier-classes.yml` + CI gates) | **First-audit** (read-only task `7eaa5c11`, run 2026-10-18 after onboarding): worker reported 5 findings; F0 verification confirmed **1** — `docs/OPERATIONS.md:191` says workflow_dispatch does not run cargo-dist, but `.github/workflows/ci.yml:36-97` exposes `run_packaging` which runs `cargo-dist` — and found 1 partially valid (batch exit-code doc vs `src/main.rs:191` exit 1 on command failure); 3 rejected as wrong/inconclusive. Note: issues #63/#53/#52/#54 resolved in PR #80 were pre-existing GitHub issues, NOT first-audit catches. | M1 ✓ · M2 ✓ (1 verified `[CODE]` catch) — wave 5 live |

## Falsification tracker

- M1: **5/5 target repos onboarded** across all 4 domain profiles (`knowledge`, `security`, `infra`, `utility`; target ≥3/5 **HIT** 2026-10-06, completed 2026-10-18)
- M2: **5/5 first-audits caught ≥1 real `[CODE]`-cited issue day-one** (`tamld-llm-wiki` ~135 findings, `defense-in-depth` 5+ CLI/doc drifts, `aegis` 4 stale artifacts + repo-name drift, `homelab-proxmox` 1 dead link + 4 missing service dirs, `tuneflow` 5 code↔doc/script drifts, `hash-checker` #63 conflicting algorithm override + #53 + #52/#54) — metric **SATISFIED** across all waves
- M3: **16/2 contribution packets shipped back to `g8s`** (target ≥2 **SATISFIED**):
- M1: **5/5 original target repos onboarded** (`tamld-llm-wiki`, `defense-in-depth`, `aegis`, `tuneflow`, `hash-checker`) across all 4 domain profiles, plus `homelab-proxmox` as an extra infra repo (target ≥3/5 **HIT** 2026-10-06, completed 2026-10-18)
- M2: **6/6 first-audits caught ≥1 real `[CODE]`-cited issue day-one** (`tamld-llm-wiki` ~135 findings, `defense-in-depth` 5+ CLI/doc drifts, `aegis` 4 stale artifacts + repo-name drift, `homelab-proxmox` 1 dead link + 4 missing service dirs, `tuneflow` 5 code↔doc/script drifts, `hash-checker` 1 verified doc↔CI drift (3 of 5 worker findings rejected on F0 verification)) — metric **SATISFIED** across all waves
- M3: **14/2 contribution packets shipped back to `g8s`** (target ≥2 **SATISFIED**):
1. `#566`: Worker `(model, effort)` realignment + empty-provider claim affinity (from `aegis` Wave 3)
2. `#567`: Sibling `.g8s/effort-classes.yml` resolution from `--scope-root` in `g8s submit` (from `aegis` Wave 3; verified live on `homelab-proxmox` & `tuneflow`)
3. `#569`: `g8s deliver` deletion propagation from worktree to target repository (from `aegis` Wave 3)
Expand All @@ -31,9 +31,8 @@ Source: issue #442.
11. `tools/ci_spec_code_sync.sh` (`f40aac7`): Polyglot test-file pin resolution (`tests/<file>`) + `docs/openspec/specs` auto-detection + nested `<domain>/spec.md` scanning
12. `tools/ci_structure_sync.sh`: Non-Go `docs/README.md` taxonomy coverage verification — when `go.mod` is absent and `docs/README.md` exists, verifies every first-level `docs/<subdir>/` directory is documented in `docs/README.md` (verified live on `tuneflow#148` covering 9 `docs/` directories)
13. `cmd/g8s/deliver.go`: `--prune-worktree` flag on `g8s deliver` to atomically remove the isolated worktree (`wt-<hash>`) and temporary `wt/<task-id>` branch after receipt-verified file delivery
14. `#611`: Preserved worktree receipt override (`--receipt-id`) and cross-repo deliverable isolation
15. `#614`: Wall-clock receipt expiry bypass for overnight worker tasks pre-consumed before deadline
16. `#617`: Sibling verifier classes scaffolding in `g8s offer init` with default CI gate script checks
14. `#617` (partial, PR #618): `g8s deliver` accepts a receipt already consumed by the same task after wall-clock expiry, plus `--receipt-id` override; `g8s offer init` now scaffolds `.g8s/verifier-classes.yml` (from Wave-1 cross-repo dogfood). Session-pool worktree cleanup (criterion 1) is still open — #617 reopened.
- Not counted as sibling Mode-3 returns: `#611` (sa-006 probe now checks the `fsops.Edit` result and final file content — found by red-team review of PR #604, an internal finding) and `#614` (`conv.Run` cleans up `blind/*` worktrees/branches — a g8s self-hygiene finding from its own test suite).
- M4: retention clock starts 2026-10-05 (3-month check ~2027-01-05; 0 gate removals across all 4 onboarded repositories)

## Empirical F1 Worker Quotas (3-Round Parallel Cross-Repo Benchmark: `aegis` + `tuneflow`)
Expand Down
Loading