mame · 251478025-coder · Mar 15, 2026 · Mar 15, 2026 · Mar 15, 2026 · Mar 15, 2026
diff --git a/.env.example b/.env.example
@@ -0,0 +1,8 @@
+# Get your API key from https://ai.google.dev/
+GOOGLE_API_KEY=your-api-key-here
+
+# Get your API key from https://console.groq.com/
+GROQ_API_KEY=your-api-key-here
+
+# Get your API key from https://platform.openai.com/
+OPENAI_API_KEY=your-api-key-here
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,5 @@
 generated/
-logs/
+artifacts/generated/
+.env
+config/codexes.local.yml
+_doc/
diff --git a/AGENT.md b/AGENT.md
@@ -0,0 +1,140 @@
+# Agent & Contributor Protocol
+
+> Rules of engagement for human and AI (codex) contributors.
+> Read [program.md](./program.md) first for the high-level iteration loop.
+
+---
+
+## Golden rule
+
+**Every session must leave the project better documented than it found it.**
+
+After each iteration (experiment run, code change, bug fix):
+
+1. **Append** a dated entry to [walkthrough.md](./walkthrough.md)
+2. **Update** [plan.md](./plan.md) — mark completed items, add new next steps
+
+This is **mandatory**, not optional. Skipping documentation means the next contributor (human or AI) starts blind.
+
+---
+
+## Session protocol
+
+### Start of session
+
+```
+1. Read program.md         → understand the current research question
+2. Read plan.md            → see active experiments + backlog
+3. Read walkthrough.md     → see recent results + decisions
+4. Pick a task from plan.md and mark it [/] (in progress)
+```
+
+### During session
+
+```
+5. Do the work (run benchmarks, add adapters, fix bugs, add problems)
+6. Run tests: `bin/which-language run <codex> <problem> --lang <lang> --trials 1`
+7. Observe metrics: time, cost, LOC, pass rate
+```
+
+### End of session
+
+```
+8. Append walkthrough.md entry (template below)
+9. Update plan.md: mark completed [x], add new items [ ], note blockers
+10. Commit with descriptive message
+```
+
+---
+
+## Walkthrough entry template
+
+```markdown
+## YYYY-MM-DD — [Brief title]
+
+**Contributor**: [human / codex-name]
+**What was done**: [1–3 sentences]
+**Codex/Problem/Language**: [e.g., gemini / minigit / python]
+**Key metrics**: [time, cost, LOC, pass rate — if applicable]
+**Observations**: [what worked, what didn't]
+**Decisions made**: [any design choices or trade-offs]
+**Next**: [immediate next step, links to plan.md items]
+```
+
+---
+
+## Code conventions
+
+### Ruby (benchmark runner + adapters)
+
+- `# frozen_string_literal: true` at the top of every `.rb` file
+- Codex adapters extend `BaseCodex` (`lib/codexes/base_codex.rb`)
+- Required interface: `run_generation(prompt, dir:, log_path:)`, `version`
+- Optional interface: `warmup(dir)`, `parse_metrics(raw_output)`
+- Use `config/codexes.local.yml` for secrets — never commit API keys
+
+### Python (plot.py)
+
+- Python 3.10+, stdlib + matplotlib
+
+### Problem layout
+
+```
+problems/<problem>/
+  problem.json       # metadata + prompt templates
+  SPEC-v1.txt        # v1 specification
+  SPEC-v2.txt        # v2 specification (extends v1)
+  test-v1.sh         # v1 test suite
+  test-v2.sh         # v2 test suite
+```
+
+---
+
+## New codex adapter checklist
+
+When adding support for a new AI coding system:
+
+- [ ] Create `lib/codexes/<name>_codex.rb` extending `BaseCodex`
+- [ ] Implement `run_generation(prompt, dir:, log_path:)`
+- [ ] Implement `version`
+- [ ] Implement `parse_metrics(raw_output)` if the API returns token/cost data
+- [ ] Add configuration block to `config/codexes.yml`
+- [ ] Add API key handling (env var expansion in config)
+- [ ] Test dry-run: `bin/which-language benchmark <name> minigit --lang python --trials 1 --dry-run`
+- [ ] Run 1 real trial: `bin/which-language run <name> minigit --lang python --trials 1`
+- [ ] Run 3+ trials for variance data
+- [ ] Document pricing in `CLAUDE.md`
+- [ ] Document the new adapter in relevant `.md` files (e.g., `program.md`, `README.md`) to update project context
+- [ ] Update `plan.md` status
+- [ ] Add walkthrough.md entry
+
+## New problem checklist
+
+- [ ] Create `problems/<problem>/problem.json` with all required keys
+- [ ] Write `SPEC-v1.txt` and `SPEC-v2.txt`
+- [ ] Write `test-v1.sh` and `test-v2.sh` (deterministic, self-contained)
+- [ ] Test with at least 1 codex × 1 language × 1 trial
+- [ ] Document the new problem in relevant `.md` files (e.g., `program.md`, `CLAUDE.md`, `README.md`) to update project context
+- [ ] Add walkthrough.md entry
+- [ ] Update `plan.md` status
+
+---
+
+## PR guidelines
+
+- Title format: `[component] Brief description` (e.g., `[codex] Add DeepSeek adapter`)
+- Include benchmark results if adding/modifying adapters
+- Keep PRs focused — one adapter or one problem per PR
+- Run `bin/which-language benchmark <name> minigit --dry-run` before submitting
+
+---
+
+## For AI agents specifically
+
+If you are an AI agent (Claude, Gemini, Codex, etc.) working in this repo:
+
+1. **Do not modify `prepare.py`-equivalent files** — `report.rb` and `plot.py` are reporting tools, modify only if explicitly asked
+2. **Scope your changes** — touch only the files relevant to your task
+3. **Be explicit about failures** — if a benchmark run fails, document the failure mode in walkthrough.md
+4. **Prefer small iterations** — one codex adapter or one problem per session
+5. **Always update documentation** — this is the most important rule
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -1,63 +1,135 @@
-# Claude Code Language Benchmark
+# AI Coding Language Benchmark
+
+> **Contributors**: Read [AGENT.md](./AGENT.md) for the contribution protocol and [program.md](./program.md) for the current research focus.
 
 ## Overview
 
-Benchmark that has Claude Code (Opus) implement "MiniGit" (a minimal git clone) in multiple languages, comparing generation time, LOC, token usage, and pass rate.
+Multi-codex benchmark that has various AI coding assistants (Claude Code, Gemini, etc.) implement coding problems (MiniGit, MiniGrades, etc.) in multiple languages, comparing generation time, LOC, token usage, and pass rate.
 
 ## Repository Structure
 
 ```
-SPEC-v1.txt          # MiniGit v1 spec (init/add/commit/log)
-SPEC-v2.txt          # MiniGit v2 spec (v1 + status/diff/checkout/reset/rm/show)
-test-v1.sh           # v1 test suite (11 tests)
-test-v2.sh           # v2 test suite (30 tests)
-benchmark.rb         # Benchmark runner (Ruby)
-report.rb            # Report generator (results.json -> report.md)
-plot.py              # Graph generator (results.json -> figures/*.png)
-results/
-  results.json       # Raw result data
-  meta.json          # Environment metadata
-  report.md          # Generated report
-figures/             # Generated graphs
+bin/
+  which-language     # Unified CLI entry point (benchmark / report / plot / run)
+src/
+  benchmark.rb       # Benchmark runner (Ruby)
+  report.rb          # Report generator (results.json -> report.md)
+  plot.py            # Graph generator (results.json -> figures/*.png)
+lib/
+  codexes/
+    base_codex.rb    # Abstract interface
+    claude_codex.rb  # Claude Code CLI adapter
+    gemini_codex.rb  # Google Gemini API adapter
+    openai_codex.rb  # OpenAI Responses API adapter
+    groq_codex.rb    # Groq API adapter
+    aider_codex.rb   # Aider CLI adapter
+  codex_loader.rb    # Loads and instantiates adapters
+  language_loader.rb # Loads language config from YAML
+config/
+  codexes.yml        # Codex configuration
+  languages.yml      # Language toolchain configuration
+problems/
+  minigit/
+    problem.json     # Problem-specific prompt + asset config
+    SPEC-v1.txt      # v1 spec
+    SPEC-v2.txt      # v2 spec (extends v1)
+    test-v1.sh       # v1 test suite
+    test-v2.sh       # v2 test suite
+artifacts/
+  <codex>/<model>/<problem>/
+    generated/       # Generated source/build artifacts
+    logs/            # Codex logs
+    results/         # Raw result data + meta + report
+    figures/         # Generated graphs
 ```
 
-The `data` branch (orphan) contains:
-```
-generated/           # Source code generated by Claude (binaries removed)
-  minigit-{lang}-{trial}-v1/
-  minigit-{lang}-{trial}-v2/
-logs/                # Claude JSON output logs
-  minigit-{lang}-{trial}-v1.json
-  minigit-{lang}-{trial}-v2.json
-```
+The `data` branch (orphan) contains generated artifacts and logs.
 
 ## How It Works
 
-1. Run `ruby benchmark.rb`
-2. For each language x trial:
-   - v1: Create `generated/minigit-{lang}-{trial}-v1/`, copy spec and tests, invoke `claude -p`
-   - v2: Copy v1 result to `minigit-{lang}-{trial}-v2/`, invoke `claude -p` to extend
+1. Run `bin/which-language run <codex> <problem>`
+2. For each language × trial:
+   - v1: Create working dir, copy problem assets, invoke the selected codex
+   - v2: Copy v1 result, invoke the codex to extend
 3. Run test scripts independently to verify
 4. Measure wall-clock time, LOC, token usage, and cost
-5. Run `ruby report.rb` to generate the report
-6. Run `python3 plot.py` to generate graphs
+5. Report and figures are generated automatically by the `run` subcommand
 
 ## Key Commands
 
 ```bash
-ruby benchmark.rb                                    # All languages x 3 trials
-ruby benchmark.rb --lang python --trials 1           # Single language test
-ruby benchmark.rb --trials 10 --start 11             # Trials 11-20
-ruby benchmark.rb --dry-run                           # Dry run
-ruby report.rb                                        # Generate report
-python3 plot.py                                       # Generate graphs
+bin/which-language run claude minigit                        # Full pipeline (default: all langs × 3 trials)
+bin/which-language benchmark claude minigit --lang python --trials 1  # Benchmark only
+bin/which-language report claude minigit                     # Generate report from existing results
+bin/which-language plot claude minigit                       # Generate figures from existing results
+bin/which-language benchmark claude minigit --dry-run        # Dry run (no API calls)
 ```
 
-## Supported Languages (LANGUAGES hash in benchmark.rb)
+Prefer `config/codexes.local.yml` for local secrets and enablement overrides.
+
+## Multi-Codex Architecture
+
+Each codex adapter implements:
+- `run_generation(prompt, dir:, log_path:)` — Generate code
+- `version` — Get codex version
+- `warmup(warmup_dir)` — Optional warmup
+- `parse_metrics(raw_output)` — Extract token/cost data
+
+To add a new codex:
+1. Create `lib/codexes/your_codex.rb` extending `BaseCodex`
+2. Add configuration to `config/codexes.yml`
+3. Run: `bin/which-language benchmark your_codex minigit --dry-run`
+
+See [AGENT.md](./AGENT.md) for the full integration checklist.
+
+## Codex Specifications
+
+### Implemented
+
+| Codex | Provider | Type | Context | Pricing (in/out per 1M) |
+|-------|----------|------|---------|------------------------|
+| **Claude Code** | Anthropic | CLI | 200K | ~$15/$75 |
+| **Gemini** | Google | API | 1M | $0.25/$1.50 |
+| **OpenAI** | OpenAI | API | 128K | $5/$15 (GPT-4o) |
+| **Groq** | Groq | API | 128K | varies by model |
+| **Aider** | Open Source | CLI | model-dependent | N/A (wraps other models) |
+
+### Benchmark Results (historical)
 
-rust, go, c, typescript, javascript, java, perl, python, python/mypy, ruby, ruby/steep, lua, scheme, ocaml, haskell
+**Claude Code** (Original Study):
+- Ruby: 73.1s, $0.36, 219 LOC, 40/40 pass
+- Python: 74.6s, $0.38, 235 LOC, 40/40 pass
+- JavaScript: 81.1s, $0.39, 248 LOC, 40/40 pass
 
-To add a language, add an entry to the `LANGUAGES` hash. Tests just call `./minigit`, so the implementation only needs to produce an executable with that name.
+**Gemini** (Flash-Lite):
+- JavaScript: 173.3s, $0.005, 125 LOC, 40/40 pass
+- Python: 167.9s, $0.005, 138 LOC, 39/40 pass
+- Ruby: 136.5s, $0.004, 154 LOC, 29/40 pass
+
+### Research Questions
+
+1. Which codex is **fastest** for different languages?
+2. Which is most **cost-effective**?
+3. Do **specialized models** (e.g., Qwen Coder) outperform general ones?
+4. How do **open source** models compare to proprietary ones?
+5. What's the overhead of **CLI tools** (Aider, Cline) vs direct API?
+
+See [plan.md](./plan.md) for planned codex integrations and current experiment status.
+
+## Supported Languages
+
+`rust`, `go`, `c`, `typescript`, `javascript`, `java`, `perl`, `python`, `python/mypy`, `ruby`, `ruby/steep`, `lua`, `scheme`, `ocaml`, `haskell`
+
+To add a language, add an entry to `config/languages.yml`.
+
+## Problem Model
+
+Problems are loaded from `problems/<problem>/problem.json` and assume a two-phase structure:
+
+- `v1_spec`, `v1_test`, `v1_prompt`
+- `v2_spec`, `v2_test`, `v2_prompt`
+
+Each run writes outputs under `artifacts/<codex>/<model>/<problem>/`, while dry-runs are isolated under `artifacts/<codex>/<model>/<problem>/dry-run/`.
 
 ## MiniGit Technical Notes
 
@@ -68,5 +140,7 @@ To add a language, add an entry to the `LANGUAGES` hash. Tests just call `./mini
 
 ## Notes
 
-- This is not a git repository for MiniGit itself; individual implementations under `generated/` may use `git init` as part of their build process
+- This is not a git repository for MiniGit itself; individual implementations under `artifacts/<codex>/<model>/<problem>/generated/` may use `git init` as part of their build process
 - The `data` branch is an orphan branch with no common history with `main`
+- Originally focused on Claude Code, now a **multi-codex benchmark platform**
+