docs: stage-safe live demo command (arithmetic-mc) instead of GPQA max_gen=3 (#111) - #157
Merged
Merged
Conversation
…earch path Issue #111: GPQA (198 questions, per-question provider calls) is minutes per generation and unreliable for a 3-4 minute stage demo. Use the already-bundled tiny arithmetic-mc task (5 self-contained questions, no dataset download) as the primary stage-safe live command with --max_gen 2, and demote GPQA to the warm-run / offline `sia web` replay research/credibility path. Updates HACKATHON_DEMO.md (run-of-show, pre-demo warm-up, backup Path A, one-slide summary, results narration, checklist) and HACKATHON_DECK.md (Step 3 reproducibility, flags table, task list, runtime guidance, checklist) so the live command, checklist, and runtime notes all agree. Runtime is marked "to be measured on the demo machine" rather than fabricating a number. Docs-only change; arithmetic-mc is already in BUNDLED_TASKS in both src/layout.rs and sia/layout.py. `cargo build` confirmed clean. https://claude.ai/code/session_01WX3ZN9azxr1fyWuZAaRJ7Y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #111.
The live run-of-show used
--task gpqa --max_gen 3, but GPQA (198 questions, per-question provider calls) is minutes/generation — not a reliable 3–4 min stage demo. Docs-only fix (the tinyarithmetic-mctask — 5 questions — is already bundled, so no new task/layout/parity changes):docs/HACKATHON_DEMO.md: primary live command is now--task arithmetic-mc --max_gen 2; GPQA demoted to a pre-warmed/offlinesia webreplay ("credibility" path, do not run live). Run-of-show, backup paths, results narration, and the pre-demo checklist all updated; checklist has a blank to record measured wall-clock on the demo machine.docs/HACKATHON_DECK.md: reproducibility step + runtime guidance contrast GPQA-minutes vs arithmetic-mc-fast; task lists corrected to five bundled tasks.Per the issue, no API key here, so runtimes are marked "to be measured on the demo machine" rather than fabricated.
cargo buildclean (docs-only).https://claude.ai/code/session_01WX3ZN9azxr1fyWuZAaRJ7Y
Generated by Claude Code