Send the empty answer, and actually run the browser-compat tests - #10
Merged
Conversation
…loop
The lookup for the widget's own submit control sat INSIDE the loop over the
model's actions, while the decision to press it sat outside:
for (const action of actionList) {
...
const lookup = frame ?? (slid ? null : scope);
if (lookup) verifyButton = await this.getVerifyButton(lookup);
}
const shouldClickSubmit = !slid && (answered || !performedAction);
if (shouldClickSubmit && verifyButton) { ... }
A plan with NO actions never enters that loop. `shouldClickSubmit` then computes
TRUE — via `!performedAction`, the branch that exists precisely for "we had
nothing to do and want the round to advance" — and presses nothing, because the
control it needs was never resolved. `performedAction` stays false and the
caller aborts on "still detected but the solver performed no interactions".
So the one answer shape the driver could not send was the empty one.
WHERE IT BIT, AND WHY IT LOOKED LIKE A MODEL REGRESSION
reCAPTCHA's 3x3 has a `none_present` variation: the prompt names a class, no
tile contains it, and the control reads SKIP rather than VERIFY. Selecting
nothing and pressing it IS the correct answer. Tier 3 fixture seed 20260730 is
exactly that shape — target `traffic light`, `target_ids: []`,
`submit_label: SKIP`.
It surfaced when CaptchaKrakenFinetune fixed font resolution on the macOS Tier 3
runner. Before, that box had no `/usr/share/fonts`, every face missed, and the
prompt rendered in a fallback bitmap face reading "Selectall images with /
traffic lights". After, the widget draws real reCAPTCHA chrome with the target
term bolded and a legible "If there are none, click skip."
This client runs at `temperature: 0`, so the model is a function of the picture.
A correct picture got a correct EMPTY answer — and the empty answer was the one
the driver dropped. `recaptcha_grid_3x3` js went 2/3 -> 1/3 in that run and read
as the font fix causing a regression. It was this, uncovered rather than caused.
`getVerifyButton` was never at fault: 'Skip' has been in its list since the
GeeTest OK fix (js/src/geetest-submit-button.test.ts pins the finder itself).
The finder was simply never called.
Both ports had it, identically — `page_solver.py` nested its `_get_verify_button`
call the same way — and both are fixed here, per CLAUDE.md 1c.
Tests are structural, and deliberately so: what is wrong is where a call sits
relative to a loop, and `solveSingle` / `_execute_plan` is ~200 lines wrapped
around a screenshot, a planner round-trip and a live page. Mocking all of that
observes the nesting far less directly than reading it. The Python half asserts
over a real AST; the JS half brace-scans, ignoring braces inside strings so an
xpath template cannot shift the depth. Each also pins that the lookup still
happens at all, so deleting the call does not pass by vacuum.
js 91/91. python 402 passed, 15 skipped; the 4 test_browser_compat errors are a
missing Playwright browser binary and reproduce identically on origin/main.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These four reported as ERRORS, not skips:
ERROR tests/test_browser_compat.py::test_a_real_page_provides_every_member...
4 errors
and the cause was not a missing browser. Playwright's Python bindings resolve
ONE pinned Chromium build and refuse to launch anything else, so
`p.chromium.launch()` with no path fails on a box that HAS Chromium — just not
the pinned one. This box has four (chromium-1148/1208/1228/1234) and 1234
launches fine when passed as `executable_path`. `playwright install` fetching
build N+1 does not make build N stop working; what looked like an availability
check was a version check.
So the fixture now tries the pinned build first — correct on a properly
provisioned box, and needs no path — then every build actually installed,
newest first. That is the same resolution CaptchaKrakenFinetune's
`tests/test_fixtures_are_solvable.py` already uses, and this file should not
have been the odd one out.
Skipping is reserved for a box with NO browser at all, which is a real case:
the package ships with no browser dependency and an end user is not required to
have one. Skipping because a build number moved is not that case, and it left
the ONE test that checks the compatibility claim against something real quietly
doing nothing. A fake page happily keeps agreeing with a driver that no longer
matches the library; that is the entire reason this file exists.
python 406 passed, 15 skipped, 0 errors — was 402 passed, 15 skipped, 4 errors.
The four now run and pass against real Chromium.
The JS twin needed no change: `playwright`, `puppeteer` and `camoufox-js` all
resolve from `js/node_modules`, and tests 10-12 already run against real
Playwright and real Puppeteer with 0 skipped. Only the Python bindings pin a
build this box no longer has.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JWriter20
force-pushed
the
fix/empty-answer-never-submits
branch
from
August 20, 2026 23:42
54ca324 to
08c1dce
Compare
Owner
Author
✓ Tier 3 driver-gate — passAggregate: 0.841 · 6/10 families passing · 14 pair(s) not solved
|
JWriter20
added a commit
that referenced
this pull request
Aug 23, 2026
…in half (#12) `overallSolveTimeoutMs` / `overall_solve_timeout_ms` is 45s and it is sized for ROUNDS — "a round costs ~4-7s, so six is the budget", which no-progress.test.ts pins. Recording an animated challenge is not a round. It is a fixed extra stage costing the burst, the slice, one MULTI-IMAGE inference (six keyframes, several times a still's) and the wait for the widget to come back round to the frame the model chose. Nothing in the 45s was ever set aside for it. So every escalation to a recording ran the clock out, and reported a TIMEOUT — a message about the model being slow, for a budget that had no room for what the solver had just decided to do. MEASURED, CaptchaKrakenFinetune Tier 3 run 32596340560 (2026-08-22): python hcaptcha_fish_swim_different 45.7s, 48.5s, 49.2s hcaptcha_number_with_highest_value_video 49.4s, 50.5s hcaptcha_tile_flip_video 52.6s, 52.1s -> "exceeded overall_solve_timeout_ms during recording the animated challenge" js hcaptcha_click_image_by_traits 52.4s, 49.7s hcaptcha_connect_path 50.2s hcaptcha_grid_3x3_property 49.7s, 49.4s -> "Captcha solve timed out after 45000ms (attempt 6/6)" The same fixtures solve in 11-20s on the rounds where the still path happens to answer them. It is the escalation that does not fit, not the puzzle. ## The escalation buys its own budget, once `video_budget_ms()` = burst + keyframe wait + one multi-image inference; 18s on the defaults, granted the first time a recording starts and never again in the same solve. DERIVED, so a longer burst carries its own budget rather than quietly reintroducing this. Worst case is 63s, still bounded. An EXTENSION rather than a looser default: a solve that never escalates keeps exactly the deadline the caller configured, and a caller who set `video_solve_enabled=False` gets no grant at all — that switch already means "fail fast rather than spend the recording time". Both ports, same arithmetic. CLAUDE.md 1c: a fixture that passed on one port and timed out on the other reads as a driver bug. ## A half-recorded burst is worthless The python port checked the overall deadline PER FRAME, so a burst that started at 41s died at frame 27 of 40 — throwing away both the frames and the ~3s spent making them. The slicer reads a clip's temporal structure; stopping early does not yield a shorter answer, it yields a recording that may not contain the screen the answer is on. Checked ONCE now, before the first frame, and if the budget cannot fit a whole burst it says so and names the knobs. The burst is fixed-length, so it is not one of the places that "can legitimately spin for a long time" that `_check_deadline` exists for; a separate, looser bound catches a genuinely hung screenshot and reports it as that. 9 new tests. Both regression tests fail on the old behaviour with the exact production message. Pre-existing on main and untouched here: 4 errors in test_browser_compat.py (fixed on #10's branch). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
JWriter20
added a commit
that referenced
this pull request
Aug 24, 2026
dev was four commits behind main and the two had diverged, which is what made every later merge conflict. Brings PRs #10-#13: the empty-answer submit and the browser-compat run, camoufox-only naming, the animated-solve budget, and the refreshed registry pages. One conflict, in python/README.md: both sides added a DIFFERENT new section immediately after the intro blockquote — this branch added "Watch it work" (the captchakraken.com demo clips) and main added "What it solves" (the 44-type vendor table). They answer different questions, so both are kept, clips first. README.md and js/README.md took both automatically. Verified after the merge: the three-vendor detection work is intact on both ports (tencent/yandex/mtcaptcha selectors, VENDOR_URL_MARKERS, TEXT_INPUT_VENDOR_SELECTORS all present), tsc builds, and 458 tests pass. python/README.md still renders through readme_renderer with all three demo images surviving sanitization. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes uncovered while diagnosing a Tier 3 result that looked like a model regression and wasn't.
1. The empty answer was the one shape the driver could not send
The submit-control lookup sat INSIDE the loop over the model's actions; the decision to press it sat outside:
A plan with no actions never enters that loop.
shouldClickSubmitthen computes true — through!performedAction, the branch that exists precisely for "we had nothing to do and want the round to advance" — and presses nothing, because the control it needs was never resolved.performedActionstays false and the caller aborts on"still detected but the solver performed no interactions".Why it looked like a model regression
reCAPTCHA's 3x3 has a
none_presentvariation: the prompt names a class, no tile contains it, and the control reads SKIP rather than VERIFY. Selecting nothing and pressing it is the correct answer. Tier 3 fixture seed 20260730 is exactly that — targettraffic light,target_ids: [],submit_label: SKIP.It surfaced when CaptchaKrakenFinetune fixed font resolution on the macOS Tier 3 runner. That box has no
/usr/share/fonts, so every face missed and the prompt had been rendering in a fallback bitmap face reading "Selectall images with / traffic lights". After the fix it draws real reCAPTCHA chrome, with the target term bolded and a legible "If there are none, click skip."This client runs at
temperature: 0, so the model is a function of the picture: a correct picture invites a correct empty answer — and the empty answer is the one shape the driver drops.Scope note, from re-running Tier 3 on identical code.
recaptcha_grid_3x3js went 2/3 → 1/3 in the run that prompted this investigation, and I initially read that as this bug biting. A re-run of the same commit put it back at 2/3, and 8 of 88 pairs flipped between those two identical runs, so that particular movement was within Tier 3's own noise and is not evidence for this fix. The defect stands on the code: seed 20260730 really isnone_presentwithtarget_ids: []andsubmit_label: SKIP, and an empty plan really does leaveverifyButtonnull and abort. What is fixed here is a path that provably cannot submit, not a measured regression.getVerifyButtonwas never at fault:'Skip'has been in its list since the GeeTest OK fix (geetest-submit-button.test.tspins the finder itself). The finder was simply never called.Both ports had it identically; both are fixed, per CLAUDE.md 1c.
2. browser-compat tests errored because a build number moved
Those four reported as ERRORS, and the cause was not a missing browser. Playwright's Python bindings resolve one pinned Chromium build and refuse to launch anything else, so
p.chromium.launch()with no path fails on a box that has Chromium — just not the pinned one. This box has four (chromium-1148/1208/1228/1234) and 1234 launches fine when passed asexecutable_path.playwright installfetching build N+1 does not make build N stop working; what looked like an availability check was a version check.The fixture now tries the pinned build first, then every build actually installed, newest first — the same resolution CaptchaKrakenFinetune's
tests/test_fixtures_are_solvable.pyalready uses. This file should not have been the odd one out.Skipping stays reserved for a box with no browser at all, which is a real case: the package ships with no browser dependency and an end user is not required to have one. Skipping because a build number moved is not that case, and it left the one test that checks the compatibility claim against something real quietly doing nothing. A fake page happily keeps agreeing with a driver that no longer matches the library — which is the entire reason this file exists.
The JS twin needed no change.
playwright,puppeteerandcamoufox-jsall resolve fromjs/node_modules, and tests 10–12 already run against real Playwright and real Puppeteer with 0 skipped. Only the Python bindings pin a build this box no longer has.While confirming that: there is no browser dependency in this package, and it is verified rather than asserted.
pyproject.tomllists pydantic, pillow, numpy, opencv-python-headless, requests, python-dotenv; thedevextra is pytest, mypy, ruff.js/package.jsonhasdependencies: {}andpeerDependencies: {}. Nothing insrc/imports a browser package in either port — every remaining mention is prose, andpage_solver.py's docstring says so outright.Tests
Structural, and deliberately so: what is wrong in (1) is where a call sits relative to a loop, and
solveSingle/_execute_planis ~200 lines wrapped around a screenshot, a planner round-trip and a live page — mocking that observes the nesting far less directly than reading it. The Python half asserts over a real AST; the JS half brace-scans, ignoring braces inside strings so an xpath template cannot shift the depth. Each also pins that the lookup still happens at all, so deleting the call does not pass by vacuum. Both confirmed RED first, naming the exact offending lines (2468 and 1030).🤖 Generated with Claude Code