Skip to content

Send the empty answer, and actually run the browser-compat tests - #10

Merged
JWriter20 merged 3 commits into
mainfrom
fix/empty-answer-never-submits
Aug 23, 2026
Merged

Send the empty answer, and actually run the browser-compat tests#10
JWriter20 merged 3 commits into
mainfrom
fix/empty-answer-never-submits

Conversation

@JWriter20

@JWriter20 JWriter20 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Two fixes uncovered while diagnosing a Tier 3 result that looked like a model regression and wasn't.

1. The empty answer was the one shape the driver could not send

The submit-control lookup sat INSIDE the loop over the model's actions; the decision to press it sat outside:

for (const action of actionList) {
  ...
  const lookup = frame ?? (slid ? null : scope);
  if (lookup) verifyButton = await this.getVerifyButton(lookup);
}
const shouldClickSubmit = !slid && (answered || !performedAction);
if (shouldClickSubmit && verifyButton) { ... }

A plan with no actions never enters that loop. shouldClickSubmit then computes true — through !performedAction, the branch that exists precisely for "we had nothing to do and want the round to advance" — and presses nothing, because the control it needs was never resolved. performedAction stays false and the caller aborts on "still detected but the solver performed no interactions".

Why it looked like a model regression

reCAPTCHA's 3x3 has a none_present variation: the prompt names a class, no tile contains it, and the control reads SKIP rather than VERIFY. Selecting nothing and pressing it is the correct answer. Tier 3 fixture seed 20260730 is exactly that — target traffic light, target_ids: [], submit_label: SKIP.

It surfaced when CaptchaKrakenFinetune fixed font resolution on the macOS Tier 3 runner. That box has no /usr/share/fonts, so every face missed and the prompt had been rendering in a fallback bitmap face reading "Selectall images with / traffic lights". After the fix it draws real reCAPTCHA chrome, with the target term bolded and a legible "If there are none, click skip."

This client runs at temperature: 0, so the model is a function of the picture: a correct picture invites a correct empty answer — and the empty answer is the one shape the driver drops.

Scope note, from re-running Tier 3 on identical code. recaptcha_grid_3x3 js went 2/3 → 1/3 in the run that prompted this investigation, and I initially read that as this bug biting. A re-run of the same commit put it back at 2/3, and 8 of 88 pairs flipped between those two identical runs, so that particular movement was within Tier 3's own noise and is not evidence for this fix. The defect stands on the code: seed 20260730 really is none_present with target_ids: [] and submit_label: SKIP, and an empty plan really does leave verifyButton null and abort. What is fixed here is a path that provably cannot submit, not a measured regression.

getVerifyButton was never at fault: 'Skip' has been in its list since the GeeTest OK fix (geetest-submit-button.test.ts pins the finder itself). The finder was simply never called.

Both ports had it identically; both are fixed, per CLAUDE.md 1c.

2. browser-compat tests errored because a build number moved

Those four reported as ERRORS, and the cause was not a missing browser. Playwright's Python bindings resolve one pinned Chromium build and refuse to launch anything else, so p.chromium.launch() with no path fails on a box that has Chromium — just not the pinned one. This box has four (chromium-1148/1208/1228/1234) and 1234 launches fine when passed as executable_path. playwright install fetching build N+1 does not make build N stop working; what looked like an availability check was a version check.

The fixture now tries the pinned build first, then every build actually installed, newest first — the same resolution CaptchaKrakenFinetune's tests/test_fixtures_are_solvable.py already uses. This file should not have been the odd one out.

Skipping stays reserved for a box with no browser at all, which is a real case: the package ships with no browser dependency and an end user is not required to have one. Skipping because a build number moved is not that case, and it left the one test that checks the compatibility claim against something real quietly doing nothing. A fake page happily keeps agreeing with a driver that no longer matches the library — which is the entire reason this file exists.

The JS twin needed no change. playwright, puppeteer and camoufox-js all resolve from js/node_modules, and tests 10–12 already run against real Playwright and real Puppeteer with 0 skipped. Only the Python bindings pin a build this box no longer has.

While confirming that: there is no browser dependency in this package, and it is verified rather than asserted. pyproject.toml lists pydantic, pillow, numpy, opencv-python-headless, requests, python-dotenv; the dev extra is pytest, mypy, ruff. js/package.json has dependencies: {} and peerDependencies: {}. Nothing in src/ imports a browser package in either port — every remaining mention is prose, and page_solver.py's docstring says so outright.

Tests

Structural, and deliberately so: what is wrong in (1) is where a call sits relative to a loop, and solveSingle / _execute_plan is ~200 lines wrapped around a screenshot, a planner round-trip and a live page — mocking that observes the nesting far less directly than reading it. The Python half asserts over a real AST; the JS half brace-scans, ignoring braces inside strings so an xpath template cannot shift the depth. Each also pins that the lookup still happens at all, so deleting the call does not pass by vacuum. Both confirmed RED first, naming the exact offending lines (2468 and 1030).

  • js 91/91
  • python 406 passed, 15 skipped, 0 errors (was 402 passed, 15 skipped, 4 errors) — the four browser-compat tests now RUN and pass against real Chromium

🤖 Generated with Claude Code

JWriter20 and others added 2 commits August 20, 2026 17:32
…loop

The lookup for the widget's own submit control sat INSIDE the loop over the
model's actions, while the decision to press it sat outside:

    for (const action of actionList) {
      ...
      const lookup = frame ?? (slid ? null : scope);
      if (lookup) verifyButton = await this.getVerifyButton(lookup);
    }
    const shouldClickSubmit = !slid && (answered || !performedAction);
    if (shouldClickSubmit && verifyButton) { ... }

A plan with NO actions never enters that loop. `shouldClickSubmit` then computes
TRUE — via `!performedAction`, the branch that exists precisely for "we had
nothing to do and want the round to advance" — and presses nothing, because the
control it needs was never resolved. `performedAction` stays false and the
caller aborts on "still detected but the solver performed no interactions".

So the one answer shape the driver could not send was the empty one.

WHERE IT BIT, AND WHY IT LOOKED LIKE A MODEL REGRESSION

reCAPTCHA's 3x3 has a `none_present` variation: the prompt names a class, no
tile contains it, and the control reads SKIP rather than VERIFY. Selecting
nothing and pressing it IS the correct answer. Tier 3 fixture seed 20260730 is
exactly that shape — target `traffic light`, `target_ids: []`,
`submit_label: SKIP`.

It surfaced when CaptchaKrakenFinetune fixed font resolution on the macOS Tier 3
runner. Before, that box had no `/usr/share/fonts`, every face missed, and the
prompt rendered in a fallback bitmap face reading "Selectall images with /
traffic lights". After, the widget draws real reCAPTCHA chrome with the target
term bolded and a legible "If there are none, click skip."

This client runs at `temperature: 0`, so the model is a function of the picture.
A correct picture got a correct EMPTY answer — and the empty answer was the one
the driver dropped. `recaptcha_grid_3x3` js went 2/3 -> 1/3 in that run and read
as the font fix causing a regression. It was this, uncovered rather than caused.

`getVerifyButton` was never at fault: 'Skip' has been in its list since the
GeeTest OK fix (js/src/geetest-submit-button.test.ts pins the finder itself).
The finder was simply never called.

Both ports had it, identically — `page_solver.py` nested its `_get_verify_button`
call the same way — and both are fixed here, per CLAUDE.md 1c.

Tests are structural, and deliberately so: what is wrong is where a call sits
relative to a loop, and `solveSingle` / `_execute_plan` is ~200 lines wrapped
around a screenshot, a planner round-trip and a live page. Mocking all of that
observes the nesting far less directly than reading it. The Python half asserts
over a real AST; the JS half brace-scans, ignoring braces inside strings so an
xpath template cannot shift the depth. Each also pins that the lookup still
happens at all, so deleting the call does not pass by vacuum.

js 91/91. python 402 passed, 15 skipped; the 4 test_browser_compat errors are a
missing Playwright browser binary and reproduce identically on origin/main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These four reported as ERRORS, not skips:

    ERROR tests/test_browser_compat.py::test_a_real_page_provides_every_member...
    4 errors

and the cause was not a missing browser. Playwright's Python bindings resolve
ONE pinned Chromium build and refuse to launch anything else, so
`p.chromium.launch()` with no path fails on a box that HAS Chromium — just not
the pinned one. This box has four (chromium-1148/1208/1228/1234) and 1234
launches fine when passed as `executable_path`. `playwright install` fetching
build N+1 does not make build N stop working; what looked like an availability
check was a version check.

So the fixture now tries the pinned build first — correct on a properly
provisioned box, and needs no path — then every build actually installed,
newest first. That is the same resolution CaptchaKrakenFinetune's
`tests/test_fixtures_are_solvable.py` already uses, and this file should not
have been the odd one out.

Skipping is reserved for a box with NO browser at all, which is a real case:
the package ships with no browser dependency and an end user is not required to
have one. Skipping because a build number moved is not that case, and it left
the ONE test that checks the compatibility claim against something real quietly
doing nothing. A fake page happily keeps agreeing with a driver that no longer
matches the library; that is the entire reason this file exists.

python 406 passed, 15 skipped, 0 errors — was 402 passed, 15 skipped, 4 errors.
The four now run and pass against real Chromium.

The JS twin needed no change: `playwright`, `puppeteer` and `camoufox-js` all
resolve from `js/node_modules`, and tests 10-12 already run against real
Playwright and real Puppeteer with 0 skipped. Only the Python bindings pin a
build this box no longer has.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JWriter20
JWriter20 force-pushed the fix/empty-answer-never-submits branch from 54ca324 to 08c1dce Compare August 20, 2026 23:42
@JWriter20 JWriter20 changed the title Send the empty answer, and skip browser tests without a browser Send the empty answer, and actually run the browser-compat tests Aug 20, 2026
@JWriter20

JWriter20 commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

✓ Tier 3 driver-gate — pass

Aggregate: 0.841 · 6/10 families passing · 14 pair(s) not solved

Port Pairs solved
js 0.864
python 0.818
Vendor family Solve rate
botdetect 1.000
geetest 0.714
hcaptcha 0.885
lemin 1.000
mtcaptcha 1.000
prosopo 1.000
recaptcha 0.500
tencent 0.000
yandex 1.000
yidun 1.000

JWriter20 added a commit that referenced this pull request Aug 23, 2026
…in half (#12)

`overallSolveTimeoutMs` / `overall_solve_timeout_ms` is 45s and it is sized for
ROUNDS — "a round costs ~4-7s, so six is the budget", which no-progress.test.ts
pins. Recording an animated challenge is not a round. It is a fixed extra stage
costing the burst, the slice, one MULTI-IMAGE inference (six keyframes, several
times a still's) and the wait for the widget to come back round to the frame the
model chose. Nothing in the 45s was ever set aside for it.

So every escalation to a recording ran the clock out, and reported a TIMEOUT —
a message about the model being slow, for a budget that had no room for what the
solver had just decided to do.

MEASURED, CaptchaKrakenFinetune Tier 3 run 32596340560 (2026-08-22):

  python  hcaptcha_fish_swim_different            45.7s, 48.5s, 49.2s
          hcaptcha_number_with_highest_value_video 49.4s, 50.5s
          hcaptcha_tile_flip_video                 52.6s, 52.1s
            -> "exceeded overall_solve_timeout_ms during recording the
                animated challenge"
  js      hcaptcha_click_image_by_traits           52.4s, 49.7s
          hcaptcha_connect_path                    50.2s
          hcaptcha_grid_3x3_property               49.7s, 49.4s
            -> "Captcha solve timed out after 45000ms (attempt 6/6)"

The same fixtures solve in 11-20s on the rounds where the still path happens to
answer them. It is the escalation that does not fit, not the puzzle.

## The escalation buys its own budget, once

`video_budget_ms()` = burst + keyframe wait + one multi-image inference; 18s on
the defaults, granted the first time a recording starts and never again in the
same solve. DERIVED, so a longer burst carries its own budget rather than
quietly reintroducing this. Worst case is 63s, still bounded.

An EXTENSION rather than a looser default: a solve that never escalates keeps
exactly the deadline the caller configured, and a caller who set
`video_solve_enabled=False` gets no grant at all — that switch already means
"fail fast rather than spend the recording time".

Both ports, same arithmetic. CLAUDE.md 1c: a fixture that passed on one port and
timed out on the other reads as a driver bug.

## A half-recorded burst is worthless

The python port checked the overall deadline PER FRAME, so a burst that started
at 41s died at frame 27 of 40 — throwing away both the frames and the ~3s spent
making them. The slicer reads a clip's temporal structure; stopping early does
not yield a shorter answer, it yields a recording that may not contain the
screen the answer is on.

Checked ONCE now, before the first frame, and if the budget cannot fit a whole
burst it says so and names the knobs. The burst is fixed-length, so it is not
one of the places that "can legitimately spin for a long time" that
`_check_deadline` exists for; a separate, looser bound catches a genuinely hung
screenshot and reports it as that.

9 new tests. Both regression tests fail on the old behaviour with the exact
production message. Pre-existing on main and untouched here: 4 errors in
test_browser_compat.py (fixed on #10's branch).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@JWriter20
JWriter20 merged commit 026c3d1 into main Aug 23, 2026
7 checks passed
@JWriter20
JWriter20 deleted the fix/empty-answer-never-submits branch August 23, 2026 05:00
JWriter20 added a commit that referenced this pull request Aug 24, 2026
dev was four commits behind main and the two had diverged, which is what made
every later merge conflict. Brings PRs #10-#13: the empty-answer submit and the
browser-compat run, camoufox-only naming, the animated-solve budget, and the
refreshed registry pages.

One conflict, in python/README.md: both sides added a DIFFERENT new section
immediately after the intro blockquote — this branch added "Watch it work" (the
captchakraken.com demo clips) and main added "What it solves" (the 44-type
vendor table). They answer different questions, so both are kept, clips first.
README.md and js/README.md took both automatically.

Verified after the merge: the three-vendor detection work is intact on both
ports (tencent/yandex/mtcaptcha selectors, VENDOR_URL_MARKERS,
TEXT_INPUT_VENDOR_SELECTORS all present), tsc builds, and 458 tests pass.
python/README.md still renders through readme_renderer with all three demo
images surviving sanitization.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant