[jehrr]

Problem

The canary's three-page walk through VRBO_PROXY ended with exit 6 on both 2026-10-20 and 2026-10-21. On 2026-10-21 the no-proxy first-page job ended the same way.

Run	Job	Page	Cards (of 50)	Stop reason
2026-10-20	pagination	2	19	next_page_throttled (the turn to page 3 got 2× HTTP 429 from /graphql)
2026-10-21	pagination	2 and 3	19 each	cards_missing (9.6 MB of the 10 MB cap)
2026-10-21	first-page	1	13	cards_missing (no proxy)

The cards missing from those pages were not throttled. Every snapshot that stopped short carries 6 data-stid="search-listing-placeholder" cards below the last real one, and no full page carries any. The site fills that lazy tail only when it scrolls into view. The scroll loop sets scrollTop = scrollHeight every round. After the first round that value no longer changes, so nothing moves and the grid waits for good. Count and height held still for six rounds, and the loop then gave up.

Measured on 2026-10-22 with a plain connection and no proxy:

• A load stalled at 13 cards with 6 placeholders. Count and height held still for eight rounds.
• One nudge took it to 50 of 50: the container went back to the top, then down a screen at a time.
• No /graphql request is behind those cards.

Two other findings from the same day:

• A throttled /graphql answer is 429 {"error":"Too Many Requests","message":"Provisioned request rate has been exceeded"} with no Retry-After header, so there is no header to honour.
• From this datacentre address, the page-turn query SearchResultsAndFilters stayed 429 when retried after 4, 8, 16 and 32 s. This PR therefore adds no request-level retry.

Resulting behaviour

• All three engines check for a stall: count and height still for three rounds with the site's placeholders showing. They then scroll the results container to the top and back down a screen at a time. This happens at most 3 times per page, and those rounds do not come out of the normal scroll budget.
• Each page's scroll entry in the sidecar gains nudges and placeholders_left. The warning now says which case it was: placeholders left after the nudges, or a short page with no placeholders.
• CI only: the pagination canary passes --block-media. A full page loads about 100 images (1.8 MB measured), and three full pages with images would not fit the 10 MB cap. image_url is not a column that job checks.

Evidence

• New sanitized fixture LISTING_LAZY_TAIL, cut by make_fixtures.build_lazy_tail from the 2026-10-21 first-page capture. It keeps 2 cards, the counter and the 6 placeholders. The kept rows are verified to parse identically to the untrimmed capture, and long hex runs are scrubbed.
• New offline group test_grid_lazy_tail:
  • the placeholder selector matches 6 placeholders in the stall and 0 in the four full-page fixtures;
  • the stall stays at 13 without a nudge and reaches 50 with one;
  • a nudge that never helps stops after 3;
  • a page with no placeholders is not nudged;
  • the nudge goes to the top and then down in steps no larger than one screen;
  • each engine's _scroll_the_grid reaches 50 with nudges: 1 through its own named operations.
• Planted-fault controls. Each one made the suite red on the named check, and the restored tree is green:
  • nudge never called;
  • nudge unbounded (9 nudges);
  • nudge skips the top;
  • wrong placeholder selector;
  • nudge on a page with no placeholders;
  • Selenium does not pass the nudge;
  • pyppeteer does not pass the nudge;
  • Playwright drops the trace.
• The suite (python3 smoke_test.py) passes in a core-only venv and in one venv each for playwright, selenium and pyppeteer, with only the other engines skipped. No browser was downloaded.
• One live run of the fixed Playwright engine, no proxy, 2026-10-22: 50 of 50 cards, complete. That load did not stall, so it shows the path still works. It is not a live nudge. The stalled case is covered by the probe described above.

Not changed

• No Retry-After handling, because the site sends none. No request-level retry of the page turn, because from a datacentre address the throttle outlasted 60 s of backoff.
• The pacing (--delay 15) and the proxy session. Through VRBO_PROXY the page turns arrived in 2–3 s on both days. The short pages were the lazy tail and do not point to the exit.
• The VRBO_PROXY secret has no sticky-session part in its login. Whether that matters for the intermittent page-turn 429 has not been measured.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8