Fill results pages that stop at the grid's loading placeholders

.github/workflows/canary.yml

@@ -158,14 +158,18 @@ jobs:
         # Three pages, not two: one turn only proves the button was found
         # once; the second turn is where a cursor that does not advance shows
         # up. --delay 15 because the turn's /graphql POST is rate limited
-        # separately from the HTML.
+        # separately from the HTML. --block-media because a full page loads
+        # about 100 images (1.8 MB measured 2026-10-22, no proxy): the
+        # 2026-10-21 walk used 9.6 MB of the 10 MB cap with two of its three
+        # pages only 19 cards deep, so three full pages with images would
+        # not fit. image_url is not one of the columns this job checks.
         run: |
           set +e
           start=$(date -u -d '+45 days' +%F)
           end=$(date -u -d '+47 days' +%F)
           url="https://www.vrbo.com/search?destination=Orlando%2C%20Florida%2C%20United%20States%20of%20America&regionId=2693&sort=RECOMMENDED&startDate=${start}&endDate=${end}&adults=2"
           xvfb-run -a python3 playwright_scraper.py --url "$url" \
-            --pages 3 --delay 15 --retries 4 --proxy-budget-mb 10 \
+            --pages 3 --delay 15 --retries 4 --proxy-budget-mb 10 --block-media \
             --out canary_walk --dump-html canary_walk.html
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
 

CHANGELOG.md

@@ -75,6 +75,27 @@ that is a fix, and the release notes lead with it.
 
 ### Fixed
 
+- A results page that stops filling partway now gets the rest of its cards
+  in all three engines. Below the last card the grid keeps loading
+  placeholders (`data-stid="search-listing-placeholder"`). It fills them only
+  when they scroll into view, and scrolling to the bottom again moves
+  nothing. The 2026-10-20 and 10-21 canaries stopped this way: 6 placeholders
+  under 13 cards on page 1 with no proxy, and under 19 cards on pages 2 and 3
+  through a residential exit. Each run ended as `cards_missing`, exit 6.
+  - When count and height hold still for three rounds with placeholders
+    showing, the scraper scrolls the container to the top and back down a
+    screen at a time. It does this at most three times per page.
+  - Measured 2026-10-22 without a proxy: a 13-card stall held for eight
+    rounds, and one nudge took it to 50 of 50.
+  - Each page's `scroll` entry in the sidecar now records `nudges` and
+    `placeholders_left`, and the warning says which case it was.
+  - New sanitized fixture `LISTING_LAZY_TAIL`, cut from the stalled canary
+    capture. There are offline checks for the stall, the bound, short pages,
+    the walk and each engine's wiring.
+- CI only: the canary's three-page walk through `VRBO_PROXY` passes
+  `--block-media`. A full page loads about 100 images (1.8 MB measured), and
+  the 2026-10-21 walk had already used 9.6 MB of its 10 MB cap with two short
+  pages.
 - The pyppeteer and Selenium proxy meter counts only `http://` / `https://` requests and responses that came over the network. Chrome's own loads (the `chrome://` new-tab page, `data:`, `blob:`), cache hits (`requestServedFromCache`, `fromDiskCache`, `fromPrefetchCache`, `fromServiceWorker`) and Chrome's error page count nothing. A sibling repo's Selenium run had counted about 2.8 MB of new-tab scripts while its proxy carried about 6 KB. The pyppeteer block sent once the budget is spent now reaches the page's own event loop even when the budget runs out on another thread.
 
 ## [0.3.0] — 2026-10-15

TROUBLESHOOTING.md

@@ -76,12 +76,21 @@ parse got 40, so ten cards never loaded.
 
 Almost always the scroll did not finish. First paint on this site is 3 to 18
 cards and the rest arrive by scrolling an **inner** container
-(`.scrollable-result-section`) — the page body never scrolls at all. If the
-batch behind a scroll is throttled, the grid stops growing early.
-
-Same remedies as above: slow down, or use a proxy pool. Re-run with
-`--dump-html` and open the snapshot to confirm the cards really were absent
-rather than unparsed.
+(`.scrollable-result-section`) — the page body never scrolls at all.
+
+Below the last card the grid keeps a block of loading placeholders
+(`data-stid="search-listing-placeholder"`) and fills it only when that block
+comes **into** view. Scrolling to the bottom again and again does not do that,
+because nothing moves after the first time. When the grid stops with those
+placeholders still showing, the scraper scrolls the container back to the
+top and down again a screen at a time, up to three times. The sidecar's
+`scroll` entry for each page records `nudges` and `placeholders_left`. If
+`placeholders_left` is above 0, the grid never filled in even after the
+nudges. If it is 0, the site served a short page.
+
+If it persists: slow down, or use a proxy pool. Re-run with `--dump-html`
+and open the snapshot to confirm the cards really were absent rather than
+unparsed.
 
 ---
 

make_fixtures.py

@@ -86,6 +86,10 @@ def _unpriced(card) -> bool:
 # exit 4, "the search matched nothing".
 SITE_ERROR_SOURCE = ("vrbo_site_error_2026-10-07.html",
                      "https://www.vrbo.com/search?destination=Orlando%2C%20Florida%2C%20United%20States%20of%20America&regionId=2693&sort=RECOMMENDED&startDate=2026-11-14&endDate=2026-11-16&adults=2")
+# A grid the scroll loop left at 13 of 50 cards with the site's own loading
+# placeholders under them (canary, 2026-10-21, no proxy).
+LAZY_TAIL_SOURCE = ("vrbo_search_lazy_tail_stall_2026-10-21.html",
+                    "https://www.vrbo.com/search?destination=Orlando%2C%20Florida%2C%20United%20States%20of%20America&regionId=2693&sort=RECOMMENDED&startDate=2026-12-05&endDate=2026-12-07&adults=2")
 
 # A card that keeps the SLUG route must survive into LISTING_DE: it is the
 # fifth URL shape and only the local brands use it, so a fixture without one
@@ -270,6 +274,52 @@ def build_site_error(filename, url):
     return trimmed
 
 
+def build_lazy_tail(filename, url, keep=2):
+    """A grid that stopped short: `keep` real cards and the lazy tail.
+
+    Cut from a page that the scroll loop left at 13 of 50 cards with the
+    site's own placeholders still under them. Keeps the placeholders exactly
+    as served (their parent block) so the selector is tested against the
+    real markup, and the first `keep` cards so the page still parses.
+    """
+    full = (CAPTURES / filename).read_text(encoding="utf-8")
+    soup = BeautifulSoup(full, "html.parser")
+    placeholders = soup.select(P.SELECTORS["grid_placeholder"])
+    if not placeholders:
+        raise SystemExit(f"{filename}: no grid placeholder — is this the stalled capture?")
+    tail = placeholders[0].parent
+    cards = soup.select(P.SELECTORS["item_card"])[:keep]
+    header = soup.select_one(P.RESULTS_HEADER_SELECTOR)
+    pagination = soup.select_one(P.PAGINATION_SELECTOR)
+    block = ((str(header) if header is not None else "")
+             + '<div data-stid="property-listing-results">'
+             + "".join(str(c) for c in cards) + str(tail) + "</div>"
+             + (str(pagination) if pagination is not None else ""))
+    trimmed = scrub(_head(soup, full) + block + "</body></html>")
+    small = BeautifulSoup(trimmed, "html.parser")
+    if len(small.select(P.SELECTORS["grid_placeholder"])) != len(placeholders):
+        raise SystemExit("LISTING_LAZY_TAIL: trimming changed the placeholder count")
+    want = {r.sku: asdict(r) for r in P.parse_products(full, url, page=1)}
+    got = {r.sku: asdict(r) for r in P.parse_products(trimmed, url, page=1)}
+    if len(got) != keep or set(got) - set(want):
+        raise SystemExit(f"LISTING_LAZY_TAIL: kept {len(got)} rows, not the {keep} cut")
+    for sku, row in got.items():
+        for column, value in row.items():
+            expected = want[sku][column]
+            if isinstance(expected, str):
+                expected = scrub(expected)  # the rate-plan id is a long hex run
+            if column not in ("scraped_at", "position") and expected != value:
+                raise SystemExit(f"LISTING_LAZY_TAIL: trimming changed {column!r} on {sku}")
+    if P.detect_page_state(trimmed, 200, url) != "content":
+        raise SystemExit("LISTING_LAZY_TAIL: trimmed fixture does not classify as content")
+    leftovers = re.findall(r"\b[0-9a-fA-F]{24,}\b|duaid", trimmed)
+    if leftovers:
+        raise SystemExit(f"LISTING_LAZY_TAIL: session material left in: {leftovers[:3]}")
+    print(f"  LISTING_LAZY_TAIL: {len(full)} -> {len(trimmed)} bytes, "
+          f"{len(got)} rows and {len(placeholders)} placeholders")
+    return trimmed
+
+
 def build_property(filename, url):
     full = (CAPTURES / filename).read_text(encoding="utf-8")
     soup = BeautifulSoup(full, "html.parser")
@@ -336,6 +386,8 @@ def main():
     urls["PROPERTY_US"] = PROPERTY_SOURCE[1]
     fixtures["SITE_ERROR"] = build_site_error(*SITE_ERROR_SOURCE)
     urls["SITE_ERROR"] = SITE_ERROR_SOURCE[1]
+    fixtures["LISTING_LAZY_TAIL"] = build_lazy_tail(*LAZY_TAIL_SOURCE)
+    urls["LISTING_LAZY_TAIL"] = LAZY_TAIL_SOURCE[1]
     # Chromium's own answer when a proxy is dead. 39 bytes and no markup at
     # all — Playwright does not expose the interstitial, so this is what a
     # caller actually gets. It must classify as blocked, which is the

page_flow.py

@@ -162,24 +162,69 @@ def wait_for_count(count: Callable[[str], int], sleep: Callable[[int], None],
 SCROLL_MAX_ROUNDS = 20
 # How many stable rounds to tolerate while the page's own counter says there
 # are still cards to come. Earned on the first live run of this engine: page
-# 2 settled at 19 cards against a counter that said 50, because the next
-# batch was in flight behind a throttled `/graphql` POST and three quiet
-# rounds went by while it was. A pause is not an ending when the site has
-# told us how many there are — so the stable-round heuristic is the
-# termination condition only for a page that published no counter, and below
-# a known target the loop is twice as patient before giving up.
+# 2 settled at 19 cards against a counter that said 50. That was read at the
+# time as a batch in flight behind a throttled `/graphql` POST. The 2026-10-22
+# measurement found the lazy tail instead (see PLACEHOLDER_SELECTOR): no
+# request is behind those cards. A pause is still not an ending when the site
+# has told us how many there are, so below a known target the loop is twice
+# as patient before giving up.
 SCROLL_STABLE_ROUNDS_BELOW_TARGET = 6
 # The next batch takes longer to arrive than a single pause.
 SCROLL_PAUSE_MS = 1_600
 
+# The grid's lazy tail. Below the last rendered card the site keeps a block
+# of placeholder cards and fills it only when it scrolls INTO view. Scrolling
+# the container to its bottom puts it in view once; after that every further
+# "scroll to bottom" sets the same scrollTop, nothing moves, and the grid
+# waits for good. Measured: the 2026-10-20 and 10-21 canaries stopped with 6
+# placeholders under 13 cards (page 1, no proxy) and 19 cards (pages 2 and 3,
+# residential exit) while count and height held still; on 2026-10-22, from a
+# plain connection, the same 13-card stall held for eight rounds and ONE
+# nudge (back to the top, then down a screen at a time) took it to 50 of 50.
+# No /graphql call is behind those cards, so waiting longer cannot help.
+PLACEHOLDER_SELECTOR = SELECTORS["grid_placeholder"]
+# Bounded: a nudge that does not help three times will not on the fourth.
+SCROLL_MAX_NUDGES = 3
+NUDGE_TOP_PAUSE_MS = 800
+NUDGE_STEP_PAUSE_MS = 300
+NUDGE_STEP_SHARE = 0.8        # of the container's visible height
+NUDGE_MAX_STEPS = 40          # a 12,000px grid in ~550px steps is ~22
+
+
+def nudge_grid(geometry: Callable[[], Optional[tuple]],
+               set_top: Callable[[int], None],
+               sleep: Callable[[int], None]) -> None:
+    """Move the lazy tail out of view and walk it back in, a screen at a time.
+
+    `geometry` returns the container's (scrollTop, scrollHeight,
+    clientHeight), or None when there is no container; `set_top` sets its
+    scrollTop. Both are named driver operations, so this reads the same in
+    every engine.
+    """
+    if not geometry():
+        return
+    set_top(0)
+    sleep(NUDGE_TOP_PAUSE_MS)
+    for _ in range(NUDGE_MAX_STEPS):
+        g = geometry()
+        if not g:
+            return
+        top, height, client = (int(v or 0) for v in g)
+        if top + client >= height - 2:
+            return
+        set_top(min(height, top + max(1, int(client * NUDGE_STEP_SHARE))))
+        sleep(NUDGE_STEP_PAUSE_MS)
+
 
 def scroll_until_settled(count: Callable[[str], int],
                          scroll_results: Callable[[], None],
                          results_height: Callable[[], Optional[int]],
                          sleep: Callable[[int], None],
                          selector: str = READY_SELECTOR_LISTING,
                          target: Optional[int] = None,
-                         max_rounds: int = SCROLL_MAX_ROUNDS) -> int:
+                         max_rounds: int = SCROLL_MAX_ROUNDS,
+                         nudge: Optional[Callable[[], None]] = None,
+                         trace: Optional[dict] = None) -> int:
     """Scroll the results container until the grid stops growing.
 
     Requires the card count AND the container height to hold still for
@@ -192,19 +237,40 @@ def scroll_until_settled(count: Callable[[str], int],
     NOT reaching it does not end anything early, because the gap is reported
     rather than chased (the rank-gap arithmetic).
 
+    `nudge`, when given, is called (at most `SCROLL_MAX_NUDGES` times) when
+    the grid has held still for `SCROLL_STABLE_ROUNDS` rounds with the site's
+    own placeholders still on the page. See PLACEHOLDER_SELECTOR. Its rounds
+    do not come out of the ordinary budget.
+
+    `trace`, when given, receives `nudges` and `placeholders_left`.
+
     Returns the final card count.
     """
     seen = count(selector)
     height = results_height()
-    stable = 0
-    for _ in range(max_rounds):
+    stable = nudges = rounds = 0
+    budget = max_rounds + (SCROLL_MAX_NUDGES * SCROLL_STABLE_ROUNDS if nudge else 0)
+    while rounds < budget:
         if target is not None and seen >= target:
             break
         scroll_results()
         sleep(SCROLL_PAUSE_MS)
+        rounds += 1
         now, now_height = count(selector), results_height()
         if now == seen and now_height == height:
             stable += 1
+            if (nudge is not None and nudges < SCROLL_MAX_NUDGES
+                    and stable >= SCROLL_STABLE_ROUNDS
+                    and count(PLACEHOLDER_SELECTOR) > 0):
+                nudges += 1
+                logger.info("The grid stopped at %d card(s) with the site's "
+                            "loading placeholders still showing; scrolling it "
+                            "back through (nudge %d/%d).", now, nudges,
+                            SCROLL_MAX_NUDGES)
+                nudge()
+                stable = 0
+                seen, height = count(selector), results_height()
+                continue
             # More patience while the site's own counter says cards are
             # still owed. See SCROLL_STABLE_ROUNDS_BELOW_TARGET.
             allowed = (SCROLL_STABLE_ROUNDS_BELOW_TARGET
@@ -215,9 +281,26 @@ def scroll_until_settled(count: Callable[[str], int],
         else:
             stable = 0
         seen, height = now, now_height
+    if trace is not None:
+        trace["nudges"] = nudges
+        trace["placeholders_left"] = count(PLACEHOLDER_SELECTOR)
     return seen
 
 
+def unsettled_reason(trace: dict) -> str:
+    """Why a page stopped short of its counter, in words for the log."""
+    left = trace.get("placeholders_left")
+    if left:
+        return (f"the site's own loading placeholders ({left}) were still on "
+                f"the page after {trace.get('nudges', 0)} nudge(s), so the "
+                f"grid never filled in")
+    if left == 0:
+        return ("no loading placeholder was left on the page, so the site "
+                "served a short page or the grid was still filling")
+    return ("the grid was still filling when the scroll budget ran out, or "
+            "the site served a short page")
+
+
 # ---------------------------------------------------------------------------
 # Classification and the policy that follows from it
 # ---------------------------------------------------------------------------

playwright_scraper.py

@@ -251,6 +251,31 @@ def _results_height(page) -> Optional[int]:
         return None
 
 
+def _results_geometry(page) -> Optional[tuple]:
+    """(scrollTop, scrollHeight, clientHeight) of the results container."""
+    try:
+        g = page.evaluate(
+            """(selector) => {
+                const el = document.querySelector(selector);
+                return el ? [el.scrollTop, el.scrollHeight, el.clientHeight] : null;
+            }""", SELECTORS["scroll_container"])
+        return tuple(g) if g else None
+    except (PWError, PWTimeout):
+        return None
+
+
+def _set_results_top(page, top: int) -> None:
+    """Set the results container's scrollTop."""
+    try:
+        page.evaluate(
+            """([selector, top]) => {
+                const el = document.querySelector(selector);
+                if (el) { el.scrollTop = top; }
+            }""", [SELECTORS["scroll_container"], top])
+    except (PWError, PWTimeout):
+        pass
+
+
 # The data behind a page turn comes over POST /graphql, and that endpoint is
 # rate limited SEPARATELY from the HTML: measured on this site, the search
 # page kept answering HTTP 200 while every /graphql POST behind a next-press
@@ -673,25 +698,33 @@ def _scroll_the_grid(session, args, html: str, page_num: int) -> dict:
     """
     target = page_flow.expected_cards(html)
     before = _count(session.page, page_flow.READY_SELECTOR_LISTING)
+    trace: dict = {}
     reached = page_flow.scroll_until_settled(
         lambda sel: _count(session.page, sel),
         lambda: _scroll_results(session.page),
         lambda: _results_height(session.page),
         session.page.wait_for_timeout,
         selector=page_flow.READY_SELECTOR_LISTING,
-        target=target)
+        target=target,
+        nudge=lambda: page_flow.nudge_grid(
+            lambda: _results_geometry(session.page),
+            lambda top: _set_results_top(session.page, top),
+            session.page.wait_for_timeout),
+        trace=trace)
     settled = target is None or reached >= target
     logger.info("Scrolled page %d: %d card(s) at first paint, %d after "
-                "scrolling%s.", page_num, before, reached,
-                f" (the page says it holds {target})" if target else "")
+                "scrolling%s%s.", page_num, before, reached,
+                f" (the page says it holds {target})" if target else "",
+                f", {trace['nudges']} nudge(s)" if trace.get("nudges") else "")
     if not settled:
         logger.warning(
             "Page %d settled at %d card(s) but its own counter says it holds "
-            "%d. Those %d are missing from this run — the grid was still "
-            "filling when the scroll budget ran out, or the site served a "
-            "short page.", page_num, reached, target, target - reached)
+            "%d. Those %d are missing from this run — %s.",
+            page_num, reached, target, target - reached,
+            page_flow.unsettled_reason(trace))
     return {"first_paint": before, "reached": reached, "target": target,
-            "settled": settled}
+            "settled": settled, "nudges": trace.get("nudges", 0),
+            "placeholders_left": trace.get("placeholders_left")}
 
 
 def _fetch_one_page(session, args, pool, page_num: int, url: Optional[str]) -> PageOutcome: