fix: scroll element into view before click, hover, tap, and drag#1073
Open
jin-2-kakaoent wants to merge 7 commits intovercel-labs:mainfrom
Open
fix: scroll element into view before click, hover, tap, and drag#1073jin-2-kakaoent wants to merge 7 commits intovercel-labs:mainfrom
jin-2-kakaoent wants to merge 7 commits intovercel-labs:mainfrom
Conversation
Clicking off-screen elements silently dispatched mouse events to coordinates outside the viewport, causing the click to have no effect without returning an error. - Add `scroll_into_view_if_needed` using CDP `DOM.scrollIntoViewIfNeeded` with JS `scrollIntoView` fallback for unsupported environments - Add `assert_in_viewport` to return an error when an element cannot be scrolled into view - Unify `resolve_element_center` to always use `getBoundingClientRect` (viewport-relative) instead of `DOM.getBoxModel` (page-absolute) - Extract `get_center_and_viewport` to fetch element center and viewport dimensions in a single CDP call Closes vercel-labs#1044
Contributor
|
@hyunjinee is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
…abs#1044) Verify that click, hover, and click-by-ref on off-screen elements auto-scroll into view before dispatching the interaction.
DOM.resolveNode succeeds for detached nodes, which broke the stale-ref fallback path. Instead of adding an extra CDP round-trip to check isConnected, fold the check into the existing getBoundingClientRect JS call (zero overhead on the happy path) and retry with a fresh accessibility tree lookup when a detached node is detected.
Collaborator
|
@jin-2-kakaoent LGTM! One minor follow-up to consider: |
…ck-1044 # Conflicts: # cli/src/native/e2e_tests.rs
jin-2-kakaoent
pushed a commit
to hyunjinee/agent-browser
that referenced
this pull request
Mar 30, 2026
…alidation - tap_touch and handle_drag now use resolve_scroll_and_center for auto-scroll + detached-node retry (addresses review feedback on vercel-labs#1073) - Fix drag ordering: mouseDown at source before scrolling to target, preventing source coordinates from being invalidated - Remove unused resolve_element_center (dead code after migration) - Unify e2e offscreen tests into a single e2e_offscreen_scroll_before_interactions
…alidation - tap_touch and handle_drag now use resolve_scroll_and_center for auto-scroll + detached-node retry (addresses review feedback on vercel-labs#1073) - Fix drag ordering: mouseDown at source before scrolling to target, preventing source coordinates from being invalidated - Remove unused resolve_element_center (dead code after migration) - Unify e2e offscreen tests into a single e2e_offscreen_scroll_before_interactions
jin-2-kakaoent
force-pushed
the
fix/scroll-before-click-1044
branch
from
2ee77ed to
f2c4d7d
Compare
- Remove unused box_model_center and its test - Propagate JS scrollIntoView fallback error instead of silently ignoring
jin-2-kakaoent
changed the title
Contributor
Author
|
@ctate Thanks for the feedback! I've applied All e2e + manual tests pass. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1044
Summary
click,hover,tap, anddragusing CDPDOM.scrollIntoViewIfNeededwith JS fallbackisConnectedcheck inside the existinggetBoundingClientRectJS call — zero extra CDP round-trips on the happy pathRoot cause
Two issues combined to produce the silent failure:
1. No auto-scroll before interactions
snapshotusesAccessibility.getFullAXTreewhich returns all elements regardless of viewport position. When interacting with an off-screen element,Input.dispatchMouseEvent/Input.dispatchTouchEventwas dispatched to coordinates outside the viewport — CDP doesn't error on this, so the interaction silently had no effect.2. Coordinate system mismatch in
resolve_element_centerDOM.getBoxModelreturns page-absolute coordinates whileInput.dispatchMouseEventexpects viewport-relative coordinates. After scrolling, these diverge and interactions miss the target.Changes
get_center_and_viewport: usesgetBoundingClientRectfor viewport-relative coordinates, returnsCenterResult::FoundorCenterResult::Detachedscroll_into_view_if_needed: CDPDOM.scrollIntoViewIfNeededwith JSscrollIntoViewfallback (errors propagated)assert_in_viewport: validates element is within viewport after scrollingresolve_element_object_id_fresh: skips cachedbackend_node_idfor retry after detached node detectionresolve_scroll_and_center: shared helper for click/hover/tap/drag — resolve, scroll, get center, retry on detachhandle_drag: reordered to mouseDown at source before scrolling to target, preventing source coordinate invalidationresolve_element_center,box_model_centerTest plan
e2e_offscreen_scroll_before_interactions— click, hover, click-by-ref, tap, and drag on off-screen elements in a single unified teste2e_click_stale_ref_falls_back_to_role_name— stale ref fallback still workscargo fmt+cargo clippyclean