Skip to content

feat: implement frameLocator support for cross-origin iframe interaction#1030

Open
giulio-leone wants to merge 4 commits intovercel-labs:mainfrom
giulio-leone:feat/279-frame-locator-rust
Open

feat: implement frameLocator support for cross-origin iframe interaction#1030
giulio-leone wants to merge 4 commits intovercel-labs:mainfrom
giulio-leone:feat/279-frame-locator-rust

Conversation

@giulio-leone
Copy link
Copy Markdown
Contributor

Summary

Implements the frameLocator feature request from #279, enabling cross-origin iframe interaction (Apple auth, OAuth providers, embedded forms, etc.) via CDP execution context scoping.

Supersedes #460 (which targeted the old TypeScript codebase).

Changes

Core: Frame-scoped evaluation

  • Added context_id: Option<i64> to EvaluateParams for frame-scoped Runtime.evaluate
  • Added BrowserManager::get_frame_context_id() using Page.createIsolatedWorld with grantUniversalAccess: true
  • Added BrowserManager::evaluate_in_context() for optional frame-scoped JS evaluation

New command: frame locator

  • frame locator <css-selector> — scope all subsequent commands to the matched iframe
  • frame locator clear — return to main frame context
  • Equivalent to Playwright's page.frameLocator(selector)

Frame-aware command execution

  • evaluate — JS evaluation scoped to active frame
  • getByRole, getByText, getByLabel, getByPlaceholder, getByTestId, getByTitle, getByAltText — semantic locators resolve elements within iframe DOM
  • click, fill, check, hover, text subactions — coordinate resolution accounts for iframe viewport offset
  • Attribute cleanup (data-agent-browser-located) runs in the correct frame context

Improved frame resolution

  • Multi-strategy matching: name → id → src URL → nth child frame
  • Handles anonymous iframes without name or id attributes
  • Extracted find_frame_in_tree() and find_nth_child_frame() as reusable helpers

How it works

When frame locator <selector> is invoked:

  1. The CSS selector resolves the iframe element in the main page
  2. The iframe's name, id, and src are extracted
  3. The frame tree (Page.getFrameTree) is searched using multiple strategies
  4. Page.createIsolatedWorld creates an execution context in the target frame
  5. All subsequent getBy*, evaluate, and interaction commands use this context
  6. frame locator clear resets to the main frame

This uses CDP's built-in cross-origin access (grantUniversalAccess), bypassing same-origin restrictions at the browser level — no proxy or content injection needed.

Testing

# Navigate to a page with an iframe
> navigate https://example.com/with-oauth-iframe

# Scope to the iframe
> frame locator "#auth-iframe"

# Interact with elements inside the iframe
> getByRole button --name "Sign in"
> click

# Return to main frame
> frame locator clear

Closes #279

⚠️ This reopens #834 which was accidentally closed due to fork deletion.

@vercel
Copy link
Copy Markdown
Contributor

vercel bot commented Mar 25, 2026

@giulio-leone is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

giulio-leone and others added 4 commits March 26, 2026 10:48
…ion (vercel-labs#279)

Implement the frameLocator feature request from issue vercel-labs#279, enabling
cross-origin iframe interaction via CDP execution context scoping.

Core changes:
- Add context_id to EvaluateParams for frame-scoped JS evaluation
- Add BrowserManager::get_frame_context_id() using Page.createIsolatedWorld
- Add BrowserManager::evaluate_in_context() for frame-scoped evaluation
- Add frame locator CLI command: 'frame locator <selector>' / 'frame locator clear'
- Add handle_framelocator() action handler with set/clear semantics

Frame-aware execution:
- handle_evaluate() now scopes evaluation to active frame
- handle_getbyrole() locates elements within frame context
- handle_semantic_locator() (getByText, getByLabel, etc.) frame-aware
- execute_subaction() resolves clicks/fills/hovers within iframe coordinates
- Viewport-absolute coordinate calculation for cross-origin iframe clicks

Improved frame resolution:
- Multi-strategy matching: name -> id -> src URL -> nth child frame
- Extracted find_frame_in_tree() and find_nth_child_frame() helpers
- Handles anonymous iframes without name/id attributes
- Fix resolve_element_in_frame to target the specific iframe via stored
  selector instead of grabbing the first visible iframe on the page
- Reorder frame matching to prefer src URL over id-as-name fallback,
  since CDP's name field maps to iframe.name, not iframe.id
- Run cargo fmt for consistent formatting

Co-authored-by: Copilot <[email protected]>
- Run cargo fmt across 4 files (commands.rs, element.rs, snapshot.rs, storage.rs)
- Refactor active_frame_selector assignment to use Option::map/or_else
  instead of manual if-let chain (actions.rs:3590)
@giulio-leone giulio-leone force-pushed the feat/279-frame-locator-rust branch from fe4b79d to dd96ebe Compare March 26, 2026 10:10
@giulio-leone
Copy link
Copy Markdown
Contributor Author

Rebased this branch onto current main, resolved the conflicts locally, and force-pushed the refreshed head (dd96ebe).

While validating the rebased diff I found and fixed one real regression in the new frameLocator path: ref-based frame selection now preserves the active iframe target for offset calculation, so cross-origin iframe interactions no longer risk falling back to the first visible iframe when the frame was selected via @ref.

Local Rust validation passed on the pushed head:

  • cargo fmt --manifest-path cli/Cargo.toml -- --check
  • cargo clippy --manifest-path cli/Cargo.toml -- -D warnings
  • cargo test --profile ci --manifest-path cli/Cargo.toml (544 passed)

GitHub has now rerun CI / Rust on the updated head and it is green. The remaining red checks still appear to be the two Vercel authorization gates, which look team-scoped rather than code-related from contributor side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Add frameLocator support for cross-origin iframe interaction

1 participant