Skip to content

Commit 82ef425

Browse files
[refactor]: use progress tracker in act/extract/observe (#3109)
# why this PR wires the `progress` tracker into `act()`, `extract()`, `observe()`. before this PR, `act()`, `observe()`, & `extract()` had their own timeout tracking mechanism, which only checked remaining time between between major IO steps. a stalled snapshot, model request, or browser action could keep the call waiting past its timeout. nested work also did not share the caller's remaining time. # what changed - gave each call one shared deadline across snapshots, cache reads & writes, model requests, & browser work. each step gets only the time left. - carried act's deadline through page settling, supplied actions, cached replay, two-step actions, & self-healing. extract's optional screenshot & second model request share its deadline too. - made expiry reject with the caller's timeout error & prevent later steps. late model or cache responses cannot start another action or request. browser commands & model requests already sent may still finish. - cleaned up settling listeners, temporary element references, & pressed inputs when interrupted. kept snapshot scope fallback within the same budget. - kept the existing defaults: omitted timeouts & `0` remain unlimited. - removed the unused timeout guard # test plan - `service-progress.test.ts` checks expiry during snapshots, cache work, & model calls for all three services. it verifies that nested work uses the remaining budget, omitted/zero timeouts stay unlimited, & late responses cannot resume work. act cases cover cached replay, self-healing, two-step actions, interrupted fill/drag/key presses, & settling cleanup. extract cases cover screenshots, image encoding, & its second model request. - `serviceTimeouts.test.ts` uses a local browser, a controlled iframe response, & controlled model responses. it checks that readiness expiry prevents inference, full-page fallback does not restart the timeout, & late responses cannot cause clicks or a second extraction request. it also verifies that act can use the full-page fallback while time remains & that an expired supplied action stays stopped after its iframe loads. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Gives `act()`, `extract()`, and `observe()` a shared `progress` deadline so timeouts apply across snapshots, cache reads/writes, model requests, and browser actions. Before, each service checked remaining time only between major steps, so stalled work could outlive its timeout and nested work never shared the caller's budget. - One deadline now covers all work in a call; nested steps inherit the caller's remaining time. - Expiry rejects with the caller's timeout error and blocks any later steps, though already-sent browser commands and model requests may still finish. - Interruption cleans up settling listeners and temporary element references, releases held keys and a pressed mouse button exactly once (the mouse at its last dispatched position) when a drag expires mid-way, and drops held cache keys. - Omitted or `0` timeouts remain unlimited; the unused `timeoutGuard` helper is removed. - Added unit and integration coverage for expiry during each step, budget inheritance, late responses, and cleanup. <sup>Written for commit f9753f8. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/browserbase/stagehand/pull/3109?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
1 parent 0336409 commit 82ef425

16 files changed

Lines changed: 1116 additions & 483 deletions

File tree

‎packages/extension/handlers/handlerUtils/actHandlerUtils.ts‎

Lines changed: 96 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { Locator } from "../../understudy/locator.js";
55
import type { MouseButton } from "@browserbasehq/stagehand-protocol/types";
66
import { resolveLocatorWithHops } from "../../understudy/deepLocator.js";
77
import type { Page } from "../../understudy/page.js";
8+
import type { Progress } from "../../understudy/progress.js";
89
import type { StagehandLogger } from "../../logger.js";
910
import { toTitleCase } from "../../utils.js";
1011

@@ -17,6 +18,7 @@ export interface UnderstudyMethodHandlerContext {
1718
page: Page;
1819
initialUrl: string;
1920
logger: StagehandLogger;
21+
progress: Progress;
2022
domSettleTimeoutMs?: number;
2123
}
2224

@@ -49,6 +51,7 @@ export async function performUnderstudyMethod(
4951
rawXPath: string,
5052
args: ReadonlyArray<unknown>,
5153
logger: StagehandLogger,
54+
progress: Progress,
5255
domSettleTimeoutMs?: number,
5356
): Promise<void> {
5457
const selectorRaw = normalizeRootXPath(rawXPath);
@@ -59,8 +62,10 @@ export async function performUnderstudyMethod(
5962
{ target: selectorRaw },
6063
async (spanLogger) => {
6164
// Unified resolver: supports '>>' hops and XPath across iframes.
62-
const locator: Locator = await resolveLocatorWithHops(page, frame, selectorRaw);
63-
const initialUrl = await getFrameUrl(frame);
65+
const locator: Locator = await resolveLocatorWithHops(page, frame, selectorRaw, progress);
66+
const initialUrl = await progress.run("reading frame URL", () =>
67+
frame.evaluate<string>("location.href", undefined, progress),
68+
);
6469

6570
spanLogger.debug("Performing understudy method", {
6671
category: "action",
@@ -79,11 +84,12 @@ export async function performUnderstudyMethod(
7984
initialUrl,
8085
logger: spanLogger,
8186
domSettleTimeoutMs,
87+
progress,
8288
};
8389
const handler = METHOD_HANDLER_MAP[method] ?? null;
8490

8591
if (handler) {
86-
await handler(ctx);
92+
await progress.run(`performing ${method}`, () => handler(ctx));
8793
return;
8894
}
8995

@@ -134,7 +140,7 @@ export async function selectOption(ctx: UnderstudyMethodHandlerContext) {
134140
const { locator, xpath, args, logger } = ctx;
135141
try {
136142
const text = args[0]?.toString() || "";
137-
await locator.selectOption(text);
143+
await locator.selectOption(text, ctx.progress);
138144
} catch (e) {
139145
const msg = e instanceof Error ? e.message : String(e);
140146
const stack = e instanceof Error ? e.stack : undefined;
@@ -154,10 +160,15 @@ async function scrollIntoView(ctx: UnderstudyMethodHandlerContext): Promise<void
154160
category: "action",
155161
xpath,
156162
});
157-
const { objectId } = await locator.resolveNode();
163+
const { objectId } = await locator.resolveNode(ctx.progress);
158164
const ownerSession = locator.getFrame().session;
159-
await ownerSession.send("DOM.scrollIntoViewIfNeeded", { objectId });
160-
await ownerSession.send("Runtime.releaseObject", { objectId }).catch(() => {});
165+
try {
166+
await ctx.progress.run("scrolling element into view", () =>
167+
ownerSession.send("DOM.scrollIntoViewIfNeeded", { objectId }),
168+
);
169+
} finally {
170+
await ctx.progress.cleanup(() => ownerSession.send("Runtime.releaseObject", { objectId }));
171+
}
161172
}
162173

163174
async function scrollElementToPercentage(ctx: UnderstudyMethodHandlerContext): Promise<void> {
@@ -169,7 +180,7 @@ async function scrollElementToPercentage(ctx: UnderstudyMethodHandlerContext): P
169180
});
170181

171182
const [yArg = "0%"] = args;
172-
await locator.scrollTo(yArg);
183+
await locator.scrollTo(yArg, ctx.progress);
173184
}
174185

175186
/** Scroll the page by pixel offset, starting from the element's center. */
@@ -178,8 +189,8 @@ async function scrollByPixelOffset(ctx: UnderstudyMethodHandlerContext): Promise
178189
const dx = Number(args[0] ?? 0);
179190
const dy = Number(args[1] ?? 0);
180191

181-
const { x, y } = await locator.centroid();
182-
await page.scroll(x, y, dx, dy);
192+
const { x, y } = await locator.centroid(ctx.progress);
193+
await page.scroll(x, y, dx, dy, ctx.progress);
183194
}
184195

185196
async function wheelScroll(ctx: UnderstudyMethodHandlerContext): Promise<void> {
@@ -189,20 +200,22 @@ async function wheelScroll(ctx: UnderstudyMethodHandlerContext): Promise<void> {
189200
category: "action",
190201
deltaY,
191202
});
192-
await frame.session.send<never>("Input.dispatchMouseEvent", {
193-
type: "mouseWheel",
194-
x: 0,
195-
y: 0,
196-
deltaY,
197-
deltaX: 0,
198-
} as Protocol.Input.DispatchMouseEventRequest);
203+
await ctx.progress.run("dispatching mouse wheel", () =>
204+
frame.session.send<never>("Input.dispatchMouseEvent", {
205+
type: "mouseWheel",
206+
x: 0,
207+
y: 0,
208+
deltaY,
209+
deltaX: 0,
210+
} as Protocol.Input.DispatchMouseEventRequest),
211+
);
199212
}
200213

201214
async function fillOrType(ctx: UnderstudyMethodHandlerContext): Promise<void> {
202215
const { locator, xpath, args, logger } = ctx;
203216
try {
204-
await locator.fill(""); // clear
205-
await locator.fill(args[0] ?? "");
217+
await locator.fill("", ctx.progress); // clear
218+
await locator.fill(args[0] ?? "", ctx.progress);
206219
} catch (e) {
207220
const msg = e instanceof Error ? e.message : String(e);
208221
logger.debug("Error filling element", {
@@ -217,7 +230,7 @@ async function fillOrType(ctx: UnderstudyMethodHandlerContext): Promise<void> {
217230
async function typeText(ctx: UnderstudyMethodHandlerContext): Promise<void> {
218231
const { locator, xpath, args, logger } = ctx;
219232
try {
220-
await locator.type(args[0] ?? "");
233+
await locator.type(args[0] ?? "", undefined, ctx.progress);
221234
} catch (e) {
222235
const msg = e instanceof Error ? e.message : String(e);
223236
logger.debug("Error typing into element", {
@@ -238,7 +251,7 @@ async function pressKey(ctx: UnderstudyMethodHandlerContext): Promise<void> {
238251
key,
239252
xpath,
240253
});
241-
await page.keyPress(key);
254+
await page.keyPress(key, undefined, ctx.progress);
242255
} catch (e) {
243256
const msg = e instanceof Error ? e.message : String(e);
244257
logger.debug("Error pressing key", {
@@ -254,7 +267,7 @@ async function pressKey(ctx: UnderstudyMethodHandlerContext): Promise<void> {
254267
async function clickElement(ctx: UnderstudyMethodHandlerContext): Promise<void> {
255268
const { locator, xpath, args, logger } = ctx;
256269
try {
257-
await locator.click({ button: (args[0] as MouseButton) || undefined });
270+
await locator.click({ button: (args[0] as MouseButton) || undefined }, ctx.progress);
258271
} catch (e) {
259272
const msg = e instanceof Error ? e.message : String(e);
260273
logger.error("Error performing click", {
@@ -269,7 +282,7 @@ async function clickElement(ctx: UnderstudyMethodHandlerContext): Promise<void>
269282
async function doubleClick(ctx: UnderstudyMethodHandlerContext): Promise<void> {
270283
const { locator, xpath, logger } = ctx;
271284
try {
272-
await locator.click({ clickCount: 2 });
285+
await locator.click({ clickCount: 2 }, ctx.progress);
273286
} catch (e) {
274287
const msg = e instanceof Error ? e.message : String(e);
275288
logger.error("Error performing double click", {
@@ -286,12 +299,12 @@ async function dragAndDrop(ctx: UnderstudyMethodHandlerContext): Promise<void> {
286299
const toXPath = String(args[0] ?? "").trim();
287300
if (!toXPath) throw new Error("dragAndDrop requires a target XPath arg");
288301

289-
const targetLocator = await resolveLocatorWithHops(page, frame, toXPath);
302+
const targetLocator = await resolveLocatorWithHops(page, frame, toXPath, ctx.progress);
290303

291304
try {
292305
// 1) Centers in local (owning-frame) viewport
293-
const { x: fromLocalX, y: fromLocalY } = await locator.centroid();
294-
const { x: toLocalX, y: toLocalY } = await targetLocator.centroid();
306+
const { x: fromLocalX, y: fromLocalY } = await locator.centroid(ctx.progress);
307+
const { x: toLocalX, y: toLocalY } = await targetLocator.centroid(ctx.progress);
295308

296309
// 2) Convert to main-viewport absolute coordinates
297310
const fromAbs = await locator
@@ -312,6 +325,7 @@ async function dragAndDrop(ctx: UnderstudyMethodHandlerContext): Promise<void> {
312325
return { x: Math.round(X), y: Math.round(Y) };
313326
},
314327
{ x: fromLocalX, y: fromLocalY },
328+
ctx.progress,
315329
);
316330

317331
const toAbs = await targetLocator
@@ -332,13 +346,21 @@ async function dragAndDrop(ctx: UnderstudyMethodHandlerContext): Promise<void> {
332346
return { x: Math.round(X), y: Math.round(Y) };
333347
},
334348
{ x: toLocalX, y: toLocalY },
349+
ctx.progress,
335350
);
336351

337352
// 3) Perform drag in main session
338-
await page.dragAndDrop(fromAbs.x, fromAbs.y, toAbs.x, toAbs.y, {
339-
steps: 10,
340-
delay: 5,
341-
});
353+
await page.dragAndDrop(
354+
fromAbs.x,
355+
fromAbs.y,
356+
toAbs.x,
357+
toAbs.y,
358+
{
359+
steps: 10,
360+
delay: 5,
361+
},
362+
ctx.progress,
363+
);
342364
} catch (e) {
343365
const msg = e instanceof Error ? e.message : String(e);
344366
logger.error("Error performing drag and drop", {
@@ -369,12 +391,13 @@ async function scrollByElementHeight(
369391
xpath,
370392
});
371393

372-
const { objectId } = await locator.resolveNode();
394+
const { objectId } = await locator.resolveNode(ctx.progress);
373395
try {
374396
const ownerSession = locator.getFrame().session;
375-
await ownerSession.send<Protocol.Runtime.CallFunctionOnResponse>("Runtime.callFunctionOn", {
376-
objectId,
377-
functionDeclaration: `
397+
await ctx.progress.run("scrolling by element height", () =>
398+
ownerSession.send<Protocol.Runtime.CallFunctionOnResponse>("Runtime.callFunctionOn", {
399+
objectId,
400+
functionDeclaration: `
378401
function(dir) {
379402
const waitForScrollEnd = (el) => new Promise((resolve) => {
380403
let last = el.scrollTop ?? 0;
@@ -399,20 +422,21 @@ async function scrollByElementHeight(
399422
return waitForScrollEnd(this);
400423
}
401424
`,
402-
arguments: [{ value: direction }],
403-
awaitPromise: true,
404-
returnByValue: true,
405-
});
425+
arguments: [{ value: direction }],
426+
awaitPromise: true,
427+
returnByValue: true,
428+
}),
429+
);
406430
} finally {
407431
const ownerSession = locator.getFrame().session;
408-
await ownerSession.send("Runtime.releaseObject", { objectId }).catch(() => {});
432+
await ctx.progress.cleanup(() => ownerSession.send("Runtime.releaseObject", { objectId }));
409433
}
410434
}
411435

412436
export async function hover(ctx: UnderstudyMethodHandlerContext) {
413437
const { locator, xpath, logger } = ctx;
414438
try {
415-
await locator.hover();
439+
await locator.hover(ctx.progress);
416440
} catch (e) {
417441
const msg = e instanceof Error ? e.message : String(e);
418442
const stack = e instanceof Error ? e.stack : undefined;
@@ -428,20 +452,15 @@ export async function hover(ctx: UnderstudyMethodHandlerContext) {
428452

429453
/* ===================== Helpers ===================== */
430454

431-
async function getFrameUrl(frame: Frame): Promise<string> {
432-
// Evaluate from within the frame's isolated world
433-
const url = await frame.evaluate<string>("location.href");
434-
return url;
435-
}
436-
437455
/**
438456
* More robust DOM settle using Network + Page events to detect network quiet.
439457
* Closely modeled after the provided snippet, adapted to our Frame/session + logger.
440458
*/
441459
export async function waitForDomNetworkQuiet(
442460
frame: Frame,
443461
logger: StagehandLogger,
444-
timeout?: number,
462+
timeout: number | undefined,
463+
progress: Progress,
445464
): Promise<void> {
446465
const overallTimeout =
447466
typeof timeout === "number" && Number.isFinite(timeout) ? Math.max(0, timeout) : 5_000;
@@ -451,13 +470,18 @@ export async function waitForDomNetworkQuiet(
451470
// Ensure a document exists; if not, wait for DOMContentLoaded on this frame.
452471
let hasDoc: boolean;
453472
try {
454-
const rs = await frame.evaluate<string>("document.readyState");
473+
const rs = await progress.run("checking document readiness", () =>
474+
frame.evaluate<string>("document.readyState", undefined, progress),
475+
);
455476
hasDoc = rs === "interactive" || rs === "complete";
456477
} catch {
478+
progress.throwIfStopped();
457479
hasDoc = false;
458480
}
459481
if (!hasDoc && overallTimeout > 0) {
460-
await frame.waitForLoadState("domcontentloaded", overallTimeout).catch(() => {});
482+
await frame.waitForLoadState("domcontentloaded", overallTimeout, progress).catch(() => {
483+
progress.throwIfStopped();
484+
});
461485
}
462486

463487
const elapsed = Date.now() - settleStart;
@@ -466,22 +490,27 @@ export async function waitForDomNetworkQuiet(
466490
return;
467491
}
468492

469-
await client.send("Network.enable").catch(() => {});
470-
await client.send("Page.enable").catch(() => {});
493+
await progress.run("enabling network events", () =>
494+
client.send("Network.enable").catch(() => {}),
495+
);
496+
await progress.run("enabling page events", () => client.send("Page.enable").catch(() => {}));
471497
// Best-effort; some sessions may not support Target.setAutoAttach here.
472-
await client
473-
.send("Target.setAutoAttach", {
474-
autoAttach: true,
475-
waitForDebuggerOnStart: false,
476-
flatten: true,
477-
filter: [
478-
{ type: "worker", exclude: true },
479-
{ type: "shared_worker", exclude: true },
480-
],
481-
})
482-
.catch(() => {});
483-
484-
return new Promise<void>((resolve) => {
498+
await progress.run("attaching to frames", () =>
499+
client
500+
.send("Target.setAutoAttach", {
501+
autoAttach: true,
502+
waitForDebuggerOnStart: false,
503+
flatten: true,
504+
filter: [
505+
{ type: "worker", exclude: true },
506+
{ type: "shared_worker", exclude: true },
507+
],
508+
})
509+
.catch(() => {}),
510+
);
511+
512+
progress.throwIfStopped();
513+
await new Promise<void>((resolve) => {
485514
const inflight = new Set<string>();
486515
const meta = new Map<string, { url: string; start: number }>();
487516
const docByFrame = new Map<string, string>();
@@ -576,7 +605,10 @@ export async function waitForDomNetworkQuiet(
576605
if (quietTimer) clearTimeout(quietTimer);
577606
if (stalledRequestSweepTimer) clearInterval(stalledRequestSweepTimer);
578607
clearTimeout(guard);
608+
progress.signal.removeEventListener("abort", resolveDone);
579609
resolve();
580610
};
611+
progress.signal.addEventListener("abort", resolveDone, { once: true });
581612
});
613+
progress.throwIfStopped();
582614
}

‎packages/extension/handlers/handlerUtils/timeoutGuard.ts‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

0 commit comments

Comments
 (0)