docs: Add guide about integrating Stagehand #1290
Conversation
|
I had to use |
docs/guides/code_examples/playwright_crawler_stagehand/support_classes.py
Outdated
Show resolved
Hide resolved
docs/guides/code_examples/playwright_crawler_stagehand/browser_classes.py
Outdated
Show resolved
Hide resolved
docs/guides/code_examples/playwright_crawler_stagehand/support_classes.py
Show resolved
Hide resolved
docs/guides/code_examples/playwright_crawler_stagehand/support_classes.py
Show resolved
Hide resolved
There was a problem hiding this comment.
Good job Max!
The integration itself is not as easy as I expected. Maybe this could show us the direction in which we could improve/simplify the browsers/Playwright-related interface.
And/or we could introduce a dedicated crawler to this directly in Crawlee, something like PlaywrightStagehandCrawler. Then the guide could focus solely on its usage, showing how to use AI-based selectors for web scraping.
Let's further discuss it with @B4nan and maybe @janbuchar once they're back from their vacations.
| @@ -0,0 +1,66 @@ | |||
| --- | |||
There was a problem hiding this comment.
I suppose we can't use "Run on Apify" for these examples as it contains more than 1 file.
There was a problem hiding this comment.
That's right.
And in one file, it would look very cumbersome
I think the integration comes out more complicated because of the current I hope that they will improve their API and then the guide can be simplified |
|
|
||
| self._total_opened_pages += 1 | ||
|
|
||
| # Wrap StagehandPage to provide Playwright Page interface |
There was a problem hiding this comment.
This comment seems inaccurate
| pw_page = page._page # noqa: SLF001 | ||
|
|
||
| # Handle page close event | ||
| pw_page.on(event='close', f=self._on_page_close) | ||
|
|
||
| # Update internal state | ||
| self._pages.append(pw_page) | ||
| self._last_page_opened_at = datetime.now(timezone.utc) | ||
|
|
||
| self._total_opened_pages += 1 |
There was a problem hiding this comment.
There is quite a bit of code copied over from PlaywrightBrowserController, isn't it? Any chance we could improve the PlaywrightBrowserController internal API so that integrating libraries that extend Playwright is easier?
There was a problem hiding this comment.
Perhaps context creation should be put into a separate public method. As well as updating states. That would make the same thing a bit cleaner.
But I would say that the main problem with this integration is that you have to do for example, this - pw_page = page._page.
docs/guides/code_examples/playwright_crawler_stagehand/stagehand_run.py
Outdated
Show resolved
Hide resolved
…nd_run.py Co-authored-by: Jan Buchar <[email protected]>
Description
stagehand-pythonv.0.4.0Issues