CDP MODE with Scrapy Framework

[Kamran-ov]

Hi All,

Question to the scraping geeks.
Was anybody able to use CDP mode (SB or Driver, does not matter) within the Scrapy framework?
Tried a lot to research online or with AI, but could not reach any conclusion on the solution.

If anybody is unaware, below is the error you get when trying to activate CDP mode within Scrapy script
I assume it is related to the fact that Scrapy already runs everything in async, and adding another async Python does not allow.

File "C:\Users\kamranov\Documents\Python Scripts\retailers\_venv\Lib\site-packages\seleniumbase\core\browser_launcher.py", line 631, in uc_open_with_cdp_mode driver.cdp_base = loop.run_until_complete( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\kamranov\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 661, in run_until_complete self._check_running() File "C:\Users\kamranov\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 622, in _check_running raise RuntimeError( RuntimeError: Cannot run the event loop while another loop is running 2025-09-02 12:52:18 [py.warnings] WARNING: C:\Users\kamranov\Documents\Python Scripts\retailers\_venv\Lib\site-packages\scrapy\core\engine.py:190: RuntimeWarning: coroutine 'start' was never awaited logger.error(

Many thanks for considering my request.

[mdmintz]

If you're looking to use async/await with SeleniumBase CDP Mode, there's an async format. Eg:

import asyncio
import time
from seleniumbase.undetected import cdp_driver

async def main():
    driver = await cdp_driver.cdp_util.start_async()
    page = await driver.get("about:blank")
    await page.set_locale("en")
    await page.get("https://www.priceline.com/")
    time.sleep(3)
    print(await page.evaluate("document.title"))
    element = await page.select('[data-testid*="endLocation"]')
    await element.click_async()
    time.sleep(1)
    await element.send_keys_async("Boston")
    time.sleep(2)

if __name__ == "__main__":
    loop = asyncio.new_event_loop()
    loop.run_until_complete(main())

Otherwise, if you want nested event loops, see https://stackoverflow.com/a/61847103/7058266, which tells you to use nest_asyncio: (pip install nest-asyncio):

import nest_asyncio
nest_asyncio.apply()