|
| 1 | +<!-- SeleniumBase Docs --> |
| 2 | + |
| 3 | +## [<img src="https://seleniumbase.github.io/img/logo6.png" title="SeleniumBase" width="32">](https://github.com/seleniumbase/SeleniumBase/) CDP Mode 🐙 |
| 4 | + |
| 5 | +🐙 <b translate="no">SeleniumBase</b> <b translate="no">CDP Mode</b> (Chrome Devtools Protocol Mode) is a special mode inside of <b><a href="https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/uc_mode.md" translate="no"><span translate="no">SeleniumBase UC Mode</span></a></b> that lets bots appear human while controlling the browser with the <b translate="no">CDP-Driver</b>. Although regular <span translate="no">UC Mode</span> can't perform <span translate="no">WebDriver</span> actions while the <code>driver</code> is disconnected from the browser, the <span translate="no">CDP-Driver</span> can still perform actions (while maintaining its cover). |
| 6 | + |
| 7 | +👤 <b translate="no">UC Mode</b> avoids bot-detection by first disconnecting WebDriver from the browser at strategic times, calling special <code>PyAutoGUI</code> methods to bypass CAPTCHAs (as needed), and finally reconnecting the <code>driver</code> afterwards so that WebDriver actions can be performed again. Although this approach works for bypassing simple CAPTCHAs, more flexibility is needed for bypassing bot-detection on websites with advanced protection. (That's where <b translate="no">CDP Mode</b> comes in.) |
| 8 | + |
| 9 | +🐙 <b translate="no">CDP Mode</b> is based on <a href="https://github.com/HyperionGray/python-chrome-devtools-protocol" translate="no">python-cdp</a>, <a href="https://github.com/HyperionGray/trio-chrome-devtools-protocol" translate="no">trio-cdp</a>, and <a href="https://github.com/ultrafunkamsterdam/nodriver" translate="no">nodriver</a>. <code>trio-cdp</code> was an early implementation of <code>python-cdp</code>, whereas <code>nodriver</code> is a modern implementation of <code>python-cdp</code>. (Refactored CDP code is imported from <a href="https://github.com/mdmintz/MyCDP" translate="no">MyCDP</a>.) |
| 10 | + |
| 11 | +🐙 <b translate="no">CDP Mode</b> includes multiple updates to the above, such as: |
| 12 | + |
| 13 | +* Sync methods. (Using `async`/`await` is not necessary!) |
| 14 | +* The ability to use WebDriver and CDP-Driver together. |
| 15 | +* Backwards compatibility for existing UC Mode scripts. |
| 16 | +* More configuration options when launching browsers. |
| 17 | +* More methods. (And bug-fixes for existing methods.) |
| 18 | +* Faster response time for support. (Eg. [Discord Chat](https://discord.gg/EdhQTn3EyE)) |
| 19 | + |
| 20 | +-------- |
| 21 | + |
| 22 | +### 🐙 <b translate="no">CDP Mode</b> initialization: |
| 23 | + |
| 24 | +* `sb.activate_cdp_mode(url)` |
| 25 | + |
| 26 | +> (Call that from a **UC Mode** script) |
| 27 | +
|
| 28 | +-------- |
| 29 | + |
| 30 | +### 🐙 <b translate="no">CDP Mode</b> examples: |
| 31 | + |
| 32 | +> [SeleniumBase/examples/cdp_mode](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode) |
| 33 | +
|
| 34 | +### 🔖 Example 1: (Pokemon site using Incapsula/Imperva protection with invisible reCAPTCHA) |
| 35 | + |
| 36 | +> [SeleniumBase/examples/cdp_mode/raw_pokemon.py](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode/raw_pokemon.py) |
| 37 | +
|
| 38 | +<div></div> |
| 39 | +<details> |
| 40 | +<summary> ▶️ (<b>Click to expand code preview</b>)</summary> |
| 41 | + |
| 42 | +```python |
| 43 | +from seleniumbase import SB |
| 44 | + |
| 45 | +with SB(uc=True, test=True, locale_code="en") as sb: |
| 46 | + url = "https://www.pokemon.com/us" |
| 47 | + sb.activate_cdp_mode(url) |
| 48 | + sb.sleep(1) |
| 49 | + sb.cdp.click_if_visible("button#onetrust-reject-all-handler") |
| 50 | + sb.cdp.click('a[href="https://www.pokemon.com/us/pokedex/"]') |
| 51 | + sb.sleep(1) |
| 52 | + sb.cdp.click('b:contains("Show Advanced Search")') |
| 53 | + sb.sleep(1) |
| 54 | + sb.cdp.click('span[data-type="type"][data-value="electric"]') |
| 55 | + sb.cdp.click("a#advSearch") |
| 56 | + sb.sleep(1) |
| 57 | + sb.cdp.click('img[src*="img/pokedex/detail/025.png"]') |
| 58 | + sb.cdp.assert_text("Pikachu", 'div[class*="title"]') |
| 59 | + sb.cdp.assert_element('img[alt="Pikachu"]') |
| 60 | + sb.cdp.scroll_into_view("div.pokemon-ability-info") |
| 61 | + sb.sleep(1) |
| 62 | + sb.cdp.flash('div[class*="title"]') |
| 63 | + sb.cdp.flash('img[alt="Pikachu"]') |
| 64 | + sb.cdp.flash("div.pokemon-ability-info") |
| 65 | + name = sb.cdp.get_text("label.styled-select") |
| 66 | + info = sb.cdp.get_text("div.version-descriptions p.active") |
| 67 | + print("*** %s: ***\n* %s" % (name, info)) |
| 68 | + sb.sleep(2) |
| 69 | + sb.cdp.highlight_overlay("div.pokemon-ability-info") |
| 70 | + sb.sleep(2) |
| 71 | + sb.cdp.click('a[href="https://www.pokemon.com/us/play-pokemon/"]') |
| 72 | + sb.cdp.click('h3:contains("Find an Event")') |
| 73 | + location = "Concord, MA, USA" |
| 74 | + sb.cdp.type('input[data-testid="location-search"]', location) |
| 75 | + sb.sleep(1) |
| 76 | + sb.cdp.click("div.autocomplete-dropdown-container div.suggestion-item") |
| 77 | + sb.cdp.click('img[alt="search-icon"]') |
| 78 | + sb.sleep(2) |
| 79 | + events = sb.cdp.select_all('div[data-testid="event-name"]') |
| 80 | + print("*** Pokemon events near %s: ***" % location) |
| 81 | + for event in events: |
| 82 | + print("* " + event.text) |
| 83 | + sb.sleep(2) |
| 84 | +``` |
| 85 | + |
| 86 | +</details> |
| 87 | + |
| 88 | +### 🔖 Example 2: (Hyatt site using Kasada protection) |
| 89 | + |
| 90 | +> [SeleniumBase/examples/cdp_mode/raw_hyatt.py](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode/raw_hyatt.py) |
| 91 | +
|
| 92 | +<div></div> |
| 93 | +<details> |
| 94 | +<summary> ▶️ (<b>Click to expand code preview</b>)</summary> |
| 95 | + |
| 96 | +```python |
| 97 | +from seleniumbase import SB |
| 98 | + |
| 99 | +with SB(uc=True, test=True, locale_code="en") as sb: |
| 100 | + url = "https://www.hyatt.com/" |
| 101 | + sb.activate_cdp_mode(url) |
| 102 | + sb.sleep(1) |
| 103 | + sb.cdp.click_if_visible('button[aria-label="Close"]') |
| 104 | + sb.sleep(0.5) |
| 105 | + sb.cdp.click('span:contains("Explore")') |
| 106 | + sb.sleep(1) |
| 107 | + sb.cdp.click('a:contains("Hotels & Resorts")') |
| 108 | + sb.sleep(2.5) |
| 109 | + location = "Anaheim, CA, USA" |
| 110 | + sb.cdp.press_keys("input#searchbox", location) |
| 111 | + sb.sleep(1) |
| 112 | + sb.cdp.click("div#suggestion-list ul li a") |
| 113 | + sb.sleep(1) |
| 114 | + sb.cdp.click('div.hotel-card-footer button') |
| 115 | + sb.sleep(1) |
| 116 | + sb.cdp.click('button[data-locator="find-hotels"]') |
| 117 | + sb.sleep(4) |
| 118 | + hotel_names = sb.cdp.select_all( |
| 119 | + 'div[data-booking-status="BOOKABLE"] [class*="HotelCard_header"]' |
| 120 | + ) |
| 121 | + hotel_prices = sb.cdp.select_all( |
| 122 | + 'div[data-booking-status="BOOKABLE"] div.rate-currency' |
| 123 | + ) |
| 124 | + sb.assert_true(len(hotel_names) == len(hotel_prices)) |
| 125 | + print("Hyatt Hotels in %s:" % location) |
| 126 | + print("(" + sb.cdp.get_text("ul.b-color_text-white") + ")") |
| 127 | + if len(hotel_names) == 0: |
| 128 | + print("No availability over the selected dates!") |
| 129 | + for i, hotel in enumerate(hotel_names): |
| 130 | + print("* %s: %s => %s" % (i + 1, hotel.text, hotel_prices[i].text)) |
| 131 | +``` |
| 132 | + |
| 133 | +</details> |
| 134 | + |
| 135 | +### 🔖 Example 3: (BestWestern site using DataDome protection) |
| 136 | + |
| 137 | +* [SeleniumBase/examples/cdp_mode/raw_bestwestern.py](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode/raw_bestwestern.py) |
| 138 | + |
| 139 | +<div></div> |
| 140 | +<details> |
| 141 | +<summary> ▶️ (<b>Click to expand code preview</b>)</summary> |
| 142 | + |
| 143 | +```python |
| 144 | +from seleniumbase import SB |
| 145 | + |
| 146 | +with SB(uc=True, test=True, locale_code="en") as sb: |
| 147 | + url = "https://www.bestwestern.com/en_US.html" |
| 148 | + sb.activate_cdp_mode(url) |
| 149 | + sb.sleep(1.5) |
| 150 | + sb.cdp.click_if_visible("div.onetrust-close-btn-handler") |
| 151 | + sb.sleep(0.5) |
| 152 | + sb.cdp.click("input#destination-input") |
| 153 | + sb.sleep(1.5) |
| 154 | + location = "Palm Springs, CA, USA" |
| 155 | + sb.cdp.press_keys("input#destination-input", location) |
| 156 | + sb.sleep(0.6) |
| 157 | + sb.cdp.click("ul#google-suggestions li") |
| 158 | + sb.sleep(0.6) |
| 159 | + sb.cdp.click("button#btn-modify-stay-update") |
| 160 | + sb.sleep(1.5) |
| 161 | + sb.cdp.click("label#available-label") |
| 162 | + sb.sleep(4) |
| 163 | + print("Best Western Hotels in %s:" % location) |
| 164 | + summary_details = sb.cdp.get_text("#summary-details-column") |
| 165 | + dates = summary_details.split("ROOM")[0].split("DATES")[-1].strip() |
| 166 | + print("(Dates: %s)" % dates) |
| 167 | + flip_cards = sb.cdp.select_all(".flipCard") |
| 168 | + for i, flip_card in enumerate(flip_cards): |
| 169 | + hotel = flip_card.query_selector(".hotelName") |
| 170 | + price = flip_card.query_selector(".priceSection") |
| 171 | + if hotel and price: |
| 172 | + print("* %s: %s => %s" % ( |
| 173 | + i + 1, hotel.text.strip(), price.text.strip()) |
| 174 | + ) |
| 175 | +``` |
| 176 | + |
| 177 | +</details> |
| 178 | + |
| 179 | +(<b>Note:</b> Extra <code translate="no">sb.sleep()</code> calls have been added to prevent bot-detection because some sites will flag you as a bot if you perform actions too quickly.) |
| 180 | + |
| 181 | +(<b>Note:</b> Some sites may IP-block you for 36 hours or more if they catch you using regular <span translate="no">Selenium WebDriver</span>. Be extra careful when creating and/or modifying automation scripts that run on them.) |
| 182 | + |
| 183 | +-------- |
| 184 | + |
| 185 | +### 🐙 CDP Mode API / Methods |
| 186 | + |
| 187 | +(Some method args have been left out for simplicity. Eg: <code translate="no">timeout</code>) |
| 188 | + |
| 189 | +```python |
| 190 | +sb.cdp.get(url) |
| 191 | +sb.cdp.reload() |
| 192 | +sb.cdp.refresh() |
| 193 | +sb.cdp.add_handler(event, handler) |
| 194 | +sb.cdp.find_element(selector) |
| 195 | +sb.cdp.find_all(selector) |
| 196 | +sb.cdp.find_elements_by_text(text, tag_name=None) |
| 197 | +sb.cdp.select(selector) |
| 198 | +sb.cdp.select_all(selector) |
| 199 | +sb.cdp.click_link(link_text) |
| 200 | +sb.cdp.tile_windows(windows=None, max_columns=0) |
| 201 | +sb.cdp.get_all_cookies(*args, **kwargs) |
| 202 | +sb.cdp.set_all_cookies(*args, **kwargs) |
| 203 | +sb.cdp.save_cookies(*args, **kwargs) |
| 204 | +sb.cdp.load_cookies(*args, **kwargs) |
| 205 | +sb.cdp.clear_cookies(*args, **kwargs) |
| 206 | +sb.cdp.sleep(seconds) |
| 207 | +sb.cdp.bring_active_window_to_front() |
| 208 | +sb.cdp.get_active_element() |
| 209 | +sb.cdp.get_active_element_css() |
| 210 | +sb.cdp.click(selector) |
| 211 | +sb.cdp.click_active_element() |
| 212 | +sb.cdp.click_if_visible(selector) |
| 213 | +sb.cdp.mouse_click(selector) |
| 214 | +sb.cdp.nested_click(parent_selector, selector) |
| 215 | +sb.cdp.get_nested_element(parent_selector, selector) |
| 216 | +sb.cdp.flash(selector) |
| 217 | +sb.cdp.focus(selector) |
| 218 | +sb.cdp.highlight_overlay(selector) |
| 219 | +sb.cdp.remove_element(selector) |
| 220 | +sb.cdp.remove_from_dom(selector) |
| 221 | +sb.cdp.remove_elements(selector) |
| 222 | +sb.cdp.scroll_into_view(selector) |
| 223 | +sb.cdp.send_keys(selector, text) |
| 224 | +sb.cdp.press_keys(selector, text) |
| 225 | +sb.cdp.type(selector, text) |
| 226 | +sb.cdp.evaluate(expression) |
| 227 | +sb.cdp.js_dumps(obj_name) |
| 228 | +sb.cdp.maximize() |
| 229 | +sb.cdp.minimize() |
| 230 | +sb.cdp.medimize() |
| 231 | +sb.cdp.set_window_rect() |
| 232 | +sb.cdp.reset_window_size() |
| 233 | +sb.cdp.get_window() |
| 234 | +sb.cdp.get_text() |
| 235 | +sb.cdp.get_title() |
| 236 | +sb.cdp.get_current_url() |
| 237 | +sb.cdp.get_origin() |
| 238 | +sb.cdp.get_page_source() |
| 239 | +sb.cdp.get_user_agent() |
| 240 | +sb.cdp.get_cookie_string() |
| 241 | +sb.cdp.get_locale_code() |
| 242 | +sb.cdp.get_screen_rect() |
| 243 | +sb.cdp.get_window_rect() |
| 244 | +sb.cdp.get_window_size() |
| 245 | +sb.cdp.get_window_position() |
| 246 | +sb.cdp.get_element_rect(selector) |
| 247 | +sb.cdp.get_element_size(selector) |
| 248 | +sb.cdp.get_element_position(selector) |
| 249 | +sb.cdp.get_gui_element_rect(selector) |
| 250 | +sb.cdp.get_gui_element_center(selector) |
| 251 | +sb.cdp.get_document() |
| 252 | +sb.cdp.get_flattened_document() |
| 253 | +sb.cdp.get_element_attributes(selector) |
| 254 | +sb.cdp.get_element_html(selector) |
| 255 | +sb.cdp.set_attributes(selector, attribute, value) |
| 256 | +sb.cdp.internalize_links() |
| 257 | +sb.cdp.is_element_present(selector) |
| 258 | +sb.cdp.is_element_visible(selector) |
| 259 | +sb.cdp.assert_element(selector) |
| 260 | +sb.cdp.assert_element_present(selector) |
| 261 | +sb.cdp.assert_text(text, selector="html") |
| 262 | +sb.cdp.assert_exact_text(text, selector="html") |
| 263 | +sb.cdp.save_screenshot(name, folder=None, selector=None) |
| 264 | +``` |
| 265 | + |
| 266 | +-------- |
| 267 | + |
| 268 | +### 🐙 CDP Mode WebElement API / Methods |
| 269 | + |
| 270 | +```python |
| 271 | +element.clear_input() |
| 272 | +element.click() |
| 273 | +element.flash() |
| 274 | +element.focus() |
| 275 | +element.highlight_overlay() |
| 276 | +element.mouse_click() |
| 277 | +element.mouse_drag(destination) |
| 278 | +element.mouse_move() |
| 279 | +element.query_selector(selector) |
| 280 | +element.querySelector(selector) |
| 281 | +element.query_selector_all(selector) |
| 282 | +element.querySelectorAll(selector) |
| 283 | +element.remove_from_dom() |
| 284 | +element.save_screenshot(*args, **kwargs) |
| 285 | +element.save_to_dom() |
| 286 | +element.scroll_into_view() |
| 287 | +element.select_option() |
| 288 | +element.send_file(*file_paths) |
| 289 | +element.send_keys(text) |
| 290 | +element.set_text(value) |
| 291 | +element.type(text) |
| 292 | +element.get_position() |
| 293 | +element.get_html() |
| 294 | +element.get_js_attributes() |
| 295 | +``` |
| 296 | + |
| 297 | +-------- |
| 298 | + |
| 299 | +<img src="https://seleniumbase.github.io/cdn/img/sb_text_f.png" alt="SeleniumBase" title="SeleniumBase" align="center" width="335"> |
| 300 | + |
| 301 | +<div><a href="https://github.com/seleniumbase/SeleniumBase"><img src="https://seleniumbase.github.io/cdn/img/sb_logo_gs.png" alt="SeleniumBase" title="SeleniumBase" width="335" /></a></div> |
0 commit comments