Selenium-Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Selenium Tutorial\n",
    "\n",
    "Selenium is a suite of tools and libraries that can web scrape and automate web browser interactions. Web browser automation includes website testing, searching, gathering data, and any other interaction you have with a website. For example, you might use selenium to test various features of a website you build, or you might use selenium to help in gathering data or papers for a literature review. The Selenium WebDriver contains the most useful features for us. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./types-of-selenium.png\" alt=\"types of selenium\" width=\"700\" class=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capabilities\n",
    "\n",
    "Selenium is a popular choice for web browser automation because of it's power and flexibility. Selenium:\n",
    "-   is free and open source\n",
    "-   supports multiple languages (Python, Java, CSS, Ruby, etc.)\n",
    "-   works on multiple operating systems (Mac, Windows, Linux)\n",
    "-   works on multiple web browsers (Chrome, Firefox, Internet Explorer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "1) Run this in terminal: `pip install selenium` \n",
    "    (Use `pip3` for MacOS)\n",
    "\n",
    "2) Import `selenium` at beginning of project\n",
    "\n",
    "### Web Driver Installation\n",
    "\n",
    "1) Download ChromeWebDriver (or whichever browser you would like to use)\n",
    "    - Know your version of Chrome you are using\n",
    "    - Select your OS accordingly\n",
    "2) Establish path to wherever you saved the WebDriver download\n",
    "    - `path = r\"C:\\Users\\kiran\\Desktop\\eds217\\chromedriver.exe\"`   \n",
    "            *** Mac users don't need to begin the string w/ r'__'. Only had to do this using Windows because of the direction of backslashes making it a different type\n",
    "3) Import WebDriver using `from selenium import webdriver`\n",
    "4) Initialize WebDriver: `driver = webdriver.Chrome(path)` \n",
    "\n",
    "**Note: make sure to use the path on your own local machine**\n",
    "\n",
    "*Check out the troubleshooting tips at the bottom of this notebook if you run into any errors during the installation and setup*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial\n",
    "\n",
    "After you've successfully installed chromedriver and imported selenium and webdriver into your environment, you're ready to get started with some browser automation! Follow the steps below to see how you can open a browser window, access html elements to click or search, and quit out of the browser. \n",
    "\n",
    "### Import\n",
    "To run some of the functions you'll need to import some classes to make your code run more smoothly.\n",
    "\n",
    "**Import the By class to tell selenium which html or xpath elements to find**\n",
    "`from selenium.webdriver.common.by import By`\n",
    "\n",
    "**Import the Keys class to tell selenium what to input or search, such as search keywords, passwords, or login info**\n",
    "`from selenium.webdriver.common.keys import Keys`\n",
    "\n",
    "**Import the WebDriverWait, expected_conditions, and time classes so that you can make sure Selenium executes in the right order and doesn't error out because it runs faster than the browser**\n",
    "`from selenium.webdriver.support.ui import WebDriverWait`\n",
    "`from selenium.webdriver.support import expected_conditions as EC`\n",
    "`import time`\n",
    "\n",
    "Run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open an automated web browser window\n",
    "\n",
    "Now that you've imported everything you need, you're ready to open a browser window with selnium. \n",
    "Run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#access whatever web page you want by pasting the URL in quotes\n",
    "driver.get(\"https://www.dataone.org/\")\n",
    "\n",
    "#Check to see what page your on by printing the title of the webpage\n",
    "print(driver.title)\n",
    "\n",
    "#now access the search bar by class\n",
    "search = driver.find_element(By.ID, \"search-term\")\n",
    "search.send_keys(\"pisaster ochraceus\")\n",
    "search.send_keys(Keys.RETURN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Access elements\n",
    "\n",
    "You're ready to start interacting with the html elements on the web page. use the By class to interact with html elements by ID, Name, or class. If possible, use ID to select a single element becuase it is gauranteed to be a unique identifier on an html page.\n",
    "\n",
    "The code below shows you how you can open a browser window, and search for papers. In this example, we're looking for sea stars! But feel free to use any keywords you want.\n",
    "\n",
    "Use the try, finally structure to time our your code so that selenium waits ten seconds (enough time for the web page to load) before looking for the elements and looping through them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    results = WebDriverWait(driver, 10).until(\n",
    "        EC.presence_of_element_located((By.ID, \"results\"))\n",
    "    )\n",
    "\n",
    "    papers = results.find_elements(By.TAG_NAME, \"cite\")\n",
    "    for paper in papers:\n",
    "        title = papers.find_elements(By.CLASS_NAME, \"title\")\n",
    "        print(title)\n",
    "finally:\n",
    "    driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was great but what if you want to do more?? Check out our *advanced* tutorial in this repo: \n",
    "\n",
    "Search and take screenshots by running the code in: [search-frogs](/search-frogs.ipynb)\n",
    "\n",
    "Search and save your results to a csv file by running the code in: [storing-frogs](/storing-frogs.ipynb)\n",
    "\n",
    "When running the code in the notebook, make sure that the driver path at the top is directing to where you installed chromedriver on your computer. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Troubleshooting\n",
    "When using Selenium for the first time, we encountered some common issues to be aware of.\n",
    "\n",
    "### Pip issues\n",
    "Make sure to install selenium into the same environment that you're working in or your project environment. If you install it in your base environment on your local machine you may get an error. \n",
    "\n",
    "### Driver verification\n",
    "When running:\n",
    "`from selenium import webdriver`\n",
    "`driver = webdriver.Chrome(PATH)`\n",
    "you may get an error that says you cannot open or run the driver because the source cannot be verified. Or something along the lines of:\n",
    "\n",
    "<img src=\"./error_msg.png\" alt=\"error message\" width=\"350\" class=\"middle\"/>\n",
    "\n",
    "To resolve this issue, run the following command in the the terminal to remove the restrictions:\n",
    "`xattr -d com.apple.quarantine chromedriver`\n",
    "\n",
    "Make sure to run this from wihtin the same directory where you saved chromedriver.\n",
    "\n",
    "### Web issues\n",
    "Because websites are constantly being updated and changed, and your script depends on the arrangement and elements in the website you are interacting with, your code could break. This makes reproducibility tougher, but thoroughly commenting your code can help fixing your code esier.\n",
    "\n",
    "### Output errors\n",
    "You may find it helpful to set your code to run headless so that no visual updates are output. To do this, run:\n",
    "`options = Options()`\n",
    "`options.headless = True`\n",
    "`options.add_argument(\"--window-size=1920,1200\")`\n",
    "`driver = webdriver.Chrome(options=options, executable_path=PATH)`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More resources\n",
    "\n",
    "- About Selenium: https://www.selenium.dev/about/\n",
    "- More on Selenium: https://pypi.org/project/selenium/ \n",
    "- Selenium Github: https://github.com/SeleniumHQ/Selenium \n",
    "- Tech with Tim Tutorials: https://www.youtube.com/watch?v=Xjv1sY630Uc&list=PLzMcBGfZo4-n40rB1XaJ0ak1bemvlqumQ \n",
    "- Website tutorials: https://www.scrapingbee.com/blog/selenium-python/ \n",
    "- For general browser automation info and other links: https://github.com/angrykoala/awesome-browser-automation \n",
    "- Challenges of using Selenium for web automation: https://www.browserstack.com/guide/top-limitations-of-selenium-automation \n",
    "- Webdriver examples: https://www.lambdatest.com/blog/selenium-webdriver-tutorial-with-examples/\n",
    "- List of webdriver capabilities (all browsers):​​ https://www.selenium.dev/documentation/webdriver/capabilities/shared/  \n",
    "- Bot tutorial: https://www.lambdatest.com/blog/automated-web-bot-with-selenium-python/ \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.5 ('eds-217')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.5"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "477af6cd1d51814771407b700a15141f197761cd17758f872ce5e89f60648a59"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}