This tool scrapes selected data from apartments.com platform (https://www.apartments.com/) into the mysql database. It loads the zipcode list from the database, search for apartments within the zipcode range and stores the data back to the database. From each page of the apartments.com, we get the link to the details of each household and we scrape more information about that household using the link. Provided codes also supports tranferring the credentials of database connection from aws s3 bucket and dockerizing the application.
Please follow these steps before running the code for the first time.
- Install Python 3
- Install all the packages listed in packages.txt. If you have pip installed, you could run
pip install -r packages.txtto install all the packages. - Replace the function content of
create_database_engineso that it matches your database connection info. - Run the program by
python3 main.py.