A scraper retrieving apartment data from apartments.com

Table of Contents

• Introduction
• Getting Started

Introduction

This tool scrapes selected data from apartments.com platform (https://www.apartments.com/) into the mysql database. It loads the zipcode list from the database, search for apartments within the zipcode range and stores the data back to the database. From each page of the apartments.com, we get the link to the details of each household and we scrape more information about that household using the link. Provided codes also supports tranferring the credentials of database connection from aws s3 bucket and dockerizing the application.

Getting Started

Please follow these steps before running the code for the first time.

1. Install Python 3
2. Install all the packages listed in packages.txt. If you have pip installed, you could run pip install -r packages.txt to install all the packages.
3. Replace the function content of create_database_engine so that it matches your database connection info.
4. Run the program by python3 main.py.