YGL Aggregator

Go server + website + Python scraper designed to manage YGL listing links received from brokers and notify when new listings are added.

Run with Docker

• For now, the only option is to build from source

docker compose up -d

Develop

Setup

• Install system deps

Debian-based systems (Ubuntu)

sudo apt-get build-dep python3-lxml

RHEL-based systems (Fedora)

sudo dnf in python3-devel libxml2-devel libxslt-devel

• Install python dependencies

Tip

It's recommended to use a virtual environment

python -m venv .venv
source ./.venv/bin/activate # run `deactivate` to turn off the venv, or just `exit` the terminal
pip install -r ./scraper/requirements.txt

OPTIONAL

• Setup your .env file

cp .env-example .env

then fill your own values into .env

The .env file allows you to adjust the domain/port of the server

Run

1. Start the server

Important

This initializes the DB that the scraper needs to exist

go run .

Note

The server will immediately run the scraper with notifications off on startup.

2. Run the scraper manually (in a seperate terminal)

Important

Make sure you activate your venv first if you set one up

./scraper/main.py --db ./ygl.db