Glen Eira Bin Time Crawler

This project extracts kerbside collection schedules and supporting waste service information for Glen Eira City Council.

What the crawler returns

The Glen Eira adapter located in internal/infrastructure/crawling/gleneira/ produces a single council.CrawlResult containing:

• residential: Regions, contact phone and reference links for household services sourced from Residential_Bin_Collection.json.
• commercial: Commercial collection regions with their scheduling metadata sourced from Commercial_Bin_Collection_Regions.json.
• binInformation: Council-wide service guidance compiled from Glen Eira's public website. This section currently includes:
  • generalWaste: Red lid general waste bin collection frequency, colour, accepted/avoid lists, and reference URL.
  • recycling: Yellow lid recycling bin guidance, contamination reminder, accepted/avoid lists, and reference URLs.
  • greenCompost: Lime-green lid green compost bin weekly service summary with accepted/avoid lists and source URL.
  • hardRubbishService: Booking summary, contact phone and link for hard rubbish, bundled branches and bulk cardboard collections.

Each section contains descriptive text tailored to downstream presentation layers, with binInformation values derived from:

• https://www.gleneira.vic.gov.au/services/rubbish-and-recycling/rubbish-or-recyclable-our-a-z-guide/general-waste-bin-red-lid
• https://www.gleneira.vic.gov.au/services/rubbish-and-recycling/rubbish-or-recyclable-our-a-z-guide/recycling-bin-yellow-lid
• https://www.gleneira.vic.gov.au/about-council/news/latest-news/recycle-right-don-t-put-recycling-in-plastic-bags
• https://www.gleneira.vic.gov.au/services/rubbish-and-recycling/rubbish-or-recyclable-our-a-z-guide/green-compost-bin-lime-green-lid
• https://www.gleneira.vic.gov.au/services/rubbish-and-recycling/hard-rubbish-branches-and-cardboard

Project structure

• cmd/crawler/ – CLI entry point that wires dependencies and runs crawl modes.
• internal/application/ – Service layer that orchestrates crawlers and persistence.
• internal/domain/ – Core contracts (crawling.Crawler, council.CrawlResult, repositories).
• internal/infrastructure/crawling/gleneira/ – Glen Eira datasets, GeoJSON helpers, and payload builders.
• internal/infrastructure/crawling/bininfo/ – HTML fetcher that normalises Glen Eira guidance into reusable payload sections.
• internal/infrastructure/crawling/registry/ – Shared catalogue of dataset endpoints and bin info links per council.
• internal/infrastructure/logging/ – Structured logging abstraction.
• internal/infrastructure/persistence/ – Filesystem-backed repository writing crawl outputs to output/.

Tech stack

• Go 1.25.1.
• Standard library net/http client with custom council-specific headers and InfoHawker crawler user agent.
• encoding/json for dataset decoding.
• golang.org/x/net/html v0.44.0 for HTML tokenisation when extracting bin information.
• Custom validation in internal/infrastructure/validation to enforce dataset schemas.
• Structured logging and filesystem persistence adapters under internal/infrastructure/.

Running the crawler

Execute the CLI to crawl Glen Eira:

pwsh -Command "go run ./cmd/crawler --council glen-eira"

Optional flags (see cmd/crawler/main.go) allow targeting a specific latitude/longitude.

Verifying the payload

1. Run the crawler as above.
2. Inspect the generated JSON under output/.
3. Confirm that payload.binInformation exists and contains the bin sections listed above alongside existing residential and commercial data.