This project was created because I needed to buy a bed and wanted to learn Go.
DBA (Den blå avis) is the largest danish intermediary portal for buying and selling between private individuals. It does not expose a public API though, so it makes it difficult to create anything for tracking new offers etc. I decided to create a simple web scraper that converts the query data into a JSON object, that can be accessed via a Web API.
A simple Web API was written in Go, usin Gin HTTP web framework and Colly - Go scraper. A single request makes the server scrap all the pages related to the query on DBA, and return the extracted data as a JSON. Also a Client using the API was created, but it's not a part of this repo and is not publicly available.
You need to have docker installed. Once that's done add an entry mapping your loopback address to scraper.docker to your host file (or use any of the available proxy servers to do that). Then docker-compose up the hell out of it. The root url is scraper.docker/api, simply append the path you'd use within DBA website and get a beautiful JSON returned. Example: you can get the data from https://dba.dk/biler/biler at http://scraper.docker/api/biler/biler. Enjoy!
- Go
- Gin
- Colly
Created: August 2020