Crawler Changelog

master

[Improved] Varies small fixes and improvements

v1.0.0 [2017-08-31]

[Added] Pause / resume / stop Crawler
[Improved] Varies small fixes and improvements

v0.4.0 [2017-08-28]

[Added] :scraper option to allow scraping content
[Improved] Varies small fixes and improvements

v0.3.1 [2017-08-28]

[Improved] Crawler.Store.DB now stores the opts meta data
[Improved] Code documentation
[Improved] Varies small fixes and improvements

v0.3.0 [2017-08-27]

[Added] :retrier option to allow custom fetch retrying logic
[Added] :url_filter option to allow custom url filtering logic
[Improved] Parser is now more stable and skips unparsable files
[Improved] Varies small fixes and improvements

v0.2.0 [2017-08-21]

[Added] :workers option
[Added] :interval option
[Added] :timeout option
[Added] :user_agent option
[Added] :save_to option
[Added] :assets option
[Added] :parser option to allow custom parsing logic
[Improved] Renamed :max_levels to :max_depths
[Improved] Varies small fixes and improvements

v0.1.0 [2017-07-30]

[Added] A semi-functioning prototype
[Added] Finished the very basic crawling function
[Added] :max_levels option