Web Scraper Implementation - Margarita Surina

[surina-margarita]

Description

A command-line web scraper built in Rust that downloads and processes web pages concurrently. The tool follows links up to a specified depth and organizes downloaded content in a hierarchical directory structure.

Features

• Concurrent Processing: Uses async/await with configurable worker threads
• Depth-Limited Crawling: Controls crawling depth to avoid infinite loops
• Same-Domain Filtering: Only follows links within the starting domain
• Organized Storage: Saves pages in depth-based directory structure
• CLI Interface: Easy-to-use command-line interface with help system

Usage

# Install the tool
cargo install --path .

# Basic usage
webcrawl https://example.com

# Advanced usage
webcrawl --output ./crawled_data --depth 3 --workers 6 https://example.com