Multithreaded Java Web Crawler

A high-performance, concurrent web crawler built in Java. This project demonstrates core Object-Oriented Programming principles, Java concurrency, and HTML parsing using the Jsoup library.

Features

• Multithreading: Utilizes Java's Runnable interface and Thread instances to crawl multiple web pages simultaneously, significantly speeding up the data extraction process.
• Thread-Safe Memory Management: Implements a ConcurrentHashMap backed Set to track visited URLs, preventing race conditions and duplicate processing across multiple bot threads.
• Recursive Depth Control: Features a configurable MAX_DEPTH limit to prevent infinite crawling loops and stack overflow errors.
• DOM Parsing: Leverages Jsoup to connect to URLs, handle HTTP responses, and extract specific HTML elements (like <a> tags and document titles).

Prerequisites

• Java Development Kit (JDK) 8 or higher
• Jsoup Library (Added to your IDE classpath, or via Maven/Gradle)

Project Structure

• mtWebCrawler (Package)
  • WebCrawler.java: The core engine implementing the Runnable interface. It handles network requests, tracks shared visited links safely across threads, and recursively parses DOM trees for nested links.
  • Main.java (or equivalent entry point): Initializes the target seed URLs and spawns the WebCrawler threads.

Getting Started

1. Clone the repository.
2. Ensure the Jsoup .jar file is added to your project's build path (or add the dependency to your pom.xml if using Maven).
3. Compile the mtWebCrawler package.
4. Run your main class to start the bot threads.

// Example usage:
WebCrawler bot1 = new WebCrawler("[https://example.com](https://example.com)", 1);
WebCrawler bot2 = new WebCrawler("[https://news.ycombinator.com](https://news.ycombinator.com)", 2);