feat: Configurable rate limiting per domain

[Snider]

Summary

Fine-grained rate limiting with per-domain configuration.

Use Case

Different sites have different tolerance. GitHub API has strict limits, static sites can handle more.

Commands

# Global rate limit
borg collect website https://example.com --rate-limit 2/s

# Per-domain config file
borg collect website https://example.com --rate-config .borg-rates.yaml

Configuration

# .borg-rates.yaml
defaults:
  requests_per_second: 1
  burst: 5

domains:
  api.github.com:
    requests_per_second: 0.5  # 1 req per 2 seconds
    burst: 1
    
  bitcointalk.org:
    requests_per_second: 0.2  # 1 req per 5 seconds
    burst: 1
    reason: "aggressive anti-scraping"
    
  eprint.iacr.org:
    requests_per_second: 2
    burst: 10
    
  "*.archive.org":
    requests_per_second: 1
    burst: 3

Token Bucket Algorithm

• requests_per_second: sustained rate
• burst: max concurrent before throttling

CLI Options

Flag	Description
--rate-limit N/s	Requests per second
--rate-limit N/m	Requests per minute
--burst N	Burst allowance
--rate-config	Config file path

Automatic Detection

• Parse Retry-After headers
• Detect 429 responses
• Adjust rate dynamically

Acceptance Criteria

• Token bucket rate limiter
• Per-domain configuration
• Wildcard domain matching
• Dynamic adjustment on 429
• Retry-After header support

[google-labs-jules]

Jules is on it. When finished, you will see another comment and be able to review a PR.

[google-labs-jules]

Ready for a review! A PR has been created.