Skip to content

feat: Connection pooling and keep-alive #56

@Snider

Description

@Snider

Summary

Reuse HTTP connections for efficiency when collecting from same domain.

Use Case

Collecting 1000 files from same domain - should reuse connections instead of TCP handshake each time.

Implementation

  • HTTP/1.1 keep-alive
  • HTTP/2 multiplexing
  • Connection pool per domain

Configuration

connections:
  max_per_host: 6        # Max connections per domain
  max_idle: 100          # Max idle connections total
  idle_timeout: 90s      # Close idle connections after
  
http2:
  enabled: true
  max_streams: 100       # Max concurrent streams per conn

Commands

# Adjust pool size
borg collect website https://example.com --max-connections 10

# Disable keep-alive (for debugging)
borg collect website https://example.com --no-keepalive

# Force HTTP/1.1
borg collect website https://example.com --http1

Benefits

  • Faster collection (no repeated handshakes)
  • Lower server load
  • Better rate limit compliance

Acceptance Criteria

  • Connection pooling per domain
  • HTTP/2 support with multiplexing
  • Configurable pool sizes
  • Idle connection cleanup
  • Connection reuse metrics

Metadata

Metadata

Assignees

No one assigned

    Labels

    julesFor Jules AI to work on

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions