Merge branch 'forward'

Author: julien-duponchelle

README.md

@@ -18,6 +18,7 @@ It's a fundation layer for building your own proxy-based tools.
 - **Certificate generation**: Dynamically generate and sign certificates for intercepted HTTPS traffic
 - **Request/Response interception**: Modify, log, or block HTTP(S) traffic in real-time
 - **Lightweight**: Pure Python implementation with only cryptography as a direct external dependency
+- **Optional implementation of forward proxy**: The implementation to forward HTTP request can be replace in order to use different HTTP client implementation
 
 ## Use Cases
 

docs/advanced-usage.md

@@ -0,0 +1,75 @@
+# Advanced Usage
+
+This guide covers advanced usage of asyncio-https-proxy using the lower-level `HTTPSProxyHandler` for complete control over request handling and forwarding logic.
+
+## When to Use HTTPSProxyHandler
+
+Use the base `HTTPSProxyHandler` when you need:
+
+- **Complete control** over request forwarding logic
+- **Custom HTTP client implementations** (using httpx, aiohttp, curl_cffi, ...)
+- **Fine-grained error handling** and retry logic
+
+For most use cases, [HTTPSForwardProxyHandler](reference/https_forward_proxy_handler.md) with automatic forwarding is recommended. See the [Getting Started](getting-started.md) guide.
+
+## HTTPSProxyHandler Overview
+
+The `HTTPSProxyHandler` is the base class that provides:
+
+- Request parsing and lifecycle hooks
+- Response writing utilities
+- Request body reading capabilities
+- **No automatic forwarding** - you implement all logic yourself
+
+## Basic HTTPSProxyHandler Example
+
+Here's a complete example using `HTTPSProxyHandler` with httpx for request forwarding:
+
+```python title="base_usage.py"
+--8<-- "examples/base_usage.py"
+```
+
+
+## Handler Lifecycle Methods
+
+The `HTTPSProxyHandler` provides these lifecycle hooks:
+
+### `on_client_connected()`
+Called when a client connects and sends a request. This is where you implement your main request handling logic.
+
+### `on_request_received()` 
+Called after the request is fully parsed. Use for logging or request inspection.
+
+### Helper Methods
+
+- `self.read_request_body()` - Async generator for request body chunks
+- `self.write_response(data)` - Write response data to client
+- `self.flush_response()` - Flush response data to client
+
+## Error Handling Best Practices
+
+```python
+async def on_client_connected(self):
+    try:
+        await self._handle_request()
+    except httpx.ConnectTimeout:
+        await self._send_error(504, "Gateway Timeout")
+    except httpx.ConnectError:
+        await self._send_error(502, "Bad Gateway") 
+    except Exception as e:
+        print(f"Unexpected error: {e}")
+        await self._send_error(500, "Internal Server Error")
+
+async def _send_error(self, code: int, message: str):
+    """Send standardized error response."""
+    body = f"Proxy Error: {code} {message}".encode()
+    response = (
+        f"HTTP/1.1 {code} {message}\\r\\n"
+        f"Content-Type: text/plain\\r\\n"
+        f"Content-Length: {len(body)}\\r\\n"
+        f"\\r\\n"
+    ).encode() + body
+    
+    self.write_response(response)
+    await self.flush_response()
+```

docs/getting-started.md

@@ -12,7 +12,7 @@ asyncio-https-proxy is an embeddable, asyncio-based HTTPS forward proxy server t
 : The proxy can intercept HTTPS traffic by acting as a man-in-the-middle. It generates certificates on-the-fly using its own Certificate Authority (CA).
 
 **Handler-Based Architecture**
-: You implement a custom handler class that extends [HTTPSProxyHandler](reference/https_proxy_handler.md) to define how requests and responses are processed.
+: You implement a custom handler class. For most use cases, extend [HTTPSForwardProxyHandler](reference/https_forward_proxy_handler.md) which provides automatic request forwarding. For advanced control, use the lower-level [HTTPSProxyHandler](advanced-usage.md).
 
 **Asyncio Native**
 : Built using Python's asyncio framework for high-performance, non-blocking operations.
@@ -36,74 +36,63 @@ pip install asyncio-https-proxy
 
 ## Quick Start
 
-Here's a complete working example that creates a basic HTTPS proxy server that use HTTPX to forward requests.
+Here's a complete working example that creates a basic HTTPS proxy server with automatic request forwarding and response analysis.
 
 **Step 1: Create the Handler**
 
-First, create a custom handler that extends [HTTPSProxyHandler](reference/https_proxy_handler.md):
+First, create a custom handler that extends [HTTPSForwardProxyHandler](reference/https_forward_proxy_handler.md):
 
 ```python
 import asyncio
-import httpx
-from asyncio_https_proxy import start_proxy_server, HTTPSProxyHandler, TLSStore
+from asyncio_https_proxy import start_proxy_server, HTTPSForwardProxyHandler, TLSStore
 
 
-class BasicProxyHandler(HTTPSProxyHandler):
-    """A basic proxy handler that forwards requests and logs activity."""
+class LoggingProxyHandler(HTTPSForwardProxyHandler):
+    """A proxy handler that automatically forwards requests and logs activity."""
+    
+    def __init__(self):
+        super().__init__()
+        self.response_size = 0
 
-    async def on_client_connected(self):
-        """Called when a client connects to the proxy."""
-        print(f"Client connected: {self.request.method} {self.request.url()}")
-
     async def on_request_received(self):
         """Called when a complete request has been received from the client."""
-        # Log request headers
-        print("Request Headers:")
+        print(f"📤 Request: {self.request.method} {self.request.url()}")
+        print("   Request Headers:")
         for key, value in self.request.headers:
-            print(f"  {key}: {value}")
+            print(f"     {key}: {value}")
+        
+        # Reset response size counter
+        self.response_size = 0
 
-        # Forward the request to the target server
-        await self._forward_request()
+        await self.forward_http_request()
+
+    async def on_response_received(self):
+        """Called when response headers are received from the upstream server."""
+        if self.response:
+            print(f"📥 Response: {self.response.status_code} {self.response.reason_phrase}")
+            print("   Response Headers:")
+            if self.response.headers:
+                for key, value in self.response.headers:
+                    print(f"     {key}: {value}")
 
-    async def _forward_request(self):
-        """Forward the request to the target server and relay the response."""
-        # Create HTTP client for forwarding requests
-        async with httpx.AsyncClient() as client:
-            # Forward the request with all original headers and body
-            async with client.stream(
-                method=self.request.method,
-                url=self.request.url(),
-                headers=self.request.headers.to_dict(),
-                content=self.read_request_body(),  # Stream request body
-            ) as response:
-                print(f"Response: {response.status_code} {response.reason_phrase}")
-                
-                # Send response status line
-                self.write_response(
-                    f"HTTP/1.1 {response.status_code} {response.reason_phrase}\\r\\n".encode()
-                )
-                
-                # Forward response headers
-                for key, value in response.headers.items():
-                    self.write_response(f"{key}: {value}\\r\\n".encode())
-                self.write_response(b"\\r\\n")
-                
-                # Stream response body
-                async for chunk in response.aiter_bytes():
-                    self.write_response(chunk)                        
+    async def on_response_chunk(self, chunk: bytes) -> bytes:
+        """Called for each chunk of response data."""
+        self.response_size += len(chunk)
+        # Could modify content here if needed
+        return chunk
+    
+    async def on_response_complete(self):
+        """Called when response forwarding is complete."""
+        print(f"✅ Response complete: {self.response_size} bytes")
+        print("---")
 ```
 
-The handler is where you implement the custom logic by overriding the `on_` methods.
-
-By default no forwarding capacity is provided you need to implement your own.
-This give you full control on the behaviors.
-
+**Key Benefits of HTTPSForwardProxyHandler:**
 
-```python
-self.write_response(chunk)
-```
-
-Send back to the browser/HTTP client the response.
+- **Automatic Forwarding**: No need to implement HTTP client logic yourself
+- **Zero External Dependencies**: Uses only Python's built-in asyncio and ssl modules  
+- **Streaming Support**: Efficiently handles large responses and chunked encoding
+- **Response Processing Hooks**: Intercept and analyze response content in real-time
 
 **Step 2: Start the Proxy Server**
 
@@ -118,32 +107,32 @@ async def main():
     port = 8888
 
     print(f"Starting HTTPS proxy on {host}:{port}")
-    print("\\nThe proxy will intercept both HTTP and HTTPS traffic.")
+    print("\nThe proxy will intercept both HTTP and HTTPS traffic.")
     print("For HTTPS, it generates certificates on-the-fly using a built-in CA.")
 
     # Initialize TLS store (creates CA certificate automatically)
     tls_store = TLSStore()
-    print(f"\\nGenerated CA certificate. Clients may show security warnings.")
+    print(f"\nGenerated CA certificate. Clients may show security warnings.")
     print("Use --insecure with curl")
 
     # Start the proxy server
     server = await start_proxy_server(
-        handler_builder=lambda: BasicProxyHandler(),
+        handler_builder=lambda: LoggingProxyHandler(),
         host=host,
         port=port,
         tls_store=tls_store,
     )
 
-    print(f"\\nProxy server started. Test with:")
+    print(f"\nProxy server started. Test with:")
     print(f"  curl --insecure --proxy http://{host}:{port} https://httpbin.org/get")
-    print("\\nPress Ctrl+C to stop...")
+    print("\nPress Ctrl+C to stop...")
 
     # Run the server
     async with server:
         try:
             await server.serve_forever()
         except KeyboardInterrupt:
-            print("\\nShutting down proxy server...")
+            print("\nShutting down proxy server...")
         finally:
             server.close()
             await server.wait_closed()
@@ -165,12 +154,17 @@ curl --proxy http://127.0.0.1:8888 http://httpbin.org/get
 Expected output shows the JSON response from httpbin.org, and your proxy will log:
 
 ```text
-Client connected: GET http://httpbin.org/get
-Request Headers:
-  Host: httpbin.org
-  User-Agent: curl/7.68.0
-  Accept: */*
-Response: 200 OK
+📤 Request: GET http://httpbin.org/get
+   Request Headers:
+     Host: httpbin.org
+     User-Agent: curl/7.68.0
+     Accept: */*
+📥 Response: 200 OK
+   Response Headers:
+     Content-Type: application/json
+     Content-Length: 312
+✅ Response complete: 312 bytes
+---
 ```
 
 **Test HTTPS Requests**
@@ -187,16 +181,48 @@ Configure your browser to use `127.0.0.1:8888` as an HTTP proxy. You'll need to
 
 ## Understanding the Handler Lifecycle
 
-The `HTTPSProxyHandler` has a well-defined lifecycle:
+The `HTTPSForwardProxyHandler` has a well-defined lifecycle with automatic forwarding:
+
+1. **Client Connection**: When a client connects, the request is automatically parsed
+2. **Request Processing**: `on_request_received()` is called, then automatic forwarding begins
+3. **Response Headers**: `on_response_received()` is called when response headers arrive
+4. **Response Body**: `on_response_chunk()` is called for each piece of response data
+5. **Response Complete**: `on_response_complete()` is called when forwarding finishes
+
+This makes it much easier to get started compared to the lower-level [HTTPSProxyHandler](advanced-usage.md).
+
+## Common Customizations
 
-1. **Client Connection**: When a client connects, `on_client_connected()` is called
-2. **Request Parsing**: The server parses the HTTP request to [HTTPRequest](reference/http_request.md) and assigns it to `self.request`
-3. **Request Processing**: `on_request_received()` is called with the complete request
-4. **Response Generation**: Your handler processes the request and writes the response
-5. **Connection Cleanup**: The connection is automatically cleaned up
+### Content Analysis
 
-See the [HTTPSProxyHandler](reference/https_proxy_handler.md) for more details on available methods and attributes.
-And [HTTPRequest](reference/http_request.md) for request structure.
+```python
+async def on_response_chunk(self, chunk: bytes) -> bytes:
+    # Analyze content in real-time
+    if b"error" in chunk.lower():
+        print("⚠️ Error detected in response")
+    return chunk
+```
+
+### Response content Modification
+
+```python
+async def on_response_chunk(self, chunk: bytes) -> bytes:
+    # Replace text on the fly
+    return chunk.replace(b"old_text", b"new_text")
+```
+
+### Request Blocking
+
+```python
+async def on_request_received(self):
+    # Block requests to certain domains
+    if "blocked.com" in self.request.host:
+        self.write_response(b"HTTP/1.1 403 Forbidden\r\n\r\nBlocked")
+        await self.flush_response()
+        return  # Don't forward the request
+    
+    await self.forward_http_request()
+```
 
 ## Next Steps
 
@@ -208,4 +234,6 @@ Now that you have a working proxy, you can:
 - Build web scraping or testing tools
 - Create security analysis tools
 
-For more advanced usage, see the [API Reference](reference/index.md).
+For more advanced usage requiring complete control over request forwarding, see [Advanced Usage](advanced-usage.md).
+
+For detailed API documentation, see the [API Reference](reference/index.md).

docs/index.md

@@ -18,6 +18,8 @@ It's a fundation layer for building your own proxy-based tools.
 - **Certificate generation**: Dynamically generate and sign certificates for intercepted HTTPS traffic
 - **Request/Response interception**: Modify, log, or block HTTP(S) traffic in real-time
 - **Lightweight**: Pure Python implementation with only cryptography as a direct external dependency
+- **Optional implementation of forward proxy**: The implementation to forward HTTP request can be replace in order to use different HTTP client implementation
+
 
 ## Use Cases
 

docs/reference/http_response.md

@@ -0,0 +1,3 @@
+# HTTPResponse
+
+::: asyncio_https_proxy.HTTPResponse

docs/reference/https_forward_proxy_handler.md

@@ -0,0 +1,3 @@
+# HTTPSForwardProxyHandler
+
+::: asyncio_https_proxy.HTTPSForwardProxyHandler

docs/reference/index.md

@@ -9,6 +9,8 @@ This section contains the complete API documentation for asyncio-https-proxy.
 ## Classes
 
 - [HTTPSProxyHandler](https_proxy_handler.md) - Base handler class for processing requests
+- [HTTPSForwardProxyHandler](https_forward_proxy_handler.md) - Built-in forward proxy handler with automatic request forwarding
 - [HTTPRequest](http_request.md) - HTTP request representation
+- [HTTPResponse](http_response.md) - HTTP response representation
 - [HTTPHeader](http_header.md) - HTTP header collection
 - [TLSStore](tls_store.md) - TLS certificate management

examples/basic_usage.py renamed to examples/custom_http_client.py

@@ -1,13 +1,16 @@
 #!/usr/bin/env python3
 """
-Basic usage example for asyncio-https-proxy.
+Basic usage example for asyncio-https-proxy using HTTPX as
+the HTTP client to forward requests
 
 This example shows how to set up a simple HTTPS proxy server.
 """
 
 import asyncio
+
 import httpx
-from asyncio_https_proxy import start_proxy_server, HTTPSProxyHandler, TLSStore
+
+from asyncio_https_proxy import HTTPSProxyHandler, TLSStore, start_proxy_server
 
 
 class BasicProxyHandler(HTTPSProxyHandler):